For most people, downloading traditionally looks like this:
- looking at a traditional, more or less static mirror list, and picking a mirror 🙁
- trying the mirror and see that it is too slow, outdated, or not reachable 🙁
- looking at the mirror list again, and picking another mirror 🙁
- downloading with a web browser or FTP program
- restarting a failed download, after loosing network connection for some reason 🙁
- ditching the download because it never finishes, starting from scratch from another mirror 🙁
- finally having a completed download, but for some reason it doesn’t install… 🙁
- finding the MD5 sum and manually verifing the download 🙁
- finding it broken and don’t know whether to start from scratch, repair the download with rsync, … 🙁
- scratching head… and be frustrated 🙁
Manually proceeding like pictured above is no longer needed, nowadays. At least not with openSUSE.
All you need is a Metalink client. This is a wonderful technology that fixes all the above issues, and makes downloading “just work”. A Wikipedia article explains how that is achieved.
The openSUSE download server fully supports this technology, by using MirrorBrain. Mirrorbrain is a download redirector and metalink generator which is open source and supports all advanced Metalink features. Features as embedding of Torrent links, verification hashes, cryptographical signatures and transparent negotiation, so that no separate links are needed on our web sites. Most of these features were added during the course of 2008.
There is a number of Metalink client programs out there. There is a FireFox extension called DownThemAll which works in FireFox on all platforms. There is aria2, a commandline program which is the most powerful of all of them. Our wiki has a list with more clients. I tend to recommend aria2, because it is the most powerful one. It is very simple to use, nevertheless.
aria2 deserves special notice, because it has the full support for all goodies that one might think of. These include:
- downloading from several mirrors at the same time (so it also makes you faster) 🙂
- automatically noticing mirror problems, and resuming from other mirrors 🙂
- simultaneously downloading via Peer-to-Peer (BitTorrent) 🙂
- error checking for transferred data is not only done in the end – but already during downloading. Each part of the file which has arrived is already checked, and if it’s found to be broken, it is scheduled to be refetched from another mirror. 🙂
- creating a local *.asc file which contains the cryptographical signature which can be used to verify the authenticity of the file 🙂
- automatically noticing if a server supports metalinks (if not, it will just act as “normal” download client)
- being robust against all sorts of network failures 🙂
- avoiding head-scratching of its user 🙂
Both aria2 and MirrorBrain are “location aware”, and work together to select mirrors that are as close to you as possible. In addition, mirrors known to be more powerful are assigned more users.
What else do you need to know? Not much. The command that you run to download an image is as simple as:
aria2c http://download.opensuse.org/distribution/11.1/iso/openSUSE-11.1-DVD-i586.iso
(For some other clients, you need to append “.metalink” to the URL.)
Note, aria2 tries to maximize utilization of your Internet connection for download bandwidth. This is wanted for most people, but it may be unwanted if you want to use the connection for other work, or if you are in a company with shared Internet access. In that case, use acia2’s -C command line option to limit the number of simultaneous servers being used.
Special note for Torrent users: you don’t need to bother downloading Torrent files. Aria2 does this automatically… since the Torrent link is embedded in the Metalink!
If you want to see what the magic behind all this is, look at http://download.opensuse.org/distribution/11.1/iso/openSUSE-11.1-DVD-i586.iso.metalink with an editor. You’ll see an XML file containing everything that the Metalink client needs. This file transfers the knowledge of the download server (and mirror database) to the client. With this knowledge, the client is enabled to work its way to a successful download even under adverse circumstances. In contrast, a traditional HTTP redirect to a mirror does convey only extremely minimal information – one link to one server, and there is no provision in the HTTP protocol to handle failures, or to add checksums that make problems detectable. An Internet Draft documents the Metalinks.
Many thanks to Tatsuhiro Tsujikawa for aria2!
This technology would be even easier to use, when web browsers would implement native support for it. Let’s hope that we will see that in the future. The technical challenges are solved and the way is paved. Ask your favourite browser vendor for it today…!
And since this is so powerful, we intend to employ it for other downloads as well — those done by the openSUSE package management tool, YaST respectively zypper. A prototype for this is available in openSUSE 11.1. Please test it – it is enabled by installing aria2 and setting ZYPP_ARIA2C=1 in the environment.
