Home Home > 2008 > 12 > 16
Sign up | Login

Deprecation notice: openSUSE Lizards user blog platform is deprecated, and will remain read only for the time being. Learn more...

Archive for December 16th, 2008

Comments on Phoronix Benchmarking openSUSE 11.1

December 16th, 2008 by

Phoronix has run some tests comparing the openSUSE 11.1 release candidate (RC1), Ubuntu 8.10, Fedora 10 and Mandriva 2009.0 on Intel Atom.

We have looked at the results and they are not good for openSUSE 11.1. I’ve talked with a few engineers and want to present below our first analysis.

While the benchmarks were done on a specific hardware, they might be relevant for other hardware as well.

Note that the numbers I cite below are not benchmark numbers comparable to the one Phoronix measured, they are measured on totally different machines by different engineers and not all are done as real benchmarks.  But they show some of the problems.


Best Way to Download openSUSE

December 16th, 2008 by

For most people, downloading traditionally looks like this:

  1. looking at a traditional, more or less static mirror list, and picking a mirror 🙁
  2. trying the mirror and see that it is too slow, outdated, or not reachable 🙁
  3. looking at the mirror list again, and picking another mirror 🙁
  4. downloading with a web browser or FTP program
  5. restarting a failed download, after loosing network connection for some reason 🙁
  6. ditching the download because it never finishes, starting from scratch from another mirror 🙁
  7. finally having a completed download, but for some reason it doesn’t install… 🙁
  8. finding the MD5 sum and manually verifing the download 🙁
  9. finding it broken and don’t know whether to start from scratch, repair the download with rsync, … 🙁
  10. scratching head… and be frustrated 🙁

Manually proceeding like pictured above is no longer needed, nowadays. At least not with openSUSE.

All you need is a Metalink client. This is a wonderful technology that fixes all the above issues, and makes downloading “just work”. A Wikipedia article explains how that is achieved.

The openSUSE download server fully supports this technology, by using MirrorBrain. Mirrorbrain is a download redirector and metalink generator which is open source and supports all advanced Metalink features. Features as embedding of Torrent links, verification hashes, cryptographical signatures and transparent negotiation, so that no separate links are needed on our web sites. Most of these features were added during the course of 2008.

There is a number of Metalink client programs out there. There is a FireFox extension called DownThemAll which works in FireFox on all platforms. There is aria2, a commandline program which is the most powerful of all of them. Our wiki has a list with more clients. I tend to recommend aria2, because it is the most powerful one. It is very simple to use, nevertheless.

aria2 deserves special notice, because it has the full support for all goodies that one might think of. These include:

  • downloading from several mirrors at the same time (so it also makes you faster) 🙂
  • automatically noticing mirror problems, and resuming from other mirrors 🙂
  • simultaneously downloading via Peer-to-Peer (BitTorrent) 🙂
  • error checking for transferred data is not only done in the end – but already during downloading. Each part of the file which has arrived is already checked, and if it’s found to be broken, it is scheduled to be refetched from another mirror. 🙂
  • creating a local *.asc file which contains the cryptographical signature which can be used to verify the authenticity of the file 🙂
  • automatically noticing if a server supports metalinks (if not, it will just act as “normal” download client)
  • being robust against all sorts of network failures 🙂
  • avoiding head-scratching of its user 🙂

Both aria2 and MirrorBrain are “location aware”, and work together to select mirrors that are as close to you as possible. In addition, mirrors known to be more powerful are assigned more users.

What else do you need to know? Not much. The command that you run to download an image is as simple as:

aria2c http://download.opensuse.org/distribution/11.1/iso/openSUSE-11.1-DVD-i586.iso

(For some other clients, you need to append “.metalink” to the URL.)

Note, aria2 tries to maximize utilization of your Internet connection for download bandwidth. This is wanted for most people, but it may be unwanted if you want to use the connection for other work, or if you are in a company with shared Internet access. In that case, use acia2’s -C command line option to limit the number of simultaneous servers being used.

Special note for Torrent users: you don’t need to bother downloading Torrent files. Aria2 does this automatically… since the Torrent link is embedded in the Metalink!

If you want to see what the magic behind all this is, look at http://download.opensuse.org/distribution/11.1/iso/openSUSE-11.1-DVD-i586.iso.metalink with an editor. You’ll see an XML file containing everything that the Metalink client needs. This file transfers the knowledge of the download server (and mirror database) to the client. With this knowledge, the client is enabled to work its way to a successful download even under adverse circumstances. In contrast, a traditional HTTP redirect to a mirror does convey only extremely minimal information – one link to one server, and there is no provision in the HTTP protocol to handle failures, or to add checksums that make problems detectable. An Internet Draft documents the Metalinks.

Many thanks to Tatsuhiro Tsujikawa for aria2!

This technology would be even easier to use, when web browsers would implement native support for it. Let’s hope that we will see that in the future. The technical challenges are solved and the way is paved. Ask your favourite browser vendor for it today…!

And since this is so powerful, we intend to employ it for other downloads as well — those done by the openSUSE package management tool, YaST respectively zypper. A prototype for this is available in openSUSE 11.1. Please test it – it is enabled by installing aria2 and setting ZYPP_ARIA2C=1 in the environment.

Fate Internal, Up- and Downstream

December 16th, 2008 by

motivated by Aaron’s blog post More downstream fun I was thinking about how Fate could be a more important part of infrastructure in the Linux landscape. Fate is now an important part of the Novell/SUSE infrastructure and we are currently in the process to open it up for the openSUSE community. But could Fate also be useful for upstream integration? To let you participate in the discussion I think I should start with some explanations what Fate is and in which environment we are with it.

Fate is a system developed at SUSE over the last few years to track features and requirements for Novell Linux products.  The term “feature” is already is a topic for scientific papers, but how we understand a feature is a functionality  that is not yet in the product but required or wanted. It references future products, in most cases more than one such as SLE and openSUSE.

Fate Feature Tracking EnvironmentThe little sketch illustrates the dilemma in which we are when it comes to product planning. Basically it is all about one  thing: decision taking. Decisions have to be taken about the new functionality  that goes into a product and the tasks internal people work on. This is based on the decision how the product should look alike from a high level point of view. To make a solid decision about the high level product it needs to be clear what we are actually able to put into the product at a given time. That is only a part of what is really going on but let’s leave it with that for simplicity.

You see that lots of the base information which is needed to make
good decisions comes from different people: Product managers have a strong idea of how products should be, the technical project manager knows about dates and technical possibilities and can plan with the engineering managers how that can be achieved with the given amount of people in a given time frame. Technical feasibility is worked out with the developers as they’re the experts. The colored arrows try to visualize the communication ways, different colors mark different topics.

Since we work with the communities we have more input of information: The user community tells what is needed and the upstream communities announce what they plan to do by when.

The part with the internal decision taking is very much based on Fate in the Linux part of Novell and that is working fine. Features come in by a requester and all involved parts can give their thoughts in  a discussion forum. The key functionalities add their priority for the feature and finally PM and TPM come to a decision. Features with a high priority have to make it into the product. Engineering managers can assign developers and they can mark features as finished. All the processes are covered by a set of configurable rules. The Fate system is integrated into other infrastructure parts. There are several clients for different needs, the most mature is the KDE fat client.

What is not yet optimal are two things: There is no good way yet for the user community to community their wishes for upcoming products. We are facing that with opening up a new web based openSUSE Fate client soon. That will involve the user community not only in testing and using the product but already in its planing in a defined way.

A more tricky part is how to involve the upstream communities. It would be great for a Product Manager of a Linux distribution to see the feature plans for upcoming releases of big upstream projects, maybe somehow integrated into the Fate for his product. Would the Fate model as described here in a nutshell suitable for upstream projects?

For example, could KDE or GNOME make use of parts of the process for them internally and provide a structured interface to the downstream parties? If so, that could add a lot of transparency. Transparency is the precondition for flexibility and trust and as a result for better collaboration which would benefit all.

I hope that helps for a basic understanding and would love to hear your opinion. I promise to come up with more information about the Fate system and improve my drawing skills 😉

openSUSE Mirrorlist Improved

December 16th, 2008 by

Peter Pöml has created a dynamic list of our mirrors with accurate information about which repositories they mirror.  You can reach the link via the openSUSE wiki.  The data in the mirror list is used by our download redirector download.opensuse.org to point users automatically to mirrors close to them.

Btw. if anybody else wants to mirror openSUSE, please read these Instructions about it.  We love to increase our mirror infrastructure and there are some regions that could use additional mirrors.  Even if you only mirror parts of the openSUSE tree, you’re welcome!

Thanks a lot to all the mirror admins that mirror openSUSE – and to Peter for his continous improvements to the infrastructure!