I had a side project the last two weeks: Make the build service more fun to use.

No matter how much fun you have creating packages, if they don’t build, there is little point in using a Service that has Build in its name, no? So one of the major goals of the service is actually to help those that want to build packages as good as possible. But there is a problem:

Let me quote from the landing page of build.opensuse.org: “The openSUSE Build Service hosts 16,414 projects, with 107,691 packages, in 26,259 repositories and is used by 25,967 confirmed users.”. That are quite some high numbers – especially in the relation to the ~25 servers we have for actually building.

If you look at the build statistics of the last month (and this is just i586, x86_64 has around the same), you notice that there is not much purple in the “Busy workers / Idle workers” graphic:

Every 2nd weekend or so we have some pause where the servers actually idle around, the rest of the time they are usually under full load and it’s not exceptional that we have over 20,000 build jobs for said 25 servers at the same time. So if Sue comes wants to build her packages at that time, she competes with quite some other packages and she gets frustrated to still see “scheduled” when she goes away. So we use some algorithms in the so called dispatcher to distribute the power to the right packages.

Over years of its existence the dispatcher used the algorithm I would dub “Randomness with exceptions” – it would if the job’s filename matches a regexp and if so, preferred it, otherwise picked a random job. Such algorithms create some fairness if you have 28.000 users all active at the same time, because there is usually not a really good balance between those.

But with 2.0 this changed: we got load and priorities. A script of mine parses the logs of download.opensuse.org and counts how many users are for the repositories. From that I calculate priorities, so that repositories of interest to people get more build power than others. E.g. KDE:Release:45 for 11.3 is downloaded 65 more often than for 11.1, so the 11.3 packages should see more attention. For that the build service calculates how many workers were busy for the repositories and then allows a factor to lower that load while picking the next repository to build for. This is much more complex than “pick a random one”, but it lead to faster return times for those projects that see actual downloads (and new projects as they have no load registered). To give some fun for those actually working on their packages, we also lower the registered load if we see commits.

But there was one problem left: with so many projects registered you also have quite some that aren’t interesting at all. They are not downloaded and very often not even their maintainers care, e.g. for some testing subproject they created in 2009 and then forgot about it. But those repositories build against the often changing openSUSE:Factory and see rebuilds because of that. Those repositories had a very low load because they have few packages, so they are often preferred over projects that have a lot of packages.

To free up more power to recent changes, we now experimented with various ways. It turned out to be useful to look at the relation between since last source change in the project and time since the jobs are “scheduled”. From this we calculate a staleness penalty – when the job is freshly scheduled, it’s basically one chance for a worker for 2 months since the last commit to the project. But this chance rises quickly, the penalty gets smaller the longer the job is in scheduled. Even those projects that have no source commits are valid “customers” and deserve to be rebuild against the latest gcc from openSUSE:Factory.

So what does this mean to you as user of the OpenSUSE Build Service?

• Don’t add repositories unless you really plan to use it. I know that clicking the checkbox to also add all SLE versions is easily done, but remember this is build power and disk space on various mirrors you’ll be using
• If you really care for your project, it’s a good idea to fix build failures from time to time. You’ll get more build power in return
• If you plan to do a larger update of some package in your project and want to test the resulting packages building against it, it’s a good idea to disable all other repositories while you do so. The fewer build jobs you create, the lower is your load, the higher are your chances to get more build power
• Make your repository popular by telling the world about it. More users means more build power
• And last but not least: don’t get frustrated, remember there are almost 26000 other users

If you look at the list of binaries for a package (e.g. icecream), you may think that you can download the RPM right away – but if you follow the link in a browser you get to see details about the rpm.

Now if you only want to download it, you may already know the details and don’t care. So I added a little shortcut: if you request the binary url with a client not accepting html explicitly (e.g. curl, wget…), you get the file directly. Just copy & paste the link to your console and be done.

And due to the joy of rails, it’s just a couple of lines and now I get:


--2010-07-09 13:51:31-- https://build.opensuse.org/stage/package/binary?arch=i586&filename=icecream-0.9.5-11.1.i586.rpm&package=icecream&project=home%3Acoolo&repository=openSUSE_11.3
Resolving build.opensuse.org... 195.135.221.34
Connecting to build.opensuse.org|195.135.221.34|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://download.opensuse.org/repositories/home:/coolo/openSUSE_11.3/i586/icecream-0.9.5-11.1.i586.rpm [following]
--2010-07-09 13:51:32-- http://download.opensuse.org/repositories/home:/coolo/openSUSE_11.3/i586/icecream-0.9.5-11.1.i586.rpm
Resolving download.opensuse.org... 195.135.221.130
Connecting to download.opensuse.org|195.135.221.130|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://widehat.opensuse.org/repositories/home:/coolo/openSUSE_11.3/i586/icecream-0.9.5-11.1.i586.rpm [following]
--2010-07-09 13:51:32-- http://widehat.opensuse.org/repositories/home:/coolo/openSUSE_11.3/i586/icecream-0.9.5-11.1.i586.rpm
Resolving widehat.opensuse.org... 62.146.92.202, 2a01:138:a004:0:21a:a0ff:fe26:efa9
Connecting to widehat.opensuse.org|62.146.92.202|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 184745 (180K) [application/x-rpm]
Saving to: `icecream-0.9.5-11.1.i586.rpm'


The build service (and any other of openSUSE infrastructure software using RoR) is using rails 2.3.5, because we once decided to harmonize on the version of SLE11 SP1. Of course the latest version has less bugs (usually), but mixing RoR versions between different developers and deployment is a nightmare, so we had to decide on one.

Now comes the catch: 11.3 has rails 2.3.8 and as such you can’t develop the build service on factory/11.3 as is. But the good news, openSUSE:Tools has all the right versions, so you can add the repo (zypper ar -r http://r.opensu.se/openSUSE:Tools/f/r) and then install zypper in -f rubygem-rack-1.0.1 rubygem-activesupport-2_3-2.3.5

To make sure, the next zypper dup is not going to take it away, use zypper al rubygem-rack rubygem-activesupport-2_3

Part of our “umbrella” milestone, Pavol and Robert ported software.opensuse.org to the Bento theme. To get more feedback for it, I now deployed it as http://software.opensuse.org/stage.

Please note two things:

• it also includes a new feature from the Education project: a link to openSUSE derivates
• the language box is experimental and we kind of decided already to kill it again

On top of that of course: the Bento theme is not yet finished – it’s only a stage deployment to get feedback.

Have fun!