Home Home
Sign up | Login

Author Archive

GSOC Idea: Improve Clic Filesystem

March 24th, 2011 by

I really look for the perfect gsoc student: someone who has a brilliant idea, noone else thought of how to improve linux in general or openSUSE in specific and just needs my guidance.

But for those that need some ideas, my best bet is really to rewrite clicfs to make it even better. Right now clicfs is a FUSE filesystem hosting a loop image. So a page read goes from ext4->loop->fuse->clicfs->CD->clicfs->fuse->loop->ext4.
If it lived in the kernel, it could offer a block device you mount your ext4 on, leaving aside loop and fuse:


modprobe clicfs packfile=/read-only/openSUSE-kde.i586 cowfile=/read-write/.COWfile cowlog=/dev/ttyS0
mount /dev/clicfs/0 /mnt
chroot /mnt /bin/init

Great idea, no? For someone who wants to become famous, it would be the perfect target!

How we use our power

December 16th, 2010 by

I had a side project the last two weeks: Make the build service more fun to use.

No matter how much fun you have creating packages, if they don’t build, there is little point in using a Service that has Build in its name, no? So one of the major goals of the service is actually to help those that want to build packages as good as possible. But there is a problem:

Let me quote from the landing page of build.opensuse.org: “The openSUSE Build Service hosts 16,414 projects, with 107,691 packages, in 26,259 repositories and is used by 25,967 confirmed users.”. That are quite some high numbers – especially in the relation to the ~25 servers we have for actually building.

If you look at the build statistics of the last month (and this is just i586, x86_64 has around the same), you notice that there is not much purple in the “Busy workers / Idle workers” graphic:

OBS load

Every 2nd weekend or so we have some pause where the servers actually idle around, the rest of the time they are usually under full load and it’s not exceptional that we have over 20,000 build jobs for said 25 servers at the same time. So if Sue comes wants to build her packages at that time, she competes with quite some other packages and she gets frustrated to still see “scheduled” when she goes away. So we use some algorithms in the so called dispatcher to distribute the power to the right packages.

Over years of its existence the dispatcher used the algorithm I would dub “Randomness with exceptions” – it would if the job’s filename matches a regexp and if so, preferred it, otherwise picked a random job. Such algorithms create some fairness if you have 28.000 users all active at the same time, because there is usually not a really good balance between those.

But with 2.0 this changed: we got load and priorities. A script of mine parses the logs of download.opensuse.org and counts how many users are for the repositories. From that I calculate priorities, so that repositories of interest to people get more build power than others. E.g. KDE:Release:45 for 11.3 is downloaded 65 more often than for 11.1, so the 11.3 packages should see more attention. For that the build service calculates how many workers were busy for the repositories and then allows a factor to lower that load while picking the next repository to build for. This is much more complex than “pick a random one”, but it lead to faster return times for those projects that see actual downloads (and new projects as they have no load registered). To give some fun for those actually working on their packages, we also lower the registered load if we see commits.

But there was one problem left: with so many projects registered you also have quite some that aren’t interesting at all. They are not downloaded and very often not even their maintainers care, e.g. for some testing subproject they created in 2009 and then forgot about it. But those repositories build against the often changing openSUSE:Factory and see rebuilds because of that. Those repositories had a very low load because they have few packages, so they are often preferred over projects that have a lot of packages.

To free up more power to recent changes, we now experimented with various ways. It turned out to be useful to look at the relation between since last source change in the project and time since the jobs are “scheduled”. From this we calculate a staleness penalty – when the job is freshly scheduled, it’s basically one chance for a worker for 2 months since the last commit to the project. But this chance rises quickly, the penalty gets smaller the longer the job is in scheduled. Even those projects that have no source commits are valid “customers” and deserve to be rebuild against the latest gcc from openSUSE:Factory.

So what does this mean to you as user of the OpenSUSE Build Service?

  • Don’t add repositories unless you really plan to use it. I know that clicking the checkbox to also add all SLE versions is easily done, but remember this is build power and disk space on various mirrors you’ll be using
  • If you really care for your project, it’s a good idea to fix build failures from time to time. You’ll get more build power in return
  • If you plan to do a larger update of some package in your project and want to test the resulting packages building against it, it’s a good idea to disable all other repositories while you do so. The fewer build jobs you create, the lower is your load, the higher are your chances to get more build power
  • Make your repository popular by telling the world about it. More users means more build power
  • And last but not least: don’t get frustrated, remember there are almost 26000 other users

git bisect in action

July 19th, 2010 by

We have a little regression in the buildservice API test suite. And looking at the commit log I didn’t find the obvious problem, so I tried bisect, because I know it worked very recently:


git bisect start master 1dfd9717348d8425492b556d3a183b4d265007d
git bisect run sh -c "cd src/api; ruby test/functional/build_controller_test.rb --name=test_project_index"

And it gave me:
25347c2d712e6e4b598e05442dd95b5e82f14aed is the first bad commit
commit 25347c2d712e6e4b598e05442dd95b5e82f14aed
Author: Jan-Simon Möller
Date: Sun Jul 18 01:14:01 2010 +0200

[api] Add fixtures for request controller ACL tests.

Now if you look at the commit, it is not obvious why it’s a problem. But now that git has proven, it has to be buggy I looked at the other fixture data and then it’s obvious and the fix is simple:


--- a/src/api/test/fixtures/package_user_role_relationships.yml
+++ b/src/api/test/fixtures/package_user_role_relationships.yml
@@ -11,8 +11,8 @@ testpack_maintainer:
bs_user_id: 267
role_id: 3
-testpack_maintainer:
- db_package_id: 10095
+testpack1_maintainer:
+ db_package_id: 11006
bs_user_id: 42
role_id: 3

Thanks git :)

Shortcut for the package download

July 9th, 2010 by

If you look at the list of binaries for a package (e.g. icecream), you may think that you can download the RPM right away – but if you follow the link in a browser you get to see details about the rpm.

Now if you only want to download it, you may already know the details and don’t care. So I added a little shortcut: if you request the binary url with a client not accepting html explicitly (e.g. curl, wget…), you get the file directly. Just copy & paste the link to your console and be done.

And due to the joy of rails, it’s just a couple of lines and now I get:


--2010-07-09 13:51:31-- https://build.opensuse.org/stage/package/binary?arch=i586&filename=icecream-0.9.5-11.1.i586.rpm&package=icecream&project=home%3Acoolo&repository=openSUSE_11.3
Resolving build.opensuse.org... 195.135.221.34
Connecting to build.opensuse.org|195.135.221.34|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://download.opensuse.org/repositories/home:/coolo/openSUSE_11.3/i586/icecream-0.9.5-11.1.i586.rpm [following]
--2010-07-09 13:51:32-- http://download.opensuse.org/repositories/home:/coolo/openSUSE_11.3/i586/icecream-0.9.5-11.1.i586.rpm
Resolving download.opensuse.org... 195.135.221.130
Connecting to download.opensuse.org|195.135.221.130|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://widehat.opensuse.org/repositories/home:/coolo/openSUSE_11.3/i586/icecream-0.9.5-11.1.i586.rpm [following]
--2010-07-09 13:51:32-- http://widehat.opensuse.org/repositories/home:/coolo/openSUSE_11.3/i586/icecream-0.9.5-11.1.i586.rpm
Resolving widehat.opensuse.org... 62.146.92.202, 2a01:138:a004:0:21a:a0ff:fe26:efa9
Connecting to widehat.opensuse.org|62.146.92.202|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 184745 (180K) [application/x-rpm]
Saving to: `icecream-0.9.5-11.1.i586.rpm'

Buildservice development on 11.3

June 26th, 2010 by

The build service (and any other of openSUSE infrastructure software using RoR) is using rails 2.3.5, because we once decided to harmonize on the version of SLE11 SP1. Of course the latest version has less bugs (usually), but mixing RoR versions between different developers and deployment is a nightmare, so we had to decide on one.

Now comes the catch: 11.3 has rails 2.3.8 and as such you can’t develop the build service on factory/11.3 as is. But the good news, openSUSE:Tools has all the right versions, so you can add the repo (zypper ar -r http://r.opensu.se/openSUSE:Tools/f/r) and then install zypper in -f rubygem-rack-1.0.1 rubygem-activesupport-2_3-2.3.5

To make sure, the next zypper dup is not going to take it away, use zypper al rubygem-rack rubygem-activesupport-2_3

What people search for

April 23rd, 2010 by

http://software.opensuse.org/stage includes now a little box to show what other people search for. It was a little idea The Boosters had, but I found the results so far pretty suprising. I didn’t really expect subversion in the top 3, but perhaps it’s only the moment.

The results shown are from actual downloads, for that every download url is a little redirect to save the query the download was for.

http://software.opensuse.org/stage

March 3rd, 2010 by

Part of our “umbrella” milestone, Pavol and Robert ported software.opensuse.org to the Bento theme. To get more feedback for it, I now deployed it as http://software.opensuse.org/stage.

Please note two things:

  • it also includes a new feature from the Education project: a link to openSUSE derivates
  • the language box is experimental and we kind of decided already to kill it again

On top of that of course: the Bento theme is not yet finished – it’s only a stage deployment to get feedback.

Have fun!

About Patterns versus Packages

October 7th, 2009 by

Just a quick note to everyone using factory and wonder what patterns-openSUSE-kde4_basis is all about: our patterns install packages now.

(more…)

Cliced Hybrids

August 13th, 2009 by

Marcus and me have been busy improving the USB experience. So here is how to use it:

download the live cd of your choice from http://download.opensuse.org/factory/iso/ and make sure you grab build218 or later. And then do something like I did:

linux-vgqb:~ # ls -l /dev/disk/by-id/*usb*
lrwxrwxrwx 1 root root 9 13. Aug 10:04 /dev/disk/by-id/usb-Kingston_DataTraveler_II+_5B751D8C1994-0:0 -> ../../sdb

So I use /dev/sdb in my commands to shorten it, normally I would use /dev/disk/by-id/.. directly to avoid overwriting my hard disk image.
(more…)

-fomit-all-instructions

May 27th, 2009 by

Now that milestone2 is almost done, we get into the next round and we decided (basically during coffee break) to try something and change the compile flags for our i586 distribution (which is still the major one for openSUSE). If there aren’t major problems coming up with it, the next milestone will be compiled with -fomit-frame-pointer -mtune=generic instead of the old -march=i586 -mtune=i586.

It will feel around 20% faster, I’m sure :)

On unrelated news: I’m away for a very long Pentecost weekend. If there are issues with Factory, I won’t be able to fix them. See you in June!