The last two weeks saw two improvements on speeding up the building of packages in the openSUSE Build Service: An ultra-fast scheduler and a binary cache for the worker. Both changes on its own should speed up the server that allocates jobs to the client – and the binary cache improves also the clients since they don’t have to download every package for every build.

New Scheduler

The role of the scheduler is to get informed about package build changes, about package build changes, e.g. source changes or some build is finished and succeeded and then decide which packages need to be rebuilt because of those changes.

Michael Schröder integrated the SAT solver code into the build service scheduler.  The SAT solver code is used in our package management as well and helps with fast decision making and reduces memory footprint as well.

Parts of the scheduler itself are more than 100 times faster which results in some operations being more than 10 times faster now.  We have almost no queue of high or medium priority events anymore, so you should not see anymore a process for ages in the state finished.
However, this is pretty much new code, please report behavior changes or problems. Changed parts are:

• package dependency expansion calculation
• package ordering
• meta file generation
• replacement of :full.cache of package meta data (perl structured) with solv files

Binary Caches for Build Clients

We have all the built packages on a central server and so far each client downloaded for each build all packages from the central server.  The server coped with it but still this produces a base load and also congests our network.  Michael has now implemented a binary cache on the worker.  In our setup we have on each client machine a couple of build worker (each in their own Xen instance) running and these worker share now a common cache of built packages.

I have no data on how much improvement it brings overall but it does show on a reduced load on the server – so that it can schedule faster – and also on far less trafic on the network.

Planned Build Improvements

Michael and Coolo consider improving the dispatcher in the near future.  The dispatcher basically decides which job runs next on which machine once.  This is at the moment a random selection of all available “scheduled” packages with some heuristics.  They plan on improving the heuristics to prioritize certain types of jobs.

If you’re interested in helping in any ways to improve the openSUSE Build Service, please join the opensuse-buildservice mailing list.

Thanks to Michael for implemting these – and for a review of this article.

Both comments and pings are currently closed.

• Tags:No tags available
• Category: Build Service