Home Home > Build-service
Sign up | Login

Archive for the ‘Build Service’ Category

The issues with contributing to projects only once

June 4th, 2017 by

I work to improve the openSUSE Tumbleweed (GNU/)Linux distribution. Specifically I make sure that all packages can be built twice on different hosts and still produce identical results, which has multiple benefits. This generates a lot of patches in a single week.

Sometimes it is enough to adjust the .spec file – that is a small text file usually specific to us. Then it is straight-forward

  1. osc bco
  3. optional: spec_add_patch $MY.patch $SOME.spec
  4. edit *.spec
  5. osc build
  6. osc vc
  7. osc ci
  8. osc sr

And OBS will even auto-clean the branch when the submit-request is accepted. And it has a ‘tasks’ page to see and track SRs in various stages. For the spec_add_patch to work, you need to do once
ln -s /usr/lib/build/spec_add_patch /usr/local/bin/

When you want to contribute patches upstream, so that other distributions benefit from your improvements as well, then you first need to find out, where they collaborate. A good starting point is the URL field in the .spec file, but a google search for ‘contribute $PROJECT’ often is better.

Then there are those many projects hosted on github, where it is also pretty low effort, because I already have the account and it even remains signed in. But some repos on github are only read-only mirrors.

  1. check pull requests, if some have been merged recently
  2. fork the project
  3. git clone git@github.com:…
  4. cd $REPO
  5. edit $FILES
  6. git commit -a
  7. git push
  8. open pull request
  9. maybe have to sign a CLA for the project
  10. When change is accepted, delete fork to not clutter up repository list too much (on github under settings)

The older brother of github. They integrate various ways of contributing. The easiest one is to open a Ticket (Patch or Bug) and attach the .patch you want them to merge with a good description. While many developers do not have the time and energy to debug every bug you file, applying patches is much easier, so gets your issue fixed with a higher chance.

devel Mailinglist
Some projects collaborate mainly through their development MLs, then I need to

  1. subscribe
  2. confirm the subscription
  3. git format-patch origin/master
  4. git send-email –to $FOO-devel@… –from $MYSUBSCRIBEDEMAIL 000*.patch
  5. wait for replies
  6. if it is a high-volume ML, also add an IMAP folder and an entry to .procmailrc
  7. unsubscribe
  8. confirm

project bugtracker
Like https://bugzilla.gnome.org/ https://bugs.python.org/ https://bugs.ruby-lang.org/ https://bz.apache.org/bugzilla/

  1. create unique email addr
  2. sign up for account
  3. add info to my account list
  4. optional: search for existing bug (90% of the time there is none)
  5. file bug
  6. attach patch

So as you can see there is a wide range of ways. And most of them have some initial effort that you would only have once… But then I only contribute once per project, so I always pay that.

Thus, please make it easy for people to contribute one simple fix.

Reducing YaST rebuild time by 30%

October 11th, 2016 by

Here comes the YaST team again trying to flood your aggregator with blog post! Now it’s time for the story of how we reduced the critical path of the rebuild time of YaST RPM packages from 42min 2s to 29min 40s.

Chapter 1: where to optimize

Of course, the first step to start fixing something is finding out what exactly needs to be fixed. In our case, this breaks down into

  1. knowing the dependencies, and
  2. knowing the individual build times.


It is tempting to figure out the dependencies by yourself, by parsing the spec files. But it is hard to do right, and, more importantly, a reinvention of the wheel. The Build Service must know all this to be able to schedule the builds, and provides a convenient way to access it, with osc dependson:

$ osc dependson YaST:Head openSUSE_Factory x86_64
yast2-x11 :
yast2-xml :
yast2-ycp-ui-bindings :

Individual build times

For each source package, the Build Service produces not only binary RPMs but also a _statistics file, available in the web UI or via osc getbinaries. We were interested in the total build time, although the data was of limited use because packages can be built on machines with vastly different power and this information is not included.

Chapter 2: how to optimize

Once we knew which screws needed to be tightened, it was time to do it. Fortunately we had more than one tool for the job.

Stop using Autotools

Autotools (automake, autoconf and configure) took up a majority of the time needed for building YaST packages. Now that most of those packages are written in pure Ruby, we don’t need autotools there checking for portability problems that we don’t have. Autotools are a leftover from the times 15 years back when they were the only sensible option. We have wiped them out where possible and have been switching to our own set of Rake tasks.

Stub the APIs used in tests

We run a mixture of unit and integration tests at RPM build time. The downside of this is that we pull in many of the run time dependencies. Fortunately Ruby is a dynamic language and makes it easy to replace interfaces by stubs. That enables us to cut those dependencies.

In fact, we also have some Perl code, notably in yast2-users. Although the stubbing techniques across languages are messier than with pure ruby, they are still effective for our purposes.

Do not build specialized documentation

This one is simple: if the development documentation is only useful for people that will check out the git repo anyway, then leave it out from the RPM.

Appendix: the details

Enough of high-level explanations, we we promised you graphs, code and all kind of gory details, and a promise made is a debt unpaid. So there we go.

Dependency graphs

A picture is worth a thousand words. That’s why we came up with this small tool to generate a graphical representation of the dependencies of the YaST packages. In the resulting graphs displayed below, a node is a source package in the build service, and an arrow means “needs for its build”. Redundant arrows are omitted (that is, we’ve erased an A→C if both A→B and B→C existed).

We can see that the most prominent conclusion is that there is a large number of packages that depend on yast2, a collection of basic libraries.

But on top of that, in the original graph there are 6 more layers, and the graph is not very dense there. After our fixes, there are only 4 layers that are more dense.

Is worth mentioning that the “layer” concept only works if the packages take roughly the same time to build; it would not be helpful if there were huge variations. To get a more accurate picture, we should have generated a histogram of build times. But the graph was good enough in our scenario… and we had to stop the analysis at some point. 🙂

The build dependency graph before our fixes:

YaST dependencies graph (before)

The build dependency graph after our fixes:

YaST dependencies graph (after)

Build statistics

If those graph are not geeky enough for you, here you are the detailed build statistics from the build service

      <size unit="M">1118</size>
    <usage>      <size unit="M">580</size> </usage>
    <total>      <time unit="s">756</time> </total>         <!-- THIS -->
    <preinstall> <time unit="s">8</time>   </preinstall>
    <install>    <time unit="s">72</time>  </install>
    <main>       <time unit="s">555</time> </main>
    <download>   <time unit="s">4</time>   </download>
    <size unit="k">33564</size>


This was definitely an interesting journey. We learned quite some things. Specially we learned that there is still room for improvement, but most likely the time reduction will not pay off for the time invested implementing those improvements.

We have to be realistic and keep working in other interesting stuff to fuel the next sprint report, coming next week!

targitter project – about OBS, tars and git

October 5th, 2015 by

In OBS we use source tarballs everywhere to build rpms (and debs) from.

This has at least two major downsides:

  1. Storing all old tar files takes up a lot of disk space
  2. OBS workflows with .tar files and patches are rather different and somewhat disconnected from the git workflows we usually use everywhere else these days. E.g. for the SUSE OpenStack Cloud team we have a “trackupstream” jenkins job, that pulls the latest git version into a tarball once every day.

Fedora already keeps their metadata in git, but only a hash of the tarball.

So as one first step, I used two rather different projects to see how different the space usage would be. On the slow side I used 20 gtk2 tarballs from the last 5 years and on the fast side, I used 31 openstack-nova tarballs from Cloud:OpenStack:Master project from the last 5 months.
I used scripts that uncompressed each tarball, added it to a git repo and used git gc to trigger git’s compression.

Here are the resulting cumulative size graphs:

gtk2 size graph
nova size graph

The raw numbers after 20 tarballs: for nova the ratio is 89772:7344 = 12.2 and for gtk2 the ratio is 296836:53076 = 5.6

What do you think: would it be worth the effort to use more git in our OBS workflows?

Do we care about being able to reproduce the original tarballs? While this is possible, it has some challenges in differing file-ordering, timestamps, file-ownerships and compression levels.
Or would it be enough if OBS converted a tarball into a signed commit (so it cannot be forged without people being able to notice)?

Do you know of a tool that can uncompress tarballs in a way that allows to track the content as single files, and allows to later re-create the original verbatim tarball, such that upstream signatures would still match?

oSC15 – 200, Why not packaging workshop like mini hack sprint

January 17th, 2015 by

welcomeHello Geekos.

I’m contacting you personally, as an openSUSE Board member.

You certainly already know that we want to have a kicking openSUSE conference next 1st-4th May 2015 at the Haag (NL).

Thus I’ve found that creating special workshop organized by development project could foster our beloved distribution.

oSC is the unique case in the year, where Geekos from all around the world meet together.
Let’s imagine you, meeting perhaps for the first time your fellows, having nice discussions, and hacking around the software you maintain.
There’s high level of chance to meet also your end users, and have constructive exchanges.

That’s why I invite you to propose a workshop directly to our event tools:

Having a workshop run like a mini-hack sprint, would help any of us in the distribution and the project.
Be it like learn people how to submit nice package, how to do maintenance, or how to do bug triage.
I feel confident that you will have the creative approach to resolve your own problematic.
The event place has small rooms for unattended sessions and they could be used to extend your workshop to get some more work done.

Some practical aspects:
oSC website : https://events.opensuse.org/conference/oSC15

And soon the travel support program for oSC 15 will be opened to handle your request about getting financial support for going to oSC.

If you have any questions, thoughts or ideas, don’t hesitate to ask on -project mailing list
or ping me by reply.

A final note about the why you should do it? Well beside being one of our “heroes” even if nobody need them 🙂
You and your co-maintainers will be able to explain your “job” on the project.
Don’t you want to inspire new comers, lead them directly to the right direction, and share the load.
Meeting you there, will also help our “marketing” force to light up a bit the work done in the shadow.

I really will appreciate your presence, afterwards, it’s you that create our distribution.
The time has come for you to be warmly thank.

I’m looking forward to see you there.

Standing for Re-Election to the openSUSE Board

January 12th, 2015 by

Hi Fellow Geekos,

This post is a summary of my wish to continue to serve on your behalf on the openSUSE board.

My term has been a short one, as I was appointed to serve out the remainder of Vincent Untz’ term.

While I now work for SUSE (and it has been a fabulous experience), that does not change my view or efforts to contribute to openSUSE.  Prior to joining SUSE, as a part of the Sales Engineering team, I was elected to the board for a two year term. To avoid having too many SUSE employees, I had to step down.  A rule I completely support.

Having been an openSUSE member long before joining SUSE, I think I have a keen awareness of what the community is about and where we can improve together.

My take is we have an awesome group of contributors who want to see the community grow and prosper.  With things like Tumbleweed and OBS, among others, we have shown real innovation and technical leadership in the Linux world. We have arguably one the top distros available.  It is solid, polished and usable for a wide variety of use cases. We need to keep the openness and solid collaboration which enables everyone to participate and succeed.

My efforts within openSUSE have been mostly focused on the Open Build Server, maintaining several projects, as well as, being part of the Factory review team.

Going forward, I want to concentrate on reaching out beyond our community to build more awareness of what an awesome distro we have, along with a pretty friendly community. I see other, less compelling, distros getting more visibility than perhaps is deserved.

Along the same lines, my take is we as a community can take a more active role in bringing in new members, who might not be technical folks, but can help in the marketing and outreach. I’ve started a local SUSE group through meetup.com which is a different way to find new users and contributors.

One other reason I wish to remain on the board, is we have a solid working relationship and there is a lot of mutual respect and good collaboration.

No matter who is elected to the board, I am very pleased with the caliber of the folks running and know the community will be in good hands moving forward.

Thank you in advance for your vote!

Peter Linnell

osc: speedup update of a project working copy

June 26th, 2014 by


recently, I pushed a commit that speeds up the update of an osc project
working copy, if most of the packages in the working copy are already up to
date (that is no update has to be performed).
The following table shows the improvements of the new code (in terms of
wall-clock time). Both project working copies were already up to date
and the packages in the home:Marcus_H project were unexpanded.

project       # number of packages  #    old code # new code
home:Marcus_H                   66        51.135s    10.653s
d:l:r:e                       1245     7:07.07min    17.804s

(the numbers for the devel:languages:ruby:extensions (d:l:r:e) project
were kindly provided by darix – thanks!).

Technically, we just reduced the number of http requests for packages
that are already up to date by using the backend’s getprojectsourceinfo
call (/source/project?view=info&package=pkg_1…&package=pkg_n).
Note: currently, such a reduction is not possible for packages that have
a _service file, because a small change in the backend is needed (see [1]).
Consequently, there are no time improvements for such packages.

If you want to test the new code, use the osc package from the
devel:tools:scm repo (http://download.opensuse.org/repositories/devel:/tools:/scm/).
Feedback is always welcome! 🙂

Next, my plan is to improve the speed of an update of a single package
working copy (again by reducing the number of http requests).

[1] http://lists.opensuse.org/opensuse-buildservice/2014-06/msg00067.html

First fruits – update on openQA and Staging work

February 19th, 2014 by

In our previous summary, we talked about some basic research and some ground work to build on. This time we have some first exciting results!


Last week we rearranged the repository a little bit, creating a new branch called "devel" where all the exciting (and not so exciting) changes are taking place. Our little Factory 😉

The main difference between this branch as master is that, as you could read in the previous blog, the devel branch is openQA build on Mojolicious, a nice web development framework. And having a proper web framework is starting to show its benefits: we have openID login available! Unfortunately the current openSUSE openID provider is a little bit weird, so it doesn’t play well with our tool yet, but some others are working and openSUSE accounts will be the next step. Having working user accounts is necessary to be able to start defining permissions and to make openQA truly multiuser. And to be able to deploy the new version on a public server!

The other main focus of this week has been internal polishing. We have revamped the database layout and changed the way in which the different openQA components communicate with each other. The openQA functionality is spread out over several parts: the workers are responsible of actually executing the tests in virtual machines reporting the result after every execution; some client utilities are used to load new ISO images and similar tasks and, finally, we have the one openQA Server to rule them all. Until now, the communication between the server and the other components was done using JSON-RPC (a lightweight alternative to XML-RPC). We have dropped JSON-RPC in favor of a REST-like API with just JSON over plain HTTP. This change allowed us to implement exactly the same functionality in a way that is simpler, perfectly supported by any web framework, natively spoken by browsers and easier to authenticate (using, for example, plain HTTP authentication or openID). This is also the first step to future integration with other services (think OBS, as the ultimate goal is to use openQA to test staging projects).

But, who tests the tester? openQA is growing and changing quite fast so we have continued with the task of creating a good testing infrastructure to tests openQA itself to make sure that all our changes do not result in breakage. We only have a few tests right now, but we now have a solid foundation to write more and more tests.

Staging and package manipulation

In the last blog post we told you we were investigating a code test suite to test the abilities of a osc plugin we are writing. osc is the commandline tool for handling the Open Build Service, and this plugin is meant to help with the administration of staging projects. We’ve been thinking about how to move forward with the testing part as we want to make sure the functionality works as advertised. More important, we write tests to make sure that our additions and changes do not break existing functionality. We have started merging functionality from various scripts handling staging thingies and rings we had into this plugin. This is partially done so we can do basic staging project tasks! We can take a request (be it submit or delete) and put it to test in a staging project. We can also move packages between staging projects and we have a simple YAML in project descriptions to indicate what packages and what requests are there.

We're all green!
Coolo already started using the plugin for some tasks, so you can see pretty titles and metadata in staging projects descriptions. Not impressive enough? Let me provide you a good headline:

Thanks to staging projects, no single regression have been introduced this year in the core of Factory

You can enjoy a more detailed description in this announcement written by coolo to the Factory mailing list and have some visual joy in the included screenshot (genuine pr0n for QA engineers).

Last but not least, we also did some cleanup of the sources in the repo and of course we added more tests (as functionality grows). And there has been work on other parts of the plugin, like taking rings into account.


We already have some useful functionality which we are going to expand and build on. Not that much to show yet, but we are progressing towards our goal. You can follow our progress either in the way of tasks (openQA and staging projects) or just follow our commit history on github (again for both openQA and staging plugin).

We are very much looking forward to feedback and thoughts – these changes aim to make Factory development easier and thus we need to hear from all you!

Some news from the trenches

February 7th, 2014 by

As you might know, we are focusing our development efforts in two fronts, namely openQA and staging projects. As we just started we don’t have fireworks for you (yet) but we did some solid ground work that we are going to build upon.

Working on openQA

We are organizing our daily job in openQA into highly focused sprints of two weeks. The focus of the first sprint was clear: cleanup the current codebase to empower future development and lower the entry barrier for casual contributors, which can be translated as “cleaning our own mess”. We created some tasks in progress, grouped in a version with a surprising and catchy name: Sprint 01.

Got my mojo workin’

Up to now, openQA web interface was written using just a bunch of custom CGI scripts and some configuration directives in Apache. We missed a convenient way to access to all the bell and whistles of modern web development and some tools to make the code more readable, reliable and easier to extend and test. In short, we missed a proper web development framework. We evaluated the most popular Perl-based alternatives and we finally decided to go for Mojolicious for several reasons:

  • It provides all the functionality we demand from a web framework while being lightweight and unbloated.
  • It’s stated as a “real time framework” which, buzzwords apart, means that is designed from the ground to fully support Comet (long-polling), EventSource and WebSockets. Very handy technologies for implementing some features in openQA.
  • It really “feels” very close to Sinatra, which makes Ruby on Rails developers feel like at home. And we have quite some Rails developers hanging around, don’t we? Just think about OBS, software.o.o, WebYast, progress.o.o, OSEM, the TSP app…
  • Mojolicious motto is “web development can be fun again”. Who could resist to that?

We’ve now reached the end of the sprint and we already have something that looks exactly the same than what we had before, but using Mojolicious internally. We are very happy with the framework and we are pretty confident that future development of openQA will be easier and faster than ever. OpenQA has mojo!

The database layer in openQA

Another part that we worked on during first sprint was the database layer. The user interface part of openQA use a SQLite database to store the jobs and workers registered in the system. The connection between the code and the database was expressed directly in SQL using a simple API.

We have replaced this layer with another equivalent that uses an ORM (Object-relational mapping) in Perl (DBIx::Class). Every data model in openQA is now a true object that can be created, copied and moved between the different layers of the application. Quite handy.

To make sure we don’t forget anything, we created a bunch of tests covering the whole functionality of the original code, running this test suite after each step of the migration. In this way we have achieved two goals: we now have a simple way to share and update information through the whole system and we can migrate very easily to a different database engine (something that we plan to do in the future).

What to do with staging

Over the time, coolo had accumulated quite some scripts that helped him with Factory. Most of them are actually related to something we are doing right now: the staging projects. So in the end we basically migrated all relevant ones to github and one by one we are merging their functionality to staging plugin. We also experimented with test frameworks that we could use to test the plugin itself, selected few and we even have a first test! The final plan is to have the whole plugin functionality covered with a proper test suite, so we will know when something breaks. Currently, there is a lot of mess in our repo and the plugin itself need big cleanup, but we are working on it.

Contributions are welcome

If you want to help but wonder where to start, we identified tasks that are good to dive into the topic and named them “easy hacks”: mostly self contained tasks we expect to have little effort but we lack the time to do right now. Just jump over the list for openQA or staging projects.

For grabbing the code related to staging project, you only need to clone the already mentioned repository. The openQA code is spread in several repositories (one, two, three and four), but setting up your own instance to play and hack is a piece of cake using the packages available in OBS (built automatically for every git push).

If you simply want to see what we are doing in more detail, take a look at progress.o.o, we have both openQA and Staging projects there.

We are having a lot of fun, and we encourage you to join us!

spec-cleaner: hide all your precious cruft!

January 31st, 2014 by

As we stated in our communication over the time, our team’s main focus for foreseeable future is Factory and how to manage all those contributions. Goal is not to increase the number of SRs that is coming to Factory, but to make sure we can process more and to make sure we see even well hidden consequences to make sure that Factory is “stable” and “usable”.


Not really part of our current sprints, but something that will hopefully help us is spec-cleaner that Tomáš Chvátal and Tomáš Čech were working on lately during their free time/hackweek. What is it trying to address? Currently, there are some packaging guidelines, but when you write a spec file for your software, you still have plenty of choices. How do you order all the information in the header? Do you use curly brackets around macros? Do you use macros? Which ones do you use and which not? Do you use binaries in dependencies? Package config? Perl symbols? Package names? There is format_spec_file obs service that tries to unify a little bit the coding style but leaves quite some of the stuff up to you. Not necessarily a bad thing, but if you have to compare changes and review packages that are using completely different coding styles the process becomes harder and slower.

spec-cleaner is format_spec_file taken to another level. It tries to unify coding style as much as it can. It uses consistent conventions, makes most of the decisions mentioned previously for you and if you already decided for one way in the past, it will try to convert your spec file to follow the conventions that it specifies. It’s not enforcing anything, it’s standalone script and therefore you don’t have to be worried that you spec file will be out of your control. You can run it, verify the result (actually, you should verify the results as there might still be some bugs) and commit it to OBS. If we all do it, our packages will all look more alike and it will be easier to read and review them.

How to try it? How to help? Well, code is on GitHub and packages are in OBS. You may have a version of it in your distribution, but that one is heavily outdated (even the 13.1 version), so add openSUSE:Tools repo and try the version from there.

zypper ar -f obs://openSUSE:Tools/openSUSE_13.1 openSUSE-Tools
zypper in spec-cleaner

You can then go to some local checkout and try what changes does it propose for your spec file. Easiest way is to just let it do stuff by calling it and taking a look at changes afterwards.

spec-cleaner -p -i *.spec
osc diff

If it works, great, we will have more unified spec files. If it doesn’t, file a bug 😉

Magic: It is now possible to use MS Silverlight based websites via pipelight

September 13th, 2013 by

It has long been a challenge to use MS Silverlight based websites on linux systems. Especially in The Netherlands this is a big hurdle as many (>80%) of the secondary school websites that pupils must use to communicate with their school (for homework, marks, etc) are equipped with Silverlight. Yes, really… 🙁

Fortunately at the end of August 2013 I discovered pipelight, a very smart idea to use MS Silverlight based website natively on Linux. The problem was however to find a working pipelight package for openSUSE. As there was none, I decided to build one myself using the incredible openSUSE Build Service. It was quite a quest to obtain a working package, but due to very good cooperation with the pipelight developers, I’m now able to present a working pipelight package to the openSUSE community. Oh, and while working on the package I reported a bug via the bug report system, that was solved and published via an rpm package within 1 hour after reporting it (that was during out of office hours). Indeed within 1 hour after reporting the problem it was; accepted, investigated, analysed, fixed, tested, handed over to me, packaged, tested and published! The amazing world of Open Source Software!

Pipelight works okay for the following sites (among many others): arte, LOVEFiLM, Netflix, Magister based NL schoolwebsites, WATCHEVER, etc. View the complete list on the pipelight website.

The installation instructions are on the pipelight website. Be aware though, that pipelight requires the wine package that is provided via the home:rbos:pipelight repository. With any other wine package, pipelight will (very likely) not work. If you rely on your currently installed wine package and installed MS applications and are unsure that the wine package provided via the home:rbos:pipelight repository will leave your currently in use MS applications untouched: don’t install pipelight (or only after making very good backups). You can always start by installing pipelight in a virtual machine.

Have fun with pipelight.