There were several participants from companies, including Microsoft, Huawei and Google.
Also some researchers from universities that work on tools like DetTrace, tuf and in-toto.
But the majority still came from various open-source projects – with Fedora/RedHat being notably absent.

We had many good discussion rounds, one of which spawned my writeup on the goal of reproducible builds

Another session was about our wish to design a nice interface, where people can easily find the reproducibility status of a package in various distributions. I might code a Proof-of-Concept of that in the next weeks (when I have time).
I also got some help with java patches in openSUSE and made several nice upstream reproducibility fixes – showing some others how easy that can be.

This whole event also was good team-building, getting to know each other better. This will allow us to better collaborate in the Future.

Later there will be a larger report compiled by others.

In openSUSE developers do not perform manual builds on some specially crafted machine in their basement and then upload the result somewhere. Instead all sources are stored in a version control system inside the open build service (OBS) instance at build.opensuse.org. OBS then automatically builds the sources including all dependencies according to defined build instructions (eg spec files for rpms). OBS also automatically adds cryptographic signatures to files that support it to make sure nobody can tamper with those files.

The WSL appx files are basically zip files that contain a tarball of a Linux system (like a container) and a Windows exe file, the so called launcher. Building a container is something OBS can already do fully automatic by means of Kiwi. The launcher as well as the final appx however is typically built on a Windows machine using Visual Studio by the developer.

Since the goal of the openSUSE WSL offering is to have the appx files officially and automatically be produced along with other images such as the DVD installer, Live images or containers, the appx files have to be built from sources in OBS.

Fortunately there’s already a MinGW cross toolchain packaged as rpms OBS and a tool to generate appx files on Linux.

Combining that all together OBS can actually build the WSL appx from sources. The current state of development can be found in the Virtualization:WSL project in OBS. The generated appx files are published on download.opensuse.org.
The current images for Leap 15.2 Alpha and Tumbleweed there are good enough for some testing already so please go ahead and do so, feedback welcome!
Note that since the appx files are signed by OBS rather than Microsoft, there are a few steps required to install them.

Going forward there is still quite some work needed to polish this up. Kiwi for example can’t build the appx directly itself but rather the fb-util-for-appx is called by a spec file. That requires some hacks with the OBS project config to work. On Linux side there’s currently no password set for the root user, so we need a better “first boot” solution. More details on that in a later article. Meanwhile, remember to have a lot of fun…

So I made an experimental openSUSE:Factory git mirror to see how well it goes and how using it feels.
The repo currently needs around 1GB but will slowly grow over time. I did not want to spend effort to import all history.

Binary files are replaced by cryptographically secure symlinks into IPFS
and I am currently providing files up to 9MB there.

If you can not run ipfs, you can still get these files through any of the public gateways like this:
curl https://ipfs.io$(readlink packages/a/aubio/aubio-0.4.9.tar.bz2) > OUTPUT

So some benefits are already obvious.
It is now much easier to find and download our patches.
Downloading and seaching all of openSUSE is now much faster.
And it works even on Thursdays (when our maintenance window often causes OBS downtimes).

It is meant to be a read-only mirror, so there is no point in opening pull-requests on github.

I hope, you enjoy it and have a lot of fun…

OBS
Sometimes it is enough to adjust the .spec file – that is a small text file usually specific to us. Then it is straight-forward

1. osc bco
2. cd $PROJECT/$PACKAGE
3. optional: spec_add_patch $MY.patch $SOME.spec
4. edit *.spec
5. osc build
6. osc vc
7. osc ci
8. osc sr

And OBS will even auto-clean the branch when the submit-request is accepted. And it has a ‘tasks’ page to see and track SRs in various stages. For the spec_add_patch to work, you need to do once
ln -s /usr/lib/build/spec_add_patch /usr/local/bin/

When you want to contribute patches upstream, so that other distributions benefit from your improvements as well, then you first need to find out, where they collaborate. A good starting point is the URL field in the .spec file, but a google search for ‘contribute $PROJECT’ often is better.

github
Then there are those many projects hosted on github, where it is also pretty low effort, because I already have the account and it even remains signed in. But some repos on github are only read-only mirrors.

1. check pull requests, if some have been merged recently
2. fork the project
3. git clone git@github.com:…
4. cd $REPO
5. edit $FILES
6. git commit -a
7. git push
8. open pull request
9. maybe have to sign a CLA for the project
10. When change is accepted, delete fork to not clutter up repository list too much (on github under settings)

sourceforge
The older brother of github. They integrate various ways of contributing. The easiest one is to open a Ticket (Patch or Bug) and attach the .patch you want them to merge with a good description. While many developers do not have the time and energy to debug every bug you file, applying patches is much easier, so gets your issue fixed with a higher chance.

devel Mailinglist
Some projects collaborate mainly through their development MLs, then I need to

1. subscribe
2. confirm the subscription
3. git format-patch origin/master
4. git send-email –to $FOO-devel@… –from $MYSUBSCRIBEDEMAIL 000*.patch
5. wait for replies
6. if it is a high-volume ML, also add an IMAP folder and an entry to .procmailrc
7. unsubscribe
8. confirm

project bugtracker
Like https://bugzilla.gnome.org/ https://bugs.python.org/ https://bugs.ruby-lang.org/ https://bz.apache.org/bugzilla/

1. create unique email addr
2. sign up for account
3. add info to my account list
4. optional: search for existing bug (90% of the time there is none)
5. file bug
6. attach patch

So as you can see there is a wide range of ways. And most of them have some initial effort that you would only have once… But then I only contribute once per project, so I always pay that.

Thus, please make it easy for people to contribute one simple fix.

Chapter 1: where to optimize

Of course, the first step to start fixing something is finding out what exactly needs to be fixed. In our case, this breaks down into

1. knowing the dependencies, and
2. knowing the individual build times.

Dependencies

It is tempting to figure out the dependencies by yourself, by parsing the spec files. But it is hard to do right, and, more importantly, a reinvention of the wheel. The Build Service must know all this to be able to schedule the builds, and provides a convenient way to access it, with osc dependson:

$ osc dependson YaST:Head openSUSE_Factory x86_64
[...]
yast2-x11 :
   yast2-devtools
yast2-xml :
   yast2-core
   yast2-devtools
yast2-ycp-ui-bindings :
   libyui
   yast2-core
   yast2-devtools

Individual build times

For each source package, the Build Service produces not only binary RPMs but also a _statistics file, available in the web UI or via osc getbinaries. We were interested in the total build time, although the data was of limited use because packages can be built on machines with vastly different power and this information is not included.

Chapter 2: how to optimize

Once we knew which screws needed to be tightened, it was time to do it. Fortunately we had more than one tool for the job.

Stop using Autotools

Autotools (automake, autoconf and configure) took up a majority of the time needed for building YaST packages. Now that most of those packages are written in pure Ruby, we don’t need autotools there checking for portability problems that we don’t have. Autotools are a leftover from the times 15 years back when they were the only sensible option. We have wiped them out where possible and have been switching to our own set of Rake tasks.

Stub the APIs used in tests

We run a mixture of unit and integration tests at RPM build time. The downside of this is that we pull in many of the run time dependencies. Fortunately Ruby is a dynamic language and makes it easy to replace interfaces by stubs. That enables us to cut those dependencies.

In fact, we also have some Perl code, notably in yast2-users. Although the stubbing techniques across languages are messier than with pure ruby, they are still effective for our purposes.

Do not build specialized documentation

This one is simple: if the development documentation is only useful for people that will check out the git repo anyway, then leave it out from the RPM.

Appendix: the details

Enough of high-level explanations, we we promised you graphs, code and all kind of gory details, and a promise made is a debt unpaid. So there we go.

Dependency graphs

A picture is worth a thousand words. That’s why we came up with this small tool to generate a graphical representation of the dependencies of the YaST packages. In the resulting graphs displayed below, a node is a source package in the build service, and an arrow means “needs for its build”. Redundant arrows are omitted (that is, we’ve erased an A→C if both A→B and B→C existed).

We can see that the most prominent conclusion is that there is a large number of packages that depend on yast2, a collection of basic libraries.

But on top of that, in the original graph there are 6 more layers, and the graph is not very dense there. After our fixes, there are only 4 layers that are more dense.

Is worth mentioning that the “layer” concept only works if the packages take roughly the same time to build; it would not be helpful if there were huge variations. To get a more accurate picture, we should have generated a histogram of build times. But the graph was good enough in our scenario… and we had to stop the analysis at some point.

The build dependency graph before our fixes:

The build dependency graph after our fixes:

Build statistics

If those graph are not geeky enough for you, here you are the detailed build statistics from the build service

<buildstatistics>
  <disk>
    <usage>
      <size unit="M">1118</size>
      <io_requests>15578</io_requests>
      <io_sectors>2156642</io_sectors>
    </usage>
  </disk>
  <memory>
    <usage>      <size unit="M">580</size> </usage>
  </memory>
  <times>
    <total>      <time unit="s">756</time> </total>         <!-- THIS -->
    <preinstall> <time unit="s">8</time>   </preinstall>
    <install>    <time unit="s">72</time>  </install>
    <main>       <time unit="s">555</time> </main>
    <download>   <time unit="s">4</time>   </download>
  </times>
  <download>
    <size unit="k">33564</size>
    <binaries>53</binaries>
    <cachehits>24</cachehits>
    <preinstallimage>preinstallimage.preinstallimage.tar.gz</preinstallimage>
  </download>
</buildstatistics>

Epilogue

This was definitely an interesting journey. We learned quite some things. Specially we learned that there is still room for improvement, but most likely the time reduction will not pay off for the time invested implementing those improvements.

We have to be realistic and keep working in other interesting stuff to fuel the next sprint report, coming next week!

This has at least two major downsides:

1. Storing all old tar files takes up a lot of disk space
2. OBS workflows with .tar files and patches are rather different and somewhat disconnected from the git workflows we usually use everywhere else these days. E.g. for the SUSE OpenStack Cloud team we have a “trackupstream” jenkins job, that pulls the latest git version into a tarball once every day.

Fedora already keeps their metadata in git, but only a hash of the tarball.

So as one first step, I used two rather different projects to see how different the space usage would be. On the slow side I used 20 gtk2 tarballs from the last 5 years and on the fast side, I used 31 openstack-nova tarballs from Cloud:OpenStack:Master project from the last 5 months.
I used scripts that uncompressed each tarball, added it to a git repo and used git gc to trigger git’s compression.

Here are the resulting cumulative size graphs:

The raw numbers after 20 tarballs: for nova the ratio is 89772:7344 = 12.2 and for gtk2 the ratio is 296836:53076 = 5.6

What do you think: would it be worth the effort to use more git in our OBS workflows?

Do we care about being able to reproduce the original tarballs? While this is possible, it has some challenges in differing file-ordering, timestamps, file-ownerships and compression levels.
Or would it be enough if OBS converted a tarball into a signed commit (so it cannot be forged without people being able to notice)?

Do you know of a tool that can uncompress tarballs in a way that allows to track the content as single files, and allows to later re-create the original verbatim tarball, such that upstream signatures would still match?

I’m contacting you personally, as an openSUSE Board member.

You certainly already know that we want to have a kicking openSUSE conference next 1st-4th May 2015 at the Haag (NL).

Thus I’ve found that creating special workshop organized by development project could foster our beloved distribution.

oSC is the unique case in the year, where Geekos from all around the world meet together.
Let’s imagine you, meeting perhaps for the first time your fellows, having nice discussions, and hacking around the software you maintain.
There’s high level of chance to meet also your end users, and have constructive exchanges.

That’s why I invite you to propose a workshop directly to our event tools:
https://events.opensuse.org/conference/osc15/proposal

Having a workshop run like a mini-hack sprint, would help any of us in the distribution and the project.
Be it like learn people how to submit nice package, how to do maintenance, or how to do bug triage.

I feel confident that you will have the creative approach to resolve your own problematic.
The event place has small rooms for unattended sessions and they could be used to extend your workshop to get some more work done.

Some practical aspects:
oSC website : https://events.opensuse.org/conference/oSC15

And soon the travel support program for oSC 15 will be opened to handle your request about getting financial support for going to oSC.
https://connect.opensuse.org/travel-support/

If you have any questions, thoughts or ideas, don’t hesitate to ask on -project mailing list
or ping me by reply.

A final note about the why you should do it? Well beside being one of our “heroes” even if nobody need them
You and your co-maintainers will be able to explain your “job” on the project.
Don’t you want to inspire new comers, lead them directly to the right direction, and share the load.
Meeting you there, will also help our “marketing” force to light up a bit the work done in the shadow.

I really will appreciate your presence, afterwards, it’s you that create our distribution.
The time has come for you to be warmly thank.

I’m looking forward to see you there.

This post is a summary of my wish to continue to serve on your behalf on the openSUSE board.

My term has been a short one, as I was appointed to serve out the remainder of Vincent Untz’ term.

While I now work for SUSE (and it has been a fabulous experience), that does not change my view or efforts to contribute to openSUSE.  Prior to joining SUSE, as a part of the Sales Engineering team, I was elected to the board for a two year term. To avoid having too many SUSE employees, I had to step down.  A rule I completely support.

Having been an openSUSE member long before joining SUSE, I think I have a keen awareness of what the community is about and where we can improve together.

My take is we have an awesome group of contributors who want to see the community grow and prosper.  With things like Tumbleweed and OBS, among others, we have shown real innovation and technical leadership in the Linux world. We have arguably one the top distros available.  It is solid, polished and usable for a wide variety of use cases. We need to keep the openness and solid collaboration which enables everyone to participate and succeed.

My efforts within openSUSE have been mostly focused on the Open Build Server, maintaining several projects, as well as, being part of the Factory review team.

Going forward, I want to concentrate on reaching out beyond our community to build more awareness of what an awesome distro we have, along with a pretty friendly community. I see other, less compelling, distros getting more visibility than perhaps is deserved.

Along the same lines, my take is we as a community can take a more active role in bringing in new members, who might not be technical folks, but can help in the marketing and outreach. I’ve started a local SUSE group through meetup.com which is a different way to find new users and contributors.

One other reason I wish to remain on the board, is we have a solid working relationship and there is a lot of mutual respect and good collaboration.

No matter who is elected to the board, I am very pleased with the caliber of the folks running and know the community will be in good hands moving forward.

Thank you in advance for your vote!

Peter Linnell

recently, I pushed a commit that speeds up the update of an osc project
working copy, if most of the packages in the working copy are already up to
date (that is no update has to be performed).
The following table shows the improvements of the new code (in terms of
wall-clock time). Both project working copies were already up to date
and the packages in the home:Marcus_H project were unexpanded.

project       # number of packages  #    old code # new code
home:Marcus_H                   66        51.135s    10.653s
d:l:r:e                       1245     7:07.07min    17.804s

(the numbers for the devel:languages:ruby:extensions (d:l:r:e) project
were kindly provided by darix – thanks!).

Technically, we just reduced the number of http requests for packages
that are already up to date by using the backend’s getprojectsourceinfo
call (/source/project?view=info&package=pkg_1…&package=pkg_n).
Note: currently, such a reduction is not possible for packages that have
a _service file, because a small change in the backend is needed (see [1]).
Consequently, there are no time improvements for such packages.

If you want to test the new code, use the osc package from the
devel:tools:scm repo (http://download.opensuse.org/repositories/devel:/tools:/scm/).
Feedback is always welcome!

Next, my plan is to improve the speed of an update of a single package
working copy (again by reducing the number of http requests).

[1] http://lists.opensuse.org/opensuse-buildservice/2014-06/msg00067.html

openQA

Last week we rearranged the repository a little bit, creating a new branch called "devel" where all the exciting (and not so exciting) changes are taking place. Our little Factory

The main difference between this branch as master is that, as you could read in the previous blog, the devel branch is openQA build on Mojolicious, a nice web development framework. And having a proper web framework is starting to show its benefits: we have openID login available! Unfortunately the current openSUSE openID provider is a little bit weird, so it doesn’t play well with our tool yet, but some others are working and openSUSE accounts will be the next step. Having working user accounts is necessary to be able to start defining permissions and to make openQA truly multiuser. And to be able to deploy the new version on a public server!

The other main focus of this week has been internal polishing. We have revamped the database layout and changed the way in which the different openQA components communicate with each other. The openQA functionality is spread out over several parts: the workers are responsible of actually executing the tests in virtual machines reporting the result after every execution; some client utilities are used to load new ISO images and similar tasks and, finally, we have the one openQA Server to rule them all. Until now, the communication between the server and the other components was done using JSON-RPC (a lightweight alternative to XML-RPC). We have dropped JSON-RPC in favor of a REST-like API with just JSON over plain HTTP. This change allowed us to implement exactly the same functionality in a way that is simpler, perfectly supported by any web framework, natively spoken by browsers and easier to authenticate (using, for example, plain HTTP authentication or openID). This is also the first step to future integration with other services (think OBS, as the ultimate goal is to use openQA to test staging projects).

But, who tests the tester? openQA is growing and changing quite fast so we have continued with the task of creating a good testing infrastructure to tests openQA itself to make sure that all our changes do not result in breakage. We only have a few tests right now, but we now have a solid foundation to write more and more tests.

Staging and package manipulation

In the last blog post we told you we were investigating a code test suite to test the abilities of a osc plugin we are writing. osc is the commandline tool for handling the Open Build Service, and this plugin is meant to help with the administration of staging projects. We’ve been thinking about how to move forward with the testing part as we want to make sure the functionality works as advertised. More important, we write tests to make sure that our additions and changes do not break existing functionality. We have started merging functionality from various scripts handling staging thingies and rings we had into this plugin. This is partially done so we can do basic staging project tasks! We can take a request (be it submit or delete) and put it to test in a staging project. We can also move packages between staging projects and we have a simple YAML in project descriptions to indicate what packages and what requests are there.

Coolo already started using the plugin for some tasks, so you can see pretty titles and metadata in staging projects descriptions. Not impressive enough? Let me provide you a good headline:

Thanks to staging projects, no single regression have been introduced this year in the core of Factory

You can enjoy a more detailed description in this announcement written by coolo to the Factory mailing list and have some visual joy in the included screenshot (genuine pr0n for QA engineers).

Last but not least, we also did some cleanup of the sources in the repo and of course we added more tests (as functionality grows). And there has been work on other parts of the plugin, like taking rings into account.

Result

We already have some useful functionality which we are going to expand and build on. Not that much to show yet, but we are progressing towards our goal. You can follow our progress either in the way of tasks (openQA and staging projects) or just follow our commit history on github (again for both openQA and staging plugin).

We are very much looking forward to feedback and thoughts – these changes aim to make Factory development easier and thus we need to hear from all you!