This week, the openSUSE Team Blog is written by Agustin Benito who talks about how we’ve been documenting our work around openSUSE.

Development a GNU/Linux based distribution like openSUSE can be divided in two major phases:

• openSUSE Development process: creation and integration of the different packages. Milestones (alphas) and Beta.
• openSUSE Release process: stabilization, QA and task related with the availability of the different images. Update process. From Beta to the Release.

There are also three major continuous actions throughout the process:

• Management of the process.
• Communication and marketing related tasks.
• Development of new features

The development of new features usually happens upstream or within SUSE (in collaboration with community members) before it is added to the distribution. This phase takes place before or in parallel with the Development process (of packages and integration).

Focus: the openSUSE Team

The decision taken by SUSE during 12.2 to concentrate the SUSE employees working on openSUSE in a single department by creating the openSUSE Team had as consequence that new people has joined us to work in openSUSE. One of the early decisions within the openSUSE Team at SUSE was to put more focus on the distribution. The Release Team was created as a subset of the openSUSE Team, formed by Coolo and Tomas (Ismail Dönmez and Michal Hrušecký were also part of it in the past) to take care of the Development process. It was also decided that the whole Team would have as major goal to drive the Release process.

Documenting the process

Starting in July 2012, the openSUSE Team at SUSE has put effort in documenting the Development + Release process. Throughout the years, the process has evolved and some of those changes were not documented or the documentation was not up to date. We have taken the opportunity to analyze the the Dev+Release process, so we could learn from it and being able to design and execute changes to improve it.

The release process

As Team, we focused first on the Release process. We made an initial effort during the last few weeks of the 12.2 Release process, and we improved it during the first milestones of the 12.3 Release. For this 13.1 Release, only minor updates have been required.

Increasing the efficiency of the management side of things and structuring our work as a formal project has been another aspect of this. We set up a management tool that now is used together with the openSUSE sysadmin team. Ludwig Nussel is the controller of our efforts as team in the Release, planned through progress.opensuse.org. The combination of this planning and the improved release documentation has helped us to increase our efficiency despite new people joining our team.

The development process

During the past few weeks we have concentrated our efforts in analyzing and documenting the Development process. Our goal has been to analyze it, providing a high level view and just the minimum amount of details required to understand it by people with some knowledge of these kind of software integration processes.

We have recently published a draft of the Development process on the openSUSE Factory mailing list and are open for feedback.

Writing process

How was this document done?

1.- The first action was to analyze existing documentation about the Development process. We took as sources several articles from openSUSE wiki and some previous analysis done by the former Boosters Team in this area.
2.- We organized two sessions in which different people described the different steps of the process. These sessions were taped.
3.- We transcribed the sessions and created a first document containing all the information about the process.
4.- We processed that “story” in order to get the high level view of the process.
5.- Using the ETVX methodology, we elaborated a first draft of the document.
6.- We analyzed the result within the openSUSE Team and, after introducing some changes, we opened the discussion about it to other SUSE employees involved in the Development process.
7.- Finally, in order to make the document easier to read (it is a complex process, so the documentation needs time to digest), we introduced improvements in the format and published in the openSUSE wiki, together with the .pdf version.

This process has allowed us to analyze and discuss the process as a team, learning not just about the hows but the whys. It has also worked as a test for documenting future changes in how openSUSE is developed. We also hope that the effort can be worth it to contributors that want to get a high level view of how openSUSE works, since some of the tasks are done in house. This document has to be seen also as an exercise of transparency.

Improvements

At openSUSE Team, we are strong believers of the principle “no data -> no analysis -> no improvements“. We think we are better prepare now to propose to the community improvements in factory to make it more usable for a wider range of contributors.

Statistics Time

And like every week, we present you the top-ten of contributors to Factory!

Our team blog this week is written by Stephan ‘coolo’ Kulow and he talks about work in the team done on the Open Build Service.

Search

For years one of the biggest complaint about the webUI was that it is impossible to find packages. The search ability has been part of the interface from the beginning, but with over 200.000 packages being build today it is crucial to get the right package.

Where is my kernel?

Especially for developers new to openSUSE and the build service it is common to have to search for the package to fix for a specific bug. So you find yourself looking for kernel in the webUI and you are prompted with tons of results that are displayed in a rather random order and the notion that your search resulted in more than 200 hits and is basically invalid. Huh? home:foobar:latest-experiments:kernel is surely not the openSUSE kernel to fix, but then what is?

Now if you ask google about “kernel site:build.opensuse.org” you get closer to the problem at hand: “About 16,800 results” – that is a lot to pick the first 20 results to display from. The OBS webUI tried to find a good pick with an algorithm that might have been clever when build.opensuse.org had 100 projects. Today, it can only be called old and useless.

Ancor for world fame

So I tricked Ancor to look into the problem by claiming he would get all the praises in the OBS world for implementing a sane search.

The problem is far from trivial, but there are good tools to get a better result than what we had now and Ancor has a lot of experience with these (and Rails in general). So it seemed like he could attain a great balance between effort and outcome.

But as always the devil lay in the details, so this post is also about getting feedback about the actual result.

What we did

Ancor integrated Thinking Sphinx into the OBS, so the name, title and description can be combined with other attributes into one big index that allows page ranking.

Additionally there is no limit of 200 results anymore, the webUI will display all results now, but only 20 at a time as you might have seen in larger sites offering search results display…

We collected attributes which are most likely relevant for people searching for packages. For example, we gather the linkcount of a package into the database (so far only the backend knows what is a link and what is a plain package). The idea is to move links down in the source results.

Coming back to the kernel example, the kernel-source package is the real package, while kernel-default, kernel-desktop, kernel-xen, …. exist too but are all links to kernel-source. So it is fair to present kernel-source first.

Problem is: there are still 228 kernel-source packages in the build service (yes, people like branching the kernel – a lot), so the number of links pointing to the package is another attribute. Packages that other packages branch go up in the list while the resulting branches move down. What also plays a role in the calculation: is a package the devel package of another? (which is the final punch to have Kernel:HEAD/kernel-source as first result displayed, as opposed to the old searching algorithm displaying a discontinued “linux-kernel-nutshell” as shown in the screen shots).

To sort the vast majority of results that are all _links, not linked to and not devel packages, we take the activity index. This is a number the OBS tracks for every package, but is nowhere displayed. It goes from 0 to 100 and goes down with time and goes up with regular commits. So if you look for kde, you will actually see KDE:Unstable:Playground as first project to match. This is because of two things:

• kde is indeed a very bad search term
• the unstable playground sees a lot of commits, so your chance of getting something fresh there is the highest

Your feedback wanted

Of course nobody is perfect, and while the code of Ancor is close to, the weights given to the attributes were my choice, so all problems in the sorting you see are my fault. Please take some time and redo some searches you might have done in the past and report if the results are sane to your experienced eye. Within the HTML of the search results is a hidden span with the raw attributes used in the calculation, so if you find something strange, look for weight, linked_count, activity_index, is_devel and co. Possibly the package that looks bogus to you in the top results is just very active.

Depending on the feedback we get, we might need to change the weights or add yet more attributes in the search and ranking. Do your own experiments on build.opensuse.org today!

Statistics

And as always, we finish with the top-ten contributors to openSUSE Factory of last week!

Michal “|Miska|” Hrusecky writes this week about the openSUSE release process. He gave a talk about this subject at the openSUSE conference this year and the content of this talk is reproduced below (the marketing section, just like with the presentation, is by Jos Poortvliet).

Release Process

To get openSUSE out is a lot of work. We already shared part of what we are doing to keep Factory rolling. But as you can guess, there is much more to it. But let’s pretend it is a simple three-step process:

Step one: developing Factory

When release openSUSE, we immediately start working on the next version: a never ending story. First thing that happens during a new release cycle is coolo announcing the road map. This is the schedule of the release and important checkpoints that we have to reach on our way. After the release Factory (our development version) is not frozen anymore and people can start submitting new stuff. Usually they go crazy and submit a lot of bleeding edge and experimental packages and quite some parts of Factory will get broken.

Now comes the time for keeping Factory rolling. As one picture can say more than thousand words, take a look at how packages get into factory:

A developer creates a package in his home repository and then sends it to the devel project where it get’s some basic review (depends on how the team is setup there). From there, package goes to the factory, where are several automatic checks (will mention them later) and manual review by review team. Once the package passes all these checks, it gets into Factory. And breaks something else there. So somebody has to branch it, fix it, and go through the same process now with a different package…

We’re working on documenting the Factory development process in a little more detail so you can expect a blog on that subject in the coming months!

Checking up on you…

There are several bots that check packages send to Factory. First is factory-auto, which does some basic checks regarding you spec file, rpmlint warnings and similar. If you pass this quick test, it’s time for more thorough testing. A bot named factory-repo-checker tries to actually install your package in a testing environment to make sure it is possible and it also looks for possible file conflicts, so you wouldn’t overwrite somebody elses package. Last check before a package gets in front of the review team is legal-auto. This one checks the licence (did it change? is it a new package?) and if needed calls in our legal team to take a look at the package. The final step is manual review by members of review team which will try to spot mistakes that automatic checks overlook.

Step two: freezing

We could be breaking and fixing stuff forever, but as we have to release at some point we have to do some freezes. First one is the Toolchain Freeze. When we reach that, no new version of compilers will get in, only fixes. This way the rest can fix all the new compiler warning and errors without fear of getting new ones. Next in the line is Base System Freeze which means frozen kernel, KDE, Gnome, X11 and bellow… All these are core components with many packages depending on them. The rest of packages (after getting fixed/upgraded) gets frozen last with release of Beta 1 when we reach Full Feature Freeze. At that point only fixes go in, no new versions and features or anything else that could break anything.

Step three: branching and selecting an ISO

As we get close to the release, Factory gets branched into separate repository, currently 13.1. Factory then gets unfrozen and starts to live it own life again. The newly born release goes through the last stabilization and deep testing phase (both manual and through openQA). Every automatically build iso gets tested until the Gold Master is selected. Gold Master has to be installable andcannot contain any critical ship stopper bugs. It still can contain known bugs, we would never release if we would wait for bug free release. But all these bugs can be fixed via updates and some of them will even get fixed before official release date. After selecting Gold Master it takes us usually about a week to get it synchronized to our mirrors and to get everything ready for the big day. In that time, developers keeps fixing bugs and when you get our fresh release, we already have many fixes ready and waiting.

Marketing process

Let’s talk release marketing now. We’ve got a HUGE list of things we work on in the marketing team.

Building excitement

There is preparation for the release, stuff we do usually in the 2 months before the release. Think about material like the release counters, posters and badges for on websites or at conferences; writing the sneak peek articles and organizing release parties.

Releasing openSUSE

Then there is the release marketing itself. We write a feature guide, extract feature highlights, create screen shots, write announcements for our news site and for the press. Then we put all of the above, combined with a list of people to be interviewed and a link to the gold master into a press kit we send one week before the release to journalists. This way we can have reviews published on the release date.

How it all comes about

All that is essentially build upon the list of new features for openSUSE. We gather the developers’ brain dumps at this wiki page. They share with us a bullet list of what changed since the previous release and how important it is for their users.

Pirates

We take these features to a pirate pad and start looking for everything that is missing (which, usually, is a lot: much of the features we don’t know about because developers don’t tell us). We gather info from the sites of projects like KDE and GNOME about their new versions and put it in. We work on the bullet points to make pretty text and we try to bring in some structure.

Once it is reasonably complete, we move it to a wiki page where we start to polish it up and add screenshots, links and video’s if we have em. The end result you are probably familiar with: the Features page. Note that most of this does not really need much experience, almost anybody can help!

Highlighting

Then the next step is to look at that draft and the structure it has gotten and put in some deep thinking to pick the major features for the highlights. There is a lot of whiteboarding in this step: figuring out what the message of our release is, the theme, that is not easy.

These are the main part of the announcement and everything else, which is then ‘just’ hard work – a lot of it. This step needs the most experienced writing and a lot of feedback-improvement-feedback-improvement cycles.

Timeline: work gets crazy at Beta

The biggest problem with all this is that it can’t be done in a very relaxed way because we’re in a hurry. Beta 1 is about 6 weeks before we have the Goldmaster! When the Goldmaster is done, we need to send our stuff out to the press. At that point we can make only VERY minor changes. After all – just a week later we do the release. In that week, we only write social media messages (and translations) and get our websites, servers and everything else in order. Marketing is ready for the release!

Releasing a conclusion

We hope the above gives some sense of how the openSUSE release process works. Yes, it is long, laborious, but also a lot of fun! And of course it results in the awesome operating system which is openSUSE!

Weekly statistics

And it is that time again: the unveiling of our statistics of openSUSE top contributors of the last week! Here’s the top-ten most active Factory hackers:

Redmine in action

This week we present Ancor Gonzales Sosa who writes about how we coordinate our work and introduces progress.opensuse.org!

A distributed team

The openSUSE Team at SUSE is a combat force spread all over the world: we have members in Berlin, Prague, Malaga, Nuremberg and Taiwan. And we are planning to reach new territories in the following months (beware, a openSUSE Team member could be standing behind you just right now). Coordinating the work of a distributed team is not always easy. On the other hand, we are disperse not only from a physical point of view, we also work simultaneously in a lot of different tasks and projects. We also have members with different skills and interactions with SUSE employees and community members outside our team. (more…)

While we’re internally working on an openSUSE Conference report, we share a few more thoughts on the event.

Booths

The openSUSE both was located near the reception desk. The booth featured:

• Merchandise, including fresh 12.3 DVDs, openSUSE flyers, calendars, stickers and bookmarks. The team brought a stack of ‘old’ flyers from Nuremberg which were ordered burned by Jos (who brought new style flyers from Berlin, initially printed for LinuxTag 2013). These old flyers are so outdated, both in terms of artwork and text, that it is pointless to keep giving them out…
• We are Hiring flyers: Due to the demand of the flyers, we printed 30 more jobs descriptions, all were delivered.
• The booth was dressed up in the openSUSE booth cloth, had a standing X banner and on the table was one of our big touch screens. The booth cloth and standing X banner were ‘test versions’ for what will probably be included in future Booth Boxes. The designs were temporary, and in case of the table cloth, quite horrible…
• Last but not least, the booth staff of rotating SUSE employees (rotating shifts) informed visitors of what was going on in the openSUSE world.

Of course there were other booths. Mozilla presented the new Mozilla OS, allowing us to be grabby with two prototype phones; there were KDE and GNOME booths with some merchandising, an Oracle booth where a very knowledgeable Oracle employee answered questions about MySQL and MariaDB. There also was an LPI booth for those hungry for knowledge and an FSFE booth for the hungry-for-freedom folk.

We kind’a like having a booth area for in between the talks. What do you think?

Schedule and content

The openSUSE team organized and gave quite a number of talks and sessions. A quick list:

• Alberto Planas discussed openSUSE project statistics and gave a workshop on creating tests for openQA
• Michal Hrušecký explained how openSUSE does releases, spoke about how to travel light and finally about keysigning parties
• Tomáš Chvátal gave a workshop on Autotools, shared his experience in dealing with toxic behavior and detailed what’s new in LibreOffice
• Jos Poortvliet gave two workshops on giving awesome presentations and a talk about how to run a successful booth and what materials SUSE will be shipping around. He also gave the community keynote on Sunday and demoed the travel support program application with Izabel Valverde

Future

We plan to increase our efforts especially in training workshops next year as there’s a lot of knowledge in SUSE that should be shared with the community.

We’d again like to thank everybody involved in the booth staffing, talks, workshops and everything else, of course!

Weekly statistics

And as usual, we bring you the top-10 contributors to openSUSE Factory. Great work, all!