Michal “|Miska|” Hrusecky writes this week about the openSUSE release process. He gave a talk about this subject at the openSUSE conference this year and the content of this talk is reproduced below (the marketing section, just like with the presentation, is by Jos Poortvliet).
To get openSUSE out is a lot of work. We already shared part of what we are doing to keep Factory rolling. But as you can guess, there is much more to it. But let’s pretend it is a simple three-step process:
Step one: developing Factory
When release openSUSE, we immediately start working on the next version: a never ending story. First thing that happens during a new release cycle is coolo announcing the road map. This is the schedule of the release and important checkpoints that we have to reach on our way. After the release Factory (our development version) is not frozen anymore and people can start submitting new stuff. Usually they go crazy and submit a lot of bleeding edge and experimental packages and quite some parts of Factory will get broken.
Now comes the time for keeping Factory rolling. As one picture can say more than thousand words, take a look at how packages get into factory:
A developer creates a package in his home repository and then sends it to the devel project where it get’s some basic review (depends on how the team is setup there). From there, package goes to the factory, where are several automatic checks (will mention them later) and manual review by review team. Once the package passes all these checks, it gets into Factory. And breaks something else there. So somebody has to branch it, fix it, and go through the same process now with a different package…
We’re working on documenting the Factory development process in a little more detail so you can expect a blog on that subject in the coming months!
Checking up on you…
There are several bots that check packages send to Factory. First is factory-auto, which does some basic checks regarding you spec file, rpmlint warnings and similar. If you pass this quick test, it’s time for more thorough testing. A bot named factory-repo-checker tries to actually install your package in a testing environment to make sure it is possible and it also looks for possible file conflicts, so you wouldn’t overwrite somebody elses package. Last check before a package gets in front of the review team is legal-auto. This one checks the licence (did it change? is it a new package?) and if needed calls in our legal team to take a look at the package. The final step is manual review by members of review team which will try to spot mistakes that automatic checks overlook.
Step two: freezing
We could be breaking and fixing stuff forever, but as we have to release at some point we have to do some freezes. First one is the Toolchain Freeze. When we reach that, no new version of compilers will get in, only fixes. This way the rest can fix all the new compiler warning and errors without fear of getting new ones. Next in the line is Base System Freeze which means frozen kernel, KDE, Gnome, X11 and bellow… All these are core components with many packages depending on them. The rest of packages (after getting fixed/upgraded) gets frozen last with release of Beta 1 when we reach Full Feature Freeze. At that point only fixes go in, no new versions and features or anything else that could break anything.
Step three: branching and selecting an ISO
As we get close to the release, Factory gets branched into separate repository, currently 13.1. Factory then gets unfrozen and starts to live it own life again. The newly born release goes through the last stabilization and deep testing phase (both manual and through openQA). Every automatically build iso gets tested until the Gold Master is selected. Gold Master has to be installable andcannot contain any critical ship stopper bugs. It still can contain known bugs, we would never release if we would wait for bug free release. But all these bugs can be fixed via updates and some of them will even get fixed before official release date. After selecting Gold Master it takes us usually about a week to get it synchronized to our mirrors and to get everything ready for the big day. In that time, developers keeps fixing bugs and when you get our fresh release, we already have many fixes ready and waiting.
Let’s talk release marketing now. We’ve got a HUGE list of things we work on in the marketing team.
There is preparation for the release, stuff we do usually in the 2 months before the release. Think about material like the release counters, posters and badges for on websites or at conferences; writing the sneak peek articles and organizing release parties.
Then there is the release marketing itself. We write a feature guide, extract feature highlights, create screen shots, write announcements for our news site and for the press. Then we put all of the above, combined with a list of people to be interviewed and a link to the gold master into a press kit we send one week before the release to journalists. This way we can have reviews published on the release date.
How it all comes about
All that is essentially build upon the list of new features for openSUSE. We gather the developers’ brain dumps at this wiki page. They share with us a bullet list of what changed since the previous release and how important it is for their users.
We take these features to a pirate pad and start looking for everything that is missing (which, usually, is a lot: much of the features we don’t know about because developers don’t tell us). We gather info from the sites of projects like KDE and GNOME about their new versions and put it in. We work on the bullet points to make pretty text and we try to bring in some structure.
Once it is reasonably complete, we move it to a wiki page where we start to polish it up and add screenshots, links and video’s if we have em. The end result you are probably familiar with: the Features page. Note that most of this does not really need much experience, almost anybody can help!
Then the next step is to look at that draft and the structure it has gotten and put in some deep thinking to pick the major features for the highlights. There is a lot of whiteboarding in this step: figuring out what the message of our release is, the theme, that is not easy.
These are the main part of the announcement and everything else, which is then ‘just’ hard work – a lot of it. This step needs the most experienced writing and a lot of feedback-improvement-feedback-improvement cycles.
Timeline: work gets crazy at Beta
The biggest problem with all this is that it can’t be done in a very relaxed way because we’re in a hurry. Beta 1 is about 6 weeks before we have the Goldmaster! When the Goldmaster is done, we need to send our stuff out to the press. At that point we can make only VERY minor changes. After all – just a week later we do the release. In that week, we only write social media messages (and translations) and get our websites, servers and everything else in order. Marketing is ready for the release!
Releasing a conclusion
We hope the above gives some sense of how the openSUSE release process works. Yes, it is long, laborious, but also a lot of fun! And of course it results in the awesome operating system which is openSUSE!
And it is that time again: the unveiling of our statistics of openSUSE top contributors of the last week! Here’s the top-ten most active Factory hackers:
|8||Vitezslav Cizek, Ludwig Nussel|
|10||Todd R, Hillwood Yang|
Both comments and pings are currently closed.
I’m still confused why the above left ‘Get it’ banner / logo is inappropriately linked to:
..that then ends at:
..? Can someone please correct this? THANKS!
M. Hill | Hacim Llih | tech9iner
Ahem.. please know “..above RIGHT..” was intended; woe is me. ;]
From the above blog entry we get a link to Keeping Factory in shape, which states:
“Another example is GTK which keeps deprecating old API functions and you have to keep up and replace them with correct counterparts from the new API. Sometimes even the kernel changes API and third party modules stops building. All these errors will get eventually resolved upstream (if upstream is alive), but as we follow upstream quite closely before feature freeze, it may happen that we are first facing these issues because sometime we are the first ones who tried to compile this software using new GTK”.
The GTK developers may very well be in their right to remove deprecated functions. However, why on earth is openSUSE linking new GTK with old/unmaintained software that relies on old GTK features? Surely gtk v2 and v3 can live side-by-side?
This brings up another point. Instead of discovering breakage as a side-product of building software, why not have deliberate and explicit API and ABI regression tests?
Reasoning: much of open source software is done on a volunteer basis, and the upstream developers may not have the time/energy/knowledge to write (or maintain) their own regression tests. Wouldn’t it be more useful for a distribution to provide such regression tests, rather than simply packaging upstream software? This will also help keep certain developers honest, by exposing how much unnecessary API breakage they cause.
I’m currently running 13.1 Milestone 4. When the final version of 13.1 drops, will I have to reinstall or will my milestone version get updated?
My question has been answered.