Home Home > 2016 > 09
Sign up | Login

Archive for September, 2016

Highlights of YaST development sprint 25

September 28th, 2016 by

Another development sprint is over. Time flies! In our previous post we already reported about the branching of Tumbleweed and the upcoming releases and about the expected consequences: the landing of some cool features in a less conservative Tumbleweed.

We are still dedicating quite some effort to polish the upcoming stable releases (SLE12-SP2 and Leap 42.2), but in this sprint we finally found some time to play. Which is great because blogging about new features is more fun than doing it about bug fixes. 🙂

Importing Authorized Keys with AutoYaST

When logging in via SSH, public key authentication should be preferred over password authentication. Until now, the best way of setting up the required authorized_keys files in AutoYaST was using the files section.

However, that approach is tedious and error prone, as you need to make sure you set the correct owner, permissions, etc. Moreover you need to keep in sync the user definition (username and home directory) with the file definition.

AutoYaST now supports the specification of a set of public keys for each user with a pretty straightforward syntax:

<user>
  <username>suse<username>
  <authorized_keys config:type="list">
    <listentry>ssh-rsa your-public-key-1</listentry>
    <listentry>ssh-rsa your-public-key-2</listentry>
  <authorized_keys>
<user>

AutoYaST takes care of writing the files and setting the ownership and the proper permissions.

While documenting this new feature we realized the AutoYaST documentation about users management could be more detailed, which leads us to…

Improving the documentation

Usually developers love to create programs loaded with cool features but hate to write documentation. Fortunately there are people out there who enjoy writing documentation and bringing all those features to light. We have already mentioned in previous reports how grateful we are for having the SUSE documentation team polishing and publishing our documentation drafts and how open and straightforward the process is.

We updated the YaST documentation to include information about the installer self-update feature, which will debut in SUSE Linux Enterprise 12 SP2 and openSUSE Leap 42.2. As part of the same pull request and in the AutoYaST side, some additional improvements were made, including cleaning-up some duplicated information about SUSE registration.

On the other hand and as a consequence of the above mentioned new feature, the AutoYaST documentation regarding users management has been rewritten adding missing information like groups, user defaults and login settings.

All our pull requests are already merged in the doc-sle repository. At a later point in time, the SUSE documentation team will review and polish all the new content (including ours) and will publish an up-to-date version of the online documentation. If you don’t want to wait, you can easily generate an HTML or PDF version of the documentation including all the non-reviewed contributions just following the very simple instructions in the README file of the doc-sle repository.

Did we already mention we love the open source, programmer-friendly processes of the documentation team? 😉

Storage reimplementation: something you can touch

We promised news about the storage reimplementation and here they are. Our customized Tumbleweed image (labeled as NewStorage) in the storage-ng OBS repository can now perform some simple actions during installation and display the result to the user.

First of all, when proposing the timezone settings it will, as usual, check for MS Windows installations in the disk to guess if the hardware clock should be set to UTC. The news is that check is performed using the new storage stack, that offers more functionality in every sprint.

More important is that the installer will show the partitioning proposal calculated also using the new stack. As you can see in the screenshot below, the screen is way more simpler than the one you would find in a regular Tumbleweed. There are no buttons to change the settings or to run the expert partitioner yet. That doesn’t mean the functionality is not there, it’s simply that we prefer to focus first on modifying all the installer steps to use the new stack (what will enable us to use openQA) before refining every screen to add all options there.

The new partitioning proposal

Right now the system works only in disks containing a MS-DOS style partition table and will always propose a partition-based (no LVM) setup. That’s because we prefer to solve the hardest scenarios first. Using LVM and/or GPT partition tables is less challenging than the already supported scenario.

Reduce global warming by saving OBS build power

As you may know, we use the awesome Open Build Service (OBS) to generate both the YaST rpm packages and the openSUSE/SLE ISO images. Every time the source code of any component changes, OBS rebuilds that component and all the packages that depend on it.

Our beloved openSUSE and SLE release managers told us that there were several YaST packages that often triggered rebuild of other YaST packages, that triggered yet another rebuild, that triggered… you got the idea. 😉

The mentioned problem slows down the creation of new ISO images, interferes with the continuous integration process (specially visible in Tumbleweed) and wastes valuable OBS resources.

During this sprint we reduced the rebuild time of YaST by 30%. That’s for sure interesting, but knowing the details about how we did it could be even more interesting for many readers. We feared the explanation could be too complex and technical to fit into this report… which gives us just another opportunity for blogging. So expect an upcoming post including interesting technical stuff and crazy graphs like this one.

YaST dependencies graph

Some adjustment for the installer in the LiveCDs

One of the good things about working in open source is that sometimes the evolution of the projects you have created can surprise you. Quite some time ago, the YaST team dropped support for the live installer. It was simply too demanding to keep it alive while still doing our regular work (bug fixes and new features for YaST and the regular installer).

Recently the live installer was removed from Tumbleweed, the only system in which it was still available (after having been dropped in the past from stable openSUSE releases). One could have expected that somebody would decide then to step up and take the maintainership of the live installer.

But what actually happened was that Fabian Vogt decided to try a different approach we haven’t considered – adding the standard network installer to the LiveCDs images of Tumbleweed. He managed to make it work well enough and asked us for help to debug some problems. We fixed the initial problems by disabling the self-update functionality in the LiveCDs (it’s simply not designed to work on that scenario).

There are still quite some problems to be resolved to make everything work flawlessly, but if Fabian and others don’t give up, we will keep assisting them in order to bring the installation back to the LiveCDs… even in unexpected ways.

UI Designer

The YaST user interface is quite difficult to design and code. The main problem is that there is was no interactive UI designer where you could build a dialog or modify an existing one. Instead, you had to write new code or modify the existing code which creates and opens the dialog. Then, to see your changes you had to start the YaST module, go to the respective dialog and check its content. If it didn’t look like you intended, you had to close it, modify the code and start it again. And again… and again. Very annoying especially if the dialog is hidden deep in the module and you need to take several steps to get there.

During Hack Week 14, a project to improve the situation a bit was started. We already had a dialog spy which can be opened by Ctrl+Shift+Alt+Y keyboard shortcut, but that was read-only. You could only inspect the dialog tree and see the details of the selected widget but you could not change anything.

During that Hack Week project it was improved so it could edit the properties of the existing widgets, remove them or even add some new widgets. However the code was more or less a proof of concept than ready to be merged into the YaST UI and released to public. So we decided to finish it in this sprint.

As usual, it was harder than expected… but we made it and here is a short demo showing how it works and what you can do there:

The new UI designer in action

The new tool is still far from being perfect. The most obvious missing feature is that the dialog is changed in place and you cannot save or export you changes. After closing the dialog everything is lost. But it can still help to try things in the UI or make a quick prototype, specially when discussing solutions with interface designers. Hopefully we find some more time in the future to make it even better.

Storage reimplementation: encryption support

Although the partitioning proposal still does not support encryption or LVM, we implemented full LUKS (encryption) support in the underlying library (libstorage-ng). Together with the LVM support implemented in the previous sprint, that makes the new library already a valid replacement for the old libstorage in many situations and scenarios. Now it’s mainly a matter of switching from one version to another in every single YaST component, starting with the expert partitioner that we plan to start redesigning in the next sprint.

As usual, new features in the library are hard to illustrate, unless you accept the action diagrams as screenshots. In that case, here you can see the sequence of actions performed by the library when creating an encrypted home volume.

Creation of an encrypted home with libstorage-ng

Syncing keyboard layouts and console fonts in Leap and Tumbleweed

In parallel to our Scrum sprints, we have been for some time steady working, together with the openSUSE maintainers of X.Org and systemd, in redesigning how keyboard maps and console fonts are managed by YaST. Some changes were introduced in Tumbleweed some time ago but never made it to SLE (or Leap) because they needed more testing.

Recently Ludwig Nussel, the Leap’s release manager, decided that he wanted to sync 42.2 with Tumbleweed in that regard, using the new approach instead of the more conservative SLE one. Thus, we also invested quite some time coordinating again with Stefan Dirsch (X.Org) and Franck Bui (systemd) to push the changes just in time for the beta2 version of Leap 42.2… just in time to introduce bug#1000565, that was honored with its inclusion in the list of most annoying bugs in 42.2 Beta2.

The bright side is that a fix for the bug has already been provided (see bug report) and you can now finally test the new fonts and keyboard maps. Please, do so and provide feedback in order to get a properly localized Leap 42.2 release.

See you soon

As usual, this post was just a quick overview of the most interesting part of the sprint, because most people (including ourselves) don’t want to read about the boring part of the work, which is mainly fixing bugs.

The good news is that this time you will not have to wait another three weeks to read interesting stuff about YaST. As mentioned, we plan to publish a blog post about the reduction of the build time of YaST. And we will probably also publish a post about the visit of a YaST geecko to Euruko 2016.

So this time more than ever… stay tuned!

Highlights of YaST development sprint 24

September 7th, 2016 by

We are back to this blog after another three weeks of (mainly) bug-fixing. In the previous post we promised some news about the self-update functionality and about the LVM support in the new storage stack. We have that… and much more!

So this will be a long post, but it also hides some gems. You will have to keep reading in order to find them.

Self-update improvements

We have already mentioned in several previous reports the new self-update feature in YaST, which allows updating the installer itself before performing installation of the system.

But it turned out that the initial implementation had an important drawback. The self-update process happened after having performed some of the installation steps. Then, after updating the installer it was restarted and several of those steps lost their configuration or simply did their operations twice.

After some discussions we decided to move the self-update step earlier, at the very beginning. For downloading the updates we basically need just working network connection and initialized package management. So we moved the self-update step after the initial automatic network setup (DHCP) and added package initialization to the self-update step.

The self-update in action

As you can see the self-update step is the very first step in the installation workflow, the language selection and the EULA dialog is displayed after the self update is finished and YaST is restarted. That means all the following steps do not need to remember their state as they will not be called twice after the restart.

The disadvantage is that we had to drop some features. The self-update step happens before the language selection and the optional disk activation. That means by default the self-update progress (and potential error messages) will be displayed in English. But you can still use the “language” boot option and set the language manually via linuxrc.

On the bright side, we fixed like half a dozen of reported bugs just by relocating the self-update process. So we are pretty sure it’s worth the price.

For more details see the updated documentation.

Gem one: using the info boot parameter

The info boot parameter is a pretty old linuxrc option but it is probably not known well. The parameter is an URL which points to a text file which can contain more boot options.

When we tested the updated self-update described above we needed to build a driver update disk and pass several boot options. To avoid repeating the same options on the boot command line and to share the boot options across the team we created an info.txt file with content like

insecure=1
startshell=1
dud=ftp://example.com/self_update.dud

Then you simply boot the installation with info=ftp://example.com/info.txt and linuxrc will read the additional parameters from the file. This can save you a lot of typing, especially if you need to repeat the tests many times.

Fixed a security bug for 7 (yes, seven) different SLE releases

Some weeks ago, during a routine code review, our security experts found a vulnerability in YaST’s libstorage related to the way we provide the encryption passwords to some external commands. It is debatable how dangerous this threat really is. It was never a problem during system installation, but it would affect admins who create encrypted partitions (mostly encrypted LVM physical volumes) or crypto files in the installed system.

A potential attacker with access to /tmp could intercept the password in the very precise moment in which the “cryptsetup” or “losetup” command are invoked by YaST. It’s really only a matter of milliseconds. But we don’t want to take any risks, however small they may be.

So not only did we fix that for the current code streams, we backported it to all the SLE releases out there that are still supported (even though in some cases it’s just a single customer) – back to SLES-10 SP3 from late 2009. That meant backporting the fix to no less than 7 SLE releases (for Leap, those fixes are picked automatically).

As you can imagine, this got more difficult the farther back in history we went: In a central library like libstorage, things are constantly changing because the tools and environment (kernel, udev, you name it) are constantly changing. There was only a single case where the patch applied cleanly; in all other cases, it involved massive manual work (including testing, of course).

Was this fun? No, it certainly was not. It was a tedious and most frustrating experience. Do we owe it to our users (paying customers as well as community users) to fix security problems, however theoretical they are? Yes, of course. That’s why we do those things.

Storage reimplementation: every LVM piece in its place

As time permits, we keep adding new features to the future libstorage replacement. During previous sprints we added support to read and manipulate all kinds of LVM block devices (PVs, VGs and LVs) but an important aspect was missing: deciding the order of the operations is as important as performing them. We instructed the library about the dependencies between operations and implemented several automated test cases to ensure we don’t try to do not-so-smart things like removing a physical volume from a volume group and shrinking its logical volumes afterwards.

The good thing about our automated test-cases is that they generate nice graph that are quite useful to illustrate blog posts. 🙂

One of the several added test-cases

Gem two: enjoy Google Summer of Code result

As you may know, openSUSE is one of the Free Software organizations selected to take part in Google Summer of Code 2016. For YaST that means we had the huge pleasure of having Joaquín Yeray as student. You can know more about him and his experience diving into YaST and Open Source in his GSoC blog.

But the openSUSE community is not only gaining a new member, we also have a new YaST module. The yast2-alternatives package has already been accepted into Tumbleweed and will be also part of Leap 42.2. So we have a new gadget in our beloved configuration Swiss Army knife!

We liked Joaquín and his module so much that we are revamping the YaST development tutorial to be based on his module (instead of yast2-journal). He is already working on that, so hopefully we will have Joaquín around quite some time still. 😉

Unify license handling screens

We got a report about the license agreement screen in automatic installation (AutoYaST) being different to the one showed during common installation. So we decided to take a look to the problem and unify them. We are in a quite late phase of the development process of both the next SLE and the next Leap, so we decided to not unify the code but simply adapt one dialog to look like the other. Also we are after string freeze due to translations, so we had to use a trick and reuse another already translated text. We also took the opportunity to fix some small usability problems.

This is one of those cases in which some images are worth a thousand words, so in order to understand what we did, take a look at the description of this pull request, which includes many images (too many for this blog post).

The new AutoYaST license screen

Smarter check to avoid duplicated repositories

The openSUSE software server defines the online repositories which can be added during installation. The openSUSE DVD also specifies its own online repositories that are always added to the system. And these repositories overlap.

In openSUSE 42.1 it happened that one repository was added twice, even though there was already a check to avoid that. So we investigated why.

We found that the URLs for the problematic repository were not exactly the same, one of them had a trailing slash. Therefore we made the URL comparison more tolerant and if the URLs differ only by the trailing slash, they are still considered the same.

After the fix all repositories are added only once, without any duplicates.

Gem three: we are looking for new teammates!

After 12 sprint reports, most readers would have already realized that the life as a full-time YaST developer is everything but boring… and that we are always pretty busy. The fun and the work are better when you share them so… we are looking for a new hero to join us in our journey.

Even if you don’t feel hacking in YaST would be your thing, maybe you are interested in any of the other jobs at SUSE.

Improved documentation about YaST environment variables

The behaviour of YaST can be affected by several environment variables, but not all of them are well known by everybody. During this sprint we also decided to invest some time documenting them better. The resulting document will be soon properly integrated in our centralized documentation for developers, but you can sneak it already here.

Branching Tumbleweed and the upcoming stable releases

Most of the features and bug-fixes we have blogged about in the last months were incorporated to Tumbleweed, the upcoming Leap 42.2 and the future SLE 12-SP2, since we always try to keep those three codebases as close as possible to each other.

Now Leap 42.2 and SLE 12-SP2 are close enough to their release date, so we plan to be more conservative with the changes. At the end of this sprint we decided to branch the code for Tumbleweed and for the stable siblings. From now on, most exciting stuff will appear only in Tumbleweed, with SLE 12-SP2 and Leap 42.2 becoming more and more boring.

And the wheel keeps on turning

So that was a very minimal selection of the most interesting stuff from the just finished sprint. What comes next? Another sprint, of course! We have already planned some interesting stuff for it, like integrating the new partitioning proposal into the installer or finishing the ultra-cool UI designer that was started during latest Hack Week.

As always, you can follow development in a daily basis in the usual channels (#yast IRC channel and the yast-devel mailing list) or wait another three weeks for the next sprint report. Meanwhile… have a lot of fun!