This is the fifth report since we started blogging about our development sprints and we have to admit that is the less impressive one so far. We probably underestimated the impact of the combination of Easters holidays and vacations of some team members.

But although we were less productive than expected, we still have a couple of cool things to show to our beloved users and fellow developers.

Handling of file conflicts in packages

Until now the package installation in YaST ignored possible file conflicts in the installed packages. In contrast zypper already supports that check for some time.

File conflicts happen when two packages attempt to install files with the same name but different contents. If such conflicting packages are installed the conflicting files will be replaced losing the previous content. The final file content will also depend on the installation order so some issues might look “random”. The package which file has been overwritten is actually broken.

YaST now displays a confirmation dialog which asks whether to continue with installation despite the conflicts or abort. Previously YaST silently continued with the package installation which could cause serious troubles later.

File conflicts should normally not happen, at least when you use the original distribution repositories. The OBS build checks for some file conflicts during package build and if there really is a file conflict that it should be marked on the RPM level (so you should not be allowed to select the conflicting packages for installation at first place).

It is up to the user to decide whether it is OK to ignore the conflict or not. If the conflict is for example in a documentation file then ignoring the conflict is usually no problem, but if the conflict is in a binary file or in a system library then it is potentially risky. If you are not sure “Abort” is the safe choice here.

In the description of this pull request you can see several additional animations showcasing the new feature in a variety of interfaces (Qt, NCurses, command line) and scenarios (software manager, inline installation of extra packages).

Improvements in the installer self-update

During this sprint, the self-update feature has received several improvements and changes. The most important one is that now it uses libzypp to fetch the updates, delegating signatures checking and that kind of stuff. Obviously, it also means that installer updates will be distributed using regular RPM repositories (instead of Driver Update Disks).

On the other hand, user’s driver updates (specified through dud option) will take precedence over installer updates. They will be applied by Linuxrc anyway, but they’ll remain on top of installer updates.

Fun with Ruby and proxies

This is not exactly a new feature or fix in YaST, but something we learned and we decided could be worth sharing in order to save headaches to other people.

We got a report about YaST ignoring the no_proxy setting in /etc/sysconfig/proxy. After some investigation, it turned out the problem was not in YaST but in the underlying tools, that are also implemented in Ruby. Looks like Ruby have an unexpected (by us, at least) behavior dealing with proxy settings. If you are interested in the details, don’t miss the information we collected in this page in the YaST2-Registration wiki, which includes some background and a set of recommendations to follow when setting no_proxy.

Unification of network setup during installation

As a result of the analysis about how the network settings affect different installation modes and steps, we unified the position and shortcuts of the “Network Setup” button. That affected three installation steps.

A button was added to “Add On Product” to avoid going back and forth just to setup some special configuration for some of our network interfaces.

In the “Disk Activation” step, the button was moved to the top-right corner to be consistent with other steps.

And to round off consistency we also adjusted the keyboard shortcut in the registration screen.

New storage library keeps evolving

This time we don’t have any big headline about the development of the new storage layer. We keep collaborating with experts in our attempt to ensure solid solutions for all situations. In addition to booting experts, the input from Parted guru Petr Uzel was really valuable during this sprint. We took some important decisions about the integration of the new libstorage and libparted and we made progress in implementing a partitioning proposal that ensures a bootable system in all architectures and configurations, backed with highly readable tests and specs.

If time and bug reports permit, we’ll have much more to show after the next sprint… but that would be in three weeks from now. Meanwhile, have a lot of fun!

After three weeks of work, another development sprint is over. So time for another report for our fellow geckos. As usual, quite some time was invested in boring bug fixes and small non-obvious improvements, but we also have some interesting stuff to talk about.

Improved UI for the encrypted partitioning proposal

We wanted to talk about this feature not only because it has a visible impact in the user interface, but also because it’s a great example of collaboration among the different roles present in a Scrum Team. Formerly our Scrum development team was only formed by the developers of the YaST Team at SUSE. For this sprint (and future ones) the Scrum Team has been powered up with the addition of Ken Wimer as UI/UX expert and Jozef Pupava, one of the openQA.opensuse.org and openQA.suse.de operators.

We got a feature request to make encryption more visible in this dialog.

Being software developers used to tools like Vim and Git, we have to admit that the YaST team found the dialog perfectly usable and was having hard times thinking on a better alternative. Fortunately, we now have a UI/UX expert able to bring better alternatives like this one we finally implemented.

This kind of visual changes in the installer used to cause delays in openQA, because adapting the tests while keeping the openQA machinery running is not always trivial. The great news is that it didn’t happen this time because our particular openQA superhero was already watching over our steps all along the process.

So welcome into our Scrum process, Ken and Jozef!

System roles

A new feature was added to the installer making it possible to quickly adjust several settings for the installation with one shot. You can see a detailed description of the feature, including several screenshots and configurations options in this wiki page at the Github repository of yast2-installation. And yes, for the impatient we also have a glorious screenshot!

Storage reimplementation: resizing partitions

We have already explained in previous reports that we are performing an integral rewrite of the code managing partitioning and other storage tasks. During this sprint, the brand new library gained the ability to resize all kind of partitions (Linux, Windows, swap, etc.). It’s nothing that is going to hit the users in the short term but at least we have a couple of screenshots to see the premiere working (yes, we know that screenshots of terminals are not the most fancy stuff).

Installer self-update

Starting on version 3.1.175, YaST is able to update itself during system installation. This feature will help to solve problems in the installer even after the media has been released. That’s a huge step towards improving YaST reliability.

The workflow is pretty simple: during system analysis YaST will search automatically for an update. If such update is found, YaST will download, verify (using GPG) and apply it. Finally, the installation will be resumed using the new version. Nice!

In the future, self update will be enabled by default. However, if for some reason you don’t want such a nice feature, you’re free to disable it. What is more: you can also craft your own update and use it instead the official one passing a custom URL to the installer.

If you’re curious, you can check the technical details.

Storage reimplementation: the search for the perfect bootloader

One of the goals of rewriting the storage layer was to make possible to cope with all the over-complicated requirements involved in the proposal of a good bootloader configuration. This time we don’t want the different scenarios to simply pop-up over time in a bug-report-oriented basis and start aggregating more and more branches to the existing code in order to support every one of those “new” scenarios.

Changes will come, for sure, but we need a solid ground based in experts knowledge to start building a flexible future-proof code to handle partitioning regarding bootloader. Thus, we started a round of contacts with several experts in all the hardware architectures supported by YaST in order to capture all the knowledge from their brains into a set of Ruby classes. It’s still a work in progress since squeezing people’s brains is not always easy, but we already have some preliminary document.

Consolidating continuous integration tools

Continuous integration is a key aspect of software development, specially with methodologies like Scrum. Currently we use both Travis and Jenkins for it. Travis builds the pushed commits and pull requests at GitHub, while Jenkins takes care of the integration with the Open Build Service.

We are investing quite some effort trying to use Jenkins for everything. If you want to know more about the reasons or how the progress is going, check the detailed documentation.

And much more!

As usual, this is just a small sample of the total work delivered by the team during the latest sprint (for Scrum and statistic’s lovers, it represents 34 story points out of a total of 87 delivered ones). Hopefully enough to keep you informed… and if it’s not, you know where you can reach us for more information!

We know you have missed the usual summary from the YaST trenches. But don’t panic, here you got it! As usual, we will only cover some highlights, saving you from the gory details of the not so exciting regular bugfixing.

Package notifications

libzypp has a nice feature that enables packages to display notifications when they’re installed/upgraded. Zypper takes advantages of this feature and shows that information when a package is installed/upgraded. For example, if you install mariadb package, Zypper will inform you about setting up a database root password and so on.

If you installed any of those packages with YaST, you missed that piece of information… until now! Starting on yast2 3.1.175 YaST will show packages notifications.

The only exception is when doing a regular installation (or autoinstallation), as we want to show as few dialogs as possible.

Registration Codes from a USB Stick

During the installation of a SUSE Linux Enterprise product, you are asked for a registration code. Previously you had to remember it and type it by hand. Now the code can be read from USB storage.

Insert a USB stick at installation boot time or at the latest before you proceed from the first installation screen (Language, Keyboard and License Agreement). That stick should contain the registration codes either at /regcodes.txt or at /regcodes.xml. In the registration dialogs, the input fields will be prefilled.

The syntax of the files is as follows. In the file identify the product with the name quoted by zypper search --type product or SUSEConnect --list-extensions (without the /version/architecture part).

regcodes.txt:

SLES    cc36aae1
SLED    309105d4

sle-we  5eedd26a
sle-live-patching 8c541494

regcodes.xml: (xml wins if both xml and txt are present)

<?xml version="1.0"?>
<profile xmlns="http://www.suse.com/1.0/yast2ns" xmlns:config="http://www.suse.com/1.0/configns">
  <suse_register>
    <!-- See https://www.suse.com/documentation/sles-12/singlehtml/book_autoyast/book_autoyast.html#CreateProfile.Register.Extension -->
    <addons config:type="list">
      <addon>
        <!-- Name of add-on as listed by "zypper search --type product" -->
        <name>sle-we</name>
        <reg_code>5eedd26a</reg_code>
      </addon>
      <addon>
        <name>sle-live-patching</name>
        <reg_code>8c541494</reg_code>
      </addon>
      <addon>
        <!-- SLES is not an add-on but listing it here allows for combining
             several base product registration codes in a single file -->
        <name>SLES</name>
        <reg_code>cc36aae1</reg_code>
      </addon>
      <addon>
        <name>SLED</name>
        <reg_code>309105d4</reg_code>
      </addon>
    </addons>
  </suse_register>
</profile>

Lot of Btrfs-related improvements in the expert partitioner

We also invested quite some time improving the support for Btrfs in the expert partitioner. Implementing one requested feature and closing five bugs.

The following animation shows the feature #320296 (user friendly handling of subvolumes) in action, together with the fix to 965279 (Btrfs settings always overridden with default values).

But we have even more screenshots and animations for the improvements in the expert partitioner. In the description of this pull request, you have screenshots displaying the new dialog that was implemented to fix bug#928641. And in this other pull request, you can see in action the fixes for bug#944252 (snapshots were offered for partitions other than root) and for bug#954691 (fstab options being forgotten for Btrfs partitions).

Improving testability of the new storage code

In recent posts, we reported how we are about to refactor the storage subsystem of YaST. The improved partition proposal for installation presented in the previous summary performs a lot of operations – like analyzing what disks are there and what is on each one of them, checking if there already is enough free space and making a best guess on what partitions may be candidates to be removed to make space for a new Linux installation.

If there are many disks with many partitions, this can get complicated really quickly. So we need a reliable way to test it. Thus, we created a testing framework to build fake storage hardware (disks) with fake partitions and file systems. Although it’s fake hardware (we can’t create hard disks out of thin air… yet), it enables us to do unit tests without setting up virtual machines. With those tests we can cover a lot more scenarios that would otherwise be really difficult to test, with one or many disks, with many partitions of different kinds, with a previously existing RAID array or whatever.

One nice thing about the new libstorage is that it operates on “device graphs” that can be transformed into the GraphViz format for easy visualization. Here you have a nice diagram generated by libstorage based on some fake hardware created from this YAML specification.

Better handling of wrong registration code for extensions

We also spent some time improving the usability of the section for registration of extensions and modules. Now if the user selects several modules and the registration of some of them fails, the user will be kindly redirected back to the same dialog but only with inputs for the unregistered ones. From there, they can go back to unselect the failing extensions or retry with different (or even with the same) codes.

Say goodbye to the “receive system mail” checkbox

As the last step of the improvements done to the user creation dialog (see the previous post for more details). We removed the long-ago meaningless checkbox titled “Receive System Mail”. That leaded to the removal of quite some code… and removing code is usually a good thing. 🙂

Many other things

As usual, this is just a short summary with some highlights. Many other stuff was implemented and several other bugs were fixed but, you know, we cannot blog about everything if we want to invest some time in debugging and coding. 🙂

See you in the highlights for next sprint, in around three weeks.

PS.- If you want to be part of the fun, take a look to the YaST-related summer projects we have on the openSUSE mentoring page.

Another three weeks period and another report from the YaST Team (if you don’t know what we are talking about, see highlights of sprint 13 and the presentation post). This was actually a very productive sprint although, as usual, not all changes have such an obvious impact on final users, at least in the short term.

Redesign and refactoring of the user creation dialog

One of the most visible changes, at least during the installation process, is the revamped dialog for creating local users. There is a full screenshots-packed description of the original problems (at usability and code levels) and the implemented solution in the description of this pull request at Github.com.

Spoilers: the new dialog looks like the screenshot below and the openSUSE community now needs to decide the default behavior we want for Tumbleweed regarding password encryption methods. To take part in that discussion, read the mentioned description and reply to this thread in the openSUSE Factory mailing list.

Beyond the obvious changes for the final user, the implementation of the new dialogs resulted in a much cleaner and more tested code base, including a new reusable class to greatly streamline the development of new installation dialogs in the future.

One step further in the new libstorage: installation proposal

In the highlights of the previous sprint, we already explained the YaST team is putting a lot of effort in rewriting the layer that access to disks, partitions, volumes and all that. One important milestone in such rewrite is the ability to examine a hard disk with a complex partitioning schema (including MS Windows partitions, a Linux installation and so on) and propose the operations that need to be performed in order to install (open)SUSE. It’s a more complex topic that it could look at the first glance.

During this sprint we created a command line tool that can perform that task. Is still not part of the installation process and will take quite some time until it gets there, but it’s already a nice showcase of the capabilities of the new library.

Fixed a crash when EULA download fails

If a download error occurred during the installation of any module or extension requiring an EULA, YaST simply exited after displaying a pop-up error, as you can see here.

Now the workflow goes back to the extension selection, to retry or to deselect the problematic extension. Just like this.

Continuous integration for Snapper and (the current) libstorage

Snapper and libstorage now build the Git “master” branch continuously on ci.opensuse.org (S, L), and commit a passing build to the OBS development project (S, L), and if the version number has changed, submit the package to Factory (S, L).

The same set-up works on ci.suse.de (S, L), committing to the SUSE’s internal OBS instance (S, L) and submitting to the future SLE12-SP2 (S, L).

This brings these two packages up to the level of automation that the YaST team uses for the yast2-* packages, and allows releasing simple fixes even by team members who do not work on these packages regularly.

Disk usage stats in installation and software manager: do not list subvolumes

While installing software, YaST provides a visual overview of the free space in the different mount points of the system. With Btrfs and its subvolumes feature, the separation between a physical disk and its mount point is not that clear anymore. That resulted in wrong reports in YaST, like the one displayed in the left bottom corner of this screen.

The actual calculation of free space is performed by libzypp (the Zypper library), YaST only takes care of rendering the result of that calculation in the screen. Thus, we closely collaborated with the Zypper developer, Michael Andres, to teach libzypp how to deal with Btrfs subvolumes. Thanks to his work, with any version of libzypp >= 15.21 (already available in Tumbleweed), you can enjoy an error-free disk usage report.

To ensure we have no regressions, the YaST team also contributed a new test to openQA. See it in action.

Cleanup dependencies in YaST systemd services

We have received several bug reports about problems executing AutoYaST or YasT2-Firstboot in combination with complex network settings or third party services… and even in some simple scenarios. Many of these problems boil down to a single culprit – the dependencies of the YaST related systemd units.

During this sprint we have simplified the unit files for Tumbleweed and SLE, as you can see in this pull request. It’s a small change but with a huge impact and several implications, so a lot of time was invested into testing during the sprint. The problems seem to be gone, but more AutoYaST and YaST-Firstboot testing would be highly appreciated.

Many other things

As usual, this is only a brief collection of highlights of all the job done during the sprint. Using Scrum terminology, this represents only 27 story points out of a total of 81 story points delivered by the team during this sprint. Using more mundane words, this is just a third part of what we have achieved during the last three weeks.

Hopefully, enough to keep you entertained until the next report in other three weeks!

As promised in the previous post on this blog, we’ll try to keep you updated about what is happening in the YaST world. Before Christmas we finished an specially short sprint, interrupted by another successful Hackweek. Although we always reserve some time for bug fixing, the last two sprints has been quite focused in looking into the future, implementing new solutions for old problems and trying to prepare replacements for some legacy stuff we have been carrying on for too long. Here you are the highlights.

SCR replacement

For low level access to the system, YaST uses its own library called SCR, inherited from the old YCP days. It’s used to call scripts and also to read and write files. Its design is showing its age and using it from Ruby is unnecessary complex. We feel SCR is one of the biggest source of confusion for newcomers to YaST development. Last but not least, SCR is only used by YaST which means all the maintenance work fall to us.

We want to use a new approach for the future. For running scripts we plan to use a Ruby gem called Cheetah and for accessing configuration files we plan to rely on Augeas.

Taking some improvements that were needed in YaST2-Bootloader and the drop of perl-bootloader as starting points (or as an excuse, if you prefer), we have used this sprint to develop all the moving parts that will allow us to easily use Cheetah and Augeas within YaST.

For Cheetah, we have contributed two features to the upstream project: chroot support and the ability to provide environment variables.

For Augeas we have developed an object oriented layer called Config Files Api (shortly CFA). The idea of CFA is to provide specific functionality by mean of plugins. Of course, the first plugin we have developed is aimed to manipulation of all the Grub2 configuration files.

The next step will be to integrate these new components into the next versions of YaST2-Bootloader, hitting your Tumbleweed repositories soon. Of course, after all the usual manual and openQA testing.

Libstorage replacement

Another low level layer that has been a constant source of headache for YaST developers is libstorage. We use it -specially in the installer and the partitioner- to access disks, partitions, volumes and all that. Once again, the original design have fundamental flaws that limit us in many ways and we have been dreaming for quite some time about writing a replacement for it.

To make this rewrite fit into the Scrum process, we are using the new redesigned library (find the code at Github and the packages at OBS) to write prototypes for the installer partitioning proposal and for a new partitioning YaST module.

This new module is only intended as a testbed to showcase the development of the new library and to drive its integration process. It’s not intended for end users, but after this sprint it can already do some things that are impossible with the current partitioner and even shows some nice graphs really useful for debugging and verifying the behavior of the library.

If you don’t mind to break your system using unsupported software, you can always fetch the code or the packages.

AutoYaST integration tests

Testing is crucial in software development. And integration tests are a must when you are developing an installer. During the last sprints, we have been developing and improving our own solution for testing AutoYaST-driven installations, consisting on a set of tests and a framework to run them.

The goal for this sprint was to de-couple the tests and the framework. Making it possible to reuse our tests in openQA. As a side effect, we wanted to ease the installation and usage of our testing framework.

Both goals were achieved, now you can install AutoYaST Integration Tests (not a very original name) following the instructions available in the repository and there is already an openQA instance directly running the separately available tests.

Snapper development documentation

Last but not least, as a side effect of the development (and the Scrum principles), we have improved the Snapper’s development documentation. Enjoy it at the usual Snapper repository.

That’s all folks. Next sprint starts next week and will be three weeks long, so expect more news during the first days of February.