Home Home > Programming
Sign up | Login

Deprecation notice: openSUSE Lizards user blog platform is deprecated, and will remain read only for the time being. Learn more...

Archive for the ‘Programming’ Category

Highlights of YaST Development Sprint 94

March 6th, 2020 by

After some time of silent work (our previous blog post was published a month ago), the YaST Team is back with some news about the latest development sprint and some Hack Week experiments. Those news include:

  • Enabling YaST on the Windows Subsystem for Linux
  • Usability improvements for the Online Search, the Partitioner and the Kdump module
  • Better control of overridden sysctl configuration values
  • Improvements in the default selections of the upcoming SLE 15 SP2 installer
  • New features for zSeries mainframes like Secure Boot and I/O devices auto-configuration
  • And, as a bonus, a couple of Hack Week projects related to YaST, Ruby and Crystal

So, as you can see, we have a little bit of everything in the menu, from WSL to mainframes, from new features to small usability improvements, from installation to system fine-tuning… So let’s dive into the details!

Improved compatibility with WSL

Have you ever heard about WSL, the Windows Subsystem for Linux? To be honest, before this sprint we haven’t payed much attention to it either. But as both openSUSE Leap and SUSE Linux Enterprise (SLE) are available to Windows users via WSL images and the 15.2 releases of both distributions are approaching, we decided it was time to dive into WSL to research how it works and how can YaST be useful there.

Setting up an (open)SUSE test system inside a WSL environment was a piece of cake thanks to the excellent documentation at the openSUSE Wiki.

Many components of YaST are useless in WSL because not everything can actually be configured from the Linux system itself and because systemd is not available (we are talking exclusively about WSL1 here). But YaST is still very useful for the initial setup of the system when running the (open)SUSE image for the first time. It can be used to setup the first user, to confirm the license and, in the SLE case, also to register the system. The YaST modules for software management can also be very handy to customize the image at any point after that initial setup.

So far, we have done three changes to improve the experience of executing YaST within WSL.

  • We increased the speed of the initial boot by removing calls to systemd when it is not available.
  • We fixed the registration process for YaST Firstboot.
  • We implemented a feature to explicitly mark YaST modules that work in WSL and show only those modules in the YaST control center.

We also documented all our findings about WSL in this document.

As always, we are hungry for feedback. Please reach out to us and tell us what’s your experience using YaST inside WSL and which modules do you miss the most.

As we announced one month ago, YaST will offer a mechanism to search for packages through all SUSE Linux Enterprise modules, even if they are not registered. This feature, known as package online search, was already available using zypper’s search-packages command or through the SCC web interface.

After gathering some feedback during the sprint review meeting, we decided to invest some time improving the overall UX experience. Perhaps the most relevant change is the new summary screen, which shows the list of modules to activate and packages to install.

New summary screen for the online search

Additionally, we improved error handling and, by the way, we fixed the case sensitive filter.

…And the Partitioner as Well

The online search is not the only part of YaST that has received some love in the UX area. We also tried to improve a bit the usability of the Partitioner. In this occasion, based on the feedback coming from our users via openSUSE’s Bugzilla.

On one hand, we got a report about this dialog been too long to properly fit in low screen resolutions.

Previous dialog for mount options

The result was even worse in a text console with a resolution of 80 columns and 24 lines, which is the minimum size we design all YaST screens to work on.

Previous text-based dialog for mount options

So we dropped some obsolete options and made others more compact. Now the dialog fits in 24 lines again.

Revamped text-based dialog for mount options

And, as you can see below, it looks also nicer (or at least less overwhelming) in graphical mode as well. It’s worth mentioning we also took the opportunity to fix other related dialogs that had similar problems.

Revamped dialog for mount options

On the other hand, we also got a report about how inconvenient was to always jump to the first tab when a device was selected in the devices tree at the left of the Partitioner, forcing the user to click in the “Partitions” tab (or any other desired one) over and over.

In that regard and as you may remember, a couple of sprints ago we made the overview screen actionable, avoiding the navigation to the device page just to perform a simple action over it. But navigating through the different devices back and forth is still possible and useful. Now such navigation has been improved by remembering the last tab and row selected per section or device whenever possible, which will save you a bunch of clicks when working with multiple devices.

Related to this, we started a public discussion about what should be the default tab the first time a device is visited. Once again, we are looking for opinions. So we would be grateful if you read the thread and contribute to the discussion.

Showing Suggested Values for Kdump Configuration

But the Partitioner was not the only YaST module for which our users pointed usability problems via Bugzilla. After some changes in how Kdump works after the migration from openSUSE Leap 42.3 to 15.0, it turned out that using YaST to re-adjust the values was not as helpful as it should be. YaST Kdump displayed the current size of the memory reservations, as well as the min and max margins. But it did not show the recommended default values for the current system, so if the user has adjusted the limits in the past it was impossible to get an up-to-date proposal from YaST calculated for the current system.

We have adapted the dialog to show those suggested values. As you can see below, we also took the opportunity to extend the help text to explain the meaning of the different values.

New Kdump configuration screen

Better Control of Overridden Kernel Parameter Values

And talking about YaST pieces we are improving step by step, you may remember from our report of sprint 86 that we are adapting YaST to deal with the new structure of the sysctl configuration.

Up to now YaST has stored sysctl values mainly in /etc/sysctl.conf and /etc/sysctl.d/70-yast.conf. But this reflects only a part of the possibilities for storing those values. The truth is that there are many more locations where these settings can be stored: /run/sysctl.d, /etc/sysctl.d, /usr/local/lib/sysctl.d, /usr/lib/sysctl.d, /lib/sysctl.d, /etc/sysctl.conf

Now YaST also takes care of these locations and informs the user if there are some conflicting values, as you can see in the following screenshot.

YaST alerting about conflicts in sysctl

The Default Pre-selected SLE Modules

We have also invested some time smoothing some rough edges off the installation process for the upcoming openSUSE 15.2 and SLE 15 SP2. For example, if you register your SLE 15 product during installation you will see the available modules and extensions in the following dialog. Some of them are by default pre-selected because they either contain the base system components (kernel, glibc,…) or the product specific packages (e.g. GNOME for SLE Desktop).

Modules selection during SLE installation

However, if you skip the registration and use the packages from the DVD medium there were no modules or extension pre-selected. The problem is that the information about the default modules was only available in the SCC data which obviously is not available in an offline installation.

In SLE 15 SP2 we added this extra information to the installer configuration files so now also in an offline installation YaST can preselect the default modules for each product.

Proposing NTP Servers During Installation

And talking about offering sensible defaults for installation, we also improved the situation regarding the configuration of the NTP server. For openSUSE based systems (including Kubic) and a few SUSE products, like CaaSP or SLE High Performance Computing, YaST sets up the NTP daemon during installation. YaST tries to determine which server to use through the DHCP information but, when it is not available, it will propose one from openSUSE and SUSE pools (e.g., n.opensuse.pool.ntp.org where n is a number between 0 and 3).

However, we still were using the novell.pool.ntp.org pool for SUSE based products. During this sprint, we have switched to the suse.pool.ntp.org pool of servers and, additionally, we have refactored some code in order to reduce duplication and improve testability.

Secure Boot Support for IBM zSeries

You may have noticed by the recent sprint reports that we are improving several aspects related to the installation and configuration of zSeries mainframes. This sprint was not an exception… and will certainly not be the last one in that regard.

As a result of that effort, YaST now supports the Secure Boot feature found on the latest zSeries machines. It’s rather similar to the existing UEFI Secure Boot so we took the opportunity to unify the Secure Boot handling found on different architectures.

This means you get this checkbox if your zSeries machine does have Secure Boot support.

Configuring Secure Boot in zSeries

In addition, we added a shortcut link on the installation summary screen that lets you enable Secure Boot with just a click.

Secure Boot in the proposal screen

As mentioned, we took the opportunity to unify the management of Secure Boot in all platforms, so this new shortcut link is also available in x86_64 or aarch64 machines that have UEFI Secure Boot.

Automatic Configuration of I/O Devices in zSeries

And talking about zSeries mainframes, anyone having used Linux in one of those systems know that input/output devices, like disks or network cards, must be configured and activated before they can be detected and used normally by the operating system.

But thanks to the new I/O device auto-configuration mechanism, users can now specify IDs and settings of I/O devices that should be automatically enabled in Linux. We modified the installer to detect such configuration and trigger the corresponding configuration actions, removing the need of manually activating disks and network devices during the installation process.

This is still an experimental feature and we are waiting for feedback to make sure the current implementation works in all the desired scenarios. If everything goes as expected, the feature will debut in SLE 15 SP2.

Hack Week

As said at the beginning of the post, the main reason for spending almost a month without publishing any report was that the whole YaST Team at SUSE was diving into completely different topics due to Hack Week 19, which theme was “Simplify, Modernize and Accelerate”.

There were not many projects related to YaST in this edition of Hack Week, but there are at least two that could be interesting for YaST fans and contributors. Fortunately, we have published reports for both of them in the yast-devel mailing list. So check out the results of “Learn Crystal by Porting Part of YaST to that Language” and “YaST Logs Analyzer“.

More to come

Now that we are back to our usual development pace, we should have more news about YaST development in a couple of weeks. The plan is to focus on fixing bugs for the upcoming releases of openSUSE Leap and SUSE Enterprise Linux, but we are pretty sure we will still find interesting bits of information for you.

Meanwhile, keep in touch through the usual channels and have a lot of fun!

Highlights of YaST Development Sprint 91

December 18th, 2019 by

The last two weeks of the year, and also the first one of the new year, are vacation season in many parts of the world and YaSTland is not an exception. But before we enter hibernation mode, let’s take a look to the most important features and bugfixes we implemented in the last sprint of 2019. That includes:

  • bringing back to life some sections of the Software Manager,
  • implementing system upgrade with the new SLE media types,
  • making the installation in Raspberry Pi and IBM Z System even better,
  • improving usability of encryption,
  • reducing the footprint of the Snapper plugin for ZYpp,
  • as always, many other small improvements and fixes.

Restored Some Package Views: Recommended, Suggested, etc.

Let’s start with a redemption story. Some time ago we implemented feature fate#326485 which requested dropping the “Package Groups” view from the package manager UI. That was quite an easy task.

However, a few weeks later we got a bug report that the lists of recommended, suggested, etc… packages couldn’t be displayed anymore. It turned out that, in the Qt package manager front-end, the removed “Package Groups” view not only used to display the static group data from the packages but it also contained some special computed package lists like orphaned, suggested or recommended packages. So these lists were lost as a collateral damage of removing the “Package Groups” view.

The ncurses package manager was not affected by the same problem because, in that front-end, those views are grouped in a separate “Package Classification” section. So the task for this sprint was to somehow revive the lists in Qt and make them again available to the users.

We partly reverted the Package Groups removal and restored displaying those special package groups. To make it consistent we also use the “Package Classification” name for the view, like in the ncurses package manager.

The new Package Classification view in Qt

On the other hand, the ncurses front-end was missing some lists like the “Multiversion Packages” and “All Packages”. To take consistency another step further, we added these missing lists and did some small cleanup and fixes so now both the Qt and the ncurses package managers should offer the same functionality and should look similar.

The revamped Package Classification view in ncurses

User-friendly Encryption Device Names

And talking about bug reports that trigger some usability revamp, some users had pointed that, when the system is booting and prompts for the password of an encrypted device, it’s not always that easy to identify which exact device it is referring to:

Booting password prompt before the change

The root of the problem is that when YaST creates an encryption device (during the installation by means of the storage proposal, or manually with the Expert Partitioner), the device mapper name for the new encrypted device is generated from the udev id of the underlying device (e.g., cr_ccw-0XAF5E-part2).

We decided to improve the encryption naming generation in YaST for Tumbleweed and future releases of Leap and SLE. From now on, the name will be based on the mount point of the device. For example, if an encrypted device is going to be mounted at root, its device mapper name would be cr_root. In general, when the encrypted device is mounted, the device mapper name would be cr_mount_point (e.g., cr_home_linux for an encrypted device mounted at /home/linux).

Booting password prompt after the change

Note that udev-based names might still be used for some scenarios. For example, when the device is not mounted or for an indirectly used encrypted device (e.g., an encrypted LVM Physical Volume).

And related to the identification of encryption devices, we have also added more information about the device when the encryption devices are activated during the installation process. Providing the password for the correct device was very difficult because the user needed to know the UUID of the encryption device. Now on, the activation popup also informs about the kernel name of the underlying device, making much easier to identify it.

New password prompt during installation

Because names matter… which leads us to the next topic.

How does it Feel to Run a Mainframe?

As you may know, (open)SUSE runs in a vast range of hardware, including powerful mainframes like the IBM Z family. One of the strengths of our beloved distributions is that, despite the differences in hardware and scope, the installation and usage experience is very similar in all the supported systems.

Consistency and ease of use are good, but when you drive a luxury car you want to see the brand’s badge on top of the hood. So in future versions of the installer, the model of the machine will be displayed when installing in an IBM Z system. See the right-top corner of the following screenshot.

IBM Z Model in the installer

The text-based installer also has been modified to include the same banner in a similar place.

IBM Z Model in the text-mode installer

But in the same way that (open)SUSE enables you to install and use Linux in a mainframe “just like in any other computer”, we also target to do the same in the other extreme of the hardware spectrum.

Better Support for Raspberry Pi in the Partitioning Proposal

One year ago we announced that openSUSE Leap 15.1 and SLE-15-SP1 would be the first Linux distributions that could be installed in Raspberry Pi devices following the standard installation procedure, instead of deploying a Raspberry-specific pre-built image. The only prerequisite was the existence in the target SD card (or disk) of a partition containing the Raspberry Pi boot code.

But we are now able to go one step further for SLE-15-SP2 (and Leap 15.2). Thanks to the technologies included in those upcoming releases, (open)SUSE will not longer need a separate partition with the boot code in all cases. Now the installer can make a reasonable installation proposal in all situations, even if the target storage device doesn’t contain a booting partition in advance. See, for example, what the installer suggests by default for installing a fully standard SLE-15-SP2 Beta1 in a 32 GiB SD card that contained initially a GPT partition table (tip: GPT partition tables cannot be used to boot in a Raspberry Pi device… and the installer knows it).

Installer proposal for a Raspberry Pi

With that, the installation of the standard SLE-15-SP2 Beta1 (the aarch64 version, of course) in a Raspberry Pi 3 or 4 is as easy as “next”, “next”, “next”… with the only exception of a couple of packages that must be manually selected for installation (raspberrypi-* and u-boot-rpi3). Hopefully, future beta images of both SLE and openSUSE Leap 15.2 will select those packages automatically when installing in a Pi, which will make the (open)SUSE experience in those devices basically identical to any other computer.

SLE Upgrade with the New Media Types

And talking about the standard installation images of the upcoming SLE-15-SP2, we explained in our previous blog post that those versions of SUSE Linux Enterprise (SLE) and all its associated products will be distributed in two new kinds of media – Full and Online. The Full Media contains many repositories and the system can be installed without network connectivity. The Online Media is similar to the openSUSE’s net installer, it contains no repository and it must download everything from the network. The big difference with openSUSE is that SLE systems need to be registered in order to have access to remote repositories.

But apart from installation, those two new media types can also be used to upgrade an existing system… at least after all the improvements implemented during the latest sprint.

In the case of the Online Media, if the system is registered the upgrade process will switch all repositories to point to their corresponding versions at the SUSE Customer Center (SCC) and will get the new software from there. If the system is not registered, the upgrade process is cancelled and the user is advised to either register the system or use the Full Media.

The Full Media can be used to upgrade any system, registered or not, but the process is different in each case. For a non-registered system, the repositories will be switched to the ones included in the media and the system will be upgraded from there. For registered systems the process is the same that with the Online Media, so the software will be fetched from the remote repositories at the SUSE Customer Center.

Last but not least, we also made sure the process with both medias works with an AutoYaST upgrade (yes, you can also use AutoYaST to perform an unattended upgrade, in addition to the better known unattended installation). For a registered system, we simplified the procedure as much as possible and it only needs access to SCC and an empty AutoYaST profile. For non-registered systems it is a little bit more complex because the profile must specify which repositories from the media should be used for the upgrade. But other than that, the process works quite smooth.

And, of course, we used the opportunity to improve the unit test coverage of the code and to improve the documentation, including the profiles we used for testing.

The Snapper Plugin for ZYpp Becomes More Compact and Future-proof

Snapper lets you make filesystem snapshots. It has a companion, snapper-zypp-plugin, a plugin for ZYpp that makes snapshots automatically during commits. See the “zypp” descriptions in this listing:


# snapper list

  # | Type   |Pre # | Date                     | User | Used Space | Cleanup  | Description  | Userdata     
----+--------+------+--------------------------+------+------------+----------+--------------+-------------
  0 | single |      |                          | root |            |          | current      |              
[...]
824 | pre    |      | Tue Dec 17 10:00:27 2019 | root |  16.00 KiB | number   | zypp(zypper) | important=no
826 | post   |  824 | Tue Dec 17 10:02:19 2019 | root |  16.00 KiB | number   |              | important=no
827 | single |      | Tue Dec 17 11:00:01 2019 | root |  16.00 KiB | timeline | timeline     |             
828 | single |      | Tue Dec 17 11:00:01 2019 | root |  16.00 KiB | timeline | timeline     |             

To make our enterprise products supportable for a looong time, we have rewritten this plugin to C++, starting with snapper-0.8.7. (The original Python implementation is not dead, it is resting in old Git commits.)

As a result, Python regular expressions are no longer supported in the /etc/snapper/zypp.conf file. POSIX extended regular expressions work instead, which should work sufficiently well for the purpose of package name matching. Shell patterns continue working unchanged.

Happy new year!

During the following three weeks, the YaST team will interrupt the usual sprint-based development pace. That also means, almost for sure, that we will not publish any blog post about the development of YaST until mid January of 2020. So we want to take this opportunity to wish you a happy new year full of joy and Free Software.

See you soon and make sure to start the year with a lot of fun!

openSUSE on reproducible builds summit

December 13th, 2019 by

As in the past 3 years, I joined the r-b summit where many people interested in reproducible builds met.

There were several participants from companies, including Microsoft, Huawei and Google.
Also some researchers from universities that work on tools like DetTrace, tuf and in-toto.
But the majority still came from various open-source projects – with Fedora/RedHat being notably absent.

We had many good discussion rounds, one of which spawned my writeup on the goal of reproducible builds

Another session was about our wish to design a nice interface, where people can easily find the reproducibility status of a package in various distributions. I might code a Proof-of-Concept of that in the next weeks (when I have time).
I also got some help with java patches in openSUSE and made several nice upstream reproducibility fixes – showing some others how easy that can be.

This whole event also was good team-building, getting to know each other better. This will allow us to better collaborate in the Future.

Later there will be a larger report compiled by others.

Highlights of YaST Development Sprint 87

October 23rd, 2019 by

It’s time for another YaST team report! Let’s see what’s on the menu today.

  • More news and improvements in the storage area, specially regarding encryption support.
  • Some polishing of the behavior of YaST Network.
  • New widgets in libYUI.
  • A look into systemd timers and how we are using them to replace cron.
  • And a new cool tool for developers who have to deal with complex object-oriented code!

So let’s go for it all.

Performance Improvements in Encrypted Devices

As you may know, we have recently extended YaST to support additional encryption mechanisms like volatile encryption for swap devices or pervasive encryption for data volumes. You can find more details in our blog post titled "Advanced Encryption Options Land in the YaST Partitioner".

Those encryption mechanisms offer the possibility of adjusting the sector size of the encryption layer according to the sector size of the disk. That can result in a performance boost with storage devices based on 4k blocks. To get the best of your systems, we have instructed YaST to set the sector size to 4096 bytes whenever is possible, which should improve the performance of the encrypted devices created with the recently implemented methods.

Additionally, we took the time to improve the codebase related to encryption, based on the lessons we learned while implementing volatile and pervasive encryption. We also performed some additional tests and we found a problem that we are already fixing in the sprint that has just started.

Other improvements related to encryption

One of those lessons we have learnt recently is that resizing a device encrypted with a LUKS2 encryption layer works slightly different to the traditional LUKS1 case. With LUKS2 the password must be provided in the moment of resizing, even if the device is already open and active. So we changed how libstorage-ng handles the passwords provided by the user to make it possible to resize LUKS2 devices in several situations, although there are still some cases in which it will not be possible to use the YaST Partitioner to resize a LUKS2 device.

As a side effect of the new passwords management, now the process that analyzes the storage devices at the beginning of the installation should be more pleasant in scenarios like the one described in the report of bug#1129496, where there are many encrypted devices but the user doesn’t want to activate them all.

And talking about improvements based on our users’ feedback, we have also adapted the names of the new methods for encrypting swap with volatile keys, as suggested in the comments of our already mentioned previous blog post. We also took the opportunity to improve the corresponding warning messages and help texts.

New name and description for encryption with volatile keys

Network and Dependencies Between Devices

Similar to encryption, the network backend is another area that needed some final adjustments after the big implementation done in the previous sprints. In particular, we wanted to improve the management of devices that depend on other network devices, like VLANs (virtual LANs) or bridges.

Historically, YaST has simply kept the name of the device as a dependency, even if such device does not exist any longer. That leaded to inconsistent states. Now the dependencies are updated dynamically. If the user renames a device, then it’s automatically renamed in all its dependencies. If the user deletes a device that is needed by any other one, YaST will immediately ask the user whether to modify (in the case of bonding and bridges) or to remove (in the case of VLANs) those dependent devices.

New libYUI Widget: ItemSelector

Now that we mention the user experience, it’s fair to note that it has been quite a while since we created the last new widget for libYUI, our YaST UI toolkit. But we identified a need for a widget that lets the user select one or many from a number of items with not only a short title, but also a descriptive text for each one (and optionally an icon), and that can scroll if there are more items than fit on the screen.

So say hello to the new SingleItemSelector.

SingleItemSelector in graphical mode

As you would expect from any libYUI widget, there is also a text-based (ncurses) alternative.

SingleItemSelector in text mode

Please, note the screenshots above are just short usage examples. We are NOT planning to bring back the desktop selection screen. On the other hand, now we have the opportunity to make a prettier screen to select the computer role. Stay tuned for more news about that.

There is also an alternative version of the new widget that allows to select several items. The unsurprisingly named MultiItemSelector.

MultiItemSelector in graphical mode

Which, of course, also comes with an ncurses version.

MultiItemSelector in text mode

In the near future, we are planning to use that for selecting products and add-on modules. But this kind of widgets will find other uses as well.

Fun with Systemd Timers

And talking about the close future, many of you may know there is a plan coming together to replace the usage of cron with systemd timers as the default mechanism for (open)SUSE packages to execute periodic tasks.

In our case, we decided to start the change with yast2-ntp-client, which offers the possibility to synchronize the system time once in a while. So let’s take a look to how systemd timers work and how we used them to replace cron.

When defining a service in systemd it is possible to specify a type for that service to define how it behaves. When started, a service of type oneshot will simply execute some action and then finish. Those services can be combined with the timers, which invoke any service according to monotonous clock with a given cadence. To make that cadence configurable by the user, the YaST module overrides the default timer with another one located at /etc/systemd/system.

As a note for anyone else migrating to systemd timers, our first though was to use the EnvironmentFile directive instead of overriding the timer. But that seems to not be possible for timers.

One clear advantage of using a systemd service to implement this is the possibility of specifying dependencies and relations with other services. In our case, that allows us to specify that one time synchronization cannot be used if the chrony daemon is running, since they would both conflict. So the new system is slightly more complex than a one-liner cron script, but it’s also more descriptive and solid.

And another tip for anyone dealing with one-shot services and systemd timers, you can use systemd-cat to catch the output of any script and redirect it to the systemd journal.

Everybody Loves Diagrams

But apart from tips for sysadmins and packagers, we also have some content for our fellow developers. You know YaST is a huge project that tries to manage all kind of inter-related pieces. Often, the average YaST developer needs to jump into some complex module. Code documentation can help to know your way around YaST internals that you don’t work with every day. To generate such documentation, we use the YARD tool, and its output is for example here, for yast-network. Still, for large modules with many small classes, this is not enough to get a good overview.

Enter yard-medoosa, a plugin for YARD that automatically creates UML class diagrams, clickable to get you to the classes textual documentation.

The yast2-network medoosa

It is still a prototype but it has proven useful for navigating a certain large pull request. We hope to soon tell you about an improved version.

More Solid Device Names in fstab and crypttab

Back to topics related to storage management, you surely know there are several ways to specify a device to be mounted in the /etc/fstab file or a device to be activated in the /etc/crypttab. Apart from using directly the name of the device (like /dev/sda1) or any of its alternative names based on udev, you can also use the UUID or the label of the file-system or of the LUKS device.

By default, YaST will use the udev path in s390 systems and the UUID in any other architecture. Although that’s something that can be configured modifying the /etc/sysconfig/storage file or simply using this screen of the Partitioner, which makes possible to change how the installation (both the Guided Setup and the Expert Partitioner) writes the resulting fstab and crypttab files.

Changing the way devices are referenced

But, what happens when the default option (like the udev path) is not a valid option for some particular device? So far, YaST simply used the device name (e.g. /dev/sda1) as an immediate fallback. That happened at the very end of the process, when already writing the changes to disk.

We have improved that for Tumbleweed, for SLE-15-SP1 (which implies Leap 15.1) and for the upcoming versions of (open)SUSE. Now, if the default value is not suitable for a particular device because the corresponding udev path does not exists, because using a given name is incompatible with the chosen encryption method, or for any other reason, YaST will fall back to the most reasonable and stable alternative. And it will do it from the very beginning of the process, being immediately visible in the Partitioner.

Stay Tuned for More… and Stay Communicative

As usual, when we publish our sprint report we are already working on the next development sprint. So in approximately two weeks you will have more news about our work, this time likely with a strong focus in AutoYaST.

Don’t forget to keep providing us feedback. As commented above, it’s very valuable for us and we really use it as an input to plan subsequent development sprints.

Advanced Encryption Options Land in the YaST Partitioner

October 9th, 2019 by

Welcome to a new sneak peek on the YaST improvements you will enjoy in SLE-15-SP2 and openSUSE Leap 15.2… or much earlier if you, as most YaST developers, are a happy user of openSUSE Tumbleweed.

In our report of the 84th sprint we mentioned some changes regarding the encryption capabilities of the YaST Partitioner, like displaying the concrete encryption technology or the possibility to keep an existing encryption layer.

Keeping the previous encryption layer

And the report of sprint 85 contained a promise about a separate blog post detailing the new possibilities we have been adding when it comes to create encrypted devices.

So here we go! But let’s start with a small disclaimer. Although some of the new options are available for all (open)SUSE users, it’s fair to say that this time the main beneficiaries are the users of s390 systems, which may enjoy up to four new ways of encrypting their devices.

Good Things don’t Need to Change

As you may know, so far the YaST Partitioner offered an “Encrypt Device” checkbox when creating or editing a block device. If such box is marked, the Partitioner asks for an encryption password and creates a LUKS virtual device on top of the device being encrypted.

LUKS (Linux Unified Key Setup) is the standard for Linux hard disk encryption. By providing a standard on-disk-format, it facilitates compatibility among distributions. LUKS stores all necessary setup information in the partition header, enabling to transport or migrate data seamlessly. So far, there are two format specifications for such header: LUKS1 and LUKS2. YaST uses LUKS1 because is established, solid and well-known, being fully compatible with the (open)SUSE installation process and perfectly supported by all the system tools and by most bootloaders, like Grub2.

You should not fix what is not broken. Thus, in most cases, the screen for encrypting a device has not changed at all and it still works exactly in the same way under the hood.

Editing an encrypted device

But using an alternative approach may be useful for some use cases, and we wanted to offer an option in the Partitioner for those who look for something else. So in some special cases that screen will include a new selector to choose the encryption method. Let’s analyze all those new methods.

Volatile Swap Encryption with a Random Key

When a swap device has been marked to be encrypted, the user will be able to choose between “Regular LUKS1” and “Volatile Encryption with Random Key”. Both options will be there for swap devices on all hardware architectures. The first option simply uses the classical approach described above.

Selecting the encryption method

The second one allows to configure the system in a way in which the swap device is re-encrypted on every boot with a new randomly generated password.

Encrypt swap with a random password

Some advanced users may point that configuring such a random encryption for swap was already possible in versions of openSUSE prior to Leap 15.0. But the procedure to do so was obscure to say the least. The encryption with a random password was achieved by simply leaving blank the “Enter a Password” field in the encryption step. The exact implications were not explained anywhere in the interface and the help text didn’t mention all the risks.

Now the same configuration can be achieved with a more explicit interface, relevant information is provided as you can see in the screenshot below and some extra internal controls are in place to try to limit the potential harm.

Help text for the random swap method

With this approach, the key used to encrypt the content of the swap is generated on every boot using /dev/urandom which is extremely secure. But you can always go a bit further…

Swap Encryption with Volatile Protected AES Keys

One of the nice things about having a mainframe computer (and believe us there are MANY nice things) is the extra security measures implemented at all levels. In the case of IBM Z or IBM LinuxONE that translates into the so-called pervasive encryption. Pervasive encryption is an infrastructure for end-to-end data protection. It includes data encryption with protected and secure keys.

In s390 systems offering that technology, the swap can be encrypted on every boot using a volatile protected AES key, which offers an extra level of security compared to regular encryption using data from /dev/urandom. This document explains how to setup such system by hand. But now you can just use YaST and configure everything with a single click, as shown in the following screenshot.

Encrypting swap with volatile protected keys

The good thing about this method is that you can use it even if your s390 system does not have a CCA cryptographic coprocessor. Oh, wait… you may not know what a cryptographic coprocessor is. Don’t worry, just keep reading.

Pervasive Encryption for Data Volumes

Have you ever wondered how James Bond would protect his information from the most twisted and resourceful villains? We don’t know for sure (or, at least, we are supposed to not disclosure that information), but we would bet he has an s390 system with at least one Crypto Express cryptographic coprocessor configured in CCA mode (shortly referred as a CCA coprocessor).

Those dedicated pieces of hardware, when properly combined with CPU with CPACF support, make sure the information at-rest in any storage device can only be read in the very same system where that information was encrypted. They even have a physical mechanism to destroy all the keys when the hardware is removed from the machine, like the self-destruction mechanisms in the spy movies!

As documented here, the process to enjoy the full power of pervasive encryption for data volumes in Linux can be slightly complex… unless you have the YaST Partitioner at hand!

Pervasive volume encryption

As you can see in the screenshot above, the process with YaST is as simple as choosing “Pervasive Volume Encryption” instead of the classic LUKS1 that YaST uses regularly for non-swap volumes. If YaST finds in the system a secure AES key already associated to the volume being encrypted, it will use that key and the resulting encryption device will have the DeviceMapper name specified for that key. If such secure keys don’t exist, YaST will automatically register a new one for each volume.

Help for pervasive volume encryption

Pervasive encryption can be used on any volume of the system, even the root partition.

I want it all!

So far we have seen you can use protected AES keys for randomly encrypting swap and registered secure keys for protecting data volumes. But what if you want your swap to be randomly encrypted with a volatile secure AES key? After all, you have already invested time and money installing those great CCA coprocessors, let’s use them also for the random swap encryption!

If your hardware supports it, when encrypting the swap you will see a “Volatile Encryption with Secure Key” option, in addition to the other four possibilities commented above. As easy as it gets!

All possible methods for encrypting a swap

More Booting Checks in non-s390 Systems

As described in the help for pervasive volume encryption showed above, that encryption method uses LUKS2 under the hood. So we took the opportunity to improve the Partitioner checks about encryption and booting. Now, in any architecture that is not s390 the following warning will be displayed if the expert partitioner is used to place the root directory in a LUKS2 device without a separate plain /boot.

New check for booting

As mentioned, that doesn’t apply to s390 mainframes. The usage of zipl makes possible to boot Linux in those systems as long as the kernel supports the encryption technology, independently of the Grub2 (lack of) capabilities.

What’s next?

We are still working to smooth off the rough edges of the new encryption methods offered by YaST and to add AutoYaST support for them. You may have noticed that most of the improvements currently implemented will only directly benefit s390 systems… even just a subset of those. But at the current stage, we already have built the foundation for a new era of encryption support in YaSTland.

We are thinking about adding more encryption methods that could be useful for all (open)SUSE users, with general support for LUKS2 being an obvious candidate. But that’s not something we will see in the short term because there are many details to iron up first in those technologies to make then fit nicely into the current installation process.

But hey, meanwhile you can play with all the other new toys!

Highlights of YaST Development Sprint 84

September 16th, 2019 by

The YaST Team finished yet another development sprint last week and we want to take the opportunity to let you all glance over the engine room to see what’s going on.

Today we will confess an uncomfortable truth about how we manage the Qt user interface, will show you how we organize our work (or at least, how we try to keep the administrative part of that under control) and will give you a sneak peak on some upcoming YaST features and improvements.

Let’s go for it!

There Be Dragons: YaST Qt UI Event Handling

The YaST Qt UI is different in the way that it uses Qt. Normal Qt applications are centered around a short main program that after initializing widgets passes control to the Qt event loop. Not so YaST: it is primarily an interpreter for the scripts (today in Ruby, in former times in YCP) that are executed for the business logic. Those scripts, among other things, also create and destroy widget trees. But the control flow is in the script, not in a Qt event loop. So YaST uses different execution threads to handle both sides: graphic’s system events for Qt widgets and the control flow from the scripts.

This was always quite nonstandard, and we always needed to do weird things to make it happen. We always kind of misused Qt to hammer it into shape for that. And we always feared that it might break with the next Qt release, and that we might have a hard time to make it work again.

This time has now come with bug#1139967, but we were lucky enough to find a Qt call to bring it back to life; a QEventLoop::wakeUp() call fixed it. We don’t quite know (yet) why, though. Any hint, anyone?.

Should we even tell you that? Well, we think you deserve to know. After all, it worked well for about 20 years (!)… and now it’s working again.

The Refactoring of YaST Network Keeps Going

What is still not working that fine is the revamped network management. We have been working on it during the latest sprint, but it will take at least another one before it’s stable enough to be submitted to openSUSE Tumbleweed.

On which parts have we be working during the this sprint? Glad you asked! 😉 We are cleaning the current mess in wireless configuration. Soon that part will be more intuitive and consistent with other types of network. We are also making sure we propose meaningful defaults for the new cards added to the network configuration (all types of cards, not only wireless). The functionality to configure udev in order to enjoy stable names for the network interfaces has also received some love. The new version is more stable and flexible. And last but not least, we are improving the device activation in s390 systems for it to be more straightforward in the code and more clear in the user interface.

If everything goes as planned, by the end of the next sprint we will be ready to submit the improved YaST Network to Tumbleweed. That will be the right time for a dedicated blog post with screenshots and further explanations of all the changes.

Enhancing the Partitioner Experience with Encrypted Devices

And talking about ongoing work, we are currently working to broaden the set of technologies and use-cases the Partitioner supports when it comes to data encryption. As with the network area, the big headlines in that regard will have to wait for future blog posts. But if you look close enough to the user interface of the Partitioner available in Tumbleweed you can start spotting some small changes that anticipate the upcoming new features.

The first change is very subtle. When visualizing the details of an encrypted device, next to the previously existing “Encrypted: Yes” text you will now be able to see the concrete type of encryption. For all devices encrypted using YaST, that type will always be LUKS1 since that’s the only format that YaST has supported… so far. 😉

Partitioner: show the type of encryption

Some other small changes are visible when editing an encrypted device. If such a device was not originally encrypted in the system, nothing changes apart from minor adjustments in the labels. The user simply sees a form with an empty field to enter the password.

Encrypting a plain device

When editing for a second time a device that was already marked for encryption during the current execution of the Partitioner, the form is already prefilled with the password entered before. In the past, the previous encryption layer was ditched (so it’s password and other arguments were forgotten) and the user had to define the encryption again from scratch. That will become even more relevant soon, when the form for encryption becomes more than just a password field. Stay tuned for news in that regard.

Editing an encrypted device

Moreover, when editing a device that is already encrypted in the system, an option is offered to just use the existing encryption layer instead of replacing it with a (likely more limited) encryption created by the Partitioner.

Keeping the previous encryption layer

Apart from opening the door for more powerful and relevant changes in the area of encryption, these changes represent an important usability improvement by themselves.

Tidying up the YaST Team’s Trello Board

As you can see in this report and in all the previous ones, the YaST Team works constantly on many different areas like installation, network management, storage technologies… you name it. We use Trello to organize all that work. For each bug in Bugzilla or feature in Jira there is a corresponding Trello card. As it happens, sometimes when a bug is closed its Trello card is forgotten to be updated.

A check with ytrello revealed that, out of a total of 900-something cards, about 500 were outdated and could be closed. More than the half! Why so many?

We found that quite a number of these cards were open cards in closed (archived) lists. So when you archive a list, don’t forget to archive its cards before. We have just learned that Trello does not do this automatically. That’s exactly why there’s a menu item Archive All Cards in This List... besides Archive This List in the Trello user interface.

Back to work!

This has been a summer of promises on our side. We told you we are working to improve our user interface library (libYUI), revamping the code to manage the network configuration, extending the support for encryption… which means we have a lot work to be finished.

So let us go back to work while you do your part – having a lot of fun!

Highlights of YaST Development Sprint 82

August 14th, 2019 by

July and August are very sunny months in Europe… and chameleons like sun. That’s why most YaST developers run away from their keyboards during this period to enjoy vacations. Of course, that has an impact in the development speed of YaST and, as a consequence, in the length of the YaST Team blog posts.

But don’t worry much, we still have enough information to keep you entertained for a few minutes if you want to dive with us into our summer activities that includes:

  • Enhancing the development documentation
  • Extending AutoYaST capabilities regarding Bcache
  • Lots of small fixes and improvements

AutoYaST and Bcache – Broader Powers

Bcache technology made its debut in YaST several sprints ago. You can use the Expert Partitioner to create your Bcache devices and improve the performance of your slow disks. We even published a dedicated blog post with all details about it.

Apart of the Expert Partitioner, AutoYaST was also extended to support Bcache devices. And this time, we are pleased to announce that … we have fixed our first Bcache bug!

Actually, there were two different bugs in the AutoYaST side. First, the auto-installation failed when you tried to create a Bcache device without a caching set. On the other hand, it was not possible to create a Bcache with an LVM Logical Volume as backing device. Both bugs are gone, and now AutoYaST supports those scenarios perfectly.

Configuring Bcache and LVM with AutoYaST

But Bcache is a quite young technology and it is not free of bugs. In fact, it fails when the backing device is an LVM Logical Volume and you try to set the cache mode. We have already reported a bug to the Bcache crew and (as you can see in the bug report) a patch is already been tested.

Enhancing Our Development Documentation

This sprint we also touched our development documentation, specifically we documented our process for creating the maintenance branches for the released products. The new branching documentation describes not only how to actually create the branches but also how to adapt all the infrastructure around (like Jenkins or Travis) which requires special knowledge.

We will see how much the documentation is helpful next time when somebody has to do the branching process for the next release. 😉

Working for a better world YaST

We do our best to write code free of bugs… but some bugs are smarter than us and they manage to survive and reproduce. Fortunately we used this sprint to do some hunting.

Those are only some examples of the kind of bugs we have fought during this sprint. But checking bug reports has also made us think in the future…

LibYUI in 21st Century

We fixed a bug related to how the focus was managed in text mode after changing any setting via the hyperlinks available in the installation summary.

Installation summary in text mode

The implemented solution is actually not perfect, it’s just the better we can do with our set of widgets. And that was yet another example of such problem – LibYUI is an awesome library that allows us to create interfaces that work in both graphic and text modes, but it has basically not evolved for more than a decade… and it’s time to fix that!

So we have been discussing how to organize our time in the close future to leave some room for innovation and renovation regarding LibYUI and the YaST UI in general. Stay tuned for more news.

August is still not over

The YaST Team will keep working during the rest of the summer sharpening our Linux Swiss army knife. But half of the team is still on vacation or starting their vacation now. So most likely our next report will be here in two weeks and it will also be a light read.

Meanwhile, don’t forget to have a lot of fun!

SUSE Manager and the Partitioning Guided Setup

July 16th, 2019 by

Apart from our usual development sprint reports, we (the YaST Team) sometimes publish separate blog posts to summarize a new feature or to present an idea we are working on. Lately, several of those posts have been focused on new features of the YaST Partitioner, like the support for Bcache or the new Btrfs capabilities. But today it’s the turn of another part of yast2-storage-ng: the partitioning proposal, also known as the Guided Setup.

As you may know, YaST is an universal installer used to configure all the (open)SUSE and derivative products. Moreover, the installer options and steps can be refined even further by each of the system roles available for each product. The goal of this blog post is to present some ideas aimed to add new possibilities in the area of the storage guided proposal for those who configure the installer for a certain product or system role. With that we hope to ease the life for the creators of SUSE Manager, the SUSE’s purpose-specific distribution to manage software-defined infrastructures.

Although many of the presented capabilities will land soon in openSUSE Tumbleweed they will not be used by default. Not only because they are not targeted to the openSUSE use-case, but also because so far this is just a prototype. That means all texts are subject to change and most screens will get some adaptations before being used in a final product… or maybe they will even be completely revamped.

One Guided Proposal to Rule them All

Although the Expert Partitioner can be used to tweak the storage configuration of any SUSE or openSUSE distribution during installation, the installer always tries to offer a reasonable proposal about it. Moreover, the “Guided Setup” button in the “Suggested Partitioning” screen leads to a wizard that can be used to configure some aspects of such a proposal, as shown in the following diagram (some actions have been blurred just to emphasize the fact that the concrete list of actions will change after each execution of the wizard).

Default Guided Setup wizard

The exact behavior of the Guided Setup is different in every product and, potentially, in every system role. Many things can be adjusted by the creators of the product or the role, like the partitions and LVM volumes to be proposed, the options to be offered in the wizard, the default value for every option and much more. But all those possibilities were still not enough in the case of SUSE Manager and its unique approach to organize the storage devices.

The Strange Case of SUSE Manager

First of all, the SUSE Manager documentation suggests to allocate each of several data directories (/var/spacewalk, /var/lib/pgsql, /var/cache and /srv) in its own dedicated disk when installing in a production environment. For such setup to make sense, it’s absolutely crucial to choose the right disk for every data directory taking into account both the size and the speed of the disks.

The documentation also suggests to use LVM in production environments. In order to achieve a clear separation of disks when using LVM, the recommended approach is to set up a separate LVM volume group for each relevant data directory instead of allocating all the logical volumes in the usual single shared “system” group.

So, although it may look overkill when installing SUSE Manager just for evaluation purposes, the preferred setup for a final deployment of the product spreads over up to five disks – one containing an LVM volume group with the usual logical volumes of any Linux system (like the root system and the swap space) and each of the other disks containing additional LVM groups, each one dedicated to a particular data directory.

Last but not least, the SUSE Manager guided setup should never offer the possibility of keeping the preexisting partitions in any of the disks. So the usual questions “Choose what to do with existing Linux/Windows/other partitions” (see the image above) should not even be displayed to the user. The answer is always “remove even if not needed”. Period. 😉

Breaking Down the Problem into Smaller Pieces

We didn’t want to implement a completely different guided proposal for SUSE Manager. Instead, we wanted to merge the main ideas behind its approach into the current configurable system, so other products and roles could use them. We identified three different features that we turned into the corresponding optional configuration settings at disposal of anyone defining a new system role. All the new settings are independent of each other and can be combined in any way to provide a fully customized user experience.

First Piece: Explicit Selection of Disks per Volume

First of all, it was necessary to support letting the user explicitly choose a disk for every partition or LVM volume, unlike the default guided setup which automatically finds the best disk to allocate every partition given the requirements and a set of “candidate disks”. To enable that, now the product or role can choose between two values for the new allocate_volume_mode setting. A value of auto (which is the default to keep backwards compatibility) will result in the already known wizard with up to 4 steps.

  • Select the candidate disks
  • Decide what to do with existing partitions
  • Configure the schema (LVM and/or encryption)
  • Configure each file system

As always, the steps in which there is nothing for the user to decide are skipped so the wizard is usually shorter than four steps.

No surprises so far. But allocate_volume_mode can also be set to device, which will result in the alternative wizard displayed in the following image.

New possible Guided Setup wizard

As you can see, there is no initial step to select the set of disks to be used by the system to automatically allocate the needed partitions. Instead, the following screen allows to explicitly assign a disk to every partition or LVM volume group.

New step to assign volumes and partitions to disks

Second Piece: Enforcing a Behavior about Previous Partitions

No matter which allocate mode is configured (auto or device), there is always one step in which the user is asked what to do with the preexisting partitions in the affected disks. So far, the product defined the default answer for those questions, but the user always had the opportunity to change that default option.

Now, the creator of the product or the system role can disable the setting called delete_resize_configurable which is enabled by default in order to prevent the user from modifying the default behavior. The wizard will include no questions about what to do with existing Windows/Linux/other partitions. In most cases, that will imply a whole step of the wizard to be simply skipped.

Third Piece: Separate Volume Groups for some Directories

The most important setting configured by every system role is the list of so-called volumes. That list includes all the file systems (both mandatory and optional ones) that the guided setup should create as separate partitions or LVM logical volumes. Now it’s possible to specify that a volume could be created in its very own separate LVM volume group using the new attribute separate_vg_name. If any of the volumes defined for the current product and role contains such attribute, the screen for selecting the schema will contain an extra checkbox below the usual LVM-related one.

New checkbox for directories into their own separate LVM

Putting the Pieces Together for SUSE Manager

With all the above, we expanded the toolbox for anyone wanting to configure the (open)SUSE installation experience. Which means now we can fulfill the requirements of SUSE Manager maintainers by just adding separate_vg_name to some volumes, setting delete_resize_configurable to false and adjusting the allocate_volume_mode. With all that, the new SUSE Manager workflow for the guided setup will look like this.

First of all, the user will be able to specify the creation of separate LVM volume groups as suggested in the product documentation.

SUSE Manager setup - first screen

Then a second screen to select which separate file systems should be created and to fine-tune the options for every one of them, if any.

SUSE Manager setup - second screen

And finally a last step to assign the correct disk for every partition or separate volume group, depending on the selections on previous screens. With this step the user can optimize the performance by distributing the disks as explained in the SUSE Manager documentation, allocating the areas that need intensive processing to the faster disks and the greedy directories to the bigger devices.

SUSE Manager setup - third screen

As usual, the list of actions will reflect the selections of the user creating as many LVM structures as requested.

SUSE Manager setup - result

Beyond SUSE Manager

As already mentioned, all the guided proposal features can be combined within a given product in any way. For example, one product could adopt the approach of creating separate LVM volume groups while still sticking to the traditional auto allocate mode. Or a given system role could enforce to never delete any existing partition without allowing the user to change that.

But beyond the “Guided Setup” button, the availability of two different allocate modes brings back one idea that has been floating around since the introduction of Storage-ng – adding a section “Wizards” to the Expert Partitioner. That would allow to combine some manual steps with the execution of any of the two available allocate modes of the guided proposal… or with any other workflow that can be implemented in the future.

As always, we are looking forward any feedback about the new features or the guided partitioning proposal in general. And stay tuned for more news!

Highlights of the Latest YaST Development Sprints

June 25th, 2019 by

May and June have been, so far, interesting months for the YaST Team. We worked hard to polish the last details of the recently released openSUSE Leap 15.1, we attended the openSUSE Conference 2019 (with many fruitful conversations), we shared quite some time together around a table without computers (most of the time, we are a geographically distributed team), many team members enjoyed vacations (it’s spring time in Europe), we organized a Leap 15.1 launch party with technical talks in Gran Canaria… and we ran out of energy to also publish our traditional sprint reports in this blog.

We will try to fix that with this blog post in which we will try to summarize some highlights from the latest three development sprints, namely the 77th, 78th and 79th. So be warned, this is going to be a loooong post.

Support for Multi-device Btrfs File Systems

We have been working steadily during the three sprints in implementing all the necessary bits to offer a good experience installing and upgrading an (open)SUSE system on top of several block devices by means of Btrfs RAID capabilities. That includes support in the Partitioner, in AutoYaST, in the storage guided setup and more.

We decided that all that deserved a separate blog post. You can find here: Getting Further with Btrfs in YaST.

More Improvements for the Partitioner

That blog post mentions a couple of changes in the Partitioner that, although initially motivated by the introduction of multi-device Btrfs, go beyond that scope and are aimed to make the all the lists of devices more useful and informative.

Traditionally the Partitioner used two separate columns “Type” and “FS Type” to describe the function of every device. That was sometimes hard to understand. Moreover, quite often the important information (like the relationship between a partition and its RAID or LVM) was simply missing in those tables.

Traditional devices table in the Partitioner

We have merged those columns into a more informative one that identifies the devices and also gives an overview of the relationship between them at first glance. In addition, the table displaying all the system devices now includes multi-device file systems.

Revamped table of devices

Mitigating CPU vulnerabilities from YaST

If you are interested in security (or simply if you have not been living under a rock) you probably have heard about CPU based attacks like Spectre or Meltdown. The last year has seen a number of these CPU issues, all of them coming along with their own kernel options to change the Linux behavior in order to mitigate the security risks at a price of some performance loss.

However, not all users know what affects their architectures or particular models of CPU and which kernel parameters to use to gain more performance if the security risk is acceptable for them.

For that purpose, a new meta-option called “mitigations” was added to the Linux kernel. It allows to enable and disable at once several of those mitigations that prevent CPU attacks. See more information at this document published by SUSE.

We find that kernel option very useful, so we decided to provide an easy way for users to adjust it. Now the YaST bootloader screen contains a new setting which offers three pre-defined options and even a fourth one to let the users fine-tune the settings on their own. As you can see in the screenshot below, we have included extensive documentation in the help dialog, so you will not need to search for this blog post in the future.

It is also possible to modify this option directly from the installation summary. For that purpose, the “Firewall” section was renamed to “Security” and it now includes the possibility to tweak the CPU mitigation options, alongside the traditional settings for firewall and opening the SSH port.

CPU mitigations in the installation summary

Another success story of (open)SUSE offering a promptly solution for our users to easily adapt their systems to ever-changing complex needs.

Memory Optimizations during Installation

While the release of openSUSE Leap 15.1 was approaching, we got several bug reports stating the YaST installer used to freeze when using only 1GB RAM with online repositories (see bug#1136051).

It turned out that at some point YaST loads the details of all available packages. And that needs a lot of memory if you enable the online repositories during installation. For example the OSS Leap repository contains more than 35.000 binary packages!

The problem was in the YaST internal API accessing the package manager library (libzypp). It did not allow to filter the objects, YaST had to read all objects and then do the filtering in the code. And for each object it returned all attributes, even those which were not needed (like the package description, full RPM name, etc…). All that data required a lot of memory.

To fix that we have introduced new API calls that allow specifying more filters (like return all selected packages, packages from specific repository,…) and you can set which attributes should be returned. If you need to know only the name and the version then you will not get the other useless attributes. And to ease the usage of the new API in YaST we provided a nice object oriented wrapper written in Ruby.

This optimization saves a lot of memory, 1GB of RAM should be enough for future installations with the online repositories, even if they grow even more.

Unfortunately, we were only able to diagnose the problem and provide a solution a couple of weeks before the official release of Leap 15.1. Introducing a change in such a sensitive part of the installer was considered too risky (it would have invalidated many of the tests that had been already performed) so the installer included in openSUSE Leap 15.1 is still memory hungry if online repositories are used. For that release, we simply increased the official memory requirements to 1.5 GiB.

Online Migration from openSUSE Leap 15.1 to SLES15-SP1

For openSUSE Leap 15.0 it was only possible to migrate from Leap to SLES (SUSE Linux Enterprise Server) manually (see the documentation). With Leap 15.1 the goal was to also support a migration using YaST. But we got a bug report saying the online migration from openSUSE Leap 15.1 to SLES15-SP1 displayed a wrong migration summary and didn’t work well.

It turned out that YaST needed some small fixes to support this properly. The main problem was that YaST did not expect that the base product or the package vendor can be changed during online migration, so far it was only possible to upgrade to the next SLE service pack level. But that is fixed now.

Wanna try the migration from openSUSE Leap 15.1 to SLES15-SP1? Then follow these steps.

  1. Install the yast2-registration and yast2-migration packages in the Leap 15.1 installation
  2. Make sure the latest online updates are installed (to install the fixes mentioned above)
  3. Start the YaST registration module and register the openSUSE Leap 15.1 product using your registration key
  4. Then start the YaST migration module, select the migration to SLES15-SP1
  5. (There might be reported some package dependency issues in the migration summary, go to the package manager and resolve them. Usually removing the old openSUSE package is the right solution.)
  6. Start the migration, the SLES packages will be downloaded and installed
  7. At the end the system will be rebooted to start the freshly installed SLES, enjoy! 🙂
  8. (It is recommended to review the orphaned packages, leftovers from the Leap installation, with the command zypper packages --orphaned and possibly remove them.)

From Leap to SLES via YaST

Please note that only minimal server installations of Leap are supported for upgrade, full installations especially with third party packages might not work correctly.

Why Cannot I Read the Logs?

Long time ago, the logs of any Linux system were spread over several files living under the /var/log subdirectory. YaST offers its “System Log” module to inspect those files in a convenient way. Since the introduction of Systemd and its journal, that information has been gradually moved to this new mechanism by default. And YaST offers its “Systemd Journal” module to inspect and query that journal.

Both YaST modules can be executed by any user in the system, not only by root. That’s intentional because both the Systemd journal and the traditional Linux log files can register information targeted to unprivileged users. But there was some room for improvement in the error message displayed by both modules when such users were trying to access to protected information.

This is how the new more explanatory message looks in the “System Log” module.

Explanatory pop-up for log viewer

And this is the extended message for “Systemd Journal”, that now mentions the systemd-journal user’s group.

Improved message in the journal viewer

Another Day at the YaST Office: Adapting to Changes

As we usually remark in our blog posts, a significant part of the work of the YaST Team consist in keeping YaST in sync with the constant changes in the underlying system. Of course, these sprints were not an exception in that regard. Without trying to do an exhaustive list, let’s take a look to some of those adaptations since the mentioned underlying changes may be interesting for some readers.

Turns out Systemd developers has decided to change the list of possible states for Systemd services. The systemd-sigabrt state is obsolete and a new systemd-watchdog one was added. In the YaST team we learned some time ago that the list of Systemd states changes more often than what most people would expect. As a consequence we have an automated check to detect these situations. The bell ringed, we adapted the code and everything keeps working.

Systemd is not the only technology that keeps evolving. Quite some time ago, RMT replaced SMT as the default proxy technologies for the SUSE Customer Center. Although both has coexisted for quite some time, from SLE-15 onward only RMT is offered. Thus we have adapted all the references to SMT that still existed in YaST. From now on, only RMT is mentioned to avoid confusion.

Another common adaptation we have to perform in YaST is adjusting some module when the output of the command it runs under the hood has changed. Recently we found out the developers of the command iscsiadm has decided to use more than one exit code to indicate a successful execution (traditionally, only zero should mean that). After a long discussion, we decided to adapt YaST iSCSI to also be happy with the error code 21. What means for you? That future versions of YaST iSCSI should work faster in some situations, since the confusion will not longer result in a timeout.

YaST Firstboot: a Better Example File

Those were just some example of the many adaptations we did lately for changes in the system. But not all adjustments are motivated by external changes. We also realized the example configuration file provided by YaST Firstboot (at /etc/YaST2/firstboot.xml) needed some love. Due to the nature of YaST Firstboot, that file should be customized before using YaST Firstboot. But providing an example file with three different steps about acceptance of licenses (two of them enabled and the third one disabled) and other inconsistencies was definitely not helping anyone to understand how to use the module.

In fact, the status of that default example configuration and of the documentation managed to confuse even SUSE’s quality assurance team. So we improved the example file shipped in the package to make it more realistic and we also updated the YaST Firstboot documentation to clarify how to use that file. Because not all improvements are always done by coding.

Leap 15.1’s Most Annoying Bug: Create Home as Btrfs Subvolume

As all openSUSE users should know, for every Leap release a page is created in the openSUSE wiki listing the so-called Most Annoying Bugs. Leap 15.1 was a very smooth release and this time the corresponding list contains only one bug… and it’s a YaST one. 🙁

In an installed system, when YaST was creating a new user it was always trying to create its home directory as a Btrfs subvolume. Even in cases in which that was impossible. For example, if the directory to be created was not in a Btrfs file system.

Writing user error

Fortunately, the bug didn’t affect the installation process or AutoYaST. We created a fix that was quickly available as a maintenance update. So make sure your openSUSE system is updated before trying to create new users with YaST.

YaST Network refactoring: status report

Since we submitted the first bits of the yast2-network refactoring by the end of April, we have made quite some progress in this area. Although it is still an ongoing effort, we would like you to give you an update on the current state.

We might say that we have been working on two different areas the user interface implementation and the future data model.

About the user interface, the team has improved the code quite a lot to make it easier to maintain and extend. We have introduced some classes to de-couple the widgets from the data, and it pays off. Additionally, we have fixed some bugs (many of them related to validations), simplified the process of adding new devices and reorganized the hardware tab.

New hardware tab in YaST Network

Regarding the internal data model, we have been thinking about the best way to represent network configuration in an agnostic way so, in the future, we can not only support Wicked but other options too (for the time being, the NetworkManager support is quite limited). If you are curious about the details, we have added a document describing the approach to the repository.

The new data model is already being used to handle DNS and Routing configuration. So if you are using Tumbleweed you have been already using the new network code for some weeks, including the UI enhancements presented in our latest post.

New network routing dialog

Although the data model is so far only used in the mentioned parts, the plan is to submit to Tumbleweed a heavily refactored UI layer during next sprint. So stay tuned.

Added Appstream Metadata to the YaST Packages

The YaST package manager is not the only software manager in the (open)SUSE distributions. There are some more, like Discover in KDE or the GNOME software manager, not to mention the online openSUSE appstore.

While the YaST manager is package oriented, those other software managers are application oriented. That makes a huge difference, especially for the beginner users.

The full list of packages does not only include the applications (basically anything a user can start from the desktop), but also shared libraries, pieces that provide functionality for other applications or basic components needed for the system to work. With so many software packages (the openSUSE OSS repository contains over 35.000 of them!) it’s sometimes hard to find the software you need unless you know what you are looking for.

To offer an application-oriented view on top of all that, the application managers need some special data describing the applications inside the packages. The data are located in the /usr/share/metainfo/*.xml files, if you are interested in the technical details check the AppStream documentation provided by the Freedesktop.org.

Our absolutely awesome community contributor Stasiek Michalski (more famous by his nickname lcp) realized YaST was not offered in those application managers and decided to fix it. So he created an XML generator which collects the data from YaST packages and automagically generates the metainfo XML file needed by the other software managers.

YaST in GNOME Software

As a result, the Gnome Software Manager, Discover and other software managers will offer YaST in Tumbleweed just as any user-oriented application. Thanks lcp!

YaST in Discover

AutoYaST Pre-install Scripts & Storage

AutoYaST has a special feature to allow users to customize the installation process and take control in different stages of the installation. For that, the AutoYaST profile offers a section where you can insert your custom scripts. There are four types of scripts: pre-scripts, postpartitioning-scripts, chroot-scripts, post-scripts and init-scripts.

For the particular case of pre-scripts, the documentation states that “It is also possible to change the partitioning in your pre-script“. That means, for example, you could use a script to create a new partition or to configure some kind of technology. Therefore, it would be very convenient to re-analyze the storage devices after running the user pre-scripts. In fact, that was the default behavior in the old storage stack, but the new one was slightly modified to only re-analyze the system under certain conditions.

But turns out some SLE customers were using pre-scripts to configure the behavior of multipath and those changes were not being noticed by AutoYaST.

The solution was quite trivial. We simply decided to always perform a new storage re-analysis after the AutoYaST pre-scripts. We did not find strong reasons to not do it and there should not be significant performance penalty.

And, for the specific case of multipath, YaST now copies some configuration files (e.g., /etc/multipath.conf and /etc/multipath/bindings) to the target system when performing a new installation. Otherwise, the installed system would not contain the configuration applied during the installation.

Clarifying the Usage of Software Management Options

Our software manager is one of the most complex YaST modules, which makes some aspects of its usage not fully obvious for some users. You may have even notice that is the only YaST module in which the interface is clearly different when executed in text and graphical mode. Specially the menu at the top of the interface, which is organized in different sections.

Some users where confused by the fact of some options not being persisted between different executions of the module. Those options are there to modify the current operation of the module, not to change the configuration of package management in the system.

When executed in text mode that was clear because such options where labeled as “temporary change” but the graphical mode didn’t have any indication about it. As you can see in the following screenshot, that’s fixed now.

Temporary options in YaST Software

Product License Hard to Understand? Try Another Language!

Some user reported that in the text-mode installation it was impossible to switch the language of a product license. Although during graphical installation everything works flawlessly, in text mode the language switching widget was there… but disabled.

The point is that such behavior was not exactly a bug. It was in fact done on purpose. We decided to prevent such language change some time ago because on the Linux console we’re not able to display the characters of many languages. Hopefully some of our usual reader have just shout “that’s not true!” 😉 Those users remember that in the report of our 67th sprint we explained that now we always use fbiterm when installing in text mode in order to be able to display characters of almost every language.

We are now able to display all languages that currently have license translations, so we have enabled the language switching widget and now both graphical installation and text-based one deliver an equivalent user experience.

More about languages

This is not the only change related to internationalization we did in these sprints. We also added a specific warning message for situations in which YaST is used to change the language of the system but there is no repository containing the needed translation packages. Something that obviously only affects users configuring system in very restricted environments.

As you can see in this and other recent reports, we have to deal relatively often with difficulties related to translation and internationalization. To reduce the effect of those problems on our final users, we also added some extra mechanisms to detect internationalization errors introduced during development. Hopefully that will mean that in future reports the space dedicated to comment language-related problems gets shorter and shorter. 🙂

And that was just a summary!

As long as this post looks, there are many interesting things we have done in these weeks we left out, intentionally or not. We definitely should avoid skipping three reports in a row in the future!

This week it’s Hack Week at SUSE, which means regular YaST development will be put on hold… or will turn into something completely different. You never know what the result of a Hack Week can be!

But in any case we will go back to our sprint-based pace in August. So expect a new blog post in three weeks. See you then!

Getting further with Btrfs in YaST

June 19th, 2019 by

Since the YaST team rewrote the software stack for managing the storage devices, we have been adding and presenting new capabilities in that area regularly. That includes, among other features, the unpaired ability to format and partition all kind of devices and the possibility of creating and managing Bcache devices. Time has come to present another largely awaited feature that is just landing in openSUSE Tumbleweed: support for multi-device Btrfs file systems.

As our usual readers surely know, Btrfs is a modern file system for Linux aimed at implementing advanced features that go beyond the scope and capabilities of traditional file systems. Such capabilities include subvolumes (separate internal file system roots), writable and read-only snapshots, efficient incremental backup and our today’s special: support for distributing a single file system over multiple block devices.

Multi-device Btrfs at a glance

Ok, you got it. YaST now supports multi-device Btrfs file system… but, what that exactly means? Well, as simple as it sounds, it’s possible to create a Btrfs file system over several disks, partitions or any other block devices. Pretty much like a software-defined RAID. In fact, you can use it to completely replace software RAIDs.

Let’s see an example. Imagine you have two disks, /dev/sda and /dev/sdb, and you also have some partitions on the first disk. You can create a Btrfs file system over some devices at the same time, e.g., over /dev/sda2 and /dev/sdb, so you will have a configuration that looks like this.

        /dev/sda                /dev/sdb
            |                       |   
            |                       |   
     ---------------                |   
    |               |               |   
    |               |               |   
/dev/sda1       /dev/sda2           |   
                    |               |   
                    |               |   
                     ---------------
                            |   
                          Btrfs
                            |   
                            |   
                            @ (default subvolume)
                            |   
                            |   
                 -----------------------
                |       |       |       |   
                |       |       |       |   
              @/home  @/log   @/srv    ...

Once you have the file system over several devices, you can configure it to do data striping, mirroring, striping + mirroring, etc. Basically everything that RAID can do. In fact, you can configure how to treat the data and the Btrfs meta-data separately. For example, you could decide to do striping with your data (by setting the data RAID level to the raid0 value) and to do mirroring with the Btrfs meta-data (setting it as raid1 level). For both, data and meta-data, you can use the following levels: single, dup, raid0, raid1, raid10, raid5 and raid6.

The combination of this feature and Btrfs subvolumes opens an almost endless world of possibilities. It basically allows you to manage your whole storage configuration from the file system itself. Usage of separate tools and layers like software-defined RAID or LVM are simply dispensable when using Btrfs in all its glory.

Managing multi-device Btrfs with the YaST Partitioner

Interesting feature indeed, but where to start? As usual, YaST brings you the answer! Let’s see how the YaST version that is currently being integrated in openSUSE Tumbleweed will ease the management of this cool Btrfs feature. SLE and Leap users will have to wait to the next release (15.2) to enjoy all the bells and whistles.

First of all, the Btrfs section of our beloved Expert Partitioner has been revamped as shown in the following picture.

New Btrfs section of the Partitioner

It lists all the Btrfs file systems, single- and multi-device ones. You can distinguish them at first sight by the format of the name. The table contains the most relevant information about the file systems, alongside buttons to add a new file system and to delete and modify the existing ones.

Existing Btrfs file system can be inspected and modified in several ways. The “Overview” tab includes details like the mount point, file system label, UUID, data and meta-data RAID levels, etc. The file system can be edited to modify some aspects like the mount options or the subvolumes.

Overview of a Btrfs file system

In addition, the tab called “Used Devices” contains a detailed list of the block devices being used by the file system. That list can also be modified to add or remove devices. Note such operation can only be done when the file system does not exist on disk yet. Theoretically, Btrfs allows to add and remove devices from an already created file system, but a balancing operation would be needed after that. Such balancing operation could take quite a considerable amount of time. For that reason it has been avoided in the Expert Partitioner.

Devices of a Btrfs file system

Of course, you can still format a single device as Btrfs in the traditional way (using the “edit” button for such device). But let’s see how the new button for adding a Btrfs file system opens new possibilities.

Adding a Btrfs file system

Similar to the RAID dialog, you have the available devices on the left and you can select the devices where you want to create the file system, and also you can indicate the data and meta-data RAID levels. Of course, the admissible RAID levels will depend on the number of selected devices. You will go to the second step of the Btrfs creation by clicking the “Next” button. In this second step, you can select the mount options and define the subvolumes, see the next image.

Options for a new Btrfs file system

And apart of all that, the Expert Partitioner has received several small improvements after including multi-device Btrfs file systems. Now that the multi-device Btrfs file systems are considered first class citizens, they are included in the general list of devices. Note the “Type” column has also been improved to show more useful information, not only for Btrfs but for all kind of devices.

Revamped list of devices

What else works?

But YaST goes far beyond the Partitioner. We have also ensured the storage proposal (i.e. the Guided Setup) can deal with existing multi-device Btrfs configurations when you are performing a new installation. Moreover, the upgrade process is also ready to work with your multi-device Btrfs file system.

Last but not least, AutoYaST can now also be used to specify that kind of Btrfs setups. The official AutoYaST documentation will include a specific section about advanced management of Btrfs file systems on top of several block devices. The content is being reviewed by the SUSE documentation team right now.

What does not work (yet)?

There is still one scenario that is not 100% covered. As described in bug#1137997, is still not possible to use the “Import Mount Points” button in the Partitioner to recreate a multi-device Btrfs layout. But fear not, it’s in our list of things to fix in the short term!

Get it while it’s hot

Free Software development is a collaborative process and now we need YOU to do your part. Please test this new feature and report bugs if something does not work as you expected. And please, come with your ideas and further improvements and use cases. And, of course, don’t forget to have a lot of fun!