A More User Friendly Role Selector Dialog

Step by step, we continue improving the user experience making use of the newly added widgets to libyui. This sprint was the turn to update the role selection dialog to use the new item selector introduced during the sprint 87. Apart from looking better as it can be seen in the screenshots below, there are two immediate improvements:

• the vertical scroll, when present, is respected after selecting a role (instead of “jumping to the beginning”), and
• the selected role (if any) will be visible when arriving at the dialog even when the list is quite long or the available space too short.

Before	After

What is more, updating the dialog was also useful for us to realize about some needed improvements for the widget itself, mentioned in the next section. Quite a productive change

When one Bit is not Enough: The CustomStatusItemSelector

A few weeks ago, we wrote about the new ItemSelector widget that is finding its way into YaST user interfaces. It turned out that just a simple on/off status is not enough in some cases, so we had to extend that concept. For example, software modules may have dependencies, and we want to show the difference between one that was explicitly selected by the user and one that was auto-selected because some other software module requires it.

This kind of shook the foundations of the underlying classes; all of a sudden a bit is no longer just a bit, but it needs to be broken down into even smaller pieces. Well, we cheated; we now use integer values instead. Most of the class hierarchy still only uses 0 and 1, but the new YCustomStatusItemSelector also supports using higher numbers for application-defined purposes.

For each possible status value, the application defines the name of the icon to be displayed (for graphical UIs like the Qt UI), the text equivalent (for text mode / the NCurses UI), and an optional nextStatus which tells the widget what status to cycle to when the user changes the status of an item with a mouse click or with the keyboard. A value of -1 lets the application handle this.

So this is not a one-trick-pony that is useful only for that one use case (the software modules), but a generic tool that might find good uses in other places all over YaST as well.

Usage examples: C++, Ruby.

Snapper and Machine-readable Output

Most likely you already know snapper, a great tool to work with your filesystem snapshots. Some third-party scripts and tools (e.g., YaST) use the snapper CLI to get some information, but generally, snapper generates an output intended to be human-readable. Sometimes that could cause some troubles in scripts checking the snapper output. Now on, snapper also offers CLI options to generate its output in a machine-readable format, i.e., CSV and JSON. Please, check this post for more information about those new options.

Fix Boot Problems with Remote Encrypted Devices

Since we adopted systemd, the management during system boot of encrypted devices on top of network-based devices like iSCSI or FCoE disks has been less than optimal. But now we are happy to announce that we have put all the pieces together to make the experience as smooth as possible.

One of the main responsibilities of systemd is sorting the actions performed during boot and setting the dependencies between them. For example, if there are encrypted devices, systemd will first ask you for the password and activate the devices to afterwards mount the file system contained in those encrypted devices. Systemd should be able to distinguish when an encrypted device is based on a network-based storage device and, thus, can only be initialized after the network is up. In some cases that detection failed (for example network block device based mounts, such as iSCSI and FCoE disks) and systemd got stuck before initializing the network waiting for the device to be available.

Recently, SLE and openSUSE Leap has incorporated support for specifying the _netdev option in the /etc/crypttab file. With such option, systemd will recognize the encrypted device as network-based and will only try to activate it after setting up the network. That’s analogous to the corresponding _netdev option in /etc/fstab that has been already there for quite some time and that can be used to defer when a device is mounted. For it to work properly, the _netdev option must be present in all the relevant entries of both crypttab and fstab.

And that’s exactly what YaST will do now in openSUSE Tumbleweed and upcoming releases of both SLE and openSUSE Leap. From now on, the _netdev option will be added fstab for all mount points depending (directly or indirectly) on the network. In addition, that option (and also the noauto and nofail ones) will be propagated from fstab to all the corresponding crypttab entries.

This should mark the end of a dark age of encrypted iSCSI and FCoE devices timing out during boot.

AutoYaST Support for Random and Pervasive Encryption

Back in October, we announced that YaST got support for new encryption methods like random or pervasive encryption. At that time, AutoYaST was out of scope because we wanted to have a stable (and tested) API first. Fortunately, the time has come and now AutoYaST supports these encryption mechanisms.

Starting in autoyast2 4.2.17, you can specify the encryption method using a crypt_method element, as shown in the example below. Possible values are luks1 (regular LUKS1 encryption), pervasive_luks2 (pervasive volume encryption), protected_swap (encryption with volatile protected key), secure_swap (encryption with volatile secure key) and random_swap (encryption with volatile random key).

    <drive>
     <type config:type="symbol">CT_DISK</type>
     <use>all</use>
     <partitions config:type="list">
      <partition>
       <size>20G</size>
       <mount>/</mount>
       <filesystem config:type="symbol">ext4</filesystem>
       <crypt_method config:type="symbol">luks1</crypt_method> <!-- default method if crypt_key is defined -->
       <crypt_key>S3CR3T</crypt_key>
      </partition>
      <partition>
       <size>1G</size>
       <mount>swap</mount>
       <crypt_method config:type="symbol">random_swap</crypt_method> <!-- set encryption method -->
      </partition>
     </partitions>
    </drive>

As we want AutoYaST to be as user-friendly as possible, it will try to help you if you do some mistake setting the encryption configuration as when in the screenshot below.

Finally, the old crypt_fs element is deprecated, although it stills works for backward-compatibility reasons. Basically, it is equivalent to setting crypt_method to luks1.

Improve Support for AutoYaST Guided Partitioning

When it comes to partitioning, we can categorize AutoYaST use cases into three different levels:

• Automatic partitioning: the user does not care about the partitioning and trusts in AutoYaST to do the right thing.
• Guided partitioning: the user would like to set some basic settings (use LVM, set an encryption password, etc.)
• Expert partitioning: the user specifies how the layout should look, although a complete definition is not required.

To some extent, it is like using the regular installer where you can skip the partitioning screen and trust in YaST, use the Guided Proposal, or define the partitioning layout through the Expert Partitioner.

The second level (Guided partitioning) was introduced in AutoYaST with the release of SUSE Linux Enteprise 15 (and Leap 15.0) but it was not documented at all. Additionally, although it was working as designed at first sight, it was far from being that useful.

This sprint with invested quite some time improving the documentation (kudos to our awesome documentation team) and the behaviour. Now, if you want to set up an LVM system without having the specify all the details, you can use the following snippet in your profile:

<general>
  <storage>
    <lvm config:type="boolean">true</lvm>
  </storage>
</general>

If you are interested in the available options, you can check the documentation draft.

Unifying the Console Keyboard Layouts for SLE and openSUSE

The way of managing internationalization in Linux systems has changed through the years, as well as the technologies used to represent the different alphabets and characters used in every language. YaST tries to offer a centralized way of managing the system-wide settings in that regard. An apparently simple action like changing the language in the YaST interface implies many aspects like setting the font and the keyboard map to be used in the text-based consoles, doing the same for the graphical X11 environment and keeping those fonts and keyboard maps in sync, ensuring the compatibility between all the pieces.

For that purpose, YaST maintains a list with all the correspondences between keyboard layouts and its corresponding "keymap" files living under /usr/share/kbd/keymaps. Some time ago the content of that list diverged between openSUSE and SLE-based products. During this sprint we took the opportunity to analyze the situation and try to unify criteria in that regard.

We analyzed the status and origin of all the keymap files used in both families of distributions (you can see a rather comprehensive research starting in comment #18 of bug#1124921) and we came to the conclusions that:

• The openSUSE list needed some minor adjustments.
• Leaving that aside, the keymaps used in openSUSE were in general a better option because they are more modern and aligned with current upstream development.

So we decided to unify all systems to adopt the openSUSE approach. That will have basically no impact for our openSUSE users but may have some implications for users installing the upcoming SLE-15-SP2. In any case, we hope that change will be for the better in most cases. Time will tell.

Exporting User Defined Repositories to AutoYaST Configuration File.

With the call yast clone_system an AutoYaST configuration file will be generated which reflects the state of the running system. Up to now only SUSE Add-Ons have been defined in the AutoYaST configration module. Now also user defined repositories will be exported in an own subsection <add_on_others> of the <add-on> section.

<add-on>
  <add_on_others config:type="list">
    <listentry>
      <alias>yast_head</alias>
      <media_url>https://download.opensuse.org/repositories/YaST:/Head/openSUSE_Leap_15.1/</media_url>
      <name>Yast head</name>
      <priority config:type="integer">99</priority>
      <product_dir>/</product_dir>
    </listentry>
  </add_on_others>
  <add_on_products config:type="list">
    <listentry>
      <media_url>dvd:/?devices=/dev/sr1</media_url>
      <product>sle-module-desktop-applications</product>
      <product_dir>/Module-Desktop-Applications</product_dir>
    </listentry>
    <listentry>
      <media_url>dvd:/?devices=/dev/sr1</media_url>
      <product>sle-module-basesystem</product>
      <product_dir>/Module-Basesystem</product_dir>
    </listentry>
  </add_on_products>
</add-on>

The format of the <add_on_others> section is the same as the <add_on_products> section.

Better Handling of Broken Bootloader Setups during Upgrade

With the current versions of SLE and openSUSE, using the installation media to upgrade a system which contains a badly broken GRUB2 configuration (e.g. contains references to udev links that do not longer exist) can result in an ugly internal error during the process.

The first possible problem could arise in the summary screen. Like shown in this screenshot.

If the error didn’t pop-up or if the user managed to recover from it, it was possible to reach the final phase of the upgrade process. But then the same internal error could still pop up in a different place:

Those errors will be fixed in the upcoming releases of SLE-12-SP5 and SLE-15-SP2 and, of course, in the corresponding openSUSE Leap version and in Tumbleweed. Now if such a broken setup is detected in the summary screen, a proper warning is displayed, including the technical details and a tip on what to do to fix the problem.

The user can ignore the problem or click on "booting" to fix it. In the latter case, the usual pop-up with instructions will appear.

If the final stage of the upgrade process is reached without fixing the error, the wild internal error is now replaced by an informative message that does not interrupt the process.

Hopefully most of our users will never see these improvements. But users with a broken system will likely appreciate the extra guidance.

Old Storage, New Features

If you are an usual reader of this blog, most likely you already know that YaST has a completely re-implemented Storage stack (a.k.a. storage-ng). That new storage code did its debut with the SLE 15 (and openSUSE Leap 15.0) family. And thanks to this revamped code, our beloved users can enjoy today some new great features in YaST like Bcache, partitionable Software RAIDs or multi-device Btrfs file system (just to mention a few examples). But SLE 12 (openSUSE 42) products are still alive and getting improvements with every maintenance update! Of course, the old Storage stack is not an exception, and now on a new installation scenario is supported.

Thanks to a bug report, we realized that Snapper could not be configured in some cases. More specifically, the reporter was trying to install with AutoYaST over a directly formatted Software RAID by using Btrfs for root and enabling snapper. The installation was perfectly performed, but it turned out that snapper was not correctly enabled in the installed system. After having a deeper look into the problem, we discovered that this was not a bug exactly but a completely missing feature. But no problems, YaST got down to work and now it is nicely supported.

Snapper: Show Currently Mounted and Next to be Mounted Snapshot

Btrfs has some special snapshots: The snapshot currently mounted, and the snapshot that will be mounted next time (unless a snapshot is selected in grub). Now snapper informs the user about these two special snapshot when listing snapshots by a special sign after the number:

# snapper --iso list --disable-used-space
 # | Type   | Pre # | Date                | User | Cleanup | Description           | Userdata     
---+--------+-------+---------------------+------+---------+-----------------------+--------------
0  | single |       |                     | root |         | current               |              
1+ | single |       | 2018-10-18 10:33:50 | root |         | first root filesystem |              
2  | single |       | 2018-10-18 10:43:45 | root | number  | after installation    | important=yes
3- | pre    |       | 2018-10-18 11:03:11 | root |         | ruin system           |              
4  | post   |     3 | 2018-10-18 11:03:11 | root |         | ruin system           |              

For more details visit http://snapper.io/2018/10/18/show-special-snapshots.html.

More Descriptive Name for Snapper Module in YaST Control Center

Previously, the module was called just "Snapper", but users who don’t know that Snapper is could not make any sense of that. We changed it to "Filesystem Snapshots".

Funny anecdote: One team member asked if Snapper really supports LVM when he read the subtitle "Manage Btrfs / LVM filesystem snapshots". Yes, it does! (It has been doing that for a long time). You don’t need Btrfs for snapshots; LVM can also do that, albeit a little differently than Btrfs.

More Bcache Improvements

As you can see in the previous blog post, we are currently working on adding support for Bcache into the YaST partitioner. This time we allow to configure the cache mode for a new bcache device. If you are not sure what a particular cache mode means, we provide also a quite extensive help text. Beside this configuration we also limit operations to prevent data loss or operations that can result in unreliable results. Here with a couple of screenshots:

Using whole disks in AutoYaST

On one hand, now it is possible to format and mount a whole disk without creating any partition. In order to do so, you only need to set the <disklabel> element to none and AutoYaST will understand that you do not want to partition the drive but to use the whole disk as a filesystem.

<drive>
  <device>/dev/sdb</device>
  <disklabel>none</disklabel>
  <partitions config:type="list">
    <partition>
      <mount>/home</mount>
      <filesystem config:type="symbol">xfs</filesystem>
    </partition>
  </partitions>
</drive>

Given the definition above, AutoYaST will format the whole /dev/sdb disk mounting it at /home. But that is not all: it is even possible to use a whole disk as an LVM physical volume or as a software RAID member. The support for the first case was already present in previous AutoYaST versions, but it was not working correctly in SUSE Linux Enterprise 15 and openSUSE Leap 15.

<drive>
  <device>/dev/sdb</device>
  <disklabel>none</disklabel>
  <partitions config:type="list">
    <partition>
      <lvm_group>system</lvm_group>
    </partition>
  </partitions>
</drive>

AutoYaST and partitioned software RAIDs

AutoYaST is now able to create partitioned software RAIDs, something that was not possible in pre-storage-ng times. However, in order to support such a scenario, we needed to change the way in which software RAIDs are described in AutoYaST profiles, although the old format is still supported. So let’s have a look at how a RAID looks like now.

Instead of grouping all RAIDs in a single and special <drive> section, now each RAID is defined in its own section:

<drive>
  <device>/dev/md0</device>
  <raid_options>
    <raid_type>raid0</raid_type>
  </raid_options>
  <partitions config:type="list">
    <partition>
      <mount>/</mount>
      <filesystem config:type="symbol">btrfs</filesystem>
    </partition>
    <partition>
      <mount>/home</mount>
      <filesystem config:type="symbol">xfs</filesystem>
    </partition>
  </partitions>
</drive>

Of course, if you do not want the RAID to be partitioned, just set the <disklabel> element to none, as for any other device.

Better Xen Virtual Partitions support

Analogous to how software RAIDs were defined in AutoYaST until now, Xen virtual partitions with a similar name were grouped in the same <drive> section. It means the /dev/xvda1, /dev/xvda2, etc. were defined within the <drive> section for xvda, which does not exist at all.

To make things clearer, we have decided to use a separate drive section for partition:

<drive>
  <type config:type="symbol">CT_DISK</type>
  <device>/dev/xvdd1</device>
  <disklabel>none</disklabel> <!-- not really needed -->
  <use>all</use>
  <partitions config:type="list">
    <partition>
      <format config:type="boolean">true</format>
      <mount>/home</mount>
      <size>max</size>
    </partition>
  </partitions>
</drive>

AutoYaST Rules: Cleaning the profiles before being merged

AutoYaST rules offer the possibility to configure a system depending on system attributes by merging multiple control files during installation. Check the Rules and Classes section for further documentation.

The merging process is often confusing for people, and the sections in the merged XML files must be in alphabetical order for the merge to succeed.

AutoYaST was cleaning the profiles after a merge, but if the resultant profile was merged with another profiles that profiles were not cleaned before the merge. That was confusing and error prone, so we have fixed it cleaning also them before the merge.

Better explanation of the requirements to boot with GPT

As our readers know, one of the main goals of yast-storage-ng was to offer a more reliable and precise diagnosis on what partitions need to be created in order to ensure that a new system being installed will be able to boot. If something doesn’t fit with such diagnosis, the installer shows a warning message.

In the case of booting a system installed in a GPT device, using the legacy BIOS system (as opposed to EFI), that means SLE-15 and openSUSE Leap 15.0 will warn the user if there is no partition of type BIOS Boot. But there are two problems with that.

• The warning messages from the Partitioner and, specially, from AutoYaST don’t do a great job in explaining what is wrong.
• Some users have reported they have GPT systems booting fine in legacy mode without a BIOS Boot partition and, thus, our diagnosis in such cases may be wrong.

We even had a comment in our source code reinforcing the second point!

So we tried to fix our wrong diagnosis… just to end up realizing it was in fact right. After carefully evaluating all the possible setups, checking the different specifications, the Grub2 documentation and even checking the Grub2 source code, we found that layouts without a BIOS Boot partition could get broken (resulting in a non-bootable system) by some file-system level operations. So only the configurations including a BIOS Boot partition can be considered to be 100% safe, both in the short term and against future changes in the system.

We simply cannot allow our users to fall into traps without, at least, a warning message. So we kept the behavior as it was and we focused on improving the messages. After all, advanced users knowing the risks can ignore such warnings. This is how the new warning look in the Partitioner of the upcoming SLE-15-SP1 (and, thus, in openSUSE Leap 15.1).

And this is what AutoYaST will report if the profile doesn’t specify a BIOS Boot partition and it’s not possible to add one to the layout described by such profile.

CaaSP / Kubic: Propose NTP servers according to DHCP response

All-in-one dialog of CaaSP installer asks for NTP Servers. Up to now it searched for NTP servers using SLP only. Otherwise only manual configuration was possible.

Since now CaaSP installer parses DHCP response and fetches NTP servers if any was provided. NTP Servers obtained from DHCP are preferred over those discovered via SLP.

Partitioner UI is a bit faster now

We noticed that clicking around the partitioner UI feels slow. So we used the built-in (Y2PROFILER=1) as well as an external (rbspy) profiler to pinpoint the places that need optimization (mostly caching). Can you see a difference in the following screencast?

So how can you make use of it? Suppose the subvolume /home/tux is already configured for snapper and you want to allow the user tux to manage the snapshots for her home directory. This is done in two easy steps:

1. Edit /etc/snapper/configs/home-tux and add ALLOW_USERS=”tux”. Currently the server snapperd does not reload the configuration so if it’s running either kill it or wait for it to terminate by itself.
2. Give the user permissions to read and access the .snapshots directory, ‘chmod a+rx /home/tux/.snapshots’.

For details consult the snapper man-page.

Now tux can play with snapper:

  tux> alias snapper="snapper -c home-tux"

  tux> snapper create --description test

  tux> snapper list
  Type   | # | Pre # | Date                             | User | Cleanup  | Description | Userdata
  -------+---+-------+----------------------------------+------+----------+-------------+---------
  single | 0 |       |                                  | root |          | current     |         
  single | 1 |       | Tue 16 Oct 2012 12:15:01 PM CEST | root | timeline | timeline    |         
  single | 2 |       | Tue 16 Oct 2012 12:21:38 PM CEST | tux  |          | test        |

Snapper packages are available for various distributions in the filesystems:snapper project.

So long and see you at the openSUSE Conference 2012 in Prague.

Unfortunately openSUSE 12.2 RC1 does not include LVM tools with thin-provisioning so you have to compile them on your own. First install the thin-provisioning-tools. Then install LVM with thin-provisioning enabled (configure option –with-thin=internal).

To setup LVM we first have to create a volume group either using the LVM tools or YaST. I assume it’s named test. Then we create a storage pool with 3GB space.

  # modprobe dm-thin-pool
  # lvcreate --thin test/pool --size 3G

Now we can create a thin-provisioned logical volume named thin with a size of 5GB. The size can be larger than the pool since data is only allocated from the pool when needed.

  # lvcreate --thin test/pool --virtualsize 5G --name thin

  # mkfs.ext4 /dev/test/thin
  # mkdir /thin
  # mount /dev/test/thin /thin

Finally we can create a snapshot from the logical volume.

  # lvcreate --snapshot --name thin-snap1 /dev/test/thin

  # mkdir /thin-snapshot
  # mount /dev/test/thin-snap1 /thin-snapshot

Space for the snapshot is also allocated from the pool when needed. The command lvs gives an overview of the allocated space.

  # lvs
  LV         VG   Attr     LSize Pool Origin Data%  Move Log Copy%  Convert
  pool       test twi-a-tz 3.00g               4.24
  thin       test Vwi-aotz 5.00g pool          2.54
  thin-snap1 test Vwi-a-tz 5.00g pool thin     2.54

After installing snapper version 0.0.12 or later we can create a config for the logical volume thin.

  # snapper -c thin create-config --fstype="lvm(ext4)" /thin

As a simple test we can create a new file and see that snapper detects its creation.

  # snapper -c thin create --command "touch /thin/lenny"

  # snapper -c thin list
  Type   | # | Pre # | Date                          | Cleanup | Description | Userdata
  -------+---+-------+-------------------------------+---------+-------------+---------
  single | 0 |       |                               |         | current     |
  pre    | 1 |       | Tue 24 Jul 2012 15:49:51 CEST |         |             |
  post   | 2 | 1     | Tue 24 Jul 2012 15:49:51 CEST |         |             |

  # snapper -c thin status 1..2
  +... /thin/lenny

So now you can use snapper even if you don’t trust btrfs. Feedback is welcomed.

For years we had the request to provide rollbacks for YaST and zypper but things never got far due to various technical problems. With the rise of btrfs snapshots we finally saw the possibility for a usable solution. The basic idea is to create a snapshot before and after running YaST or zypper, compare the two snapshots and finally provide a tool to revert the differences between the two snapshots. That was the birth of snapper. Soon the idea was extended to create hourly snapshots as a backup system against general user mistakes.

The tool is now in a state where you can play with it. On the other hand there is still room and time for modifications and new features.

Overview

We provide a command line tool and a YaST UI module. Here is a brief tour:

First we manually create a snapshot:

# snapper create --description "initial"

Running YaST automatically creates two snapshots:

# yast2 users

Now we can list our snapshots:

# snapper list
Type   | # | Pre # | Date                     | Cleanup  | Description
-------+---+-------+--------------------------+----------+-------------
single | 0 |       |                          |          | current
single | 1 |       | Wed Mar 30 14:52:17 2011 |          | initial
pre    | 2 |       | Wed Mar 30 14:57:10 2011 | number   | yast users
post   | 3 | 2     | Wed Mar 30 14:57:35 2011 | number   |
single | 4 |       | Wed Mar 30 15:00:01 2011 | timeline | timeline

Snapshot #0 always refers to the current system. Snapshot #2 and #3 were created by YaST. Snapshot #4 was created by an hourly cronjob.

Getting the list of modified files between to snapshots is easy:

# snapper diff 2 3
c... /etc/group
c... /etc/group.YaST2save
c... /etc/passwd
c... /etc/passwd.YaST2save
c... /etc/shadow
c... /etc/shadow.YaST2save
c... /etc/sysconfig/displaymanager
c... /var/tmp/kdecache-linux/icon-cache.kcache
c... /var/tmp/kdecache-linux/plasma_theme_openSUSEdefault.kcache

You can also compare a single file between two snapshots:

# snapper diff --file /etc/passwd 2 3
--- /snapshots/2/snapshot/etc/passwd    2011-03-30 14:41:45.943000001 +0200
+++ /snapshots/3/snapshot/etc/passwd    2011-03-30 14:57:33.916000003 +0200
@@ -22,3 +22,4 @@
 uucp:x:10:14:Unix-to-Unix CoPy system:/etc/uucp:/bin/bash
 wwwrun:x:30:8:WWW daemon apache:/var/lib/wwwrun:/bin/false
 linux:x:1000:100:linux:/home/linux:/bin/bash
+tux:x:1001:100:tux:/home/tux:/bin/bash

The main feature of course is to revert the changes between two snapshots:

# snapper rollback 2 3

Finally yast2-snapper provides a YaST UI for snapper.

Testing

Playing with snapper should only be done on test machines. Both btrfs and snapper are not finished and included known bugs. Here is a step-by-step manual for installing and configuring snapper for openSUSE 11.4.

Feedback

We would like to get feedback, esp. about general problems and further ideas. There are also tasks everybody can work on, e.g. the snapper wiki page or a man-page for snapper.

For the time being there is no dedicated mailing-list so just use opensuse-factory@opensuse.org.