Another YaST sprint is over and it is our pleasure to offer a report which includes a wide ranging of topics:

• The partitioner continues getting more love and attention: now you can partition your RAID devices and we have started to work on supporting Bcache.
• A new YaST2 firewall UI is around the corner.
• The self-update feature has been extended to cover more cases.
• YaST supports now your brand new 4K display during installation.

And, of course, some bugfixes (like proper handling of problems when trying to deactivate a DASD channel) and infrastructure improvements (check out our take on Travis caching).

Managing partitions within MD RAID devices

Our beloved Partitioner was one of the stars of the previous report, with news about several improvements and refinements. During this sprint it also received some love, not only with the addition of a couple of new features but also with more discussions and prototypes about the reorganization of the UI needed to bring even more improvements.

One of those features introduced in the previous sprint was the ability to create a RAID based on whole disks with no partitions. Now we have closed the circle by allowing to manage partitions within a RAID. So now it’s possible to create RAID arrays based on any combination of disks and partitions and then either use those arrays directly to host a file system (or to act as LVM physical volume) or create partitions inside the array. Partitions that can be, of course, formatted, encrypted, used as LVM physical volumes, etc. With that, we can now say the Partitioner support every possible MD RAID setup.

In the following animated screenshot you can see how the partitions of an MD RAID are listed in the left-hand tree of the Partitioner and in the table of devices of the RAID section, similar to how partitions of a hard disk are displayed.

As you may have noticed, the main difference with other views is that, as already anticipated by the UI discussions summarized in our previous sprint report, the set of buttons adapts dynamically to the selected item on the table offering a different set of actions for MD arrays and for partitions.

Another step to define the future of the Partitioner user interface

The interface showed in the previous screenshot, with its dynamic behavior, is just the first step in the direction already defined in the document that resulted from the previous sprint. Having that first fully functional prototype enabled us to rekindle the discussion about the best way to offer all the new possibilities of storage-ng in the Partitioner while keeping the user interface recognizable and familiar to the YaST users.

After another round of discussions and several iterations of mock-ups, we ended up with the idea documented at this gist. That will be the guide for the UI changes we are already implementing as part of the next sprint as a way to unleash even more of the underlying power of storage-ng.

Initial support for Bcache in the Partitioner

But apart from revamping and improving the support for existing technologies with new possibilities, we also want the Partitioner to grow in scope, putting the new kernel technologies into the hands of our users. One of such technologies is Bcache.

Bcache allows to improve the performance of any big but relative slow storage device (so-called “backing device” in Bcache terminology) by using a faster and smaller device (so-called caching device) to speed up read and write operations. The resulting Bcache device has then the size of the backing device and (almost) the effective speed of the caching one.

As a first step to offer full Bcache support, the Partitioner can visualize all the Bcache devices and allows to manipulate its partitions. Broader support, like creating new Bcache devices, will follow in upcoming sprints.

A New YaST2 Firewall UI is Coming

firewalld replaced the venerable SuSEfirewall2 as the default firewall solution in SUSE Linux Enterprise 15 and openSUSE Leap 15.0. And there is high chance that you have noticed that YaST does not offer a user interface to manage the firewall configuration anymore. Instead, it asks the user to use firewall-config, the official firewalld UI.

But, fortunately, that’s about to change. During recent sprints we have been working in a new UI to manage firewalld configuration. It is still a work in progress, but it is capable of assigning interfaces to zones and opening services/ports for a given zone.

We plan to release a first version as soon as we finish the integration with the AutoYaST user interface. So stay tunned!

Better HiDPI (4k Display) Support

What is better than a high screen resolution? Simple: a higher screen resolution. 4k displays are arriving in the consumer mainstream, and 8k displays are the next big thing.

But there are some real-world problems with that: with that large resolution, texts and graphics may become tiny – too tiny to read, too small to fill the available space on the screen.

On your desktop, you can tweak those things – select larger fonts or larger icon sizes. But for the installer that’s another matter; it needs to autodetect the presence of such a display and then auto-scale the user interface so it is actually usable.

We had a number of such issues in YaST; one was the font size, another the size of pixel graphics such as the time zone selection map. Those are now fixed.

For backwards compatibility, the Qt library which we are using for the graphical UI of YaST does not do that completely automatically but we needed to explicitly enable a HiDPI mode. But now it does it all automagically for YaST.

Updating Additional Installation Data

As you may know, YaST is able to update itself during system installation, which enables us to fix the installer by releasing updated packages even after the official release.

However, sometimes we need to fix packages which are not part of the installer but they provide some additional data for it like installation defaults, system role definitions, etc.

In order to support those scenarios, we have extended the installer to use the self-update repository as a regular (but temporary) one during the installation process. So with the next SUSE Linux Enterprise release we will be able to update that data too.

Fixing DASD Channel Deactivation Support

Recently we discovered that YaST was getting frozen in s390 architecture when the user tried to deactivate a DASD channel in use. The problem was that dasd_configure, the underlying tool which takes care of deactivating the channel, was waiting for user confirmation.

Definitely, losing user data is not an option so, from now on, the operation will be just canceled and YaST will report the problem, including all the details, so the user can solve the issue before trying to deactivate the channel again.

Faster Travis Rebuilds using ccache

For continuous integration at GitHub we use Travis which is easy to configure and use. Unfortunately, building there a big project like libstorage-ng takes a lot of time. In this case it is almost half an hour.

Waiting so long to see the result after doing a small change in the code is quite annoying and unconvenient. And this is exactly the scenario where using the ccache tool helps a lot. ccache is a small wrapper around the standard C/C++ compiler which saves the compilation result for later. So if the same file is compiled later again then the cached result is returned immediately without calling the real compiler.

Together with the Travis caching mechanism, which we use for storing and restoring the ccache files for each build, we reached a significant speed up. From 29 minutes to about 6 minutes, that is about four times faster!

Of course, it highly depends how much the environment (compiler, libraries) or the source files have been changed since the last build. The bigger changes the more recompilation is needed, so in reality the speedup might be smaller.

Check the related pull request if you are interested in the implementation details.

More to come

The following sprint is already running and the team is working in really interesting stuff: finishing the bcache support, releasing a first version of the new firewall UI, proper support to use a whole disk as an LVM PV in AutoYaST, several improvements to the installer for CaaSP/Kubic, etc.

So stay tunned!

We know, we know… we owed you a report for the 59th sprint! But you know what? We are now delivering this summary which covers sprints 59 and 60 so we are now up to date again.

During these weeks, we have been working mainly in two different areas. On one hand, we have improved YaST services management capabilities, adding support for on demand services and making the user experience slightly better. On the other hand, we are still working in the new storage layer, extending it to support Xen virtual partitions and fixing BIOS MD devices handling, among other things.

Last but not least, our bug fighting squad (do you remember squads, right?) have fixed quite some bugs accross our full stack.

So let’s have a tour to recap some highlights from these sprints.

Services Management Gets Some Love

The team is investing quite some time improving the services management area. In the past, YaST was adapted to play nicely with Systemd (which replaced the old runlevel-based init system). However, there was a lot of room for improvement, as Systemd added some new features when it comes to services management that were not supported in YaST at all.

To explore these changes, let’s start by having a look at the new services manager user interface.

At first sight, you may have spotted some changes: there is an Start Mode menu button where the Enable/Disable used to be; new Apply or Show Logs buttons have been added; the Start/Stop button has been replaced by an Start (or Stop) one… Ok, let’s describe these changes one by one.

Perhaps the most relevant update is the support for socket activated services. But you may be asking what a “socket activated service” is, right? In a nutshell, these are services that are started on demand. For instance, the printing service could be started only when you want to use the printer. And that’s what the Start Mode button is all about. For instance, you may want the printing services to be started only when needed (On Demand) and YaST will take care of setting up Systemd units in order to do so.

Another interesting feature is browsing services logs. Beware that this button will be available only if yast2-journal is installed because it uses the browser provided by this package.

And browsing the logs brings us to the new Apply button. Sometimes, it may be useful to Start a service and check the logs looking for potential problems. In the past, the changes were applied when the OK button was pressed but, unfortunately, the services manager was then closed. Now, with the Apply button, you can ask YaST to made those changes for real without closing the services manager (and check the logs or keep adjusting services).

But that’s not all! Among other small changes (like fixing bug #1080738) we are still improving the UX for this services manager.

Make the Xen virtual partitions great again

Rewriting a piece of YaST that is 20 years old is a never-ending source of surprises. When we though that Storage-ng did already support all the storage technologies handled by the old storage stack, we just found that we had left something behind.

When setting a Xen virtualization environment, it’s possible to define block devices that are seen and named like partitions in the Xen guest, but are in fact disks (or almost-disks, since they cannot be partitioned by regular methods). In other words, the Xen guest can see some block devices called e.g. /dev/xvda1 and /dev/xvda2 without a corresponding /dev/xvda.

From the YaST point of view, you can do whatever you can do with a partition (format it, mount it, etc.), except you can’t delete them or create more (since they are not backed by a disk, from the perspective of the Xen guest). We call those devices “Xen virtual partitions”. The old storage stack used to pretend those devices were just plain partitions of a made-up /dev/xda disk that was not there in the system but still appeared in the Partitioner and in the AutoYaST profile. The new stack simply contained no support at all for Xen virtual partitions.

We fixed that during the last sprint, making the feature available on the SLE-15 installation process thanks to the installer self-update feature. So either if your Xen guest runs a fully up-to-date version of SLE-15, Leap 15.0 or Tumbleweed or if you are installing SLE-15 with access to the installer self-update repositories, virtual partitions are visible in the Hard Disks section of the Partitioner.

It works in a in a similar way to the old (pre Storage-NG) Partitioner, but without artificial disks to group the virtual partitions. See the following screenshots in which xvda is a DVD, xvdb a real disk and xvdc1 a XEN virtual partition (there is no bogus xvdc added by YaST).

Only the Edit button really works as expected, allowing the user to mount and/or format the virtual partition. The other buttons just show the appropriate error for the different unsupported operations.

I want that on my Leap 15.0 installer

As a side note on the Xen partitions topic, it’s worth mentioning that the self-update feature of the installer is always available for all YaST-based distributions. But, unlike SLE, openSUSE doesn’t offer an official self-update repository, implying the feature is effectively disabled by default in openSUSE Leap. Does it mean that if you want to install Leap on top of a Xen virtual partition you will have to wait for Leap 15.1 or switch to Tumbleweed? Not exactly.

If you really want to use the latest YaST features during the openSUSE Leap 15.0 installation process, there are a couple of ways to achieve it. For example, to install (or upgrade) using the Leap 15.0 Live images. The live images are refreshed once in a while, so they can contain an installer that is more up-to-date than the installer in the Leap 15.0 normal ISO. There are also ways to use the self-update feature with unofficial (and unsupported) repositories. For a summary of all the options, check this comment at Bugzilla. Users of SLE should not need any of these hacks, since there is an official repository for the installer self-update mechanism ensuring SLE15 can always be installed with updated version of all the crucial packages.

A New Widget To Manage Services

Did you think that services management were confined to the services manager? Of course no 🙂 As you may know, there are serveral YaST modules that allow our users to set up several services like DNS, DHCP, Samba, etc. All these modules offer a way to configure how and when those services should be started.

So as part of the effort to improve the services management experience, we have rolled out a new widget which offers some benefits:

• Allows setting a service to be started On Demand.
• Offers a consistent an unified interface accross all modules.
• It is able to handle cases where several services are involved (like yast2-samba-server or yast2-iscsi-client).

By the way, we have adapted all these modules to use the new and shiny widget.

Better Handle Large File Systems in libyui

We fixed an issue with disks larger than 8EiB in the last sprint. However, the fix was intended for SLE12 and SLE15 maintenance update so we could not do big changes and we fixed only the most important parts.

For SLE15-SP1/Leap 15.1 and openSUSE Tumbleweed we can do more changes so we did some backward incompatible improvements.

Originally the size was implemented using long long data type which is a 64-bit signed integer (with the maximum value 8EiB). We switched to the Boost multiprecision C++ library which implements arbitrary precision integers. It works like the Integer class in Ruby which adds more bits to the data when needed.

Of course, it still depends on the limits in the underlying libzypp library which uses a 64-bit signed integer but with 1KiB unit, so the limit should be 8ZiB. With this change we are prepared even for more.

Additionally we added more units for converting to a string representation. Originally it used TiB units which resulted in too big numbers, now the EiB sizes are displayed using the EiB units as expected.

Moreover we added support for writing unit tests, evaluating the code coverage and reporting the it to coveralls.io. With this support we were able to write the very first unit test in libyui! The code coverage is now at the embarrassing 2% but we have just started!

Rubocop Check Speed Up

In the previous blog post we reported that we had improved running the unit tests in the yast2-storage-ng package. The speed up was based on parallelizing the execution and using all processors. This time we have improved the Rubocop check.

Normally Rubocop scans the directory for the files to check and then processes found files sequentially. And this might take very long time if there are hundreds of files to check.

Similarly as in the previous speed up we take advantage of the available processors and run multiple Rubocop instances in parallel. The implementation is a bit more complicated as Rubocop itself does not support parallel scan at all. But it is possible to evaluate the checked files, split them into the groups according to the number of processors and start a separate Rubocop instance for each group in parallel.

If you are interested in the details or you want to use parallel Rubocop also in your project then check the implementation in the yast-rake Ruby gem.

Of course, this improvement has an important impact in the yast2-storage-ng package where Rubocop needs to check over 600 files. Here you are the numbers:

• Running Rubocop locally (with hyperthreading enabled):
  • ~3.6x faster on an older quad core CPU (from 44s to 12s)
  • ~6x faster on a new eight core CPU (from 35s to 6s)
• At Travis: ~1.5x faster (from 69s to 47s), that’s the maximum possible because Travis has about 1.5x CPU build power

.

As you can see, for big projects the speed up is very good on local system and even at Travis it is still nice.

What’s next?

Sprint 61 is already running and the new storage layer and services management are again the areas where we are focusing an important part of our resources. Hopefully, in around two weeks, we will publish what we have achieved this time.

Stay tunned!

Squads in the Team

In the previous post we explained the squads idea and said we would tell more in this report. Thus, we should mention that we finally did three squads for this sprint:

• The Sockets & Services Squad working on supporting systemd sockets properly and other systemd related tasks (all in the YaST context, of course).
• The Qt and UI Squad working on user interface things like adding a new view to the package selection to show packages managed by a service, and also some control center improvements.
• The Bug Fighting Squad handling bugs that are coming in on a daily basis and in our backlog.

Fixed Issues with Disks Larger than 8EiB

It’s quite unlikely that you have at home a disk storage larger than 8EiB (eight exbibytes, 2⁶³). But in enterprise or cloud world it might be possible.

And it turned out that the YaST package manager does not handle such large disks well. At the start you would see this false error message:

The Problems

There are two problems:

• There definitely is a lot of free space on the disk, the error telling the user the space is running out is simply lying.
• The disk sizes are displayed as wrong negative values.

It turned out that the problem was caused by using the signed 64-bit integer data type which overflows for values bigger than 8EiB and the number becomes negative.

The Fix

We had to fix several places, each required some different solution.

• Use unsigned 64-bit integers where possible, that obviously avoids overflow.
• The numbers from libzypp use KiB units, at some places we need to convert that number to MiB. But first we converted to plain bytes (by multiplying by 1024) and then divide by 1MiB. And this first multiply step might cause overflow. Instead we simply convert KiB to MiB directly by dividing by 1024 without risk of overflow in the middle.
• Use floating point double data type for converting the values to a human readable text or to percents. The double has wider range and in these cases we do not need exact precision so rounding in floating point operations does not matter.
• Ignore a negative number in the free space check. At one place the value goes through the YaST component system which uses signed integer and this cannot be easily changed. In that case we consider negative free space as enough for installing any package, more than 8EiB free space should be enough for any package™.

There is still some minor issue with the large numbers. The highest supported unit is TiB so even very big numbers are displayed in TiB units as on the screenshot above. The fix is planned to be released as a maintenance update and this change would break the backward compatibility so we will improve it later but only for the future releases.

Testing

But the problem was how to test the behavior? You could fake some numbers in the code but for full testing or QA validation it would be nice to test on a real disk. But you usually do not have such a large storage for testing…

Fortunately in Linux it is possible to fake such large file system quite easily using sparse files and loop devices. Here is a short how to:

# create two big sparse files
truncate -s 6E /tmp/huge_file1
truncate -s 6E /tmp/huge_file2

# create block devices via loopback
losetup -f /tmp/huge_file1
losetup -f /tmp/huge_file2

# get the loop back device names, the names might be different
# if the system already uses some other loop back devices
losetup -a
/dev/loop1: [0056]:1171324 (/tmp/huge_file2)
/dev/loop0: [0056]:1171323 (/tmp/huge_file1)

# create a btrfs file system over both "disks"
mkfs.btrfs -K /dev/loop0 /dev/loop1

# mount it on /mnt2 (or whatever else, do no use /mnt, that is ignored by libzypp!)
mkdir /mnt2
mount /dev/loop0 /mnt2

# verify the size
df -h /mnt2
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       12E   17M   12E   1% /mnt2
# voila! you have a 12EiB file system! enjoy 😉

Note: Obviously even if you have a 12EiB filesystem you cannot save there more data than in the real file system below (in /tmp in this case). If you try you will get write errors, there is no prepetuum mobile…

Speeding Up Unit Tests and Travis Builds in yast2-storage-ng

The new yast2-storage-ng package has a quite large set of unit tests. That’s good, it allows to have less buggy code and make sure the features work as expected.

On the other hand the drawback is that running the tests take too much time. If you have to wait for 3 or 4 minutes after any small change in the code then either you waste too much time or you do not run the tests at all. So we looked into speeding up the tests.

Running Tests in Parallel

The main problem was that all tests were executed sequentially one by one. Even if you have a multi CPU system only one processor was used. It turned out that using the parallel_test Ruby gem allows easily running the tests in parallel utilizing all available processors.

The only possible problem is that there must not be any dependencies or conflicts between the tests otherwise running them in parallel would fail. Fortunately there was only one small issue in the yast2-storage-ng tests and we could enable the parallel tests without much work.

Running Travis Jobs in Parallel

Also the Travis job took quite a lot of time. Running the tests in parallel helped a bit at Travis but still was not good enough.

Fortunately Travis allows running multiple jobs in parallel. Therefore we split the single CI job which runs the tests, builds the package, runs syntax check, etc… into three independent groups which can be started in parallel.

Documentation

If you are interested in details you might check our updated Travis documentation and the Parallel tests documentation. Or check the announcement on the YaST mailing list.

Results

Here are some real numbers to see the speedup:

• Running the test suite locally (rake test:unit): from 2:44 to 0:38 (4.3x speed up on a quad core CPU with hyper-threading enabled)
• Building the package locally (rake osc:build): from 137s to 49s (with cached RPM packages but includes chroot installation)
• Package build in OBS: from 323s-505s to 102s-235s (it highly depends on the speed of the used worker)
• Travis speed up: from 8-10 minutes to 3-4minutes (using both parallel Travis jobs and parallel tests)

This allows us to continue with adding even more tests into the package. :smiley:

What Packages are Provided by a Product Extension/Module?

In the Software Management module we have a Repositories view where you can see the packages grouped by the repository that provides them. But this is not really helpful if you want to see what is delivered with a product Extension or Module, because each product module is composed of several repositories: the originally released packages, the updates, the sources, the debuginfos.

Fortunately, the repositories for each product module are grouped together in a repository Service, and we have added a Services filter to Software Management.

Qt Service filter:

ncurses Service filter:

(Reference: https://fate.suse.com/320573)