Home Home > 2018 > 08
Sign up | Login

Archive for August, 2018

YaST Squad Sprint 61

August 29th, 2018 by

We have to admit that lately we have not been exactly regular and reliable in delivering our blog posts. But with the vacation season coming to an end, we are determined to recover the good pace. Since the proof is in the pudding, here is our latest report, just one week after the previous one.

Improving the user experience in the Services Manager

And talking about the previous report, we presented there several improvements in the YaST Services Manager module, including the new “Apply” and “Show Log” buttons. With the “Apply” button, all changes performed over the services are applied without closing the Services Manager, which allows you to continue using it and to inspect the logs of a service without relaunching the Services Manager again. But this new “Apply” button only makes sense when there is something to save, so during this sprint we have improved the UI to disable the button when nothing has been changed yet. In addition, now it is easier to know what we have edited so far in the Services Manager. For every change over a service, the new value is explicitly highlighted by using the special mark (*). For example, when you change the start mode of a service from “On boot” to “On demand”, you will see “(*) On Demand” in the corresponding column, see example.

https://lizards.opensuse.org/2018/08/22/yast-squad-sprint-59-60/

The list of services and the changes performed on them can be quite long. So in addition to the new mark, now a confirmation popup is shown up when using the “Apply” or “OK” button. This popup will present a summary with all changes that will be applied, that is, what services will be started or stopped, what services will change its start mode to “on boot”, “on demand” or “manually”, and even which will be the new default Systemd Target in case you have modified it. See an example in the following screenshot.

New summary of changes in the Services Manager

These improvements will reach openSUSE Tumbleweed soon and will be available in the upcoming versions of SLE (SLE-15-SP1) and openSUSE Leap (15.1).

Yast2 Systemd classes reorganized

Related to the changes in the Service Manager and in a more developer-oriented note. Yast2 Systemd (the set of YaST components that handle Systemd units under the hood) also has been completely reorganized in a more Ruby compliant way. Moving from YCP-style modules to a set of classes that behave like nice citizens of the Ruby ecosystem in their own proper namespace.

AutoYaST support for Xen virtual partitions

And to continue with refinements over the features introduced in the previous sprint, we have also improved the support for the so-called Xen virtual partitions that we presented in our previous post. As explained there, the old storage stack used to represent the Xen devices like /dev/xvda1 as partitions of a non-existing /dev/xvda hard disk. In the new stack, those devices are treated as they deserve, as independent block devices on themselves with no made-up disks coming from nowhere.

But AutoYaST profiles from a SLE-12 or Leap 42.x still pretend there are hard disks grouping the Xen virtual partitions. So in addition to the fixes introduced in the Partitioner during the previous sprint, we also had to teach the new storage stack how to handle fanciful AutoYaST <drive> sections like this, used to describe groups of Xen devices (xvda1 and xvda2 really exist in the system, xvda doesn’t).

 <drive>
    <device>/dev/xvda</device>

    <partition>
      <partition_nr>1</partition_nr>
      ...information about /dev/xvda1...
    </partition>

    <partition>
      <partition_nr>2</partition_nr>
      ...information about /dev/xvda2...
    </partition>

  </drive>
</pre>

The fix will be released as an installer self-update patch so users installing SLE-15 (with access to a self-update repository) can take advantage of it. In the mid term we will have to come up with a more realistic format to represent such devices in the AutoYaST profiles, but so far the limitations of the current AutoYaST format enforces us to keep the current approach.

Ignoring inactive RAID arrays

But that’s not the only new skill of Storage-ng for this sprint. It also learned how to better manage inactive RAID arrays. MD RAID arrays are built to handle failures of the underlying physical devices. When some of the devices fail, the RAID becomes “degraded” which means the data is still accessible but it’s time to fix things. When too many devices suffer a fault, the RAID becomes inactive and it cannot operate any longer until it’s repaired. Our Partitioner was not handling this situation well, popping up a generic “unexpected situation” error message.

Generic error message in Storage-ng

We have fixed that, and the storage stack doesn’t go nuts any longer if an inactive RAID array is found. Even more, it now shows an “Active: Yes/No” field under the RAID heading to inform the user in case the RAID is in such bad shape.

Partitioner displaying an inactive RAID array

All that will be soon available as a maintenance update for SLE-15 and Leap 15.0. So far, no mechanisms have been introduced to stop the user from modifying an inactive RAID array with the Partitioner. That will come in the future, together with other MD RAID improvements in Storage-ng targeting future releases of SLE and openSUSE Leap.

Media support in the Installation Server module

It’s quite embarrassing but it turned out that the Installation Server YaST module in SLE15 and openSUSE Leap 15.0 is not able to add the SLE15/Leap15 installation media. The reason is that the new media use a different repository format and the Installation Server module crashed when trying to add a new repository.

Fortunately the fix was small and allows adding the new media correctly. We plan to release a maintenance update for SLE15, openSUSE Leap 15.0 and SLE12-SP3. It turned out that the code in SLE11-SP4 is more robust and does not crash so we do not need an update there.

Improved help text for system roles

We recently got a bug report about how hard was to read the help text in the installation screen explaining the system roles. So we took it as an opportunity to try how flexible our help text system is. Adding some richtext format made it look much better in graphical mode and also surprisingly well text mode. Let’s see some screenshots from Leap 15.0, although the fix applies to as well to openSUSE Tumbleweed and the SLE15 family.

This is how it looked before the fix.

The poorly formatted help of the System Roles screen

And that how it looks now, in both Qt and Ncurses, with the new format.

The properly formatted text of the System Roles screen

Proper format even in text mode

Fixed PHP support in the YaST HTTP server module

The YaST HTTP server module allows enabling the PHP support in the Apache web server configuration. However, as the module is not actively developed it turned out that the PHP support was broken. YaST wanted to install the apache2-mod_php5 package which is not available in openSUSE Leap 15.0 or SLE15, there is a newer apache2-mod_php7 package.

After checking the other required packages it turned out that a similar problem exists for some other Apache modules. To avoid this issue in the future again we have added an additional test which checks the availability of all potentially installed packages. If there is a new version or a package is dropped we should be notified earlier by continuous integration instead of an user bug report later.

Stay tuned

Of course, in addition to the mentioned highlights we have fixed several small and medium bugs. And we plan to continue improving YaST in many ways… and to keep you punctually updated. So don’t go too far away.

YaST Squad Sprint 59 & 60

August 22nd, 2018 by

We know, we know… we owed you a report for the 59th sprint! But you know what? We are now delivering this summary which covers sprints 59 and 60 so we are now up to date again.

During these weeks, we have been working mainly in two different areas. On one hand, we have improved YaST services management capabilities, adding support for on demand services and making the user experience slightly better. On the other hand, we are still working in the new storage layer, extending it to support Xen virtual partitions and fixing BIOS MD devices handling, among other things.

Last but not least, our bug fighting squad (do you remember squads, right?) have fixed quite some bugs accross our full stack.

So let’s have a tour to recap some highlights from these sprints.

Services Management Gets Some Love

The team is investing quite some time improving the services management area. In the past, YaST was adapted to play nicely with Systemd (which replaced the old runlevel-based init system). However, there was a lot of room for improvement, as Systemd added some new features when it comes to services management that were not supported in YaST at all.

To explore these changes, let’s start by having a look at the new services manager user interface.

Services Manager UI with On Demand Activation Support

At first sight, you may have spotted some changes: there is an Start Mode menu button where the Enable/Disable used to be; new Apply or Show Logs buttons have been added; the Start/Stop button has been replaced by an Start (or Stop) one… Ok, let’s describe these changes one by one.

Perhaps the most relevant update is the support for socket activated services. But you may be asking what a “socket activated service” is, right? In a nutshell, these are services that are started on demand. For instance, the printing service could be started only when you want to use the printer. And that’s what the Start Mode button is all about. For instance, you may want the printing services to be started only when needed (On Demand) and YaST will take care of setting up Systemd units in order to do so.

Another interesting feature is browsing services logs. Beware that this button will be available only if yast2-journal is installed because it uses the browser provided by this package.

And browsing the logs brings us to the new Apply button. Sometimes, it may be useful to Start a service and check the logs looking for potential problems. In the past, the changes were applied when the OK button was pressed but, unfortunately, the services manager was then closed. Now, with the Apply button, you can ask YaST to made those changes for real without closing the services manager (and check the logs or keep adjusting services).

But that’s not all! Among other small changes (like fixing bug #1080738) we are still improving the UX for this services manager.

Make the Xen virtual partitions great again

Rewriting a piece of YaST that is 20 years old is a never-ending source of surprises. When we though that Storage-ng did already support all the storage technologies handled by the old storage stack, we just found that we had left something behind.

When setting a Xen virtualization environment, it’s possible to define block devices that are seen and named like partitions in the Xen guest, but are in fact disks (or almost-disks, since they cannot be partitioned by regular methods). In other words, the Xen guest can see some block devices called e.g. /dev/xvda1 and /dev/xvda2 without a corresponding /dev/xvda.

From the YaST point of view, you can do whatever you can do with a partition (format it, mount it, etc.), except you can’t delete them or create more (since they are not backed by a disk, from the perspective of the Xen guest). We call those devices “Xen virtual partitions”. The old storage stack used to pretend those devices were just plain partitions of a made-up /dev/xda disk that was not there in the system but still appeared in the Partitioner and in the AutoYaST profile. The new stack simply contained no support at all for Xen virtual partitions.

We fixed that during the last sprint, making the feature available on the SLE-15 installation process thanks to the installer self-update feature. So either if your Xen guest runs a fully up-to-date version of SLE-15, Leap 15.0 or Tumbleweed or if you are installing SLE-15 with access to the installer self-update repositories, virtual partitions are visible in the Hard Disks section of the Partitioner.

It works in a in a similar way to the old (pre Storage-NG) Partitioner, but without artificial disks to group the virtual partitions. See the following screenshots in which xvda is a DVD, xvdb a real disk and xvdc1 a XEN virtual partition (there is no bogus xvdc added by YaST).

Handling Xen Virtual Partitions

Only the Edit button really works as expected, allowing the user to mount and/or format the virtual partition. The other buttons just show the appropriate error for the different unsupported operations.

Handling Xen Virtual Partitions Error Message

I want that on my Leap 15.0 installer

As a side note on the Xen partitions topic, it’s worth mentioning that the self-update feature of the installer is always available for all YaST-based distributions. But, unlike SLE, openSUSE doesn’t offer an official self-update repository, implying the feature is effectively disabled by default in openSUSE Leap. Does it mean that if you want to install Leap on top of a Xen virtual partition you will have to wait for Leap 15.1 or switch to Tumbleweed? Not exactly.

If you really want to use the latest YaST features during the openSUSE Leap 15.0 installation process, there are a couple of ways to achieve it. For example, to install (or upgrade) using the Leap 15.0 Live images. The live images are refreshed once in a while, so they can contain an installer that is more up-to-date than the installer in the Leap 15.0 normal ISO. There are also ways to use the self-update feature with unofficial (and unsupported) repositories. For a summary of all the options, check this comment at Bugzilla. Users of SLE should not need any of these hacks, since there is an official repository for the installer self-update mechanism ensuring SLE15 can always be installed with updated version of all the crucial packages.

A New Widget To Manage Services

Did you think that services management were confined to the services manager? Of course no 🙂 As you may know, there are serveral YaST modules that allow our users to set up several services like DNS, DHCP, Samba, etc. All these modules offer a way to configure how and when those services should be started.

So as part of the effort to improve the services management experience, we have rolled out a new widget which offers some benefits:

  • Allows setting a service to be started On Demand.
  • Offers a consistent an unified interface accross all modules.
  • It is able to handle cases where several services are involved (like yast2-samba-server or yast2-iscsi-client).

By the way, we have adapted all these modules to use the new and shiny widget.

Better Handle Large File Systems in libyui

We fixed an issue with disks larger than 8EiB in the last sprint. However, the fix was intended for SLE12 and SLE15 maintenance update so we could not do big changes and we fixed only the most important parts.

For SLE15-SP1/Leap 15.1 and openSUSE Tumbleweed we can do more changes so we did some backward incompatible improvements.

Originally the size was implemented using long long data type which is a 64-bit signed integer (with the maximum value 8EiB). We switched to the Boost multiprecision C++ library which implements arbitrary precision integers. It works like the Integer class in Ruby which adds more bits to the data when needed.

Of course, it still depends on the limits in the underlying libzypp library which uses a 64-bit signed integer but with 1KiB unit, so the limit should be 8ZiB. With this change we are prepared even for more.

Additionally we added more units for converting to a string representation. Originally it used TiB units which resulted in too big numbers, now the EiB sizes are displayed using the EiB units as expected.

Large filesystems support in Libyui

Moreover we added support for writing unit tests, evaluating the code coverage and reporting the it to coveralls.io. With this support we were able to write the very first unit test in libyui! The code coverage is now at the embarrassing 2% but we have just started!

Rubocop Check Speed Up

In the previous blog post we reported that we had improved running the unit tests in the yast2-storage-ng package. The speed up was based on parallelizing the execution and using all processors. This time we have improved the Rubocop check.

Normally Rubocop scans the directory for the files to check and then processes found files sequentially. And this might take very long time if there are hundreds of files to check.

Similarly as in the previous speed up we take advantage of the available processors and run multiple Rubocop instances in parallel. The implementation is a bit more complicated as Rubocop itself does not support parallel scan at all. But it is possible to evaluate the checked files, split them into the groups according to the number of processors and start a separate Rubocop instance for each group in parallel.

If you are interested in the details or you want to use parallel Rubocop also in your project then check the implementation in the yast-rake Ruby gem.

Of course, this improvement has an important impact in the yast2-storage-ng package where Rubocop needs to check over 600 files. Here you are the numbers:

  • Running Rubocop locally (with hyperthreading enabled):
    • ~3.6x faster on an older quad core CPU (from 44s to 12s)
    • ~6x faster on a new eight core CPU (from 35s to 6s)
  • At Travis: ~1.5x faster (from 69s to 47s), that’s the maximum possible because Travis has about 1.5x CPU build power
  • .

As you can see, for big projects the speed up is very good on local system and even at Travis it is still nice.

What’s next?

Sprint 61 is already running and the new storage layer and services management are again the areas where we are focusing an important part of our resources. Hopefully, in around two weeks, we will publish what we have achieved this time.

Stay tunned!

zypper-upgraderepo-plugin is here

August 7th, 2018 by

zypper-upgraderepo-plugin adds to zypper the ability to check the repository URLs either for the current version or the next release, and upgrade them all at once in order to upgrade the whole system from command line.

This tool started as a personal project when a day I was in the need to upgrade my distro quicker than using a traditional ISO image, Zypper was the right tool but I got a little stuck when I had to handle repositories: some of them were not yet upgraded, others changed slightly in the URL path.

Who knows how to Bash the problem is not exactly a nightmare, and so I did until I needed to make a step further.

The result is zypper-upgraderepo Ruby gem which can be integrated as a zypper plugin just installing the zypper-upgraderepo-plugin package.

Installing zypper-upgraderepo-plugin

Installing zypper-upgraderepo-plugin is as easy as:

  1. Adding my repo:
    sudo zypper ar https://download.opensuse.org/repositories/home:/FabioMux/openSUSE_Leap_42.3/home:FabioMux.repo
  2. Install the package:
    sudo zypper in zypper-upgraderepo-plugin

How to use it

Sometime we want to know the status of current repositories, the command zypper ref does a similar job but it is primarily intended to update the repository’s data and that slow down a bit the whole process.
Instead we can type:

$ zypper upgraderepo --check-current

To know whether or not all the available repositories are upgrade-ready:

$ zypper upgraderepo --check-next


As you can see from the example above all the enabled repositories are ready to upgrade except for the OSS repo which has a slightly different URL.

# The URL used in the openSUSE Leap 42.3
http://download.opensuse.org/distribution/leap/42.3/repo/oss/suse/
# The suggested one for openSUSE Leap 15.0
http://download.opensuse.org/distribution/leap/15.0/repo/oss/

Let’s try again overriding the URL without make any real change:

$ zypper upgraderepo --check-next \
--override-url 8,http://download.opensuse.org/distribution/leap/15.0/repo/oss/

Once everything is ok, and after performed a backup including all the repositories, it’s time to upgrade all the repository at once:

$ sudo zypper upgraderepo --upgrade \
--override-url 8,http://download.opensuse.org/distribution/leap/15.0/repo/oss/

Conclusions

That’s all with the basic commands, more information is available in the wiki page of zypper-upgraderepo gem where all the commands are intended with the only use of the gem, but installing the plugin they are also available as zypper subcommands like shown above, also a man page is available as

$ zypper help upgraderepo