Home Home > Distribution > Factory
Sign up | Login

Archive for the ‘Factory’ Category

YaST Squad Sprint 62

September 12th, 2018 by
  • Jenkins commenting in GitHub pull requests
  • Intel Rapid Start Technology for better sleeping
  • Consistent storage proposal in SLE-12
  • Partitioner: designing the UI for its full potential
  • Partitioner: entire disks as members of a software MD RAID
  • Partitioner: better explanation of unusual conditions
  • A sample of bug fixes

Improved Jenkins Integration

It happened quite often that our Jenkins job failed for some reason after merging a pull request at GitHub. And because the Jenkins is supposed to submit the changes to the build service it happened that the fixes from Git were not released in the RPM packages if nobody noticed the failure. That was a bit confusing because we closed a bug at Bugzilla but the fix was not available anywhere.

To avoid this we have added a wrapper script which runs the original Jenkins command (rake osc:sr in this case) and writes the result as a comment to the respective pull request at GitHub. If a submit request is created it additionally adds a link to it.

Since now you should keep the pull request page open after merging it and wait for the Jenkins status result. If it fails for some reason then try fixing it or ask for help on IRC or the YaST mailing list.

Note: This automation works only for the code branches which are in active development and for packages which have an Jenkins job assigned (most of the YaST packages have).

Intel Rapid Start Technology Support

The Intel Rapid Start Technology allows to use a fast disk (SSD) for suspend-to-RAM to save energy. The idea is that after a given time the contents of RAM will be moved to SSD so that the system can power itself off. When powered on, RAM will be read back. So it’s something like a dynamic changing of suspend-to-RAM to suspend-to-fast-disk.

What does this technology need from the installer? It needs its own dedicated partition where it can store the contents of RAM. To support this technology we added in this sprint the ability to create and recognize such a partition. It looks like this:

Consistent Storage Proposal (SLE-12-SP4 / yast2-storage-old)

We fixed a bug where the behavior was inconsistent if you switched the storage proposal between partition-based and LVM-based / encrypted LVM-based: bsc#1085169.

The behavior was pretty irritating: Initially, it would propose to create a separate “/home” partition, but when you changed the proposal parameters and simply kept that checkbox “[x] separate /home” checked, it would complain that with the current settings a separate “/home” was not possible.

The two code paths did the calculations differently: One accounted for the other partitions that were also proposed like “swap” and “boot” (or “PrEP” or “/efi-boot”), the other did not. We unified that as much as reasonably possible without breaking things, but since calculating when and how to use any boot partitions is quite complex in that old legacy storage stack, we did not go all the way; boot partitions are pretty small, so their size matters only in very pathological fringe cases. We try not to overengineer things, in particular not with the 4th service pack for a business product.

More details in the pull request with the fix.

The Partitioner looks to the future

We have blogged a lot about Storage-NG and the possibilities and features it will bring to the users. But a significant part of its power remain dormant under the surface because we decided to clone the user interface and the functionality of the classical YaST Partitioner for the deadline marked by SLE 15 (and Leap 15.0). But now we are finally able to start exposing those long-awaited features and to bring new ones for the current Tumbleweed and for the future SLE-15-SP1 and Leap 15.1.

The user interface of the Partitioner is already rather packed with functionality, but we want to avoid a too disruptive redesign. So it was time to some pen and paper sessions, trying to find and draw a good way to add exciting new stuff to the Partitioner, including all of the following:

  • Allow entire disks (no partition table) to be added as members of a software MD RAID.
  • Manage (create, modify and delete) partitions within a software MD RAID device.
  • Make possible to format an entire disk (no partition table) and/or define a mount point for it (just as we do with partitions or LVM logical volumes).
  • Manage Bcache so the user can set and configure which devices will be used to speed up others.

As usual, we consulted some UI experts in the process and the result is this first version of a document, which summarizes how to incorporate all that to the Partitioner, including some alternatives we are considering for the near future.

That document will become the cornerstone of future developments. Sometimes you need to spend a sprint doing other stuff (like researching and documenting) before you can go ahead with writing code.

Partitioner: full disks as members of a software MD RAID

The first of the features described in the previous document is already available for Tumbleweed users (or it will be as soon as the integration process concludes) and, thus, ready for the upcoming releases of SLE and Leap. Now the Partitioner offers full disks as “Available Devices” in the RAID screen, following the criteria and considerations explained at the document.

That brings even more ways of combining devices together (disks, partitions, software RAIDs, you name it) to create a storage setup. As a result, we decided it was important to explain the situation when some combinations are not possible right away, likely because they need some previous step. Which brings us to…

Partitioner: more specific errors when a device is in use

In general, most of the checks already present in the Partitioner were already able to correctly handle situations in which the disk was a direct member of an MD RAID or an LVM volume group. But the message about the device being in use was not informative enough.

Now the message includes the name of the device that is making the operation impossible (it’s usually one, but there are corner cases in which it can be more than one), so the user has some clue about how to fix it.

Partitioner: improved creation of partition table

One part of the Partitioner that was specially bad at explaining the current situation and the possible consequences was the workflow of “Create New Partition Table”, which also used to exhibit a behavior quite inconsistent with the rest of the Partitioner actions.

In SLE-15 and openSUSE Leap 15.0, the “Create New Partition Table” button immediately presents a form to select the partition table type in case the device supports more than one.

And after the user selects one type it always shows a warning about all kind of devices to be destroyed, no matter if some device is really affected or not.

Even better, if only one partition table type is possible, it still shows the form but with no question. So creating a partition table in a completely empty DASD device result in a misleading warning (nothing is going to be destroyed) on top of an empty wizard.

So the whole action was reimplemented to display the warning only if some devices are indeed going to be affected (including the list of affected devices) and to display that warning as soon as the user clicks the button (as any other Partitioner action).

As seen in the screenshot, the check handles correctly situations in which the disk as a whole (no partitions) is part of an MD RAID or an LVM setup. And, of course, there are no empty wizard steps in the case of DASD or nothing like that. Now the workflow works in the expected way on each situation.

In short, the Storage-NG Partitioner is moving away, step by step, from being a 1:1 clone of the historical Partitioner to offer more functionality and usability. And there are more improvements to come in that area.

Partitioner: Unmounting devices in advance

The Partitioner allows to extensively configure your system storage devices. You can perform a lot of different kind of actions, from changing the label of a file system to creating a complex configuration by using LVM or RAIDs. Each modification you perform is stored in memory, so the real system is not altered at all until you confirm to apply the changes as last step. But in some circumstances, the Partitioner could not be able to perform some of the required actions, and it would fail when trying to modify the real system. One action that sometimes might fail is unmounting a device. This action might fail for several reasons, but the most common is because the file system is busy. And moreover, sometimes there are actions that require the device to be unmounted, for example, for deleting a partition, so the Partitioner would try to automatically unmount it.

During this sprint, the Partitioner has recovered its ability of unmounting devices on the fly to avoid possible failures when applying the changes. Now, if you want to delete a currently mounted device (e.g., a LVM Logical Volume) you will be asked in advance to unmount it. If you accept, the Partitioner will try to unmount the device on the fly without waiting to apply all the changes. In case the unmount action fails, you will be informed about the problem and you might try to manually solve the problem before the Partitioner applies the changes in your system. Of course, you can also continue without unmounting the device and the Partitioner will try to automatically unmount it after accepting all the changes.

Another task that might require unmounting the device is resizing the filesystem. The Partitioner will ask you about unmounting the device when the filesystem does not support resizing while being mounted. And, even when the filesystem does support it, you still might be requested to unmount the device. For example when you want to extend a device by more than 50 GiB. That task might be quite slow and it is highly recommended to unmount the device to speed up the resizing time, otherwise it could take hours.

Bug Fixes

Of course, we continue fighting against bugs. Thus, from this sprint on, alongside other minor stuff, the system

YaST Squad Sprint 61

August 29th, 2018 by

We have to admit that lately we have not been exactly regular and reliable in delivering our blog posts. But with the vacation season coming to an end, we are determined to recover the good pace. Since the proof is in the pudding, here is our latest report, just one week after the previous one.

Improving the user experience in the Services Manager

And talking about the previous report, we presented there several improvements in the YaST Services Manager module, including the new “Apply” and “Show Log” buttons. With the “Apply” button, all changes performed over the services are applied without closing the Services Manager, which allows you to continue using it and to inspect the logs of a service without relaunching the Services Manager again. But this new “Apply” button only makes sense when there is something to save, so during this sprint we have improved the UI to disable the button when nothing has been changed yet. In addition, now it is easier to know what we have edited so far in the Services Manager. For every change over a service, the new value is explicitly highlighted by using the special mark (*). For example, when you change the start mode of a service from “On boot” to “On demand”, you will see “(*) On Demand” in the corresponding column, see example.

https://lizards.opensuse.org/2018/08/22/yast-squad-sprint-59-60/

The list of services and the changes performed on them can be quite long. So in addition to the new mark, now a confirmation popup is shown up when using the “Apply” or “OK” button. This popup will present a summary with all changes that will be applied, that is, what services will be started or stopped, what services will change its start mode to “on boot”, “on demand” or “manually”, and even which will be the new default Systemd Target in case you have modified it. See an example in the following screenshot.

New summary of changes in the Services Manager

These improvements will reach openSUSE Tumbleweed soon and will be available in the upcoming versions of SLE (SLE-15-SP1) and openSUSE Leap (15.1).

Yast2 Systemd classes reorganized

Related to the changes in the Service Manager and in a more developer-oriented note. Yast2 Systemd (the set of YaST components that handle Systemd units under the hood) also has been completely reorganized in a more Ruby compliant way. Moving from YCP-style modules to a set of classes that behave like nice citizens of the Ruby ecosystem in their own proper namespace.

AutoYaST support for Xen virtual partitions

And to continue with refinements over the features introduced in the previous sprint, we have also improved the support for the so-called Xen virtual partitions that we presented in our previous post. As explained there, the old storage stack used to represent the Xen devices like /dev/xvda1 as partitions of a non-existing /dev/xvda hard disk. In the new stack, those devices are treated as they deserve, as independent block devices on themselves with no made-up disks coming from nowhere.

But AutoYaST profiles from a SLE-12 or Leap 42.x still pretend there are hard disks grouping the Xen virtual partitions. So in addition to the fixes introduced in the Partitioner during the previous sprint, we also had to teach the new storage stack how to handle fanciful AutoYaST <drive> sections like this, used to describe groups of Xen devices (xvda1 and xvda2 really exist in the system, xvda doesn’t).

 <drive>
    <device>/dev/xvda</device>

    <partition>
      <partition_nr>1</partition_nr>
      ...information about /dev/xvda1...
    </partition>

    <partition>
      <partition_nr>2</partition_nr>
      ...information about /dev/xvda2...
    </partition>

  </drive>
</pre>

The fix will be released as an installer self-update patch so users installing SLE-15 (with access to a self-update repository) can take advantage of it. In the mid term we will have to come up with a more realistic format to represent such devices in the AutoYaST profiles, but so far the limitations of the current AutoYaST format enforces us to keep the current approach.

Ignoring inactive RAID arrays

But that’s not the only new skill of Storage-ng for this sprint. It also learned how to better manage inactive RAID arrays. MD RAID arrays are built to handle failures of the underlying physical devices. When some of the devices fail, the RAID becomes “degraded” which means the data is still accessible but it’s time to fix things. When too many devices suffer a fault, the RAID becomes inactive and it cannot operate any longer until it’s repaired. Our Partitioner was not handling this situation well, popping up a generic “unexpected situation” error message.

Generic error message in Storage-ng

We have fixed that, and the storage stack doesn’t go nuts any longer if an inactive RAID array is found. Even more, it now shows an “Active: Yes/No” field under the RAID heading to inform the user in case the RAID is in such bad shape.

Partitioner displaying an inactive RAID array

All that will be soon available as a maintenance update for SLE-15 and Leap 15.0. So far, no mechanisms have been introduced to stop the user from modifying an inactive RAID array with the Partitioner. That will come in the future, together with other MD RAID improvements in Storage-ng targeting future releases of SLE and openSUSE Leap.

Media support in the Installation Server module

It’s quite embarrassing but it turned out that the Installation Server YaST module in SLE15 and openSUSE Leap 15.0 is not able to add the SLE15/Leap15 installation media. The reason is that the new media use a different repository format and the Installation Server module crashed when trying to add a new repository.

Fortunately the fix was small and allows adding the new media correctly. We plan to release a maintenance update for SLE15, openSUSE Leap 15.0 and SLE12-SP3. It turned out that the code in SLE11-SP4 is more robust and does not crash so we do not need an update there.

Improved help text for system roles

We recently got a bug report about how hard was to read the help text in the installation screen explaining the system roles. So we took it as an opportunity to try how flexible our help text system is. Adding some richtext format made it look much better in graphical mode and also surprisingly well text mode. Let’s see some screenshots from Leap 15.0, although the fix applies to as well to openSUSE Tumbleweed and the SLE15 family.

This is how it looked before the fix.

The poorly formatted help of the System Roles screen

And that how it looks now, in both Qt and Ncurses, with the new format.

The properly formatted text of the System Roles screen

Proper format even in text mode

Fixed PHP support in the YaST HTTP server module

The YaST HTTP server module allows enabling the PHP support in the Apache web server configuration. However, as the module is not actively developed it turned out that the PHP support was broken. YaST wanted to install the apache2-mod_php5 package which is not available in openSUSE Leap 15.0 or SLE15, there is a newer apache2-mod_php7 package.

After checking the other required packages it turned out that a similar problem exists for some other Apache modules. To avoid this issue in the future again we have added an additional test which checks the availability of all potentially installed packages. If there is a new version or a package is dropped we should be notified earlier by continuous integration instead of an user bug report later.

Stay tuned

Of course, in addition to the mentioned highlights we have fixed several small and medium bugs. And we plan to continue improving YaST in many ways… and to keep you punctually updated. So don’t go too far away.

YaST Squad Sprint 59 & 60

August 22nd, 2018 by

We know, we know… we owed you a report for the 59th sprint! But you know what? We are now delivering this summary which covers sprints 59 and 60 so we are now up to date again.

During these weeks, we have been working mainly in two different areas. On one hand, we have improved YaST services management capabilities, adding support for on demand services and making the user experience slightly better. On the other hand, we are still working in the new storage layer, extending it to support Xen virtual partitions and fixing BIOS MD devices handling, among other things.

Last but not least, our bug fighting squad (do you remember squads, right?) have fixed quite some bugs accross our full stack.

So let’s have a tour to recap some highlights from these sprints.

Services Management Gets Some Love

The team is investing quite some time improving the services management area. In the past, YaST was adapted to play nicely with Systemd (which replaced the old runlevel-based init system). However, there was a lot of room for improvement, as Systemd added some new features when it comes to services management that were not supported in YaST at all.

To explore these changes, let’s start by having a look at the new services manager user interface.

Services Manager UI with On Demand Activation Support

At first sight, you may have spotted some changes: there is an Start Mode menu button where the Enable/Disable used to be; new Apply or Show Logs buttons have been added; the Start/Stop button has been replaced by an Start (or Stop) one… Ok, let’s describe these changes one by one.

Perhaps the most relevant update is the support for socket activated services. But you may be asking what a “socket activated service” is, right? In a nutshell, these are services that are started on demand. For instance, the printing service could be started only when you want to use the printer. And that’s what the Start Mode button is all about. For instance, you may want the printing services to be started only when needed (On Demand) and YaST will take care of setting up Systemd units in order to do so.

Another interesting feature is browsing services logs. Beware that this button will be available only if yast2-journal is installed because it uses the browser provided by this package.

And browsing the logs brings us to the new Apply button. Sometimes, it may be useful to Start a service and check the logs looking for potential problems. In the past, the changes were applied when the OK button was pressed but, unfortunately, the services manager was then closed. Now, with the Apply button, you can ask YaST to made those changes for real without closing the services manager (and check the logs or keep adjusting services).

But that’s not all! Among other small changes (like fixing bug #1080738) we are still improving the UX for this services manager.

Make the Xen virtual partitions great again

Rewriting a piece of YaST that is 20 years old is a never-ending source of surprises. When we though that Storage-ng did already support all the storage technologies handled by the old storage stack, we just found that we had left something behind.

When setting a Xen virtualization environment, it’s possible to define block devices that are seen and named like partitions in the Xen guest, but are in fact disks (or almost-disks, since they cannot be partitioned by regular methods). In other words, the Xen guest can see some block devices called e.g. /dev/xvda1 and /dev/xvda2 without a corresponding /dev/xvda.

From the YaST point of view, you can do whatever you can do with a partition (format it, mount it, etc.), except you can’t delete them or create more (since they are not backed by a disk, from the perspective of the Xen guest). We call those devices “Xen virtual partitions”. The old storage stack used to pretend those devices were just plain partitions of a made-up /dev/xda disk that was not there in the system but still appeared in the Partitioner and in the AutoYaST profile. The new stack simply contained no support at all for Xen virtual partitions.

We fixed that during the last sprint, making the feature available on the SLE-15 installation process thanks to the installer self-update feature. So either if your Xen guest runs a fully up-to-date version of SLE-15, Leap 15.0 or Tumbleweed or if you are installing SLE-15 with access to the installer self-update repositories, virtual partitions are visible in the Hard Disks section of the Partitioner.

It works in a in a similar way to the old (pre Storage-NG) Partitioner, but without artificial disks to group the virtual partitions. See the following screenshots in which xvda is a DVD, xvdb a real disk and xvdc1 a XEN virtual partition (there is no bogus xvdc added by YaST).

Handling Xen Virtual Partitions

Only the Edit button really works as expected, allowing the user to mount and/or format the virtual partition. The other buttons just show the appropriate error for the different unsupported operations.

Handling Xen Virtual Partitions Error Message

I want that on my Leap 15.0 installer

As a side note on the Xen partitions topic, it’s worth mentioning that the self-update feature of the installer is always available for all YaST-based distributions. But, unlike SLE, openSUSE doesn’t offer an official self-update repository, implying the feature is effectively disabled by default in openSUSE Leap. Does it mean that if you want to install Leap on top of a Xen virtual partition you will have to wait for Leap 15.1 or switch to Tumbleweed? Not exactly.

If you really want to use the latest YaST features during the openSUSE Leap 15.0 installation process, there are a couple of ways to achieve it. For example, to install (or upgrade) using the Leap 15.0 Live images. The live images are refreshed once in a while, so they can contain an installer that is more up-to-date than the installer in the Leap 15.0 normal ISO. There are also ways to use the self-update feature with unofficial (and unsupported) repositories. For a summary of all the options, check this comment at Bugzilla. Users of SLE should not need any of these hacks, since there is an official repository for the installer self-update mechanism ensuring SLE15 can always be installed with updated version of all the crucial packages.

A New Widget To Manage Services

Did you think that services management were confined to the services manager? Of course no 🙂 As you may know, there are serveral YaST modules that allow our users to set up several services like DNS, DHCP, Samba, etc. All these modules offer a way to configure how and when those services should be started.

So as part of the effort to improve the services management experience, we have rolled out a new widget which offers some benefits:

  • Allows setting a service to be started On Demand.
  • Offers a consistent an unified interface accross all modules.
  • It is able to handle cases where several services are involved (like yast2-samba-server or yast2-iscsi-client).

By the way, we have adapted all these modules to use the new and shiny widget.

Better Handle Large File Systems in libyui

We fixed an issue with disks larger than 8EiB in the last sprint. However, the fix was intended for SLE12 and SLE15 maintenance update so we could not do big changes and we fixed only the most important parts.

For SLE15-SP1/Leap 15.1 and openSUSE Tumbleweed we can do more changes so we did some backward incompatible improvements.

Originally the size was implemented using long long data type which is a 64-bit signed integer (with the maximum value 8EiB). We switched to the Boost multiprecision C++ library which implements arbitrary precision integers. It works like the Integer class in Ruby which adds more bits to the data when needed.

Of course, it still depends on the limits in the underlying libzypp library which uses a 64-bit signed integer but with 1KiB unit, so the limit should be 8ZiB. With this change we are prepared even for more.

Additionally we added more units for converting to a string representation. Originally it used TiB units which resulted in too big numbers, now the EiB sizes are displayed using the EiB units as expected.

Large filesystems support in Libyui

Moreover we added support for writing unit tests, evaluating the code coverage and reporting the it to coveralls.io. With this support we were able to write the very first unit test in libyui! The code coverage is now at the embarrassing 2% but we have just started!

Rubocop Check Speed Up

In the previous blog post we reported that we had improved running the unit tests in the yast2-storage-ng package. The speed up was based on parallelizing the execution and using all processors. This time we have improved the Rubocop check.

Normally Rubocop scans the directory for the files to check and then processes found files sequentially. And this might take very long time if there are hundreds of files to check.

Similarly as in the previous speed up we take advantage of the available processors and run multiple Rubocop instances in parallel. The implementation is a bit more complicated as Rubocop itself does not support parallel scan at all. But it is possible to evaluate the checked files, split them into the groups according to the number of processors and start a separate Rubocop instance for each group in parallel.

If you are interested in the details or you want to use parallel Rubocop also in your project then check the implementation in the yast-rake Ruby gem.

Of course, this improvement has an important impact in the yast2-storage-ng package where Rubocop needs to check over 600 files. Here you are the numbers:

  • Running Rubocop locally (with hyperthreading enabled):
    • ~3.6x faster on an older quad core CPU (from 44s to 12s)
    • ~6x faster on a new eight core CPU (from 35s to 6s)
  • At Travis: ~1.5x faster (from 69s to 47s), that’s the maximum possible because Travis has about 1.5x CPU build power
  • .

As you can see, for big projects the speed up is very good on local system and even at Travis it is still nice.

What’s next?

Sprint 61 is already running and the new storage layer and services management are again the areas where we are focusing an important part of our resources. Hopefully, in around two weeks, we will publish what we have achieved this time.

Stay tunned!

YaST Squad Sprint 58

July 23rd, 2018 by

Squads in the Team

In the previous post we explained the squads idea and said we would tell more in this report. Thus, we should mention that we finally did three squads for this sprint:

  • The Sockets & Services Squad working on supporting systemd sockets properly and other systemd related tasks (all in the YaST context, of course).
  • The Qt and UI Squad working on user interface things like adding a new view to the package selection to show packages managed by a service, and also some control center improvements.
  • The Bug Fighting Squad handling bugs that are coming in on a daily basis and in our backlog.

Fixed Issues with Disks Larger than 8EiB

It’s quite unlikely that you have at home a disk storage larger than 8EiB (eight exbibytes, 263). But in enterprise or cloud world it might be possible.

And it turned out that the YaST package manager does not handle such large disks well. At the start you would see this false error message:

The Problems


There are two problems:

  • There definitely is a lot of free space on the disk, the error telling the user the space is running out is simply lying.
  • The disk sizes are displayed as wrong negative values.

It turned out that the problem was caused by using the signed 64-bit integer data type which overflows for values bigger than 8EiB and the number becomes negative.

The Fix

We had to fix several places, each required some different solution.

  • Use unsigned 64-bit integers where possible, that obviously avoids overflow.
  • The numbers from libzypp use KiB units, at some places we need to convert that number to MiB. But first we converted to plain bytes (by multiplying by 1024) and then divide by 1MiB. And this first multiply step might cause overflow. Instead we simply convert KiB to MiB directly by dividing by 1024 without risk of overflow in the middle.
  • Use floating point double data type for converting the values to a human readable text or to percents. The double has wider range and in these cases we do not need exact precision so rounding in floating point operations does not matter.
  • Ignore a negative number in the free space check. At one place the value goes through the YaST component system which uses signed integer and this cannot be easily changed. In that case we consider negative free space as enough for installing any package, more than 8EiB free space should be enough for any package™.


There is still some minor issue with the large numbers. The highest supported unit is TiB so even very big numbers are displayed in TiB units as on the screenshot above. The fix is planned to be released as a maintenance update and this change would break the backward compatibility so we will improve it later but only for the future releases.

Testing

But the problem was how to test the behavior? You could fake some numbers in the code but for full testing or QA validation it would be nice to test on a real disk. But you usually do not have such a large storage for testing…

Fortunately in Linux it is possible to fake such large file system quite easily using sparse files and loop devices. Here is a short how to:

# create two big sparse files
truncate -s 6E /tmp/huge_file1
truncate -s 6E /tmp/huge_file2

# create block devices via loopback
losetup -f /tmp/huge_file1
losetup -f /tmp/huge_file2

# get the loop back device names, the names might be different
# if the system already uses some other loop back devices
losetup -a
/dev/loop1: [0056]:1171324 (/tmp/huge_file2)
/dev/loop0: [0056]:1171323 (/tmp/huge_file1)

# create a btrfs file system over both "disks"
mkfs.btrfs -K /dev/loop0 /dev/loop1

# mount it on /mnt2 (or whatever else, do no use /mnt, that is ignored by libzypp!)
mkdir /mnt2
mount /dev/loop0 /mnt2

# verify the size
df -h /mnt2
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       12E   17M   12E   1% /mnt2
# voila! you have a 12EiB file system! enjoy 😉

Note: Obviously even if you have a 12EiB filesystem you cannot save there more data than in the real file system below (in /tmp in this case). If you try you will get write errors, there is no prepetuum mobile…

Speeding Up Unit Tests and Travis Builds in yast2-storage-ng

The new yast2-storage-ng package has a quite large set of unit tests. That’s good, it allows to have less buggy code and make sure the features work as expected.

On the other hand the drawback is that running the tests take too much time. If you have to wait for 3 or 4 minutes after any small change in the code then either you waste too much time or you do not run the tests at all. So we looked into speeding up the tests.

Running Tests in Parallel

The main problem was that all tests were executed sequentially one by one. Even if you have a multi CPU system only one processor was used. It turned out that using the parallel_test Ruby gem allows easily running the tests in parallel utilizing all available processors.

The only possible problem is that there must not be any dependencies or conflicts between the tests otherwise running them in parallel would fail. Fortunately there was only one small issue in the yast2-storage-ng tests and we could enable the parallel tests without much work.

Running Travis Jobs in Parallel

Also the Travis job took quite a lot of time. Running the tests in parallel helped a bit at Travis but still was not good enough.

Fortunately Travis allows running multiple jobs in parallel. Therefore we split the single CI job which runs the tests, builds the package, runs syntax check, etc… into three independent groups which can be started in parallel.

Documentation

If you are interested in details you might check our updated Travis documentation and the Parallel tests documentation. Or check the announcement on the YaST mailing list.

Results

Here are some real numbers to see the speedup:

  • Running the test suite locally (rake test:unit): from 2:44 to 0:38 (4.3x speed up on a quad core CPU with hyper-threading enabled)
  • Building the package locally (rake osc:build): from 137s to 49s (with cached RPM packages but includes chroot installation)
  • Package build in OBS: from 323s-505s to 102s-235s (it highly depends on the speed of the used worker)
  • Travis speed up: from 8-10 minutes to 3-4minutes (using both parallel Travis jobs and parallel tests)

This allows us to continue with adding even more tests into the package. :smiley:

What Packages are Provided by a Product Extension/Module?

In the Software Management module we have a Repositories view where you can see the packages grouped by the repository that provides them. But this is not really helpful if you want to see what is delivered with a product Extension or Module, because each product module is composed of several repositories: the originally released packages, the updates, the sources, the debuginfos.

Fortunately, the repositories for each product module are grouped together in a repository Service, and we have added a Services filter to Software Management.

Qt Service filter:

ncurses Service filter:

(Reference: https://fate.suse.com/320573)

Update on YaST Development Status

July 5th, 2018 by

Five weeks without blogging is certainly a quite an hiatus for the YaST Team. But fear no more, we are back! This is the first time in quite a while in which our post is not titled “Highlights of YaST Development Sprint” and there are good reasons for that.

Adapting the YaST Team Structure the Agile Way

Now that openSUSE Leap 15.0 is out and SUSE Enterprise Linux 15 is ready to be shipped, we felt it was time to rethink our activities. For the duration of the storage-ng development, we had split the YaST team into two sub-teams: Team S for Storage and Team R for the Rest. But now new challenges await us; there are some things that were pushed aside because getting storage-ng into an acceptable state had top priority.

We decided we’d try an approach that other development teams in SUSE have already been using successfully: split up the YaST team into “squads” of 3-5 people each for the duration of a couple of sprints. Each squad is centered around a big topic that needs to be addressed. There is no long-term fixed assignment of anyone to any squad; the idea is to shuffle people and thus know-how around as needed, of course taking each developer’s interests into account. So the squads and the topics will change every few weeks.

Is this the pure spirit of Scrum and the agile bible? We don’t know. And we don’t care. The agile spirit is to adapt your work based on what makes sense in every moment. We work the agile way, so the way of working also has to be agile.

The next sprint’s report will contain more information about the first set of squads and the results they are delivering. But meanwhile we have done much more than just reorganizing our forces. While the sprint-based work was suspended (thus the blog title not containing the word “sprint”), the YaST team still managed to put out of the door quite some features, improvements and bug fixes targeting mainly Tumbleweed.

Expert Partitioner: Moving Partitions

After quite some effort, the YaST team has completely rewritten the Expert Partitioner from scratch using the new storage stack (a.k.a. storage-ng). And although this new Expert Partitioner already offers practically all the same features than the old one, some last options are still coming. One them in the button for moving partitions, which saves us of a lot unnecessary work in many cases. For example, imagine you are installing openSUSE Tumbleweed and the installer automatically proposes you to create a partition for root and, just following it, a second partition for home. In case you don’t like the default proposed sizes (e.g. because you want a bigger root), you have to use the Expert Partitioner to fix the situation. You have to completely remove the home partition, resize root for enlarging it and then create home again with the same options than before.

Now, with the “Move” button, this kind of modifications are much easier. For that example, you can accomplish exactly the same by simply resizing home (without deleting it completely) and moving the resized home closer to the end of the disk (by using Move button). After moving the home partition, you have enough free space for enlarging the root partition. In the following screenshot you can see this dialog for moving partitions.

Moving partitions

One important thing to take into account is that the movement of partitions is only possible for new partitions, that is, it is not possible to move partitions that already exist on disk.

Trying to move an existing partition

YaST Masking Systemd Mount and Swap Units

And speaking about the Partitioner and its relationship with the rest of the system, the transition from SysVinit to Systemd changed the behavior of (open)SUSE concerning mounting devices. Systemd generates mount units for various file systems, e.g. those listed in /etc/fstab. The result is that Systemd may automatically mount any file system, even if that file system has been manually unmounted before. This can be problematic when the user needs the file system to be unmounted for certain operations, like resizing or unplugging.

Thus, now the Partitioner uses a new mechanism to prevent that to happen during its execution. Starting with version 4.0.194, the yast2-storage-ng package includes and uses the script /usr/lib/YaST2/bin/mask-systemd-units to mask all mount and swap units one by one. The script might also be useful for direct use of system administrators. So… profit!

Showing Logs the Systemd Way

And since we speak about how Systemd has changed the way the overall system works, it’s also worth noticing how more and more services has been adopting the Systemd journal for its logging purposes.

Some of the existing YaST modules to configure a given service include a button to show the logs of such service. In the past, they used to display the content of /var/log/messages with some basic filtering to ensure only the information relative to the service (e.g. tftp) was shown. But that didn’t work out of the box for services already using the Systemd journal, and we had gotten quite some bug reports about it.

Fortunately, the solution is really at our fingertips. You surely know by now that there is a YaST module for viewing the journal content with powerful queries for filtering, searching and so so on. The obvious solution is to use that YaST journal module also within other YaST modules, in order to show domain specific logs.

So far we adapted the YaST tftp module, but it will be easy to fix also other places that use the old approach that no longer works. And this is how it looks when you click the “Show Logs” button in the YaST module to configure tftp.

Journal entries for the tftp module

Usability Improvement in the Repositories Manager

The YaST repositories manager displays the repositories sorted by priority. But some people have a lot of repositories in their system and make no use of the priorities. Since there was not a clear second criteria, the order of the repository list looked quite arbitrary in those cases. Now all the repositories with the same priority are sorted by name, which makes more sense. See how it looked before the improvement.

List of repositories sorted only by priority

And compare to how it looks now.

List of repositories sorted by priority and name

Handling Inconsistent Boot Methods During Upgrade

We got a rather interesting amount of bug reports for openSUSE Leap 15.0 about collisions between the grub2 and grub2-efi bootloaders during the upgrade process. The root cause was that the installation medium used a different booting mode than the installed system being upgraded. For example, the installed system uses EFI boot but the upgrade is executed from a DVD booted via legacy mode (i.e. disabling EFI). In that case, the kernel running from the DVD does not expose some devices that are needed to write to the EFI boot manager. Moreover, it causes troubles to the updater itself, which does not expect this situation.

Looking at the majority of the bug reports, it is obvious that in most cases it happens by accident rather than the user consciously trying to mix both boot modes. So to improve the user experience we added a warning that will be displayed when this situation is detected, before starting the upgrade. That gives the user the possibility to fix the problem or to continue if the situation is really intentional.

Below you can see how it looks, both in graphical and text mode, in a patched openSUSE Leap 15.0 installation media, since the feature was developed too late to be included in the official installation images.

Graphical warning about inconsistent boot mode

Text-mode warning about inconsistent boot mode

What’s Next? Hack Week!

As commented at the beginning of the post, we have restarted the sprint-based work, although with a little twist to try out the squads approach. But before we come back to you to show the results of the first squad-based sprint, we have something else to do – Hack Week 17!.

Again it’s the time of the year for all SUSE Engineers (and any Open Source enthusiast willing to join) to innovate and learn new stuff. So please forgive us if we go too deep into playing and we are less responsive next week. See you again soon!

Highlights of YaST Development Sprint 57

May 31st, 2018 by

Three weeks from our last update on this blog. Time flies when you are busy! As you know, openSUSE Leap 15.0 was released in the meantime, which also means the active development of SLE15 is coming to an end… so time to look a little bit further into the future.

That’s why we had a face-to-face workshop with the whole YaST Team at the beautiful city of Prague during several days right before joining the openSUSE Conference 2018.

But we have done much more in three weeks than attending workshops and conferences. Apart from last-minute fixes, here you have a list of some interesting changes we have done in YaST in this period. Take into account that some of these changes didn’t make it into Leap 15.0, although all will be available in SLES15 and are probably already integrated into openSUSE Tumbleweed.

Fine tuning installer behavior in small disks

As you may know, the default installation of SLE and both openSUSE distribution enables Btrfs snapshots in the root partition alongside separate partitions for /home and swap. That means a default installation needs quite some space. In SLE12 and openSUSE Leap 42.X, if such disk space was not there the installer silently tries to disable the separate /home and even the snapshots in order to be able to create an initial proposal.

That behavior has become configurable for each product and role with Storage-ng and during the last sprint there was some controversy about what the configuration should be, both for openSUSE and the SLE family. It may look like a minor problem, but it becomes very relevant in virtualization environment (where virtual disks smaller than 10 GiB are not uncommon) or certain architectures with special storage devices like s390 and ARM.

The final decision was to never disable snapshots automatically in the case of openSUSE, so the user will be forced to manually go through the Guided Setup and explicitly disable snapshots to install in a small disk. In the SLE case, it was decided to keep the traditional behavior (automatically disabling snapshots if really needed) but making the situation more visible by adding a previous sentence to explain how the initial proposal was calculated.

So the installation in a normal disk would look like this.

Default initial partitioning proposal

While the installation in a very small disk displays some information similar to the following screen (the wording was slightly improved after taking the screenshots).

Adjusted initial partitioning proposal

The explanatory text preceding the list of actions will be available in all products based on SLE15, but will not be there for Leap 15.0, since the modification to the installer was not ready on time for the deadline and, moreover, would have been impossible to get the translations on time.

By the way, if you are interested in a more in-depth explanation on how the partitioning proposal adapts to all kind of situations like small disks and other scenarios, don’t hesitate to check Iván’s presentation at openSUSE Conference 2018 detailing its internals.

More parameter passing for s390

And talking about uncommon scenarios and the s390 architecture, you may remember that in the latest sprint we improved the handling of the persistent network device names kernel parameter for such systems. Shortly after, we found out a similar improvement was needed also for the FIPS parameter.

FIPS is a military encryption standard in USA. If the installation is started using the corresponding parameter, YaST will enforce strong encryption and will install an specific FIPS pattern. Moreover, after the recent fix, a system installed in hardened mode s390 will continue operating in this mode after the installation.

Fun with MD RAIDs

As SLE15 comes closer, future users start testing the system with more exotic and complex hardware setups. Same applies to openSUSE Leap 15.0 right after the official release. As a result of all that testing, we found several scenarios in which Storage-ng got confused about MD RAIDs defined by some specific hardware or manually by the user before starting the installation.

By default, the old storage didn’t handle partitions within software RAIDs and it didn’t handle software RAIDs directly on top of full disks (with no partitions in the physical disks). For the first version of Storage-ng present in Leap 15.0 and SLE15, we tried to implement the same behavior with the intention to rethink the whole thing and open new possibilities in the close future. Check more about the present and future of Storage-ng in Ancor’s talk at openSUSE Conference 2018.

Unfortunately, while trying to replicate the old storage behavior with software-defined MD RAIDs, we overlooked some heuristic that was hidden in the old implementation to recognize some special setups in which a given RAID device currently detected as regular software-defined RAIDs should be treated like hardware RAIDs. That’s the case of Software RAID Virtual Disks defined on a S130/S140 controller on DellEMC PowerEdge Servers through the BIOS Interface. We also found that some users used to produce a similar situation by manually creating software MD RAIDs and creating partitions within them before starting the installation.

With the preparation of SLE15 already in the final stages and with openSUSE Leap 15 already out, it was too late to introduce drastic changes in how MD RAIDs are detected and used. To mitigate the problem while limiting the potential breakage, we reintroduced an ancient installer parameter. Now, when we run the installer using LIBSTORAGE_MDPART=1, all existing software-defined RAIDs will be considered as BIOS RAIDs.

Using LIBSTORAGE_MDPART

The new parameter is not available in Leap 15.0 (we added it too late) and will hopefully not be necessary anymore in future versions of SLE and openSUSE, since the short term plan is to redesign everything about how MD RAIDs are handled during installation.

And even more fun with MD RAIDs

Another example of RAID that looks like defined by software but is indeed assembled by BIOS is the Intel RSTe technology. In this case, the usage of LIBSTORAGE_MDPART is not needed, but still we found the bootloader installation to be broken because YaST was once again getting confused by the mixed RAID setup.

Fortunately it was possible to fix the issue and verify the solution in only two days, despite the YaST Team not having direct access to the hardware, thanks to the outstanding help of the user reporting the bug. Connecting users and developers directly always produces great results… and that’s one of the reasons open source rocks so much!

Improved error reporting for wrong bootloader in AutoYaST

That was not the only improvement in the bootloader handling done during this sprint. We also invested some time improving the user experience in AutoYAST, since the error message displayed when using an EFI variant not supported in the system architecture was far from being useful or even informative.

So alongside a more clear message, AutoYaST will now list all the possible values supported on the given architecture to better guide the user.

More precise bootloader error in AutoYaST

Setting the default subvolume name in AutoYaST

AutoYaST also received improvements in other areas, like making use of the new possibilities offered by Storage-ng. The new storage layer allows the user to set different default subvolumes (or none at all) for every Btrfs file system. As shown in the example below, a prefix name can be specified for each partition using the subvolumes_prefix.

<partition>
  <mount>/</mount>
  <filesystem config:type="symbol">btrfs</filesystem>
  <size>max</size>
  <subvolumes_prefix>@</subvolumes_prefix>
</partition>

To omit the subvolume prefix, set the subvolumes_prefix tag:

<partition>
  <mount>/</mount>
  <filesystem config:type="symbol">btrfs</filesystem>
  <size>max</size>
  <subvolumes_prefix><![CDATA[]]></subvolumes_prefix>
</partition>

As a consequence of the new behaviour, the old btrfs_set_default_subvolume_name tag is not needed and, therefore, it is not supported in Leap 15.0 and SLE15.

Skipping Btrfs subvolume creation

And more changes in AutoYaST that arrived just in time for SLE15 and openSUSE Leap 15.0. Recently, we have introduced a new flag in AutoYaST partition sections to skip the creation of Btrfs subvolumes because, due to a known limitation of our XML parser, it is not possible to specify an empty list.

So from now on, setting create_subvolumes to false will prevent AutoYaST from creating any Btrfs subvolumes in a given partition.

<partition>
  <mount>/</mount>
  <filesystem config:type="symbol">btrfs</filesystem>
  <size>max</size>
  <create_subvolumes config:type="boolean">false</create_subvolumes>
</partition>

Keep it rolling!

As usual, the content of this post is just a small part of everything we did during the sprint. There were also many other fixes and improvements, from auto-repairing wrong partition tables (with different sizes than the underlying disk) during installation to better interaction with other components like udisk or mdadm auto-assembling and many other things in between.

But it’s time to go back to work and start implementing all the new ideas that emerged from the YaST Team Workshop and the openSUSE Conference. See you in the next report!

Highlights of YaST Development Sprint 56

May 8th, 2018 by

LEAP/SLE 15 is getting more stable and closer to be released, but to keep this process flowing, our team of bug killers is having a lot of work to do!

This last sprint we had several fixes for really special scenarios. The kind of problems that you can find once most of things are working fine! So let’s take a look at some of these cases and how we’re working to stabilize this upcoming release.

Keep predictable network devices settings on S390x

A not well-known feature in YaST is that many specific boot parameters for installation are also used on the target system. However, this approach has one exception: the S390 mainframe. In this case, many installation specific parameters should not go to the target system and, therefore, we ignore all installation parameter for this system.
In the last sprint, we worked on a bug, which reported that at least systemd predictable names for network devices should be also used for S390 systems, otherwise the configuration done during the installation won’t be valid in a running system as network names will differ. So, from SLE15 we start to keep the settings of predictable network names for S390 systems.

Fun with console configuration of GRUB2

Another report that helped us to learn about other not well-known features was the one reporting that bootloader module does not support multiple console outputs of GRUB2. After digging into some code, we found out that YaST bootloader takes only native terminal, gfxterm or serial console in consideration, but not a mix of them.
Once we looked at the manual of GRUB2, we learned that it supports some funny outputs such as morse code, PC beeper or simple data protocol using system speaker. Of course, YaST2 bootloader does not support all these options and when it gets one of them, it is treated as an unexpected value and bootloader fails.

As we are really close to Leap/SLE 15 release, we want to avoid big changes in the system. So we decided to handle this issue by showing a popup, which informs that the configuration contains an unexpected value and asks if the whole proposed configuration should be proposed again or if YaST should quit and let the user edit it manually.

If you are curious about how it looks like:

For Leap 15.1 / SLE 15 SP1 we plan to extend the values that we support to provide a nicer experience for the user.

Bootloader configuration during upgrade

Another reported issue in bootloader was also solved this last sprint. When upgrading from Leap 42/SLE 12 to Leap/SLE 15, if the user clicks on booting on the proposal, the system crashes. The reason is that usually on openSUSE 13.2/SLE 11 it needs to repropose bootloader during the upgrade. This is no longer required for the latest upgrade and, therefore, YaST does not expect that the user will click on it. We would like to remove this option completely, but YaST still support upgrade from SLE11 to SLE15, so we still need it there. In the end, the solution is to show a popup informing the user that the modification of bootloader is not supported during upgrade.

In short, take a look at the screenshot:

Fixing kdump on Xen

We got a report about kdump breaking when it is used in Xen. To explain the problem, we need to go back to how we configure a parameter and the reasons we implemented it in this way: In current versions (before this fix) when a user wants to use kdump, we configure the crashkernel kernel parameter for all targets, for the common kernel, Xen PV0 domain and also Xen guests. This approach worked very well in the past because traditional xenlinux ignores crashkernel for PV0 and just pass it to Xen VMs. However, the current pvops implementation of Xen no longer ignores this parameter and consequently, it results in breaking the Xen virtualization. The solution for this issue was pretty simple: YaST stopped to propose using crashkernel for Xen PV0 and everything works again. This is a perfect example to show how to understand the issue, sometimes, takes much more time than to fix it!

Improving the upgrade from SLE11/12 to SLE15

We are still improving the migration from SLE11 or SLE12 to SLE15 and this sprint we were focused on the automated registration upgrade using AutoYaST. If you are not familiar with the autoupgrade feature you can find more details in the documentation.

We already supported manual registration upgrade from the old products, but the automated way was still not adapted to the new SUSE Customer Center (SCC) API and it did not work correctly. This sprint we have adapted the registration upgrade code to correctly work in the interactive mode and also in the automatic upgrade.

The code was adapted to skip the user interaction and do everything manually when running in the autoupgrade mode. The only problematic part was how to handle multiple migration targets, which in the interactive upgrade we ask the user to choose one. To have a simple solution we decided to take the first migration, later (SP1) we might allow configuring this as well. But as now there is only one migration possible anyway, this looks like a good enough solution.

Falling back to the guided proposal

During this sprint, we were informed that AutoYaST was unable to display a proper error message when no partitioning section was specified and there was not enough disk space. The bug was rather easy to solve, but we wanted to take the opportunity to highlight how AutoYaST works when the partitioning section is missing from the profile.

In the past, AutoYaST implemented its own logic, different from the one used during a normal installation. Fortunately, as part of the adaptation to the new storage layer, AutoYaST relies now on the same code than the regular installation in order to propose a partitioning layout when the partitioning section is not specified. What is more, you can override some values which are defined in the product’s control file by setting them in the general/storage section from the AutoYaST profile.

Leap /SLE 15 is closed, but Tumbleweed is still rolling

We are really close to release Leap/SLE 15 and we are more focused on minimal changes that fix only critical stuff. On the other hand, Tumbleweed users are looking always for the latest and greatest features. In order to satisfy both groups, YaST separated Leap 15.1/SLE 15 and Tumbleweed in two different git branches. In this way, we can easily start adding new features, bug fixes and other improvements for Tumbleweed while we keep SLE15 stabilized. Besides that, there is another planned Service Pack for SLE12, and once we start to work at it, we’ll also create a new separated branch to include these changes. We also adapted all related infrastructure around the branches, such as CI, docker testing images, among others.

This way, we are able to allow you to enjoy the stable Leap 15 or the latest and hottest Tumbleweed.

Conclusion

We’re now working on our last sprint before the openSUSE Conference 2018. In two weeks we’ll come back with the highlights of Sprint 57 and until there we hope that you have already everything planned to enjoy the conference that will occur in Prague this year (we hope that you enjoy the city too). We’re looking forward to seeing you there!

Highlights of YaST Development Sprint 55

April 24th, 2018 by

Time flies. We are almost in May and the openSUSE Conference ’18 is around the corner. So after booking your flights (if you need to) and your acommodation, you might want to know what happened in the YaST world during the Development Sprint 55th.

The YaST team is currently polishing the upcoming release, introducing some improvements and fixes. There are no breaking changes but still we have a lot of things to blog about.

Updating NFS Version Handling

Once upon a time, back in 2008 to be precise, the Large Hadron Collider (LHC) was finally ready, Raúl Castro replaced Fidel as President of Cuba, the TV show Phineas and Ferb was previewed… and yast2-nfs-client added support to configure NFSv4 mounts. Back then, the proper way of doing that was using “nfs4” as type for mounting the NFS share, i.e. writing “nfs4” in the vfstype column of the /etc/fstab file. Some time later, NFS4.1 (also known as pNFS) came out, and a new mount option “minorversion=1” was added. Very soon it was clear that such solution was not scaling and was not the way to go.

So at some point “nfs4” was deprecated as acceptable value for vfstype and “minorversion” was ditched in favor of “nfsvers”. Since the old deprecated way of doing things was still working, yast2-nfs-client was never updated to reflect this. But starting with the upcoming Leap 15 and SLE 15, some things will change in NFSland (in fact, the change landed in openSUSE Tumbleweed some time ago already). The type “nfs4” will be considered identical to “nfs” and “minorversion” will be completely ignored, so your old NFS mounts may not work as you expect them to do it. Time to refresh yast2-nfs-client!

During this sprint, yast2-nfs-client was not only fixed internally to produce valid entries in /etc/fstab, it also got a slightly revamped form to create and edit NFS mounts that should be less confusing than the old one and also more explanatory about how NFS versioning really works when defining a mount.

NFS version selection

To ensure our users don’t get fooled by old entries that seems to be enforcing a particular NFS version (because they use “nfs4” as mount type, for example), but are in fact not doing it due to the new behavior in SLE 15 and openSUSE 15, yast2-nfs-client is now able to detect such circumstance, mark such entries in the list and offer a safe migration path to users.

nfs4 warning

As you can infer from the screenshot above, all these improvements are available when yast2-nfs-client runs standalone, as well as when it runs embedded within the YaST Partitioner. Enjoy!

Fixing Broken Translations

Recently we got some bug reports about YaST crashing at some points when running in some specific locales. It turned out that the problem was caused by broken translations.

A lot of translated texts contain placeholders like %s, %{text} or %1. These tags are replaced by the real values by YaST. But that requires that the translated text contains the same tags. If they are missing the value will not be included and, what is even worst, if they are invalid the Ruby interpreter throws an exception which means YaST aborts making our users unhappy. And that’s really bad, right?

Unfortunately the Ruby gettext does not support format tags and the GNU gettext does not support Ruby at all. As a quick solution we wrote a script which checks whether all tags are included in the translated text and reports broken translations.

The script found about 160 broken translations. The most common problems were usually just typos (s% instead of %s, {%foo} instead of %{foo}, or extra space in %␣1). But some cases were not that trivial. Translators by mistake also used the Unicode ٪ instead of the ASCII % or even translated the tags, which must stay untouched (%{مساعدة}).

Some translations were obviously wrong or even contained the original English texts – we removed them. In some cases the tags were wrong but we were not sure whether the whole translation is valid. In that case instead of fixing the tags we removed the translated text completely. It is better to ask the translators for translating again than have a completely invalid translation.

In the future we plan to improve these checks, so the tags are properly handled by Ruby and/or GNU gettext directly and we do not need a separate script for that.

Installing Over VNC Using the Browser

You are surely accustomed to remote administration using SSH. And, as you may know, the (open)SUSE installation can be done over SSH too. But, additionally, YaST also have support for installing over VNC.

When using VNC for installation, you can choose between using a native VNC viewer or a web browser based one. The cool thing about the second option, is that you can follow the installation just pointing your browser to http://IP-ADDRESS:5801.

Until now, YaST was using a Java applet based implementation, which is no longer supported in browsers. But during this sprint, we have completed the switch to a JavaScript based solution.

Unfortunately, that has resulted in losing an encryption layer: the HTTP connection on port 5801 is unencrypted, but the typical VNC port (5901) continues to be encrypted.

Asking Once About Equivalent Licenses

After splitting SUSE Linux Enterprise in several modules, it was pretty common that the user had to accept a couple of equivalent licenses during the installation process. Given that the content for those licenses was pretty much the same, it was quite confusing. Actually, we got a bug report about the installation process being stuck asking the user to accept the license over and over (it was just the same license being shown for different modules).

In order to make our users happy, YaST is now able to decide whether two licenses are the same and, in that case, it will only ask once for acceptance. For the time being, YaST applies a hash function to license contents and compare the result, but most likely this mechanism will be refined in the future.

License Confirmation in CaaSP 3.0

And talking about licenses, another small change about how they are handled was introduced in CaaSP 3.0. As you know, CaaSP features a One Dialog Installer and there was no room for the license to be shown. Now, before proceeding with the installation, YaST will show the license in the confirmation screen if needed.

CaaSP 3.0 License Confirmation Popup

Improving the addon Boot Option Handling

Back in February, we improved the addon boot option to handle the SUSE Linux Packages DVD properly. However, during testing, we found out that if you are using a system which only has one DVD drive, the installation DVD will be automatically used as an addon.

In order to fix this conflict, if the installation media and the Packages DVD are going to use the same drive, YaST will ask the user to change the DVD before using it as an addon.

Additionally, we improved the documentation of the addon boot option adding new examples to clarify how the dvd:/// URLs are handled.

Echoes of Winter: White Text on a White Background

These days we fixed a bug that only allowed clairvoyant users to finish the installation of openSUSE Kubic.

The bug is pretty unremarkable but may we draw your attention to the related CSS styling engine? It powers the high-contrast color mode that you can select with F3 or with Y2STYLE:

Linuxrc Color Mode Selection

Installer in High Contrast Mode

and if you press Ctrl-Alt-Shift-S (for style) you can change the styling on the fly, as in this example of changing the background color:

Installer Stylesheet Editor

Conclusions

openSUSE Leap 15.0 release is approaching and, as usual, we need help from our dear users to give testing versions a try and report bugs. Thanks in advance!

Highlights of YaST Development Sprint 54

April 11th, 2018 by

We were in the middle of rewriting no, refactoring recompiling all of YaST into Visual Basic when we found that it was April 2nd already and had to scratch the entire project. Next year for sure. So you are left with a report of enterprise grade stabilization and we hope that your servers will be very bored running our software.

Installation and Upgrade

Clearer Description of Migration Targets

Life goes through various roads and it is same for SLE life. SLE15 is now split into multiple modules and during the upgrade it can be quite complex to pick the desired upgrade target. We have to react to this issue as customers start complaining that the upgrade overview starts to be hard to understand and we should improve it. So we did it and now you can check the changes on the attached screenshots. We modified the overview label from listing all products to just a summary with the details displayed below as it was before. Be aware that in the future and for some products or extensions/modules more migration targets will be possible.

Old screenshot:

and the new one (for a slightly different system, so it is not an exact match for the previous screenshot):

Importing the SMT Server SSL Certificate at Upgrade

We are still improving and fixing bugs in the migration from the SLE11 or SLE12 products to the new SLE15 line. One issue we fixed this sprint was importing the SSL server certificate from the old system at upgrade.

For registration you can use a local SMT server (Subscription Management Tool) instead of the usual SCC server (SUSE Customer Center).

The SMT servers usually use a self-signed SSL certificate to save some money for buying a real certificate signed by a well-known certificate authority. This self-signed certificate is imported to the system by YaST during the initial registration so the registration process and the repositories from the server can be properly accessed.

But during the offline upgrade to SLE15 the old system is not running, the installer runs from the installation medium. In that case we need to import the SSL certificate from the old system to the installer so it can properly access the registration server and do the upgrade.

The certificate import is quite easy, we just need to be careful as SLE11 uses a different (old) path for storing the imported certificates than in SLE12 or SLE15.

As the result you should be now able not only to upgrade the systems registered against the SCC server but also the systems registered against your local SMT server.

Many System Roles

Various products that we’re able to install have grown so many groups of presets, called System Roles, that they no longer fit on the screen. We applied some dark gray magic to make them fit in a scrollable box, at the expense of losing the keyboard shortcuts, sorry.

Storage

Better Message for Multipath (and other) Problems

While scanning the storage hardware, or at a later stage while manipulating it, there is always a chance of finding problems in the system that make it very hard to continue with the installation or the execution of YaST. In that case, previous versions of storage-ng used to show you a pop-up message with some technical details about what went wrong (for example, the command that failed and its output) and with options to abort YaST or continue despite the error.

But we found that for some situations we could do better in trying to understand what went wrong and explain it to you, instead of directly showing those raw technical details. One clear example is finding the same LVM physical volume twice, something that should never happen. Apart from double vision problems (libstorage-ng doesn’t drink alcohol), the most likely cause is that a multipath system is not being correctly detected and thus every one of the connections to the disk is being detected as a different disk, duplicating the content in the eyes of YaST.

Now such a circumstance is detected and explained to you, advising to use LIBSTORAGE_MULTIPATH_AUTOSTART (see linuxrc documentation) or the corresponding entry in the AutoYaST profile if it has not been used. By the way, during this sprint we also instructed storage-ng about LIBSTORAGE_MULTIPATH_AUTOSTART, since it used to ignore that ancient libstorage modifier.

The technical details are still available under the "Details" button, as you can see below. They are simply not displayed at first sight, which should make the whole experience less daunting for less-experienced users. That change applies to all the severe errors found during the three critical phases of storage-ng: hardware activation, system probing, and commit (when the partitions and other devices are created).

Of course, the new pop-up messages have full support for AutoYaST. The most appropriate default option (continue or abort) is automatically selected depending on which one of the mentioned phases is being executed and, if AutoYaST is configured to display pop-ups, the usual countdown is displayed before doing such selection. See below the new generic error (for a different, unidentified problem) in action in AutoYaST.

AutoYaST is now Able to Reuse Encrypted Devices

As you may know, AutoYaST is quite flexible when it comes to partitioning, so we are still writing the final bits of the adaptation with the new storage layer. And this time, we were working on teaching AutoYaST how to reuse encrypted devices properly.

However, the implementation was not that straightforward, as the hardware probing occurs even before the partitioning section of the profile has been analyzed. And, in some scenarios, it is not clear which key should be used to unlock a device (for instance, this can happen when more than one encryption key is defined). To solve this problem, AutoYaST will try all defined keys on all encrypted devices until a working key is found.

Of course, this behavior is properly documented now in the AutoYaST handbook.

Miscellaneous

Fixed AutoYaST profiles validation issues.

In our previous blog entry we already mentioned that there are significant changes between SLE12 and SLE15 profiles which have been documented in this appendix.

It is very common to adapt the profiles by hand which is error-prone and sometimes it is also hard to identify where the errors are just running an installation and looking deeply into the logs. That is why profiles validation using xmllint or jing is recommended (more info here).

During this sprint we have fixed some errors with the cloned profiles after installation which were not validating.

Translation Issues

We are receiving quite a lot of bugs regarding the translations. The usual problem is that some text is not translated at all and the original English text is displayed. This sprint we fixed several issues in this area, two of them are worth sharing in the blog.

The XSL File Format

The first problem was reported for missing translations in the role descriptions in the SLES4SAP product. The SLES4SAP installation basically behaves like the standard SLES installation just with changed few defaults. To avoid the duplication and make the SLES4SAP maintenance easier we simply take the original SLES XML control file, which describes the installer behavior and the defaults, and change just few values using a XSL transformation into the resulting SLES4SAP installer control file.

It turned out that the roles with missing translations were located in that XSL file. And unfortunately YaST did not support extracting the translatable strings from XSL files. However, we support translations in XML files and because a XSL file is actually a valid XML file we could easily extend the translation support in YaST to also cover the XSL files. So now the SLES4SAP roles are correctly translated.

Missing textdomain Call

We fixed several bugs with missing translations which were caused by missing textdomain call in the code. This call defines which POT file should be loaded and searched for the translations. If the YaST code does not use this call then obviously no text can be translated as YaST does not know which POT file should be used and it silently used the original untranslated text.

That means it was quite difficult to find why some text was not translated. And because that was quite common bug we had a nice idea to improve the situation by logging a warning into the YaST log with the exact message which could not be translated. And more importantly the log now also contains the location of the code which was trying to use the translation wrongly. See the pull request for more details.

With the openQA team we discussed also the possibility to add a new check to openQA which would scan the YaST log for this particular warning and report a problem. Which means we should not overlook this quite important warning in the future.

Highlights of YaST Development Sprint 53

March 23rd, 2018 by

As the release dates for SUSE Linux Enterprise 15 and openSUSE Leap 15 approach, we keep adapting YaST to make easier for our users to take advantage of all the new features that these rock-solid operating systems will bring.

During the last two weeks that has implied, apart from regular bug fixing that we usually don’t cover here, working on AutoYaST, improving Storage-ng and polishing several aspects related to modules and extensions, like their registration and licenses.

Let’s start with the rewritten Partitioner that is part of yast2-storage-ng.

Partitioner: more flexibility with the partition id

Setting the right partition id (also known as partition type) for each partition is an important part of the system setup that is often overlooked. Our Partitioner has always displayed in a prominent place the widget allowing to set that id, suggesting always the best value based on the selected role and the chosen file system type. But in many cases, that was more than a simple suggestion. In the old Partitioner (and in the new one until this sprint) the value of the partition id field (Linux, swap, Linux LVM, etc.) could only be manually edited in case the user had selected to not format the partition. When the option “Format device” was selected, the automatically chosen value could not be changed.

In SLE15 and openSUSE Leap 15 (and quite soon in openSUSE Tumbleweed), it will be possible to modify the id, no matter if the partition is going to be formatted or not. Of course, the logic to propose the best option every time the user selects a file system type is still there, but now it can be always overridden if the user wish. That change resulted in a small rearrangement of the widgets in that screen, as you can see below (remember we are trying to be very conservative with the UI changes in the Partitioner).

UI adjustments for the partition id

Partitioner: better support for DASD

In our previous report we explained some of the aspects in which the Direct-access storage devices (DASD) used in s390 mainframes are different from regular hard disks. But as you can imagine, there are more differences… and we know our readers love to learn new stuff while enjoying our reports. 😉

In short, there are two possible kinds of DASDs devices: Extended Count Key Data (ECKD) and Fixed Block Architecture (FBA). As explained in the previous report, the ECKD devices need to be formatted at low-level in order to be used by the operating system and, moreover, there are two possible low-formats for them: Compatible Disk Layout (CDL) and Linux Disk Layout (LDL).

And now the fun – ECKD devices formatted as LDL do NOT have a partition table. FBA devices can potentially have one, but it’s also often skipped. To manage those DASDs without partition table, the Linux kernel simulates an implicit single partition taking the whole disk. Of course, working with such implicit partitions implies some restrictions, and we have introduced several controls to make sure things stay under control in the storage-ng Partitioner. For example, an error message is now shown if the user tries to remove an implicit partition.

Trying to delete an implicit partition

For curious readers, there is more information about DASD available in this link.

Partitioner: can’t resize a partition… but why?

In SLE15 and openSUSE Leap 15 we will report very detailed reasons why a partition or a file system cannot be resized, as you can see in this screenshot.

Detailed description of resizing restrictions

This used to be just a very simplistic message “Device cannot be resized”. But there may be many reasons for that, and sometimes different restrictions might contradict each other: While some type of file system only lets you grow, not shrink (e.g. XFS), the partition that the file system is on might not be able to grow, for example because there is another partition right next to it. We want to minimize user frustration that might happen when we only report the first reason, and when the user somehow managed to fix that problem, show another one that can’t be fixed.

As usual, this feature will be available in Tumbleweed in a matter of days.

Handling registration rollback in SLE15 Migration

Of course, the Partitioner was not the only YaST area to get attention during this sprint. Several aspects related to products, modules and extensions were also worked, with all the implications they have about registration, migration and licenses.

For the offline migration to SLE15 we reused some parts from the online migration which handles service pack upgrade. But it tuned out that the reused part was not correctly integrated into the installer and in some corner cases (registration errors) it did not behave correctly.

Moreover if the upgrade failed early then the system still contained a SLE12 installation but was registered as a SLE15 system on the SCC server. After booting the original SLE12 system the access to the online repositories was broken.

This sprint we fixed that so in case of registration error or when going back the original registration is restored. Now you can go back and choose a different system to upgrade and it will work as expected.

Additionally we fixed some small issues with custom repositories (add-on or driver updates) used at upgrade.

More fun with hiding/showing beta versions in SLE15

Usual readers of our blog already know that SUSE is taking extensions and modules to a whole new level in SLE15, making them a cornerstone of the system installation and upgrade process. As already explained in previous posts, that implies more complex dependencies between extensions and modules. All those mechanisms usually work nice… except a small problem we found out with beta versions.

If a given extension was in beta phase and some of its dependencies were also in beta, if the “Hide Beta Versions” checkbox was unchecked the system was displaying only the extension selected by the user, but not the auto-selected dependent beta extensions. Our SLE testers found that quite confusing. So to make everyone’s life easier, we fixed the behavior as shown in the following screenshot.

Displaying selected and auto-selected beta extensions

A look into the future: analyzing how we display licenses

Currently there are many different ways to handle and display licenses. That can happen during the installation or upgrade process, while adding additional products to an installed system and, last but not least, while using YaST2 Firstboot to perform additional installation steps on the first system execution.

Additional there are 3 different locations from which these licenses come from. They can be provided by the SUSE Customer Center, be provided by libzypp or come from a repository using a legacy approach.

To simplify and unify all that in a close future, the first step was researching all those possibilities and how they are handled in (Auto)YaST. The result of such research can be found in this document hosted on Github.

AutoYaST product selection and installer update improvements

As you probably already know, starting with SLE-15, all products are distributed using one medium and you need to choose explicitly which product to install. Of course, if the medium only contains one product that would not be needed.

In AutoYaST profile the product is selected using the /software/products/product XML node:

<software>
  <products config:type="list">
    <product>SLED</product>
  </products>
</software>

Due to a bug, the cloned system exported the product short_name instead of the name, resulting in an internal error reported by the installer update and a later error during the auto-installation which aborted it because no product was selected.

So, during this sprint we have made improvements for both scenarios.

  1. The installer update will not rely in the product selection at all (the installer is the same for all the products) but will use the self_update_id from the control file and the version and architecture from the first product available on the media. The installer update documentation has been also updated according the last changes and it is probably the best place for knowing more about its behavior.
  2. The wrong product selection error reported was not very useful and it was decided to provide more information about the list of available products from the media. Just see the image below with the latest implementation:

Warning about wrong product in AutoYaST

Document main differences in AutoYaST profiles between SLE12 and SLE15

The need to select a product is not the only relevant change affecting AutoYaST profiles for SLE15. There are many other significant changes in SLE15 compared to SLE12. Like the new modules concept, replacing SuSEfirewall2 with firewalld, replacing ntp with Chrony… Users wanting to reuse existing SLE12 profiles with SLE-15, will probably need to adjust them.

We have created this summary describing some of the most important changes in order to help with the conversion.

That document is just a preliminary and temporary work that is currently being reviewed and improved by the awesome documentation team at SUSE. Very soon (probably already done at the time you are reading this) the content will be merged and a new section titled “Main differences between SLES 12 and 15 profiles” will be available in the current guide for AutoYaST. Have we ever mentioned how much the doc team rocks? So please, use that last link as final reference instead of our temporary summary.

Cron config for NTP client

It is possible to setup the YaST-ntp-client module to sync the system clock at regular intervals. If that feature is used, YaST writes the needed configuration to a cron.d config file. We were still using “novell” as part of the name of such file, which was reported as a bug. It turned to be a good opportunity to take a look to a module that, as you can guess from that bug, we don’t update very often. 😉

First of all, we made sure that newly written files will have a more up-to-date name. Straightforward and easy.

The second part was to provide an upgrade path if the file already existed. We integrated that with the existing ntp to chrony conversion. That means the existing configuration is updated when a new version of the yast2-ntp-client package is installed, so the user does not need to run the module again to start using chrony with an existing configuration.

Last but not least, the third part was to adapt the package to be a better citizen in the RPM world, marking that file as ghost file in RPM spec. Now this command can recognize that yast2-ntp-client is responsible for that configuration file.

  rpm -qf <file>

Two months… and counting

Only two months of countdown until the release date of openSUSE Leap 15! That means a lot of hard work ahead of us, so stay tuned for more updates.