Home Home > Systems-management > Software-management
Sign up | Login

Archive for the ‘Software Management’ Category

Highlights of the Latest YaST Development Sprints

June 25th, 2019 by

May and June have been, so far, interesting months for the YaST Team. We worked hard to polish the last details of the recently released openSUSE Leap 15.1, we attended the openSUSE Conference 2019 (with many fruitful conversations), we shared quite some time together around a table without computers (most of the time, we are a geographically distributed team), many team members enjoyed vacations (it’s spring time in Europe), we organized a Leap 15.1 launch party with technical talks in Gran Canaria… and we ran out of energy to also publish our traditional sprint reports in this blog.

We will try to fix that with this blog post in which we will try to summarize some highlights from the latest three development sprints, namely the 77th, 78th and 79th. So be warned, this is going to be a loooong post.

Support for Multi-device Btrfs File Systems

We have been working steadily during the three sprints in implementing all the necessary bits to offer a good experience installing and upgrading an (open)SUSE system on top of several block devices by means of Btrfs RAID capabilities. That includes support in the Partitioner, in AutoYaST, in the storage guided setup and more.

We decided that all that deserved a separate blog post. You can find here: Getting Further with Btrfs in YaST.

More Improvements for the Partitioner

That blog post mentions a couple of changes in the Partitioner that, although initially motivated by the introduction of multi-device Btrfs, go beyond that scope and are aimed to make the all the lists of devices more useful and informative.

Traditionally the Partitioner used two separate columns “Type” and “FS Type” to describe the function of every device. That was sometimes hard to understand. Moreover, quite often the important information (like the relationship between a partition and its RAID or LVM) was simply missing in those tables.

Traditional devices table in the Partitioner

We have merged those columns into a more informative one that identifies the devices and also gives an overview of the relationship between them at first glance. In addition, the table displaying all the system devices now includes multi-device file systems.

Revamped table of devices

Mitigating CPU vulnerabilities from YaST

If you are interested in security (or simply if you have not been living under a rock) you probably have heard about CPU based attacks like Spectre or Meltdown. The last year has seen a number of these CPU issues, all of them coming along with their own kernel options to change the Linux behavior in order to mitigate the security risks at a price of some performance loss.

However, not all users know what affects their architectures or particular models of CPU and which kernel parameters to use to gain more performance if the security risk is acceptable for them.

For that purpose, a new meta-option called “mitigations” was added to the Linux kernel. It allows to enable and disable at once several of those mitigations that prevent CPU attacks. See more information at this document published by SUSE.

We find that kernel option very useful, so we decided to provide an easy way for users to adjust it. Now the YaST bootloader screen contains a new setting which offers three pre-defined options and even a fourth one to let the users fine-tune the settings on their own. As you can see in the screenshot below, we have included extensive documentation in the help dialog, so you will not need to search for this blog post in the future.

It is also possible to modify this option directly from the installation summary. For that purpose, the “Firewall” section was renamed to “Security” and it now includes the possibility to tweak the CPU mitigation options, alongside the traditional settings for firewall and opening the SSH port.

CPU mitigations in the installation summary

Another success story of (open)SUSE offering a promptly solution for our users to easily adapt their systems to ever-changing complex needs.

Memory Optimizations during Installation

While the release of openSUSE Leap 15.1 was approaching, we got several bug reports stating the YaST installer used to freeze when using only 1GB RAM with online repositories (see bug#1136051).

It turned out that at some point YaST loads the details of all available packages. And that needs a lot of memory if you enable the online repositories during installation. For example the OSS Leap repository contains more than 35.000 binary packages!

The problem was in the YaST internal API accessing the package manager library (libzypp). It did not allow to filter the objects, YaST had to read all objects and then do the filtering in the code. And for each object it returned all attributes, even those which were not needed (like the package description, full RPM name, etc…). All that data required a lot of memory.

To fix that we have introduced new API calls that allow specifying more filters (like return all selected packages, packages from specific repository,…) and you can set which attributes should be returned. If you need to know only the name and the version then you will not get the other useless attributes. And to ease the usage of the new API in YaST we provided a nice object oriented wrapper written in Ruby.

This optimization saves a lot of memory, 1GB of RAM should be enough for future installations with the online repositories, even if they grow even more.

Unfortunately, we were only able to diagnose the problem and provide a solution a couple of weeks before the official release of Leap 15.1. Introducing a change in such a sensitive part of the installer was considered too risky (it would have invalidated many of the tests that had been already performed) so the installer included in openSUSE Leap 15.1 is still memory hungry if online repositories are used. For that release, we simply increased the official memory requirements to 1.5 GiB.

Online Migration from openSUSE Leap 15.1 to SLES15-SP1

For openSUSE Leap 15.0 it was only possible to migrate from Leap to SLES (SUSE Linux Enterprise Server) manually (see the documentation). With Leap 15.1 the goal was to also support a migration using YaST. But we got a bug report saying the online migration from openSUSE Leap 15.1 to SLES15-SP1 displayed a wrong migration summary and didn’t work well.

It turned out that YaST needed some small fixes to support this properly. The main problem was that YaST did not expect that the base product or the package vendor can be changed during online migration, so far it was only possible to upgrade to the next SLE service pack level. But that is fixed now.

Wanna try the migration from openSUSE Leap 15.1 to SLES15-SP1? Then follow these steps.

  1. Install the yast2-registration and yast2-migration packages in the Leap 15.1 installation
  2. Make sure the latest online updates are installed (to install the fixes mentioned above)
  3. Start the YaST registration module and register the openSUSE Leap 15.1 product using your registration key
  4. Then start the YaST migration module, select the migration to SLES15-SP1
  5. (There might be reported some package dependency issues in the migration summary, go to the package manager and resolve them. Usually removing the old openSUSE package is the right solution.)
  6. Start the migration, the SLES packages will be downloaded and installed
  7. At the end the system will be rebooted to start the freshly installed SLES, enjoy! 🙂
  8. (It is recommended to review the orphaned packages, leftovers from the Leap installation, with the command zypper packages --orphaned and possibly remove them.)

From Leap to SLES via YaST

Please note that only minimal server installations of Leap are supported for upgrade, full installations especially with third party packages might not work correctly.

Why Cannot I Read the Logs?

Long time ago, the logs of any Linux system were spread over several files living under the /var/log subdirectory. YaST offers its “System Log” module to inspect those files in a convenient way. Since the introduction of Systemd and its journal, that information has been gradually moved to this new mechanism by default. And YaST offers its “Systemd Journal” module to inspect and query that journal.

Both YaST modules can be executed by any user in the system, not only by root. That’s intentional because both the Systemd journal and the traditional Linux log files can register information targeted to unprivileged users. But there was some room for improvement in the error message displayed by both modules when such users were trying to access to protected information.

This is how the new more explanatory message looks in the “System Log” module.

Explanatory pop-up for log viewer

And this is the extended message for “Systemd Journal”, that now mentions the systemd-journal user’s group.

Improved message in the journal viewer

Another Day at the YaST Office: Adapting to Changes

As we usually remark in our blog posts, a significant part of the work of the YaST Team consist in keeping YaST in sync with the constant changes in the underlying system. Of course, these sprints were not an exception in that regard. Without trying to do an exhaustive list, let’s take a look to some of those adaptations since the mentioned underlying changes may be interesting for some readers.

Turns out Systemd developers has decided to change the list of possible states for Systemd services. The systemd-sigabrt state is obsolete and a new systemd-watchdog one was added. In the YaST team we learned some time ago that the list of Systemd states changes more often than what most people would expect. As a consequence we have an automated check to detect these situations. The bell ringed, we adapted the code and everything keeps working.

Systemd is not the only technology that keeps evolving. Quite some time ago, RMT replaced SMT as the default proxy technologies for the SUSE Customer Center. Although both has coexisted for quite some time, from SLE-15 onward only RMT is offered. Thus we have adapted all the references to SMT that still existed in YaST. From now on, only RMT is mentioned to avoid confusion.

Another common adaptation we have to perform in YaST is adjusting some module when the output of the command it runs under the hood has changed. Recently we found out the developers of the command iscsiadm has decided to use more than one exit code to indicate a successful execution (traditionally, only zero should mean that). After a long discussion, we decided to adapt YaST iSCSI to also be happy with the error code 21. What means for you? That future versions of YaST iSCSI should work faster in some situations, since the confusion will not longer result in a timeout.

YaST Firstboot: a Better Example File

Those were just some example of the many adaptations we did lately for changes in the system. But not all adjustments are motivated by external changes. We also realized the example configuration file provided by YaST Firstboot (at /etc/YaST2/firstboot.xml) needed some love. Due to the nature of YaST Firstboot, that file should be customized before using YaST Firstboot. But providing an example file with three different steps about acceptance of licenses (two of them enabled and the third one disabled) and other inconsistencies was definitely not helping anyone to understand how to use the module.

In fact, the status of that default example configuration and of the documentation managed to confuse even SUSE’s quality assurance team. So we improved the example file shipped in the package to make it more realistic and we also updated the YaST Firstboot documentation to clarify how to use that file. Because not all improvements are always done by coding.

Leap 15.1’s Most Annoying Bug: Create Home as Btrfs Subvolume

As all openSUSE users should know, for every Leap release a page is created in the openSUSE wiki listing the so-called Most Annoying Bugs. Leap 15.1 was a very smooth release and this time the corresponding list contains only one bug… and it’s a YaST one. 🙁

In an installed system, when YaST was creating a new user it was always trying to create its home directory as a Btrfs subvolume. Even in cases in which that was impossible. For example, if the directory to be created was not in a Btrfs file system.

Writing user error

Fortunately, the bug didn’t affect the installation process or AutoYaST. We created a fix that was quickly available as a maintenance update. So make sure your openSUSE system is updated before trying to create new users with YaST.

YaST Network refactoring: status report

Since we submitted the first bits of the yast2-network refactoring by the end of April, we have made quite some progress in this area. Although it is still an ongoing effort, we would like you to give you an update on the current state.

We might say that we have been working on two different areas the user interface implementation and the future data model.

About the user interface, the team has improved the code quite a lot to make it easier to maintain and extend. We have introduced some classes to de-couple the widgets from the data, and it pays off. Additionally, we have fixed some bugs (many of them related to validations), simplified the process of adding new devices and reorganized the hardware tab.

New hardware tab in YaST Network

Regarding the internal data model, we have been thinking about the best way to represent network configuration in an agnostic way so, in the future, we can not only support Wicked but other options too (for the time being, the NetworkManager support is quite limited). If you are curious about the details, we have added a document describing the approach to the repository.

The new data model is already being used to handle DNS and Routing configuration. So if you are using Tumbleweed you have been already using the new network code for some weeks, including the UI enhancements presented in our latest post.

New network routing dialog

Although the data model is so far only used in the mentioned parts, the plan is to submit to Tumbleweed a heavily refactored UI layer during next sprint. So stay tuned.

Added Appstream Metadata to the YaST Packages

The YaST package manager is not the only software manager in the (open)SUSE distributions. There are some more, like Discover in KDE or the GNOME software manager, not to mention the online openSUSE appstore.

While the YaST manager is package oriented, those other software managers are application oriented. That makes a huge difference, especially for the beginner users.

The full list of packages does not only include the applications (basically anything a user can start from the desktop), but also shared libraries, pieces that provide functionality for other applications or basic components needed for the system to work. With so many software packages (the openSUSE OSS repository contains over 35.000 of them!) it’s sometimes hard to find the software you need unless you know what you are looking for.

To offer an application-oriented view on top of all that, the application managers need some special data describing the applications inside the packages. The data are located in the /usr/share/metainfo/*.xml files, if you are interested in the technical details check the AppStream documentation provided by the Freedesktop.org.

Our absolutely awesome community contributor Stasiek Michalski (more famous by his nickname lcp) realized YaST was not offered in those application managers and decided to fix it. So he created an XML generator which collects the data from YaST packages and automagically generates the metainfo XML file needed by the other software managers.

YaST in GNOME Software

As a result, the Gnome Software Manager, Discover and other software managers will offer YaST in Tumbleweed just as any user-oriented application. Thanks lcp!

YaST in Discover

AutoYaST Pre-install Scripts & Storage

AutoYaST has a special feature to allow users to customize the installation process and take control in different stages of the installation. For that, the AutoYaST profile offers a section where you can insert your custom scripts. There are four types of scripts: pre-scripts, postpartitioning-scripts, chroot-scripts, post-scripts and init-scripts.

For the particular case of pre-scripts, the documentation states that “It is also possible to change the partitioning in your pre-script“. That means, for example, you could use a script to create a new partition or to configure some kind of technology. Therefore, it would be very convenient to re-analyze the storage devices after running the user pre-scripts. In fact, that was the default behavior in the old storage stack, but the new one was slightly modified to only re-analyze the system under certain conditions.

But turns out some SLE customers were using pre-scripts to configure the behavior of multipath and those changes were not being noticed by AutoYaST.

The solution was quite trivial. We simply decided to always perform a new storage re-analysis after the AutoYaST pre-scripts. We did not find strong reasons to not do it and there should not be significant performance penalty.

And, for the specific case of multipath, YaST now copies some configuration files (e.g., /etc/multipath.conf and /etc/multipath/bindings) to the target system when performing a new installation. Otherwise, the installed system would not contain the configuration applied during the installation.

Clarifying the Usage of Software Management Options

Our software manager is one of the most complex YaST modules, which makes some aspects of its usage not fully obvious for some users. You may have even notice that is the only YaST module in which the interface is clearly different when executed in text and graphical mode. Specially the menu at the top of the interface, which is organized in different sections.

Some users where confused by the fact of some options not being persisted between different executions of the module. Those options are there to modify the current operation of the module, not to change the configuration of package management in the system.

When executed in text mode that was clear because such options where labeled as “temporary change” but the graphical mode didn’t have any indication about it. As you can see in the following screenshot, that’s fixed now.

Temporary options in YaST Software

Product License Hard to Understand? Try Another Language!

Some user reported that in the text-mode installation it was impossible to switch the language of a product license. Although during graphical installation everything works flawlessly, in text mode the language switching widget was there… but disabled.

The point is that such behavior was not exactly a bug. It was in fact done on purpose. We decided to prevent such language change some time ago because on the Linux console we’re not able to display the characters of many languages. Hopefully some of our usual reader have just shout “that’s not true!” 😉 Those users remember that in the report of our 67th sprint we explained that now we always use fbiterm when installing in text mode in order to be able to display characters of almost every language.

We are now able to display all languages that currently have license translations, so we have enabled the language switching widget and now both graphical installation and text-based one deliver an equivalent user experience.

More about languages

This is not the only change related to internationalization we did in these sprints. We also added a specific warning message for situations in which YaST is used to change the language of the system but there is no repository containing the needed translation packages. Something that obviously only affects users configuring system in very restricted environments.

As you can see in this and other recent reports, we have to deal relatively often with difficulties related to translation and internationalization. To reduce the effect of those problems on our final users, we also added some extra mechanisms to detect internationalization errors introduced during development. Hopefully that will mean that in future reports the space dedicated to comment language-related problems gets shorter and shorter. 🙂

And that was just a summary!

As long as this post looks, there are many interesting things we have done in these weeks we left out, intentionally or not. We definitely should avoid skipping three reports in a row in the future!

This week it’s Hack Week at SUSE, which means regular YaST development will be put on hold… or will turn into something completely different. You never know what the result of a Hack Week can be!

But in any case we will go back to our sprint-based pace in August. So expect a new blog post in three weeks. See you then!

zypper-upgraderepo 1.2 is out

February 23rd, 2019 by

Fixes and updates applied with this second minor version improved and extended the main functions, let’s see what’s new.

If you are new to the zypper-upgraderepo plugin, give a look to the previous article to better understand the mission and the basic usage.

Repository check

The first important change is inherent the way to check a valid repository:

  • the HTTP request sent is HEAD instead of GET in order to have a more lightweight answer from the server, being the HTML page not included in the packet;
  • the request point directly to the repodata.xml file instead of the folder, that because some server security setting could hide the directory listing and send back a 404 error although the repository works properly.

Check just a few repos

Most of the times we want to check the whole repository’s list at once, but sometimes we want to check few of them to see whether or not they are finally available or ready to be upgraded without looping through the whole list again and again. That’s where the –only-repo switch followed by a list of comma-separated numbers comes in help.

--only-repo switch

–only-repo switch

All repo by default

The disabled repositories now are shown by default and a new column highlights which of them are enabled or not, keeping their number in sync with the zypper lr output. To see only the enabled ones just use the switch –only-enabled.

Table view

Table view

Report view

Beside the table view, the switch –report introduce a new pleasant view using just two columns and spanning the right cell to more rows in order to improve the number of info and the reading quality.

Report view

Report view

Other changes

The procedure which tries to discover an alternative URL now dives back and forth the directory list in order to explore the whole tree of folders wherever the access is allowed by the server itself. The side effect is a general improvement also in repo downgrade.

The output in upgrade mode is now verbose and shows a table similar to the checking one, giving details about the changed URLs in the details column.

The server timeout error is now handled through the switch –timeout which allows tweaking the time before to consider an error any late answer from the server itself.

Final thoughts and what’s next

This plugin is practically completed, achieving all the goals needed for its main purpose no other relevant changes are scheduled, so I started thinking of other projects to work in my spare time.

Among them, there is one I am interested in: bring the power of openSUSE software search page to the command line.

However, there are some problems:

  • This website doesn’t implement any web API so will be a lot of scraping job;
  • There are missing packages while switching from the global research (selecting All distribution) to the specific distribution;
  • Packages from Packman are not included.

I have already got some ideas to solve them and did lay down several lines of code, so let’s see what happens!

Highlights of YaST Development Sprint 67

November 20th, 2018 by

Time flies and another development sprint has finished for the YaST team. During it we have focused on improving the installation process, refining it and adding new features to unleash new possibilities. That includes:

  • More options to configure Kubic during installation.
  • Several improvements in the storage proposal.
  • Easing the network configuration and the usage of online repositories in openSUSE.
  • Improving installation in text mode for CJK and other languages.
  • Configuring SSH access during installation and in a running system.
  • And many other small fixes here and there!
  • Many of these features will be already available in openSUSE Tumbleweed during November (the sprint just finished November 16th and the integration process usually takes some days), others will be visible for the first time in the next SLE-15-SP1 and Leap 15.1 Alpha versions.

    New Dialogs in the openSUSE Kubic Installation

    The SUSE CaaSP and the openSUSE Kubic products got a new installation workflow some time ago. Originally they used a specific installation workflow (a single all-in-one configuration dialog), but the problem was that many openSUSE/SLE installation features were missing because the installer code was completely different.

    That has been changed some time ago to use the usual installation as in standard SLE or openSUSE products. You can read the details about the new installation workflow at the Kubic wiki page.

    However, the CaaSP or Kubic products required some more specific settings depending on the selected role. During this sprint we have added the corresponding steps to the installation.

    New installation steps for openSUSE Kubic

    Currently the extra dialog only asks for the NTP server address, but some more options can be added later.

    Improve Textmode Support for CJK and Other Languages

    YaST is able to cope with many languages even when running in textmode. When a user wants to run the installer in textmode using some specific languages -like Chinese, Japanese or Korean-, YaST will use a special terminal emulator called fbiterm which is able to display the characters which are needed in those languages.

    Now, instead of keeping two different approaches depending on the language, YaST will try to use this special terminal emulator whenever is possible for all text based installations. Unfortunately, there is a small set of languages that are not properly handled by fbiterm. In those cases, YaST will inform the user about the problem and it will fall back to English.

    YaST installing SLE-15-SP1 in Japanese

    As a side effect of this unification, the font used during the textmode installation has changed for all the languages that didn’t used fbiterm in the past. So your SUSE or openSUSE installation may now look slightly different.

    Storage proposal: playing nicely with Intel Rapid Start Technology

    Intel Rapid Start Technology (shorted as IRST) enables systems to quickly resume from deep sleep (e.g. if your battery runs out). It’s a firmware-driven technology that relies on the existence of a special partition located on a SSD (solid state device).

    But our partitioning proposal (the so-called Guided Setup) was not aware of the important role of such partition, so it sometimes proposed to delete it in order to use the freed space for installing (open)SUSE on top.

    We have improved that and now the IRST partition will only be deleted as a last resort, if deleting all other allowed partitions in the disk is still not enough to make the new system fit. Moreover, an IRST partition located in the same disk than a Windows system will only be deleted if the user explicitly allows the storage proposal to delete that Windows installation.

    Running the storage proposal on top of software RAIDs and USB devices

    But that’s not the only improvement done in the area of the partitioning guided setup. We have also broadened its usefulness by allowing it to be used on top of new kinds of devices.

    First of all, it’s now possible to execute the guided setup on top of a software MD RAID. For that, the candidate RAIDs has to fulfill one of two conditions: to contain a partition table or to be completely empty (which includes not being formatted, encrypted or used for any other purpose). That is not only another step to take advantage of the relatively recent capability of YaST to partition software RAIDs (thanks to libstorage-ng) but also a very natural way to support the usage of Dell EMC PowerEdge RAID controllers S130 and S140, which offer an hybrid solution based on software RAIDs but backed up by a firmware-driven configuration.

    Partitioning proposal on top of a software MD RAID

    Almost for the same price, we decided that it was time to offer the available USB devices as candidates for the guided setup, even when there are also non-USB candidates. Excluding, of course, the installation media (except when performing a network-based installation, in that case the USB device can be reused to install the system).

    As usual, these features will be soon available in openSUSE Tumbleweed, as well as in the next stable versions of openSUSE Leap (15.1) and SUSE Linux Enterprise (SLE-15-SP1).

    Fine-tuning the initial storage proposal

    All the described changes open new possibilities for the users to decide where to execute the storage proposal. But as our users already know, when the installer reaches the storage proposal step for first time (before the user has had the chance to run the Guided Setup or the Expert Partitioner), an initial proposal is automatically calculated and displayed. This proposal is performed without user intervention, and all necessary information is taken from the control file of the current product. For example, whether to create a separate home, or whether to enable snapshots for root, etc.

    During the previous sprint, this initial proposal was already improved to make an attempt over each individual disk instead of considering all disks together. Now we have improved this behavior to not discard the first disk so early, even if that means getting a smaller (open)SUSE system.

    For example, let’s image we have a system with a first small SSD disk of 40 GiB and a second big Hard Disk of 1 TiB, which is a relatively common configuration for laptops. Since the first disk is too small to allocate a system with the optimal partition sizes and a separate /home partition, the installer used to switch to the second disk, in which it could propose a root partition of the recommended size to use Btrfs snapshots (50 GiB) and a partition for /home bigger than the 40GiB specified as minimum for it in the product description.

    Initial storage proposal before the latest change

    Now, more attempts will be performed in the first disk before giving up and moving to the next one. So, when the proposal is not possible with the desired sizes and with all the optional devices enabled, it will try to disable some optional settings and to relax the size requirements (using the minimum sizes instead of the recommended ones). In the following screenshot you can see the result of this adjusted initial proposal.

    Initial storage proposal after the latest change

    Tumbleweed: Add Online Repositories during Installation

    We added a step in the installation workflow to enable online repositories. This means you can now get the latest packages installed immediately during installation instead of installing older versions from the installation media first and later updating them to the latest versions from the openSUSE servers.

    Also, because of disk space limitations, the installation media don’t contain all the packages that openSUSE provides and thus not all system roles; some of the less frequently used desktops, for example, had not been available in the role selection (formerly desktop selection) dialog. With online repositories, they are now available immediately during installation.

    For details, see the corresponding pull request.

    Choosing the network backend

    And more changes to the installer you will be able to enjoy very soon in openSUSE Tumbleweed and also in the upcoming openSUSE Leap 15.1. Now it’s possible to choose the network backend, wicked or NetworkManager, to be used in the final installed system from the Network proposal.

    As easy as a click on a hyperlink, as you can see in the images below.

    New shortcut to change the network backend

    Managing SSH public keys

    When you want to access to a system via SSH, it is a good practice to use public key based authentication instead of the common password-based approach. For that reason, we have added support to handle authorized SSH public keys in the user and group management module, also known as the users module.

    Configuring SSH in yast2-users

    Additionally, you will be able to set up a public key for the root user at installation time, something that can be especially handy for products like the SUSE CaaS Platform.

    And it keeps moving!

    The YaST crew keeps working hard on fixing bugs and, specially, on building new features for everybody to enjoy in the upcoming versions of all the products of the SLE and openSUSE families. So, as much as we would like to keep talking to you all about all the improvements… it’s time to go back to coding! See you again in two weeks with a new report.

zypper-upgraderepo-plugin is here

August 7th, 2018 by

zypper-upgraderepo-plugin adds to zypper the ability to check the repository URLs either for the current version or the next release, and upgrade them all at once in order to upgrade the whole system from command line.

This tool started as a personal project when a day I was in the need to upgrade my distro quicker than using a traditional ISO image, Zypper was the right tool but I got a little stuck when I had to handle repositories: some of them were not yet upgraded, others changed slightly in the URL path.

Who knows how to Bash the problem is not exactly a nightmare, and so I did until I needed to make a step further.

The result is zypper-upgraderepo Ruby gem which can be integrated as a zypper plugin just installing the zypper-upgraderepo-plugin package.

Installing zypper-upgraderepo-plugin

Installing zypper-upgraderepo-plugin is as easy as:

  1. Adding my repo:
    sudo zypper ar https://download.opensuse.org/repositories/home:/FabioMux/openSUSE_Leap_42.3/home:FabioMux.repo
  2. Install the package:
    sudo zypper in zypper-upgraderepo-plugin

How to use it

Sometime we want to know the status of current repositories, the command zypper ref does a similar job but it is primarily intended to update the repository’s data and that slow down a bit the whole process.
Instead we can type:

$ zypper upgraderepo --check-current

To know whether or not all the available repositories are upgrade-ready:

$ zypper upgraderepo --check-next


As you can see from the example above all the enabled repositories are ready to upgrade except for the OSS repo which has a slightly different URL.

# The URL used in the openSUSE Leap 42.3
http://download.opensuse.org/distribution/leap/42.3/repo/oss/suse/
# The suggested one for openSUSE Leap 15.0
http://download.opensuse.org/distribution/leap/15.0/repo/oss/

Let’s try again overriding the URL without make any real change:

$ zypper upgraderepo --check-next \
--override-url 8,http://download.opensuse.org/distribution/leap/15.0/repo/oss/

Once everything is ok, and after performed a backup including all the repositories, it’s time to upgrade all the repository at once:

$ sudo zypper upgraderepo --upgrade \
--override-url 8,http://download.opensuse.org/distribution/leap/15.0/repo/oss/

Conclusions

That’s all with the basic commands, more information is available in the wiki page of zypper-upgraderepo gem where all the commands are intended with the only use of the gem, but installing the plugin they are also available as zypper subcommands like shown above, also a man page is available as

$ zypper help upgraderepo

Highlights of YaST Development Sprint 49

January 25th, 2018 by

Time goes by and the YaST wheel keeps rolling. So let’s take a look to what have moved since our previous development report.

More flexible NET installation ISOs

Network installation media for Tumbleweed or Leap only work properly with the exact repository they have been built for – which for Tumbleweed may mean they could be outdated after just one day.

You would then run into this message:

Linuxrc warning

To improve the situation the installer can now offer to download matching boot files (kernel and initrd, to be precise) from the repository if it detects this situation:

Linuxrc offering a solution, as always

Of course, you can say ‘No’ here – but then you’re back to the red dialog. 😀

Technically, what’s done is to download a new kernel/initrd pair from the repository and restart the installation process with them (using kexec). So be prepared for a slight déjà vu.

This feature is controlled by the kexec boot option.

Storage-ng lands into Tumbleweed: handle with care

But that’s not the only news we have about openSUSE Tumbleweed. Our usual readers already know about Storage-ng, our effort to rewrite the whole YaST storage stack from scratch. And they also know it’s still a work in progress. But since there were too many valuable changes blocked by the adoption of Storage-ng, it was decided it was time to push the red button. So we are glad to announce the Storage-ng era has started with its inclusion in the first official (open)SUSE product – starting with snapshot 20180117, libstorage-ng has replaced libstorage and, thus, yast2-storage-ng has replaced yast2-storage.

They say forewarned is forearmed, so an article was published in advance in news.opensuse.org to set the expectations and to provide and overview of the current status. We would like to encourage all openSUSE Tumbleweed users to (re)visit the article to get a better picture of the situation.

Alignment of partitions in the expert partitioner

An important part of that work in progress is the re-implementation of the Expert Partitioner with Storage-ng technologies. As mentioned many times in previous posts, this is mainly a 1:1 clone, with the same functionality presented in exactly the same way than the classic YaST partitioner. But some times we take the opportunity to introduce some improvement here and there, as we did this week with a topic that can have a very noticeable impact in the system performance: partitions alignment.

Although many people is not aware of it, the partitions in a system must be properly aligned to avoid the performance drop caused by excessive read-modify-write cycles. For details please refer to the great article at Wikipedia explaining the topic, especially the sections titled “4 KB sector alignment” and “SSD page partition alignment”. Moreover, leaving performance considerations aside, some partition tables require alignment to simply work, like DASD partition tables which need alignment to tracks (usually 12 sectors).

The new expert partitioner takes all that into consideration when creating and resizing partitions, ensuring always the required alignment (like the DASD tracks) and encouraging the optional performance-related one, avoiding undesired gaps between partitions in the process.

Detail of the Expert Partitioner dialog to create a partition

Above you can see the dialog for choosing the size for a new partition that, unsurprisingly, looks very much like the same dialog in the pre-storage-ng Expert Partitioner. If a size is specified by the user in that dialog (any of the two first options in the form), the start and end of the partition will be aligned to ensure optimal performance and to minimize gaps. That may result in a slightly smaller partition (with the difference being usually less than 1MiB). If a custom region is specified, the start and end will be honored as closely as possible, with no performance optimizations (although mandatory alignment, like DASD tracks, still will take place). This third option is the best to create very small partitions.

The same considerations for optimal alignment will also be taken into account while resizing an existing partition and calculating the minimal and maximal sizes suggested by the partitioner during that process.

Choosing the new size of a resized partition

Sanity checks for the storage setup

The possibility of bypassing the performance optimizations in the Expert Partitioner is just one example of the (potentially unleashed) power that tool provides. As a consequence of that flexibility, sometimes the user can overlook some important setup configurations or even make mistakes. To help with that, the Expert Partitioner recovered this week its ability to check the entered storage setup.

Once the user has set partitions, LVM volumes, file systems, mount points, etc. and decides to proceed, the Partitioner will validate that setup to ensure it fulfills all necessary requirements for booting and running the system. When some issue is detected, a popup message is presented to show what the problem is, offering the option to ignore the warning and move forward.

The resurrected partitioner sanity checks

Two kind of checks are carried out to ensure the partitioning setup validity. First, the presence of needed partitions for booting is checked. Booting requirements depends on the current architecture (x86, PowerPC, AArch, etc.) and other technical details like the partition table type (GPT vs MS-DOS). Then, the mandatory volumes for the current product are checked. The mandatory volumes are defined in the revamped partitioning section of the control file. Typically, only a volume for root and another for swap used to be mandatory, but now this is totally configurable by anyone defining the product (SLE, Leap, Tumbleweed, your own custom openSUSE derivative…).

As a bonus, all the sanity checks are now centralized (they used to be scattered around the YaST source code) and it’s easier to add new ones (you will miss some old checks at this moment) and to use them from other parts of YaST (like the bootloader module or AutoYaST).

More improvements in the Expert Partitioner

The new warnings and the alignment improvements commented above are not the only news on the evolution of the Expert Partitioner clone this week. Resizing of LVM devices has also been brought back to life, both for volume groups and logical volumes. In the case of logical volumes, the functionality is not much different, at least in the surface, from the partition resizing that was already present and that you can see in the screenshot of the alignment section.

On the other hand, in the context of the Partitioner, resizing a volume group actually means adding or removing physical volumes. Actions that are now possible again, including the corresponding checks. For example, a physical volume cannot be removed if it already exists on disk (that could destroy your data) or if the resulting size of the volume group is not enough to cover all its logical volumes.

Trying to remove the wrong PV

Apart from the mentioned functionality, there has also been improvements in how the Expert Partitioner presents the information. For example, now the “type” column shows the correct label and icon for each device instead of that useless TODO label. Moreover, similar TODO marks were replaced by proper data in the device overview tab.

TODO labels are gone

Minimize changes between the SLE15 “Installer” and “Packages” DVDs

The SUSE Enterprise Server 15 (SLES15) product can be installed from a bootable “Installer” DVD medium which contains the installer and a subset of packages needed for a very minimal system. The other packages are available either from a registration server (after registering the SLES product) or via a separate “Packages” DVD medium.

Due to the structure of those DVDs (with some packages being in present in both) the SLES installer was asking the user to change the medium several times during the installation process. Ideally the installer should use all packages from the “Packages” medium without changing the media.

In addition, there is yet another requirement for preferring the packages from the installation DVD to the packages available via a remote repository. Downloading a package from the internet is usually much slower than the DVD and can be problematic in network connections with a download limit or with a price based on the bandwidth usage.

Now the installer properly adjust the priority of all the repositories to achieve the desired behavior. To avoid possible side effects we decided to change the repository priority only when more than one repository is used and all repositories are local (e.g. DVD, hard disk, USB flash disk…). That means in some less common cases (2 DVDs + a remote repository) you will still need to change the medium but this is a safer solution.

Add On products in AutoYaST

For those using SLE Add On products, we have improved the error message if an Add On Product cannot be added during an AutoYaST installation. The user can see now which wrongly configured Add On Product has produced the error.

AutoYaST reporting which Add On is wrong

This will be specially useful with the upcoming SLE15, in which the concepts of Add Ons and Modules will become more relevant than ever.

Fixed a crash when shutting down the YaST user interface

And now it’s time for the corresponding dose of technical insights for those who enjoy that part of our reports.

When UI::OpenDialog() and UI::CloseDialog() calls didn’t match when shutting down the UI (user interface YaST component), you’d get a segmentation fault with a core dump. Well, you did want to shut down YaST, but probably not like that. This is now fixed.

After tracking this down, it was surprisingly simple to reproduce: Just use the YaST version of the trivial “Hello, World” program and comment out the UI::CloseDialog() call.

This was a case of providing additional error reporting causing more problems than the original error: leaving dialogs open while terminating the program is an error, of course. But fixing this little problem by cleaning up the remaining dialogs lead to handling widgets after some of the underlying infrastructure (in this case the QApplication) was already destroyed, so all the QWidgets were also destroyed (because the QApplication takes care of that), but YaST’s generic UI layer was still unaware of that fact and tried to destroy them again.

This is now fixed by properly cleaning up the widget tree in YaST’s generic UI layer first which will also clean up the associated QWidgets so there is nothing left to clean up for the QApplication.

This might also fix a number of similar segfaults in other situations where the YaST Ruby engine would need to shut down because of other problems, e.g. when there is an unhandled Ruby exception.

Surprisingly enough, this must have been a very old (10+ years?) bug, but it never became quite obvious, or at least nobody was ever annoyed enough to try to track it down.

If you want even more details, check the conversation in the bug report.

More to come

The end of this sprint caught up with a lot of almost finished stuff. But following the Scrum principle of “nothing is done until it fits the Definition of Done”, we don’t blog about such stuff. Fortunately, that means the next report will likely be quite juicy. So, see you again in a couple of weeks!

Highlights of YaST Development Sprint 48

January 9th, 2018 by

The YaST team finished its 47th Sprint right before the Christmas break but, sadly, we had not published the corresponding report… until now. The last sprint of the year brought some interesting changes, like Chrony support for AutoYaST, better multi-products medium handling, etc. So let’s recap those changes.

Chrony support in AutoYaST

As part of our effort to support Chrony as the default NTP service for (open)SUSE, we have revamped how AutoYaST handles the configuration of such a service. The first noticeable change is that we have redesigned the schema which, instead of containing low level configuration options, is now composed of a set of high level ones that are applied on top of the default settings.

And here is how the new (and nicer) configuration looks like:

<ntp-client>
  <ntp_policy>auto</ntp_policy>
  <ntp_servers config:type="list">
    <ntp_server>
      <iburst config:type="boolean">false</iburst>
      <address>cz.pool.ntp.org</address>
      <offline config:type="boolean">true</offline>
    </ntp_server>
  </ntp_servers>
  <ntp_sync>15</ntp_sync>
</ntp-client>

Updating the Remote Administration Capabilities

During this sprint, the remote administration client has been deeply modified. To begin with, as xinetd is being replaced by systemd sockets, we have dropped that dependency (adjusting the code accordingly).

Additionally the VNC handling have been improved too. Until now, YaST offered the possibility to connect through a web browser using a Java applet. Now YaST allows the user to enable/disable this feature (check the screenshot below to see how it looks now). It is worth to mention that Michal Srb has replaced the old viewer with novnc, a JavaScript based one. Thanks a lot for that, Michal!

And last but not least, we have seized the occasion to do some code cleaning, reimplementing some dialogs using the Common Widget Manipulation object oriented API.

Modifying AutoYaST Profile During Installation

AutoYaST offers a cool feature that allows the profile to be modified during the initial stages of the installation using an user script. So you can run a script which adjusts the profile and AutoYaST will read it again. If you are interested in such a feature, you could find more information in the official documentation.

On the other hand, in our previous report, we mentioned that AutoYaST was able again to use multipath devices using the new storage stack. But we didn’t count that it was possible to modify the profile on runtime so the initialization happened too early.

Now the bug is fixed so you can again adjust any storage setting using the aforementioned feature.

Properly Handling Selected Modules

As you may know, some time ago we added a support for the multi-product media (DVDs which contain more than one repository/product in separate subdirectories). This time we fixed some issues regarding this functionality.

Originally after selecting several products only one of them was actually selected to install and only one product was displayed in the installation proposal. Fortunately, those issues have been addressed now.

Unified Look & Feel for Multi-Product Selection Dialog

For the multi-product DVD media we used this selection dialog:

The functionality is very similar to the on-line product selection dialog displayed after registering the system:

As you can see the look & feel is quite different, but from the usability point of view the dialogs should look the same regardless whether the products are added from a DVD medium or from an on-line source (a registration server).

This sprint we improved the DVD media dialog to better match the registration dialog:

.

The dialog is still not exactly the same, but now it looks more similar so users should feel more familiar with it. There is also displayed an additional note about not handling the product dependencies automatically. This note was already present in the help text but that is hidden by default. As we got quite a lot of bug reports about this issue we decided to make this fact more prominent.

(Note: The dialogs actually cannot look the very same as the DVD media currently lack some information like the dependencies, beta status, detailed descriptions…).

Dropping SYSTEMCTL_OPTIONS Variable Support

We have been using the environment variable called SYSTEMCTL_OPTIONS (which is SUSE-specific) in our systemd services to prevent locks in dependencies. As this hack will not be necessary for upcoming the (open)SUSE 15 version, it will be dropped from systemd and, therefore, we already removed it from our systemd services.

Unifying Disklabel Handling in AutoYaST

When specifying an AutoYaST partitioning schema, you can select which partition type you want to use for each device (MSDOS, GPT, DASD, etc.). In the past, AutoYaST implemented its own logic to select which one to use in case that it was not specified by the user. After this sprint, AutoYaST relies on the new storage stack in order to decide which option fits better when the user does not specify one.

Bonus: Automatically Checking the Defined Systemd States

Some time ago we had serious issues with service management in YaST (see bug#1012047 and bug#1017166). The problem was caused by introducing a new systemd service state which was not expected by YaST. We fixed the problem by correctly handling the new state.

But the main problem was that we (as the YaST developers) were not notified about this major system change and we found this change later after we got bug reports in Bugzilla. To avoid this problem again in the future we decided to implement a script which would regularly check the defined systemd states notifying us if a unknown state was detected.

To implement the regular check we use the Travis cron job feature which allows running continuous integration builds not only after a change is pushed into the repository but also in regular intervals, even when there is no change in the code.

Alternatively you can use any CI service, but we chose Travis because it is easy to use and we already use Docker for normal CI jobs which allows us to run the latest systemd from openSUSE Tumbleweed in an easy way.

In this case we could possibly run the script in OBS when building the yast2-services-manager package, but that would need adding systemd to BuildRequires which does not sound as a good idea…

So if you also need to run some check scripts regularly you can see more details in this pull request.

Conclusion

2017 was an exciting year with a lot of interesting stuff: a new storage layer, multi-product installation medium, integration of new components (firewalld, chrony, etc.). And it looks like 2018 will not be different and we will have a lot of fun.

Thanks for your support and happy new year!

Highlights of YaST development sprint 38

July 14th, 2017 by

Here we go again with a new report from the YaST trenches. This time with the storage reimplementation as the clear star of the show.

Storage reimplementation: the proposal adapts, you succeed

As we have announced in our previous sprints and as you probably already know, the YaST team is working hard to rewrite the whole storage stack on time for SLE15 and openSUSE Leap 15. As part of this reimplementation we have designed a brand new storage proposal that automagically offers the user the best combination of partitions and LVM volumes based on the current configuration of the system and the user preferences.

The storage proposal in action

When we are working with very small disks or with special technologies like DASD (which doesn’t accept more than three partitions by device), the storage proposal might not be able to generate a valid initial proposal honoring the initial requirements of the product (e.g. creating a separate home partition and enabling btrfs snapshots for the root partition in the openSUSE Leap case). Now the proposal is not limited to fail when it is not possible to satisfy the default product requirements. Before giving up, the new system looks for alternatives, like discarding the separate home partition or disabling snapshots. Moreover, now the proposal is able to automatically adjust the size requirements not only for root, but also for swap and home. And, of course, the guided setup continues there for fine tuning the proposal settings.

Desktop selection improvements

As our usual readers also know, we recently introduced a more fair desktop selection screen for openSUSE, both Leap 42.3 and Tumbleweed. We used part of the latest sprint to implement some feedback we gathered about the wording and behavior of that dialog.

Revamped desktop selection screen

That feedback gathering included some discussions on how to make user experience nicer after selecting one of the user interfaces available through the “custom” option. As a result, the awesome openSUSE crew created a new mechanism for selecting the default window manager on each graphical login, so YaST can delegate the details to the maintainers of those alternative interfaces.

How everything works now? Glad you asked. 🙂

If the user select KDE or GNOME in the YaST dialog, /etc/sysconfig/windowmanager is configured to point to that desktop by default. If the “custom” option is selected, then YaST does not enforce any interface in that file and the new mechanism comes into play. It relies in the default.desktop file, which defines the default window manager and can be managed by the common update-alternatives workflow. Meaning it can be easily tweaked by the package maintainers and by the users, specially since YaST includes a nice module for managing alternatives.

Storage reimplementation: improvements in the expert partitioner

Although, as explained above, we keep improving the storage automatic proposal to support more and more situations, we cannot ignore that flexibility and adaptability have always been two of the flagships of (open)SUSE. And one of the most prominent examples is the YaST Expert Partitioner.

As detailed in our report of the sprint 36, we have been rewriting that powerful Swiss Army knife using the new storage stack but keeping the same user interface and functionality. So far, the new implementation was only able to display information about the existing partitions, LVM systems and MD RAIDs. But now we added many options to create, edit and delete partitions.

Using the expert partitioner during installation

It is still a work in progress because the number of possibilities offered by the YaST Partitioner is sometimes overwhelming and implementing them all takes time, but it’s progressing nicely.

Beside the improvements in the Partitioner itself, we also worked on its integration in the installation workflow. Now the Expert Partitioner can be used to refine the schema automatically proposed by the installer. As a bonus, the behavior of the “abort” and “finish” buttons has been improved in relation to the Expert Partitioner currently available in (open)SUSE, which historically shows a usability inconsistency there compared to the rest of the installation process.

Fixed Automatic Patch Installation

In both SLE and openSUSE Leap, the online updates can be installed either manually or automatically at some regular intervals. For the automatic installation we provide the yast2-online-update-configuration package which provides a cron job script and an YaST module for configuring it (hint: it is not installed by default, you might give it a try, maybe it is something “new” for you).

Configuring online updates via YaST

You can configure how often the patches should be installed or filter the patch categories in the YaST module, but we got a bug report saying that when multiple patch categories are selected only one is actually used.

It turned out to be a trivial mistake in the cron job and we fixed it for SLE12-SP2 and Leap 42.2, as well as for the upcoming SLE12-SP3 and Leap 42.3. So if you use this module and want to use the category filter then it’s recommended to upgrade the package.

Storage reimplementation: many steps forward in the AutoYaST integration

And going back to our new storage stack, we keep working to integrate it better with other parts of YaST, specially AutoYaST. During this sprint we polished some rough edges and we added support for MD RAID. With all that, now is possible to automatically setup a system based on an AutoYaST profile containing any combination of partitions, LVM systems and MD arrays, including encryption at any of the levels.

AutoYaST summary of actions on partitions, MD RAID and  LVM

But the relation of AutoYaST with the system works in both directions. Apart from being able to install a system based on an AutoYaST profile, it also offers the interface to export the current system configuration, including the storage layout, to a profile in order to reproduce the system later. During this sprint we also ported that logic to rely on the new storage layer.

It was harder than it sounds, due to the need of keeping the backwards-compatibility with several behaviors that have been introduced along the AutoYaST history to adapt to several specific situations. On the bright side, the new code is easier to follow, includes behavior-driven automated tests (RSpec) and contains information about the rationale of each decision… which in some cases required some archaeology.

Storage reimplementation: fixed bootloader proposal in S/390 and PowerPC

Another part of YaST we are constantly working to integrate better with the new storage stack is yast2-bootloader. Although the new storage system was already able to correctly setup a valid disk layout for most situations and architectures (each one with it’s own requirements for booting), our bootloader module is still not fully compatible with the new system in all those scenarios.

During this sprint, we adapted it to ensure all the combinations suggested by the new storage stack (partitions, LVM, encryption and so on) are correctly covered by YaST2-Bootloader. As a result, we can already say that our testing ISO images are fully installable in any x86_64, PowerPC and S/390 system by just clicking “next” a few times. And we have automated integration tests (a separate openQA instance) to ensure the resulting system boots just fine.

UDEV device id on PowerPC

We know some of our readers enjoy our more technical posts and like to lurk into the kitchen to see how we deal with all kind of surprises maintaining a complex tool like YaST. Today’s chapter in that regard started with a bug report about the bootloader being installed in the wrong device name (/dev/vda instead of /dev/vdb) in an emulated PowerPC machine in openQA.

After a lot of investigation and with help from our PowerPC expert, we found the culprit, that turned out to be an emulator quirk. The next couple of paragraphs may be interesting or daunting, depending how much virtualization and PowerPC jargon you know. Be warned.

On POWER, the PReP partition containing the bootloader has no unique identifier other than the serial number of the disk on which it created. QEMU virtualization does not provide any disk serial number when the user does not explicitly specify one. This means that the PReP partition in QEMU installation does not have any unique identification and the partition name may change when a disk is added or removed from the virtual machine or the storage configuration otherwise changes. This may lead to system errors related to bootloader installation and updates.

It is recommended to assign a unique serial number to each disk in a QEMU virtual machine on POWER when it is expected the storage configuration of the virtual machine may change. Otherwise, there is nothing YaST or any other tool running within the virtual machine can do to avoid the problems. So this time we only did the investigation part, with the fix coming from the openQA side, which was improved to explicitly set serial numbers when needed.

Storage reimplementation: better support for advanced scenarios

As exposed above, our StorageNG testing image already works in all the supported architectures and in scenarios combining plain partitions, LVM, RAID and encryption in any way. But there are even more situations and technologies currently supported by YaST that we need to incorporate into the new stack.

First of all, we used this sprint to add Multipath I/O support to the new libstorage. Now it can also be combined with all the other mentioned technologies (like having an LVM system on top of an encrypted multipath device), although the storage proposal still needs to be adapted to play nicely with preexisting multipath setups.

As usual with the library stuff, the best “screenshot” we have to offer is one of its geeky autogenerated graphs.

A multipath layout in libstorage-ng

Another scenario that goes beyond the most regular use cases is installing the system into a network storage device, instead of a local hard disk. Now, the new storage system can report whether a root filesystem via a network is used. When that happens, YaST sets the network interfaces with the start mode nfsroot, which is used to avoid interface shutdown and, therefore, the unavailability of the system.

That’s all, folks!

Once again, we omitted the boring parts about bugfixing (with yast2-ntp-client being the star in that regard) and similar stuff. We hope you enjoyed the report and we hope to reach you again in two weeks. Meanwhile…

Have a lot of fun testing the Leap 42.3 pre-releases and reporting bugs!

Highlights of YaST development sprint 37

July 3rd, 2017 by

Since we announced in the latest report, we decided to shorten our development sprints from three to only two weeks. As a natural consequence, this is the first report of a series of shorter ones. But shorter doesn’t have to mean less juicy! Keep reading and enjoy.

Displaying very long changelog

We got a bug report about YaST not responding when a very long package changelog was displaying in the package manager. It turned out that some packages have a huge change log history with several thousands entries (almost 5000 for the kernel-default package). That produces a very long table which takes long time to parse and display in the UI.

The solution is to limit the maximum number of displayed items in the UI. You cannot easily read that very long text anyway, for such a long text you would need some search functionality which the YaST UI does not provide.

Finding the limit, that magic number, was not easy as we want to be backward compatible and display as much as possible but still avoid that pathological cases with a huge list.

In the end we decided to go for a limit of 512 change entries. The vast majority of packages have way fewer entries, so you should not notice this clipping functionality. When the limit is reached a command to display the full log is displayed at the end so you can easily see the missing part when needed. (Hint: the widget supports the usual copy&paste functionality, you can copy (with Ctrl+C) the displayed command and paste it into a terminal window directly.)

Clipping the kernel changelog

Storage reimplementation: MD RAID support in the rewritten partitioner

In the previous report we told you we are rewriting the YaST partitioner to use the new storage stack and modern reusable CWM widgets under the hood while retaining exactly the same behavior and look&feel on the surface. We are reimplementing one feature at a time, and this sprint it was the MD RAID’s turn.

Now the partitioner displays current RAIDs, with details about them and the devices used to construct each RAID. If a picture is worth a thousand words, here you have 3,000 words:

RAID list in the rewritten partitioner

Devices in a given RAID in the rewritten partitioner

RAID details in the rewritten partitioner

No, you can’t have a separate partition for /var/lib/docker for CaaSP – so here you have it

Our brand new containers-oriented solutions, SUSE CaaSP and its openSUSE counterpart Kubic, need to have /var/lib/docker as a separate partition for several reasons. It was a separate Btrfs subvolume already, but that wasn't quite separate enough. 🙂

The problem was just that the automated storage proposal in the old YaST storage subsystem is not prepared for anything like this; it can deal with root, swap and optionally a separate home partition (or volume if using LVM). That's it. Extending the current syntax of the control.xml file to be able to specify arbitrary partitions and adapting the code of the old proposal to understand and handle this new specification was unfortunately almost impossible. Sadly, we have to admit the old code is hardly maintainable these days and not flexible enough to accommodate this kind of changes in a safe way.

This is one (out of many) reasons why we are currently in the process of rewriting that entire YaST storage subsystem, as you already know from many previous posts in this blog. But, as you also know, the new system will debut in SLES15 and openSUSE Leap 15.0, too late for the current SUSE CaaSP and openSUSE Kubic.

We decided we'd introduce a hack based in the old proposal, well knowing that hacks accumulated in code can develop their own life just like Dust Puppy. But we plan to kill that hack immediately once StorageNG arrives.

So the hack was to simply use the logic for the separate home partition and repurpose it, keeping all the respective parameters in control.xml and introducing yet another one called home_path where the actual mount point for that partition can be specified — in this case /var/lib/docker. The tradeoff is that there can be no separate /home anymore in parallel to that.

That "feature" should not be used outside of that very specific use case in CaaSP and we even considered the possibility of not documenting it too publicly to avoid misuse. But we are living the open source spirit and whatever we do, we do in public. Even if it's the (quite embarrassing) fact that we consider changes to certain parts of our own code too risky. But again, that's why we are pushing the storage layer reimplementation (Storage-NG) so hard: we want to regain control over that part.

Storage reimplementation: custom partitioning with AutoYaST

And talking about the new storage layer and our previous posts, you already know we are working hard to integrate it with AutoYaST. For the time being, custom partitioning layouts are supported, but only using plain partitions and LVM. Other features, like RAID or Btrfs subvolumes support, are still missing.

A nice thing about the new code is that it relies as much as possible on the new storage layer. On older versions, AutoYaST implemented some logic on its own and that caused some unexpected troubles. Fortunately, that is not the case anymore, and the new code looks way easier to extend and maintain.

YaST does not write directly in /etc/vconsole.conf anymore

When configuring the system keyboard, YaST used to write the keyboard map configuration directly in the /etc/vconsole.conf file. However, this approach is no longer appropriated since it may cause undesired effects to other tools. Now YaST uses the Systemd tool localectl to set the keyboard map in /etc/vconsole.conf, instead of writing in it directly. Another step to make YaST a good citizen of the Systemd world.

Storage reimplementation: named RAID arrays

If you use MD RAID arrays you probably know there are two ways of identifying them – by number or assigning a meaningful name to them. Named arrays use device names like /dev/md/<name> instead of /dev/md<number>.

During this sprint, we taught the new libstorage how to create and manage named RAID array, in addition to the already supported numbered ones. If the user decides so, the MD RAID name is used in fstab instead of the UUID (which was the only option with the old libstorage). We also made sure to improve the behavior in several scenarios, so we consider some bugs of the old storage to be fixed (or obsoleted, if you prefer) with storage-ng.

Linux also supports names of the form /dev/md_<name>. The new library is also able to handle this format, but the feature is intentionally disabled and documented to be unsupported because other parts of the system could not be 100% verified to work in that scenario. And we take quality assurance very seriously before labeling a feature as “supported”.

If you are not happy enough with the screenshots showing the RAID support in the partitioner, here you have more pictures. But, as usual with stuff implemented in the library, here “pictures” doesn’t mean screenshots, but fancy automatically generated graphs.

Stay tuned

Of course there are a lot of other things we did during the last two weeks, although some of them were considered not interesting enough for this report or were not finished on time. We are already working in finishing the unfinished stuff and implementing new exciting improvements. And the bright side is that you only have to wait two weeks to know more… so stay tuned.

Highlights of YaST development sprint 31

February 20th, 2017 by

As we announced in the previous report, our 31th Scrum sprint was slightly shorter than the usual ones. But you would never say so looking to this blog post. We have a lot of things to talk you about!

Teaching the installer self-update to be a better citizen

As you may know, YaST includes a nice feature which allows the installer to update itself in order to solve bugs in the installation media. This mechanism has been included in SUSE Linux Enterprise 12 SP2, although it’s not enabled by default (you need to pass an extra option selfupdate=1 to make it work).

So after getting some feedback, we’re working toward fixing some usability problems. The first of them is that, in some situations, the self-update mechanism is too intrusive.

Consider the following scenario: you’re installing a system behind a firewall which prevents the machine to connect to the outside network. As the SUSE Customer Center will be unreachable, YaST complains about not being able to get the list of repositories for the self-update. And, after that, you get another complain because the fallback repository is not accessible. Two error messages and 2 timeouts.

And the situation could be even worse if you don’t have access to a DNS server (add another error message).

So after some discussion we’ve decided to show such errors only if the user has specified SMT or a custom self-update repository (which is also possible). In any other case, the error is logged and the self-update is skipped completely.

You can find further information in our Self-Update Use Cases and Error Handling document.

During upcoming sprints, we’ll keep working on making the self-update feature great!

Configuring workers during CaaSP installation

While CaaSP release approaches, our team is still working hard to satisfy the new requirements. Thankfully, YaST is a pretty flexible tool and it allows us to change a lot of things.

For CaaSP installation, YaST features a one-dialog installation screen. During this sprint, configuration of the Worker role has been implemented, including validation of the entered URL and writing the value to the installed system. You can check the animated screenshot for details.

The CaaSP worker configuration

New desktop selection screen in openSUSE installer

The world of Linux desktop environments change relatively quick, with new options popping-up and some projects being abandoned. Thanks to the openSUSE’s community of packagers we have a lot of these new desktop environments available on the openSUSE distributions. But the status of those packages for openSUSE is also subject to changes: some desktop environments are poorly maintained while others have a strong and active group of packagers and maintainer behind.

The YaST Team does not have enough overview and time to watch all these desktop environment and evaluate which one is ready or not for being in the installer’s desktop selection screen. So the openSUSE Release Team decided to replace this dialog with something a bit more generic but still useful for newcomers.

They asked the YaST Team to come up with a new dialog featuring the two openSUSE main desktops (KDE Plasma and GNOME) and allowing the easy selection of other environments without reworking the dialog in the future. The goal of that new dialog was to replace the existing one you can see in the following screenshot.

The old desktop selection screen

We decided the new dialog should rely on patterns for several reasons. The main one is that the set of patterns is under the close control of the openSUSE community, which looks more closely than us to the desktop environments and their integration into the distribution. Moreover, each pattern specifies its own icon and description that can be somehow re-used by the installer.

We also took the opportunity to merge this almost empty and outdated dialog with the new one.

The old additional repositories screen

Addons are no longer produced for openSUSE, so only the second checkbox made any sense nowadays. Moreover, the functionality of that second checkbox directly influence the available selection of patterns, so it made more sense to merge everything in a single screen that keeping an extra step in the installation just to accommodate a checkbox.

So we sent a proposal for the new dialog to the opensuse-factory mailing list and, after implementing many of the ideas discussed there (like better wording or using a button instead of a checkbox for the online repositories), this is the new dialog that replaces the two ones mentioned above.

The new desktop selection dialog

Selecting “custom” will take the users to the already existing patterns selection screen. Just in case you don’t remember how that screen looks like, you can check this image.

The patterns selection screen

If these screenshots are not enough to make your mind about the change, you can check the following animation, in which KDE Plasma is initially chosen to be changed at a later point (going back in the workflow) to LXQt.

Desktop selection animation

It will take some time before the changes hit the Tumbleweed installer, since they obviously have a non-trivial impact on the openQA tests, that will need some adaptation.

We would like to thank everybody that contributed to this new feature by providing feedback and suggestions through the mailing list. Once again, the openSUSE community has proved to be simply awesome!

Storage reimplementation: improved handling of logical partitions

Our reimplementation of the storage layer keeps getting improvements here and there. With the base x86 case working, we are now implementing the tricky parts, like the partitions renumbering that takes place when dealing with logical partitions in a MBR style partition table.

With GPT partition tables or with primary partitions in a MBR partition table, the partition gets a number (like sda1) when it’s created and keeps that number for its whole lifetime. But logical partition gets constantly renumbered when other partitions are created and destroyed. We need to know in advance what device name every partition will get in order to configure everything beforehand and only commit the changes to the disk when the user clicks in the “install” (during the installation process) or “commit” (if running the expert partitioner).

Now, libstorage-ng is able to simulate in memory the re-numbering process, so we can calculate all the settings related to partitioning (like the configuration of the bootloader) before really touching the disk.

Making kdump work in CaaSP

When you enable the Kernel Crash Dump (kdump), you probably expect that you will get a kind of core dump, like you do for regular processes. What you might not expect is an automatic reboot. That is a nice bonus. If you are tired of waiting for an actual kernel crash, you can test your kdump setup by triggering a crash yourself. Just run this as root:

echo c > /proc/sysrq-trigger

The way kdump works is by allocating some extra memory at boot time and putting a second kernel+initrd there. When the main kernel realizes it is crashing, it transfers control (by kexec) to the other one, which has only one purpose, to dump the memory of the crashed kernel.

In the upcoming Containers as a Service Platform, kdump was not working because the root filesystem is read-only there and we were not able to create the kdump initrd. We fixed it by creating it just after the RPMs are installed, when the FS is still read-write.

Fixes for Snapper in CaaSP

Kdump was not the only component affected by the fact of having a read-only filesystem in CaaSP. Snapper was also running into problems when trying to execute the rollback and cleanup operations. Now the problems are fixed. While working on that we did enough interesting findings to deserve a separate blog entry. So you can expect a new entry in the Snapper.io blog soon.

Snapshot-related improvements in the expert partitioner

While we work on the new storage system, we are still taking care and improving the current one that will continue to be shipped with SLE12-SP3, SUSE CaaSP and openSUSE Leap 42.3. During this sprint we introduced a couple of usability improvements in the expert partitioner related to Btrfs subvolumes.

First of all, we moved the “Enable Snapshots” checkbox that was pretty much hidden in the “Subvolume Handling” dialog to the place where it really belongs – below the selector of the file-system type.

New location for enabling snapshots

In addition, we added the warning you can see in the screenshot below for users enabling snapshots in a potentially problematic setup.

Warning for snapshots in a small partition

Bring back the beta warning on CaaSP

And talking about warnings, we improved the CaaSP installation dialog to show the Alpha/Beta product warning at the beginning. After changing the CaaSP installation to use the all-in-one dialog described in our previous post, this feature was lost as it is part of the initial EULA dialog. Now it is back and the users should now properly see the current product status.

The CaaSP alpha/beta warning pop-up

Storage reimplementation: encrypted LVM proposal

Back to our storage layer reimplementation, the new system is now able to propose a setup with encrypted LVM. Since some sprints ago, we were already able to propose a partition-based and a LVM setup. That means the new proposal is now feature-pair with the old one, with the only exceptions of Btrfs sub-volumes (that shouldn’t be a big challenge) and s/390 storage (that is still not properly managed by the underlying libstorage-ng).

The bright part is that the new code is written with re-usability in mind, which means implementing an encrypted partition-based proposal (with no LVM) would be trivial. The new code is 100% covered by automatic unit tests and scores to the top in all the automatic quality checkers we have run (like Rubycritics, CodeClimate, and Rubocop).

So now we are prepared for whatever the future brings, having lost no single feature in the way.

Storage reimplementation: prototype if the UI for the proposal settings

The next challenge is to make the power of that new storage proposal available to the users through the user interface. In the previous post we presented the document describing the general ideas we wanted to discuss with UX experts.

It’s our pleasure to announce we met the experts and we very easily reached an agreement on how the user interaction should be. During this sprint we already implemented a prototype of the future proposal settings wizard that is, as usual, included in our testing ISO.

Now that we have solid foundations, it’s very likely that the following sprint will result in the fully working wizard, with almost-final wording and design.

Support for CaaSP added to the AutoYaST integration tests

While fixing a problem with AutoYaST and CaaSP, we decided to extend our AutoYaST integration testing tool to support CaaSP. But, as a side effect, we also made some additional improvements that were long overdue (like improving the procedure to download ISOs, reducing timeouts, etc.).

Now, our internal Jenkins instance takes care of running AutoYaST integration tests for the development version of CaaSP (as it already does for SLE 12 SP2).

Services manager moved to first auto-installation stage

AutoYaST allows to define a set of services to be enabled/disabled in the installed system, applying this configuration during the 2nd stage (after the first reboot).

Unfortunately, this approach won’t work for CaaSP because, in that case, the 2nd stage won’t be available. For that reason, during this sprint, we have adapted AutoYaST to write services configuration during 1st stage.

As usual, not only CaaSP, but other future (open)SUSE versions will benefit of this change.

Faster YaST installation when the release notes cannot be downloaded

Sometimes a very small simple change in a program makes a very noticeable impact in its everyday use. That’s the case of a fix implemented during this sprint. YaST usually re-tries to download the distribution release notes several times if the first attempt was unsuccessful. Although that’s correct from a general point of view, there are cases in which retrying makes no sense and only delays the installation. We have now added failed DNS resolution to that list of cases, which should result in a noticeable faster installation in many scenarios.

Moreover, in the case of AutoYaST we now skip downloading the release notes completely. Downloading them don’t make us much sense in the kind of unattended scenarios handled by AutoYaST and skipping this step and all the possible associated problems result in a more robust process.

Better continuous integration for Libyui

In the previous sprint we switched to Docker at Travis so we could build and test our packages in the native openSUSE system instead of the default Ubuntu. This sprint we did the same change also for the Libyui library which implements the UI part of YaST.

Originally we could not build the Libyui packages at Travis as either the required build dependencies were missing or were present at a very old unusable version. With this switch we can easily use the latest packages from openSUSE Tumbleweed and thus enable Travis builds for all Libyui packages.

See you after Hack Week!

As already mentioned, this week is Hack Week 15! So our Scrum process will be on hold for some days. That doesn’t necessarily mean the YaST development will stall. Since there are quite some YaST-related projects in the Hack Week page, you can expect YaST to simply go in unexpected directions. 🙂

And remember Hack Week is not a SUSE internal initiative, we are open to collaboration from anybody within or outside the company. So jump in and have fun with us!

Highlights of YaST development sprint 29

December 22nd, 2016 by

It’s Christmas time and since (open)SUSE users have been nice, the YaST team brings some gifts for them. This is the result of the last development sprint of 2016.

As you may have noticed, in the latest sprints we have been focusing more and more in making SUSE CASP possible. That’s even more obvious in this last sprint of the year. For those that have not been following this blog recently, it’s probably worth to remember that SUSE CASP will be a Kubernetes based Container As a Service Platform.

But our daily work goes beyond CASP, so let’s take a look to all the highlights.

More improvements in the management of DHCLIENT_SET_HOSTNAME

In the previous report we presented the changes introduced in yast2-network to make the configuration parameter DHCLIENT_SET_HOSTNAME configurable in a per-interface basis.

One of the great things about working in an agile an iterative way, presenting and evaluating the result every three weeks, is that it allows us to detect room for improvements in our work. In this case we noticed some discrepancy in the expectations of Linuxrc and yast2-network and also some room for improvement in the code documentation and in the help texts.

Thus, we used this sprint to refine the work done in the previous one and tackle those problems down.

Improved error message

Ensure installation of needed packages

Another example of iterative development. We already presented in the report of the 26th development sprint a new mechanism to detect when the user had deselected during installation some package that was previously pre-selected by YaST in order to install the bootloader. Since the new functionality proved to work nicely, we decided to extend it to cover other parts of the system beyond the bootloader.

The software proposal now contains an error message including a list of missing packages or patterns, in case the user deselects some needed items.

Warning about missing packages

After clicking the Install button the installation is blocked, the user must resolve the problem either by selecting the packages back or by adjusting the respective YaST configuration (e.g. do not install any bootloader and disable the firewall).

Blocking an incomplete installation

Rethinking the expert partitioner

May we insist one more time on the topic of using Scrum to organize our work in an iterative way? 😉 As our usual readers should already know, we structure the work into minimal units that produce a valuable outcome called PBIs in Scrum jargon. That valuable outcome doesn’t always have to be a piece of software, an implemented feature or a fixed bug. Sometimes a document adds value to YaST, specially if it can be used as base to collaborate with people outside the team.

Our readers also know that we are putting a lot of effort in rewriting the whole storage layer of YaST. That also implies rewriting the most powerful tool known by humanity to define partitions, volumes, RAIDs and similar stuff – the YaST expert partitioner.

It would be great if we could use the opportunity to make it both more powerful and more usable. You can take the first part for granted, but we are not so sure about our UI design skills. That’s why we wanted to have a base to discuss possible changes and alternative approaches with UX (user experience) experts. And we decided that it was worth to invest some time to create a document collecting the state of the art and some ideas for the future and to send that to SUSE experts in UX and to anybody with some interest in the topic.

Here you can find that fine piece of documentation. Take a look to that document if you want to peek into YaST developers’ mind. That’s the kind of stuff we discuss when we are about to start rewriting something… specially something that need to serve hundreds of different use cases.

And of course we would like to know your ideas or thoughts. We usually discuss this stuff at the public #yast IRC channel and at the yast-devel mailing list. But if you prefer so, you can simply open an issue at the repository hosting the document. Whatever works for you.

Rethinking yast2-network

But that was not the only documentation PBI finished during this sprint. Inspired by the first fruits of the storage layer reimplementation, we decided yast2-network also deserves a reincarnation.

As we did in the past with yast2-storage and libstorage, the first step is to collect as much information as possible about what can be currently done with the module and how it behaves in several situations, specially in tricky or complex scenarios. The outcome was three documents, one about the behavior during installation (installation.md), a second one about AutoYaST (autoinstallation.md) and another collecting general features (features.md).

CASP: merged dialogs for root password and keyboard layout

CASP is a product targeted to a quite specific use case with simplicity as a main priority. The installation process has been streamlined to a minimal set of dialogs to configure just the very basic stuff. Among other removed things, there is no step to configure the system language. That can be a problem when entering the root password (the only user that will be created during installation), since the language settings screen is normally also used to select the keyboard layout.

The implemented solution is shown in the screenshot below. As you can see, the keyboard layout and root passwords selections are merged into a single step. As a bonus, we made both widgets more reusable, opening the possibility to place the root password widget or the keyboard layout selection anywhere.

Keyboard layout and root password screen

Storage reimplementation: handling GPT disks in the installation proposal

After several sprints reporting small steps forward, in the 27th sprint we were happy to announce that our testing ISO for the new storage stack was fully installable under certain circumstances. As we reported, it worked in UEFI or legacy systems with the only requirement of having a pre-existing MBR partition table in the disk.

Now we can say it also works with GPT partition tables and even with systems with a mixture of both technologies.

Making the GPT scenario work was much harder that it sounds due to several factors, like the strange way in which parted handles partition types in GPT or some peculiarities in the way the space is distributed in such partition tables.

But now our test ISO can install a fully functional system in the four combinations of MBR/GPT partition table and UEFI/Legacy boot, as it can be seen in the next image.

Storage proposal in several scenarios

The storage reimplementation gets its own openQA instance

But there are better ways than screenshots to prove that something is working, even to prove it keeps working after future modifications. And in (open)SUSE we have one of the best tools for that – openQA.

We have always considered having the new stack tested in openQA as the first big milestone in its development (and we are finally there!) but we are aware that openQA.opensuse.org is already quite busy testing a huge combination of products, architectures and scenarios… even testing releases of openQA itself. Fortunately openQA is free software and can be installed anywhere so we created our own instance of openQA to test YaST stuff, specially the new storage layer.

So far, that instance is hosted in the internal SUSE network, which is enough for us to get continuous feedback about the changes we introduce. In addition to installing the new instance and configuring it to continuously grab and check the latest testing ISO, we had to introduce several changes in the ISO itself with the goal of keeping our tests as aligned as possible with the tests performed in the current Tumbleweed version by openQA.opensuse.org.

For example, we made sure the ISO was properly signed to avoid the need to always pass the insecure=1 boot argument. We also included several packages that were missing in order to make sure the ISO included all the software checked during the so-called MinimalX test and to make sure it shared the look and feel with a regular Tumbleweed, since many openQA checks are screenshot-based.

From now on, we can back every new feature with the corresponding integration tests, something crucial to ensure the quality of a piece of software meant to handle storage hardware.

Making Snapper work without DBus

As you may know, some YaST team members are also the main developers and maintainers of Snapper, the ultimate file-system snapshot tool for GNU/Linux systems.

Normally the snapper command line tool uses DBus to connect to snapperd which does most of the actual work. This allows non-root users to work with snapper.

There are however situations when using DBus is not possible and not being able to work in those situations was limiting Snapper’s usefulness. Now with the latest version all snapper commands support the –no-dbus option. This evolution is worth a blog post by itself… and, of course, we have it. To know all the details check this post at Snapper’s blog.

CASP (and beyond): improved roles

Do you remember the system roles feature introduced during development sprint 16 and improved in subsequent sprints? In case you don’t, let us remind you that system roles allow to define many settings of the installation just by choosing one of the offered roles. That’s only possible, of course, in products making use of that feature, like SLES.

For CASP we will have 3 different roles, as shown in the following screenshot.

CASP system roles

The main difference between these three roles is the selection of patterns to be installed. But apart from that, the Worker role will offer an extra step during installation allowing the user to specify the address of the so-called Administration Dashboard.

Configuration screen for the Worker role

That relatively small detail implied the development of a full new feature in the installer – the ability of a given role to define it’s own specific configuration, including the dialog to interact with the user. As expected from any other installation dialog, you can go back and forward without loosing the entered information. If the user goes back and selects a different role, then this additional dialog is not run again.

That new feature is, of course, not specific to CASP and could eventually be used in other products and roles. Just as a crazy example, openSUSE could decide to introduce a role called “NTP server”, running the YaST NTP server configuration right after the user selecting the role.

Other CASP related features

As already said, we have been focusing quite a lot on introducing features that are needed for CASP. It’s worth mentioning, in case it’s still unclear, that CASP will NOT ship its own adapted version of YaST. All the features introduced in the installer are in fact configurable and available for all other products as well. There is only one YaST codebase to rule them all.

Let’s briefly describe some of the introduced CASP-specific (at least for the time being) features.

CASP always uses Btrfs as filesystem for the root partition. At the end of the installation, the root btrfs subvolume will become read-only. All the other subvolumes will stay as read-write, as shown in this screenshot taken right after rebooting at the end of the installation process.

CASP subvolumes

It makes no sense to update from any existing product to CASP. Thus, CASP media should not show an “update” option when booting, even if it’s still possible for advanced users to pass the UPDATE boot parameter. Since we needed to modify the installation-images package, we took the opportunity to make the “update” option and other settings configurable in a per product basis and we unified SLES and openSUSE packages, so now they share a single branch in the source code repository.

CASP is targeted to big deployments extended all over the world. To make possible the synchronization of geographically distributed nodes, the UTC timezone is enforced in every CASP installation. Thus, we implemented support for products to enforce a given timezone in the installer. Take into account this is different from a default timezone.

Last but not least, it has already been mentioned that the CASP installation workflow will have very few steps. That also affects the screen displaying the installations settings summary. In comparison to a regular SLES, some options must disappear because they are not configurable and some other sections must be added because they are not longer presented as a separate previous step. So far, this is the appearance of the installation settings screen in the current CASP prototype.

Installation settings in CASP prototype

…and a surprise about the blog

We also prepared a Christmas gift related to the blog. The technical aspects are solved, but we are ironing out the administrative details. So you will have to wait until the next sprint report to see it in full glory. But, as the Spanish proverb says, “good things are worth waiting for”.

See you next year

That’s enough to report from our December sprint, we don’t want to bore you with every small bug fix. And talking about things that are worth waiting for, our next report will very likely be published at the beginning of February 2017.

That’s because we will put our Scrum process on hold during the Christmas session. We will restart it on the second week of the year, after the visit of the Three Wise Men. In several countries, it’s a tradition that the Three Kings bring gifts to the kids that have been nice, so let’s expect they bring us some new members for the team!