openSUSE Lizards

Yast Team

Author Archive

Highlights of YaST Development Sprint 46

November 10th, 2017 by Yast Team

It’s Hack Week time at SUSE! But before we dive into all kind of ~~crazy~~innovative experiments, let’s take a look to what we achieved during the latest development sprint.

User-friendly error messages in AutoYaST

During recent weeks, the AutoYaST version for the upcoming SLE 15 family has received quite some love regarding the integration with the new storage layer, from fixing bugs to adding some missing (and even some new) features. So let’s have a look at what we have done so far.

First of all, a new error reporting mechanism will debut in the upcoming AutoYaST version. Until now, when a problem occurred during partitioning, you got a message like “Error while configuring partitions. Try again.“. It does not help at all, right? At that point, you were on your own to find out the problem.

Now AutoYaST is able to identify and report different problems to the user in a convenient way. What is more, in many situations it is even able to point to the offending section of the AutoYaST profile.

The error reporting mechanism can distinguish between two different kind of issues: warnings and errors. When a warning is detected, a message is shown to the user but the installation will not be stopped (it honors the settings in the <reporting> section). Errors, on the other hand, will block the installation entirely.

Please, bear in mind that this error reporting mechanism is only available for the <partitioning> section. Maybe it could be extended in the future to cover other parts of the auto-installation process.

Bringing back skip lists to AutoYaST partitioning

When defining a partitioning schema, you can let AutoYaST decide which device should be used for installation. Thanks to that, you can use the same profile to install machines with, for instance, different storage devices kernel names (like /dev/sda and /dev/vda).

Needless to say that, in such a situation, we might want to influence the decision process. For example, we would like to avoid considering USB devices for installation. AutoYaST offers a feature known as skip lists which allow the user to filter out devices using properties like name, driver, size, etc.

Unfortunately, skip lists support in SLE 15 Beta1 is rather limited. But these days we have extended yast2-storage-ng to offer additional hardware information and now AutoYaST is able to use it to filter devices.

As a side effect, the ayast_probe client has been fixed to show (again) which keys you can use in your skip lists.

More on AutoYaST

Apart of adding or bringing back features, we have fixed several bugs. You can check the recent entries in the yast2-storage-ng changes file if you are interested in the details.

We know that a few features are still missing and more bugs should be addressed sooner or later, but hopefully AutoYaST must work in most use cases.

SLE15 media based upgrade for unregistered system

This sprint we also continued implementing the upgrade from SUSE Linux Enterprise (SLE) 12 products to the version 15. Particularly we solved the upgrade of unregistered systems.

In that case you need the “SLE15 Installer” medium and additionally also the “SLE 15 Packages” medium. The installer medium contains only the minimal packages for installing just a very minimal system. The rest is available either via the registration server or via the extra medium. Obviously for unregistered systems only the second option makes sense.

In this sprint we were focused on making all pieces to work together. You can see the result in the following screencast.

Upgrading an unregistered system

Fixed an installer crash in systems with 512MB RAM

We got a bug report that the beta version of the upcoming SUSE Linux Enterprise Server 15 was sometimes crashing during installation on a system with 512MB RAM. That’s bad, the 512MB is a required limit which should be enough to install a minimal system in text mode.

At first we thought that the crash was caused by insufficient memory, but the reported memory usage was OK, there was still enough free memory.

It turned out that the problem was in the pkg-bindings which tried to evaluate undefined callback function. The fix was quite simple, however, the question was why that happened only in systems with 512MB RAM and not when there was more memory.

Later we found out that the difference was caused by the extra inst-sys cleanup (mentioned in the Sprint 22 report) which YaST runs when there is low memory. In that case YaST removed the libzypp raw repository metadata cache. The assumption was that when the data is already parsed and cached in the binary solv cache the original files are not needed anymore. However, libzypp still might use some raw files later.

So we changed the inst-sys cleanup algorithm to remove only the files which we know are not needed later and keep the rest untouched.

Expert partitioner: the some boys are back in town

Several features have been brought back to the expert partitioner during this sprint.

Allow to create and delete logical volumes.
Allow to delete MD RAIDs.
Allow to work with multipath devices.

Now you can create logical volumes using the expert partitioner. When you go to the LVM overview or visit a specific volume group, a button for adding a logical volume is available. Clicking on it, you will be taken through a wizard for the creation of a logical volume. Note that although the logical volume type can be selected in the first wizard step, only normal volumes can be created. Thin logical volumes and thin pools will come soon. And apart of creating logical volumes, now there is also a button for deleting them.

Delete action has been also implemented for MD RAIDs. For that reason, you have a delete button in the RAID overview and also when you access to a specific MD RAID. And of course, you will be asked for confirmation before removing the device.

Additionally, another important feature recovered during this sprint is the possibility to work with multipath devices. Now, multipaths are listed together with other disks in the tree view of the expert partitioner, allowing you to manage them as regular disks. For example, you can create or remove partitions over them. Moreover, when a multipath device is selected, a new tab is showed to list the so-called “wires” that belong to the multipath.

Improving the product upgrade workflow

Although the possibility to offer an upgrade option from openSUSE Leap to SLE is on both SUSE and openSUSE radars for the future, the reality is that it has been, and still is, an unsupported scenario.

But with previous versions of SUSE Linux Enterprise, you could take a SLES DVD, boot it in the Upgrade mode, and select to upgrade an openSUSE partition. YaST would let you proceed several screens before telling you that it actually will not let you upgrade from openSUSE to SLES.

Starting with recent SLE15 pre-releases, the incompatible products are filtered out in the partition selector already (overridable with a Show All Partitions checkbox), letting you know earlier whether you will be able to upgrade your system to the new SLES.

Fix of a registration issue during installation process

In SLE 15 Installer, there is a product selection dialog at the very beginning of the installation. After that, you can register the selected product but you cannot change the product later as unregistering the product and registering another one is not supported. Our awesome QA squad found out that when the installation was aborted and then started again from Linuxrc without rebooting, the installer thought that the product had been already selected and did not offer any product for installation. A little fix made it work again – now we always execute the following SUSEConnect command at the start of the installer.

SUSEConnect --cleanup

That removes all traces of previous registration attempts from the Installer. This also means that you might still want to unregister directly at SUSE Customer Center if needed.

Improving help texts in the registration process

As you have seen so far, we have been working hard to polish the registration experience in many aspects and scenarios. That also includes a better communication with the user. Thus, the help text in the registration module has been improved to also include the description of the check box states. This is especially important for the “auto selected” state which is specific to this dialog and is not used anywhere else.

The help texts in YaST use an HTML subset which allows also including images. In this case we included the check box images directly from the UI stylesheet. But in the text mode we have to use text replacement instead of the images. That means the help text content must be created dynamically depending on the current UI.

Here you can see examples of both interfaces.

Twisting the storage proposal: this time for real

In our report of sprint 42 (to be precise, in the section titled “Twisting the storage proposal”) we presented our plans to make the software proposal more customizable in a per-product basis and the draft document of the new format for control.xml that would allow release managers to define the installer behavior in that regard.

Now this goes further than a mere specification and the new format is actually being used to define the partitioning proposal of both the KVM/Xen role of SUSE Linux Enterprise 15 and the upcoming SLE15-based CaaSP.

In the following screenshot you can see the corresponding step of the guided setup for the mentioned KVM/Xen role, in which the classical controls for the /home and Swap partitions have been replaced by more goal-specific volumes defined in the section of the control file describing the role.

And, as you can see below, the installer now honors those settings to propose a reasonable partition layout.

The new format and the corresponding implementation of both the logic and the UI are flexible enough to empower the release managers to define all kind of products and to make possible for everybody to create a more customized derivative of openSUSE without renouncing to the power of the automatic proposal. See another example below (not corresponding to any product or derivative planned in the short term) with more possibilities and note how the wording was automatically adapted to talk about LVM volumes instead of partitions, based on the user choice in a previous step.

Replacing ntpd with Chrony in yast2 ntp client

Chrony will replace the classical ntpd as default NTP client starting with SLE15 and openSUSE Leap 15. That will offer several advantages to system administrators and other users, as can be seen in this comparison. In order to make this replacement possible, we started a research to find out what is supported in Chrony and how to allow our users to configure it through YaST.

The research phase is now complete and we have already a plan to proceed with the adaptation of the existing yast2-ntp-client module. Also a few bits of code, which allows to set the NTP service during installation, are now in a feature branch (so not yet in Tumbleweed).

The next step will be a huge improvement (and simplification) of the YaST module, which will go further than adapting a list of options. In the screenshot below you can see the not yet finished prototype in action.

Configuring the keyboard in the installer via systemd

Originally the keyboard configuration was written directly by YaST in the corresponding Systemd-related configuration files. But we got a bug report that YaST should not touch the config directly and rather call the localectl tool for changing it. (See the details in the localectl man page).

However, this works only in the installed system, it does not work in the system installation as it needs a running Systemd that is not available during the installation process. Changing the setting for a not running system must be done using the systemd-firstboot command.

But this did not supported modifying the keyboard settings. Fortunately one of the SUSE developer helped us and implemented this feature to Systemd (pull request). Currently the feature is available in (open)SUSE packages but later it will be available in the upstream release for others.

Another related change was that YaST not only set the console keyboard but also constructed the keyboard settings for X11 (GUI). But this is actually a duplicated functionality, localectl itself includes this feature. So we have removed it from YaST and let the localectl tool to set both keyboard setting automatically.

And now for something completely different

Hack Week 0x10 (that is, the 16th edition) is starting just right now. Which means most developers of the YaST team will spend a week working on topics that may or may not have a direct and visible impact in our beloved users in the short term. But hey, maybe we will build a robot or a space rocket!

After that week, we will restart our Scrum activity. So if nothing goes wrong, you will have another update about the YaST development in approximately four weeks. Meanwhile, join us at Hack Week and let’s have a lot of fun together!

Posted in Distribution, Factory, Hackweek, Programming, Systems Management, YaST | Comments Off on Highlights of YaST Development Sprint 46

Highlights of YaST Development Sprint 45

October 25th, 2017 by Yast Team

The wait is over: finally a new YaST team report, with news about our 45th sprint!
Our team is still focused on the development of the upcoming SUSE Linux Enterprise (SLE) 15 products family and openSUSE Leap 15, which in this sprint resulted in new dialogs to select modules and extensions, changes for the multi-product medium, and fixes for issues that have been found during our development phase. So let’s check out the most interesting things that came out of our last sprint.

New Modules/Extensions Selection

SLE has a specific dialog that allows the user to select additional modules or extensions. When we first introduced this selection dialog, the extensions could have only a single dependency, which resulted in a maximum of only two levels of dependency. During this last sprint, we implemented changes to allow a chain of dependencies. You can check on the image below this new selection dialog in action:

Reliable Self Update for Multi Product medium

Until now, the self-update URL depends on the product which is ship in the medium. However, as you may know, SLE 15 product family will be shipped in a multi-product installation medium. For that reason, sometimes self-update was failing as only a single product is defined in SCC.
Now we have fixed this issue by a defining self-update identifier that is used instead of a product name, which allows the self-update feature to work in a reliable way.

Welcome screen adapted for upgrading

Some sprints ago we announced the addition of product selection to the initial screen for LeanOS installer.

The welcome dialog is shared between different workflows like between installation and upgrading. The problem is that for upgrading we need to find the target system or root partition before selecting the product to migrate. Now it does not require any selection if there are no products to select, so it will work when upgrading. Besides that, we have polished some presentation details like the dialog title and the product selector caption.

Check out the screenshots below to see the final result:

LeanOS:

Tumbleweed:

Unavailable Packages in AutoYaST

AutoYaST needs to make sure that, after rebooting into the 2nd stage (when needed), the user can access to the installation process using the same tools that he/she used during the 1st installation stage. Apart AutoYaST packages it self, it may need to install other additional tools like VNC, SSH or the X.org system.

Unfortunately, as SLE 15 is split into modules, it’s not guaranteed that the VNC, SSH or X.org packages can be installed, which resulted in AutoYaST failing when trying to install those packages.

We have improved the package handling and now YaST displays a warning that the packages are missing and the system (and later the installer) could not be accessed as expected. However, the AutoYaST installation can still proceed although you cannot watch AutoYaST running during 2nd stage.

Improved Handling of Multi-repository Media

We got some bug reports about the new multi-repository media handling in YaST (mentioned in the sprint 43 report). Some of the problems were delegated to the underlying libzypp library, but we got our share of real YaST issues.

One of those problems was, for instance, the inconsistent styling of the MultiSelectionBox widget used in that dialog, which was pretty confusing. Fortunately, the issue has been fixed and now it looks the same than any standard checkbox widget.

More to come

The 46th sprint has already started and has many new items planned to be developed, especially for the SLE15 and openSUSE Leap 15 installer. We are looking forward to bring these planned features to life and tell you about all the details of this sprint. Meanwhile, have fun and stay tuned!

Posted in Distribution, Programming, Systems Management, YaST | Comments Off on Highlights of YaST Development Sprint 45

Highlights of YaST Development Sprint 44

October 11th, 2017 by Yast Team

Here is the YaST team again with a new report from the trenches, this time with a small delay over the usual two weeks. Most of the team keeps focused on the development of the upcoming SUSE Linux Enterprise 15 products family, including openSUSE Leap 15. That means finishing and polishing the new storage stack, implementing the new rich ecosystem of products, modules, extensions and roles (one of the biggest highlights of the SLE15 family) and much more. So let’s dive into the most interesting bits coming out of the sprint.

Hostname configuration during installation

And let’s start with one of those stories that illustrate the complexity hidden below the user-oriented YaST surface. During installation is very common to assign a hostname to the machine being installed to identify it clearly and unequivocally in the future.

Usually it is a fixed hostname (stored in /etc/hostname) but in some circumstances is preferable to set it dynamically by DHCP. Since some time ago (as you read in a previous report and is shown in the image below) YaST allows to set the hostname selecting a concrete interface or with a system-wide variable named DHCLIENT_SET_HOSTNAME which is defined in /etc/sysconfig/network/dhcp. The value to be set for such variable during installation can be optionally read from the distribution control file. Last but not least, as you already know, Linuxrc can also be used to enforce a particular network configuration.

Most users have a simple setup that works flawlessly, but we recently got a bug report about a wrong network configuration after installing the system if the hostname configuration was set via Linuxrc. After some research we found that the value of DHCLIENT_SET_HOSTNAME coming from the control file was overwriting the Linuxrc configuration at the end of the installation. Now it’s fixed and the global variable will be set by the linuxrc sethostname option if provided or loaded from the control file if not. And all that happens now at the very beginning of the installation to give the user to chance to modify it and to ensure the user’s choice is respected at the end.

Take into account that with multiple DHCP interfaces the resulting value for DHCLIENT_SET_HOSTNAME is not fully predictable. Hence, in that scenario we recommend to explicitly select the interface which is expected/allowed to modify the hostname.

Extending the installation process via RPM packages

As we have mentioned (a couple of times) during latest reports, we are implementing multi-product support for the installer. It means that SUSE will ship several products on a single installation media.

One interesting feature is that products, modules and extensions can define its own installation roles. For instance, if you select the desktop extension, you will be able to select GNOME as system role.

During this sprint, we have improved roles definitions handling, displaying a different list of roles depending on which product was selected.

As a side effect, we added support for sorting roles assigning them a display order.

Getting Release Notes from the Installation Repository

As part of our effort to drop SUSE tags from the installation media, we improved the way in which release notes are handled during installation.

Release notes are downloaded from openSUSE or SUSE websites in order to show always the latest version. Of course, the installation media includes a copy of them, which may be outdated, to be used when there is no network connection.

From now on, instead using some additional files, this offline copy of release notes will be retrieved from the release-notes package which lives in the packages repository. So we do not need to ship additional files containing release notes in the installation media anymore.

Moreover, although the old approach worked just fine in almost all cases, there was an uncovered scenario. Let’s consider a system which have access to an updated packages repository but is not connected to Internet. That could be the case, for instance, if you are using SUSE Subscription Management Tool (SMT). With the new approach, the installer will get release notes from that repository instead of displaying the (potentially outdated) ones included in the installation media.

Additionally, we refactored and clean-up a lot of old code, improving also test coverage.

Storage reimplementation: bringing more features back

We are also working hard to make sure the brand new yast2-storage-ng includes all the features from yast2-storage, in addition to the new ones. That means that, after this 44th sprint, SLE15 is already able to perform the following operations using the new module.

Creating MD software RAID devices in the expert partitioner. This feature is specially relevant for many openQA tests that rely on it.
Displaying the compact description of the partitioning proposal in the one-click-installer screen used by SUSE CaaSP and openSUSE Kubic
Importing users and SSH system keys from a previous (open)SUSE installation.

Rethinking LVM thin provisioning

When trying to create a thin-pool using all free space the metadata has to be accounted for. In contrast to linear LVs the metadata for thin-pool uses space of the VG. For instance, if there are 2048 GiB free in the VG, the metadata for a maximal size thin-pool is about 128 MiB and the pool can be about 2047.9 GiB big.

Additionally LVM creates a spare metadata with the same size. This spare metadata is shared between all pools and thus has the size of the biggest pool metadata. The spare metadata can be deleted manually and all pool metadata can also be resized.

When starting with an empty VG it is relative easy to account for the metadata. But how to handle this with an already existing volume group? Also take into account a volume group containing e.g. RAID LVs or cache pools (which also have metadata).

We finally decided that, during probing, YaST will check how much free space the VG has and then it will calculate “reserved” value for the volume group:

reserved = total size - used by LVs the library handles - free

So when calculating available space for a normal or thin pool, it will take the “reserved” into account:

max size = total size - reserved - used by LVs the library handles

The only drawback is that the maximal size for the pool can be smaller than actually possible since e.g. the spare metadata might be shared with an already existing thin pool.

More to come

The 45th sprint has already started and you can expect more and more work in the installer for SLE15 and openSUSE Leap 15 and more news regarding the revamped storage stack. Meanwhile, don’t forget to have a lot of fun!

Posted in Uncategorized | Comments Off on Highlights of YaST Development Sprint 44

Highlights of YaST Development Sprint 43

September 21st, 2017 by Yast Team

The summer is about to end (in Europe) and it is time for another YaST Development Sprint report. As usual, storage-ng has been one of the stars of the show, but new installer features for upcoming SUSE/openSUSE versions have received a lot of attention too. Also CaaSP 2.0 got some love from us during this sprint.

storage-ng: Udev mapping and ARM64 support

The new storage layer is getting better everyday. After the big amount of work that came with the re-implementation, the team is trying to unleash the power of the new design.

Udev Mapping is Back

The bootloader module supports using persistent device names provided by Udev. It is a pretty useful feature that comes in handy in many situations but it was missing in the storage-ng based version of the module.

But fear not: this feature has been re-implemented taking advantage of the much improved API of the new storage layer. And that’s not all: the team also took this opportunity to clean up some code and document the strategy for picking Udev device names in a proper way.

Do you want more details? Here you have them. Let’s start with the scenarios we support:

S1: Disk with the booting configuration is moved to different machine
S2: Disk dies and its content is loaded to new disk from a backup (because you have a backup, right?)
S3: Path to disk breaks and is moved to different one

Given these scenarios, let’s have a look at the strategies:

If the device has a filesystem with its mount_by, do not change it.
If the device names includes a by-label, just use it. This behaviour enables us to handle the three scenarios.
If there is by-uuid, then use it. It can also handle the three scenarios.
If there is by-id, use it. It can handle S3, but not always.
If there is by-path, just use it. It is the last supported Udev symlink and, at least, it will prevent the name changing during boot.
As fallback, just use the kernel name (for instance, /dev/sda3).

storage-ng now works also on ARM64

For quite some time, the storage-ng code has been tested (and worked) on x86_64, ppc64 and s390x architectures. Now, we have added aarch64 to match the list of supported architectures in SUSE Linux Enterprise and openSUSE Leap.

Speeding up the Service Manager Startup

As you may know, YaST includes a nice module for managing systemd services. Compared to systemadm, included in the system-ui package, there are some key differences:

It displays only services, not other unit types such as sockets.
It can enable/disable them for the next boot.
It works, as any YaST module, in textmode (even on 80×25 terminals).

Some time ago we got a report that the presented information was inaccurate in some corner cases. We fixed that but also made the module a lot slower at the same time. It did not take long for that to be reported.

We tested the following scenario: 286 services (SUSE Linux Enterprise 12 SP3 with nearly all software patterns installed) on a not very fast virtual machine. Normally, you should have fewer services and probably a faster system, but we wanted to fix the issue even for the worst scenario.

After analyzing the problem, we found out the root cause: too many calls of systemctl, at least 3 times per service (show, is-active, is-enabled). With a couple dozen milliseconds per call, it quickly adds up.

The fix was to combine all show calls into one and correctly interpreting the ActiveState property to eliminate all of the is-active calls. But you want to know the numbers, right? After the fix, the startup time went down from 69 to 15 seconds (bear in mind that it is a slow virtual machine).

So even if your system is not that slow or you have fewer services installed, you may benefit of a shorter startup time for this module.

Multi Repository Media

There are some hidden gems in YaST that are maybe not well known although they have been there for a long time. One of those features is support for multi repository media.

What actually is a multi repository medium? Imagine a CD or DVD with several independent repositories. The advantage is that, if you want to release several add-ons, you can put them all on a single DVD medium. Really nice stuff, isn’t it?

Up to now YaST added all repositories found on the medium automatically without any user interaction. In this sprint we have added a new dialog into the workflow which asks the user to select which repositories should be used. Of course, if there is only one repository, it does not make sense to ask and the repository is added automatically.

Selecting which add-ons should be installed

i18n support for CaaSP 2.0

On June 2017, SUSE released the first version of the promising SUSE CaaS (Container as a Service) Platform. The YaST team actively worked on this project by adding several new features to the installer, like the one-dialog installation screen.

That very first version of CaaSP was only available in English. However, version 2.0, which is around the corner, will support more languages. For the YaST team, it means that we needed to add the language selector to the
installer, as you can see in the screenshot below, and to mark every string in the yast2-caasp for translation.

YaST2 CaaSP features a language selector

Finally, if you are interested in CaaSP, maybe you would like to check out Kubic, its Tumbleweed based variant.

More bug fixes

Apart from the new and shiny stuff, the YaST team was able to fix quite some issues during this sprint. Let’s have a look at some of them.

Taking Care of Small Details

Usability is a critical point for a project like YaST. From time to time, we receive a bug report about some usability problem that needs to be addressed and we took them very serious. In this case, the bootloader module had a problem when showing long device names in the dialog to change the order of disks. So we just needed to do a minor fix in order to ensure that there is enough space.

Please, keep reporting usability related issue you find in order to make YaST even better.

Fixing the INSECURE mode

It sounds scary, but YaST supports an insecure mode during installation. What does it mean? YaST is pretty flexible when it comes to system installation. You, as a user, has the power to modify/tweak the installer (using a Driver Update Disk), add custom repositories, etc. And sometimes it can be useful to skip some checks.

Some time ago, libzypp introduced a new callback to inform about bad (or missing) GPG signatures. This callback was properly handled by AutoYaST but it was ignored in regular installations, so the user always got the warning about the failing signature, even when running on insecure mode.

Now the problem has been fixed and you can run the installer in insecure mode if you want to do so.

Learning about FCOE

YaST deals with a lot of moving parts and, although it can be daunting for the newcomer, it also has a bright side: we regularly learn new stuff to play with.

In this case, Martin Vidner, one of our engineers, had to deal with a fix related to the Fibre Channel over Ethernet support in YaST. But instead of blindly applying the patch that we already had, Martin decided to learn more about the topic sharing his findings in its own blog. Sure it will be a valuable resource to check in the future.

Posted in Uncategorized | Comments Off on Highlights of YaST Development Sprint 43

Highlights of YaST Development Sprint 42

September 7th, 2017 by Yast Team

Kids started school in Prague, and we’re energized to bring you news from the YaST team.

This sprint we made progress in:

selecting one of several products to present on a DVD
adapting to the switch to the rpm-md package metadata format, regarding licenses and the Beta notice
notifying about not being able to support ReiserFS also in the case of an upgrade via AutoYaST
not reporting missing optional patterns

As always, the new storage stack deserves a separate section.

Expert partitioner has a better initial summary and it handles Btrfs subvolumes better.
Snapper can create and restore snapshots now.
A more flexible installation proposal is being developed.
It has landed in SLE15/Leap15 already.

Selecting the base product in the first screen

As we reported three sprints ago, YaST will support having multiple products on the same installation medium. This feature will allow to ship several SUSE products on the same DVD asking the user which one to install.

But, as an (open)SUSE user, you may know that the first screen of the installer allows you to select the language/keyboard and, additionally, it shows the product’s license. But which license should we show for multi-product media?

After asking our UX expert, we decided to allow the selection of the product in the welcome screen as you can see in the image below.

Obviously, this behavior applies only to multi-product medias. When using a single product one, the license is shown directly in the welcome screen.

Finally, as developers, we would like to highlight that we mainly re-implemented the welcome screen using modern YaST techniques (like our object-oriented API to YaST widgets), improving test coverage and code quality.

Getting licenses from repositories

In the sprint 40 report, we announced that YaST was dropping support for SUSE tags because the plan is to use RPM metadata and packages to store all that information (licenses, release notes, etc.).

During this sprint (and the previous one) we focused on improving how base product licenses are handled. Until now, licenses lived in a tarball which was included in the installation media. But that is not the case anymore: now YaST relies on (the awesome) libzypp to get products licenses directly from the repositories. There are still some rough edges: for instance, multi-language support should be improved, but we will tackle them pretty soon.

Finally, bear in mind that only base product licenses have been adapted, but we plan to do basically the same for modules, extensions and add-ons.

Updated README.BETA Support

The YaST installer supports displaying the README.BETA file from the installation medium. This file is added during Beta phase so the users know this is not the final release. This is also useful after the final product is released if you by mistake boot the old medium.

Unfortunately the original YaST code supported only the so called SUSE-Tags repository format which was used for the CD/DVD media. However, the new SUSE Linux Enterprise media will use the RPM-MD format which is something completely different.

In this sprint we have added the support for the RPM-MD format and additionally we display the Beta warning popup also in the AutoYaST installations. Of course, blocking AutoYaST with a popup would be a bad idea so in the AutoYaST mode the popup is automatically closed after a timeout. (The default is 10 seconds but can be configured in the XML profile.)

Drop support for ReiserFS autoupgrade

The drop of ReiserFS support for manual installation was announced in our last post, and now is the turn to autoupgrades.

With SUSE Linux Enterprise 15 the autoupgrade will be blocked in case of ReiserFS presence were detected suggesting a manual conversion to another filesystem.

Software proposal does not report missing optional patterns

YaST guides you through the installation process of your system making you a proposal based on the different selections like the product, addons, system role, desktop etc.

This proposal contains some software patterns that are mandatory and some that are optional but only the mandatory ones will be reported when the proposal is shown.

During this sprint we solved a bug which reported not only missing mandatory patterns but also optional ones.

The Storage Stack (storage-ng)

Storage reimplementation: Expert Partitioner

As you probably already know, in the YaST team we are rewriting our powerful expert partitioner tool from scratch to adapt it to the new storage layer. Sprint by sprint we are bringing back some of the many great features the expert partitioner offers, and this time it will not be different.

Now, when entering to the expert partitioner, all available storage devices are presented in an initial summary. Moreover, in this first screen you could find an option to rescan your devices, which allows the expert partitioner to be aware about the changes in your system, for example when you plug in a USB stick.

The management of Btrfs subvolumes was also improved during this sprint. Now, you will be alerted when a new subvolume is shadowed by an existing mount point. Moreover, some subvolumes could be automatically deleted or created when a mount point changes in the system.

Snapper can snap again

We reintroduced one of the most important features about using Btrfs: Being able to create snapshots, so you can go back to a previous state of the system if anything went wrong during a package upgrade or when you changed anything about your system configuration. This means we are now again setting up and configuring snapper correctly, installing everything into a subvolume and creating an initial snapshot when the installation is complete.

Why install into a subvolume in the first place? When snapshots are made in the future (e.g., during package upgrade or installation) and at some point you decide to roll back to one of those snapshots, you might want to delete previous snapshots to save disk space. If we didn’t install into a subvolume, the initial files would always be left over and consume disk space, so there would be a considerable resource leak.

Twisting the storage proposal

As you should know, one important part of the rewritten storage stack is the partitioning proposal. The old one was designed with a rather narrow scope in mind. Targeting desktops and old-school servers, it always proposes a root file-system, a swap volume and an optional separate /home. That’s not the most useful schema for new innovative products with a different focus like SUSE CaaS Platform, openSUSE Kubic, SUSE Manager, etc.

As we remind quite often, all the products supported by YaST (SLES, SLED, Leap, Tumbleweed, Kubic, CaaSP, SUSE Manager, SLES4SAP, you name it) share an identical installer. That installer is fully configurable using the file control.xml that is included in the media and that allows to define the sequence of installer steps, the default values and much more.

One of the goals of the new storage proposal is to give product creators and release managers more freedom and flexibility configuring the behavior of the guided setup. And for that we need a better format for control.xml, so they can express themselves with fewer limitations.

Of course, that new format is not something for the YaST team do decide on its own, but something being designed openly in collaboration with all the involved parties. To make easier for anyone to jump into the subject we have prepared this detailed document which includes all the present and historical information to understand the topic, as well as an explanation of the new format with several examples based on existing or hypothetical use cases.

As everything in YaST, that document is alive and expected to continue changing and evolving based on everybody’s feedback and contributions. So feel free to take a look and suggest improvements or future use cases we may have overlooked.

The storage stack is dead, long live the storage stack

In our previous sprint report we told you we were working to integrate the new storage stack into the future SLE15 and openSUSE Leap 15 codebase. Now the submission process is over and the preliminary images of both future distributions are fully based on libstorage-ng. For openSUSE Tumbleweed, that is the Staging:E project. That means we now have many more eyes looking into it, finding bugs, pointing what is missing and providing feedback about the new behavior. Of course, every couple of eyes comes with a little bit more of pressure for the YaST Team to get things done as soon as possible, but also with a pair of hands to help us getting there.

Posted in Systems Management, YaST | 2 Comments »

Highlights of YaST development sprint 41

August 24th, 2017 by Yast Team

We all know that everything slows down in summertime and software development is not an exception. But heat is not enough to stop the YaST team from turning the Scrum wheel and delivering the corresponding sprint reports. Let’s take a look to what we have been doing the last two weeks.

The storage reimplementation gets on the launchpad

As already anticipated in the previous report, one of the goals of this sprint was to merge the new storage stack into the codebase of SUSE Linux Enterprise 15 and openSUSE Leap 15. That implies submitting everything to Factory first and making sure the result looks harmless and good enough there. Thanks to the awesome openSUSE tools and processes, that kind of experiments can be isolated in a dedicated staging project allowing us to reach useful conclusions without risking the stability and features of Tumbleweed.

So we submitted two new source packages libstorage-ng and yast2-storage-ng to Factory, together with new versions of all the affected packages (already adapted to use the new system, instead of the old yast2-storage) and a modified version of the list of packages to be used during installation.

Everything was mixed and cooked in the Staging:E project and… guess what! We got brand new Factory ISOs with storage-ng, successfully building and verified to work by openQA, as you can see in this screenshot of the Staging Dashboard.

Yes, we know there are two failing tests in that dashboard, but that was fully expected since those tests use the expert partitioner to configure an installation of openSUSE on top of a MD RAID system and the reimplemented partitioner still lacks some controls to configure MD RAID arrays.

The new stack will live in Factory:Staging:E (or any other staging project the Tumbleweed crew decides) for quite some time, until it’s feature-pair with the old storage layer and, thus, can progress further in its travel to Tumbleweed. But Factory was just the first stop, the ultimate goal of this sprint was getting into the preliminary versions of the next SLE and openSUSE Leap.

That second integration is taking a little bit longer because it has coincided on time with other important changes in the installer and the base system… and the fact that August is the typical European vacation period is not exactly helping to iron all the details out. But since the new storage system works for Factory, we are certain it will do it for SUSE Linux and Leap.

As readers familiar with the Tumbleweed development process may have noticed already, having all those packages in Staging:E implies that newer versions of them will only reach Tumbleweed all at once, when yast2-storage-ng is considered mature enough for that. Somehow, that will block us from delivering new features for the packages you see in the list in the mentioned image of the dashboard. But don’t worry, if something serious happens and a critical update is needed we will not let our beloved Tumbleweed users down.

But there is much more happening in YaSTland beyond the storage reimplementation. Let’s take a look to the improvements in other areas.

Installation without Grub packages

Sometimes, users have already Linux installed in their system and they do not want to install Grub in MBR again with a new Linux distribution since the installed Linux can manage the bootloader. For this case, the user may decide to not install grub packages at all in the system. However, until now the user was obligated to install this package otherwise an error message would appear, as the image below shows.

For some specific scenarios, as you may find here, even other packages are required, and when the user decided for not installing the bootloader, these packages were still required for the installation.

We changed this behavior in Tumbleweed and SLE 15, and now the users will be able to install the system without the packages that are not required, in case they decide to manage bootloader through another operational system.

But that’s not the only improvement introduced in the bootloader management during this sprint.

Improve how YaST finds disk to install Grub in MBR

In Leap 42.3 and SLE 12.3, we found out that, in some very specific cases (check the bug report for more details), YaST was not finding the correct disk to install Grub in MBR. When it happened, an error message appeared at the end of the installation, showing that Grub could not be installed in /dev/btrfs disk.

We improved our approach to finding the correct MBR device, by adding a specific search for the disk where the partition /boot or / (in case /boot does not exist) is located.

Such a change will be released as maintenance update and self-update, and it affects only Leap 42.3 and SLE 12.3, since SLE 15 will use the new storage layer, which does not need this double check for the correct disk.

And talking about the new storage system…

Remove support for ReiserFS

The support of new installations with ReiserFS was removed from YaST in SUSE Linux Enterprise 12 and openSUSE Leap 42 but upgrades were still supported.

With SUSE Linux Enterprise 15 and Leap 15 the support of ReiserFS will be completely removed from YaST and the installer will block the upgrade of systems formatted with ReiserFS.

If some of the entries in the /etc/fstab file of the system to be upgraded is using ReiserFS, the installer will suggest to convert them to another filesystem type before migrating the system to SUSE Linux Enterprise 15 or openSUSE Leap 15.

A similar blocking error will be reported for ReiserFS root partitions.

Another Ruby 2.4 fix

This may be interesting for Ruby developers in general. We got a bug report about crashing YaST which in the end turned out to be caused by upgrade to Ruby 2.4. The tricky part was that YaST crashed randomly and it was difficult to reproduce the problem.

It turned out that the crash happened when Ruby wanted to print a warning on the error output, which in some situations failed. We did not fix the race condition, as it likely would be too difficult to debug the Ruby internals, but we at least fixed the code to not produce the warnings anymore.

So if you are a Ruby developer take this free advise from your YaST fellows – if your code crashes randomly with Ruby 2.4 then check for the Ruby warnings first.

A heads-up about network devices names

Two sprints ago we told you about the new possibility of configuring the network with AutoYaST already in the first stage, avoiding an extra restart of the system in most cases.

During this sprint we spent some time trying to test old AutoYaST profiles (with complex network configurations) with the upcoming version of SUSE Linux Enterprise Server, using our suite of automatic AutoYaST Integration Tests. But we found some issues caused by the current architecture of our test suite that may be of interest for some of our readers.

Let’s see some technical background first.

Tumbleweed has been using ‘predictable network interface names’ for some time now and it fits most regular use cases. Inspired or following the scheme idea introduced by ‘biosdevname’, Predictable Network Interface Names was adopted in systemd/udev v197 trying to solve an historical problem with the non deterministic classic naming scheme for network interfaces (eth0, eth1, eth2 …)

Basically it will assign fixed names based on firmware, topology, and location information making them stable between system reboots, hardware additions or removals and also between kernel or drivers updates.

For the upcoming SLE15, we are giving predictable network interface names a try (they are disabled in SLE12 and openSUSE Leap 42.x). For us that turned to be a problem because our AutoYaST testsuite dynamically creates new virtual machines on every system reboot (instead of really rebooting the virtual machine created in the previous step). So from the point of view of the operating system being tested, all the network devices are replaced by new ones in every reboot and that drives the network settings nuts.

That was only our case (arguably “our fault”), but there might be other situations in which going back to the old naming scheme (with names like ‘eth0’) would be more convenient than adapting the preexisting AutoYaST profiles to the new one. In such cases you still can use the old scheme (not fully predictable but very well known by Linux veterans) by just booting the SLE15 installation with this parameters.

biosdevname=0 net.ifnames=0

More to come

In addition to everything reported in this post, we have been working hard to get some new cool features to the upcoming SLE15 and to get the storage reimplementation full-featured enough to substitute the old one in all possible situations.

So, although it would still be summertime (in Europe), stay tuned for more news in two weeks.

Posted in Distribution, Factory, Programming, Systems Management, YaST | 3 Comments »

Highlights of YaST development sprint 40

August 10th, 2017 by Yast Team

Doubtlessly, these are pretty exciting times for the YaST team. The merge of the new storage layer into the main codebase is around the corner and we are working on other features that will debut on the next open(SUSE) major release. So let’s summarize what happened during the last sprint.

New storage layer is coming

As you may already know, the YaST team has invested a lot of time and effort preparing our storage layer for the future and we have started to merge the new layer into the main code base during the current sprint. But that’s something for our next report, right? By now, we will just focus on the stuff that got added and fixed during the last two weeks.

Storage reimplementation: BIOS RAID support

libstorage-ng, the low level library in which our new storage layer relies on, got support for BIOS RAID (handled in Linux via MD devices). Now, YaST could take advantage of such a feature to allow the installation of open(SUSE) systems on this kind of devices, including the bootloader.

Storage reimplementation: managing BtrFS subvolumes in new Expert Partitioner

The new Expert Partitioner is getting a lot of attention these days and, during sprint 40, it got initial support for Btrfs subvolumes management.

Now, when you select the BtrFS section in the general menu placed on the left, all BtrFS filesystems are presented allowing you to edit its subvolumes through a dialog which contains the list of subvolumes that belongs to the filesystem. Apart from the usual stuff, like adding and deleting subvolumes, it is also possible to set the noCoW property when you are creating a new one.

However, some features are still missing. For instance the partitioner will not prevent you to create a subvolume which is shadowed by an already existing mount point. Consider the current implementation as the first step towards a really cool Btrfs subvolume handling.

Dropping SUSE tags support

During installation, YaST uses a mechanism known as SUSE tags as source of information for media handling. For instance, a /content file contains information about the product, languages, etc. Additionally, information like release notes or the slide-show texts are stored in the installation media.

Some time ago, SUSE decided to drop SUSE tags and use RPM metadata and packages to store all that information. To make it possible, the installation media would use REPOMD repositories.

Obviously, YaST needs some adaptation. As a first step, support for the /content has been dropped, cleaning up some old and even unused code.

In the upcoming sprints, YaST will be adapted to retrieve licensing, release notes, etc. from RPM repositories and packages, which is also an opportunity to do some refactoring and to improve test coverage.

AutoYaST support for add-on products on same installation media

Nowadays YaST supports having add-on products on the same media than the base product. The problem is that the EULA for those products is displayed too early, even before AutoYaST had been initialized at all.

To solve this issue, now the EULA acceptance of included add-ons is performed at the same time than other add-ons which are not included in the installation media. As a side effect, now the user needs to define those add-ons on the AutoYaST profile in order to handle the EULA acceptance.

Bug squashing and 80×24 terminals

As developers, we enjoy working on new features and, of course, we are committed to fix critical bugs as fast as possible. But there are many small (and annoying) bugs out there that deserve our attention. Additionally, there are several bug reports that are no longer valid (the bug was fixed, it is not reproducible, it is a duplicate, the affected product is not supported anymore, etc.). In order to reduce the list of open issues, the team decided some sprints ago to reserve one day to do some bug squashing.

Among the bugs we closed during this sprint, we would like to highlight a usability problem in YaST services manager. Bear in mind that, along with the graphical interface, YaST ships a text based one which is supposed to fit in good old-fashioned 80×24 terminals. That’s an interesting constraint when you are designing interfaces for YaST.

Needlessly to say that, from time to time, we get a bug report about some element that just do not fit. In this case, YaST services manager had a problem when the service name was too long as you can see in the screenshot below.

Now, if there is not enough space, the name will be truncated and the rest of the information will be shown in an proper way.

Do not miss the next report!

As you may have noticed, a lot of interesting things are currently happening in the YaST world and more cool stuff is about to come. So you should not miss our next sprint report.

By now, enjoy openSUSE 42.3 (you already upgraded your system, right?) and see you in two weeks.

Posted in Distribution, Factory, Programming, Systems Management, Usability, YaST | Comments Off on Highlights of YaST development sprint 40

Highlights of YaST development sprint 39

July 31st, 2017 by Yast Team

openSUSE 42.3 is out! Do you need some reading material while you wait for the download of the new release to finish? Don’t worry, we have the solution right here – another YaST team report. 😉

Several products in one installation medium

Obviously, we stopped adding new features to SLE12-SP3 and Leap 42.3 some days ago, because everything needed to be tested properly before the release. So now we are mainly looking into the future. And one of the plans for that future regarding SUSE Linux Enterprise (SLE) is offering several products packed in a single installation medium.

SUSE offers several mission-specific products based on SLE and, so far, every product needs to be installed from its own medium (usually a DVD or a virtual image). So if you use SLE Server, SLE Desktop and SLE Server for SAP, you have to have three DVDs which is a bit cumbersome.

After some discussions about the technical implementation details, we created a first prototype of the installer with an extra dialog that allows to select one of the detected products and continues the installation from there according to the installation workflow of the chosen product. It is still a proof of concept, but we can at least share screenshots showing how it looks for the time being.

So far, there are no plans to use the new feature in openSUSE, mainly because the project does not deliver separate mission-specific products in the same way than SUSE does with SUSE Linux Enterprise.

Storage reimplementation: numbers after repatriating the Expert Partitioner

In the Sprint 36 report we presented the rewrite of the YaST Partitioner and we have been informing about its evolution in subsequent reports. We told you back then that we decided to split it in a separate yast2-partitioner package. But time has proved that decision to have too many drawbacks so we decided to bring the Partitioner back home, to yast2-storage-ng. As part of the process, we got rid of an old previous prototype of the Partitioner that was still lying in the yast2-storage-ng repository and some code that was there just to support that old prototype.

You may be asking why is all that relevant. It is because that means the repository (and thus the package) is finally approaching the final structure it will have when released into Tumbleweed. And that implies that all the systems we use to automatically measure the quality and reliability of our repositories are now providing trustworthy results… as trustworthy as automatic quality evaluations can be.

And according to those tools:

93% of the code in yast2-storage-ng is covered with automated unit tests in addition to openQA (this number is expected to raise in close future as we polish the new Partitioner),
Code Climate reports a code quality GPA of 3.91 out of a possible maximum of 4
and 76% of all the classes, modules and functions, including the internal ones, are properly documented (with that developer documentation being available here, by the way).

If you wonder about the numbers for the old codebase we want to replace, its code quality is 0.94 and only a 31% of it is covered by unit tests. A perfect example of legacy code.

Storage reimplementation: Btrfs subvolumes in AutoYaST

As reported in previous posts, the new storage stack can already process AutoYaST profiles including partitions, LVM and MD arrays, but some details are still missing. The first of those details we wanted to address was the definition and creation of subvolumes in a Btrfs file-system.

Now it works according to the official documentation – both syntaxes for the <subvolumes> section are supported, it never creates subvolumes that would be shadowed by any other mounted file-system and it uses the list in control.xml as fallback for the root partition if Btrfs is used but subvolumes are not specified.

All that, as usual in the re-implemented stack, with fully tested and documented code.

Storage reimplementation: handle multipath I/O in the proposal

In the previous report we showed you how the support for Multipath I/O looked at the library level, which usually means just geeky graphs. During this sprint, we have taught the installer to use that new library feature, so we now have real screenshots to show!

During the installation, now multipath hardware is detected and the user is asked for activation.

If the user agrees, the installer will never use the individual disk devices to propose a partitioning layout and it will not offer them as an option during the guided setup. The installer always works on the final (compound) multipath device, proposing the correct names for the partitions and so on (which, being a devicemapper device, follows a
different pattern when compared to raw devices).

The resulting system is still not fully bootable because yast2-bootloader has still not been adapted to this scenario. Very likely, something for the upcoming sprint, so stay tuned.

Support for Ruby 2.4

The world changes every day and we are always adapting YaST to remain shiny. The 2.4 version of Ruby is about to land in openSUSE Tumbleweed and is expected to be the default Ruby for SUSE Linux Enterprise 15 and openSUSE Leap 15. We found that some of the YaST packages were not fully ready for this new Ruby, so it was time for some tweaking.

After dealing with quite some details too technical and boring for this blog (but feel free to ask if you want the gory bits), YaST is shining again in Factory, which means we are no longer blocking the adoption of Ruby 2.4 in Tumbleweed.

Add-on Creator and Product Creator

As our team keeps always developing new features, solving bugs, and receiving feedback, we always evaluate our priorities and product. Sometimes, during this evaluation, we see that some YaST modules do not bring enough value or do not shine enough as part of our standard package.

After some evaluation, we come to announce that the modules Add-on Creator and Product Creator will no longer be part of YaST. These packages use Kiwi as backend and we have high competition on UI sides – SUSE Studio and Open Build Service. So it no longer makes sense to have these packages and we recommend for users of these modules to use one of the alternatives or Kiwi directly if you already have an XML definition file for Kiwi.

Adapting YaST to accept 12 digits Service Request Number

As Service Request Numbers can be now composed of 11 or 12 digits, instead of only 11 digits as before, we had to adapt YaST to handle this change. YaST module Support can now accept 11 or 12 digit service request numbers. We implemented such a change for all products dating back from SLE 10 SP3 until the most recent SUSE Linux versions. Updates with this change will be soon released.

Network Setup in the 1st Stage of Autoinstallation

The YaST installation used to have two stages, separated by a reboot. Starting with SLE 12 and openSUSE Leap 42.1 we have eliminated the
second stage. But it was still needed for AutoYaST, controlled by the setting

<profile><general><mode><second_stage>true | false</...>

We have fixed those parts of the networking setup and now you can explicitly set AutoYaST to not use a second stage anymore.

User settings in AutoYaST

An issue that we have found out is that GDM has problems with the system when different users have the same UIDs. If it happens, GDM does not start properly. As a solution, we decided that either UIDs will be defined in the AutoYaST configuration file for ALL users or this tag will not be used at all for ALL users since a mix of both can result into duplicate UIDs.

And we just keep YaSTing!

We hope you liked our report as much as we loved to build all of that. We’ll continue YaSTing so we can reach you again in two weeks with much more cool stuff to show.

By now, enjoy your openSUSE 42.3 and all the cool features that came with it!

Posted in Distribution, Factory, Packaging, Programming, Systems Management, YaST | 3 Comments »

Highlights of YaST development sprint 38

July 14th, 2017 by Yast Team

Here we go again with a new report from the YaST trenches. This time with the storage reimplementation as the clear star of the show.

Storage reimplementation: the proposal adapts, you succeed

As we have announced in our previous sprints and as you probably already know, the YaST team is working hard to rewrite the whole storage stack on time for SLE15 and openSUSE Leap 15. As part of this reimplementation we have designed a brand new storage proposal that automagically offers the user the best combination of partitions and LVM volumes based on the current configuration of the system and the user preferences.

When we are working with very small disks or with special technologies like DASD (which doesn’t accept more than three partitions by device), the storage proposal might not be able to generate a valid initial proposal honoring the initial requirements of the product (e.g. creating a separate home partition and enabling btrfs snapshots for the root partition in the openSUSE Leap case). Now the proposal is not limited to fail when it is not possible to satisfy the default product requirements. Before giving up, the new system looks for alternatives, like discarding the separate home partition or disabling snapshots. Moreover, now the proposal is able to automatically adjust the size requirements not only for root, but also for swap and home. And, of course, the guided setup continues there for fine tuning the proposal settings.

Desktop selection improvements

As our usual readers also know, we recently introduced a more fair desktop selection screen for openSUSE, both Leap 42.3 and Tumbleweed. We used part of the latest sprint to implement some feedback we gathered about the wording and behavior of that dialog.

That feedback gathering included some discussions on how to make user experience nicer after selecting one of the user interfaces available through the “custom” option. As a result, the awesome openSUSE crew created a new mechanism for selecting the default window manager on each graphical login, so YaST can delegate the details to the maintainers of those alternative interfaces.

How everything works now? Glad you asked. 🙂

If the user select KDE or GNOME in the YaST dialog, /etc/sysconfig/windowmanager is configured to point to that desktop by default. If the “custom” option is selected, then YaST does not enforce any interface in that file and the new mechanism comes into play. It relies in the default.desktop file, which defines the default window manager and can be managed by the common update-alternatives workflow. Meaning it can be easily tweaked by the package maintainers and by the users, specially since YaST includes a nice module for managing alternatives.

Storage reimplementation: improvements in the expert partitioner

Although, as explained above, we keep improving the storage automatic proposal to support more and more situations, we cannot ignore that flexibility and adaptability have always been two of the flagships of (open)SUSE. And one of the most prominent examples is the YaST Expert Partitioner.

As detailed in our report of the sprint 36, we have been rewriting that powerful Swiss Army knife using the new storage stack but keeping the same user interface and functionality. So far, the new implementation was only able to display information about the existing partitions, LVM systems and MD RAIDs. But now we added many options to create, edit and delete partitions.

It is still a work in progress because the number of possibilities offered by the YaST Partitioner is sometimes overwhelming and implementing them all takes time, but it’s progressing nicely.

Beside the improvements in the Partitioner itself, we also worked on its integration in the installation workflow. Now the Expert Partitioner can be used to refine the schema automatically proposed by the installer. As a bonus, the behavior of the “abort” and “finish” buttons has been improved in relation to the Expert Partitioner currently available in (open)SUSE, which historically shows a usability inconsistency there compared to the rest of the installation process.

Fixed Automatic Patch Installation

In both SLE and openSUSE Leap, the online updates can be installed either manually or automatically at some regular intervals. For the automatic installation we provide the yast2-online-update-configuration package which provides a cron job script and an YaST module for configuring it (hint: it is not installed by default, you might give it a try, maybe it is something “new” for you).

You can configure how often the patches should be installed or filter the patch categories in the YaST module, but we got a bug report saying that when multiple patch categories are selected only one is actually used.

It turned out to be a trivial mistake in the cron job and we fixed it for SLE12-SP2 and Leap 42.2, as well as for the upcoming SLE12-SP3 and Leap 42.3. So if you use this module and want to use the category filter then it’s recommended to upgrade the package.

Storage reimplementation: many steps forward in the AutoYaST integration

And going back to our new storage stack, we keep working to integrate it better with other parts of YaST, specially AutoYaST. During this sprint we polished some rough edges and we added support for MD RAID. With all that, now is possible to automatically setup a system based on an AutoYaST profile containing any combination of partitions, LVM systems and MD arrays, including encryption at any of the levels.

But the relation of AutoYaST with the system works in both directions. Apart from being able to install a system based on an AutoYaST profile, it also offers the interface to export the current system configuration, including the storage layout, to a profile in order to reproduce the system later. During this sprint we also ported that logic to rely on the new storage layer.

It was harder than it sounds, due to the need of keeping the backwards-compatibility with several behaviors that have been introduced along the AutoYaST history to adapt to several specific situations. On the bright side, the new code is easier to follow, includes behavior-driven automated tests (RSpec) and contains information about the rationale of each decision… which in some cases required some archaeology.

Storage reimplementation: fixed bootloader proposal in S/390 and PowerPC

Another part of YaST we are constantly working to integrate better with the new storage stack is yast2-bootloader. Although the new storage system was already able to correctly setup a valid disk layout for most situations and architectures (each one with it’s own requirements for booting), our bootloader module is still not fully compatible with the new system in all those scenarios.

During this sprint, we adapted it to ensure all the combinations suggested by the new storage stack (partitions, LVM, encryption and so on) are correctly covered by YaST2-Bootloader. As a result, we can already say that our testing ISO images are fully installable in any x86_64, PowerPC and S/390 system by just clicking “next” a few times. And we have automated integration tests (a separate openQA instance) to ensure the resulting system boots just fine.

UDEV device id on PowerPC

We know some of our readers enjoy our more technical posts and like to lurk into the kitchen to see how we deal with all kind of surprises maintaining a complex tool like YaST. Today’s chapter in that regard started with a bug report about the bootloader being installed in the wrong device name (/dev/vda instead of /dev/vdb) in an emulated PowerPC machine in openQA.

After a lot of investigation and with help from our PowerPC expert, we found the culprit, that turned out to be an emulator quirk. The next couple of paragraphs may be interesting or daunting, depending how much virtualization and PowerPC jargon you know. Be warned.

On POWER, the PReP partition containing the bootloader has no unique identifier other than the serial number of the disk on which it created. QEMU virtualization does not provide any disk serial number when the user does not explicitly specify one. This means that the PReP partition in QEMU installation does not have any unique identification and the partition name may change when a disk is added or removed from the virtual machine or the storage configuration otherwise changes. This may lead to system errors related to bootloader installation and updates.

It is recommended to assign a unique serial number to each disk in a QEMU virtual machine on POWER when it is expected the storage configuration of the virtual machine may change. Otherwise, there is nothing YaST or any other tool running within the virtual machine can do to avoid the problems. So this time we only did the investigation part, with the fix coming from the openQA side, which was improved to explicitly set serial numbers when needed.

Storage reimplementation: better support for advanced scenarios

As exposed above, our StorageNG testing image already works in all the supported architectures and in scenarios combining plain partitions, LVM, RAID and encryption in any way. But there are even more situations and technologies currently supported by YaST that we need to incorporate into the new stack.

First of all, we used this sprint to add Multipath I/O support to the new libstorage. Now it can also be combined with all the other mentioned technologies (like having an LVM system on top of an encrypted multipath device), although the storage proposal still needs to be adapted to play nicely with preexisting multipath setups.

As usual with the library stuff, the best “screenshot” we have to offer is one of its geeky autogenerated graphs.

Another scenario that goes beyond the most regular use cases is installing the system into a network storage device, instead of a local hard disk. Now, the new storage system can report whether a root filesystem via a network is used. When that happens, YaST sets the network interfaces with the start mode nfsroot, which is used to avoid interface shutdown and, therefore, the unavailability of the system.

That’s all, folks!

Once again, we omitted the boring parts about bugfixing (with yast2-ntp-client being the star in that regard) and similar stuff. We hope you enjoyed the report and we hope to reach you again in two weeks. Meanwhile…

Have a lot of fun testing the Leap 42.3 pre-releases and reporting bugs!

Posted in Desktop, Distribution, Factory, Packaging, Programming, Software Management, Systems Management, YaST | 1 Comment »

Highlights of YaST development sprint 37

July 3rd, 2017 by Yast Team

Since we announced in the latest report, we decided to shorten our development sprints from three to only two weeks. As a natural consequence, this is the first report of a series of shorter ones. But shorter doesn’t have to mean less juicy! Keep reading and enjoy.

Displaying very long changelog

We got a bug report about YaST not responding when a very long package changelog was displaying in the package manager. It turned out that some packages have a huge change log history with several thousands entries (almost 5000 for the kernel-default package). That produces a very long table which takes long time to parse and display in the UI.

The solution is to limit the maximum number of displayed items in the UI. You cannot easily read that very long text anyway, for such a long text you would need some search functionality which the YaST UI does not provide.

Finding the limit, that magic number, was not easy as we want to be backward compatible and display as much as possible but still avoid that pathological cases with a huge list.

In the end we decided to go for a limit of 512 change entries. The vast majority of packages have way fewer entries, so you should not notice this clipping functionality. When the limit is reached a command to display the full log is displayed at the end so you can easily see the missing part when needed. (Hint: the widget supports the usual copy&paste functionality, you can copy (with Ctrl+C) the displayed command and paste it into a terminal window directly.)

Storage reimplementation: MD RAID support in the rewritten partitioner

In the previous report we told you we are rewriting the YaST partitioner to use the new storage stack and modern reusable CWM widgets under the hood while retaining exactly the same behavior and look&feel on the surface. We are reimplementing one feature at a time, and this sprint it was the MD RAID’s turn.

Now the partitioner displays current RAIDs, with details about them and the devices used to construct each RAID. If a picture is worth a thousand words, here you have 3,000 words:

No, you can’t have a separate partition for /var/lib/docker for CaaSP – so here you have it

Our brand new containers-oriented solutions, SUSE CaaSP and its openSUSE counterpart Kubic, need to have /var/lib/docker as a separate partition for several reasons. It was a separate Btrfs subvolume already, but that wasn't quite separate enough. 🙂

The problem was just that the automated storage proposal in the old YaST storage subsystem is not prepared for anything like this; it can deal with root, swap and optionally a separate home partition (or volume if using LVM). That's it. Extending the current syntax of the control.xml file to be able to specify arbitrary partitions and adapting the code of the old proposal to understand and handle this new specification was unfortunately almost impossible. Sadly, we have to admit the old code is hardly maintainable these days and not flexible enough to accommodate this kind of changes in a safe way.

This is one (out of many) reasons why we are currently in the process of rewriting that entire YaST storage subsystem, as you already know from many previous posts in this blog. But, as you also know, the new system will debut in SLES15 and openSUSE Leap 15.0, too late for the current SUSE CaaSP and openSUSE Kubic.

We decided we'd introduce a hack based in the old proposal, well knowing that hacks accumulated in code can develop their own life just like Dust Puppy. But we plan to kill that hack immediately once StorageNG arrives.

So the hack was to simply use the logic for the separate home partition and repurpose it, keeping all the respective parameters in control.xml and introducing yet another one called home_path where the actual mount point for that partition can be specified — in this case /var/lib/docker. The tradeoff is that there can be no separate /home anymore in parallel to that.

That "feature" should not be used outside of that very specific use case in CaaSP and we even considered the possibility of not documenting it too publicly to avoid misuse. But we are living the open source spirit and whatever we do, we do in public. Even if it's the (quite embarrassing) fact that we consider changes to certain parts of our own code too risky. But again, that's why we are pushing the storage layer reimplementation (Storage-NG) so hard: we want to regain control over that part.

Storage reimplementation: custom partitioning with AutoYaST

And talking about the new storage layer and our previous posts, you already know we are working hard to integrate it with AutoYaST. For the time being, custom partitioning layouts are supported, but only using plain partitions and LVM. Other features, like RAID or Btrfs subvolumes support, are still missing.

A nice thing about the new code is that it relies as much as possible on the new storage layer. On older versions, AutoYaST implemented some logic on its own and that caused some unexpected troubles. Fortunately, that is not the case anymore, and the new code looks way easier to extend and maintain.

YaST does not write directly in `/etc/vconsole.conf` anymore

When configuring the system keyboard, YaST used to write the keyboard map configuration directly in the /etc/vconsole.conf file. However, this approach is no longer appropriated since it may cause undesired effects to other tools. Now YaST uses the Systemd tool localectl to set the keyboard map in /etc/vconsole.conf, instead of writing in it directly. Another step to make YaST a good citizen of the Systemd world.

Storage reimplementation: named RAID arrays

If you use MD RAID arrays you probably know there are two ways of identifying them – by number or assigning a meaningful name to them. Named arrays use device names like /dev/md/<name> instead of /dev/md<number>.

During this sprint, we taught the new libstorage how to create and manage named RAID array, in addition to the already supported numbered ones. If the user decides so, the MD RAID name is used in fstab instead of the UUID (which was the only option with the old libstorage). We also made sure to improve the behavior in several scenarios, so we consider some bugs of the old storage to be fixed (or obsoleted, if you prefer) with storage-ng.

Linux also supports names of the form /dev/md_<name>. The new library is also able to handle this format, but the feature is intentionally disabled and documented to be unsupported because other parts of the system could not be 100% verified to work in that scenario. And we take quality assurance very seriously before labeling a feature as “supported”.

If you are not happy enough with the screenshots showing the RAID support in the partitioner, here you have more pictures. But, as usual with stuff implemented in the library, here “pictures” doesn’t mean screenshots, but fancy automatically generated graphs.

Stay tuned

Of course there are a lot of other things we did during the last two weeks, although some of them were considered not interesting enough for this report or were not finished on time. We are already working in finishing the unfinished stuff and implementing new exciting improvements. And the bright side is that you only have to wait two weeks to know more… so stay tuned.

Posted in Distribution, Factory, Programming, Software Management, Systems Management, YaST | Comments Off on Highlights of YaST development sprint 37

Advertisement
Tags
11.3 11.4 12.1 12.2 12.3 13.1 13.2 amd ARM ATI Beta buildservice Build Service C-Language cloud Collaboration Community conference Education event Events Factory fglrx fun GNOME gsoc Hackweek KDE Kernel Kraft Linux LXDE obs openSUSE Package PostgreSQL radeon raspberry Raspberry Pi rpm Ruby Tumbleweed XML xorg YaST
Lizards
- Adrian Schröter (12)
- Agustin Chavarria (6)
- Alessandro de Oliveira Faria (13)
- Alex Barrios (12)
- Alexander Naumov (10)
- Alexander Orlovskyy (3)
- Alin M Elena (5)
- Andrea Florio (27)
- Andreas Jaeger (70)
- Andreas Stieger (12)
- Andrew Wafaa (31)
- Arvin Schnell (9)
- Atri Bhattacharya (3)
- Bernhard Wiedemann (31)
- Bonnie Kurniawan (1)
- Bruno Friedmann (98)
- Calumma Brevicorne (29)
- Carl Fletcher (1)
- Christopher Hobbs (17)
- Ciaran Farrell (3)
- Stephan Kulow (17)
- craig gardner (2)
- Stephan Barth (2)
- Thomas Schmidt (2)
- Dinar Valeev (1)
- Dirk Mueller (2)
- Dmitry Serpokryl (7)
- Efstathios Iosifidis (21)
- Fabio Mucciante (5)
- Federico Lucifredi (9)
- Greg Freemyer (1)
- Holger Sickenberg (2)
- Hubert Mantel (1)
- Ilya Chernykh (5)
- Ismail Donmez (1)
- J. Daniel Schmidt (2)
- James Tremblay (7)
- Jan Blunck (4)
- Jan Loeser (3)
- Jan Madsen (1)
- Jan-Christoph Bornschlegel (3)
- Jan-Simon Möller (20)
- Javier Llorente (12)
- Jigish Gohil (85)
- Jiri Srain (1)
- Jiří Suchomel (3)
- Johan Kotze (5)
- José Oramas M. (6)
- Josef Reidinger (16)
- Juergen Weigert (1)
- Julio Vannini (9)
- Dinar Valeev (5)
- Kevin "Yeaux" Dupuy (11)
- Klaas Freitag (55)
- Lars Vogdt (11)
- Ludwig Nussel (13)
- M. Edwin Zakaria (4)
- Marcus Hüwe (39)
- Marcus Meissner (2)
- Marcus Moeller (3)
- Marcus Schaefer (4)
- Martin Lasarsch (8)
- Martin Mohring (11)
- Masim "Vavai" Sugianto (20)
- Michael Andres (1)
- Michael Löffler (7)
- Michal Marek (7)
- Michal Vyskocil (12)
- Miguel Angel Barajas Hernandez (2)
- P Linnell (2)
- Nelson Marques (55)
- Nenad Latinović (1)
- Nikanth Karthikesan (2)
- Przemyslaw Bojczuk (1)
- Peter Pöml (4)
- Petr Gajdos (2)
- Petr Mladek (60)
- Petr Uzel (5)
- Ray Wang (1)
- Raymond Wooninck (1)
- Ricardo Chung (7)
- Ricardo Varas Santana (7)
- Richard Bos (11)
- Robert Schweikert (16)
- Rossana Motta (1)
- Rupert Horstkötter (10)
- Sascha Manns (66)
- saydul akram (3)
- Sebastian Siebert (6)
- Shawn Dunn (2)
- Stanislav Visnovsky (7)
- Stefan Haas (1)
- Stefan Hundhammer (5)
- Stefan Schubert (7)
- Steffen Winterfeldt (8)
- Suresh Jayaraman (3)
- Susanne Oberhauser (3)
- Thomas Göttlicher (6)
- Thomas Schraitle (26)
- Togan Muftuoglu (3)
- Tuukka Pasanen (36)
- Will Stephenson (22)
- YaST Team (90)
Archives
- March 2020 (1)
- February 2020 (2)
- January 2020 (1)
- December 2019 (3)
- November 2019 (2)
- October 2019 (4)
- September 2019 (3)
- August 2019 (3)
- July 2019 (4)
- June 2019 (2)
- April 2019 (4)
- March 2019 (3)
- February 2019 (5)
- January 2019 (1)
- December 2018 (2)
- November 2018 (2)
- October 2018 (3)
- September 2018 (1)
- August 2018 (3)
- July 2018 (2)
- May 2018 (2)
- April 2018 (2)
- March 2018 (2)
- February 2018 (2)
- January 2018 (2)
- December 2017 (1)
- November 2017 (2)
- October 2017 (2)
- September 2017 (3)
- August 2017 (4)
- July 2017 (4)
- June 2017 (2)
- May 2017 (4)
- April 2017 (2)
- March 2017 (3)
- February 2017 (3)
- January 2017 (2)
- December 2016 (5)
- November 2016 (3)
- October 2016 (6)
- September 2016 (2)
- August 2016 (3)
- July 2016 (4)
- June 2016 (2)
- May 2016 (2)
- April 2016 (1)
- March 2016 (2)
- February 2016 (4)
- January 2016 (4)
- December 2015 (6)
- November 2015 (2)
- October 2015 (3)
- September 2015 (2)
- August 2015 (2)
- July 2015 (2)
- June 2015 (3)
- May 2015 (12)
- April 2015 (7)
- March 2015 (6)
- February 2015 (6)
- January 2015 (7)
- December 2014 (5)
- November 2014 (3)
- October 2014 (5)
- September 2014 (3)
- August 2014 (5)
- July 2014 (5)
- June 2014 (7)
- May 2014 (9)
- April 2014 (2)
- March 2014 (9)
- February 2014 (9)
- January 2014 (10)
- December 2013 (9)
- November 2013 (10)
- October 2013 (10)
- September 2013 (6)
- August 2013 (7)
- July 2013 (3)
- June 2013 (7)
- May 2013 (4)
- April 2013 (4)
- March 2013 (7)
- February 2013 (6)
- January 2013 (3)
- December 2012 (3)
- October 2012 (6)
- September 2012 (6)
- August 2012 (5)
- July 2012 (12)
- June 2012 (6)
- May 2012 (4)
- April 2012 (4)
- March 2012 (5)
- February 2012 (2)
- January 2012 (5)
- December 2011 (10)
- November 2011 (6)
- October 2011 (5)
- September 2011 (9)
- August 2011 (12)
- July 2011 (14)
- June 2011 (11)
- May 2011 (18)
- April 2011 (15)
- March 2011 (26)
- February 2011 (16)
- January 2011 (23)
- December 2010 (27)
- November 2010 (18)
- October 2010 (21)
- September 2010 (16)
- August 2010 (21)
- July 2010 (20)
- June 2010 (33)
- May 2010 (29)
- April 2010 (24)
- March 2010 (29)
- February 2010 (22)
- January 2010 (20)
- December 2009 (15)
- November 2009 (21)
- October 2009 (17)
- September 2009 (22)
- August 2009 (28)
- July 2009 (36)
- June 2009 (38)
- May 2009 (40)
- April 2009 (30)
- March 2009 (20)
- February 2009 (21)
- January 2009 (27)
- December 2008 (23)
- November 2008 (12)
- October 2008 (23)
- September 2008 (40)
- August 2008 (24)
- July 2008 (12)
- June 2008 (28)
- May 2008 (26)
- April 2008 (1)