Home Home
Sign up | Login

Highlights of YaST Development Sprint 51

February 22nd, 2018 by
  1. Can you perform an online offline migration?
  2. Will you understand the license better if you display a different translation?
  3. Is your /home mounted or haunted?

Welcome back to our linguistic blog disguised as a YaST team sprint report.

Installation and Upgrade

Offline Upgrade Using Bootable Media via SCC

The last few weeks we spent quite some time implementing the offline migration from the old SUSE Linux Enterprise products (versions 11 and 12) to the upcoming version 15.

Note: the offline migration term actually means that your production system is not booted and running, it is not about the network status. At offline migration a different system is booted (usually from a DVD). See more details in the official SUSE documentation.

We implemented and tested the upgrade from SLE 12 SP3. The offline migration workflow is similar to the online migration as implemented in SLE 12 release. The only difference is that you boot the SLE 15 medium, select Upgrade in the boot menu and then select the disk partition to upgrade. The rest should be the same as in the online migration.

The migration from SLE 11 is a bit more complex as it is registered against the older Novell Customer Center and requires some additional changes in the installer. This is work in progress.

And the last note: for testing the offline migration to SLE 15, your system needs to be registered using the Beta registration keys. For regular SLE 12 and 11 registrations, the migration to SLE 15 is blocked. It will be unblocked after the SLE15 has been officially released.

Made the addon Linuxrc Option Work Well with the Packages DVD

In SLE 15 there are numerous modules and extensions, such as the Live Patching module, or the Web Scripting module. On physical DVDs, we are putting all of them on a single Packages DVD.

When you are installing SLES and choose to add such a module during the installation from the DVD, you will be presented with a screen to select from all the modules found on the DVD.

You can also automate this step by passing the addon=dvd:/// option at the installation boot prompt. (See the Linuxrc reference). Formerly this worked only with single-product media. Starting now, the addon option will work also with multi-product media such as the Packages DVD.

Improve Licenses Translations Support

Some months ago, YaST started to use libzypp to get product licenses, instead of using the old SUSE Tags approach. However, until this sprint, this feature was somehow incomplete.

On one hand, the complete list of supported languages was shown, no matter whether a translation was available for a given language or not. On the other hand, the "Licenses Translation" button was missing (it is still used in single product media).

Now both problems are solved and, as soon as new translations are included in the installation media, they should be handled gracefully in the installer.

Replaced Components

Finalize Xinetd Conversion to Systemd Sockets

This sprint we finished our change from xinetd to systemd sockets for starting services on demand. To finalize it there is basically two main tasks.

The first one was dropping the YaST module for xinetd. That required a conversion of yast2-ftp-server that used this module and also adding a note to AutoYaST that xinetd configuration is no longer supported, so if you have it in your AutoYaST profile, it won’t be applied. The FTP server part was harder because, as mentioned in the last report, one of the two backends does not support systemd sockets, but we found that this backend is a bit ancient and support for us was quite painful. The result is that we dropped pure-ftp and kept only vsftpd backend, which makes the code much simplier and our life better (the final diff-stat is +1100/-3700). And then we converted the usage to systemd sockets. Then we could proceed to dropping yast2-inetd because there was no dependency anymore.

The second task was xinetd usage directly, with an API for starting on demand. It is not used too often and in the end the biggest parts were dropping xinetd usage for VNC based installation and yast2-inst-server which is now converted to use systemd sockets. And here we can give you a nice trick we discovered during the implementation: If a systemd service has a parameter (often the case with services started by a systemd socket) you can stop all of them with a wildcard, e.g. systemctl stop vnc@*.

So here is a happy end – after this sprint there is no xinetd usage and we can support only one tool for starting on demand, allowing us to focus on improving other parts.

Added support for exporting firewalld AutoYaST configuration

In the previous blog entry we announced AutoYaST support for configuring firewalld but cloning the firewall configuration was not supported yet and also the AutoYaST summary concerning the Firewall module needed some love and that is basically what we have implemented during this sprint.

It should be noted that editing of the AutoYaST Firewall configuration is not allowed since the firewall configuration is now done with the firewalld graphical configuration tool (firewall-config)

Storage and Partitioner

Added the Format Options Dialog to the Partitioner

One of the missing things in the new partitioner was the format options dialog letting you tune the file system a bit when it is created.

The options itself are more or less the same as in the old partitioner. This feature is intended more for the experts. As an average user you will rarely find a need to fine-tune file system parameters. But in case you do, the dialog is there to aid you (remember the help button).

Here is a screenshot of the ext4 options:

Removed the Empty Views in the Partitioner

In the process of rewriting the YaST storage stack, we also rewrote the partitioner. Some views in the partitioner were taken over from the old one, but not filled with any functionality so far – they only showed empty pages. We now removed those empty pages:

  • Crypt Files
  • Device Mapper
  • Unused Devices
  • Mount Graph
  • Settings (this one will be back soon with content)

This is what it looks like now:

Compared to the previous version with the empty views:

See also Bug #1078849.

Installation Summary in the Partitioner

One of the sections that survived the Partitioner sifting mentioned above was the Installation Summary. During this sprint we re-implemented this useful view that shows the changes that would be performed in the system, including packages to install in order for the system to work with the chosen technologies. One image is worth a thousand words.

Of course, the information in both lists is updated with every change done in the Partitioner and, as usual, everything works as a charm also in text mode. Including the possibility of collapsing or extending the (usually lengthy) list of operations on subvolumes.

Blocker Errors in Partitioner

A few sprints ago we implemented warnings in the partitioner that inform you if something looks like probably not working, but with expert knowledge it can be made working. Now we add also blocking errors where we are sure it won’t work. It is just a first draft so it will be adapted as needed and as problems appear. Some checks are already moved from the bootloader to the partitioner, so you can fix the partitioning quickly. But enough words, check out the screenshots, where the first one is an error which prevents continuing and the second one is just a warning.

Extending the AutoYaST <device> Element

When defining a <drive> section in an AutoYaST profile, the <device> element should determine to which disk you want to apply that partitioning schema. Usually, it is a device kernel name, like /dev/sda, or a link which resolves to a disk (for instance, /dev/disk/by-id/*, /dev/disk/by-path, etc.).

However, AutoYaST supports specifying other names, like by-partlabel, by-label, etc. Those links won’t resolve directly to a disk, but the storage layer will be able to find out which disk they belong to.

Although SLE 12 supported this behaviour, it was missing from SLE 15 until this sprint.

Fine-tuning of the Requirements to Boot a System

In some situations, an extra partition or a given disk layout is needed to boot an (open)SUSE operating system. Like the separate /boot partition needed in some corner-case legacy scenarios, the ESP partition that must be mounted at /boot/efi in EFI systems or the PReP partition needed by some PowerPC systems. The partitioning proposal performed by the installer tries to ensure those booting requirements are met (so does AutoYaST in some cases) and our beloved Partitioner also includes some checks to warn you user if you forget to create or mount any of those partitions.

But in some situations, the installer was suggesting partitions with a suboptimal size or even partitions that were not strictly needed. On the other hand, the Partitioner was sometimes being too picky, warning about situations that were not such a big problem. So during this sprint we refined our list of booting requirements, updating the corresponding documentation, relaxing the partitioner checks and fine-tuning the proposal outcome. Specifically the PowerPC requirements were revamped based on the input from several experts and bug reports from manufacturers of some systems.

So no matter if you usually trust the installer proposal, if you like to handcraft stuff with the Partitioner or if you install using AutoYaST, the experience should be more smooth now, which fewer (if any) ugly surprises at boot time.

Mount Options Revisited: When the Old Demons Come back to Haunt You

Recently, we reintroduced per-filesystem-type defaults for mount options. We thought this would be a great chance to clean up old code that had become messy over time; we thought we could provide a clean, well-structured and easy-to-understand way to do this.

Then people began to test some more scenarios, and we found out the hard way that in some situations, those mount options depend on the mount point for various reasons: Some quirks of underlying kernel modules like the VFAT filesystem or the way systemd handles mounting the root filesystem during booting made this necessary.

So we had to bite the bullet and reintroduce some of that old code which was kind of messy; for example, for ext4 or ext3 root filesystems, we no longer specify any data=... mount options because this might make remounting the root filesystem read/write at boot time fail; in another case, a mkdir -p /boot/efi/EFI (when installing the boot loader) on a VFAT partition failed despite VFAT technically being case-insensitive (omitting iocharset=utf8 fixed this).

Lesson learned: Some messy old code is messy for a reason. Trying to streamline it may break some scenarios.

Conclusion

The answers to the initial questions are

  1. Yes.
  2. 说不定.
  3. Most of the time.

As the SLE release cycle is shifting from the "all features are mandatory" phase to the "all bugs are top priority" phase, expect less of feature news and more bugfix news in the next report, due in two weeks.

openSUSE will be at LibrePlanet 2018

February 14th, 2018 by

I’m glad to let you know my participation this year at #LibrePlanet 2018. I have been invited to give a talk. It will be on March 24-25 in Cambridge, Massachusetts.

Proudly representing to openSUSE

LibrePlanet is an annual conference hosted by the Free Software Foundation for free software enthusiasts and anyone who cares about the intersection of technology and social justice. For the past ten years, LibrePlanet has brought together software developers, law and policy experts, activists, students and computer users to learn skills, celebrate free software accomplishments, and face challenges to software freedom. Newcomers are always welcome, and LibrePlanet 2018 will feature programming for all ages and experience levels.

 

Highlights of YaST Development Sprint 50

February 9th, 2018 by

Sprint 50 is finished and we are ready to share with you all the highlights of this last sprint!

We are close to finishing features for the new SUSE Linux Enterprise 15, so this time we came up with a lot of information about the new storage, partitioner, firewall enhancements, and roles, which now is also present for the Desktop version of SUSE Linux Enterprise. Besides that, we fought another issue regarding memory consumption in Tumbleweed installation and once again we were able to solve it. So, let’s take a look into details!

Filesystem type specific default mount options in /etc/fstab are back

We reimplemented another feature that was still missing with the rewrite of the YaST storage stack: Reasonable mount options in /etc/fstab that may be different for each filesystem type. For example, for an Ext4 filesystem we used to set options data=ordered,acl,user_xattr by default. Of course, you can still change those options in the partitioner if the defaults don’t work well for you.

For most filesystem types, this was straightforward: Just filling a table with fixed default options. But for some others, most notably the legacy Microsoft filesystem types FAT/VFAT, this involved some not-so-trivial heuristics to figure out locale data so we could provide reasonable values for iocharset=... and codepage=...; those are necessary to recode filenames with non-ASCII international characters between whatever a MS Windows system might use and Linux; otherwise, you might not even see those files when mounted on a Linux system.

Resurrected sections in the Expert Partitioner

We keep bringing the functionality from the old Partitioner back to the Storage-ng reimplementation. During this sprint, it was the turn of the NFS and Device Graph sections.

The NFS case is a little bit special from the technical point of view since it’s the only section not managed directly by the Expert Partitioner. Instead, an embedded version of yast2-nfs-client is executed when visiting that section. The same approach has been followed in this reimplementation, which means NFS is the only section in which the old Partitioner and its Storage-ng clone will offer an absolutely identical experience and behavior (including both features and bugs). Just a heads-up: NFSv4 support is still not fully functional in libstorage-ng, so the check-boxes about the NFS version may be ignored until that support reaches your distribution.

On the other hand, the “Device Graph” section was, as most of the new Partitioner, rewritten from scratch. In addition to the already known graph showing how the system is going to look after applying the changes, it includes now a representation of the current system in a separate tab. Both graphs are interactive, double-click on any node takes the user to the corresponding section of the Partitioner.

As you can see in the previous screenshot, the usual button to export the graph to Graphviz format is not alone anymore. Its new friend allows exporting the graph to the very detailed XML format used by the YaST Team to reproduce test scenarios.

Of course, the “Device Graph” section is not available if using the text-based ncurses interface, as you can see comparing the left part of the following screenshot with the previous one.

Partitioner: LVM thin provisioning

Our new Expert Partitioner was already able to manage LVM volume groups and logical volumes since several weeks ago. But now it also recovers its great ability to work with LVM thin pools and volumes. LVM thin provisioning is a powerful technology, very useful in cases where you need to administrate storage resources for a large group of users. Basically, thin provisioning allows you to provide more storage space than it is actually available in the system. You can increase the real hardware storage only when it is really required. For more information about LVM thin provisioning, you can find a great guide in this link.

With the Expert Partitioner, you can create a thin pool over a LVM volume group in a similar way you create a normal logical volume. You only have to add a new logical volume and check the “Thin pool” option in the first dialog. Once the volume group contains at least a thin pool, you will be able to create thin volumes. Once again, the process is exactly the same. Add a new logical volume and select “Thin volume” option and in which pool to create it. The rest of steps are exactly the same as for creating a normal logical volume.

Apart from creating new thin pools and volumes, options for editing, resizing and deleting are now also available for all kind of logical volumes: normal volumes, thin pools and thin volumes. In the case of resizing a thin pool, you will be warned when the resulting pool is overcommitted. Take a look at the following screenshot.

Adapting partitions and logical volumes sizes in AutoYaST

In older versions, when the proposed partitioning in a profile and it did not fit in the disk, AutoYaST tried its best to reduce the biggest partition. A typical use case was to set the root (/) partition to a huge value, so the profile could be used in systems with different disk sizes (as AutoYaST will take care of reducing that partition to make it fit). To be honest, the best solution in that scenario would be to set the size to max, so AutoYaST will do what’s expected.

However, if for any reason the wanted layout does not fit, AutoYaST is now smarter about what to do: instead of blindly reducing the biggest one, it will try to reduce all partitions in a (kind of) proportional way, informing the user about the new sizes.

Small behavior change in AutoYaST installation

Up to now, not installable packages have been ignored silently by AutoYaST, which has been defined in the AutoYaST configuration file. From now on, the user will be informed that these packages cannot be installed.

System roles for SUSE Linux Enterprise Desktop 15

SUSE Linux Enterprise uses system roles to define, during the installation process, the packages to install on the system, so it can have the necessary software to play its “role”. This feature was already available for SUSE Linux Enterprise Server and now it is also available for SUSE Linux Enterprise Desktop. In order to choose a specific role for your system, you need to select a module or extension which contains this role during the installation process. There are four available roles for Desktop:

  • GNOME Desktop (Wayland): available when Desktop Productivity (on SLED) or Workstation Extension are selected.
  • GNOME Desktop (X11): available when Desktop Productivity (on SLED) or Workstation Extension are selected.
  • GNOME Desktop (Basic): available when the Desktop Application module is selected.
  • IceWM Desktop (Minimal): available when Basesystem module is selected.

Firewalld enhancements

During the last sprints, our team has been working hard in the integration of firewalld with AutoYaST and with the CWM library for opening ports / services in our different modules (http-server, squid, dns-server etc..). Here are some changes we did during this last sprint:

YaST firewall module is discontinued

As we now adopted firewalld in our distribution, there is no more reasons to have a YaST module for the firewall. Therefore, as you may see here, we discontinued the YaST Firewall module and we recommend to use firewall-config to configure your firewall via a user interface or firewall-cmd for the command line.

AutoYaST

A new AutoYaST schema has been defined for firewalld configuration although the features supported are very limited.

We can configure properties like the default zone, the service state, the type of packages to be logged and zones specific configuration like interfaces, services, ports, protocols, and sources.

For example, a SuSEFirewall2 based profile defining some interfaces, services, and ports in the EXT zone like this one:

<firewall> 
  <enable_firewall config:type="boolean">true</enable_firewall>
  <start_firewall config:type="boolean">true</start_firewall>
  <FW_DEV_EXT>eth0</FW_DEV_EXT>
  <FW_SERVICES_EXT_TCP>443 80 8080</FW_SERVICES_EXT_TCP>
  <FW_SERVICES_EXT_UDP>21 22</FW_SERVICES_EXT_UDP>
  <FW_CONFIGURATIONS_EXT>dhcp dhcpv6 samba vnc-server</FW_CONFIGURATIONS_INT>
</firewall>

will be translated to something like this:

<firewall>
  <enable_firewall config:type="boolean">true</enable_firewall>
  <start_firewall config:type="boolean">true</start_firewall>
  <default_zone>public</default_zone>
  <zones config:type="list">
    <zone>
      <name>public</name>
      <interfaces config:type="list">
        <interface>eth0</interface>
      </interfaces>
      <services config:type="list">
        <service>dhcp</service>
        <service>dhcpv6</service>
        <service>samba</service>
        <service>vnc-server</service>
      </services>
      <ports config:type="list">
        <port>21/udp</port>
        <port>22/udp</port>
        <port>80/tcp</port>
        <port>443/tcp</port>
        <port>8080/tcp</port>
      </ports>
    </zone>
  </zones>
</firewall>

SuSEFirewall2 based profiles will continue working although limited to the configuration currently supported by YaST.

During the autoinstallation, an error will be shown if the profile has some property not supported, however, we will able to continue with the installation and also a warning will be shown suggesting the use of the new schema even when all the properties are translatable.

Opening ports / services in zones through interfaces selection

In YaST, CWMFirewallInterfaces module provides a set of widget definitions for opening services in zones through a selection of interfaces (each interface belongs to a ZONE).

The module has been adapted to work properly with the new firewalld API.

If a firewalld service is not defined (probably because the package shipping the service definition has not been adapted yet), then the CWMFirewallInterfaces widget will show a list of missing services suggesting to deploy them to be able to configure the firewall.

Move from Xinetd to Systemd sockets

As requested by our friends from packages, now the future of starting service on demand is systemd sockets instead of xinetd. There are multiple reasons for that, but for us the most important one is that we can concentrate in one thing and do it properly.

It’s hard to support different ways to start services on demand and especially to handle such a situation when a user can set it up with sockets. Our old approach to make it happen was to set up the start of services on demand with xinetd. However, this approach is really hard to debug. Besides this problem, we also would like to unify our approach with the one suggested by packagers people.

So what have we done in this last sprint? The goal was to do some research, to create a proof of concept on one specific module and to find potential obstacles when unifying approaches to start services on demand. We found out that the conversion from xinetd to systemd sockets is quite easy, but also cannot be done automatically. Basically, what we need to do is to use systemctl call (which we already support) to activate the socket that is provided by the package instead of our old approach, which is to write a file into /etc/xinetd.d/ and to reload xinetd. Such a change will make our life much easier than previously.

The hardest part is that xinetd configuration file also often contains the service configuration that YaST can and would like to modify. This is no longer possible with systemd sockets and it is also the reason that we cannot automate this conversion.

So how will we proceed with the conversion of the YaST modules? We first checked all modules that use xinetd and we verified that, for the majority, the conversion is straight-forward. YaST also has a module for inetd configuration, which no longer makes sense, so the plan is to drop it in the near future and to allow socket activation in services manager module.

A problematic module is ftp-server, which supports two backends: vsftp and pure-ftp. This is problematic because pure-ftp does not support systemd-sockets, only xinetd. We plan to discuss how to handle it, as we are also not happy that we support two backends, because it makes the life of our users harder when they just want to quickly configure a ftp server. For SLE users we already support only vstftp, so if we agree on it, we will probably also support only this server in YaST. But right now nothing is set in stone.

Memory eating strikes again

Once again we have to mention our battle with memory consumption. This time the problem happened within the NET iso for Tumbleweed, which should need no more than 1 GiB of memory during the installation process.

In the beginning of our research, we had no suspicious code to look at, we just knew that it could be related to the fact that the NET iso was using the whole repository of Tumbleweed (over 60k packages) while the DVD iso uses only a subset of packages.

So which technique did we use to find the problematic part of the code? We changed logging in order to append the memory status of the whole process to each log line. By using this approach we quickly identified the problematic part of the code. This problematic code was responsible only for logging, but as it was searching for all packages and loading all packages properties, it was consuming a huge amount of data and causing this memory issue.

As a hotfix, we removed this logging from packages and kept it only for patterns and products, which is more important for us and has low memory consumption. We also looked into all code that searches through all packages properties and we reduced it as much as we could. Finally, we forced the Ruby garbage collector to run before the processes of disk preparation and rpm installation, reducing the chances that the memory consumption of the installation process goes up.

Once again we had a happy end and we are now able to install the NET iso with 1 GiB of memory when using the graphical installer.

Concluding

As we’re getting closer to finish SLES 15, we’re getting busier and busier with features and bugs to finish. Therefore, YaST team is already working hard on Sprint 51 and we’re looking forward to come back to you in two weeks with much more cool stuffs that we’re doing. Meanwhile, have fun and stay tuned!

Highlights of YaST Development Sprint 49

January 25th, 2018 by

Time goes by and the YaST wheel keeps rolling. So let’s take a look to what have moved since our previous development report.

More flexible NET installation ISOs

Network installation media for Tumbleweed or Leap only work properly with the exact repository they have been built for – which for Tumbleweed may mean they could be outdated after just one day.

You would then run into this message:

Linuxrc warning

To improve the situation the installer can now offer to download matching boot files (kernel and initrd, to be precise) from the repository if it detects this situation:

Linuxrc offering a solution, as always

Of course, you can say ‘No’ here – but then you’re back to the red dialog. 😀

Technically, what’s done is to download a new kernel/initrd pair from the repository and restart the installation process with them (using kexec). So be prepared for a slight déjà vu.

This feature is controlled by the kexec boot option.

Storage-ng lands into Tumbleweed: handle with care

But that’s not the only news we have about openSUSE Tumbleweed. Our usual readers already know about Storage-ng, our effort to rewrite the whole YaST storage stack from scratch. And they also know it’s still a work in progress. But since there were too many valuable changes blocked by the adoption of Storage-ng, it was decided it was time to push the red button. So we are glad to announce the Storage-ng era has started with its inclusion in the first official (open)SUSE product – starting with snapshot 20180117, libstorage-ng has replaced libstorage and, thus, yast2-storage-ng has replaced yast2-storage.

They say forewarned is forearmed, so an article was published in advance in news.opensuse.org to set the expectations and to provide and overview of the current status. We would like to encourage all openSUSE Tumbleweed users to (re)visit the article to get a better picture of the situation.

Alignment of partitions in the expert partitioner

An important part of that work in progress is the re-implementation of the Expert Partitioner with Storage-ng technologies. As mentioned many times in previous posts, this is mainly a 1:1 clone, with the same functionality presented in exactly the same way than the classic YaST partitioner. But some times we take the opportunity to introduce some improvement here and there, as we did this week with a topic that can have a very noticeable impact in the system performance: partitions alignment.

Although many people is not aware of it, the partitions in a system must be properly aligned to avoid the performance drop caused by excessive read-modify-write cycles. For details please refer to the great article at Wikipedia explaining the topic, especially the sections titled “4 KB sector alignment” and “SSD page partition alignment”. Moreover, leaving performance considerations aside, some partition tables require alignment to simply work, like DASD partition tables which need alignment to tracks (usually 12 sectors).

The new expert partitioner takes all that into consideration when creating and resizing partitions, ensuring always the required alignment (like the DASD tracks) and encouraging the optional performance-related one, avoiding undesired gaps between partitions in the process.

Detail of the Expert Partitioner dialog to create a partition

Above you can see the dialog for choosing the size for a new partition that, unsurprisingly, looks very much like the same dialog in the pre-storage-ng Expert Partitioner. If a size is specified by the user in that dialog (any of the two first options in the form), the start and end of the partition will be aligned to ensure optimal performance and to minimize gaps. That may result in a slightly smaller partition (with the difference being usually less than 1MiB). If a custom region is specified, the start and end will be honored as closely as possible, with no performance optimizations (although mandatory alignment, like DASD tracks, still will take place). This third option is the best to create very small partitions.

The same considerations for optimal alignment will also be taken into account while resizing an existing partition and calculating the minimal and maximal sizes suggested by the partitioner during that process.

Choosing the new size of a resized partition

Sanity checks for the storage setup

The possibility of bypassing the performance optimizations in the Expert Partitioner is just one example of the (potentially unleashed) power that tool provides. As a consequence of that flexibility, sometimes the user can overlook some important setup configurations or even make mistakes. To help with that, the Expert Partitioner recovered this week its ability to check the entered storage setup.

Once the user has set partitions, LVM volumes, file systems, mount points, etc. and decides to proceed, the Partitioner will validate that setup to ensure it fulfills all necessary requirements for booting and running the system. When some issue is detected, a popup message is presented to show what the problem is, offering the option to ignore the warning and move forward.

The resurrected partitioner sanity checks

Two kind of checks are carried out to ensure the partitioning setup validity. First, the presence of needed partitions for booting is checked. Booting requirements depends on the current architecture (x86, PowerPC, AArch, etc.) and other technical details like the partition table type (GPT vs MS-DOS). Then, the mandatory volumes for the current product are checked. The mandatory volumes are defined in the revamped partitioning section of the control file. Typically, only a volume for root and another for swap used to be mandatory, but now this is totally configurable by anyone defining the product (SLE, Leap, Tumbleweed, your own custom openSUSE derivative…).

As a bonus, all the sanity checks are now centralized (they used to be scattered around the YaST source code) and it’s easier to add new ones (you will miss some old checks at this moment) and to use them from other parts of YaST (like the bootloader module or AutoYaST).

More improvements in the Expert Partitioner

The new warnings and the alignment improvements commented above are not the only news on the evolution of the Expert Partitioner clone this week. Resizing of LVM devices has also been brought back to life, both for volume groups and logical volumes. In the case of logical volumes, the functionality is not much different, at least in the surface, from the partition resizing that was already present and that you can see in the screenshot of the alignment section.

On the other hand, in the context of the Partitioner, resizing a volume group actually means adding or removing physical volumes. Actions that are now possible again, including the corresponding checks. For example, a physical volume cannot be removed if it already exists on disk (that could destroy your data) or if the resulting size of the volume group is not enough to cover all its logical volumes.

Trying to remove the wrong PV

Apart from the mentioned functionality, there has also been improvements in how the Expert Partitioner presents the information. For example, now the “type” column shows the correct label and icon for each device instead of that useless TODO label. Moreover, similar TODO marks were replaced by proper data in the device overview tab.

TODO labels are gone

Minimize changes between the SLE15 “Installer” and “Packages” DVDs

The SUSE Enterprise Server 15 (SLES15) product can be installed from a bootable “Installer” DVD medium which contains the installer and a subset of packages needed for a very minimal system. The other packages are available either from a registration server (after registering the SLES product) or via a separate “Packages” DVD medium.

Due to the structure of those DVDs (with some packages being in present in both) the SLES installer was asking the user to change the medium several times during the installation process. Ideally the installer should use all packages from the “Packages” medium without changing the media.

In addition, there is yet another requirement for preferring the packages from the installation DVD to the packages available via a remote repository. Downloading a package from the internet is usually much slower than the DVD and can be problematic in network connections with a download limit or with a price based on the bandwidth usage.

Now the installer properly adjust the priority of all the repositories to achieve the desired behavior. To avoid possible side effects we decided to change the repository priority only when more than one repository is used and all repositories are local (e.g. DVD, hard disk, USB flash disk…). That means in some less common cases (2 DVDs + a remote repository) you will still need to change the medium but this is a safer solution.

Add On products in AutoYaST

For those using SLE Add On products, we have improved the error message if an Add On Product cannot be added during an AutoYaST installation. The user can see now which wrongly configured Add On Product has produced the error.

AutoYaST reporting which Add On is wrong

This will be specially useful with the upcoming SLE15, in which the concepts of Add Ons and Modules will become more relevant than ever.

Fixed a crash when shutting down the YaST user interface

And now it’s time for the corresponding dose of technical insights for those who enjoy that part of our reports.

When UI::OpenDialog() and UI::CloseDialog() calls didn’t match when shutting down the UI (user interface YaST component), you’d get a segmentation fault with a core dump. Well, you did want to shut down YaST, but probably not like that. This is now fixed.

After tracking this down, it was surprisingly simple to reproduce: Just use the YaST version of the trivial “Hello, World” program and comment out the UI::CloseDialog() call.

This was a case of providing additional error reporting causing more problems than the original error: leaving dialogs open while terminating the program is an error, of course. But fixing this little problem by cleaning up the remaining dialogs lead to handling widgets after some of the underlying infrastructure (in this case the QApplication) was already destroyed, so all the QWidgets were also destroyed (because the QApplication takes care of that), but YaST’s generic UI layer was still unaware of that fact and tried to destroy them again.

This is now fixed by properly cleaning up the widget tree in YaST’s generic UI layer first which will also clean up the associated QWidgets so there is nothing left to clean up for the QApplication.

This might also fix a number of similar segfaults in other situations where the YaST Ruby engine would need to shut down because of other problems, e.g. when there is an unhandled Ruby exception.

Surprisingly enough, this must have been a very old (10+ years?) bug, but it never became quite obvious, or at least nobody was ever annoyed enough to try to track it down.

If you want even more details, check the conversation in the bug report.

More to come

The end of this sprint caught up with a lot of almost finished stuff. But following the Scrum principle of “nothing is done until it fits the Definition of Done”, we don’t blog about such stuff. Fortunately, that means the next report will likely be quite juicy. So, see you again in a couple of weeks!

Highlights of YaST Development Sprint 48

January 9th, 2018 by

The YaST team finished its 47th Sprint right before the Christmas break but, sadly, we had not published the corresponding report… until now. The last sprint of the year brought some interesting changes, like Chrony support for AutoYaST, better multi-products medium handling, etc. So let’s recap those changes.

Chrony support in AutoYaST

As part of our effort to support Chrony as the default NTP service for (open)SUSE, we have revamped how AutoYaST handles the configuration of such a service. The first noticeable change is that we have redesigned the schema which, instead of containing low level configuration options, is now composed of a set of high level ones that are applied on top of the default settings.

And here is how the new (and nicer) configuration looks like:

<ntp-client>
  <ntp_policy>auto</ntp_policy>
  <ntp_servers config:type="list">
    <ntp_server>
      <iburst config:type="boolean">false</iburst>
      <address>cz.pool.ntp.org</address>
      <offline config:type="boolean">true</offline>
    </ntp_server>
  </ntp_servers>
  <ntp_sync>15</ntp_sync>
</ntp-client>

Updating the Remote Administration Capabilities

During this sprint, the remote administration client has been deeply modified. To begin with, as xinetd is being replaced by systemd sockets, we have dropped that dependency (adjusting the code accordingly).

Additionally the VNC handling have been improved too. Until now, YaST offered the possibility to connect through a web browser using a Java applet. Now YaST allows the user to enable/disable this feature (check the screenshot below to see how it looks now). It is worth to mention that Michal Srb has replaced the old viewer with novnc, a JavaScript based one. Thanks a lot for that, Michal!

And last but not least, we have seized the occasion to do some code cleaning, reimplementing some dialogs using the Common Widget Manipulation object oriented API.

Modifying AutoYaST Profile During Installation

AutoYaST offers a cool feature that allows the profile to be modified during the initial stages of the installation using an user script. So you can run a script which adjusts the profile and AutoYaST will read it again. If you are interested in such a feature, you could find more information in the official documentation.

On the other hand, in our previous report, we mentioned that AutoYaST was able again to use multipath devices using the new storage stack. But we didn’t count that it was possible to modify the profile on runtime so the initialization happened too early.

Now the bug is fixed so you can again adjust any storage setting using the aforementioned feature.

Properly Handling Selected Modules

As you may know, some time ago we added a support for the multi-product media (DVDs which contain more than one repository/product in separate subdirectories). This time we fixed some issues regarding this functionality.

Originally after selecting several products only one of them was actually selected to install and only one product was displayed in the installation proposal. Fortunately, those issues have been addressed now.

Unified Look & Feel for Multi-Product Selection Dialog

For the multi-product DVD media we used this selection dialog:

The functionality is very similar to the on-line product selection dialog displayed after registering the system:

As you can see the look & feel is quite different, but from the usability point of view the dialogs should look the same regardless whether the products are added from a DVD medium or from an on-line source (a registration server).

This sprint we improved the DVD media dialog to better match the registration dialog:

.

The dialog is still not exactly the same, but now it looks more similar so users should feel more familiar with it. There is also displayed an additional note about not handling the product dependencies automatically. This note was already present in the help text but that is hidden by default. As we got quite a lot of bug reports about this issue we decided to make this fact more prominent.

(Note: The dialogs actually cannot look the very same as the DVD media currently lack some information like the dependencies, beta status, detailed descriptions…).

Dropping SYSTEMCTL_OPTIONS Variable Support

We have been using the environment variable called SYSTEMCTL_OPTIONS (which is SUSE-specific) in our systemd services to prevent locks in dependencies. As this hack will not be necessary for upcoming the (open)SUSE 15 version, it will be dropped from systemd and, therefore, we already removed it from our systemd services.

Unifying Disklabel Handling in AutoYaST

When specifying an AutoYaST partitioning schema, you can select which partition type you want to use for each device (MSDOS, GPT, DASD, etc.). In the past, AutoYaST implemented its own logic to select which one to use in case that it was not specified by the user. After this sprint, AutoYaST relies on the new storage stack in order to decide which option fits better when the user does not specify one.

Bonus: Automatically Checking the Defined Systemd States

Some time ago we had serious issues with service management in YaST (see bug#1012047 and bug#1017166). The problem was caused by introducing a new systemd service state which was not expected by YaST. We fixed the problem by correctly handling the new state.

But the main problem was that we (as the YaST developers) were not notified about this major system change and we found this change later after we got bug reports in Bugzilla. To avoid this problem again in the future we decided to implement a script which would regularly check the defined systemd states notifying us if a unknown state was detected.

To implement the regular check we use the Travis cron job feature which allows running continuous integration builds not only after a change is pushed into the repository but also in regular intervals, even when there is no change in the code.

Alternatively you can use any CI service, but we chose Travis because it is easy to use and we already use Docker for normal CI jobs which allows us to run the latest systemd from openSUSE Tumbleweed in an easy way.

In this case we could possibly run the script in OBS when building the yast2-services-manager package, but that would need adding systemd to BuildRequires which does not sound as a good idea…

So if you also need to run some check scripts regularly you can see more details in this pull request.

Conclusion

2017 was an exciting year with a lot of interesting stuff: a new storage layer, multi-product installation medium, integration of new components (firewalld, chrony, etc.). And it looks like 2018 will not be different and we will have a lot of fun.

Thanks for your support and happy new year!

Highlights of YaST Development Sprint 47

December 7th, 2017 by

On the night of December 5th, teams of three are sprinting through the streets of Czech towns, visiting homes: Mikuláš (the developer), čert (the scrum master), and anděl (the product owner). If you’ve been a bad kid, you will get pieces of coal or brown potatoes. Good kids, on the other hand, will get new YaST features:

  • Performance: adapting to jemalloc, a better Ruby memory allocator
  • Replaced components: s/ntpd/chrony/g s/SUSEFirewall2/firewalld/g
  • Storage: resizing partitions, resizing RAID
  • Software modules and extensions: preselecting them, resolving dependencies automatically, creating your own installation ISO with mksusecd
  • Security related: tarballs for reproducible builds, SSH keys for root
  • AutoYaST: asking for a disk

Performance

Using Jemalloc Memory Allocator in Ruby

The MRI Ruby (the original C implementation) interpreter allows using the jemalloc library for allocating the memory instead of the usual glibc default. The advantage is better memory usage (30-40% less used memory was reported here) and it has some extra debugging features.

But after enabling this feature in Ruby in openSUSE Factory and SLE15 many YaST packages failed to build. It turned out to be an issue caused by dynamically loading the library by YaST. A workaround was to directly link the YaST against libjemalloc.

Later we found out that some packages still do not build. This time the problem was caused by a warning printed by the library on the error output. That happened only when running the old YaST test suite which still uses an embedded Ruby interpreter (normal YaST usage or the new unit tests use full Ruby which does not have this problem). The fix was just to ignore that warning when checking the error output in the tests.

If you want to check whether your Ruby uses the jemalloc feature then run this command:

ruby -r rbconfig -e "puts RbConfig::CONFIG['LIBS']"

If the output includes the -ljemalloc option then your Ruby is using the jemalloc library.

Replaced Components

Switching NTP from ntpd to chrony

It was decided that it is a good time now to switch the Network Time Protocol (NTP) implementation from ntpd to chrony as it is more secure, nicer and better. So also YaST has to adapt and modify its NTP client module to work with chrony. The module also shows its age and it is a bit hairy, so we took the opportunity to do some cleaning.

The most visible change is the UI on the running system. Please note that the UI is not final yet and it is mostly just a proof of concept. But the goals are clear: The first goal is to be really client only, as previously NTP also allowed to configure an NTP server, making the UI complex and confusing for the majority of users that just want to select a server to sync against. The second goal is to focus on the majority of users who want a simple UI to configure a common NTP client, leaving fine tuning for experts who modify configuration files directly.

In the installer, chrony is now called instead of ntpdate. This change is invisible to users and it simply works the same as previously.

We also decided to drop the command-line interface (CLI) of the NTP client module as chrony itself comes which a nice CLI chronyc. Instead of duplicating the work our CLI just suggests to use chronyc.

Please also note that changes are not finished yet and more will come. One of such changes is AutoYaST support, which is currently just skipped. But let us show you some pictures to get an idea how it will look soon in Tumbleweed and explain our decisions along the way:

The first image is from the main screen. It shows three main areas:

  1. The way the time is synchronized. It allows a pure manual run, synchronizing in a given interval or using the daemon that runs always.
  2. Configuration of sources for synchronization servers. It can be dynamic or static. For static, only the ones in the servers table (see point 3) are used. For dynamic, it also adds NTP servers from the network (like ones from DHCP). So when the user uses only public servers, the static configuration works well. But when internal NTP servers are used and they can change, then using the dynamic option makes sense.
  3. A list of synchronization servers. This specifies a statically defined list of NTP servers to sync against. You can use the familiar Add/Edit/Delete buttons.

The second image is from the Edit dialog. It was also simplified to the most common options. The Test button allows a quick check if a server is reachable and there are no typos in its address. The Quick Initial Sync option is useful to synchronize time quickly when connected to network. And Starting Offline is a nice chrony feature that allows to define the server as offline during the start-up and only when connected to the network to mark it as online and synchronize. It greatly helps with boot time and is a good option for notebooks where fast boot is important and the network is managed by Network Manager.

Firewalld will be the default firewall in SLE15 (Installer adapted)

Firewalld has some nice advantages over SuSEFirewall2: it can be dynamically managed, permits more zones, has a separation between runtime and permanent configuration options, it is well maintained, and it is already supported in some applications or libraries.

These are some of the main reasons why it has been decided to replace completely SuSEFirewall2 with firewalld.

In a past sprint report we announced that yast-firewall was adapted to support firewalld, but that was mainly an adaptation of the code to supporting both backends.

During this sprint we have completely replaced SuSEFirewall2 in the installer using a new firewalld API and although the UI seems to not have changed at all, the firewall dialogs have been completely reimplemented using CWM (our object-oriented API to YaST widgets).

During next sprints we will continue adapting the firewalld API and YaST modules to complete the replacement. The next target will be supporting firewalld in AutoYaST.

Storage

Expert Partitioner: Resizing

The Expert Partitioner has regained the ability to resize partitions. A Resize button is now available in the Hard Disks view as well as in the view of either a particular disk or a specific partition. When you try to resize a partition, a new dialog is shown with three options: maximum size, minimum size and custom size.

When the Custom Size option is selected, you have to specify manually the new partition size, but do not worry, the Expert Partitioner will prevent you from entering wrong sizes. Also, it will automatically align the partitions for a better performance.

Improvements in RAID management

As with partitions, it is now also possible to resize software RAIDs, although in the case of RAIDs resizing is a completely different operation performed by adding or removing devices to it. The partitioner will ensure you have selected the minimum amount of devices necessary for the current RAID type. Take into account that resizing a Software RAID will be only possible for new RAIDs being created, that is, resize action is not allowed when the RAID exists on disk.

As you can see in that screenshot, buttons for sorting the devices within a RAID have also been added at the right of the device list during this sprint. Those buttons are available both when creating a new RAID and when resizing it.

Last but not least, now it is also possible to work with BIOS RAIDs in the same way you do with regular disks. You will find all BIOS RAIDs in the Hard Disks section of the Expert Partitioner, where you could add or remove partition to the RAID device.

Software Modules and Extensions

The SLE15 products were split into several repositories and the installation medium contains only the minimal set of packages required to boot the system. If you want to have something more then you have to either register the system or use an additional medium.

To make that easier for users the registration module now pre-selects the recommended modules or extensions. However, if you for some reason really need a minimal system you can still deselect the pre-selected items.

The list of recommended addons is maintained at the server side (SCC) so it is possible to change that without updating the YaST installer.

Adding Modules and Extensions in AutoYaST

The list of available modules and extensions for SLE15 is already quite long as the base SLE15 products have been split into several modules. That makes adding the modules from the registration server more complicated compared to SLE12.

Originally the AutoYaST registered the addons in the XML profile exactly in the specified order. And that made troubles because SCC requires the dependent modules to be registered first. Writing the modules in the correct registration order in the profile was not easy for SLE15.

To make it easier AutoYaST now automatically reorders the modules according to their dependencies and even registers the dependent modules which are missing in the profile. Obviously that works well only if the missing module does not require a registration key.

mksusecd and linuxrc Supporting the New Media Layout

Starting with SLE15 the installation media have a slightly different layout. In short: the content file that was holding information about the product and a list of checksums is gone and has been replaced by .treeinfo and CHECKSUMS. Also, the package repository is now a repomd repository.

Also, SLE15 has been split into several so-called modules. The installation medium itself contains only a minimal set of packages. You have to register during the installation process to access other modules or use a separate medium containing them.

To save you from carrying several USB sticks around, mksusecd lets you build your own customized installation medium.

First, check what is on the media. The installation iso has just the base product:

mksusecd --list-repos SLE-15-Installer.iso
Repositories:
  SLES15 [15-0]

and you need an extra image for the modules:

# mksusecd --list-repos SLE-15-Packages.iso 
Repositories:
  Basesystem-Module [15-0]
  Desktop-Productivity-Module [15-0]
  Desktop-Applications-Module [15-0]
  Development-Tools-Module [15-0]
  HA-Module [15-0]
  HPC-Module [15-0]
  Legacy-Module [15-0]
  Public-Cloud-Module [15-0]
  SAP-Applications-Module [15-0]
  Server-Applications-Module [15-0]

Then pick the modules you want to add to the installation medium:

mksusecd --create new.iso \
  --include-repos Basesystem-Module,Legacy-Module \
  SLE-15-Installer.iso SLE-15-Packages.iso
Repositories:
  SLES15 [15-0]
  Basesystem-Module [15-0]
  Legacy-Module [15-0]
assuming repo-md sources
El-Torito legacy bootable (x86_64)
El-Torito UEFI bootable (x86_64)
building: 100%
calculating sha256...

Now use new.iso to start an installation.

Security / Development

Creating Reproducible Tarballs

We automatically submit the YaST packages from the Git repositories to the OBS projects using Jenkins. We use two Jenkins instances, the public Jenkins submits to openSUSE Factory/Tumbleweed, the internal Jenkins submits the same packages to SLE15.

The OBS compares the SLE15 packages with the openSUSE version to make sure the same packages are built in both projects. But this check compares only the file checksums, not the archive content.

The problem was that the internal and external Jenkins created a tarball with different checksum although the files inside were exactly same. If you run simple tar cfjv archive.tar.bz2 files you will find out that the tarball checksum is different for each run. So how to create a tarball in a reproducible way?

To build a reproducible tarball we use these tar options:

  • --owner=root --group=root – to make sure the file owner and the group is always the same, it does not matter who builds the tarball or who owns the files on the system
  • --mtime=<timestamp> – to have constant file time stamps, it does not matter when you downloaded the source files. Having fixed time stamps (like 0) is not a good idea as you do not know how the files are old actually, but using the current time makes the tarball not reproducible. So we use the date of the last commit in the Git repository (git show -s --format=%ci), it is stable across multiple runs and still describes how old the files are.
  • --format=gnu – use the GNU tar format, the default POSIX format contains some archive time stamps
  • --null --files-from - – this tells tar to read the file names from standard input, we use this options together with find ... -print0 | LC_ALL=C sort -z to sort the files to always have the very same file order.

Note: it also depends which compression method you use for compressing the tarball. For example gzip by default also adds some time stamps to the archive, bzip2 does not. If you want to use gzip then use the -n option.

See the tarball Rake task for the implementation details.

AutoYaST supports root SSH Authorized Keys deployment

From a security point of view, it is recommended to use SSH for logging into a remote machine instead of user and password in plain text format. But having to memorize the user and password specially if you try to use a strong non predictable password is also hard and hard to maintain.

For that reason SSH provides authentication based on public key cryptography and AutoYaST supports the deployment of SSH Authorized keys for normal users since SLE-12-SP2 but it was not working for the root user.

We have fixed this issue in SLES-12-SP3 allowing to deploy the SSH authorized keys also for the root user and exporting them into the cloned system profile.

AutoYaST keeps improving

The storage-ng based AutoYaST version has received a couple of improvements like, for instance, an improved resizing handling (supporting logical volumes) or multipath support. Additionally, we’ve squashed some bugs, like setting size=max for LVs (bsc#1070131) or some problems when not defining any partition (bsc#1070790 and bsc#1065061).

But even more interesting is the fact that we have brought back a hidden (and undocumented) feature. Did you know that if you set the <device/> element to ask, AutoYaST will ask you about which device to use for installation? Cool, right? This feature will continue working in (open)SUSE 15 and now it is documented!

Conclusion

My colleagues are telling me that practicing Scrum may be getting just ever so slightly on my nerves because I have several bugs in my description of the St. Nicholas tradition. Surely they are wrong. Stay tuned for the next report from a new and improved Scrum team: Ježíšek, Santa, und das Christkind.

Encrypted installation media

November 17th, 2017 by

Hackweek project: create encrypted installation media

  • You’re still carrying around your precious autoyast config files on an unencrypted usb stick?
  • You have a customized installation disk that could reveal lots of personal details?
  • You use ad blockers, private browser tabs, or even tor but still carry around your install or rescue disk unencrypted for everyone to see?
  • You have your personal files and an openSUSE installation tree on the same partition just because you are lazy and can’t be bothered to tidy things up?
  • A simple Linux install stick is just not geekish enough for you?

Not any longer!

mksusecd can now (well, once this pull request has been merged) create fully encrypted installation media (both UEFI and legacy BIOS bootable).

Everything (but the plain grub) is on a LUKS-encrypted partition. If you’re creating a customized boot image and add sensitive data via --boot or add an add-on repo or autoyast config or some secret driver update – this is all safe now!

You can get the latest mksusecd-1.54 already here to try it out! (Or visit software.opensuse.org and look for (at least) version 1.54 under ‘Show other versions’.

It’s as easy as

mksusecd --create crypto.img --crypto --password=xxx some_tumbleweed.iso

And then dd the image to your usb stick.

But if your Tumbleweed or SLE/Leap 15 install media are a bit old (well, as of now they are) check the ‘Crypto notes’ in mksusecd --help first! – You will need to add two extra options.

This is how the first screen looks then

Highlights of YaST Development Sprint 46

November 10th, 2017 by

It’s Hack Week time at SUSE! But before we dive into all kind of crazyinnovative experiments, let’s take a look to what we achieved during the latest development sprint.

User-friendly error messages in AutoYaST

During recent weeks, the AutoYaST version for the upcoming SLE 15 family has received quite some love regarding the integration with the new storage layer, from fixing bugs to adding some missing (and even some new) features. So let’s have a look at what we have done so far.

First of all, a new error reporting mechanism will debut in the upcoming AutoYaST version. Until now, when a problem occurred during partitioning, you got a message like “Error while configuring partitions. Try again.“. It does not help at all, right? At that point, you were on your own to find out the problem.

Now AutoYaST is able to identify and report different problems to the user in a convenient way. What is more, in many situations it is even able to point to the offending section of the AutoYaST profile.

The error reporting mechanism can distinguish between two different kind of issues: warnings and errors. When a warning is detected, a message is shown to the user but the installation will not be stopped (it honors the settings in the <reporting> section). Errors, on the other hand, will block the installation entirely.

Please, bear in mind that this error reporting mechanism is only available for the <partitioning> section. Maybe it could be extended in the future to cover other parts of the auto-installation process.

Bringing back skip lists to AutoYaST partitioning

When defining a partitioning schema, you can let AutoYaST decide which device should be used for installation. Thanks to that, you can use the same profile to install machines with, for instance, different storage devices kernel names (like /dev/sda and /dev/vda).

Needless to say that, in such a situation, we might want to influence the decision process. For example, we would like to avoid considering USB devices for installation. AutoYaST offers a feature known as skip lists which allow the user to filter out devices using properties like name, driver, size, etc.

Unfortunately, skip lists support in SLE 15 Beta1 is rather limited. But these days we have extended yast2-storage-ng to offer additional hardware information and now AutoYaST is able to use it to filter devices.

As a side effect, the ayast_probe client has been fixed to show (again) which keys you can use in your skip lists.

More on AutoYaST

Apart of adding or bringing back features, we have fixed several bugs. You can check the recent entries in the yast2-storage-ng changes file if you are interested in the details.

We know that a few features are still missing and more bugs should be addressed sooner or later, but hopefully AutoYaST must work in most use cases.

SLE15 media based upgrade for unregistered system

This sprint we also continued implementing the upgrade from SUSE Linux Enterprise (SLE) 12 products to the version 15. Particularly we solved the upgrade of unregistered systems.

In that case you need the “SLE15 Installer” medium and additionally also the “SLE 15 Packages” medium. The installer medium contains only the minimal packages for installing just a very minimal system. The rest is available either via the registration server or via the extra medium. Obviously for unregistered systems only the second option makes sense.

In this sprint we were focused on making all pieces to work together. You can see the result in the following screencast.

Upgrading an unregistered system

Fixed an installer crash in systems with 512MB RAM

We got a bug report that the beta version of the upcoming SUSE Linux Enterprise Server 15 was sometimes crashing during installation on a system with 512MB RAM. That’s bad, the 512MB is a required limit which should be enough to install a minimal system in text mode.

At first we thought that the crash was caused by insufficient memory, but the reported memory usage was OK, there was still enough free memory.

It turned out that the problem was in the pkg-bindings which tried to evaluate undefined callback function. The fix was quite simple, however, the question was why that happened only in systems with 512MB RAM and not when there was more memory.

Later we found out that the difference was caused by the extra inst-sys cleanup (mentioned in the Sprint 22 report) which YaST runs when there is low memory. In that case YaST removed the libzypp raw repository metadata cache. The assumption was that when the data is already parsed and cached in the binary solv cache the original files are not needed anymore. However, libzypp still might use some raw files later.

So we changed the inst-sys cleanup algorithm to remove only the files which we know are not needed later and keep the rest untouched.

Expert partitioner: the some boys are back in town

Several features have been brought back to the expert partitioner during this sprint.

  • Allow to create and delete logical volumes.
  • Allow to delete MD RAIDs.
  • Allow to work with multipath devices.

Now you can create logical volumes using the expert partitioner. When you go to the LVM overview or visit a specific volume group, a button for adding a logical volume is available. Clicking on it, you will be taken through a wizard for the creation of a logical volume. Note that although the logical volume type can be selected in the first wizard step, only normal volumes can be created. Thin logical volumes and thin pools will come soon. And apart of creating logical volumes, now there is also a button for deleting them.

LVM management in the reimplemented partitioner

Creating an LVM LV in the reimplemented partitioner

Deleting an LVM LV in the reimplemented partitioner

Delete action has been also implemented for MD RAIDs. For that reason, you have a delete button in the RAID overview and also when you access to a specific MD RAID. And of course, you will be asked for confirmation before removing the device.

Deleting an MD RAID in the reimplemented partitioner

Additionally, another important feature recovered during this sprint is the possibility to work with multipath devices. Now, multipaths are listed together with other disks in the tree view of the expert partitioner, allowing you to manage them as regular disks. For example, you can create or remove partitions over them. Moreover, when a multipath device is selected, a new tab is showed to list the so-called “wires” that belong to the multipath.

Multipath devices in the reimplemented partitioner

Improving the product upgrade workflow

Although the possibility to offer an upgrade option from openSUSE Leap to SLE is on both SUSE and openSUSE radars for the future, the reality is that it has been, and still is, an unsupported scenario.

But with previous versions of SUSE Linux Enterprise, you could take a SLES DVD, boot it in the Upgrade mode, and select to upgrade an openSUSE partition. YaST would let you proceed several screens before telling you that it actually will not let you upgrade from openSUSE to SLES.

Starting with recent SLE15 pre-releases, the incompatible products are filtered out in the partition selector already (overridable with a Show All Partitions checkbox), letting you know earlier whether you will be able to upgrade your system to the new SLES.

Fix of a registration issue during installation process

In SLE 15 Installer, there is a product selection dialog at the very beginning of the installation. After that, you can register the selected product but you cannot change the product later as unregistering the product and registering another one is not supported. Our awesome QA squad found out that when the installation was aborted and then started again from Linuxrc without rebooting, the installer thought that the product had been already selected and did not offer any product for installation. A little fix made it work again – now we always execute the following SUSEConnect command at the start of the installer.

SUSEConnect --cleanup

That removes all traces of previous registration attempts from the Installer. This also means that you might still want to unregister directly at SUSE Customer Center if needed.

Improving help texts in the registration process

As you have seen so far, we have been working hard to polish the registration experience in many aspects and scenarios. That also includes a better communication with the user. Thus, the help text in the registration module has been improved to also include the description of the check box states. This is especially important for the “auto selected” state which is specific to this dialog and is not used anywhere else.

The help texts in YaST use an HTML subset which allows also including images. In this case we included the check box images directly from the UI stylesheet. But in the text mode we have to use text replacement instead of the images. That means the help text content must be created dynamically depending on the current UI.

Here you can see examples of both interfaces.

The graphical version of the new registration message

Text-based version of the new registration message

Twisting the storage proposal: this time for real

In our report of sprint 42 (to be precise, in the section titled “Twisting the storage proposal”) we presented our plans to make the software proposal more customizable in a per-product basis and the draft document of the new format for control.xml that would allow release managers to define the installer behavior in that regard.

Now this goes further than a mere specification and the new format is actually being used to define the partitioning proposal of both the KVM/Xen role of SUSE Linux Enterprise 15 and the upcoming SLE15-based CaaSP.

In the following screenshot you can see the corresponding step of the guided setup for the mentioned KVM/Xen role, in which the classical controls for the /home and Swap partitions have been replaced by more goal-specific volumes defined in the section of the control file describing the role.

Partitioning configuration for the KVM/Xen role

And, as you can see below, the installer now honors those settings to propose a reasonable partition layout.

Storage proposal for the KVM/Xen role

The new format and the corresponding implementation of both the logic and the UI are flexible enough to empower the release managers to define all kind of products and to make possible for everybody to create a more customized derivative of openSUSE without renouncing to the power of the automatic proposal. See another example below (not corresponding to any product or derivative planned in the short term) with more possibilities and note how the wording was automatically adapted to talk about LVM volumes instead of partitions, based on the user choice in a previous step.

LVM-based example of the new proposal

Replacing ntpd with Chrony in yast2 ntp client

Chrony will replace the classical ntpd as default NTP client starting with SLE15 and openSUSE Leap 15. That will offer several advantages to system administrators and other users, as can be seen in this comparison. In order to make this replacement possible, we started a research to find out what is supported in Chrony and how to allow our users to configure it through YaST.

The research phase is now complete and we have already a plan to proceed with the adaptation of the existing yast2-ntp-client module. Also a few bits of code, which allows to set the NTP service during installation, are now in a feature branch (so not yet in Tumbleweed).

The next step will be a huge improvement (and simplification) of the YaST module, which will go further than adapting a list of options. In the screenshot below you can see the not yet finished prototype in action.

Configuring the keyboard in the installer via systemd

Originally the keyboard configuration was written directly by YaST in the corresponding Systemd-related configuration files. But we got a bug report that YaST should not touch the config directly and rather call the localectl tool for changing it. (See the details in the localectl man page).

However, this works only in the installed system, it does not work in the system installation as it needs a running Systemd that is not available during the installation process. Changing the setting for a not running system must be done using the systemd-firstboot command.

But this did not supported modifying the keyboard settings. Fortunately one of the SUSE developer helped us and implemented this feature to Systemd (pull request). Currently the feature is available in (open)SUSE packages but later it will be available in the upstream release for others.

Another related change was that YaST not only set the console keyboard but also constructed the keyboard settings for X11 (GUI). But this is actually a duplicated functionality, localectl itself includes this feature. So we have removed it from YaST and let the localectl tool to set both keyboard setting automatically.

And now for something completely different

Hack Week 0x10 (that is, the 16th edition) is starting just right now. Which means most developers of the YaST team will spend a week working on topics that may or may not have a direct and visible impact in our beloved users in the short term. But hey, maybe we will build a robot or a space rocket!

After that week, we will restart our Scrum activity. So if nothing goes wrong, you will have another update about the YaST development in approximately four weeks. Meanwhile, join us at Hack Week and let’s have a lot of fun together!

Highlights of YaST Development Sprint 45

October 25th, 2017 by

The wait is over: finally a new YaST team report, with news about our 45th sprint!
Our team is still focused on the development of the upcoming SUSE Linux Enterprise (SLE) 15 products family and openSUSE Leap 15, which in this sprint resulted in new dialogs to select modules and extensions, changes for the multi-product medium, and fixes for issues that have been found during our development phase. So let’s check out the most interesting things that came out of our last sprint.

New Modules/Extensions Selection

SLE has a specific dialog that allows the user to select additional modules or extensions. When we first introduced this selection dialog, the extensions could have only a single dependency, which resulted in a maximum of only two levels of dependency. During this last sprint, we implemented changes to allow a chain of dependencies. You can check on the image below this new selection dialog in action:

Reliable Self Update for Multi Product medium

Until now, the self-update URL depends on the product which is ship in the medium. However, as you may know, SLE 15 product family will be shipped in a multi-product installation medium. For that reason, sometimes self-update was failing as only a single product is defined in SCC.
Now we have fixed this issue by a defining self-update identifier that is used instead of a product name, which allows the self-update feature to work in a reliable way.

Welcome screen adapted for upgrading

Some sprints ago we announced the addition of product selection to the initial screen for LeanOS installer.

The welcome dialog is shared between different workflows like between installation and upgrading. The problem is that for upgrading we need to find the target system or root partition before selecting the product to migrate. Now it does not require any selection if there are no products to select, so it will work when upgrading. Besides that, we have polished some presentation details like the dialog title and the product selector caption.

Check out the screenshots below to see the final result:

LeanOS:

Tumbleweed:

Unavailable Packages in AutoYaST

AutoYaST needs to make sure that, after rebooting into the 2nd stage (when needed), the user can access to the installation process using the same tools that he/she used during the 1st installation stage. Apart AutoYaST packages it self, it may need to install other additional tools like VNC, SSH or the X.org system.

Unfortunately, as SLE 15 is split into modules, it’s not guaranteed that the VNC, SSH or X.org packages can be installed, which resulted in AutoYaST failing when trying to install those packages.

We have improved the package handling and now YaST displays a warning that the packages are missing and the system (and later the installer) could not be accessed as expected. However, the AutoYaST installation can still proceed although you cannot watch AutoYaST running during 2nd stage.

Improved Handling of Multi-repository Media

We got some bug reports about the new multi-repository media handling in YaST (mentioned in the sprint 43 report). Some of the problems were delegated to the underlying libzypp library, but we got our share of real YaST issues.

One of those problems was, for instance, the inconsistent styling of the MultiSelectionBox widget used in that dialog, which was pretty confusing. Fortunately, the issue has been fixed and now it looks the same than any standard checkbox widget.

More to come

The 46th sprint has already started and has many new items planned to be developed, especially for the SLE15 and openSUSE Leap 15 installer. We are looking forward to bring these planned features to life and tell you about all the details of this sprint. Meanwhile, have fun and stay tuned!

Highlights of YaST Development Sprint 44

October 11th, 2017 by

Here is the YaST team again with a new report from the trenches, this time with a small delay over the usual two weeks. Most of the team keeps focused on the development of the upcoming SUSE Linux Enterprise 15 products family, including openSUSE Leap 15. That means finishing and polishing the new storage stack, implementing the new rich ecosystem of products, modules, extensions and roles (one of the biggest highlights of the SLE15 family) and much more. So let’s dive into the most interesting bits coming out of the sprint.

Hostname configuration during installation

And let’s start with one of those stories that illustrate the complexity hidden below the user-oriented YaST surface. During installation is very common to assign a hostname to the machine being installed to identify it clearly and unequivocally in the future.

Usually it is a fixed hostname (stored in /etc/hostname) but in some circumstances is preferable to set it dynamically by DHCP. Since some time ago (as you read in a previous report and is shown in the image below) YaST allows to set the hostname selecting a concrete interface or with a system-wide variable named DHCLIENT_SET_HOSTNAME which is defined in /etc/sysconfig/network/dhcp. The value to be set for such variable during installation can be optionally read from the distribution control file. Last but not least, as you already know, Linuxrc can also be used to enforce a particular network configuration.

YaST DHCP configuration with several network interfaces

Most users have a simple setup that works flawlessly, but we recently got a bug report about a wrong network configuration after installing the system if the hostname configuration was set via Linuxrc. After some research we found that the value of DHCLIENT_SET_HOSTNAME coming from the control file was overwriting the Linuxrc configuration at the end of the installation. Now it’s fixed and the global variable will be set by the linuxrc sethostname option if provided or loaded from the control file if not. And all that happens now at the very beginning of the installation to give the user to chance to modify it and to ensure the user’s choice is respected at the end.

Setting hostname in Linuxrc

Take into account that with multiple DHCP interfaces the resulting value for DHCLIENT_SET_HOSTNAME is not fully predictable. Hence, in that scenario we recommend to explicitly select the interface which is expected/allowed to modify the hostname.

Extending the installation process via RPM packages

As we have mentioned (a couple of times) during latest reports, we are implementing multi-product support for the installer. It means that SUSE will ship several products on a single installation media.

One interesting feature is that products, modules and extensions can define its own installation roles. For instance, if you select the desktop extension, you will be able to select GNOME as system role.

During this sprint, we have improved roles definitions handling, displaying a different list of roles depending on which product was selected.

As a side effect, we added support for sorting roles assigning them a display order.

Getting Release Notes from the Installation Repository

As part of our effort to drop SUSE tags from the installation media, we improved the way in which release notes are handled during installation.

Release notes are downloaded from openSUSE or SUSE websites in order to show always the latest version. Of course, the installation media includes a copy of them, which may be outdated, to be used when there is no network connection.

From now on, instead using some additional files, this offline copy of release notes will be retrieved from the release-notes package which lives in the packages repository. So we do not need to ship additional files containing release notes in the installation media anymore.

Moreover, although the old approach worked just fine in almost all cases, there was an uncovered scenario. Let’s consider a system which have access to an updated packages repository but is not connected to Internet. That could be the case, for instance, if you are using SUSE Subscription Management Tool (SMT). With the new approach, the installer will get release notes from that repository instead of displaying the (potentially outdated) ones included in the installation media.

Additionally, we refactored and clean-up a lot of old code, improving also test coverage.

Storage reimplementation: bringing more features back

We are also working hard to make sure the brand new yast2-storage-ng includes all the features from yast2-storage, in addition to the new ones. That means that, after this 44th sprint, SLE15 is already able to perform the following operations using the new module.

  • Creating MD software RAID devices in the expert partitioner. This feature is specially relevant for many openQA tests that rely on it.
  • Displaying the compact description of the partitioning proposal in the one-click-installer screen used by SUSE CaaSP and openSUSE Kubic
  • Importing users and SSH system keys from a previous (open)SUSE installation.

One-click-installer view on SUSE CaaSP 4.0 (yast2-storage-ng)

Rethinking LVM thin provisioning

When trying to create a thin-pool using all free space the metadata has to be accounted for. In contrast to linear LVs the metadata for thin-pool uses space of the VG. For instance, if there are 2048 GiB free in the VG, the metadata for a maximal size thin-pool is about 128 MiB and the pool can be about 2047.9 GiB big.

Additionally LVM creates a spare metadata with the same size. This spare metadata is shared between all pools and thus has the size of the biggest pool metadata. The spare metadata can be deleted manually and all pool metadata can also be resized.

When starting with an empty VG it is relative easy to account for the metadata. But how to handle this with an already existing volume group? Also take into account a volume group containing e.g. RAID LVs or cache pools (which also have metadata).

We finally decided that, during probing, YaST will check how much free space the VG has and then it will calculate “reserved” value for the volume group:

reserved = total size - used by LVs the library handles - free

So when calculating available space for a normal or thin pool, it will take the “reserved” into account:

max size = total size - reserved - used by LVs the library handles

The only drawback is that the maximal size for the pool can be smaller than actually possible since e.g. the spare metadata might be shared with an already existing thin pool.

More to come

The 45th sprint has already started and you can expect more and more work in the installer for SLE15 and openSUSE Leap 15 and more news regarding the revamped storage stack. Meanwhile, don’t forget to have a lot of fun!