Home Home > 2018 > 02
Sign up | Login

Archive for February, 2018

Highlights of YaST Development Sprint 51

February 22nd, 2018 by
  1. Can you perform an online offline migration?
  2. Will you understand the license better if you display a different translation?
  3. Is your /home mounted or haunted?

Welcome back to our linguistic blog disguised as a YaST team sprint report.

Installation and Upgrade

Offline Upgrade Using Bootable Media via SCC

The last few weeks we spent quite some time implementing the offline migration from the old SUSE Linux Enterprise products (versions 11 and 12) to the upcoming version 15.

Note: the offline migration term actually means that your production system is not booted and running, it is not about the network status. At offline migration a different system is booted (usually from a DVD). See more details in the official SUSE documentation.

We implemented and tested the upgrade from SLE 12 SP3. The offline migration workflow is similar to the online migration as implemented in SLE 12 release. The only difference is that you boot the SLE 15 medium, select Upgrade in the boot menu and then select the disk partition to upgrade. The rest should be the same as in the online migration.

The migration from SLE 11 is a bit more complex as it is registered against the older Novell Customer Center and requires some additional changes in the installer. This is work in progress.

And the last note: for testing the offline migration to SLE 15, your system needs to be registered using the Beta registration keys. For regular SLE 12 and 11 registrations, the migration to SLE 15 is blocked. It will be unblocked after the SLE15 has been officially released.

Made the addon Linuxrc Option Work Well with the Packages DVD

In SLE 15 there are numerous modules and extensions, such as the Live Patching module, or the Web Scripting module. On physical DVDs, we are putting all of them on a single Packages DVD.

When you are installing SLES and choose to add such a module during the installation from the DVD, you will be presented with a screen to select from all the modules found on the DVD.

You can also automate this step by passing the addon=dvd:/// option at the installation boot prompt. (See the Linuxrc reference). Formerly this worked only with single-product media. Starting now, the addon option will work also with multi-product media such as the Packages DVD.

Improve Licenses Translations Support

Some months ago, YaST started to use libzypp to get product licenses, instead of using the old SUSE Tags approach. However, until this sprint, this feature was somehow incomplete.

On one hand, the complete list of supported languages was shown, no matter whether a translation was available for a given language or not. On the other hand, the "Licenses Translation" button was missing (it is still used in single product media).

Now both problems are solved and, as soon as new translations are included in the installation media, they should be handled gracefully in the installer.

Replaced Components

Finalize Xinetd Conversion to Systemd Sockets

This sprint we finished our change from xinetd to systemd sockets for starting services on demand. To finalize it there is basically two main tasks.

The first one was dropping the YaST module for xinetd. That required a conversion of yast2-ftp-server that used this module and also adding a note to AutoYaST that xinetd configuration is no longer supported, so if you have it in your AutoYaST profile, it won’t be applied. The FTP server part was harder because, as mentioned in the last report, one of the two backends does not support systemd sockets, but we found that this backend is a bit ancient and support for us was quite painful. The result is that we dropped pure-ftp and kept only vsftpd backend, which makes the code much simplier and our life better (the final diff-stat is +1100/-3700). And then we converted the usage to systemd sockets. Then we could proceed to dropping yast2-inetd because there was no dependency anymore.

The second task was xinetd usage directly, with an API for starting on demand. It is not used too often and in the end the biggest parts were dropping xinetd usage for VNC based installation and yast2-inst-server which is now converted to use systemd sockets. And here we can give you a nice trick we discovered during the implementation: If a systemd service has a parameter (often the case with services started by a systemd socket) you can stop all of them with a wildcard, e.g. systemctl stop vnc@*.

So here is a happy end – after this sprint there is no xinetd usage and we can support only one tool for starting on demand, allowing us to focus on improving other parts.

Added support for exporting firewalld AutoYaST configuration

In the previous blog entry we announced AutoYaST support for configuring firewalld but cloning the firewall configuration was not supported yet and also the AutoYaST summary concerning the Firewall module needed some love and that is basically what we have implemented during this sprint.

It should be noted that editing of the AutoYaST Firewall configuration is not allowed since the firewall configuration is now done with the firewalld graphical configuration tool (firewall-config)

Storage and Partitioner

Added the Format Options Dialog to the Partitioner

One of the missing things in the new partitioner was the format options dialog letting you tune the file system a bit when it is created.

The options itself are more or less the same as in the old partitioner. This feature is intended more for the experts. As an average user you will rarely find a need to fine-tune file system parameters. But in case you do, the dialog is there to aid you (remember the help button).

Here is a screenshot of the ext4 options:

Removed the Empty Views in the Partitioner

In the process of rewriting the YaST storage stack, we also rewrote the partitioner. Some views in the partitioner were taken over from the old one, but not filled with any functionality so far – they only showed empty pages. We now removed those empty pages:

  • Crypt Files
  • Device Mapper
  • Unused Devices
  • Mount Graph
  • Settings (this one will be back soon with content)

This is what it looks like now:

Compared to the previous version with the empty views:

See also Bug #1078849.

Installation Summary in the Partitioner

One of the sections that survived the Partitioner sifting mentioned above was the Installation Summary. During this sprint we re-implemented this useful view that shows the changes that would be performed in the system, including packages to install in order for the system to work with the chosen technologies. One image is worth a thousand words.

Of course, the information in both lists is updated with every change done in the Partitioner and, as usual, everything works as a charm also in text mode. Including the possibility of collapsing or extending the (usually lengthy) list of operations on subvolumes.

Blocker Errors in Partitioner

A few sprints ago we implemented warnings in the partitioner that inform you if something looks like probably not working, but with expert knowledge it can be made working. Now we add also blocking errors where we are sure it won’t work. It is just a first draft so it will be adapted as needed and as problems appear. Some checks are already moved from the bootloader to the partitioner, so you can fix the partitioning quickly. But enough words, check out the screenshots, where the first one is an error which prevents continuing and the second one is just a warning.

Extending the AutoYaST <device> Element

When defining a <drive> section in an AutoYaST profile, the <device> element should determine to which disk you want to apply that partitioning schema. Usually, it is a device kernel name, like /dev/sda, or a link which resolves to a disk (for instance, /dev/disk/by-id/*, /dev/disk/by-path, etc.).

However, AutoYaST supports specifying other names, like by-partlabel, by-label, etc. Those links won’t resolve directly to a disk, but the storage layer will be able to find out which disk they belong to.

Although SLE 12 supported this behaviour, it was missing from SLE 15 until this sprint.

Fine-tuning of the Requirements to Boot a System

In some situations, an extra partition or a given disk layout is needed to boot an (open)SUSE operating system. Like the separate /boot partition needed in some corner-case legacy scenarios, the ESP partition that must be mounted at /boot/efi in EFI systems or the PReP partition needed by some PowerPC systems. The partitioning proposal performed by the installer tries to ensure those booting requirements are met (so does AutoYaST in some cases) and our beloved Partitioner also includes some checks to warn you user if you forget to create or mount any of those partitions.

But in some situations, the installer was suggesting partitions with a suboptimal size or even partitions that were not strictly needed. On the other hand, the Partitioner was sometimes being too picky, warning about situations that were not such a big problem. So during this sprint we refined our list of booting requirements, updating the corresponding documentation, relaxing the partitioner checks and fine-tuning the proposal outcome. Specifically the PowerPC requirements were revamped based on the input from several experts and bug reports from manufacturers of some systems.

So no matter if you usually trust the installer proposal, if you like to handcraft stuff with the Partitioner or if you install using AutoYaST, the experience should be more smooth now, which fewer (if any) ugly surprises at boot time.

Mount Options Revisited: When the Old Demons Come back to Haunt You

Recently, we reintroduced per-filesystem-type defaults for mount options. We thought this would be a great chance to clean up old code that had become messy over time; we thought we could provide a clean, well-structured and easy-to-understand way to do this.

Then people began to test some more scenarios, and we found out the hard way that in some situations, those mount options depend on the mount point for various reasons: Some quirks of underlying kernel modules like the VFAT filesystem or the way systemd handles mounting the root filesystem during booting made this necessary.

So we had to bite the bullet and reintroduce some of that old code which was kind of messy; for example, for ext4 or ext3 root filesystems, we no longer specify any data=... mount options because this might make remounting the root filesystem read/write at boot time fail; in another case, a mkdir -p /boot/efi/EFI (when installing the boot loader) on a VFAT partition failed despite VFAT technically being case-insensitive (omitting iocharset=utf8 fixed this).

Lesson learned: Some messy old code is messy for a reason. Trying to streamline it may break some scenarios.

Conclusion

The answers to the initial questions are

  1. Yes.
  2. 说不定.
  3. Most of the time.

As the SLE release cycle is shifting from the "all features are mandatory" phase to the "all bugs are top priority" phase, expect less of feature news and more bugfix news in the next report, due in two weeks.

Highlights of YaST Development Sprint 50

February 9th, 2018 by

Sprint 50 is finished and we are ready to share with you all the highlights of this last sprint!

We are close to finishing features for the new SUSE Linux Enterprise 15, so this time we came up with a lot of information about the new storage, partitioner, firewall enhancements, and roles, which now is also present for the Desktop version of SUSE Linux Enterprise. Besides that, we fought another issue regarding memory consumption in Tumbleweed installation and once again we were able to solve it. So, let’s take a look into details!

Filesystem type specific default mount options in /etc/fstab are back

We reimplemented another feature that was still missing with the rewrite of the YaST storage stack: Reasonable mount options in /etc/fstab that may be different for each filesystem type. For example, for an Ext4 filesystem we used to set options data=ordered,acl,user_xattr by default. Of course, you can still change those options in the partitioner if the defaults don’t work well for you.

For most filesystem types, this was straightforward: Just filling a table with fixed default options. But for some others, most notably the legacy Microsoft filesystem types FAT/VFAT, this involved some not-so-trivial heuristics to figure out locale data so we could provide reasonable values for iocharset=... and codepage=...; those are necessary to recode filenames with non-ASCII international characters between whatever a MS Windows system might use and Linux; otherwise, you might not even see those files when mounted on a Linux system.

Resurrected sections in the Expert Partitioner

We keep bringing the functionality from the old Partitioner back to the Storage-ng reimplementation. During this sprint, it was the turn of the NFS and Device Graph sections.

The NFS case is a little bit special from the technical point of view since it’s the only section not managed directly by the Expert Partitioner. Instead, an embedded version of yast2-nfs-client is executed when visiting that section. The same approach has been followed in this reimplementation, which means NFS is the only section in which the old Partitioner and its Storage-ng clone will offer an absolutely identical experience and behavior (including both features and bugs). Just a heads-up: NFSv4 support is still not fully functional in libstorage-ng, so the check-boxes about the NFS version may be ignored until that support reaches your distribution.

On the other hand, the “Device Graph” section was, as most of the new Partitioner, rewritten from scratch. In addition to the already known graph showing how the system is going to look after applying the changes, it includes now a representation of the current system in a separate tab. Both graphs are interactive, double-click on any node takes the user to the corresponding section of the Partitioner.

As you can see in the previous screenshot, the usual button to export the graph to Graphviz format is not alone anymore. Its new friend allows exporting the graph to the very detailed XML format used by the YaST Team to reproduce test scenarios.

Of course, the “Device Graph” section is not available if using the text-based ncurses interface, as you can see comparing the left part of the following screenshot with the previous one.

Partitioner: LVM thin provisioning

Our new Expert Partitioner was already able to manage LVM volume groups and logical volumes since several weeks ago. But now it also recovers its great ability to work with LVM thin pools and volumes. LVM thin provisioning is a powerful technology, very useful in cases where you need to administrate storage resources for a large group of users. Basically, thin provisioning allows you to provide more storage space than it is actually available in the system. You can increase the real hardware storage only when it is really required. For more information about LVM thin provisioning, you can find a great guide in this link.

With the Expert Partitioner, you can create a thin pool over a LVM volume group in a similar way you create a normal logical volume. You only have to add a new logical volume and check the “Thin pool” option in the first dialog. Once the volume group contains at least a thin pool, you will be able to create thin volumes. Once again, the process is exactly the same. Add a new logical volume and select “Thin volume” option and in which pool to create it. The rest of steps are exactly the same as for creating a normal logical volume.

Apart from creating new thin pools and volumes, options for editing, resizing and deleting are now also available for all kind of logical volumes: normal volumes, thin pools and thin volumes. In the case of resizing a thin pool, you will be warned when the resulting pool is overcommitted. Take a look at the following screenshot.

Adapting partitions and logical volumes sizes in AutoYaST

In older versions, when the proposed partitioning in a profile and it did not fit in the disk, AutoYaST tried its best to reduce the biggest partition. A typical use case was to set the root (/) partition to a huge value, so the profile could be used in systems with different disk sizes (as AutoYaST will take care of reducing that partition to make it fit). To be honest, the best solution in that scenario would be to set the size to max, so AutoYaST will do what’s expected.

However, if for any reason the wanted layout does not fit, AutoYaST is now smarter about what to do: instead of blindly reducing the biggest one, it will try to reduce all partitions in a (kind of) proportional way, informing the user about the new sizes.

Small behavior change in AutoYaST installation

Up to now, not installable packages have been ignored silently by AutoYaST, which has been defined in the AutoYaST configuration file. From now on, the user will be informed that these packages cannot be installed.

System roles for SUSE Linux Enterprise Desktop 15

SUSE Linux Enterprise uses system roles to define, during the installation process, the packages to install on the system, so it can have the necessary software to play its “role”. This feature was already available for SUSE Linux Enterprise Server and now it is also available for SUSE Linux Enterprise Desktop. In order to choose a specific role for your system, you need to select a module or extension which contains this role during the installation process. There are four available roles for Desktop:

  • GNOME Desktop (Wayland): available when Desktop Productivity (on SLED) or Workstation Extension are selected.
  • GNOME Desktop (X11): available when Desktop Productivity (on SLED) or Workstation Extension are selected.
  • GNOME Desktop (Basic): available when the Desktop Application module is selected.
  • IceWM Desktop (Minimal): available when Basesystem module is selected.

Firewalld enhancements

During the last sprints, our team has been working hard in the integration of firewalld with AutoYaST and with the CWM library for opening ports / services in our different modules (http-server, squid, dns-server etc..). Here are some changes we did during this last sprint:

YaST firewall module is discontinued

As we now adopted firewalld in our distribution, there is no more reasons to have a YaST module for the firewall. Therefore, as you may see here, we discontinued the YaST Firewall module and we recommend to use firewall-config to configure your firewall via a user interface or firewall-cmd for the command line.

AutoYaST

A new AutoYaST schema has been defined for firewalld configuration although the features supported are very limited.

We can configure properties like the default zone, the service state, the type of packages to be logged and zones specific configuration like interfaces, services, ports, protocols, and sources.

For example, a SuSEFirewall2 based profile defining some interfaces, services, and ports in the EXT zone like this one:

<firewall> 
  <enable_firewall config:type="boolean">true</enable_firewall>
  <start_firewall config:type="boolean">true</start_firewall>
  <FW_DEV_EXT>eth0</FW_DEV_EXT>
  <FW_SERVICES_EXT_TCP>443 80 8080</FW_SERVICES_EXT_TCP>
  <FW_SERVICES_EXT_UDP>21 22</FW_SERVICES_EXT_UDP>
  <FW_CONFIGURATIONS_EXT>dhcp dhcpv6 samba vnc-server</FW_CONFIGURATIONS_INT>
</firewall>

will be translated to something like this:

<firewall>
  <enable_firewall config:type="boolean">true</enable_firewall>
  <start_firewall config:type="boolean">true</start_firewall>
  <default_zone>public</default_zone>
  <zones config:type="list">
    <zone>
      <name>public</name>
      <interfaces config:type="list">
        <interface>eth0</interface>
      </interfaces>
      <services config:type="list">
        <service>dhcp</service>
        <service>dhcpv6</service>
        <service>samba</service>
        <service>vnc-server</service>
      </services>
      <ports config:type="list">
        <port>21/udp</port>
        <port>22/udp</port>
        <port>80/tcp</port>
        <port>443/tcp</port>
        <port>8080/tcp</port>
      </ports>
    </zone>
  </zones>
</firewall>

SuSEFirewall2 based profiles will continue working although limited to the configuration currently supported by YaST.

During the autoinstallation, an error will be shown if the profile has some property not supported, however, we will able to continue with the installation and also a warning will be shown suggesting the use of the new schema even when all the properties are translatable.

Opening ports / services in zones through interfaces selection

In YaST, CWMFirewallInterfaces module provides a set of widget definitions for opening services in zones through a selection of interfaces (each interface belongs to a ZONE).

The module has been adapted to work properly with the new firewalld API.

If a firewalld service is not defined (probably because the package shipping the service definition has not been adapted yet), then the CWMFirewallInterfaces widget will show a list of missing services suggesting to deploy them to be able to configure the firewall.

Move from Xinetd to Systemd sockets

As requested by our friends from packages, now the future of starting service on demand is systemd sockets instead of xinetd. There are multiple reasons for that, but for us the most important one is that we can concentrate in one thing and do it properly.

It’s hard to support different ways to start services on demand and especially to handle such a situation when a user can set it up with sockets. Our old approach to make it happen was to set up the start of services on demand with xinetd. However, this approach is really hard to debug. Besides this problem, we also would like to unify our approach with the one suggested by packagers people.

So what have we done in this last sprint? The goal was to do some research, to create a proof of concept on one specific module and to find potential obstacles when unifying approaches to start services on demand. We found out that the conversion from xinetd to systemd sockets is quite easy, but also cannot be done automatically. Basically, what we need to do is to use systemctl call (which we already support) to activate the socket that is provided by the package instead of our old approach, which is to write a file into /etc/xinetd.d/ and to reload xinetd. Such a change will make our life much easier than previously.

The hardest part is that xinetd configuration file also often contains the service configuration that YaST can and would like to modify. This is no longer possible with systemd sockets and it is also the reason that we cannot automate this conversion.

So how will we proceed with the conversion of the YaST modules? We first checked all modules that use xinetd and we verified that, for the majority, the conversion is straight-forward. YaST also has a module for inetd configuration, which no longer makes sense, so the plan is to drop it in the near future and to allow socket activation in services manager module.

A problematic module is ftp-server, which supports two backends: vsftp and pure-ftp. This is problematic because pure-ftp does not support systemd-sockets, only xinetd. We plan to discuss how to handle it, as we are also not happy that we support two backends, because it makes the life of our users harder when they just want to quickly configure a ftp server. For SLE users we already support only vstftp, so if we agree on it, we will probably also support only this server in YaST. But right now nothing is set in stone.

Memory eating strikes again

Once again we have to mention our battle with memory consumption. This time the problem happened within the NET iso for Tumbleweed, which should need no more than 1 GiB of memory during the installation process.

In the beginning of our research, we had no suspicious code to look at, we just knew that it could be related to the fact that the NET iso was using the whole repository of Tumbleweed (over 60k packages) while the DVD iso uses only a subset of packages.

So which technique did we use to find the problematic part of the code? We changed logging in order to append the memory status of the whole process to each log line. By using this approach we quickly identified the problematic part of the code. This problematic code was responsible only for logging, but as it was searching for all packages and loading all packages properties, it was consuming a huge amount of data and causing this memory issue.

As a hotfix, we removed this logging from packages and kept it only for patterns and products, which is more important for us and has low memory consumption. We also looked into all code that searches through all packages properties and we reduced it as much as we could. Finally, we forced the Ruby garbage collector to run before the processes of disk preparation and rpm installation, reducing the chances that the memory consumption of the installation process goes up.

Once again we had a happy end and we are now able to install the NET iso with 1 GiB of memory when using the graphical installer.

Concluding

As we’re getting closer to finish SLES 15, we’re getting busier and busier with features and bugs to finish. Therefore, YaST team is already working hard on Sprint 51 and we’re looking forward to come back to you in two weeks with much more cool stuffs that we’re doing. Meanwhile, have fun and stay tuned!