Home Home > 2017 > 09
Sign up | Login

Archive for September, 2017

SUSE Support Lands Upstream In cloud-init

September 21st, 2017 by

Well it’s been many many years and many many releases that we’ve been carrying a large number of patches for the cloud-init package in openSUSE and SUSE Linux Enterprise. I remember the first semi serious implementation of SLES support happened when I worked with HP to get SLES into the HP Public Cloud offering, which was based on OpenStack. The offer was eventually named Helion Public Cloud and then eventually shut down. Yes, it’s been many many years and I have received many questions about when is SUSE support going to be upstreamed, and my answer was always, “when I get around to it“. Well, it finally happened, in big part thanks to the cloud-init summit which was held for the first time earlier this year. Google in Seattle was a great host and I very much appreciate that I was invited.

Anyway, long story short spending some face time with other contributors and working out the kinks that existed in the pipeline worked wonders.  Rather than sending a small patch here and there the main implementation for openSUSE and SLE, lots of code, were accepted shortly after the cloud-init summit and over the last couple of days another couple of patches took us another step forward.

There are a few more loose ends that need work but with 17 patches removed from the package build, currently building in Cloud:Tools:Next in OBS we’ve made major progress.

Well, I for one am happy about this, and those that want to install from source can do so and have openSUSE and SLE support working from the upstream sources and not just from the packages included with openSUSE and SLE.

Thanks to Canonical for organizing the summit to get everyone together and thanks to Google for hosting the summit.

Oh and before I forget, getting the changes accepted was not the only major step forward, openSUSE Leap 42.3 will, in the not too distant future, like in the next couple of days, be integrated into cloud-init testing using containers the lxd project builds, go figure who knew these even existed.

Highlights of YaST Development Sprint 43

September 21st, 2017 by

The summer is about to end (in Europe) and it is time for another YaST Development Sprint report. As usual, storage-ng has been one of the stars of the show, but new installer features for upcoming SUSE/openSUSE versions have received a lot of attention too. Also CaaSP 2.0 got some love from us during this sprint.

storage-ng: Udev mapping and ARM64 support

The new storage layer is getting better everyday. After the big amount of work that came with the re-implementation, the team is trying to unleash the power of the new design.

Udev Mapping is Back

The bootloader module supports using persistent device names provided by Udev. It is a pretty useful feature that comes in handy in many situations but it was missing in the storage-ng based version of the module.

But fear not: this feature has been re-implemented taking advantage of the much improved API of the new storage layer. And that’s not all: the team also took this opportunity to clean up some code and document the strategy for picking Udev device names in a proper way.

Do you want more details? Here you have them. Let’s start with the scenarios we support:

  • S1: Disk with the booting configuration is moved to different machine
  • S2: Disk dies and its content is loaded to new disk from a backup (because you have a backup, right?)
  • S3: Path to disk breaks and is moved to different one

Given these scenarios, let’s have a look at the strategies:

  • If the device has a filesystem with its mount_by, do not change it.
  • If the device names includes a by-label, just use it. This behaviour enables us to handle the three scenarios.
  • If there is by-uuid, then use it. It can also handle the three scenarios.
  • If there is by-id, use it. It can handle S3, but not always.
  • If there is by-path, just use it. It is the last supported Udev symlink and, at least, it will prevent the name changing during boot.
  • As fallback, just use the kernel name (for instance, /dev/sda3).

storage-ng now works also on ARM64

For quite some time, the storage-ng code has been tested (and worked) on x86_64, ppc64 and s390x architectures. Now, we have added aarch64 to match the list of supported architectures in SUSE Linux Enterprise and openSUSE Leap.

Speeding up the Service Manager Startup

As you may know, YaST includes a nice module for managing systemd services. Compared to systemadm, included in the system-ui package, there are some key differences:

  • It displays only services, not other unit types such as sockets.
  • It can enable/disable them for the next boot.
  • It works, as any YaST module, in textmode (even on 80×25 terminals).

Some time ago we got a report that the presented information was inaccurate in some corner cases. We fixed that but also made the module a lot slower at the same time. It did not take long for that to be reported.

We tested the following scenario: 286 services (SUSE Linux Enterprise 12 SP3 with nearly all software patterns installed) on a not very fast virtual machine. Normally, you should have fewer services and probably a faster system, but we wanted to fix the issue even for the worst scenario.

After analyzing the problem, we found out the root cause: too many calls of systemctl, at least 3 times per service (show, is-active, is-enabled). With a couple dozen milliseconds per call, it quickly adds up.

The fix was to combine all show calls into one and correctly interpreting the ActiveState property to eliminate all of the is-active calls. But you want to know the numbers, right? After the fix, the startup time went down from 69 to 15 seconds (bear in mind that it is a slow virtual machine).

So even if your system is not that slow or you have fewer services installed, you may benefit of a shorter startup time for this module.

Multi Repository Media

There are some hidden gems in YaST that are maybe not well known although they have been there for a long time. One of those features is support for multi repository media.

What actually is a multi repository medium? Imagine a CD or DVD with several independent repositories. The advantage is that, if you want to release several add-ons, you can put them all on a single DVD medium. Really nice stuff, isn’t it?

Up to now YaST added all repositories found on the medium automatically without any user interaction. In this sprint we have added a new dialog into the workflow which asks the user to select which repositories should be used. Of course, if there is only one repository, it does not make sense to ask and the repository is added automatically.

Selecting which add-ons should be installed

i18n support for CaaSP 2.0

On June 2017, SUSE released the first version of the promising SUSE CaaS (Container as a Service) Platform. The YaST team actively worked on this project by adding several new features to the installer, like the one-dialog installation screen.

That very first version of CaaSP was only available in English. However, version 2.0, which is around the corner, will support more languages. For the YaST team, it means that we needed to add the language selector to the
installer, as you can see in the screenshot below, and to mark every string in the yast2-caasp for translation.

YaST2 CaaSP features a language selector

Finally, if you are interested in CaaSP, maybe you would like to check out Kubic, its Tumbleweed based variant.

More bug fixes

Apart from the new and shiny stuff, the YaST team was able to fix quite some issues during this sprint. Let’s have a look at some of them.

Taking Care of Small Details

Not enough space for device name

Usability is a critical point for a project like YaST. From time to time, we receive a bug report about some usability problem that needs to be addressed and we took them very serious. In this case, the bootloader module had a problem when showing long device names in the dialog to change the order of disks. So we just needed to do a minor fix in order to ensure that there is enough space.

Device name is shown properly

Please, keep reporting usability related issue you find in order to make YaST even better.

Fixing the INSECURE mode

It sounds scary, but YaST supports an insecure mode during installation. What does it mean? YaST is pretty flexible when it comes to system installation. You, as a user, has the power to modify/tweak the installer (using a Driver Update Disk), add custom repositories, etc. And sometimes it can be useful to skip some checks.

Some time ago, libzypp introduced a new callback to inform about bad (or missing) GPG signatures. This callback was properly handled by AutoYaST but it was ignored in regular installations, so the user always got the warning about the failing signature, even when running on insecure mode.

Now the problem has been fixed and you can run the installer in insecure mode if you want to do so.

Learning about FCOE

YaST deals with a lot of moving parts and, although it can be daunting for the newcomer, it also has a bright side: we regularly learn new stuff to play with.

In this case, Martin Vidner, one of our engineers, had to deal with a fix related to the Fibre Channel over Ethernet support in YaST. But instead of blindly applying the patch that we already had, Martin decided to learn more about the topic sharing his findings in its own blog. Sure it will be a valuable resource to check in the future.

Highlights of YaST Development Sprint 42

September 7th, 2017 by

Kids started school in Prague, and we’re energized to bring you news from the YaST team.

This sprint we made progress in:

  • selecting one of several products to present on a DVD
  • adapting to the switch to the rpm-md package metadata format, regarding licenses and the Beta notice
  • notifying about not being able to support ReiserFS also in the case of an upgrade via AutoYaST
  • not reporting missing optional patterns

As always, the new storage stack deserves a separate section.

  • Expert partitioner has a better initial summary and it handles Btrfs subvolumes better.
  • Snapper can create and restore snapshots now.
  • A more flexible installation proposal is being developed.
  • It has landed in SLE15/Leap15 already.

Selecting the base product in the first screen

As we reported three sprints ago, YaST will support having multiple products on the same installation medium. This feature will allow to ship several SUSE products on the same DVD asking the user which one to install.

But, as an (open)SUSE user, you may know that the first screen of the installer allows you to select the language/keyboard and, additionally, it shows the product’s license. But which license should we show for multi-product media?

After asking our UX expert, we decided to allow the selection of the product in the welcome screen as you can see in the image below.

Obviously, this behavior applies only to multi-product medias. When using a single product one, the license is shown directly in the welcome screen.

Finally, as developers, we would like to highlight that we mainly re-implemented the welcome screen using modern YaST techniques (like our object-oriented API to YaST widgets), improving test coverage and code quality.

Getting licenses from repositories

In the sprint 40 report, we announced that YaST was dropping support for SUSE tags because the plan is to use RPM metadata and packages to store all that information (licenses, release notes, etc.).

During this sprint (and the previous one) we focused on improving how base product licenses are handled. Until now, licenses lived in a tarball which was included in the installation media. But that is not the case anymore: now YaST relies on (the awesome) libzypp to get products licenses directly from the repositories. There are still some rough edges: for instance, multi-language support should be improved, but we will tackle them pretty soon.

Finally, bear in mind that only base product licenses have been adapted, but we plan to do basically the same for modules, extensions and add-ons.

Updated README.BETA Support

The YaST installer supports displaying the README.BETA file from the installation medium. This file is added during Beta phase so the users know this is not the final release. This is also useful after the final product is released if you by mistake boot the old medium.

Unfortunately the original YaST code supported only the so called SUSE-Tags repository format which was used for the CD/DVD media. However, the new SUSE Linux Enterprise media will use the RPM-MD format which is something completely different.

In this sprint we have added the support for the RPM-MD format and additionally we display the Beta warning popup also in the AutoYaST installations. Of course, blocking AutoYaST with a popup would be a bad idea so in the AutoYaST mode the popup is automatically closed after a timeout. (The default is 10 seconds but can be configured in the XML profile.)

Drop support for ReiserFS autoupgrade

The drop of ReiserFS support for manual installation was announced in our last post, and now is the turn to autoupgrades.

With SUSE Linux Enterprise 15 the autoupgrade will be blocked in case of ReiserFS presence were detected suggesting a manual conversion to another filesystem.

Software proposal does not report missing optional patterns

YaST guides you through the installation process of your system making you a proposal based on the different selections like the product, addons, system role, desktop etc.

This proposal contains some software patterns that are mandatory and some that are optional but only the mandatory ones will be reported when the proposal is shown.

During this sprint we solved a bug which reported not only missing mandatory patterns but also optional ones.

The Storage Stack (storage-ng)

Storage reimplementation: Expert Partitioner

As you probably already know, in the YaST team we are rewriting our powerful expert partitioner tool from scratch to adapt it to the new storage layer. Sprint by sprint we are bringing back some of the many great features the expert partitioner offers, and this time it will not be different.

Now, when entering to the expert partitioner, all available storage devices are presented in an initial summary. Moreover, in this first screen you could find an option to rescan your devices, which allows the expert partitioner to be aware about the changes in your system, for example when you plug in a USB stick.

The management of Btrfs subvolumes was also improved during this sprint. Now, you will be alerted when a new subvolume is shadowed by an existing mount point. Moreover, some subvolumes could be automatically deleted or created when a mount point changes in the system.

Snapper can snap again

We reintroduced one of the most important features about using Btrfs: Being able to create snapshots, so you can go back to a previous state of the system if anything went wrong during a package upgrade or when you changed anything about your system configuration. This means we are now again setting up and configuring snapper correctly, installing everything into a subvolume and creating an initial snapshot when the installation is complete.

Why install into a subvolume in the first place? When snapshots are made in the future (e.g., during package upgrade or installation) and at some point you decide to roll back to one of those snapshots, you might want to delete previous snapshots to save disk space. If we didn’t install into a subvolume, the initial files would always be left over and consume disk space, so there would be a considerable resource leak.

Twisting the storage proposal

As you should know, one important part of the rewritten storage stack is the partitioning proposal. The old one was designed with a rather narrow scope in mind. Targeting desktops and old-school servers, it always proposes a root file-system, a swap volume and an optional separate /home. That’s not the most useful schema for new innovative products with a different focus like SUSE CaaS Platform, openSUSE Kubic, SUSE Manager, etc.

As we remind quite often, all the products supported by YaST (SLES, SLED, Leap, Tumbleweed, Kubic, CaaSP, SUSE Manager, SLES4SAP, you name it) share an identical installer. That installer is fully configurable using the file control.xml that is included in the media and that allows to define the sequence of installer steps, the default values and much more.

One of the goals of the new storage proposal is to give product creators and release managers more freedom and flexibility configuring the behavior of the guided setup. And for that we need a better format for control.xml, so they can express themselves with fewer limitations.

Of course, that new format is not something for the YaST team do decide on its own, but something being designed openly in collaboration with all the involved parties. To make easier for anyone to jump into the subject we have prepared this detailed document which includes all the present and historical information to understand the topic, as well as an explanation of the new format with several examples based on existing or hypothetical use cases.

As everything in YaST, that document is alive and expected to continue changing and evolving based on everybody’s feedback and contributions. So feel free to take a look and suggest improvements or future use cases we may have overlooked.

The storage stack is dead, long live the storage stack

In our previous sprint report we told you we were working to integrate the new storage stack into the future SLE15 and openSUSE Leap 15 codebase. Now the submission process is over and the preliminary images of both future distributions are fully based on libstorage-ng. For openSUSE Tumbleweed, that is the Staging:E project. That means we now have many more eyes looking into it, finding bugs, pointing what is missing and providing feedback about the new behavior. Of course, every couple of eyes comes with a little bit more of pressure for the YaST Team to get things done as soon as possible, but also with a pair of hands to help us getting there.