Home Home
Sign up | Login

Highlights of YaST development sprint 32

March 22nd, 2017 by

To make sure you didn’t missed us too much, in our latest blog post we summarized all the YaST-related projects worked during Hack Week 15. But after all the fun, it was time for… more fun! So let’s take a look to what the team has delivered on this first sprint after Hack Week 15.

Storage reimplementation: encrypted proposal without LVM

One of the known limitations of the current installer is that it’s only able to automatically propose an encrypted schema if LVM is used. For historical reasons, if you want to encrypt your root and/or home partitions but not to use LVM, you would need to use the expert partitioner… and hope for the best from the bootloader proposal.

But the new storage stack is here (well, almost here) to make all the old limitations vanish. With our testing ISO it’s already possible to set encryption with just one click for both partition-based and LVM-based proposals. The best possible partition schema is correctly created and everything is encrypted as the user would expect. We even have continuous tests in our internal openQA instance for it.

The part of the installer managing the bootloader installation is still not adapted, which means the resulting system would need some manual fixing of Grub before being able to boot… but that’s something for an upcoming sprint (likely the very next one).

Improved add-ons listing for SLE12-SP1

The dialog in SLES-12-SP1 for selecting the add-ons after registering the system was originally designed just for a small list of add-ons. Unfortunately (or fortunately, depending on how you look at it), the number of add-ons grew over the time and it exceeded the original limit for the text mode UI.

The equivalent screen in SLE-12-SP2 is not affected by the problem because it uses a different layout with scrollable list. But the SP1 dialog looks like this.

Broken add-ons list in SP1

If you look carefully at the screenshot you will see that the Web and Scripting Module is missing in the list and the Back, Next and Abort buttons at the bottom are also not displayed.

The fix decreased the size of the Details widget and allowed displaying more items in each column. Now there is even free space for three more add-ons.

Fixed addons list in SP1

Moreover the dialog is now dynamic and checks the current size of the screen. If there is enough free space then the list is displayed in one column so the labels are not truncated and the Details widget size is increased back to the original size.

Add-ons list in SP1 with enough space

Storage reimplementation: Btrfs subvolumes

The management of subvolumes is one of those features that make Btrfs rather unique and that need special handling when compared to more traditional file systems. That was indeed one of the several reasons to rewrite libstorage – Btrfs subvolumes never fully fitted the philosophy and data structures on the old (current) libstorage and yast2-storage.

In this sprint we introduced support for subvolumes in libstorage-ng from the ground up, taking into consideration all the specificities, use cases and scenarios found in the past. And, hopefully, in a way that is also prepared for whatever the future brings.

The new functionality is already working and tested and it’s included in the latest versions of libstorage-ng, but is still not used in the proposal or any other part yast2-storage. You will have to wait another sprint to see more visible results. At least if “more visible” means screenshots. Meanwhile, if you like images you can always enjoy the graphs generated from the internal structures managed by libstorage-ng.

Internal subvolumes representation in libstorage-ng

Storage reimplementation: system upgrade

The new storage stack has been able to install an openSUSE system for quite some time already. While we keep improving that area, the next challenge was to make the upgrade from a previous openSUSE version also possible using our testing ISO.

That implies scanning the hard disks looking for previous installations, allowing the user to select one, mounting the corresponding partitions or LVM volumes, performing the update of every package and doing some final tasks like updating the bootloader configuration.

Following the iterative approach encouraged by Scrum, we focused in the first three steps, which is something that a user (or openQA, for that matter) can test and verify. So now we are able to detect and list pre-existing systems and start the upgrade process on the selected one. And we have automated tests in openQA to ensure it works across all the combinations of partition-based vs LVM-based layout and UUID-based vs name-based fstab file.

Add-ons can define new system roles

YaST is pretty customizable when it comes to adapt/modify the installation workflow. Among other things, add-ons are allowed to adapt the workflow (adding/removing steps), define new proposals, etc. And starting now, they can also define new system roles.

Let’s see an example of adding a new mail server role:

<update>
  <system_roles>
    <insert_system_roles config:type="list">
      <insert_system_role>
        <system_roles config:type="list">
          <system_role>
            <id>mail_role</id>
            <software>
              <default_patterns>base Minimal mail_server</default_patterns>
            </software>
          </system_role>
        </system_roles>
      </insert_system_role>
    </insert_system_roles>
  </system_roles>
</update>

<!-- Don't forget to add the texts -->
<texts>
  <mail_role>
    <label>Mail Server</label>
  </mail_role>
  <mail_role_description>
    <label>• Software needed to set up a mail server
• No production ready yet!</label>
  </mail_role_description>
</texts>

And now let’s see how it looks:

A role added by an addon

Which leads us to the next section…

The list of roles becomes responsive in text mode

A really nice thing about YaST is that it’s able to run in textmode, so you don’t need a graphical interface to install or configure your system. As a consequence, YaST developers need to keep certain limitations in mind when working in the user interface.

Now that add-ons can add new system roles, we noticed a potential problem in the dialog selection screen: we eventually will get out of space if more than one system role is added. So we decided to improve how system roles are displayed to make them fit in a 80×25 mode (that is, only 25 lines of text). Let’s see the changes with some examples.

This is how the screen looks by default, with a reasonably small set of roles.

Default system roles list

If the system detects there is no space to present all the information in such a convenient way, it removes all the spaces so at least the information is all there, even if it looks a little bit packed.

Roles list with no extra space

If even that is not enough, the extra descriptions are omitted, which gives us way more room.

Compact roles list

If roles don’t fit even without the descriptions, the introductory text will be also omitted which means we can present up to eighteen (yes, 18!) roles in the screen.

Storage reimplementation: guided setup mock-up

As explained in several previous reports, we are collaborating closely with SUSE UX experts to design the revamped interfaces of the installer’s partitioning proposal and the expert partitioner. We already showed you the document we used as a base to discuss the partitioning proposal, including the conclusions, and the first very simple prototype of the so-called Guided Setup.

During this sprint, that collaborative effort focused on defining exactly how every step of that wizard should work and look like. The goal was to get some interface mock-ups to be used as starting point for the upcoming sprint. More than ever, a picture (well, four of them) is worth a thousand words.

First step of the guided partitioning setup mock-up

Second step of the guided partitioning setup mock-up

Third step of the guided partitioning setup mock-up

Fourth step of the guided partitioning setup mock-up

Prevent the installation of CaaSP if Btrfs snapshots are not possible

CaaSP is a single purpose system, and having snapshots enabled is essential. So there’s now a check in place that will simply prevent you from going on with the installation if snapshots are disabled (for example, if the disk is too small).

Blocked CaaSP installation

Storage reimplementation: better handling of /etc/fstab and /etc/cryptab

For the new storage stack, we refactored the classes to handle /etc/fstab. While this would normally not be anything to write much about, we included intelligent handling for existing comments based on this standalone GitHub project.

This means that existing comment blocks at the start and at the end of the file remain untouched, and comments before any content entry remain attached to that entry; i.e. when that entry is moved around in the file (e.g. because of mount dependencies), that comment is moving along with the entry it belongs to. While this is not 100% fool proof, it is much better than the usual strategy to simply discard such comments when the file is rewritten.

Quite some adaptations and bugfixes for CaaSP

As you already know from previous reports and other sources, a considerable part of SUSE’s development firepower is focused on building the upcoming CaaSP. As part of that heavy development process, the YaST team invested a significant part of the sprint adapting YaST for CaaSP and fixing bugs introduced by previous adaptations. A large collection of changes here and there that are hard to summarize here but that help CaaSP to be a couple of steps closer to the final goal.

Keep it rolling!

We have already planned our next sprint which will hopefully bring more features to the new storage stack, CaaSP-related improvements, a surprise about AutoYaST and more stuff. And, of course, it will be followed by its corresponding report.

So see you in three weeks. Stay tuned and have a lot of fun!

Fun things to do with driver updates

March 16th, 2017 by

Today: But what if I need a new kernel?

A driver update (DUD) can of course update a single driver. But if that’s not enough and you need a whole new kernel to run an installation?

There are two parts to solve:

  1. replace the kernel used during installation and
  2. get the new kernel installed

We’ll need two tools for this (both available in Tumbleweed or here: mksusecd and mkdud).

1. Replace the kernel used during installation

For this it’s important to know which kernel packages you’ll actually need. Typically it will be kernel-default and kernel-firmware. But older SUSE distributions (SLE 11 comes to mind) had the kernel packages split into kernel-default and kernel-default-base – you’ll need them both.

To make things confusing, modern SUSE distributions also have kernel-default-base – but it’s an alternative to kernel-default. In this case we don’t need it.

If unsure, check kernel-default. If it contains the actual kernel (e.g. /boot/vmlinuz) then you don’t need kernel-default-base.

On some architectures modules are also taken from xen-kmp-default. If that’s important for you, you can add this package to the kernel list as well.

In fact you can add any number of kernel packages or kmps you like.

In the past, sometimes a different kernel flavor was used. For example PowerPC had kernel-ppc64 for a while. Simply use the flavor you need.

It’s a good idea to gather all the kernel rpms into a single directory for easier use:

> mkdir k
> cp kernel-default.rpm kernel-firmware.rpm k
> cp kernel-default-base.rpm k    # only if needed
# add any kernel-related rpms you need

Then, take your SUSE installation iso and run

> mksusecd --create new.iso \
  --kernel k/* -- \
  original_dvd1.iso

Note that the --kernel option accepts a variable number of arguments, so you have to add an isolated -- to terminate the argument list properly.

The output could look like this:

> mksusecd --create new.iso \
  --kernel k/* -- \
  SLES-11-SP4-DVD-ppc64-GM-DVD1.iso
kernel version: 3.0.101-63-ppc64 --> 3.0.101-94-ppc64
CHRP bootable (ppc64)
building: 100%
calculating sha1...

The command above will actually get the list of required modules from the old installation iso. If you are missing some driver or the new kernel comes with some additional driver, the module will not be added to the new iso.

But there’s the --modules option. It will add the listed modules together with any implicitly required modules via module dependencies.

For example, let’s add the airport wifi-module to our PowerPC iso:

> mksusecd --create new.iso \
  --kernel k/* \
  --modules airport -- \
  SLES-11-SP4-DVD-ppc64-GM-DVD1.iso
kernel version: 3.0.101-63-ppc64 --> 3.0.101-94-ppc64
kernel modules added:
  airport, cfg80211, orinoco
CHRP bootable (ppc64)
building: 100%
calculating sha1...

As you can see, it automatically adds orinoco and cfg80211 as well.

2. Get the new kernel installed

This is relatively simple. A driver update can do this:

> mkdud --create foo.dud \
  --dist sle11 \
  --install repo \
  k/*

This creates a driver update for SLE 11 (which also applies to SP4) and the kernel rpms are installed via an auto-generated add-on repo (--install repo).

Now we have the driver update that installs our kernel packages. But how do we use it?

We integrate it into our iso above!

> mksusecd --create new.iso \
  --initrd foo.dud \
  --kernel k/* -- \
  SLES-11-SP4-DVD-ppc64-GM-DVD1.iso

mksusecd has an --initrd option that directly accepts driver updates and integrates them into the iso.

3. Can I have a choice?

Maybe you just want to test this new kernel or sometimes need the old one and sometimes the new one. Can you make an installation iso that lets you choose the kernel?

Oh yes! 🙂

> mksusecd --create new.iso \
  --add-entry 3.0.101-94 \
  --initrd foo.dud \
  --kernel k/* -- \
  SLES-11-SP4-DVD-ppc64-GM-DVD1.iso

This does not replace the old kernel but adds a new boot entry Installation - 3.0.101-94.

So you can install with old or the new kernel.

YaST development during Hack Week 15

March 7th, 2017 by

During this Hack Week, some of our team members invested quite some time working in YaST related projects. But, what’s even better, some people from outside the team worked also in YaST projects. Thank you guys!

So let’s summarize those projects and also some extra ones from our team members.

Let’s Encrypt made easy in openSUSE

Daniel Molkentin enjoyed his first Hack Week working together with Klaas Freitag to bring Let’s Encrypt integration to openSUSE. Although they didn’t had experience with YaST development, they followed the YaST tutorial and created a new brand yast-acme module. It’s still a work in progress, but it looks promising.

Managing certificates

You can get more details reading the Daniel Molkentin’s blog post about the project.

Removing perl-apparmor dependency

Goldwyn Rodrigues worked to remove the dependency of yast-apparmor on (now obsolete) perl-apparmor. There’s quite some work to do, but he’s heading in the right direction.

Find out more about the project in the Hack Week page and in Goldwyn’s fork.

gfxboot for grub2

Steffen Winterfeldt was working in gfxboot2, an attempt to provide a graphical user interface for grub2. Although it’s still a work in progress, it looks really good: module works for grub2-legacy and grub2-efi, basic font rendering is in place, language primitives are partly implemented, the integrated debugger already works…

You can find further information in the projects page.

YaST Integration Tests using Cucumber

The YaST team is always trying to improve their testing tools so Ladislav Slezák has been working in a proof of concept to use Cucumber to run YaST integration tests and the results are pretty impressive. The prototype is able to test YaST in an installed system, during installation and it can be used even for plain libyui applications outside YaST.

Test adding repository

Check out the details at project’s page and at Ladislav’s blog.

Keep working on libstorage-ng

If you follow YaST development, you know that the team is making good progress towards replacing the old libstorage. And even during Hack Week, libstorage-ng got some love from our hackers.

Arvin Schnell implemented support for more filesystems in libstorage-ng: ext2, ext3, ReiserFS, ISO 9660, UDF and basic support for NFS.

Iván López invested his first Hack Week improving yast2-storage-ng. Read this description to find out more information about the changes.

And last but not least, apart from helping Iván with his project, Ancor González worked in a new approach for yast2-storage-ng: instead of just extending the API offered by libstorage-ng, the idea is to wrap the library so the Ruby code using yast2-storage-ng does not have direct visibility on the libstorage-ng classes and methods.

More Ruby in YaST

Josef Reidinger is trying to reduce the amount of code in other languages different than Ruby. He successfully replaced the binary y2base with a Ruby version (not merged yet) and reduced the amount of Perl code in the yast2 package.

You can check the progress in yast-ruby-bindings#191 and yast-yast2#540 pull requests.

Support for Salt parametrizable formulas

YaST2 CM was born in 2016 as a proof of concept to somehow integrate AutoYaST with Software Configuration Management systems like Salt or Puppet.

During this Hack Week, Duncan Mac Vicar and Imobach González were working to implement support for Salt parametrizable formulas. You can find out more information in this post at Imobach’s blog.

Other non-YaST projects

Hack Week allows us to work in any project we want, so there’re also some non-YaST related projects that we would like to mention.

Improved QDirStat

QDirStat is a Qt-based tool which offers directory statistics. His author, Stefan Hundhammer,implemented some new features (like a better behavior when hovering over a tree map or improved logging) and fixed some bugs. Find out more details in the project’s README and the version 1.3 release notes.

He also wrote an article about how to use the application in headless systems (no X server, no Xlibs).

Current version is already available at software.opensuse.org so you could consider giving it a try.

Learning new things

Hack Week is a great opportunity to play with new stuff. With that idea in mind, Martin Vidner learned about Android development and wrote detailed instructions for starting with that: Getting Started in Android Development: Part 1: Building the First App, Part 2: Publishing the First App and Part 3: Reducing Bloat.

Knut Anderssen decided to try also some mobile development and enjoyed playing around with the Ionic.

And that’s all! After Hack Week, we’re back to SCRUM and we’re now working sprint 32. We’ll give you additional details in our next “Highlights of YaST development” report.

Stay tunned!

Highlights of YaST development sprint 31

February 20th, 2017 by

As we announced in the previous report, our 31th Scrum sprint was slightly shorter than the usual ones. But you would never say so looking to this blog post. We have a lot of things to talk you about!

Teaching the installer self-update to be a better citizen

As you may know, YaST includes a nice feature which allows the installer to update itself in order to solve bugs in the installation media. This mechanism has been included in SUSE Linux Enterprise 12 SP2, although it’s not enabled by default (you need to pass an extra option selfupdate=1 to make it work).

So after getting some feedback, we’re working toward fixing some usability problems. The first of them is that, in some situations, the self-update mechanism is too intrusive.

Consider the following scenario: you’re installing a system behind a firewall which prevents the machine to connect to the outside network. As the SUSE Customer Center will be unreachable, YaST complains about not being able to get the list of repositories for the self-update. And, after that, you get another complain because the fallback repository is not accessible. Two error messages and 2 timeouts.

And the situation could be even worse if you don’t have access to a DNS server (add another error message).

So after some discussion we’ve decided to show such errors only if the user has specified SMT or a custom self-update repository (which is also possible). In any other case, the error is logged and the self-update is skipped completely.

You can find further information in our Self-Update Use Cases and Error Handling document.

During upcoming sprints, we’ll keep working on making the self-update feature great!

Configuring workers during CaaSP installation

While CaaSP release approaches, our team is still working hard to satisfy the new requirements. Thankfully, YaST is a pretty flexible tool and it allows us to change a lot of things.

For CaaSP installation, YaST features a one-dialog installation screen. During this sprint, configuration of the Worker role has been implemented, including validation of the entered URL and writing the value to the installed system. You can check the animated screenshot for details.

The CaaSP worker configuration

New desktop selection screen in openSUSE installer

The world of Linux desktop environments change relatively quick, with new options popping-up and some projects being abandoned. Thanks to the openSUSE’s community of packagers we have a lot of these new desktop environments available on the openSUSE distributions. But the status of those packages for openSUSE is also subject to changes: some desktop environments are poorly maintained while others have a strong and active group of packagers and maintainer behind.

The YaST Team does not have enough overview and time to watch all these desktop environment and evaluate which one is ready or not for being in the installer’s desktop selection screen. So the openSUSE Release Team decided to replace this dialog with something a bit more generic but still useful for newcomers.

They asked the YaST Team to come up with a new dialog featuring the two openSUSE main desktops (KDE Plasma and GNOME) and allowing the easy selection of other environments without reworking the dialog in the future. The goal of that new dialog was to replace the existing one you can see in the following screenshot.

The old desktop selection screen

We decided the new dialog should rely on patterns for several reasons. The main one is that the set of patterns is under the close control of the openSUSE community, which looks more closely than us to the desktop environments and their integration into the distribution. Moreover, each pattern specifies its own icon and description that can be somehow re-used by the installer.

We also took the opportunity to merge this almost empty and outdated dialog with the new one.

The old additional repositories screen

Addons are no longer produced for openSUSE, so only the second checkbox made any sense nowadays. Moreover, the functionality of that second checkbox directly influence the available selection of patterns, so it made more sense to merge everything in a single screen that keeping an extra step in the installation just to accommodate a checkbox.

So we sent a proposal for the new dialog to the opensuse-factory mailing list and, after implementing many of the ideas discussed there (like better wording or using a button instead of a checkbox for the online repositories), this is the new dialog that replaces the two ones mentioned above.

The new desktop selection dialog

Selecting “custom” will take the users to the already existing patterns selection screen. Just in case you don’t remember how that screen looks like, you can check this image.

The patterns selection screen

If these screenshots are not enough to make your mind about the change, you can check the following animation, in which KDE Plasma is initially chosen to be changed at a later point (going back in the workflow) to LXQt.

Desktop selection animation

It will take some time before the changes hit the Tumbleweed installer, since they obviously have a non-trivial impact on the openQA tests, that will need some adaptation.

We would like to thank everybody that contributed to this new feature by providing feedback and suggestions through the mailing list. Once again, the openSUSE community has proved to be simply awesome!

Storage reimplementation: improved handling of logical partitions

Our reimplementation of the storage layer keeps getting improvements here and there. With the base x86 case working, we are now implementing the tricky parts, like the partitions renumbering that takes place when dealing with logical partitions in a MBR style partition table.

With GPT partition tables or with primary partitions in a MBR partition table, the partition gets a number (like sda1) when it’s created and keeps that number for its whole lifetime. But logical partition gets constantly renumbered when other partitions are created and destroyed. We need to know in advance what device name every partition will get in order to configure everything beforehand and only commit the changes to the disk when the user clicks in the “install” (during the installation process) or “commit” (if running the expert partitioner).

Now, libstorage-ng is able to simulate in memory the re-numbering process, so we can calculate all the settings related to partitioning (like the configuration of the bootloader) before really touching the disk.

Making kdump work in CaaSP

When you enable the Kernel Crash Dump (kdump), you probably expect that you will get a kind of core dump, like you do for regular processes. What you might not expect is an automatic reboot. That is a nice bonus. If you are tired of waiting for an actual kernel crash, you can test your kdump setup by triggering a crash yourself. Just run this as root:

echo c > /proc/sysrq-trigger

The way kdump works is by allocating some extra memory at boot time and putting a second kernel+initrd there. When the main kernel realizes it is crashing, it transfers control (by kexec) to the other one, which has only one purpose, to dump the memory of the crashed kernel.

In the upcoming Containers as a Service Platform, kdump was not working because the root filesystem is read-only there and we were not able to create the kdump initrd. We fixed it by creating it just after the RPMs are installed, when the FS is still read-write.

Fixes for Snapper in CaaSP

Kdump was not the only component affected by the fact of having a read-only filesystem in CaaSP. Snapper was also running into problems when trying to execute the rollback and cleanup operations. Now the problems are fixed. While working on that we did enough interesting findings to deserve a separate blog entry. So you can expect a new entry in the Snapper.io blog soon.

Snapshot-related improvements in the expert partitioner

While we work on the new storage system, we are still taking care and improving the current one that will continue to be shipped with SLE12-SP3, SUSE CaaSP and openSUSE Leap 42.3. During this sprint we introduced a couple of usability improvements in the expert partitioner related to Btrfs subvolumes.

First of all, we moved the “Enable Snapshots” checkbox that was pretty much hidden in the “Subvolume Handling” dialog to the place where it really belongs – below the selector of the file-system type.

New location for enabling snapshots

In addition, we added the warning you can see in the screenshot below for users enabling snapshots in a potentially problematic setup.

Warning for snapshots in a small partition

Bring back the beta warning on CaaSP

And talking about warnings, we improved the CaaSP installation dialog to show the Alpha/Beta product warning at the beginning. After changing the CaaSP installation to use the all-in-one dialog described in our previous post, this feature was lost as it is part of the initial EULA dialog. Now it is back and the users should now properly see the current product status.

The CaaSP alpha/beta warning pop-up

Storage reimplementation: encrypted LVM proposal

Back to our storage layer reimplementation, the new system is now able to propose a setup with encrypted LVM. Since some sprints ago, we were already able to propose a partition-based and a LVM setup. That means the new proposal is now feature-pair with the old one, with the only exceptions of Btrfs sub-volumes (that shouldn’t be a big challenge) and s/390 storage (that is still not properly managed by the underlying libstorage-ng).

The bright part is that the new code is written with re-usability in mind, which means implementing an encrypted partition-based proposal (with no LVM) would be trivial. The new code is 100% covered by automatic unit tests and scores to the top in all the automatic quality checkers we have run (like Rubycritics, CodeClimate, and Rubocop).

So now we are prepared for whatever the future brings, having lost no single feature in the way.

Storage reimplementation: prototype if the UI for the proposal settings

The next challenge is to make the power of that new storage proposal available to the users through the user interface. In the previous post we presented the document describing the general ideas we wanted to discuss with UX experts.

It’s our pleasure to announce we met the experts and we very easily reached an agreement on how the user interaction should be. During this sprint we already implemented a prototype of the future proposal settings wizard that is, as usual, included in our testing ISO.

Now that we have solid foundations, it’s very likely that the following sprint will result in the fully working wizard, with almost-final wording and design.

Support for CaaSP added to the AutoYaST integration tests

While fixing a problem with AutoYaST and CaaSP, we decided to extend our AutoYaST integration testing tool to support CaaSP. But, as a side effect, we also made some additional improvements that were long overdue (like improving the procedure to download ISOs, reducing timeouts, etc.).

Now, our internal Jenkins instance takes care of running AutoYaST integration tests for the development version of CaaSP (as it already does for SLE 12 SP2).

Services manager moved to first auto-installation stage

AutoYaST allows to define a set of services to be enabled/disabled in the installed system, applying this configuration during the 2nd stage (after the first reboot).

Unfortunately, this approach won’t work for CaaSP because, in that case, the 2nd stage won’t be available. For that reason, during this sprint, we have adapted AutoYaST to write services configuration during 1st stage.

As usual, not only CaaSP, but other future (open)SUSE versions will benefit of this change.

Faster YaST installation when the release notes cannot be downloaded

Sometimes a very small simple change in a program makes a very noticeable impact in its everyday use. That’s the case of a fix implemented during this sprint. YaST usually re-tries to download the distribution release notes several times if the first attempt was unsuccessful. Although that’s correct from a general point of view, there are cases in which retrying makes no sense and only delays the installation. We have now added failed DNS resolution to that list of cases, which should result in a noticeable faster installation in many scenarios.

Moreover, in the case of AutoYaST we now skip downloading the release notes completely. Downloading them don’t make us much sense in the kind of unattended scenarios handled by AutoYaST and skipping this step and all the possible associated problems result in a more robust process.

Better continuous integration for Libyui

In the previous sprint we switched to Docker at Travis so we could build and test our packages in the native openSUSE system instead of the default Ubuntu. This sprint we did the same change also for the Libyui library which implements the UI part of YaST.

Originally we could not build the Libyui packages at Travis as either the required build dependencies were missing or were present at a very old unusable version. With this switch we can easily use the latest packages from openSUSE Tumbleweed and thus enable Travis builds for all Libyui packages.

See you after Hack Week!

As already mentioned, this week is Hack Week 15! So our Scrum process will be on hold for some days. That doesn’t necessarily mean the YaST development will stall. Since there are quite some YaST-related projects in the Hack Week page, you can expect YaST to simply go in unexpected directions. 🙂

And remember Hack Week is not a SUSE internal initiative, we are open to collaboration from anybody within or outside the company. So jump in and have fun with us!

Fun things to do with driver updates

February 16th, 2017 by

Today: update the update process!

Yesterday a colleague asked me if it would be possible to apply a driver update (DUD) to the rescue system. He wanted to use a new btrfsprogs package.

My immediate reaction was: no, you can’t do it. But then, there’s no technical reason why it shouldn’t be possible – it actually nearly works. The updates are downloaded as usual – just not applied to the rescue system.

So I thought: “Why not make a driver update so driver updates work also for the rescue system?”

Here’s how I did it.

First, let’s find out how driver updates are usually applied. The code is here:

https://github.com/openSUSE/installation-images/blob/master/data/root/etc/inst_setup#L84-L87

We need just these three lines:

for i in /update/[0-9]*/inst-sys ; do
  [ -d "$i" ] && adddir "$i" /
done

linuxrc downloads the driver updates and stores them in an /update directory. One (numbered) subdirectory for each update.

It obviously uses some adddir script. So we’ll need it as well. Luckily, it’s not far away:

https://github.com/openSUSE/installation-images/blob/master/data/root/etc/adddir

Next, we’ll have to find the spot where the rescue system is set up. It’s done in this script:

https://github.com/openSUSE/installation-images/blob/master/data/initrd/scripts/prepare_rescue

Let’s do some copy-and-paste programming and insert the above code near the end of the script. It then might look like this

# driver update: add files to rescue system
if [ -d /mounts/initrd/update ] ; then
  cp -r /mounts/initrd/update /
  for i in /update/[0-9]*/inst-sys ; do
    [ -d "$i" ] && /mounts/initrd/scripts/adddir "$i" /
  done
fi

Some notes:

  • You have to know that prepare_rescue is run as the last thing before we exec to init. So everything is already in place, the left-over files from initrd are mounted at /mounts/initrd and will be removed at the end of the script.
  • This means we have to copy our updates into the new root directory, else they will be lost.
  • Also, we plan to make the adddir script available at /scripts/adddir by our driver update (see below).

Now let’s create the driver update:

mkdud --create dud_for_rescue.dud \
  --dist tw --dist leap42.1 --dist leap42.2 --dist sle12 \
  --name 'Apply DUD also to rescue system' \
  --exec 'cp adddir prepare_rescue /scripts' \
  adddir prepare_rescue

Here’s what this call does, line-by-line:

  • the fix works for all current SUSE distributions, so let’s support them
  • give the driver update some nice name
  • this command is run right after the driver update got loaded; we copy the scripts out of the driver update to their final location
  • add adddir and our modified prepare_rescue script

Here is the result: dud_for_rescue.dud.

Now, back to the original problem: how to use this to update a package in the rescue system? That’s easy:

mkdud --create new_btrfs.dud \
  --dist sle12 \
  dud_for_rescue.dud btrfsprogs.rpm

creates a driver update (for SLE12) that updates btrfsprogs also in the rescue system.

Highlights of YaST development sprint 30

February 3rd, 2017 by

This is our first post in 2017 and looks like we must start apologizing. In our previous post we promised news about this blog, but the administrative part slowed us down and the surprise is still not ready. On the bright side, we have quite some news about YaST. So let’s go for it!

One-click system installation for CaaSP

As you may know, SUSE has been working on making containers easier, with SUSE Container as a Service Platform. We have referred to it in several previous posts using the CASP acronym, although nowadays the correct one is CaaSP (maybe we could sell the new “a” as a shiny feature 😉 ).

Part of this upcoming product is also an interactive installation option using the good old YaST. CaaSP uses a limited subset of the SLE possibilities and we wanted to make the installation simpler to reflect that. So we reduced the number of dialogs you have to click through… to one!

One-click installation in CaaSP

As you can see, it is at the expense of stuffing the screen with more widgets than usual. Still, the only part where you must make a decision is the root password.

We expect that most of the CaaSP installations will actually not use this, because they will be done automatically with AutoYaST. But still this should be useful when you are only getting started.

Refining the read-only installation proposals

It was possible to make a proposal “read-only” for some time already. However, its black-and-white logic was not sufficient for some use cases. So, it was redesigned and you can mark a proposal hard read-only or soft read-only. The difference is that users will never be able to change hard read-only proposal. However their will be able to modify a soft read-only proposal if the proposal reports an error. It can be useful e.g. for error recovery in software proposal. It has been implemented originally for CaaSP, but it will be available for SLE-12-SP3 and Leap 42.3 too.

Installing directly from a repomd repository

When you install (open)SUSE you have up to now needed a specially prepared install repository. In addition to the repository with the RPM packages, it needed a bunch of specially prepared files containing the installation system and our beloved YaST installer.

That’s all gone now!

You can now point the installer to any plain repomd repository. For this to work you have to point the installer to the repomd repository and to the installation system (they can be completely separate now).

For example:

install=http://download.opensuse.org/tumbleweed/repo/oss instsys=disk:/boot/x86_64/root

In that example, we install Tumbleweed from the openSUSE website and use the installer from some local media (maybe the NET iso).

To make things even easier there is now a regular package (tftpboot-installation-openSUSE) that contains the installation system and some sample config files.

Check out this linuxrc documentation for technical details.

Storage reimplementation: removing stones from the installation path

In our latest post, we presented the dedicated openQA instance contantly testing the new storage layer implementation. It still doesn’t run exactly the same tests than openQA.opensuse.org because not all technologies and operations are supported yet in the new yast2-storage. But now we are a couple of steps closer to run the full-blown tests also in our dedicated instance.

During this sprint, the partitioning proposal gained the ability to deal with disks not containing a partition table in advance (it always proposes to create a GPT one in that scenario) and the software selection proposal learned how to use the new storage API, so it can properly inspect the system and the associated error pop-ups are gone from the installation workflow.

More power to the system roles

We keep extending the capabilities of the system roles, now with the ability to specify some systemd services to enable. As the roles can define which software gets installed in the system, it made sense for them to also be able to specify the desired status for the services included in that software

For example, it would be possible for a given product (let’s say a customized openSUSE) to define a “static web server” role. Choosing that role during installation would result in a system with a HTTP server already installed and enabled, so the user just need to copy the files to be served into the right directory.

Expert partitioner is now less restrictive with encryption

Setting up an encrypted LVM was always pretty easy when using the automated storage proposal – simply select “encrypted LVM” at the proposal settings and you are done.

But doing that manually was almost impossible: The expert partitioner wouldn’t allow any of the system mount points (“/”, “/usr”, “/var”, …) on any encrypted partition, and it also wouldn’t allow to encrypt, but not format a partition of type “LVM” for use as an LVM physical volume.

Both restrictions are now lifted; you can now create an LVM physical volume with encryption, or you can do the encryption layer on the logical volume if you prefer. And you can create an encrypted plain partition with a filesystem directly on it without LVM.

Over the years, Grub2 learned how to do that, so you don’t even need a /boot partition anymore. For the time being, you’ll need to enter the encryption password twice, though: once at the Grub2 prompt and once later at the graphical console so systemd can mount those filesystems. Our base system developers are working on a secure solution to avoid that.

Migrate Travis CI to Docker

That’s actually not a change in YaST itself, but in its development infrastructure. Still, we believe it would be interesting for the average reader of our blog.

So far we used Travis CI for building and testing the commits and pull requests at GitHub. But the problem was that by default Travis runs Ubuntu 12.04 or 14.04 at the build workers. That had several drawbacks for us, since compiling and testing YaST on Ubuntu is not trivial and the result is not always 100% equivalent to openSUSE. All this meant extra maintenance work for us.

Fortunately Travis allows using Docker containers at the workers and that allows using basically any Linux distribution. This sprint we spent some time converting the Travis configuration to use a dockerized openSUSE Tumbleweed at Travis.

From Github to Travis thanks to Docker

The work was successful, we switched all YaST modules to use this new builds and the result is already paying off at several levels, although it took us over 100 pull requests (all of them manually tweaked and reviewed) to make it happen.

The current solution is documented and we had also a short internal presentation about this change. The notes from the presentation are available here.

Improved continuous integration for Snapper

We also enabled Travis integration with Docker for Snapper. As you may know, Snapper is a portable software that has always offered packages for many Linux distributions in the filesystems:snapper OBS repository.

So we took the continuous integration one step further and enabled Travis builds on more distributions, currently we build for openSUSE Tumbleweed, openSUSE Leap 42.2, Fedora 25, Debian 8 and Ubuntu 16.10. You can see an example build here or more details in the documentation.

Example build result of Snapper at Travis

That means we can ensure that the package still builds on all these distributions even before merging a pull request!

Better integration with systemd for YaST Services

Systemd recognizes many possible states for a service beyond the classic Unix enabled/disabled and running/stopped, and that list of possible states grows with every systemd release. In the past YaST have had some issues displaying the services status.

Now the problems are fixed by delegating to systemd the conversion from the concrete state to the good old known Unix equivalent. So the user now gets more precise information about all services running on the system.

Storage reimplementation: redesigning the installation user experience

In the latest post we showed you the document we were using as a base to discuss the new expert partitioner UI with usability experts. Now it was turn for the proposal settings dialog. We collected the current state, had a very productive discussion and ended up with a proposal for a new interface. You can check the resulting document covering all that.

As mentioned, the SUSE UX experts will use that document as a starting point to design the final interface. But we want the process to be as open as everything around YaST, so feel free to provide feedback.

Reading product renaming information from libzypp

When performing an upgrade, YaST needs to know whether a product is renamed or replaced by a another one. For example, in the past, the Subscription Manager Tool (SMT) lived in its own product but it’s included in SUSE Linux Enterprise 12. So YaST needs to know that the suse-smt product was just replaced by sles.

This information is usually provided by the SUSE Customer Center (SCC). But what happens if, for example, we are performing an offline upgrade? Until now, YaST used a hard-coded fallback list.

From now on, before falling back to such a list, YaST will ask libzypp for that information so, hopefully, it will avoid some problems while upgrading extensions and it will reduce the hassle of maintaining a hard-coded list.

Storage reimplementation: Making sure to install storage-related packages

The YaST storage subsystem has been taking care about storage-related software packages for a long time. For example, when a specific filesystem type like Btrfs or XFS is used by the system, we need to make sure that necessary support packages like btrfsprogs or xfsprogs are installed.

Figuring out what features are used is now done by the new libstorage. In this sprint, we created one Ruby class that maps those features to respective packages and one class that handles package installation itself. One interesting technical aspect is how Ruby introspection capacities are used to avoid duplicating the list of defined features from the C++ part (i.e. libstorage).

Power the chameleon

Apart from all those changes in YaST, and many more we have not included in this summary, we have something else to celebrate. On February 1st the YaST Team at SUSE has grown with the addition of a new member, Iván, who will allow the project to evolve even faster and better… and that will not be the last announcement in that direction, so stay tuned.

But don’t forget you can also help YaST, and openSUSE in general, to keep moving. This week we added several ideas for Google Summer of Code projects to the openSUSE mentoring page, including one idea to contribute to YaST. Do you have a better plan for this summer?

See you in less than three weeks, since the next sprint will be slightly shorter due to Hack Week 15.

How we run our OpenStack cloud

January 23rd, 2017 by

This post it to document how we setup cloud.suse.de which is one of our many internal SUSE OpenStack Cloud deployments for use by R&D.

In 2016-06 we started the deployment with SOC6 on 4 nodes. 1 controller and 3 compute nodes that also served for ceph (distributed storage) with their 2nd HDD. Since the nodes are from 2012 they only have 1gbit network and spinning disks. Thus ceph only delivers ~50 MB/s which is sufficient for many use cases.

We did not deploy that cloud with HA, even though our product supports it. The two main reasons for that are

  • that it will use up two or three nodes instead of one for controller services, which is significant if you start out with only 4 (and grow to 6)
  • that it increases the complexity of setup, operations and debugging and thus might lead to decreased availability of the cloud service

Then we have a limited supply of vlans even though technically they are just numbers between 1 and 4095, in SUSE we do allocations to be able to switch together networks from further away. So we could not use vlan mode in neutron if we want to allow software defined network (=SDN) (we did not in old.cloud.suse.de and I did not hear complaints, but now I see a lot of people using SDN)
So we went with ovs+vxlan +dvr (open vSwitch + Virtual eXtensible LAN + Distributed Virtual Router) because that allows VMs to remain reachable even when the controller node reboots.
But then I found that they cannot use DNS during that time, because distributed virtual DNS was not yet implemented. And ovs has some annoying bugs are hard to debug and fix. So I built ugly workarounds that mostly hide^Wsolve the problems from our users’ point of view.
For the next cloud deployment, I will try to use linuxbridge+vlan or linuxbridge+vxlan mode.
And the uptime is pretty good. But it could be better with proper monitoring.

Because we needed to redeploy multiple times before we got all the details right and to document the setup, we scripted most of the deployment with qa_crowbarsetup (which is part of our CI) and extra files in https://github.com/SUSE-Cloud/automation/tree/production/scripts/productioncloud. The only part not in there are the passwords.

We use proper SSL certs from our internal SUSE CA.
For that we needed to install that root CA on all involved nodes.

We use kvm, because it is the most advanced and stable of the supported hypervisors. Xen might be a possible 2nd choice. We use two custom kvm patches to fix nested virt on our G3 Opteron CPUs.

Overall we use 3 vlans. One each for admin, public/floating, sdn/storage networks.
We increased the default /24 IP ranges because we needed more IPs in the fixed and public/floating networks

For authentication, we use our internal R&D LDAP server, but since it does not have information about user’s groups, I wrote a perl script to pull that information from the Novell/innerweb LDAP server and export it as json for use by the hybrid_json assignment backend I wrote.

In addition I wrote a cloud-stats.sh to email weekly reports about utilization of the cloud and another script to tell users about which instances they still have, but might have forgotten.

On the cloud user side, we and other people use one or more of

  • Salt-cloud
  • Nova boot
  • salt-ssh
  • terraform
  • heat

to script instance setup and administration.

Overall we are now hosting 70 instance VMs on 5 compute nodes that together have cost us below 20000€

Official RPM package : OWASP ZAP

January 6th, 2017 by

The world is going through a security crisis, so I make the official packages available in openSUSE, SUSE Enterprise, CentOS, Fedora e RedHat.

Thank you, Mauro Risonho de Paula Assumção e Simon Bennetts.

Link for download: https://software.opensuse.org/download.html?project=home%3Acabelo&package=owasp-zap

Watching 360 video on openSUSE

December 29th, 2016 by

In this post, there how to watch 360 videos on the Linux platform without the need for special technical magic. To do this, just use the package QMPlay 2, A player based on QT that fulfills its function very well. In addition to being versatile and effective, bringing an incredible amount of features for you to use, including touchscreen monitor recognition to navigate the video preview.

To watch 360 video, first press PLAYBACK the main menu, then choose VIDEO FILTERS, and finally enable the SPHERICAL VIEW option to use the 360 viewing function.

For testing, I provide my 360 video on the link below for download. And to finalize at the end a demonstration video of the software running as the installation links ONE CLICK INSTALL in my openSUSE repository.

Video example for download AQUI!

Source in Brazilian Portuguese (Alessandro de Oliveira Faria A.K.A. CABELO): https://assuntonerd.com.br/2016/12/29/assistindo-video-360-no-linux/

 

How to build OS images without kiwi

December 26th, 2016 by

kiwi has long been the one standard way of building images in openSUSE, but even though there exist extensive writings on how to use it, for many it is still an arcane thing better left to the Great Magicians.

Thus, I started to use a simpler alternative image building method, named altimagebuild when I built our first working Raspberry Pi images in 2013 and now I re-used that to build x86_64 VM images at
https://build.opensuse.org/package/show/home:bmwiedemann/altimagebuild
after I found out that it even works in OBS, including publishing the result to our mirror infrastructure.
It is still in rpm format because of how it is produced, so you have to use unrpm to get to the image file.

This method uses 3 parts.

  • a .spec file that lists packages to be pulled into the image
  • a mkrootfs.sh that converts the build system into the future root filesystem you want
  • a mkimage.sh that converts the rootfs into a filesystem image

The good thing about it is that you do not need specialized external tools, because everything is hard-coded in the scripts.
And the bad thing about it is that everything is hard-coded in the scripts, so it is hard to share general improvements over a wider range of images.

In the current version, it builds cloud-enabled partitionless images (which is nice for VMs because you can just use resize2fs to get a larger filesystem and if you later want to access your VM’s data from outside, you simply use mount -o loop)
But it can build anything you want.

To make your own build, do osc checkout home:bmwiedemann/altimagebuild && cd $_ && osc build openSUSE_Leap_42.2

So what images would you want to build?