Unifying the Console Keyboard Layouts for SLE and openSUSE

The way of managing internationalization in Linux systems has changed through the years, as well as the technologies used to represent the different alphabets and characters used in every language. YaST tries to offer a centralized way of managing the system-wide settings in that regard. An apparently simple action like changing the language in the YaST interface implies many aspects like setting the font and the keyboard map to be used in the text-based consoles, doing the same for the graphical X11 environment and keeping those fonts and keyboard maps in sync, ensuring the compatibility between all the pieces.

For that purpose, YaST maintains a list with all the correspondences between keyboard layouts and its corresponding "keymap" files living under /usr/share/kbd/keymaps. Some time ago the content of that list diverged between openSUSE and SLE-based products. During this sprint we took the opportunity to analyze the situation and try to unify criteria in that regard.

We analyzed the status and origin of all the keymap files used in both families of distributions (you can see a rather comprehensive research starting in comment #18 of bug#1124921) and we came to the conclusions that:

• The openSUSE list needed some minor adjustments.
• Leaving that aside, the keymaps used in openSUSE were in general a better option because they are more modern and aligned with current upstream development.

So we decided to unify all systems to adopt the openSUSE approach. That will have basically no impact for our openSUSE users but may have some implications for users installing the upcoming SLE-15-SP2. In any case, we hope that change will be for the better in most cases. Time will tell.

Exporting User Defined Repositories to AutoYaST Configuration File.

With the call yast clone_system an AutoYaST configuration file will be generated which reflects the state of the running system. Up to now only SUSE Add-Ons have been defined in the AutoYaST configration module. Now also user defined repositories will be exported in an own subsection <add_on_others> of the <add-on> section.

<add-on>
  <add_on_others config:type="list">
    <listentry>
      <alias>yast_head</alias>
      <media_url>https://download.opensuse.org/repositories/YaST:/Head/openSUSE_Leap_15.1/</media_url>
      <name>Yast head</name>
      <priority config:type="integer">99</priority>
      <product_dir>/</product_dir>
    </listentry>
  </add_on_others>
  <add_on_products config:type="list">
    <listentry>
      <media_url>dvd:/?devices=/dev/sr1</media_url>
      <product>sle-module-desktop-applications</product>
      <product_dir>/Module-Desktop-Applications</product_dir>
    </listentry>
    <listentry>
      <media_url>dvd:/?devices=/dev/sr1</media_url>
      <product>sle-module-basesystem</product>
      <product_dir>/Module-Basesystem</product_dir>
    </listentry>
  </add_on_products>
</add-on>

The format of the <add_on_others> section is the same as the <add_on_products> section.

Better Handling of Broken Bootloader Setups during Upgrade

With the current versions of SLE and openSUSE, using the installation media to upgrade a system which contains a badly broken GRUB2 configuration (e.g. contains references to udev links that do not longer exist) can result in an ugly internal error during the process.

The first possible problem could arise in the summary screen. Like shown in this screenshot.

If the error didn’t pop-up or if the user managed to recover from it, it was possible to reach the final phase of the upgrade process. But then the same internal error could still pop up in a different place:

Those errors will be fixed in the upcoming releases of SLE-12-SP5 and SLE-15-SP2 and, of course, in the corresponding openSUSE Leap version and in Tumbleweed. Now if such a broken setup is detected in the summary screen, a proper warning is displayed, including the technical details and a tip on what to do to fix the problem.

The user can ignore the problem or click on "booting" to fix it. In the latter case, the usual pop-up with instructions will appear.

If the final stage of the upgrade process is reached without fixing the error, the wild internal error is now replaced by an informative message that does not interrupt the process.

Hopefully most of our users will never see these improvements. But users with a broken system will likely appreciate the extra guidance.

Old Storage, New Features

If you are an usual reader of this blog, most likely you already know that YaST has a completely re-implemented Storage stack (a.k.a. storage-ng). That new storage code did its debut with the SLE 15 (and openSUSE Leap 15.0) family. And thanks to this revamped code, our beloved users can enjoy today some new great features in YaST like Bcache, partitionable Software RAIDs or multi-device Btrfs file system (just to mention a few examples). But SLE 12 (openSUSE 42) products are still alive and getting improvements with every maintenance update! Of course, the old Storage stack is not an exception, and now on a new installation scenario is supported.

Thanks to a bug report, we realized that Snapper could not be configured in some cases. More specifically, the reporter was trying to install with AutoYaST over a directly formatted Software RAID by using Btrfs for root and enabling snapper. The installation was perfectly performed, but it turned out that snapper was not correctly enabled in the installed system. After having a deeper look into the problem, we discovered that this was not a bug exactly but a completely missing feature. But no problems, YaST got down to work and now it is nicely supported.

Official features

First I look on official features. I think most important is automatic test suite. This is set of automatic test, which test again interface of library (it is black box testing). Before release of 11.1 this test suite contain 232 tests. I hope this increase quality of each perl-Bootloader release, because this suite catch many problems and also if new occur I add to it (this should prevent regression). Only problem with this test suite is that it doesn’t test whole kernel upgrade. It is hard to due, because kernel upgrade must fill itself hardware informations and if I want test it correctly I must simulate it with actual utilities which is used and on many types of hardware. So this part should be improved, but need some idea how test many hardware configuration (like different RAIDs (linux and bios), LVMs, multipath, different hardware architecture (like macos, efika or chrp on powerpc)) on one machine.
Another feature is consistent device names. This is most problematic feature, because it is hard to resolve many different devices given by udev (also udev is broken during some part of development). I experiment with many different sollution and at final I decide (it is after last beta) to create function which translate everything to kernel device and then compare that device to another translated device (previous solution based on filling all symlinks also work, but after device mapper problem with udev(more lower) I change it to more efficient and more reliable solution).
Next feature you can use if you have machine with chip to trusted computing (some notebooks have it). More about trusted computing you can read here. Most work did thorsten duwe (maintainer of grub) and in final I only ensure that due to security reason no splash screen is loaded (I remove message line even if user want it). If you have pc with GPT table and x86_64 processor you must for 11.0 use legacy booting. Now you can use ELILO bootloader (which have support efi) also on x86 architecture. This is also quite easy to implement, but harder to ensure it works, because get hardware with that configuration is not so easy. Only short notes for another features. Old disks which uses C/H/S is dropped and LBA is forced, kernel append during upgrade is taken from sysconfig and add support for disk remapping for windows entries.

Enhancement

Difference between feature and bug is really small and maybe some of enhancement looks more like bug and vice versa. For me interesting enhancement is check, if /boot is mounted. I use it at home quite often, because if you have separate home you needn’t mount it, but if during update also kernel is updated I often forgot mount it. First implementation is not ideal and I get many experience, that some users have really exotic entries in fstab, which of course bad match to check pattern. But after three iteration and one work-around this work quite good and I am satisfied with it. Another enhancement (really close to bug) is none bootloader. It is special bootloader settings, which ensure that nothing happen during kernel upgrade. It is quite useful if kernel update mess your configuration (which is bug, but you want prevent it until I fixed it) or take to much time (this is also bug). More important usage is for netboot, when you needn’t any bootloader, because you boot via PXE.
Important enhanced is stop kernel install, if bootloader update fail (problem is in using tee in pipe and ignore script return value). It is good, because you know that something goes wrong and reload backup. Also because usually new kernel is installed first, you still have old kernel and can boot it. Last enhancement is for anyone who want look how perl-bootloader works inside. Now you can use make doc in sources and it generates html page from comments (something like javadoc). Beside this documentation together with Jozo Uhliarik we create wiki page about interface of library and anyone who want use perl-Bootloader should use it.

Bugs

Solving bugs in perl-Bootloader is not so interesting as it looks like ;). You need analyze many long logs (if perl-Bootloader uses yast, then it is saved to y2log, if used by kernel upgrade then save to perl-BL-standalone log) and find what and when going wrong. You find what is wrong, but not who break it. Sometime it is bug in parsing configuration, sometime in parsing output of external commands and sometime is bug outside (like break udev). Source code (also with test suite) have 15k LOC and when I try look how many line I change (oneliner for it `find . -type f 2>/dev/null | grep -v .svn | xargs svn blame | grep jreidinger | wc -l `) I find, that I change 3k LOC. So code base is good, but need some improvement. I note some interesting bugs, which I solve.
One is longterm problem with chainload, when as root is current mounted root, but chainloader key is with right prefix (root of chainloaded partition). This on some hardware causes problems and now root is correctly set and I don’t have any negative responses (so I hope it works).
Another one is also quite long term problem when you change flavor of kernel, sometime previous kernel is uninstalled and after that new installed (normal work-flow is install new and uninstall old). Of course it switch default, which is quite annoying. Fix this is quite tricky, and trick is that after remove last image section and if it is default, I add comment about that and when new kernel is added, then set it as default. That fix works quite good, but after sent iso to factory (for boxes) I get bugs, that if you update via YaST, it doesn’t work. Problem is in some deprecated code which overwrite my comment by another one. Fix is easy, I only remove deprecated code, which cause it. So if you after install of opensuse11.1 see update of perl-Bootloader you know why. As late update is also solved problem with hanging kernel update. It take too much time because when I read logs I add useful logging lines to code (I hope it help me next time find problem faster) and when log records reach some level, it take too much time (due to array copying of records). I reduce debug logging and also improve whole performance of logging, so I hope it significant improve time to kernel update (at least it should not take minutes to upgrade bootloader configuration).
Not every bug is problem of perl-Bootloader, but you must solve it in code. Example is device mapper and udev inconsistency. Udev points to /dev/dm0, but device mapper to /dev/mapper/something. This became real problem when persistant names feature is implemented. Solution is also little tricky, udev define variables for links and one of that variable is DM_NAME for name of device and DM_PART for partition number and with this information I can construct whole /dev/mapper/name. I find this solution as working, but not much robust. If someone know better, (s-)he is welcome to write it to comment. Also many perl warnings is fixed, it usually doesn’t break anything, but it looks really unprofessional.
Now I have only 2 unresolved bugs (both is enhancement on which I work) and wait for another, so report all problems in opensuse11.1, I don’t throw it to bin