openSUSE Lizards

Archive for the ‘Distribution’ Category

Highlights of YaST development sprint 29

December 22nd, 2016 by Yast Team

It’s Christmas time and since (open)SUSE users have been nice, the YaST team brings some gifts for them. This is the result of the last development sprint of 2016.

As you may have noticed, in the latest sprints we have been focusing more and more in making SUSE CASP possible. That’s even more obvious in this last sprint of the year. For those that have not been following this blog recently, it’s probably worth to remember that SUSE CASP will be a Kubernetes based Container As a Service Platform.

But our daily work goes beyond CASP, so let’s take a look to all the highlights.

More improvements in the management of DHCLIENT_SET_HOSTNAME

In the previous report we presented the changes introduced in yast2-network to make the configuration parameter DHCLIENT_SET_HOSTNAME configurable in a per-interface basis.

One of the great things about working in an agile an iterative way, presenting and evaluating the result every three weeks, is that it allows us to detect room for improvements in our work. In this case we noticed some discrepancy in the expectations of Linuxrc and yast2-network and also some room for improvement in the code documentation and in the help texts.

Thus, we used this sprint to refine the work done in the previous one and tackle those problems down.

Ensure installation of needed packages

Another example of iterative development. We already presented in the report of the 26th development sprint a new mechanism to detect when the user had deselected during installation some package that was previously pre-selected by YaST in order to install the bootloader. Since the new functionality proved to work nicely, we decided to extend it to cover other parts of the system beyond the bootloader.

The software proposal now contains an error message including a list of missing packages or patterns, in case the user deselects some needed items.

After clicking the Install button the installation is blocked, the user must resolve the problem either by selecting the packages back or by adjusting the respective YaST configuration (e.g. do not install any bootloader and disable the firewall).

Rethinking the expert partitioner

May we insist one more time on the topic of using Scrum to organize our work in an iterative way? 😉 As our usual readers should already know, we structure the work into minimal units that produce a valuable outcome called PBIs in Scrum jargon. That valuable outcome doesn’t always have to be a piece of software, an implemented feature or a fixed bug. Sometimes a document adds value to YaST, specially if it can be used as base to collaborate with people outside the team.

Our readers also know that we are putting a lot of effort in rewriting the whole storage layer of YaST. That also implies rewriting the most powerful tool known by humanity to define partitions, volumes, RAIDs and similar stuff – the YaST expert partitioner.

It would be great if we could use the opportunity to make it both more powerful and more usable. You can take the first part for granted, but we are not so sure about our UI design skills. That’s why we wanted to have a base to discuss possible changes and alternative approaches with UX (user experience) experts. And we decided that it was worth to invest some time to create a document collecting the state of the art and some ideas for the future and to send that to SUSE experts in UX and to anybody with some interest in the topic.

Here you can find that fine piece of documentation. Take a look to that document if you want to peek into YaST developers’ mind. That’s the kind of stuff we discuss when we are about to start rewriting something… specially something that need to serve hundreds of different use cases.

And of course we would like to know your ideas or thoughts. We usually discuss this stuff at the public #yast IRC channel and at the yast-devel mailing list. But if you prefer so, you can simply open an issue at the repository hosting the document. Whatever works for you.

Rethinking yast2-network

But that was not the only documentation PBI finished during this sprint. Inspired by the first fruits of the storage layer reimplementation, we decided yast2-network also deserves a reincarnation.

As we did in the past with yast2-storage and libstorage, the first step is to collect as much information as possible about what can be currently done with the module and how it behaves in several situations, specially in tricky or complex scenarios. The outcome was three documents, one about the behavior during installation (installation.md), a second one about AutoYaST (autoinstallation.md) and another collecting general features (features.md).

CASP: merged dialogs for root password and keyboard layout

CASP is a product targeted to a quite specific use case with simplicity as a main priority. The installation process has been streamlined to a minimal set of dialogs to configure just the very basic stuff. Among other removed things, there is no step to configure the system language. That can be a problem when entering the root password (the only user that will be created during installation), since the language settings screen is normally also used to select the keyboard layout.

The implemented solution is shown in the screenshot below. As you can see, the keyboard layout and root passwords selections are merged into a single step. As a bonus, we made both widgets more reusable, opening the possibility to place the root password widget or the keyboard layout selection anywhere.

Storage reimplementation: handling GPT disks in the installation proposal

After several sprints reporting small steps forward, in the 27th sprint we were happy to announce that our testing ISO for the new storage stack was fully installable under certain circumstances. As we reported, it worked in UEFI or legacy systems with the only requirement of having a pre-existing MBR partition table in the disk.

Now we can say it also works with GPT partition tables and even with systems with a mixture of both technologies.

Making the GPT scenario work was much harder that it sounds due to several factors, like the strange way in which parted handles partition types in GPT or some peculiarities in the way the space is distributed in such partition tables.

But now our test ISO can install a fully functional system in the four combinations of MBR/GPT partition table and UEFI/Legacy boot, as it can be seen in the next image.

The storage reimplementation gets its own openQA instance

But there are better ways than screenshots to prove that something is working, even to prove it keeps working after future modifications. And in (open)SUSE we have one of the best tools for that – openQA.

We have always considered having the new stack tested in openQA as the first big milestone in its development (and we are finally there!) but we are aware that openQA.opensuse.org is already quite busy testing a huge combination of products, architectures and scenarios… even testing releases of openQA itself. Fortunately openQA is free software and can be installed anywhere so we created our own instance of openQA to test YaST stuff, specially the new storage layer.

So far, that instance is hosted in the internal SUSE network, which is enough for us to get continuous feedback about the changes we introduce. In addition to installing the new instance and configuring it to continuously grab and check the latest testing ISO, we had to introduce several changes in the ISO itself with the goal of keeping our tests as aligned as possible with the tests performed in the current Tumbleweed version by openQA.opensuse.org.

For example, we made sure the ISO was properly signed to avoid the need to always pass the insecure=1 boot argument. We also included several packages that were missing in order to make sure the ISO included all the software checked during the so-called MinimalX test and to make sure it shared the look and feel with a regular Tumbleweed, since many openQA checks are screenshot-based.

From now on, we can back every new feature with the corresponding integration tests, something crucial to ensure the quality of a piece of software meant to handle storage hardware.

Making Snapper work without DBus

As you may know, some YaST team members are also the main developers and maintainers of Snapper, the ultimate file-system snapshot tool for GNU/Linux systems.

Normally the snapper command line tool uses DBus to connect to snapperd which does most of the actual work. This allows non-root users to work with snapper.

There are however situations when using DBus is not possible and not being able to work in those situations was limiting Snapper’s usefulness. Now with the latest version all snapper commands support the –no-dbus option. This evolution is worth a blog post by itself… and, of course, we have it. To know all the details check this post at Snapper’s blog.

CASP (and beyond): improved roles

Do you remember the system roles feature introduced during development sprint 16 and improved in subsequent sprints? In case you don’t, let us remind you that system roles allow to define many settings of the installation just by choosing one of the offered roles. That’s only possible, of course, in products making use of that feature, like SLES.

For CASP we will have 3 different roles, as shown in the following screenshot.

The main difference between these three roles is the selection of patterns to be installed. But apart from that, the Worker role will offer an extra step during installation allowing the user to specify the address of the so-called Administration Dashboard.

That relatively small detail implied the development of a full new feature in the installer – the ability of a given role to define it’s own specific configuration, including the dialog to interact with the user. As expected from any other installation dialog, you can go back and forward without loosing the entered information. If the user goes back and selects a different role, then this additional dialog is not run again.

That new feature is, of course, not specific to CASP and could eventually be used in other products and roles. Just as a crazy example, openSUSE could decide to introduce a role called “NTP server”, running the YaST NTP server configuration right after the user selecting the role.

Other CASP related features

As already said, we have been focusing quite a lot on introducing features that are needed for CASP. It’s worth mentioning, in case it’s still unclear, that CASP will NOT ship its own adapted version of YaST. All the features introduced in the installer are in fact configurable and available for all other products as well. There is only one YaST codebase to rule them all.

Let’s briefly describe some of the introduced CASP-specific (at least for the time being) features.

CASP always uses Btrfs as filesystem for the root partition. At the end of the installation, the root btrfs subvolume will become read-only. All the other subvolumes will stay as read-write, as shown in this screenshot taken right after rebooting at the end of the installation process.

It makes no sense to update from any existing product to CASP. Thus, CASP media should not show an “update” option when booting, even if it’s still possible for advanced users to pass the UPDATE boot parameter. Since we needed to modify the installation-images package, we took the opportunity to make the “update” option and other settings configurable in a per product basis and we unified SLES and openSUSE packages, so now they share a single branch in the source code repository.

CASP is targeted to big deployments extended all over the world. To make possible the synchronization of geographically distributed nodes, the UTC timezone is enforced in every CASP installation. Thus, we implemented support for products to enforce a given timezone in the installer. Take into account this is different from a default timezone.

Last but not least, it has already been mentioned that the CASP installation workflow will have very few steps. That also affects the screen displaying the installations settings summary. In comparison to a regular SLES, some options must disappear because they are not configurable and some other sections must be added because they are not longer presented as a separate previous step. So far, this is the appearance of the installation settings screen in the current CASP prototype.

…and a surprise about the blog

We also prepared a Christmas gift related to the blog. The technical aspects are solved, but we are ironing out the administrative details. So you will have to wait until the next sprint report to see it in full glory. But, as the Spanish proverb says, “good things are worth waiting for”.

See you next year

That’s enough to report from our December sprint, we don’t want to bore you with every small bug fix. And talking about things that are worth waiting for, our next report will very likely be published at the beginning of February 2017.

That’s because we will put our Scrum process on hold during the Christmas session. We will restart it on the second week of the year, after the visit of the Three Wise Men. In several countries, it’s a tradition that the Three Kings bring gifts to the kids that have been nice, so let’s expect they bring us some new members for the team!

Posted in Distribution, Factory, Programming, Quality Assurance, Software Management, Systems Management, Usability, YaST | 3 Comments »

AMD/ATI Catalyst fglrx rpms, end of an era!

December 7th, 2016 by Bruno Friedmann

Long time not talking about fglrx rpm, mostly because they’ve got no update since last December 2015.

Short Summary

In a word as hundred, fglrx is now a dead horse!

Dead horse

We had the hope of getting it working for Leap 42.2 in October, but except freezing kernel and xorg, you will not get what you would expect: a stable xorg session

Say goodbye fglrx!, repeat after me, goodbye fglrx.

If you are locked down and forced for any reasons to use fglrx with your gpu, and are still using 42.1, then don’t upgrade to 42.2, without a plan B

It has no more support from AMD upstream, and that’s it!, if someone want to break its computer, it’s still possible to pick the last files and try it by yourself, but the repository will never contain it for 42.2 (see below how-to)

That’s said, I’m not still sure, to keep for a long time the repository, I’ve been managing since 6 years now.

A bit of history

In 2010, when we were working hard to get 11.1 out, the news that no supported ATI (at that time) will be available for end-users, as we have for nvidia gpu

I didn’t check back the irc log, but we were a few, that would like to have this still available, by pure commodity. Especially that I’ve just exchanged a non working gpu by my new hd5750.

I remember the first chaotic steps, how to build that, and create repeating builds, what about the license? Did we have the right to offer a pre-build rpm etc. I spent some time fixing all of this stuff.
And start the build on real hardware. Hey afterward kvm was really in infancy stage.

Release after release amd/ati and openSUSE, the driver was build, on hardware for each supported distribution. When beginning of 2013 Sebastian Siebert, who got some direct contacts with AMD, release
his own script, we collaborate to have the possibility to build on virtual machines, which allow me to simplify the build process, as having on kvm for each openSUSE release supported.

Afterward, AMD start to split fglrx with the fglrx for HD5xx and above, and fglrx-legacy. So 2 drivers to maintain, but as always with proprietary software, the legacy version became rapidly obsolete,
and non usable. Not that bad, in the meantime the AMD effort on the free and open source radeon driver, quickly overcome the performance of legacy.

Still from 2013, to 2016 I’ve been able to propose ready to use rpm for several version of openSUSE’s distributions. I think the repository serve quite well end users, and I never got big flames.

I can’t avoid to mention the openSUSE powered server and sponsored by Ioda-Net Sàrl that has serve this objective so well during that time frame.

Future of the repository

Now that fglrx is becoming obsolete, I think seriously about why the repository online should stay online.

At openSUSE project level, we still have 13.1, 13.2, 42.1 and 42.2 that are mostly active. 13.1 is already almost out of the game of evergreen,
13.2 will follow soon, and I don’t know yet the exact plan for 42.1, but it will certainly go out of maintenance in less than a year.

If you feel or have the need of the repository, please express that in the comments below.

Wait there’s amd-gpu-pro, no?

Yeap there’s a closed driver, called amd-gpu-pro, available, for newer cards. But there’s two things that bring me out of the game, first I don’t have those newer gpu,
and don’t have the need to replace my hd5750 for the moment. The second and certainly the most important, those drivers are only available for Ubuntu or at least in .deb format.

I will certainly not help proprietary crap, if I don’t have a solid base to work with, and a bit of help from their side. I wish good luck to those who want to try those drivers,
I’ve got a look inside, and got a blame face.

For crazy, and those who don’t love their computer

So you want to loose your time? you can! I’ve kept in raw-src directory all the script used to build the driver.
They differ a bit compared to Sebastian Siebert last version in the sense of making Leap 422 as a possible target.
If you dig a bit around, you should be able to build them, but you’re alone on that way, you’ve been warned!

I’m not against a republished version, if someone find a way to make them working, just drop me a message.

That’s all for this journey, Have Fun! 🙂

Tags: 11.4, 12.1, 12.2, 12.3, 13.1, 13.2, 42.1, 42.2, amd, ATI, fglrx, Kernel, Tumbleweed, unified, xorg
Posted in Distribution, Kernel | 15 Comments »

Highlights of YaST development sprint 28

December 2nd, 2016 by Yast Team

November is over, Santa Claus elves start to stress and the YaST team brings you one of the last reports of 2016. Let’s see what’s new in YaSTland.

Harder to ignore installation warning

The “installation settings” summary screen usually reports some non-critical errors displayed as a red text. Although the installation can proceed despite those errors, they are usually serious enough to lead to problems. That’s why we decided to introduce a change to highlight them a little bit more, making them harder to overlook.

The following screenshot shows the newly introduced confirmation dialog, presented before proceeding with installation.

Make DHCLIENT_SET_HOSTNAME configurable on a per-interface basis

But that’s not the only usability-oriented enhancement on this sprint. We also reworked a bit the network configuration dialog.

For home users is very common to use a fixed hostname -set during installation- for our beloved linux box. But in some circumstances it’s better to set the hostname of the machine dynamically using DHCP, something YaST has always allowed to do by just ticking a checkbox that used to be in the network configuration screen. See “Change Hostname via DHCP” below.

That checkbox used to modify the system-wide variable DHCLIENT_SET_HOSTNAME, which was fine in scenarios in which only one of the network interfaces was configured via DHCP. But with several network interfaces connected to different DHCP-enabled networks, some problems arose.

During installation, if network configuration is used, Linuxrc creates the ifcfg files with DHCLIENT_SET_HOSTNAME='yes' for all of the enabled or configured interfaces and this value has precedence over the global one.

So the main problem was that YaST only allowed us to modify the global variable and setting it to ‘no’ did nothing because it was enabled for some interface.

During this sprint we have fixed that and now the user interface offers the possibility of choosing which DHCP interface will be used to decide the hostname.

Apart from choosing one of the existing interfaces, the new setting can also be set to ‘no’ or to ‘any’. In any case, YaST will always configure the system-wide options and the interface specific ones in a consistent way, so the behavior is always predictable.

But YaST is not the only way of configuring the network, so it’s always possible to have an unpredictable configuration. Fortunately, those potentially problematic scenarios will be detected by YaST and reported to the user.

Partitioning in CASP

In the previous report we already explained how are we improving the installer to support the definition of the ultra-streamlined installation process of SUSE CASP, the new Kubernetes based member of the SUSE family.

In this sprint we introduced several additional changes to enable a different partitioning approach, more guided and automatic than ever. In a CASP node it makes no sense to use the advanced settings offered by our storage proposal, like encryption or LVM. Moreover, CASP relies on Btrfs to provide some of its cool and advanced features, like transactional updates.

As a result, although the regular SUSE and openSUSE releases will keep offering all the current possibilities in the same way than ever, in CASP the partitioning step will be skipped and the automatically calculated proposal will be simply displayed in the installation summary.

Clicking on the proposal will allow to re-target the installation to a different disk (or disks) in a similar way than the regular installer, but the options will be more limited. Again, no easy way to use LVM, encryption, separate home or any file system type other than Btrfs.

The expert partitioner is still available during CASP installation, but using it will show an extra warning, since it implies a much bigger risk than using it in a regular SUSE or openSUSE system.

Improved debugger integration

We have improved the Ruby debugger integration in YaST. So far you could start the debugger using the y2debugger=1 boot option or by setting the Y2DEBUGGER=1 environment variable. The new feature allows starting the Ruby debugger also later when the YaST module is already running.

Simply press the Shift+Ctrl+Alt+D keyboard shortcut (D as debug) and it will start the Ruby debugger. It works during installation and also in installed system (just make sure the byebug Ruby gem is installed).

Unfortunately this new feature works only in the Qt UI, the ncurses UI is not supported (currently it does not handle the debugging keyboard shortcut at all).

After pressing the keyboard shortcut the debugger window will pop up:

New debugger integration

Storage reimplementation: it’s alive!

It took us one more sprint than originally expected, but finally we can say the testing ISO for the new storage stack is fully installable.

We fixed the UEFI + MBR partition table scenario we already had almost working in the previous sprint (turns out it was not that broken in Tumbleweed after all) and we adapted yast2-bootloader to be also able to deal with legacy (i.e. no UEFI) booting using the new storage stack.

As a nice result, our testing ISO can be used to install a perfectly functional system in both UEFI or legacy systems with the only requirement of having a pre-existing MBR partition table in the disk. It only shows a couple of error pop-ups related to the calculation of the proposal of software to be installed, but nothing that would prevent you from replacing whatever operating system you have with a new shiny openSUSE-based experiment.

This milestone opens the door to start testing the new stack with openQA, the same system that helps to guarantee the robustness of all the recent SUSE and openSUSE versions.

Storage reimplementation: preparations for the storage proposal

Now that yast2-bootloader starts to be ready to work with the new storage stack in more and more scenarios, it’s time to adapt the only component that still complains during the installation.

In order to make that task doable during the next sprint, we invested some time in this sprint analyzing the interaction between the software proposal calculator and the old storage layer. The outcome was a small document detailing what needs to be adapted in the proposal and in the new stack. The perfect input for a task in the next sprint.

Help for power-users with short memory

Our beloved YaST is packed with magic tricks below the surface. Many of them are very useful to debug installation problems or to better understand how the YaST internals work. Unfortunately developers tend to not be that good at blindly memorizing stuff and the functionality is so well hidden that most newcomers would have hard times finding it… until now.

We have added a couple of new keyboard shortcuts to show a summary of all the advanced hotkeys, so now you only have to remember one key combination instead of a dozen of them. In both text (ncurses) and graphical (Qt) mode, it will be enough to press shift+F1 to get the advanced help displayed below. Since some terminal emulators could already use that combination, ctrl+D F1 can also be used in the ncurses interface as an alternative.

Contributions keep coming!

As we have already mentioned in previous sprint reports, an important part of our daily job as open source developers is helping casual (and not so casual) contributors to bring their ideas and code into YaST and related projects.

This time that (hopefully not casual) contributor was Devin Waas, who wanted to improve the installation to make the life of cloud-lovers easier.

For cloud guys out there retrieving logs of a failed installation is was a huge problem. Now, thanks to Devin, all you need is a running a rsyslog server and you’ll be able to easily access your installation logs from there.

A drawing is worth a thousand words

As a matter of fact, the newest Tumbleweed release allows us to specify the IP address of a remote server from the bootloader through the “Loghost flag”. Linuxrc will take care of setting up a UDP broadcast for dmesg contents and YaST installation logs.

This is just a first step. Devin promised further improvements of our newly implemented remote logging system. And he codes better than he draws, so stay tuned!

Storage reimplementation: LVM-based proposal

As we already mentioned in previous reports, when we started to develop the partitioning proposal we first focused in the scenario of a partition-based proposal with one or several MBR-style partition tables. That looked like the most complex scenario due to the limited number of primary partitions, the alignment problems, the overhead introduced by the EBR (extended boot record) of every logical partition and so on.

A couple of sprints ago, we got that working so we started to work on the LVM-based proposal. It took a little bit longer than expected but now we are able to generate LVM-based proposals for almost every possible scenario. The goal was to have them working in our mocked test cases. So probably the new LVM-based proposal cannot still be used to install a fully functional system, but it is backed by a full load of tests that prove we can handle many situations, from trivial to really tricky ones… and believe us, things can get quite tricky if you mix logical partitions with their EBR overhead and LVM volumes with their PE size rounding and their metadata overhead.

Bugs, bugs, bugs

In this sprint we kept the already commented approach of making the fix of low-priority and small bugs part of the Scrum process. As a result we accounted for approximately 50 deaths of those annoying creatures.

Conclusion

Looking at the report, we could say it was a quite successful sprint. But to be honest we were aiming even higher. Quite some interesting PBIs (features or bug-fixes in Scrum jargon) were almost done at the end of the sprint. But following Scrum philosophy, we never blog about almost-done stuff.

Thus, if nothing goes wrong things will be even better in the next report in three weeks. So have a lot of fun trying the new stuff and stay tuned for more!

Posted in Base System, Distribution, Factory, Network, Programming, Systems Management, Usability, YaST | Comments Off on Highlights of YaST development sprint 28

Highlights of YaST development sprint 27

November 10th, 2016 by Yast Team

Another three weeks of development come to an end… and our usual report starts. Take a look to what we have been cooking.

Read-only proposal modules

This week, during SUSECon 2016, SUSE announced an exciting upcoming new product. SUSE CASP – a Kubernetes based Container As a Service Platform.

That has, of course, some implications for the installer, like the need of some products (like CASP) to specify a fixed configuration for some subsystems. For example, an established selection of packages. The user should not be allowed to change those fixed configurations during installation.

We have implemented a possibility to mark some modules in the installation proposal as read-only. These read-only modules then cannot be started from the installer and therefore their configuration is kept at the default initial state.

In this sprint we have implemented a basic support in the proposal framework, in the future we could improve the respective proposal modules to better handle the read-only mode.

Per product Btrfs subvolumes handling

The ability to have read-only proposals is not the only improvement we have introduced in YaST to make SUSE CASP possible.

When Btrfs is used during the installation, YaST2 proposes a list of subvolumes to create. Unfortunatelly, that list is hard-coded and it’s the same for every (open)SUSE product… until now.

Starting on yast2-storage 3.1.103.1, a list of subvolumes can be defined in the product’s control file along with additional options (check out SLES and openSUSE examples to learn more).

<subvolumes config:type="list">
  <subvolume>
    <path>boot/grub2/i386-pc</path>
    <archs>i386,x86_64</archs>
  </subvolume>
  <subvolume>
    <path>home</path>
  </subvolume>
  <subvolume>
    <path>opt</path>
  </subvolume>
  <subvolume>
    <path>var/lib/libvirt/images</path>
    <copy_on_write config:type="boolean">false</copy_on_write>
  </subvolume>
</subvolumes>

This specification supports:

Disabling copy-on-write for a given subvolume (it’s enabled by default).
Enabling the creation of a subvolume only for a set of architectures.

Improved AutoYaST Btrfs subvolumes handling

AutoYaST Btrfs subvolumes handling has been also improved. Using almost the same syntax as for product control files, you can disable the CoW behavior for a given subvolume.

Of course, if you don’t need such a feature, you won’t need to adapt your profiles to the new syntax as it’s backward compatible. You can also mix both of them:

<subvolumes config:type="list">
  <subvolume>home</subvolume>
  <subvolume>
    <path>var/lib/libvirt/images</path>
    <copy_on_write config:type="boolean">false</copy_on_write>
  </subvolume>
</subvolumes>

On the other hand, if you’re running SLES, you’ll know that a subvolume called @ is used as the default Btrfs subvolume. Now is possible to turn-off such behavior in the profile’s general section.

<general>
  <btrfs_set_subvolume_default_name config:type="boolean">false</btrfs_set_subvolume_default_name>
</general>

Disable installer self-update by default

We have talked many times in this blog about the new self-update functionality for the installer. As expected, this functionality will debut in SLE-12-SP2 but with a small change in the original plan. In order to ensure maximum consistency in the behavior of SLE-12-SP1 and SP2 installers, the self-update functionality will be disabled by default. Not a big deal, since enabling it to get the latest fixes will be just a matter of adding a boot option in the initial installation screen.

Storage reimplementation: adapted EFI proposal in yast2-bootloader

The previous report about the status of the storage stack reimplementation finished with the following cliffhanger “we have in place all the ingredients to cook an installable ISO“. So, as expected, we worked during this sprint in adapting the bootloader proposal to the new storage layer. As you can see in the following screenshot, we succeeded and the installer can already propose a valid grub2 setup to boot an EFI system.

Does that mean that the testing ISO for the new storage stack is already fully installable? Unfortunately not. Why not, you ask? The reason is, in fact, kind of fun.

The current partitioning proposal only works with MS-DOS style partition tables because we wanted to address the most complicated case first. On the other hand, we intentionally restricted the adaptation of the bootloader proposal to the EFI scenario. And you know what? We found out that combination (MS-DOS partition table + EFI) does not currently work in Tumbleweed, so it neither does in our Tumbleweed-based testing ISO.

We will work during the following sprint to support another scenario. And hopefully we will not choose again an unsupported or broken one. 😉

Letting libYUI run free: first visible steps

As you know, YaST has both a graphical and a textual interface. They run on the same code thanks to an abstraction layer called LibYUI. Originally it served YaST only, but over time other projects started using it, notably the admin panel in Mageia. Moreover, the developers of those projects started to contribute fixes and improvements to LibYUI… faster than we can cope with.

Recently we decided to give the people outside the YaST team more power in the LibYUI project. To make that possible ensuring it does not affect YaST stability, we have been enabling more continuous integration tests.

As a first fruit of the revamped collaboration, we have merged a fix contributed by Angelo Naselli of Mageia regarding selection of tree items. As soon as we complete our continuous integration setup (of course we will keep you informed), more fixes and improvements will come for sure.

More bug squashing and bug paleontology

In the previous report we presented our new effort to integrate the fix of low-priority bugs into the Scrum process. During this sprint we refined the idea a little bit more, distinguishing two separate concepts (each of them with its own PBI) – fixing of existing bugs and closing of too old ones.

The first one doesn’t need much explanation. We managed to fix around 25 non-critical bugs in YaST and if you are using Tumbleweed you are probably already benefiting from the result.

The second task was not exactly about fixing bugs present in the software, but about cleaning bugzilla from bug reports that were simply too old to be relevant any more. Like bugs affecting very old versions of openSUSE that cannot be reproduced in openSUSE 13.2 or Leap. We must be realistic about releases that are already out of support and the limited human resources we have. We closed around 80 of those ancient bugs.

So overall, we cleaned up around one hundred bugs from our queue. Still a long way to have a bug-free YaST, but undoubtedly a step in the right direction.

More coming

We are already working in the next sprint that will hopefully bring several new improvements for CASP, quite relevant progress in the storage reimplementation, several usability improvements, some bugfixes and even some news about this blog.

To make sure you don’t get bored while waiting we are also planning to finally publish the report about our visit to Euruko 2016.

Stay tuned!

Posted in Base System, Distribution, Factory, Programming, Systems Management, YaST | 6 Comments »

Highlights of YaST development sprint 26

October 20th, 2016 by Yast Team

We did our best to keep you entertained during this development sprint with a couple of blog posts ([1] and [2]). But now the sprint is over and it’s time for a new report.

Squashing low priority bugs

One of the main reasons to adopt Scrum was to ensure we make a good use of our development resources (i.e. developers’ time and brains) focusing on things that bring more value to our users. In the past we had the feeling that many important things were always postponed because the developers were flooded by other not so important stuff. Now that feeling is gone (to a great extent) and we have a more clear and shared view of the direction of our development efforts.

But there is always a drawback. We have accumulated quite some unsolved low-priority bugs. That was an expected consequence, but still it was starting to feel wrong. On one hand, it makes us feel uncomfortable – replying “this will have to wait” so often is not nice, even if it’s for the shake a bigger goal. On the other hand, the amount of low-priority stuff was affecting the signal/noise ratio on Bugzilla, making harder to distinguish the important stuff.

So far, dealing with those low-priority bug was something that each developer did on his own, as permitted by the time dedicated to Scrum. In sprint 26 we decided to make that effort an explicit part of the process and to devote a significant portion of the sprint to it. As a result we closed or reassigned a total of 135 bugs that were just sitting in our list.

Yes, you did read it right. One hundred and thirty five bugs.

Storage reimplementation: our testing ISO can already destroy your data

On the previous sprint we already showed a screenshot of the installer using the new storage stack to calculate a partitioning proposal. Now the installer can go one step further. As you can see on this animation, the changes are now committed to the disk, meaning the system is actually partitioned, formatted and installed.

Installation with the new storage stack

The process is interrupted after installing the software, when trying to configure and install the bootloader. That was expected because yast2-bootloader has still not been adapted to use the new stack. First of all, because we wanted to leave some fun for sprint 27. But also because we used this sprint (26) to document all the requirements of yast2-bootloader in relation to the new storage stack. Now we have in place all the ingredients to cook an installable ISO.

Automatic update of translation files

Recently openSUSE has adopted Weblate to perform and coordinate the translation of the software and the project’s web pages. The openSUSE’s Weblate instance enables everybody (from dedicated translators to casual contributors) to take part in the process and makes possible to coordinate the translations of openSUSE with the ones for SUSE Enterprise Linux, boosting collaboration between the translators of both projects.

As YaST developers, is of course our responsibility to ensure that openSUSE’s Weblate contains always the latest English strings to be translated. Making our developer’s life easier sometimes not only brings advantages for us but also for our users. Until now, after each code change we had to keep in mind to trigger the translation process for every added or changed English text. Sometimes we were not quick enough so that some English leftovers remained in our awesome YaST when being used in one of the 20 languages where the translation is normally 100% complete.

Now we finally set up a Jenkins job to automate the process of triggering the translation update after code changes. This saves the developers some work and makes the update of translations even faster.

Looking at Weblate numbers, you can see we have 20 languages that are about 100% translated, another 20 that are translated more than 75% and 37 languages which are translated less than 75%. So we still need some help to bring all languages to 100% coverage. If you are willing to contribute, why not join our team of translators? Check out the localization guide to get in contact with the coordinators of your preferred language to learn about how to contribute with translations, reviews or by any other mean.

Ensure installation of packages needed for booting

We got some reports of systems not being able to work after the installation in scenarios in which the user had customized the list of packages to install. That happened because, although the bootloader component of YaST pre-selects for installation all the needed packages, the user can override that selection and manually disable the installation of those packages.

To prevent this situation, or at least to increase awareness of it, the installer now alerts in the summary screen (the last step before proceeding to installation) if some of the required packages is missing, as you can see in the screenshot below. We still allow the users to shoot their own feet if they insist, but now we warn them very clearly.

Progress in the low-vision accessibility of the installer

During this sprint, we have been working to make the (open)SUSE installer accessible to people with low-vision impairment. We already blogged about it looking for feedback.

In a few days, some changes will land in Tumbleweed:

Alternative style selection (color and font-sizes). Currently, we offer four options: default (no changes), high contrast (cyan/green/white/black), white on black and cyan on black. Those styles are just a proof of concept in order to test the code changes and, most important, to get feedback from you.
A long-standing issue, which prevented to switch to high-contrast mode during installation (shift+F4), has been fixed.

Although we have made some progress, it is still an ongoing effort and we hope to release more improvements during the upcoming weeks.

Recover from broken bootloader configuration

There are situations in which YaST Bootloader is not able to read the system configuration. For example, when the udev device originally used by Grub2 is no longer available. In the past that leaded to YaST crashes, requiring a manual fix of the bootloader configuration files. Now YaST correctly detects the situation and offers the option to propose a new configuration with correct devices.

Disable autorefresh by default in local media

We have changed the default autorefresh flag for the new repositories added by YaST. In the past the autorefresh was enabled for all repositories except CD/DVD media and ISO files.

With the new defaults the autorefresh is enabled only for remote repositories (like http, ftp, nfs,…). The reason is that some local repositories might not be always available (e.g. external hard disk, USB flash drive,…) and the automatic refresh might cause ugly errors when starting the package manager.

Of course, you can still manually change the autorefresh flag after adding a repository if you need a different value.

Note: The default has been changed in Tumbleweed distribution only, Leap 42.2 or SLE-12-SP2 keep the old defaults. The zypper behavior is unchanged as well, it by default disables autorefresh for all repositories. Only repositories imported from a .repo file have autorefresh enabled. See man zypper for more details.

Tons of improvements in network bridge handling

YaST Network is Swiss Army knife for network configuration which comprehends the management of routing, bonding, bridging and many other things. But, to be honest, the management of bridges was not in the best possible shape. It had quite some usability problems and it was not 100% consistent with the way in which bridges are managed nowadays by Wicked in (open)SUSE. Until now!

This pull request with several screenshots and animations tries to summarize all the changes that have been done during this sprint. Like adapting old configuration files to the new conventions or unifying the UI to make it consistent with the one for managing bonding.

This revamp includes also quite some usability improvements:

“NONE” is shown instead of 0.0.0.0 for old bridge configuration.
The bridge master is shown in the enslaved interface.
The interfaces overview is updated after a bridge is modified.

Optimizing read of hosts file

It was reported that yast2-network was slow in system with a lot of entries in the /etc/hosts file. We took that as an opportunity to test the new profiler support in YaST. The profiler revealed the problem was in some slow calls to SCR, the layer traditionally used by YaST to interact with the underlying system… which sounded like another opportunity. This time an opportunity to expand the use of CFA (the component we are developing to steady replace SCR) and its Augeas parser. Since Augeas already supports parsing of /etc/hosts files, it was quite straightforward to implement that into YaST… and the result is quite impressive.

The time needed to execute the next command in a system containing a huge /etc/hosts with around 10,000 entries (quite an extreme case, we know) was reduced from 75 seconds to just 20.

yast2 lan list

As you can see in this pull request, we also improved CFA itself, greatly reducing the time needed for reading configuration files with Augeas.

That’s all… until next report

Once again, we must conclude the report telling that this was just a small summary of all the work done during the sprint and that we will be back in three weeks with the next report. Or maybe before, now that we are starting to get used to blog more often.

In any case, see you soon and don’t forget to have a lot of fun!

Posted in Distribution, Factory, Localization, Programming, Systems Management, Usability, YaST | 1 Comment »

Reducing YaST rebuild time by 30%

October 11th, 2016 by Yast Team

Here comes the YaST team again trying to flood your aggregator with blog post! Now it’s time for the story of how we reduced the critical path of the rebuild time of YaST RPM packages from 42min 2s to 29min 40s.

Chapter 1: where to optimize

Of course, the first step to start fixing something is finding out what exactly needs to be fixed. In our case, this breaks down into

knowing the dependencies, and
knowing the individual build times.

Dependencies

It is tempting to figure out the dependencies by yourself, by parsing the spec files. But it is hard to do right, and, more importantly, a reinvention of the wheel. The Build Service must know all this to be able to schedule the builds, and provides a convenient way to access it, with osc dependson:

$ osc dependson YaST:Head openSUSE_Factory x86_64
[...]
yast2-x11 :
   yast2-devtools
yast2-xml :
   yast2-core
   yast2-devtools
yast2-ycp-ui-bindings :
   libyui
   yast2-core
   yast2-devtools

Individual build times

For each source package, the Build Service produces not only binary RPMs but also a _statistics file, available in the web UI or via osc getbinaries. We were interested in the total build time, although the data was of limited use because packages can be built on machines with vastly different power and this information is not included.

Chapter 2: how to optimize

Once we knew which screws needed to be tightened, it was time to do it. Fortunately we had more than one tool for the job.

Stop using Autotools

Autotools (automake, autoconf and configure) took up a majority of the time needed for building YaST packages. Now that most of those packages are written in pure Ruby, we don’t need autotools there checking for portability problems that we don’t have. Autotools are a leftover from the times 15 years back when they were the only sensible option. We have wiped them out where possible and have been switching to our own set of Rake tasks.

Stub the APIs used in tests

We run a mixture of unit and integration tests at RPM build time. The downside of this is that we pull in many of the run time dependencies. Fortunately Ruby is a dynamic language and makes it easy to replace interfaces by stubs. That enables us to cut those dependencies.

In fact, we also have some Perl code, notably in yast2-users. Although the stubbing techniques across languages are messier than with pure ruby, they are still effective for our purposes.

Do not build specialized documentation

This one is simple: if the development documentation is only useful for people that will check out the git repo anyway, then leave it out from the RPM.

Appendix: the details

Enough of high-level explanations, we we promised you graphs, code and all kind of gory details, and a promise made is a debt unpaid. So there we go.

Dependency graphs

A picture is worth a thousand words. That’s why we came up with this small tool to generate a graphical representation of the dependencies of the YaST packages. In the resulting graphs displayed below, a node is a source package in the build service, and an arrow means “needs for its build”. Redundant arrows are omitted (that is, we’ve erased an A→C if both A→B and B→C existed).

We can see that the most prominent conclusion is that there is a large number of packages that depend on yast2, a collection of basic libraries.

But on top of that, in the original graph there are 6 more layers, and the graph is not very dense there. After our fixes, there are only 4 layers that are more dense.

Is worth mentioning that the “layer” concept only works if the packages take roughly the same time to build; it would not be helpful if there were huge variations. To get a more accurate picture, we should have generated a histogram of build times. But the graph was good enough in our scenario… and we had to stop the analysis at some point. 🙂

The build dependency graph before our fixes:

The build dependency graph after our fixes:

Build statistics

If those graph are not geeky enough for you, here you are the detailed build statistics from the build service

<buildstatistics>
  <disk>
    <usage>
      <size unit="M">1118</size>
      <io_requests>15578</io_requests>
      <io_sectors>2156642</io_sectors>
    </usage>
  </disk>
  <memory>
    <usage>      <size unit="M">580</size> </usage>
  </memory>
  <times>
    <total>      <time unit="s">756</time> </total>         <!-- THIS -->
    <preinstall> <time unit="s">8</time>   </preinstall>
    <install>    <time unit="s">72</time>  </install>
    <main>       <time unit="s">555</time> </main>
    <download>   <time unit="s">4</time>   </download>
  </times>
  <download>
    <size unit="k">33564</size>
    <binaries>53</binaries>
    <cachehits>24</cachehits>
    <preinstallimage>preinstallimage.preinstallimage.tar.gz</preinstallimage>
  </download>
</buildstatistics>

Epilogue

This was definitely an interesting journey. We learned quite some things. Specially we learned that there is still room for improvement, but most likely the time reduction will not pay off for the time invested implementing those improvements.

We have to be realistic and keep working in other interesting stuff to fuel the next sprint report, coming next week!

Posted in Build Service, Distribution, Factory, Programming, Systems Management, YaST | 3 Comments »

Improving low-vision accessibility of the installer

October 7th, 2016 by Yast Team

In our latest report, we promised you would not have to wait another three weeks to hear (or read) from us. And here we are again, but not with any of the anticipated topics (build time reduction and Euruko 2016), but with a call for help in a topic that could really make a difference for (open)SUSE.

Nowadays, YaST team is trying to fix a long-standing issue in the installer: low-vision accessibility. In the past, a user could get a high-contrast mode just pressing shift+F4 during installation. Unfortunately, that feature does not work anymore and, to be honest, changing to a high-contrast palette is not enough. Other adjustments, like setting better font sizes, should be taken into account.

Another option is to use the textmode installation and set some obscure variable (Y2NCURSES_COLOR_THEME) to get the high-contrast mode. But it sounds like the opposite to user friendly.

Some days ago, the team fired up the discussion in the opensuse-factory mailing list but we would like to reach as many people as we can to gather information and feedback about this topic. Getting some affected people involved in the process would be really awesome!

For the time being we’re already working on some improvements:

Adding a Linuxrc option so the user can set the high-contrast mode from the very beginning.
Fixing shift+F4 support.
Improving the high-contrast mode appearance. Below you can see a screenshot of the work in progress.

But we would like to hear from you. You can raise your voice in the already mentioned thread at the opensuse-factory mailing list or leave a comment in the related pull request at Github. If you prefer to have a chat, we’re also available on the #yast IRC channel at Freenode… and we love to see people there. 😉

Please, join us to make YaST even better!

Posted in Accessibility, Distribution, Systems Management, Usability, YaST | 2 Comments »

Highlights of YaST development sprint 25

September 28th, 2016 by Yast Team

Another development sprint is over. Time flies! In our previous post we already reported about the branching of Tumbleweed and the upcoming releases and about the expected consequences: the landing of some cool features in a less conservative Tumbleweed.

We are still dedicating quite some effort to polish the upcoming stable releases (SLE12-SP2 and Leap 42.2), but in this sprint we finally found some time to play. Which is great because blogging about new features is more fun than doing it about bug fixes. 🙂

Importing Authorized Keys with AutoYaST

When logging in via SSH, public key authentication should be preferred over password authentication. Until now, the best way of setting up the required authorized_keys files in AutoYaST was using the files section.

However, that approach is tedious and error prone, as you need to make sure you set the correct owner, permissions, etc. Moreover you need to keep in sync the user definition (username and home directory) with the file definition.

AutoYaST now supports the specification of a set of public keys for each user with a pretty straightforward syntax:

<user>
  <username>suse<username>
  <authorized_keys config:type="list">
    <listentry>ssh-rsa your-public-key-1</listentry>
    <listentry>ssh-rsa your-public-key-2</listentry>
  <authorized_keys>
<user>

AutoYaST takes care of writing the files and setting the ownership and the proper permissions.

While documenting this new feature we realized the AutoYaST documentation about users management could be more detailed, which leads us to…

Improving the documentation

Usually developers love to create programs loaded with cool features but hate to write documentation. Fortunately there are people out there who enjoy writing documentation and bringing all those features to light. We have already mentioned in previous reports how grateful we are for having the SUSE documentation team polishing and publishing our documentation drafts and how open and straightforward the process is.

We updated the YaST documentation to include information about the installer self-update feature, which will debut in SUSE Linux Enterprise 12 SP2 and openSUSE Leap 42.2. As part of the same pull request and in the AutoYaST side, some additional improvements were made, including cleaning-up some duplicated information about SUSE registration.

On the other hand and as a consequence of the above mentioned new feature, the AutoYaST documentation regarding users management has been rewritten adding missing information like groups, user defaults and login settings.

All our pull requests are already merged in the doc-sle repository. At a later point in time, the SUSE documentation team will review and polish all the new content (including ours) and will publish an up-to-date version of the online documentation. If you don’t want to wait, you can easily generate an HTML or PDF version of the documentation including all the non-reviewed contributions just following the very simple instructions in the README file of the doc-sle repository.

Did we already mention we love the open source, programmer-friendly processes of the documentation team? 😉

Storage reimplementation: something you can touch

We promised news about the storage reimplementation and here they are. Our customized Tumbleweed image (labeled as NewStorage) in the storage-ng OBS repository can now perform some simple actions during installation and display the result to the user.

First of all, when proposing the timezone settings it will, as usual, check for MS Windows installations in the disk to guess if the hardware clock should be set to UTC. The news is that check is performed using the new storage stack, that offers more functionality in every sprint.

More important is that the installer will show the partitioning proposal calculated also using the new stack. As you can see in the screenshot below, the screen is way more simpler than the one you would find in a regular Tumbleweed. There are no buttons to change the settings or to run the expert partitioner yet. That doesn’t mean the functionality is not there, it’s simply that we prefer to focus first on modifying all the installer steps to use the new stack (what will enable us to use openQA) before refining every screen to add all options there.

Right now the system works only in disks containing a MS-DOS style partition table and will always propose a partition-based (no LVM) setup. That’s because we prefer to solve the hardest scenarios first. Using LVM and/or GPT partition tables is less challenging than the already supported scenario.

Reduce global warming by saving OBS build power

As you may know, we use the awesome Open Build Service (OBS) to generate both the YaST rpm packages and the openSUSE/SLE ISO images. Every time the source code of any component changes, OBS rebuilds that component and all the packages that depend on it.

Our beloved openSUSE and SLE release managers told us that there were several YaST packages that often triggered rebuild of other YaST packages, that triggered yet another rebuild, that triggered… you got the idea. 😉

The mentioned problem slows down the creation of new ISO images, interferes with the continuous integration process (specially visible in Tumbleweed) and wastes valuable OBS resources.

During this sprint we reduced the rebuild time of YaST by 30%. That’s for sure interesting, but knowing the details about how we did it could be even more interesting for many readers. We feared the explanation could be too complex and technical to fit into this report… which gives us just another opportunity for blogging. So expect an upcoming post including interesting technical stuff and crazy graphs like this one.

Some adjustment for the installer in the LiveCDs

One of the good things about working in open source is that sometimes the evolution of the projects you have created can surprise you. Quite some time ago, the YaST team dropped support for the live installer. It was simply too demanding to keep it alive while still doing our regular work (bug fixes and new features for YaST and the regular installer).

Recently the live installer was removed from Tumbleweed, the only system in which it was still available (after having been dropped in the past from stable openSUSE releases). One could have expected that somebody would decide then to step up and take the maintainership of the live installer.

But what actually happened was that Fabian Vogt decided to try a different approach we haven’t considered – adding the standard network installer to the LiveCDs images of Tumbleweed. He managed to make it work well enough and asked us for help to debug some problems. We fixed the initial problems by disabling the self-update functionality in the LiveCDs (it’s simply not designed to work on that scenario).

There are still quite some problems to be resolved to make everything work flawlessly, but if Fabian and others don’t give up, we will keep assisting them in order to bring the installation back to the LiveCDs… even in unexpected ways.

UI Designer

The YaST user interface is quite difficult to design and code. The main problem is that there is was no interactive UI designer where you could build a dialog or modify an existing one. Instead, you had to write new code or modify the existing code which creates and opens the dialog. Then, to see your changes you had to start the YaST module, go to the respective dialog and check its content. If it didn’t look like you intended, you had to close it, modify the code and start it again. And again… and again. Very annoying especially if the dialog is hidden deep in the module and you need to take several steps to get there.

During Hack Week 14, a project to improve the situation a bit was started. We already had a dialog spy which can be opened by Ctrl+Shift+Alt+Y keyboard shortcut, but that was read-only. You could only inspect the dialog tree and see the details of the selected widget but you could not change anything.

During that Hack Week project it was improved so it could edit the properties of the existing widgets, remove them or even add some new widgets. However the code was more or less a proof of concept than ready to be merged into the YaST UI and released to public. So we decided to finish it in this sprint.

As usual, it was harder than expected… but we made it and here is a short demo showing how it works and what you can do there:

The new UI designer in action

The new tool is still far from being perfect. The most obvious missing feature is that the dialog is changed in place and you cannot save or export you changes. After closing the dialog everything is lost. But it can still help to try things in the UI or make a quick prototype, specially when discussing solutions with interface designers. Hopefully we find some more time in the future to make it even better.

Storage reimplementation: encryption support

Although the partitioning proposal still does not support encryption or LVM, we implemented full LUKS (encryption) support in the underlying library (libstorage-ng). Together with the LVM support implemented in the previous sprint, that makes the new library already a valid replacement for the old libstorage in many situations and scenarios. Now it’s mainly a matter of switching from one version to another in every single YaST component, starting with the expert partitioner that we plan to start redesigning in the next sprint.

As usual, new features in the library are hard to illustrate, unless you accept the action diagrams as screenshots. In that case, here you can see the sequence of actions performed by the library when creating an encrypted home volume.

Syncing keyboard layouts and console fonts in Leap and Tumbleweed

In parallel to our Scrum sprints, we have been for some time steady working, together with the openSUSE maintainers of X.Org and systemd, in redesigning how keyboard maps and console fonts are managed by YaST. Some changes were introduced in Tumbleweed some time ago but never made it to SLE (or Leap) because they needed more testing.

Recently Ludwig Nussel, the Leap’s release manager, decided that he wanted to sync 42.2 with Tumbleweed in that regard, using the new approach instead of the more conservative SLE one. Thus, we also invested quite some time coordinating again with Stefan Dirsch (X.Org) and Franck Bui (systemd) to push the changes just in time for the beta2 version of Leap 42.2… just in time to introduce bug#1000565, that was honored with its inclusion in the list of most annoying bugs in 42.2 Beta2.

The bright side is that a fix for the bug has already been provided (see bug report) and you can now finally test the new fonts and keyboard maps. Please, do so and provide feedback in order to get a properly localized Leap 42.2 release.

See you soon

As usual, this post was just a quick overview of the most interesting part of the sprint, because most people (including ourselves) don’t want to read about the boring part of the work, which is mainly fixing bugs.

The good news is that this time you will not have to wait another three weeks to read interesting stuff about YaST. As mentioned, we plan to publish a blog post about the reduction of the build time of YaST. And we will probably also publish a post about the visit of a YaST geecko to Euruko 2016.

So this time more than ever… stay tuned!

Posted in Base System, Distribution, Documentation, Factory, Hackweek, Programming, Systems Management, YaST | Comments Off on Highlights of YaST development sprint 25

Highlights of YaST development sprint 24

September 7th, 2016 by Yast Team

We are back to this blog after another three weeks of (mainly) bug-fixing. In the previous post we promised some news about the self-update functionality and about the LVM support in the new storage stack. We have that… and much more!

So this will be a long post, but it also hides some gems. You will have to keep reading in order to find them.

Self-update improvements

We have already mentioned in several previous reports the new self-update feature in YaST, which allows updating the installer itself before performing installation of the system.

But it turned out that the initial implementation had an important drawback. The self-update process happened after having performed some of the installation steps. Then, after updating the installer it was restarted and several of those steps lost their configuration or simply did their operations twice.

After some discussions we decided to move the self-update step earlier, at the very beginning. For downloading the updates we basically need just working network connection and initialized package management. So we moved the self-update step after the initial automatic network setup (DHCP) and added package initialization to the self-update step.

The self-update in action

As you can see the self-update step is the very first step in the installation workflow, the language selection and the EULA dialog is displayed after the self update is finished and YaST is restarted. That means all the following steps do not need to remember their state as they will not be called twice after the restart.

The disadvantage is that we had to drop some features. The self-update step happens before the language selection and the optional disk activation. That means by default the self-update progress (and potential error messages) will be displayed in English. But you can still use the “language” boot option and set the language manually via linuxrc.

On the bright side, we fixed like half a dozen of reported bugs just by relocating the self-update process. So we are pretty sure it’s worth the price.

For more details see the updated documentation.

Gem one: using the `info` boot parameter

The info boot parameter is a pretty old linuxrc option but it is probably not known well. The parameter is an URL which points to a text file which can contain more boot options.

When we tested the updated self-update described above we needed to build a driver update disk and pass several boot options. To avoid repeating the same options on the boot command line and to share the boot options across the team we created an info.txt file with content like

insecure=1
startshell=1
dud=ftp://example.com/self_update.dud

Then you simply boot the installation with info=ftp://example.com/info.txt and linuxrc will read the additional parameters from the file. This can save you a lot of typing, especially if you need to repeat the tests many times.

Fixed a security bug for 7 (yes, seven) different SLE releases

Some weeks ago, during a routine code review, our security experts found a vulnerability in YaST’s libstorage related to the way we provide the encryption passwords to some external commands. It is debatable how dangerous this threat really is. It was never a problem during system installation, but it would affect admins who create encrypted partitions (mostly encrypted LVM physical volumes) or crypto files in the installed system.

A potential attacker with access to /tmp could intercept the password in the very precise moment in which the “cryptsetup” or “losetup” command are invoked by YaST. It’s really only a matter of milliseconds. But we don’t want to take any risks, however small they may be.

So not only did we fix that for the current code streams, we backported it to all the SLE releases out there that are still supported (even though in some cases it’s just a single customer) – back to SLES-10 SP3 from late 2009. That meant backporting the fix to no less than 7 SLE releases (for Leap, those fixes are picked automatically).

As you can imagine, this got more difficult the farther back in history we went: In a central library like libstorage, things are constantly changing because the tools and environment (kernel, udev, you name it) are constantly changing. There was only a single case where the patch applied cleanly; in all other cases, it involved massive manual work (including testing, of course).

Was this fun? No, it certainly was not. It was a tedious and most frustrating experience. Do we owe it to our users (paying customers as well as community users) to fix security problems, however theoretical they are? Yes, of course. That’s why we do those things.

Storage reimplementation: every LVM piece in its place

As time permits, we keep adding new features to the future libstorage replacement. During previous sprints we added support to read and manipulate all kinds of LVM block devices (PVs, VGs and LVs) but an important aspect was missing: deciding the order of the operations is as important as performing them. We instructed the library about the dependencies between operations and implemented several automated test cases to ensure we don’t try to do not-so-smart things like removing a physical volume from a volume group and shrinking its logical volumes afterwards.

The good thing about our automated test-cases is that they generate nice graph that are quite useful to illustrate blog posts. 🙂

Gem two: enjoy Google Summer of Code result

As you may know, openSUSE is one of the Free Software organizations selected to take part in Google Summer of Code 2016. For YaST that means we had the huge pleasure of having Joaquín Yeray as student. You can know more about him and his experience diving into YaST and Open Source in his GSoC blog.

But the openSUSE community is not only gaining a new member, we also have a new YaST module. The yast2-alternatives package has already been accepted into Tumbleweed and will be also part of Leap 42.2. So we have a new gadget in our beloved configuration Swiss Army knife!

We liked Joaquín and his module so much that we are revamping the YaST development tutorial to be based on his module (instead of yast2-journal). He is already working on that, so hopefully we will have Joaquín around quite some time still. 😉

Unify license handling screens

We got a report about the license agreement screen in automatic installation (AutoYaST) being different to the one showed during common installation. So we decided to take a look to the problem and unify them. We are in a quite late phase of the development process of both the next SLE and the next Leap, so we decided to not unify the code but simply adapt one dialog to look like the other. Also we are after string freeze due to translations, so we had to use a trick and reuse another already translated text. We also took the opportunity to fix some small usability problems.

This is one of those cases in which some images are worth a thousand words, so in order to understand what we did, take a look at the description of this pull request, which includes many images (too many for this blog post).

Smarter check to avoid duplicated repositories

The openSUSE software server defines the online repositories which can be added during installation. The openSUSE DVD also specifies its own online repositories that are always added to the system. And these repositories overlap.

In openSUSE 42.1 it happened that one repository was added twice, even though there was already a check to avoid that. So we investigated why.

We found that the URLs for the problematic repository were not exactly the same, one of them had a trailing slash. Therefore we made the URL comparison more tolerant and if the URLs differ only by the trailing slash, they are still considered the same.

After the fix all repositories are added only once, without any duplicates.

Gem three: we are looking for new teammates!

After 12 sprint reports, most readers would have already realized that the life as a full-time YaST developer is everything but boring… and that we are always pretty busy. The fun and the work are better when you share them so… we are looking for a new hero to join us in our journey.

Even if you don’t feel hacking in YaST would be your thing, maybe you are interested in any of the other jobs at SUSE.

Improved documentation about YaST environment variables

The behaviour of YaST can be affected by several environment variables, but not all of them are well known by everybody. During this sprint we also decided to invest some time documenting them better. The resulting document will be soon properly integrated in our centralized documentation for developers, but you can sneak it already here.

Branching Tumbleweed and the upcoming stable releases

Most of the features and bug-fixes we have blogged about in the last months were incorporated to Tumbleweed, the upcoming Leap 42.2 and the future SLE 12-SP2, since we always try to keep those three codebases as close as possible to each other.

Now Leap 42.2 and SLE 12-SP2 are close enough to their release date, so we plan to be more conservative with the changes. At the end of this sprint we decided to branch the code for Tumbleweed and for the stable siblings. From now on, most exciting stuff will appear only in Tumbleweed, with SLE 12-SP2 and Leap 42.2 becoming more and more boring.

And the wheel keeps on turning

So that was a very minimal selection of the most interesting stuff from the just finished sprint. What comes next? Another sprint, of course! We have already planned some interesting stuff for it, like integrating the new partitioning proposal into the installer or finishing the ultra-cool UI designer that was started during latest Hack Week.

As always, you can follow development in a daily basis in the usual channels (#yast IRC channel and the yast-devel mailing list) or wait another three weeks for the next sprint report. Meanwhile… have a lot of fun!

Posted in Distribution, Factory, GSOC, Programming, Systems Management, YaST | Comments Off on Highlights of YaST development sprint 24

Highlights of YaST development sprint 23

August 18th, 2016 by Yast Team

As already mentioned in our previous blog post, with Leap 42.2 in Alpha phase and SLE12-SP2 in Beta phase, the YaST Team is concentrating the firepower in fixing bugs in the installer. We fixed more than 40 bugs in three weeks! The dark side is that most bug fixes are not juicy enough for writing a blog post… but there is always some interesting stuff to report.

Integration of installer self-update with SCC and SMT

The installer self-update feature integrates now with SUSE Customer Center (SCC) and Subscription Management Tool (SMT) servers. Until now, there were three different mechanisms to get the URL of the installer updates repository:

User defined (using the `SelfUpdate` boot option).
Using an AutoYaST profile.
The default one, specified in the `control.xml` which is shipped in the media.

Now YaST2 is able to ask for the repository URL to SCC/SMT servers. The details of how the URL is determined are documented in the repository.

Fixes and enhanced usability in dialogs with timeout

As you may know, it’s possible to install (open)SUSE in an automatic, even completely unattended, basis using AutoYaST. AutoYaST can be configured to display custom configuration dialogs to the user and wait for the reply a certain amount of time before automatically selecting the default options. Until now, the only way for the user to stop that countdown was to start editing some of the fields in the dialog.

We got a bug report because that functionality was not working exactly as expected in some cases so, in addition to fixing the problem, we decided to revamp the user interface a little bit to improve usability. Now there are more user interactions that are taken into account to stop the counter, specially we added a new “stop” button displaying the remaining seconds. You can see an example of the result below.

Following, as usual, the Boy Scout Rule we also took the opportunity to add automated tests to make that part of YaST more robust for the future.

Automatically integrating add-on repositories during installation

Sometimes you want to extend the regular installation media by adding just a few extra packages or provide a number of fixed packages along with the media.

For this purpose, the installer can automatically register an add-on repository. All you have to do is to put the repository on the installation medium and to add a file /add_on_products.xml that points to this repository.

The file looks like this:

<?xml version="1.0"?>
<add_on_products xmlns="http://www.suse.com/1.0/yast2ns"
    xmlns:config="http://www.suse.com/1.0/configns">
    <product_items config:type="list">
        <product_item>
            <name>My Add-on</name>
            <url>relurl://myaddon?alias=MyAddon</url>
            <priority config:type="integer">70</priority>
            <ask_user config:type="boolean">false</ask_user>
            <selected config:type="boolean">true</selected>
            <check_name config:type="boolean">false</check_name>
        </product_item>
    </product_items>
</add_on_products>

You can define the following elements:

<name> is the name of your repository
<url> points the the repository location; you’ll likely want to use the relurl scheme here that gives the location relative to the main installation repository
<priority> is the repository priority (lesser number means higher priority, the main installation repository has priority 99)
<ask_user>: should the user be asked about adding the repository?
<selected>: should the repository be automatically selected?
<check_name>: should the repository’s actual name be matched against the value of the <name> element?

You can of course list several repositories in this file.

If you’re too lazy to remember all this, mksusecd can do all this for you.

For example, if you have a set of new kernel packages you would like to use, do:

mksusecd --create new.iso --addon kernel-*.rpm --addon-name 'my kernel' sles12-sp2.iso

This creates a new iso based on sles12-sp2.iso that will install your new kernel packages instead.

Storage reimplementation: small steps for the code, giant leap for continuous integration

During bug squashing we managed to find some time to keep the storage stack reimplementation rolling… slow and steady. The customized Tumbleweed images (labeled as NewStorage) in the storage-ng OBS repository are already able to analyze most systems, creating a representation of the system storage devices in memory that will be used to manipulate the disks and propose a partitioning schema. Unfortunately, this representation is only visible in the YaST logs since fixing installer bugs was more urgent than representing that information in the UI.

This turned to be an important milestone, not because of the functionality itself or the value of the code (we just added a couple of lines of Ruby code), but because for the first time the dependencies in some packages were switched from libstorage to libstorage-ng. That had important implications for the code organization and for our continuous integration infrastructure, specially the Travis CI integration, which implies the generation of .deb packages. We can now say that our continuous delivery workflow (from Github to OBS, passing through Jenkins, Travis, Coveralls and Code Climate) is free of any trace of the old storage code.

In addition, we also did some good progress in LVM support in the new library, being able to recognize and manipulate in memory all kind of LVM structures.

The joy of openness: updating the SUSE Linux Enterprise documentation

An important part of our job, specially as a new release date approaches, is helping to shape the SUSE Linux Enterprise (SLE) documentation. One of the strongest points of SUSE products is the awesome SUSE’s documentation team which, as the rest of the company, have open source in their genes. Suggesting improvements and updates for the documentation is as straightforward as creating a pull request in the completely open documentation repository at Github… and anybody can do it!

The documentation team uses Docbook, but they would accept contributions in other formats (e.g. Markdown) and transform it themselves into Docbook… just because they are that cool. 🙂

Better support for ARM systems using EFI

The world is getting full of cool ARM64 devices and both SUSE and openSUSE are actively working in supporting as many of them as possible. We took another small step during this sprint improving the installer’s partitioning proposal for ARM systems that use EFI.

That’s not all, folks

As we always say, this was just the small portion of the work done that we consider exciting enough to be part of our development reports, since we don’t want to bore you with details about every single fixed bug. During this installer bug-fixing phase, this is more true than ever and the next sprint, which is already planned, will be similar to this in that regard. Nevertheless, in the next report we expect to have some interesting news about the installer self-update functionality and about the LVM support in the new storage stack. Stay tuned.

Posted in Distribution, Documentation, Factory, Programming, Systems Management, YaST | 2 Comments »

Advertisement
Tags
11.3 11.4 12.1 12.2 12.3 13.1 13.2 amd ARM ATI Beta buildservice Build Service C-Language cloud Collaboration Community conference Education event Events Factory fglrx fun GNOME gsoc Hackweek KDE Kernel Kraft Linux LXDE obs openSUSE Package PostgreSQL radeon raspberry Raspberry Pi rpm Ruby Tumbleweed XML xorg YaST
Lizards
- Adrian Schröter (12)
- Agustin Chavarria (6)
- Alessandro de Oliveira Faria (13)
- Alex Barrios (12)
- Alexander Naumov (10)
- Alexander Orlovskyy (3)
- Alin M Elena (5)
- Andrea Florio (27)
- Andreas Jaeger (70)
- Andreas Stieger (12)
- Andrew Wafaa (31)
- Arvin Schnell (9)
- Atri Bhattacharya (3)
- Bernhard Wiedemann (31)
- Bonnie Kurniawan (1)
- Bruno Friedmann (98)
- Calumma Brevicorne (29)
- Carl Fletcher (1)
- Christopher Hobbs (17)
- Ciaran Farrell (3)
- Stephan Kulow (17)
- craig gardner (2)
- Stephan Barth (2)
- Thomas Schmidt (2)
- Dinar Valeev (1)
- Dirk Mueller (2)
- Dmitry Serpokryl (7)
- Efstathios Iosifidis (21)
- Fabio Mucciante (5)
- Federico Lucifredi (9)
- Greg Freemyer (1)
- Holger Sickenberg (2)
- Hubert Mantel (1)
- Ilya Chernykh (5)
- Ismail Donmez (1)
- J. Daniel Schmidt (2)
- James Tremblay (7)
- Jan Blunck (4)
- Jan Loeser (3)
- Jan Madsen (1)
- Jan-Christoph Bornschlegel (3)
- Jan-Simon Möller (20)
- Javier Llorente (12)
- Jigish Gohil (85)
- Jiri Srain (1)
- Jiří Suchomel (3)
- Johan Kotze (5)
- José Oramas M. (6)
- Josef Reidinger (16)
- Juergen Weigert (1)
- Julio Vannini (9)
- Dinar Valeev (5)
- Kevin "Yeaux" Dupuy (11)
- Klaas Freitag (55)
- Lars Vogdt (11)
- Ludwig Nussel (13)
- M. Edwin Zakaria (4)
- Marcus Hüwe (39)
- Marcus Meissner (2)
- Marcus Moeller (3)
- Marcus Schaefer (4)
- Martin Lasarsch (8)
- Martin Mohring (11)
- Masim "Vavai" Sugianto (20)
- Michael Andres (1)
- Michael Löffler (7)
- Michal Marek (7)
- Michal Vyskocil (12)
- Miguel Angel Barajas Hernandez (2)
- P Linnell (2)
- Nelson Marques (55)
- Nenad Latinović (1)
- Nikanth Karthikesan (2)
- Przemyslaw Bojczuk (1)
- Peter Pöml (4)
- Petr Gajdos (2)
- Petr Mladek (60)
- Petr Uzel (5)
- Ray Wang (1)
- Raymond Wooninck (1)
- Ricardo Chung (7)
- Ricardo Varas Santana (7)
- Richard Bos (11)
- Robert Schweikert (16)
- Rossana Motta (1)
- Rupert Horstkötter (10)
- Sascha Manns (66)
- saydul akram (3)
- Sebastian Siebert (6)
- Shawn Dunn (2)
- Stanislav Visnovsky (7)
- Stefan Haas (1)
- Stefan Hundhammer (5)
- Stefan Schubert (7)
- Steffen Winterfeldt (8)
- Suresh Jayaraman (3)
- Susanne Oberhauser (3)
- Thomas Göttlicher (6)
- Thomas Schraitle (26)
- Togan Muftuoglu (3)
- Tuukka Pasanen (36)
- Will Stephenson (22)
- YaST Team (90)
Archives
- March 2020 (1)
- February 2020 (2)
- January 2020 (1)
- December 2019 (3)
- November 2019 (2)
- October 2019 (4)
- September 2019 (3)
- August 2019 (3)
- July 2019 (4)
- June 2019 (2)
- April 2019 (4)
- March 2019 (3)
- February 2019 (5)
- January 2019 (1)
- December 2018 (2)
- November 2018 (2)
- October 2018 (3)
- September 2018 (1)
- August 2018 (3)
- July 2018 (2)
- May 2018 (2)
- April 2018 (2)
- March 2018 (2)
- February 2018 (2)
- January 2018 (2)
- December 2017 (1)
- November 2017 (2)
- October 2017 (2)
- September 2017 (3)
- August 2017 (4)
- July 2017 (4)
- June 2017 (2)
- May 2017 (4)
- April 2017 (2)
- March 2017 (3)
- February 2017 (3)
- January 2017 (2)
- December 2016 (5)
- November 2016 (3)
- October 2016 (6)
- September 2016 (2)
- August 2016 (3)
- July 2016 (4)
- June 2016 (2)
- May 2016 (2)
- April 2016 (1)
- March 2016 (2)
- February 2016 (4)
- January 2016 (4)
- December 2015 (6)
- November 2015 (2)
- October 2015 (3)
- September 2015 (2)
- August 2015 (2)
- July 2015 (2)
- June 2015 (3)
- May 2015 (12)
- April 2015 (7)
- March 2015 (6)
- February 2015 (6)
- January 2015 (7)
- December 2014 (5)
- November 2014 (3)
- October 2014 (5)
- September 2014 (3)
- August 2014 (5)
- July 2014 (5)
- June 2014 (7)
- May 2014 (9)
- April 2014 (2)
- March 2014 (9)
- February 2014 (9)
- January 2014 (10)
- December 2013 (9)
- November 2013 (10)
- October 2013 (10)
- September 2013 (6)
- August 2013 (7)
- July 2013 (3)
- June 2013 (7)
- May 2013 (4)
- April 2013 (4)
- March 2013 (7)
- February 2013 (6)
- January 2013 (3)
- December 2012 (3)
- October 2012 (6)
- September 2012 (6)
- August 2012 (5)
- July 2012 (12)
- June 2012 (6)
- May 2012 (4)
- April 2012 (4)
- March 2012 (5)
- February 2012 (2)
- January 2012 (5)
- December 2011 (10)
- November 2011 (6)
- October 2011 (5)
- September 2011 (9)
- August 2011 (12)
- July 2011 (14)
- June 2011 (11)
- May 2011 (18)
- April 2011 (15)
- March 2011 (26)
- February 2011 (16)
- January 2011 (23)
- December 2010 (27)
- November 2010 (18)
- October 2010 (21)
- September 2010 (16)
- August 2010 (21)
- July 2010 (20)
- June 2010 (33)
- May 2010 (29)
- April 2010 (24)
- March 2010 (29)
- February 2010 (22)
- January 2010 (20)
- December 2009 (15)
- November 2009 (21)
- October 2009 (17)
- September 2009 (22)
- August 2009 (28)
- July 2009 (36)
- June 2009 (38)
- May 2009 (40)
- April 2009 (30)
- March 2009 (20)
- February 2009 (21)
- January 2009 (27)
- December 2008 (23)
- November 2008 (12)
- October 2008 (23)
- September 2008 (40)
- August 2008 (24)
- July 2008 (12)
- June 2008 (28)
- May 2008 (26)
- April 2008 (1)