Archive for the ‘Packaging’ Category

openSUSE OBS git mirror

September 13th, 2019 by bmwiedemann

There was some discussion about our OBS and how in contrast Gentoo, VoidLinux or Fedora used git to track packages.

So I made an experimental openSUSE:Factory git mirror to see how well it goes and how using it feels.
The repo currently needs around 1GB but will slowly grow over time. I did not want to spend effort to import all history.

Binary files are replaced by cryptographically secure symlinks into IPFS
and I am currently providing files up to 9MB there.

If you can not run ipfs, you can still get these files through any of the public gateways like this:
curl https://ipfs.io$(readlink packages/a/aubio/aubio-0.4.9.tar.bz2) > OUTPUT

So some benefits are already obvious.
It is now much easier to find and download our patches.
Downloading and seaching all of openSUSE is now much faster.
And it works even on Thursdays (when our maintenance window often causes OBS downtimes).

It is meant to be a read-only mirror, so there is no point in opening pull-requests on github.

I hope, you enjoy it and have a lot of fun…

Tags: git, IPFS, mirror, obs
Posted in Build Service, Distribution, Factory, Miscellaneous, Packaging, Security | 2 Comments »

Highlights of the Latest YaST Development Sprints

June 25th, 2019 by Yast Team

May and June have been, so far, interesting months for the YaST Team. We worked hard to polish the last details of the recently released openSUSE Leap 15.1, we attended the openSUSE Conference 2019 (with many fruitful conversations), we shared quite some time together around a table without computers (most of the time, we are a geographically distributed team), many team members enjoyed vacations (it’s spring time in Europe), we organized a Leap 15.1 launch party with technical talks in Gran Canaria… and we ran out of energy to also publish our traditional sprint reports in this blog.

We will try to fix that with this blog post in which we will try to summarize some highlights from the latest three development sprints, namely the 77th, 78th and 79th. So be warned, this is going to be a loooong post.

Support for Multi-device Btrfs File Systems

We have been working steadily during the three sprints in implementing all the necessary bits to offer a good experience installing and upgrading an (open)SUSE system on top of several block devices by means of Btrfs RAID capabilities. That includes support in the Partitioner, in AutoYaST, in the storage guided setup and more.

We decided that all that deserved a separate blog post. You can find here: Getting Further with Btrfs in YaST.

More Improvements for the Partitioner

That blog post mentions a couple of changes in the Partitioner that, although initially motivated by the introduction of multi-device Btrfs, go beyond that scope and are aimed to make the all the lists of devices more useful and informative.

Traditionally the Partitioner used two separate columns “Type” and “FS Type” to describe the function of every device. That was sometimes hard to understand. Moreover, quite often the important information (like the relationship between a partition and its RAID or LVM) was simply missing in those tables.

We have merged those columns into a more informative one that identifies the devices and also gives an overview of the relationship between them at first glance. In addition, the table displaying all the system devices now includes multi-device file systems.

Mitigating CPU vulnerabilities from YaST

If you are interested in security (or simply if you have not been living under a rock) you probably have heard about CPU based attacks like Spectre or Meltdown. The last year has seen a number of these CPU issues, all of them coming along with their own kernel options to change the Linux behavior in order to mitigate the security risks at a price of some performance loss.

However, not all users know what affects their architectures or particular models of CPU and which kernel parameters to use to gain more performance if the security risk is acceptable for them.

For that purpose, a new meta-option called “mitigations” was added to the Linux kernel. It allows to enable and disable at once several of those mitigations that prevent CPU attacks. See more information at this document published by SUSE.

We find that kernel option very useful, so we decided to provide an easy way for users to adjust it. Now the YaST bootloader screen contains a new setting which offers three pre-defined options and even a fourth one to let the users fine-tune the settings on their own. As you can see in the screenshot below, we have included extensive documentation in the help dialog, so you will not need to search for this blog post in the future.

It is also possible to modify this option directly from the installation summary. For that purpose, the “Firewall” section was renamed to “Security” and it now includes the possibility to tweak the CPU mitigation options, alongside the traditional settings for firewall and opening the SSH port.

Another success story of (open)SUSE offering a promptly solution for our users to easily adapt their systems to ever-changing complex needs.

Memory Optimizations during Installation

While the release of openSUSE Leap 15.1 was approaching, we got several bug reports stating the YaST installer used to freeze when using only 1GB RAM with online repositories (see bug#1136051).

It turned out that at some point YaST loads the details of all available packages. And that needs a lot of memory if you enable the online repositories during installation. For example the OSS Leap repository contains more than 35.000 binary packages!

The problem was in the YaST internal API accessing the package manager library (libzypp). It did not allow to filter the objects, YaST had to read all objects and then do the filtering in the code. And for each object it returned all attributes, even those which were not needed (like the package description, full RPM name, etc…). All that data required a lot of memory.

To fix that we have introduced new API calls that allow specifying more filters (like return all selected packages, packages from specific repository,…) and you can set which attributes should be returned. If you need to know only the name and the version then you will not get the other useless attributes. And to ease the usage of the new API in YaST we provided a nice object oriented wrapper written in Ruby.

This optimization saves a lot of memory, 1GB of RAM should be enough for future installations with the online repositories, even if they grow even more.

Unfortunately, we were only able to diagnose the problem and provide a solution a couple of weeks before the official release of Leap 15.1. Introducing a change in such a sensitive part of the installer was considered too risky (it would have invalidated many of the tests that had been already performed) so the installer included in openSUSE Leap 15.1 is still memory hungry if online repositories are used. For that release, we simply increased the official memory requirements to 1.5 GiB.

Online Migration from openSUSE Leap 15.1 to SLES15-SP1

For openSUSE Leap 15.0 it was only possible to migrate from Leap to SLES (SUSE Linux Enterprise Server) manually (see the documentation). With Leap 15.1 the goal was to also support a migration using YaST. But we got a bug report saying the online migration from openSUSE Leap 15.1 to SLES15-SP1 displayed a wrong migration summary and didn’t work well.

It turned out that YaST needed some small fixes to support this properly. The main problem was that YaST did not expect that the base product or the package vendor can be changed during online migration, so far it was only possible to upgrade to the next SLE service pack level. But that is fixed now.

Wanna try the migration from openSUSE Leap 15.1 to SLES15-SP1? Then follow these steps.

Install the yast2-registration and yast2-migration packages in the Leap 15.1 installation
Make sure the latest online updates are installed (to install the fixes mentioned above)
Start the YaST registration module and register the openSUSE Leap 15.1 product using your registration key
Then start the YaST migration module, select the migration to SLES15-SP1
(There might be reported some package dependency issues in the migration summary, go to the package manager and resolve them. Usually removing the old openSUSE package is the right solution.)
Start the migration, the SLES packages will be downloaded and installed
At the end the system will be rebooted to start the freshly installed SLES, enjoy! 🙂
(It is recommended to review the orphaned packages, leftovers from the Leap installation, with the command zypper packages --orphaned and possibly remove them.)

Please note that only minimal server installations of Leap are supported for upgrade, full installations especially with third party packages might not work correctly.

Why Cannot I Read the Logs?

Long time ago, the logs of any Linux system were spread over several files living under the /var/log subdirectory. YaST offers its “System Log” module to inspect those files in a convenient way. Since the introduction of Systemd and its journal, that information has been gradually moved to this new mechanism by default. And YaST offers its “Systemd Journal” module to inspect and query that journal.

Both YaST modules can be executed by any user in the system, not only by root. That’s intentional because both the Systemd journal and the traditional Linux log files can register information targeted to unprivileged users. But there was some room for improvement in the error message displayed by both modules when such users were trying to access to protected information.

This is how the new more explanatory message looks in the “System Log” module.

And this is the extended message for “Systemd Journal”, that now mentions the systemd-journal user’s group.

Another Day at the YaST Office: Adapting to Changes

As we usually remark in our blog posts, a significant part of the work of the YaST Team consist in keeping YaST in sync with the constant changes in the underlying system. Of course, these sprints were not an exception in that regard. Without trying to do an exhaustive list, let’s take a look to some of those adaptations since the mentioned underlying changes may be interesting for some readers.

Turns out Systemd developers has decided to change the list of possible states for Systemd services. The systemd-sigabrt state is obsolete and a new systemd-watchdog one was added. In the YaST team we learned some time ago that the list of Systemd states changes more often than what most people would expect. As a consequence we have an automated check to detect these situations. The bell ringed, we adapted the code and everything keeps working.

Systemd is not the only technology that keeps evolving. Quite some time ago, RMT replaced SMT as the default proxy technologies for the SUSE Customer Center. Although both has coexisted for quite some time, from SLE-15 onward only RMT is offered. Thus we have adapted all the references to SMT that still existed in YaST. From now on, only RMT is mentioned to avoid confusion.

Another common adaptation we have to perform in YaST is adjusting some module when the output of the command it runs under the hood has changed. Recently we found out the developers of the command iscsiadm has decided to use more than one exit code to indicate a successful execution (traditionally, only zero should mean that). After a long discussion, we decided to adapt YaST iSCSI to also be happy with the error code 21. What means for you? That future versions of YaST iSCSI should work faster in some situations, since the confusion will not longer result in a timeout.

YaST Firstboot: a Better Example File

Those were just some example of the many adaptations we did lately for changes in the system. But not all adjustments are motivated by external changes. We also realized the example configuration file provided by YaST Firstboot (at /etc/YaST2/firstboot.xml) needed some love. Due to the nature of YaST Firstboot, that file should be customized before using YaST Firstboot. But providing an example file with three different steps about acceptance of licenses (two of them enabled and the third one disabled) and other inconsistencies was definitely not helping anyone to understand how to use the module.

In fact, the status of that default example configuration and of the documentation managed to confuse even SUSE’s quality assurance team. So we improved the example file shipped in the package to make it more realistic and we also updated the YaST Firstboot documentation to clarify how to use that file. Because not all improvements are always done by coding.

Leap 15.1’s Most Annoying Bug: Create Home as Btrfs Subvolume

As all openSUSE users should know, for every Leap release a page is created in the openSUSE wiki listing the so-called Most Annoying Bugs. Leap 15.1 was a very smooth release and this time the corresponding list contains only one bug… and it’s a YaST one. 🙁

In an installed system, when YaST was creating a new user it was always trying to create its home directory as a Btrfs subvolume. Even in cases in which that was impossible. For example, if the directory to be created was not in a Btrfs file system.

Fortunately, the bug didn’t affect the installation process or AutoYaST. We created a fix that was quickly available as a maintenance update. So make sure your openSUSE system is updated before trying to create new users with YaST.

YaST Network refactoring: status report

Since we submitted the first bits of the yast2-network refactoring by the end of April, we have made quite some progress in this area. Although it is still an ongoing effort, we would like you to give you an update on the current state.

We might say that we have been working on two different areas the user interface implementation and the future data model.

About the user interface, the team has improved the code quite a lot to make it easier to maintain and extend. We have introduced some classes to de-couple the widgets from the data, and it pays off. Additionally, we have fixed some bugs (many of them related to validations), simplified the process of adding new devices and reorganized the hardware tab.

Regarding the internal data model, we have been thinking about the best way to represent network configuration in an agnostic way so, in the future, we can not only support Wicked but other options too (for the time being, the NetworkManager support is quite limited). If you are curious about the details, we have added a document describing the approach to the repository.

The new data model is already being used to handle DNS and Routing configuration. So if you are using Tumbleweed you have been already using the new network code for some weeks, including the UI enhancements presented in our latest post.

Although the data model is so far only used in the mentioned parts, the plan is to submit to Tumbleweed a heavily refactored UI layer during next sprint. So stay tuned.

Added Appstream Metadata to the YaST Packages

The YaST package manager is not the only software manager in the (open)SUSE distributions. There are some more, like Discover in KDE or the GNOME software manager, not to mention the online openSUSE appstore.

While the YaST manager is package oriented, those other software managers are application oriented. That makes a huge difference, especially for the beginner users.

The full list of packages does not only include the applications (basically anything a user can start from the desktop), but also shared libraries, pieces that provide functionality for other applications or basic components needed for the system to work. With so many software packages (the openSUSE OSS repository contains over 35.000 of them!) it’s sometimes hard to find the software you need unless you know what you are looking for.

To offer an application-oriented view on top of all that, the application managers need some special data describing the applications inside the packages. The data are located in the /usr/share/metainfo/*.xml files, if you are interested in the technical details check the AppStream documentation provided by the Freedesktop.org.

Our absolutely awesome community contributor Stasiek Michalski (more famous by his nickname lcp) realized YaST was not offered in those application managers and decided to fix it. So he created an XML generator which collects the data from YaST packages and automagically generates the metainfo XML file needed by the other software managers.

As a result, the Gnome Software Manager, Discover and other software managers will offer YaST in Tumbleweed just as any user-oriented application. Thanks lcp!

AutoYaST Pre-install Scripts & Storage

AutoYaST has a special feature to allow users to customize the installation process and take control in different stages of the installation. For that, the AutoYaST profile offers a section where you can insert your custom scripts. There are four types of scripts: pre-scripts, postpartitioning-scripts, chroot-scripts, post-scripts and init-scripts.

For the particular case of pre-scripts, the documentation states that “It is also possible to change the partitioning in your pre-script“. That means, for example, you could use a script to create a new partition or to configure some kind of technology. Therefore, it would be very convenient to re-analyze the storage devices after running the user pre-scripts. In fact, that was the default behavior in the old storage stack, but the new one was slightly modified to only re-analyze the system under certain conditions.

But turns out some SLE customers were using pre-scripts to configure the behavior of multipath and those changes were not being noticed by AutoYaST.

The solution was quite trivial. We simply decided to always perform a new storage re-analysis after the AutoYaST pre-scripts. We did not find strong reasons to not do it and there should not be significant performance penalty.

And, for the specific case of multipath, YaST now copies some configuration files (e.g., /etc/multipath.conf and /etc/multipath/bindings) to the target system when performing a new installation. Otherwise, the installed system would not contain the configuration applied during the installation.

Clarifying the Usage of Software Management Options

Our software manager is one of the most complex YaST modules, which makes some aspects of its usage not fully obvious for some users. You may have even notice that is the only YaST module in which the interface is clearly different when executed in text and graphical mode. Specially the menu at the top of the interface, which is organized in different sections.

Some users where confused by the fact of some options not being persisted between different executions of the module. Those options are there to modify the current operation of the module, not to change the configuration of package management in the system.

When executed in text mode that was clear because such options where labeled as “temporary change” but the graphical mode didn’t have any indication about it. As you can see in the following screenshot, that’s fixed now.

Product License Hard to Understand? Try Another Language!

Some user reported that in the text-mode installation it was impossible to switch the language of a product license. Although during graphical installation everything works flawlessly, in text mode the language switching widget was there… but disabled.

The point is that such behavior was not exactly a bug. It was in fact done on purpose. We decided to prevent such language change some time ago because on the Linux console we’re not able to display the characters of many languages. Hopefully some of our usual reader have just shout “that’s not true!” 😉 Those users remember that in the report of our 67th sprint we explained that now we always use fbiterm when installing in text mode in order to be able to display characters of almost every language.

We are now able to display all languages that currently have license translations, so we have enabled the language switching widget and now both graphical installation and text-based one deliver an equivalent user experience.

More about languages

This is not the only change related to internationalization we did in these sprints. We also added a specific warning message for situations in which YaST is used to change the language of the system but there is no repository containing the needed translation packages. Something that obviously only affects users configuring system in very restricted environments.

As you can see in this and other recent reports, we have to deal relatively often with difficulties related to translation and internationalization. To reduce the effect of those problems on our final users, we also added some extra mechanisms to detect internationalization errors introduced during development. Hopefully that will mean that in future reports the space dedicated to comment language-related problems gets shorter and shorter. 🙂

And that was just a summary!

As long as this post looks, there are many interesting things we have done in these weeks we left out, intentionally or not. We definitely should avoid skipping three reports in a row in the future!

This week it’s Hack Week at SUSE, which means regular YaST development will be put on hold… or will turn into something completely different. You never know what the result of a Hack Week can be!

But in any case we will go back to our sprint-based pace in August. So expect a new blog post in three weeks. See you then!

Posted in Distribution, Factory, Miscellaneous, Packaging, Programming, Software Management, Systems Management, YaST | Comments Off on Highlights of the Latest YaST Development Sprints

Highlights of YaST development sprint 39

July 31st, 2017 by Yast Team

openSUSE 42.3 is out! Do you need some reading material while you wait for the download of the new release to finish? Don’t worry, we have the solution right here – another YaST team report. 😉

Several products in one installation medium

Obviously, we stopped adding new features to SLE12-SP3 and Leap 42.3 some days ago, because everything needed to be tested properly before the release. So now we are mainly looking into the future. And one of the plans for that future regarding SUSE Linux Enterprise (SLE) is offering several products packed in a single installation medium.

SUSE offers several mission-specific products based on SLE and, so far, every product needs to be installed from its own medium (usually a DVD or a virtual image). So if you use SLE Server, SLE Desktop and SLE Server for SAP, you have to have three DVDs which is a bit cumbersome.

After some discussions about the technical implementation details, we created a first prototype of the installer with an extra dialog that allows to select one of the detected products and continues the installation from there according to the installation workflow of the chosen product. It is still a proof of concept, but we can at least share screenshots showing how it looks for the time being.

So far, there are no plans to use the new feature in openSUSE, mainly because the project does not deliver separate mission-specific products in the same way than SUSE does with SUSE Linux Enterprise.

Storage reimplementation: numbers after repatriating the Expert Partitioner

In the Sprint 36 report we presented the rewrite of the YaST Partitioner and we have been informing about its evolution in subsequent reports. We told you back then that we decided to split it in a separate yast2-partitioner package. But time has proved that decision to have too many drawbacks so we decided to bring the Partitioner back home, to yast2-storage-ng. As part of the process, we got rid of an old previous prototype of the Partitioner that was still lying in the yast2-storage-ng repository and some code that was there just to support that old prototype.

You may be asking why is all that relevant. It is because that means the repository (and thus the package) is finally approaching the final structure it will have when released into Tumbleweed. And that implies that all the systems we use to automatically measure the quality and reliability of our repositories are now providing trustworthy results… as trustworthy as automatic quality evaluations can be.

And according to those tools:

93% of the code in yast2-storage-ng is covered with automated unit tests in addition to openQA (this number is expected to raise in close future as we polish the new Partitioner),
Code Climate reports a code quality GPA of 3.91 out of a possible maximum of 4
and 76% of all the classes, modules and functions, including the internal ones, are properly documented (with that developer documentation being available here, by the way).

If you wonder about the numbers for the old codebase we want to replace, its code quality is 0.94 and only a 31% of it is covered by unit tests. A perfect example of legacy code.

Storage reimplementation: Btrfs subvolumes in AutoYaST

As reported in previous posts, the new storage stack can already process AutoYaST profiles including partitions, LVM and MD arrays, but some details are still missing. The first of those details we wanted to address was the definition and creation of subvolumes in a Btrfs file-system.

Now it works according to the official documentation – both syntaxes for the <subvolumes> section are supported, it never creates subvolumes that would be shadowed by any other mounted file-system and it uses the list in control.xml as fallback for the root partition if Btrfs is used but subvolumes are not specified.

All that, as usual in the re-implemented stack, with fully tested and documented code.

Storage reimplementation: handle multipath I/O in the proposal

In the previous report we showed you how the support for Multipath I/O looked at the library level, which usually means just geeky graphs. During this sprint, we have taught the installer to use that new library feature, so we now have real screenshots to show!

During the installation, now multipath hardware is detected and the user is asked for activation.

If the user agrees, the installer will never use the individual disk devices to propose a partitioning layout and it will not offer them as an option during the guided setup. The installer always works on the final (compound) multipath device, proposing the correct names for the partitions and so on (which, being a devicemapper device, follows a
different pattern when compared to raw devices).

The resulting system is still not fully bootable because yast2-bootloader has still not been adapted to this scenario. Very likely, something for the upcoming sprint, so stay tuned.

Support for Ruby 2.4

The world changes every day and we are always adapting YaST to remain shiny. The 2.4 version of Ruby is about to land in openSUSE Tumbleweed and is expected to be the default Ruby for SUSE Linux Enterprise 15 and openSUSE Leap 15. We found that some of the YaST packages were not fully ready for this new Ruby, so it was time for some tweaking.

After dealing with quite some details too technical and boring for this blog (but feel free to ask if you want the gory bits), YaST is shining again in Factory, which means we are no longer blocking the adoption of Ruby 2.4 in Tumbleweed.

Add-on Creator and Product Creator

As our team keeps always developing new features, solving bugs, and receiving feedback, we always evaluate our priorities and product. Sometimes, during this evaluation, we see that some YaST modules do not bring enough value or do not shine enough as part of our standard package.

After some evaluation, we come to announce that the modules Add-on Creator and Product Creator will no longer be part of YaST. These packages use Kiwi as backend and we have high competition on UI sides – SUSE Studio and Open Build Service. So it no longer makes sense to have these packages and we recommend for users of these modules to use one of the alternatives or Kiwi directly if you already have an XML definition file for Kiwi.

Adapting YaST to accept 12 digits Service Request Number

As Service Request Numbers can be now composed of 11 or 12 digits, instead of only 11 digits as before, we had to adapt YaST to handle this change. YaST module Support can now accept 11 or 12 digit service request numbers. We implemented such a change for all products dating back from SLE 10 SP3 until the most recent SUSE Linux versions. Updates with this change will be soon released.

Network Setup in the 1st Stage of Autoinstallation

The YaST installation used to have two stages, separated by a reboot. Starting with SLE 12 and openSUSE Leap 42.1 we have eliminated the
second stage. But it was still needed for AutoYaST, controlled by the setting

<profile><general><mode><second_stage>true | false</...>

We have fixed those parts of the networking setup and now you can explicitly set AutoYaST to not use a second stage anymore.

User settings in AutoYaST

An issue that we have found out is that GDM has problems with the system when different users have the same UIDs. If it happens, GDM does not start properly. As a solution, we decided that either UIDs will be defined in the AutoYaST configuration file for ALL users or this tag will not be used at all for ALL users since a mix of both can result into duplicate UIDs.

And we just keep YaSTing!

We hope you liked our report as much as we loved to build all of that. We’ll continue YaSTing so we can reach you again in two weeks with much more cool stuff to show.

By now, enjoy your openSUSE 42.3 and all the cool features that came with it!

Posted in Distribution, Factory, Packaging, Programming, Systems Management, YaST | 3 Comments »

Highlights of YaST development sprint 38

July 14th, 2017 by Yast Team

Here we go again with a new report from the YaST trenches. This time with the storage reimplementation as the clear star of the show.

Storage reimplementation: the proposal adapts, you succeed

As we have announced in our previous sprints and as you probably already know, the YaST team is working hard to rewrite the whole storage stack on time for SLE15 and openSUSE Leap 15. As part of this reimplementation we have designed a brand new storage proposal that automagically offers the user the best combination of partitions and LVM volumes based on the current configuration of the system and the user preferences.

When we are working with very small disks or with special technologies like DASD (which doesn’t accept more than three partitions by device), the storage proposal might not be able to generate a valid initial proposal honoring the initial requirements of the product (e.g. creating a separate home partition and enabling btrfs snapshots for the root partition in the openSUSE Leap case). Now the proposal is not limited to fail when it is not possible to satisfy the default product requirements. Before giving up, the new system looks for alternatives, like discarding the separate home partition or disabling snapshots. Moreover, now the proposal is able to automatically adjust the size requirements not only for root, but also for swap and home. And, of course, the guided setup continues there for fine tuning the proposal settings.

Desktop selection improvements

As our usual readers also know, we recently introduced a more fair desktop selection screen for openSUSE, both Leap 42.3 and Tumbleweed. We used part of the latest sprint to implement some feedback we gathered about the wording and behavior of that dialog.

That feedback gathering included some discussions on how to make user experience nicer after selecting one of the user interfaces available through the “custom” option. As a result, the awesome openSUSE crew created a new mechanism for selecting the default window manager on each graphical login, so YaST can delegate the details to the maintainers of those alternative interfaces.

How everything works now? Glad you asked. 🙂

If the user select KDE or GNOME in the YaST dialog, /etc/sysconfig/windowmanager is configured to point to that desktop by default. If the “custom” option is selected, then YaST does not enforce any interface in that file and the new mechanism comes into play. It relies in the default.desktop file, which defines the default window manager and can be managed by the common update-alternatives workflow. Meaning it can be easily tweaked by the package maintainers and by the users, specially since YaST includes a nice module for managing alternatives.

Storage reimplementation: improvements in the expert partitioner

Although, as explained above, we keep improving the storage automatic proposal to support more and more situations, we cannot ignore that flexibility and adaptability have always been two of the flagships of (open)SUSE. And one of the most prominent examples is the YaST Expert Partitioner.

As detailed in our report of the sprint 36, we have been rewriting that powerful Swiss Army knife using the new storage stack but keeping the same user interface and functionality. So far, the new implementation was only able to display information about the existing partitions, LVM systems and MD RAIDs. But now we added many options to create, edit and delete partitions.

It is still a work in progress because the number of possibilities offered by the YaST Partitioner is sometimes overwhelming and implementing them all takes time, but it’s progressing nicely.

Beside the improvements in the Partitioner itself, we also worked on its integration in the installation workflow. Now the Expert Partitioner can be used to refine the schema automatically proposed by the installer. As a bonus, the behavior of the “abort” and “finish” buttons has been improved in relation to the Expert Partitioner currently available in (open)SUSE, which historically shows a usability inconsistency there compared to the rest of the installation process.

Fixed Automatic Patch Installation

In both SLE and openSUSE Leap, the online updates can be installed either manually or automatically at some regular intervals. For the automatic installation we provide the yast2-online-update-configuration package which provides a cron job script and an YaST module for configuring it (hint: it is not installed by default, you might give it a try, maybe it is something “new” for you).

You can configure how often the patches should be installed or filter the patch categories in the YaST module, but we got a bug report saying that when multiple patch categories are selected only one is actually used.

It turned out to be a trivial mistake in the cron job and we fixed it for SLE12-SP2 and Leap 42.2, as well as for the upcoming SLE12-SP3 and Leap 42.3. So if you use this module and want to use the category filter then it’s recommended to upgrade the package.

Storage reimplementation: many steps forward in the AutoYaST integration

And going back to our new storage stack, we keep working to integrate it better with other parts of YaST, specially AutoYaST. During this sprint we polished some rough edges and we added support for MD RAID. With all that, now is possible to automatically setup a system based on an AutoYaST profile containing any combination of partitions, LVM systems and MD arrays, including encryption at any of the levels.

But the relation of AutoYaST with the system works in both directions. Apart from being able to install a system based on an AutoYaST profile, it also offers the interface to export the current system configuration, including the storage layout, to a profile in order to reproduce the system later. During this sprint we also ported that logic to rely on the new storage layer.

It was harder than it sounds, due to the need of keeping the backwards-compatibility with several behaviors that have been introduced along the AutoYaST history to adapt to several specific situations. On the bright side, the new code is easier to follow, includes behavior-driven automated tests (RSpec) and contains information about the rationale of each decision… which in some cases required some archaeology.

Storage reimplementation: fixed bootloader proposal in S/390 and PowerPC

Another part of YaST we are constantly working to integrate better with the new storage stack is yast2-bootloader. Although the new storage system was already able to correctly setup a valid disk layout for most situations and architectures (each one with it’s own requirements for booting), our bootloader module is still not fully compatible with the new system in all those scenarios.

During this sprint, we adapted it to ensure all the combinations suggested by the new storage stack (partitions, LVM, encryption and so on) are correctly covered by YaST2-Bootloader. As a result, we can already say that our testing ISO images are fully installable in any x86_64, PowerPC and S/390 system by just clicking “next” a few times. And we have automated integration tests (a separate openQA instance) to ensure the resulting system boots just fine.

UDEV device id on PowerPC

We know some of our readers enjoy our more technical posts and like to lurk into the kitchen to see how we deal with all kind of surprises maintaining a complex tool like YaST. Today’s chapter in that regard started with a bug report about the bootloader being installed in the wrong device name (/dev/vda instead of /dev/vdb) in an emulated PowerPC machine in openQA.

After a lot of investigation and with help from our PowerPC expert, we found the culprit, that turned out to be an emulator quirk. The next couple of paragraphs may be interesting or daunting, depending how much virtualization and PowerPC jargon you know. Be warned.

On POWER, the PReP partition containing the bootloader has no unique identifier other than the serial number of the disk on which it created. QEMU virtualization does not provide any disk serial number when the user does not explicitly specify one. This means that the PReP partition in QEMU installation does not have any unique identification and the partition name may change when a disk is added or removed from the virtual machine or the storage configuration otherwise changes. This may lead to system errors related to bootloader installation and updates.

It is recommended to assign a unique serial number to each disk in a QEMU virtual machine on POWER when it is expected the storage configuration of the virtual machine may change. Otherwise, there is nothing YaST or any other tool running within the virtual machine can do to avoid the problems. So this time we only did the investigation part, with the fix coming from the openQA side, which was improved to explicitly set serial numbers when needed.

Storage reimplementation: better support for advanced scenarios

As exposed above, our StorageNG testing image already works in all the supported architectures and in scenarios combining plain partitions, LVM, RAID and encryption in any way. But there are even more situations and technologies currently supported by YaST that we need to incorporate into the new stack.

First of all, we used this sprint to add Multipath I/O support to the new libstorage. Now it can also be combined with all the other mentioned technologies (like having an LVM system on top of an encrypted multipath device), although the storage proposal still needs to be adapted to play nicely with preexisting multipath setups.

As usual with the library stuff, the best “screenshot” we have to offer is one of its geeky autogenerated graphs.

Another scenario that goes beyond the most regular use cases is installing the system into a network storage device, instead of a local hard disk. Now, the new storage system can report whether a root filesystem via a network is used. When that happens, YaST sets the network interfaces with the start mode nfsroot, which is used to avoid interface shutdown and, therefore, the unavailability of the system.

That’s all, folks!

Once again, we omitted the boring parts about bugfixing (with yast2-ntp-client being the star in that regard) and similar stuff. We hope you enjoyed the report and we hope to reach you again in two weeks. Meanwhile…

Have a lot of fun testing the Leap 42.3 pre-releases and reporting bugs!

Posted in Desktop, Distribution, Factory, Packaging, Programming, Software Management, Systems Management, YaST | 1 Comment »

The issues with contributing to projects only once

June 4th, 2017 by bmwiedemann

I work to improve the openSUSE Tumbleweed (GNU/)Linux distribution. Specifically I make sure that all packages can be built twice on different hosts and still produce identical results, which has multiple benefits. This generates a lot of patches in a single week.

OBS
Sometimes it is enough to adjust the .spec file – that is a small text file usually specific to us. Then it is straight-forward

osc bco
cd $PROJECT/$PACKAGE
optional: spec_add_patch $MY.patch $SOME.spec
edit *.spec
osc build
osc vc
osc ci
osc sr

And OBS will even auto-clean the branch when the submit-request is accepted. And it has a ‘tasks’ page to see and track SRs in various stages. For the spec_add_patch to work, you need to do once
ln -s /usr/lib/build/spec_add_patch /usr/local/bin/

When you want to contribute patches upstream, so that other distributions benefit from your improvements as well, then you first need to find out, where they collaborate. A good starting point is the URL field in the .spec file, but a google search for ‘contribute $PROJECT’ often is better.

github
Then there are those many projects hosted on github, where it is also pretty low effort, because I already have the account and it even remains signed in. But some repos on github are only read-only mirrors.

check pull requests, if some have been merged recently
fork the project
git clone git@github.com:…
cd $REPO
edit $FILES
git commit -a
git push
open pull request
maybe have to sign a CLA for the project
When change is accepted, delete fork to not clutter up repository list too much (on github under settings)

sourceforge
The older brother of github. They integrate various ways of contributing. The easiest one is to open a Ticket (Patch or Bug) and attach the .patch you want them to merge with a good description. While many developers do not have the time and energy to debug every bug you file, applying patches is much easier, so gets your issue fixed with a higher chance.

devel Mailinglist
Some projects collaborate mainly through their development MLs, then I need to

subscribe
confirm the subscription
git format-patch origin/master
git send-email –to $FOO-devel@… –from $MYSUBSCRIBEDEMAIL 000*.patch
wait for replies
if it is a high-volume ML, also add an IMAP folder and an entry to .procmailrc
unsubscribe
confirm

project bugtracker
Like https://bugzilla.gnome.org/ https://bugs.python.org/ https://bugs.ruby-lang.org/ https://bz.apache.org/bugzilla/

create unique email addr
sign up for account
add info to my account list
optional: search for existing bug (90% of the time there is none)
file bug
attach patch

So as you can see there is a wide range of ways. And most of them have some initial effort that you would only have once… But then I only contribute once per project, so I always pay that.

Thus, please make it easy for people to contribute one simple fix.

Tags: contributing
Posted in Build Service, Distribution, Packaging, Programming | 1 Comment »

Highlights of YaST development sprint 31

February 20th, 2017 by Yast Team

As we announced in the previous report, our 31th Scrum sprint was slightly shorter than the usual ones. But you would never say so looking to this blog post. We have a lot of things to talk you about!

Teaching the installer self-update to be a better citizen

As you may know, YaST includes a nice feature which allows the installer to update itself in order to solve bugs in the installation media. This mechanism has been included in SUSE Linux Enterprise 12 SP2, although it’s not enabled by default (you need to pass an extra option selfupdate=1 to make it work).

So after getting some feedback, we’re working toward fixing some usability problems. The first of them is that, in some situations, the self-update mechanism is too intrusive.

Consider the following scenario: you’re installing a system behind a firewall which prevents the machine to connect to the outside network. As the SUSE Customer Center will be unreachable, YaST complains about not being able to get the list of repositories for the self-update. And, after that, you get another complain because the fallback repository is not accessible. Two error messages and 2 timeouts.

And the situation could be even worse if you don’t have access to a DNS server (add another error message).

So after some discussion we’ve decided to show such errors only if the user has specified SMT or a custom self-update repository (which is also possible). In any other case, the error is logged and the self-update is skipped completely.

You can find further information in our Self-Update Use Cases and Error Handling document.

During upcoming sprints, we’ll keep working on making the self-update feature great!

Configuring workers during CaaSP installation

While CaaSP release approaches, our team is still working hard to satisfy the new requirements. Thankfully, YaST is a pretty flexible tool and it allows us to change a lot of things.

For CaaSP installation, YaST features a one-dialog installation screen. During this sprint, configuration of the Worker role has been implemented, including validation of the entered URL and writing the value to the installed system. You can check the animated screenshot for details.

The CaaSP worker configuration

New desktop selection screen in openSUSE installer

The world of Linux desktop environments change relatively quick, with new options popping-up and some projects being abandoned. Thanks to the openSUSE’s community of packagers we have a lot of these new desktop environments available on the openSUSE distributions. But the status of those packages for openSUSE is also subject to changes: some desktop environments are poorly maintained while others have a strong and active group of packagers and maintainer behind.

The YaST Team does not have enough overview and time to watch all these desktop environment and evaluate which one is ready or not for being in the installer’s desktop selection screen. So the openSUSE Release Team decided to replace this dialog with something a bit more generic but still useful for newcomers.

They asked the YaST Team to come up with a new dialog featuring the two openSUSE main desktops (KDE Plasma and GNOME) and allowing the easy selection of other environments without reworking the dialog in the future. The goal of that new dialog was to replace the existing one you can see in the following screenshot.

We decided the new dialog should rely on patterns for several reasons. The main one is that the set of patterns is under the close control of the openSUSE community, which looks more closely than us to the desktop environments and their integration into the distribution. Moreover, each pattern specifies its own icon and description that can be somehow re-used by the installer.

We also took the opportunity to merge this almost empty and outdated dialog with the new one.

Addons are no longer produced for openSUSE, so only the second checkbox made any sense nowadays. Moreover, the functionality of that second checkbox directly influence the available selection of patterns, so it made more sense to merge everything in a single screen that keeping an extra step in the installation just to accommodate a checkbox.

So we sent a proposal for the new dialog to the opensuse-factory mailing list and, after implementing many of the ideas discussed there (like better wording or using a button instead of a checkbox for the online repositories), this is the new dialog that replaces the two ones mentioned above.

Selecting “custom” will take the users to the already existing patterns selection screen. Just in case you don’t remember how that screen looks like, you can check this image.

If these screenshots are not enough to make your mind about the change, you can check the following animation, in which KDE Plasma is initially chosen to be changed at a later point (going back in the workflow) to LXQt.

Desktop selection animation

It will take some time before the changes hit the Tumbleweed installer, since they obviously have a non-trivial impact on the openQA tests, that will need some adaptation.

We would like to thank everybody that contributed to this new feature by providing feedback and suggestions through the mailing list. Once again, the openSUSE community has proved to be simply awesome!

Storage reimplementation: improved handling of logical partitions

Our reimplementation of the storage layer keeps getting improvements here and there. With the base x86 case working, we are now implementing the tricky parts, like the partitions renumbering that takes place when dealing with logical partitions in a MBR style partition table.

With GPT partition tables or with primary partitions in a MBR partition table, the partition gets a number (like sda1) when it’s created and keeps that number for its whole lifetime. But logical partition gets constantly renumbered when other partitions are created and destroyed. We need to know in advance what device name every partition will get in order to configure everything beforehand and only commit the changes to the disk when the user clicks in the “install” (during the installation process) or “commit” (if running the expert partitioner).

Now, libstorage-ng is able to simulate in memory the re-numbering process, so we can calculate all the settings related to partitioning (like the configuration of the bootloader) before really touching the disk.

Making kdump work in CaaSP

When you enable the Kernel Crash Dump (kdump), you probably expect that you will get a kind of core dump, like you do for regular processes. What you might not expect is an automatic reboot. That is a nice bonus. If you are tired of waiting for an actual kernel crash, you can test your kdump setup by triggering a crash yourself. Just run this as root:

echo c > /proc/sysrq-trigger

The way kdump works is by allocating some extra memory at boot time and putting a second kernel+initrd there. When the main kernel realizes it is crashing, it transfers control (by kexec) to the other one, which has only one purpose, to dump the memory of the crashed kernel.

In the upcoming Containers as a Service Platform, kdump was not working because the root filesystem is read-only there and we were not able to create the kdump initrd. We fixed it by creating it just after the RPMs are installed, when the FS is still read-write.

Fixes for Snapper in CaaSP

Kdump was not the only component affected by the fact of having a read-only filesystem in CaaSP. Snapper was also running into problems when trying to execute the rollback and cleanup operations. Now the problems are fixed. While working on that we did enough interesting findings to deserve a separate blog entry. So you can expect a new entry in the Snapper.io blog soon.

Snapshot-related improvements in the expert partitioner

While we work on the new storage system, we are still taking care and improving the current one that will continue to be shipped with SLE12-SP3, SUSE CaaSP and openSUSE Leap 42.3. During this sprint we introduced a couple of usability improvements in the expert partitioner related to Btrfs subvolumes.

First of all, we moved the “Enable Snapshots” checkbox that was pretty much hidden in the “Subvolume Handling” dialog to the place where it really belongs – below the selector of the file-system type.

In addition, we added the warning you can see in the screenshot below for users enabling snapshots in a potentially problematic setup.

Bring back the beta warning on CaaSP

And talking about warnings, we improved the CaaSP installation dialog to show the Alpha/Beta product warning at the beginning. After changing the CaaSP installation to use the all-in-one dialog described in our previous post, this feature was lost as it is part of the initial EULA dialog. Now it is back and the users should now properly see the current product status.

Storage reimplementation: encrypted LVM proposal

Back to our storage layer reimplementation, the new system is now able to propose a setup with encrypted LVM. Since some sprints ago, we were already able to propose a partition-based and a LVM setup. That means the new proposal is now feature-pair with the old one, with the only exceptions of Btrfs sub-volumes (that shouldn’t be a big challenge) and s/390 storage (that is still not properly managed by the underlying libstorage-ng).

The bright part is that the new code is written with re-usability in mind, which means implementing an encrypted partition-based proposal (with no LVM) would be trivial. The new code is 100% covered by automatic unit tests and scores to the top in all the automatic quality checkers we have run (like Rubycritics, CodeClimate, and Rubocop).

So now we are prepared for whatever the future brings, having lost no single feature in the way.

Storage reimplementation: prototype if the UI for the proposal settings

The next challenge is to make the power of that new storage proposal available to the users through the user interface. In the previous post we presented the document describing the general ideas we wanted to discuss with UX experts.

It’s our pleasure to announce we met the experts and we very easily reached an agreement on how the user interaction should be. During this sprint we already implemented a prototype of the future proposal settings wizard that is, as usual, included in our testing ISO.

Now that we have solid foundations, it’s very likely that the following sprint will result in the fully working wizard, with almost-final wording and design.

Support for CaaSP added to the AutoYaST integration tests

While fixing a problem with AutoYaST and CaaSP, we decided to extend our AutoYaST integration testing tool to support CaaSP. But, as a side effect, we also made some additional improvements that were long overdue (like improving the procedure to download ISOs, reducing timeouts, etc.).

Now, our internal Jenkins instance takes care of running AutoYaST integration tests for the development version of CaaSP (as it already does for SLE 12 SP2).

Services manager moved to first auto-installation stage

AutoYaST allows to define a set of services to be enabled/disabled in the installed system, applying this configuration during the 2nd stage (after the first reboot).

Unfortunately, this approach won’t work for CaaSP because, in that case, the 2nd stage won’t be available. For that reason, during this sprint, we have adapted AutoYaST to write services configuration during 1st stage.

As usual, not only CaaSP, but other future (open)SUSE versions will benefit of this change.

Faster YaST installation when the release notes cannot be downloaded

Sometimes a very small simple change in a program makes a very noticeable impact in its everyday use. That’s the case of a fix implemented during this sprint. YaST usually re-tries to download the distribution release notes several times if the first attempt was unsuccessful. Although that’s correct from a general point of view, there are cases in which retrying makes no sense and only delays the installation. We have now added failed DNS resolution to that list of cases, which should result in a noticeable faster installation in many scenarios.

Moreover, in the case of AutoYaST we now skip downloading the release notes completely. Downloading them don’t make us much sense in the kind of unattended scenarios handled by AutoYaST and skipping this step and all the possible associated problems result in a more robust process.

Better continuous integration for Libyui

In the previous sprint we switched to Docker at Travis so we could build and test our packages in the native openSUSE system instead of the default Ubuntu. This sprint we did the same change also for the Libyui library which implements the UI part of YaST.

Originally we could not build the Libyui packages at Travis as either the required build dependencies were missing or were present at a very old unusable version. With this switch we can easily use the latest packages from openSUSE Tumbleweed and thus enable Travis builds for all Libyui packages.

See you after Hack Week!

As already mentioned, this week is Hack Week 15! So our Scrum process will be on hold for some days. That doesn’t necessarily mean the YaST development will stall. Since there are quite some YaST-related projects in the Hack Week page, you can expect YaST to simply go in unexpected directions. 🙂

And remember Hack Week is not a SUSE internal initiative, we are open to collaboration from anybody within or outside the company. So jump in and have fun with us!

Posted in Desktop, Distribution, Factory, GNOME, Hackweek, KDE, Packaging, Programming, Software Management, Systems Management, XFCE, YaST | 5 Comments »

AMD Catalyst 15.12 for openSUSE – new makerpm-amd-script is available

March 17th, 2016 by Sebastian Siebert

AMD has released the new AMD Catalyst 15.12 (Radeon Crimson Edition). My script replaces the existing packaging script with an updated packaging script. It supports up to Kernel 4.5. (Official support up to Kernel 3.19)

Important note: The driver does not work on openSUSE Tumbleweed. Unfortunately, the version of X-server is too new for the driver.

SHA1 is obsolete by now. The script used SHA256 for integrity of the downloaded files.

New Feature from packaging script:

systemd support

Resolved Issues:

[SWDEV-82980] Ubuntu 15.10 fails when building the .deb packages

Link: AMD Catalyst 15.12 Release Notes

Downloads:

Script: makerpm-amd-15.12.sh
SHA1: makerpm-amd-15.12.sh.sha256

Installation guide (English):
http://en.opensuse.org/SDB:AMD_fglrx#Building_the_rpm_yourself

Bruno Friedmann will build the new RPM packages in the fglrx repository. Stay tune!

If you find any issue with the driver. Don’t hesitate to contact me. I am in contact with AMD and can forward your issue to the right place. Feedback are welcome.

A report of your system is very helpful beside your feedback. You can generate it with the script:
su -c 'sh makerpm-amd-15.12.sh -ur'

Have a lot of fun!

Sebastian
openSUSE member / Official AMD Packaging Script Maintainer for openSUSE
German Blog: openSUSE – proprietären Grafik-Treiber AMD Catalyst 15.12 als RPM installieren

Tags: 11.4, 12.1, 12.2, 12.3, 13.1, 13.2, 42.1, amd, ATI, driver, fglrx, Kernel, Linux, openSUSE, radeon, rpm, Tumbleweed
Posted in Base System, Desktop, GNOME, KDE, Kernel, Packaging, X.org | 3 Comments »

Sugar on openSUSE

February 17th, 2016 by Jigish Gohil

Built openSUSE Leap based Sugar test images on SUSE Studio, get it from here.

If you wish to get involved with the project maintaining packages, fixing/reporting bugs, follow the links on the X11:Sugar build service project page.

Posted in Derivative, Desktop, Distribution, Education, Packaging | Comments Off on Sugar on openSUSE

Proprietary AMD/ATI Catalyst fglrx rpms new release 15.11 (15.300.1025-1)

December 6th, 2015 by Bruno Friedmann

Time to get an update for fglrx: the new release has been build for 13.1, 13.2, Leap 42.1, and tumbleweed 20151201

Tumbleweed beware : broken Xorg

It seems since the release, that a number of report of broken xorg with gdm, ssdm and so… segfaulting

You can still use the previous version hanging on the server. But I doubt it would work better

As soon, with Sebastian Siebert we have a patch for, I will republish a new build

Informations & bugreport Sebastian’s blog or lizards.o.o

The proposed drivers support kernel up to version 4.4

AMD release note available

Sebastien Siebert making script

Sebastian Siebert posts about fglrx

If you have any problems with the driver, don’t be afraid to report to Sebastian (German and English bugreports are gladly accepted).
he will try, as far as I am able to reproduce the bug. Together with the necessary system information, he will go directly to the right place at AMD to have the bug fixed in the next driver release.
Thank you very much, Sebastian.

See below what to do in case of troubles.

Or you can also ping him on irc (freespacer)

Debugging troubles

I recommend in case of trouble the use of his script which can collect the whole informations needed to help you. then you just have to issue a simple commande in console to collect all informations, you can review them, and finally transmit them.
Check the website to get the latest.

su -c 'sh makerpm-amd-15.11.sh -ur'
The system report 'amd-report.txt' was generated.                                                                                                          [ OK ]
Do you want to read the system report 'amd-report.txt' now? yes/no [y/n]: y
Are you sure to upload the above-named system report to sprunge.us? yes/no [y/n]: y

The report was uploaded to sprunge.us.
   The link is:  http://sprunge.us/eMEB

Copy paste the link in the comment zone of Sebastian post

All proudly distributed by openSUSE powered server and sponsored by Ioda-Net Sàrl

Tags: 13.1, 13.2, 42.1, amd, ATI, fglrx, Kernel, Tumbleweed, unified, xorg
Posted in Base System, Desktop, Distribution, GNOME, KDE, Packaging, X.org, XFCE | Comments Off on Proprietary AMD/ATI Catalyst fglrx rpms new release 15.11 (15.300.1025-1)

AMD Catalyst 15.11 for openSUSE – new makerpm-amd-script is available

December 5th, 2015 by Sebastian Siebert

AMD has released the new AMD Catalyst 15.11 (Radeon Crimson Edition). My script replaces the existing packaging script with an updated packaging script. It supports up to Kernel 4.4. (Official support up to Kernel 3.19)

I have adapted the AMD driver to the Kernel 4.4 (rc3). For the moment it works for Kernel 4.4-rc3. Unfortunately the AMD driver has a compatibility issue in combination with the GNOME Desktopmanager and X-Server. As a workaround, I recommend for GNOME another Desktopmanager such as lightdm until the issue is hopefully fixed.