Home Home > 2019 > 04
Sign up | Login

Archive for April, 2019

Highlights of YaST Development Sprint 76

April 30th, 2019 by

While the openSUSE Conference 2019 is approaching, the YaST team is still busy not only fixing small bugs for the upcoming (open)SUSE releases but working on features and changes for later versions.

Although there is much more work behind this sprint, we will have a look at these changes in this report:

  • The first bits of the YaST Network refactoring have been submitted to factory.
  • AutoYaST gets support for specifying NFS shares in a sane way.
  • The Partitioner’s tree behaviour has been improved.
  • Libyui has been updated to support Qt 5.13 (do not miss the screenshots in this report).
  • The installer does not offer enabling autologin for text-only system roles.

But that’s not all. We have some pretty exciting news regarding the LibYUI Testing Framework which, hopefully, will reduce the hassle of maintaining YaST openQA tests in the future. If you are interested, we published a separate post a few days ago with all the details about this topic.

Submitting First Bits of the YaST Network Refactoring to Factory

yast2-network 4.2.2 is the first version which includes some code from the ongoing refactoring effort. As we announced in our last report, we have just started and there is a long road ahead of us, but you can see some results already in the routes management area.

Additionally to cleaning up and improving the codebase quality, we are fixing some bugs and introducing some minor enhancements along the process. For instance, version 4.2.2 permits to have multiple default routes and does not drop extra options for them.

New network routing dialog

If you want to know more details about the process, you might be interested in the kickstart meeting notes that we added to our repository a few days ago.

Auto-installation onto NFS Shares: the Sane Way

During the previous sprint, AutoYaST was powered again with the capability of installing onto an NFS share. That feature was available in SUSE Linux Enterprise 12 (and openSUSE 42.x) but, for several reasons, we did overlook it when the new storage stack was re-implemented for (open)SUSE 15.

The problem with this feature in older versions, apart of not being documented at all, is that it requires to use some hacks and a non-validating AutoYaST profile. For that reason, we have redesigned how NFS drives are described in the AutoYaST profile. With the new format, we have a drive section per each NFS share, for example,

<partitioning config:type="list">
  <drive>
    <device>192.168.1.1:/exports/root_fs</device>
    <type config:type="symbol">CT_NFS</type>
    <use>all</use>
    <partitions config:type="list">
      <partition>
        <mount>/</mount>
        <fstopt>nolock</fstopt>
      </partition>
    </partitions>
  </drive>
</partitioning>

Although the old format is still supported to keep backward compatibility, we encourage you to use the new one for now on. And of course, this feature will be appropriately documented in the official (open)SUSE 15 documentation.

No Longer Offering Autologin for Text-Only System Roles

Autologin is a feature that most modern display managers (KDM, GDM, SDDM, LightDM) offer as a convenience function for home users. Users reported that YaST also provided this for the “Server (text only)” system role where that does not make sense, so we fixed that.

Yet Another Qt Version

This is not actually a big deal; there is another version of the Qt libs which we use for the graphical version of YaST. Now it’s Qt 5.13, and we needed to adapt some functions that have become obsolete in the meantime. Fortunately, Qt always provides drop-in replacements for things they obsolete, so this was mostly a pretty mechanical task to go through the warnings about functions or classes that are now obsolete and replace them with the new counterpart.

But how long have we been doing that? Quite a while, actually; it all started with Qt 2.x in late 1999 for SuSE Linux 6.3. That was the first YaST2. It already came in both a Qt version with a graphical user interface and in a text-only (NCurses) version.

SuSE Linux 6.3 Installer

Back then we were more advanced than the KDE of that time: KDE 1.x was still using Qt 1.x, and the first version of YaST2 already used Qt 2.x. We kept upgrading throughout all the Qt 2.x versions and in SuSE Linux 9.0 we jumped to Qt3, which looked like this.

SuSE Linux Professional 9.0

And as you know, that’s not the end of the story. We have kept updating through all the Qt 3.x, Qt 4.x and now Qt 5.x versions, as you can see in the following screenshot of the upcoming openSUSE Leap 15.1. It’s been a while: almost 20 years of happy Qt usage, always up-to-date.

openSUSE Leap 15.1

Improving the Partitioner UX

When it comes to UX, the tree of our new and shiny partitioner needed some love. We fixed an annoying issue which caused all the branches to be expanded after every change on any device. Besides that, we also changed the initial view. See the screenshot below in which sections -levels with icons like “Hard Disks” or “RAID”- are initially expanded and items representing devices are collapsed, not displaying, for example, the full list of partitions until the user decides to expand those.

YaST Partitioner Screenshot

However, the bad news is that these changes will not be available in (open)SUSE 15.1, although you can give it a try as soon as they land in Tumbleweed.

Let’s Talk about Multi-device Btrfs

The improved behavior of the navigation tree is not the only enhancement in the Partitioner we are preparing for the future (where “future” means Tumbleweed and the 15.2 releases of both SLE and openSUSE Leap). We also want to make possible to use the YaST Partitioner to define Btrfs file-systems that spread over several disks and partitions. That’s a pretty unique feature of Btrfs that combines at file-system level certain characteristics usually offered by software RAID and LVM.

But that also implies a quite different way of organizing the storage devices, which is a challenge to represent in the already complex user interface of the Expert Partitioner. So, first of all, we have put together this document describing all the struggles of representing advanced Btrfs features in the Partitioner, together with some possible solutions for the short and mid terms.

Please, feel free to provide feedback about the proposed solutions and to offer new ones. Help us to shape the future of the YaST interface!

Conclusions

As we mentioned at the beginning of this report, we owe you a blog post about the Libyui Testing Framework status. But as you wait for it, you might want to register for the openSUSE Conference or check the schedule.

Announcing LibYUI Testing Framework

April 26th, 2019 by

The LibYUI framework brings enormous advantage when you need an application which can be used with X and ncurses. But while for Qt, integration testing is a solved problem, meaning there are multiple tools available, the situation is not that rose-colored in case of ncurses. On top, using different tooling would require additional efforts for the framework development. So everything started with openQA, which is using screen comparison with advanced fuzzy logic. This approach kept the costs of the tests development low, but maintenance is still painful when UI changes.

Considering all of the above, Ladislav Slezák and Rodion Iafarov has been working on a proper solution taking Ladislav’s proposal for Hack Week 15 as starting point. The idea is plain simple: provide an HTTP/JSON API which allows interacting with LibYUI by reading and setting different UI properties. Consequently, if some button was moved, resized or got new shortcut key assigned, we do not need to adapt the test code anymore.

Obviously, this functionality is not required in a production system, so we have created separate packages for this purpose. Here you can find GitHub repository containing some documentation.

Demo time!

As cool as it sounds, it is even better when you see it in action. In the screencast below you can see how the user can interact with YaST Host module using cURL from the command line.

LibYUI Testing Framework in Action

How to Test the New Packages

If you want to give it a try, you can use the packages we have prepared for you. However, it is recommended to use a virtual machine to not pollute (or accidentally break) your system. You have been warned. 🙂

First of all, you need to get new packages, which are currently available in YaST:Head project in OBS. Navigate to Repositories tab and get the repository which matches the distribution you are running. Afterwards you should be able to install libyui-rest-api package. Also, you need to update related packages and YaST2 modules were recompiled using latest LibYUI. To do that, simply run the following command, where YaST:Head is the repository added in the previous step:

zypper up --allow-vendor-change -r YaST:Head

With the required software in place, you need to start a YaST module setting the environment variable Y2TEST to 1:

xdg-su -c 'Y2TEST=1 YUI_HTTP_PORT=9999 yast2 host' # for Qt
sudo Y2TEST=1 YUI_HTTP_PORT=9999 yast2 host # for ncurses

To allow remote connections you can add YUI_HTTP_REMOTE=1. For security reasons, only connections from localhost are allowed by default.

Now you should be able to see something like the screenshot below by navigating with your browser to http://localhost:9999.

LibYUI Testing Framework Browser

Further steps

As you can see, we have not implemented a wrapper for the POST and GET requests to the HTTP server yet, although it will be the next step. We are trying to to find a solution which allows us running tests locally from the terminal, in CI using stable distribution versions, and finally in openQA for development builds.

Other things which are coming:

Security Notice

YaST is usually running with the root permissions, that means everybody who can connect to the HTTP/JSON API can read the values and send button clicks. And because there is no authentication yet, this is a big security problem. In a nutshell: do not use the API in production systems! It has been designed only for testing in a secure environment.

Closing Thoughts

We are pretty excited about this project for several reasons. On the one hand, because we expect it to reduce the maintenance burden of our integration tests. On the other hand, because it is a nice example of Hack Week benefits and cross-team collaboration.

If you are interested, we will try to keep you in the loop by posting information regularly as part of YaST sprints reports.

Highlights of YaST Development Sprint 75

April 10th, 2019 by

With the upcoming releases of openSUSE Leap 15.1 and SLE-15-SP1 approaching, the YaST Team at SUSE is investing a quite significant time in polishing details and fixing small (and not so small) bugs. But fortunately, that still leaves us enough time to also work in our mid term goals.

So welcome to our usual selection of selected bug-fixes (listing them all would be boring) and exciting new stuff. This edition includes:

  • A nice howto for reporting Snapper bugs
  • Tons of fixes for right-to-left languages like Arabic
  • Some adjustments and improvements in the storage area
  • A sneak peak into the future of the yast2-network code
  • Some contributor-oriented content: like our new pull request templates and revamped Docker images for testing

Snapper Bug Reporting Howto

During this sprint we fixed a bug that was causing Snapper to crash under very specific circumstances. The scenario was quite unusual so we had to request quite some information from the reporter of the bug to confirm what was happening. As a nice consequence, in addition to having now a more robust Snapper (one bug killed) you can also enjoy a new page in the openSUSE wiki listing the information you should attach to Bugzilla if you find a bug in Snapper while using (open)SUSE.

Which is also a nice excuse to remind you about the equivalent "Report a YaST bug" page.

YaST around the globe… in all directions

Many of the YaST users and of our blog readers are not native English speakers that surely appreciate the fact that YaST and (open)SUSE in general can be used in several languages. But have you ever thought about the implications of developing a multi-language software? Sure? In all of them? 😉

Human languages are so diverse as the human cultures and there are many details to take into account, from the usage of different alphabets to the various ways of dealing with genre or number (in English the words have just one form for singular and another for plural, but that can be way more complex in other languages). In today’s issue we will take a look to one of our favorite translation issues – languages that are written from right to left, like Arabic.

The installer summary in Arabic

Dealing with text that is a mixture of Latin and Arabic script is complex and sometimes we have to deal with interesting bugs. Fortunately we have our own weapon to fight those bugs. If in Star Wars they have protocol droids like C3PO, in the YaST team we have Martin Vidner, which is the closer human equivalent.

He fixed all the reported bugs and even created a tool to help debugging similar problems in the future. You can find the source code of that tool in Github. There is even a hosted instance of the tool to be used by translators or anyone who is curious.

Now, even complex interfaces like our Partitioner look correct enough in right-to-left languages, so we will not have to send mirrors to all our Arabic users.

The YaST Partitioner in Arabic

If you want to know more about this exciting but very complex problem of bidirectional texts, you can start with the following documents.

  • Martin’s great summary of the types problems found in YaST and their respective solutions.
  • Wikipedia: Bi-directional text, an overview of the concepts
  • Unicode Standard Annex #9: Unicode Bidirectional Algorithm, the gory details, 50 pages of them
  • More Arabic YaST

    On related news, we got also some reports about some problems visualizing bullet-points in Korean with the beta versions of the future SLE-15-SP1. But as we could verify, all those problems are gone now.

    SLES installer in Korean

    Storage Fixes

    Other area that has received some attention in this sprint is the storage management. Three related features needed adjustments before the upcoming (open)SUSE releases:

    • Fixed the detection of the boot disk in the Partitioner warnings.
    • The Guided Setup now works better when doing several attempts in different disks.
    • AutoYaST can now install over NFS.

    One of the last storage features that the YaST Team has developed is the support for Bcache devices in the Expert Partitioner. While our QA team was testing it, they found a bug. The Partitioner was complaining because the boot disk did not contain a partition table, which is a mandatory condition for a Legacy (non-UEFI) x86 system. But it was a bogus warning, since they had actually defined a /boot partition in another disk.

    That’s how we found that our Partitioner gets confused if there is a separate partition mounted at /boot and located in a different disk than the root file-system. The Partitioner insisted in considering the disk containing / to be the one that would be using for booting, instead of checking the structure of the disk containing /boot. Now that is fixed and the improvement will be available for the upcoming SLE 15 SP1, Leap 15.1 and, of course, openSUSE Tumbleweed.

    But that was not the only storage bug fixed just in time for the upcoming releases. Some sprints ago, the Storage Proposal algorithm for the initial proposal was modified to try installing on each of the individual disks. If the installation was not possible over a given disk, even after disabling all optional configurations (e.g., snapshots and separate /home), a new proposal is tried over the next disk and so on. The problem was that the disabled options in the previous attempt were not restored back when switching to the next disk. This caused some ugly side effects, for example, if the swap partition was disabled when trying over the first disk, then the proposal did not try to create a swap when it was performing the proposal over the next disk. But now this is also fixed and it will work as expected.

    And last but not least, AutoYaST now supports to install over a Network File System (NFS). This feature was left back when the new YaST storage stack was re-implemented for SLE 15 GA. Actually, this is a non documented feature. That’s why we overlooked that SUSE 12 was able to do it using some hacks and a non-validating AutoYaST profile. But no worries, the feature is available again and such profile will work now in any updated SLE-15 or Leap 15.0. Of course, it will also work while installing SLE-15-SP1 or openSUSE Leap 15.1 and Tumbleweed.

    Nevertheless, we are working on a better and documented way of supporting that scenario in the future, with no need to twist the specification of the AutoYaST profile. Stay tuned for more information.

    Rethinking the Location of Special Boot Partitions

    And now that the storage layer looks sane and healthy for the upcoming releases, we also took some time to think about future improvements. As you know, the storage Guided Setup always proposes to create special boot partitions as needed on each case. That can be a BIOS BOOT (for Legacy x86 systems with GPT), an ESP (for UEFI systems), PReP (for PPC systems) or zipl (for S/390 systems). Strictly speaking those partitions doesn’t have to always be in the same disk than the root partition and in some cases having it on a separate one can have some advantages (like sharing the ESP partition with another operating systems).

    But we have been reconsidering all the cases, the expectations of most users and of the majority of BIOS vendors and the known bugs in other operating systems about sharing boot partitions. We have decided to be more strict in the future about the location of those partitions. Starting today with openSUSE Tumbleweed and in the 15.2 releases of openSUSE Leap and SLE, the Guided Setup will always propose those partitions in the system disk. That is, in the disk containing /boot and the root filesystem.

    The future of YaST Network is here

    Those who follow this blog know that we invested quite some time on the last couple of years rewriting the part of YaST that was more buggy and harder to modify – the storage stack. And surely you have already noticed that since we did it we are introducing new features at a very good pace (like bcache, more powerful Partitioner, Raspberry Pi support, etc.) and fixing the reported bugs in a matter of days or even hours.

    The next in our list of YaST areas to revamp is the networking support. And we are happy to announce that we are starting to have some visible results in that. There is still a very long road ahead and we will provide more information in upcoming reports. But at least we have already a preview of a fully rewritten management of network routes. It’s still not available in openSUSE Tumbleweed. But for those who can’t wait, here you can see the first screenshot. All based in new and clean code backed by automated tests.

    New network routing dialog

    Activating Online Repositories in openSUSE Leap 15.1

    The openSUSE Tumbleweed installer asks at the beginning of the installation whether to activate and use the online repositories when a network connection is available.

    The reason is that the installation DVD does not contain all available packages because of the limited media size. Another advantage is that the installer might directly install newer packages than on the media, this avoids installing the older versions first and then upgrading them to the latest version.

    However, in some case you might not want to use the online repositories, for example if the network connection is slow or is paid.

    We got a bug report that this question was missing in the Leap 15.1. It turned out that the control.xml file which drives the installer did not contain this step. After adding few lines into the file you can now enjoy the online repositories also in Leap 15.1!

    Online repositories in Leap 15.1

    Why are we writing about this? The reason for the missing step in the Leap 15.1 was a bit surprising. Normally all YaST packages are developed in the Git master branch for both Tumbleweed and Leap. However, in this case the Leap 15.1 has been already branched and was developed separately, the changes in the master went only to the Tumbleweed. And we overlooked that small difference when adding this step.

    To avoid this in the future we added a pull request template with a reminder which informs the developers about this difference in the Git setup when opening a pull request.

    If your project also has some unusual setup then the pull request template might a good reminder for you as well.

    Building the Docker Images in OBS

    But the reminders about the correct branches and procedures is not the only news we have for YaST contributors and main developers. As you may remember, few years ago we switched to using Docker at Travis. That works well but we found some disadvantages of that initial setup.

    • You need extra account at the Docker Hub to manage the images.
    • There is no link between OBS and the Docker Hub, we cannot easily trigger image rebuild when a package is updated in OBS.
    • We only blindly triggered the rebuild every 2 hours (in some cases the rebuild is not necessary, in some cases it took too much time).
    • The Docker Hub can use the new OBS packages only after they are published by OBS.
    • The build at the Docker Hub is quite slow (~20 minutes in our case), if an image is currently being built the build is added into the queue and it will start after the previous builds are finished.

    The result is that a new package can be available in Travis several hours after merging the pull request. And even after triggering the build manually it still might take more than one hour.

    We needed a faster cycle and the solution, as usually happens, was in the openSUSE ecosystem. As you may know, the Open Build Service is able of much more than just building packages. So we decided to make use of the OBS capacity of building Docker images.

    Building both our packages and our Docker images in OBS comes with many advantages:

    • The image build is started immediately when the new packages are built, it does not wait for publishing the packages and does not wait for full rebuild (only for the needed packages).
    • No extra accounts/permissions (just use your OBS account).
    • The build in OBS is faster (6-7 minutes).
    • No need for extra Jenkins jobs periodically triggering the image rebuilds.

    This means the new packages should be available in the Docker image in about 10-15 minutes after merging a pull request (for leaf packages, changing a core package which triggers a complete YaST rebuild will of course take more time).

    If you want to learn more about this topic, take a look to the following links:

    And that was not all!

    As usual, the content of this report is just a small subset of all the work the YaST Team does in two weeks. In this sprint, most of that work went to fixing all kind of bugs in preparation for the next releases. Big bugs, small ones, hidden bugs and embarrassingly obvious ones. Hopefully, you got a fix for your reported bug. If not, you can always stay tuned for more news after the next sprint. And don’t forget to have a lot of fun!

experimental openSUSE mirror via IPFS

April 3rd, 2019 by

The InterPlanetary File System (IPFS) can be used to provide files in a more efficient and distributed way than HTTP.

Our filesystem repo already has the go-ipfs client.

You use it with
ipfs daemon --init

And then you can add my Tumbleweed mirror with
zypper ar http://127.0.0.1:8080/ipns/opensuse.zq1.de./tumbleweed/repo/oss/ ipfs-oss

You can also browse the content online at
http://opensuse.zq1.de./tumbleweed/repo/oss/ . During my testing I found that the results are sometimes inappropriately cached on the Cloudflare CDN, so if you used it under this URL without the ipfs client, this might throw signature errors in zypper.

On the server side, the mirror is updated using the syncopensuse script from
https://github.com/bmwiedemann/opensusearchive and consistency of the repo is verified with checkrepo

When a complete repo was synced, dynaname updates a DNS entry to point to the new head:

> host -t txt _dnslink.opensuse.zq1.de.
_dnslink.opensuse.zq1.de is an alias for tumbleweedipfs.d.zq1.de.
tumbleweedipfs.d.zq1.de descriptive text “Last update: 2019-04-03 12:23:43 UTC”
tumbleweedipfs.d.zq1.de descriptive text “dnslink=/ipfs/QmSXEVuU5z23rDxMyFYDhSAUaGRUPswuSXD3aVsBEzucjE”

If you got spare bandwidth and 300 GB disk on some public server, you could also host a mirror of today’s version, simply by doing ipfs pin add QmSXEVuU5z23rDxMyFYDhSAUaGRUPswuSXD3aVsBEzucjE

This is a permalink: http://127.0.0.1:8080/ipfs/QmSXEVuU5z23rDxMyFYDhSAUaGRUPswuSXD3aVsBEzucjE also browsable via any public IPFS gateway. This means, it will always remain on the 20190401 version of Tumbleweed and no changes in content are possible – similar to how a git commit ID always refers to the same data.

So why did I create this IPFS mirror? That is related to my work on reproducible builds for openSUSE. There it regularly happened that published Tumbleweed binaries were built with libraries, compilers and toolchains that were no longer available in current Tumbleweed. This prevented me from verifying that the published binaries were indeed built correctly without manipulation on the OBS build workers.

Now, with this archive of rpms easily available, it was possible to verify many more Tumbleweed packages than before. And most importantly, it remains possible to independently verify even after Tumbleweed moves on to newer versions. This data is going to stay available as long as anyone pins it on a reachable server. I’m going to pin it as long as it remains relevant to me, so probably a bit until after the next full Tumbleweed rebuild – maybe 6 to 12 months.

Thus, it now is even less easy to sneak in binary backdoors during our package build process.