Home Home > Tag > storage
Sign up | Login

Posts Tagged ‘storage’

Highlights of YaST Development Sprints 88 and 89

November 22nd, 2019 by

A More User Friendly Role Selector Dialog

Step by step, we continue improving the user experience making use of the newly added widgets to libyui. This sprint was the turn to update the role selection dialog to use the new item selector introduced during the sprint 87. Apart from looking better as it can be seen in the screenshots below, there are two immediate improvements:

  • the vertical scroll, when present, is respected after selecting a role (instead of “jumping to the beginning”), and
  • the selected role (if any) will be visible when arriving at the dialog even when the list is quite long or the available space too short.
Before After

What is more, updating the dialog was also useful for us to realize about some needed improvements for the widget itself, mentioned in the next section. Quite a productive change 🙂

When one Bit is not Enough: The CustomStatusItemSelector

A few weeks ago, we wrote about the new ItemSelector widget that is finding its way into YaST user interfaces. It turned out that just a simple on/off status is not enough in some cases, so we had to extend that concept. For example, software modules may have dependencies, and we want to show the difference between one that was explicitly selected by the user and one that was auto-selected because some other software module requires it.

This kind of shook the foundations of the underlying classes; all of a sudden a bit is no longer just a bit, but it needs to be broken down into even smaller pieces. Well, we cheated; we now use integer values instead. Most of the class hierarchy still only uses 0 and 1, but the new YCustomStatusItemSelector also supports using higher numbers for application-defined purposes.

For each possible status value, the application defines the name of the icon to be displayed (for graphical UIs like the Qt UI), the text equivalent (for text mode / the NCurses UI), and an optional nextStatus which tells the widget what status to cycle to when the user changes the status of an item with a mouse click or with the keyboard. A value of -1 lets the application handle this.

So this is not a one-trick-pony that is useful only for that one use case (the software modules), but a generic tool that might find good uses in other places all over YaST as well.

Usage examples: C++, Ruby.

Snapper and Machine-readable Output

Most likely you already know snapper, a great tool to work with your filesystem snapshots. Some third-party scripts and tools (e.g., YaST) use the snapper CLI to get some information, but generally, snapper generates an output intended to be human-readable. Sometimes that could cause some troubles in scripts checking the snapper output. Now on, snapper also offers CLI options to generate its output in a machine-readable format, i.e., CSV and JSON. Please, check this post for more information about those new options.

Fix Boot Problems with Remote Encrypted Devices

Since we adopted systemd, the management during system boot of encrypted devices on top of network-based devices like iSCSI or FCoE disks has been less than optimal. But now we are happy to announce that we have put all the pieces together to make the experience as smooth as possible.

One of the main responsibilities of systemd is sorting the actions performed during boot and setting the dependencies between them. For example, if there are encrypted devices, systemd will first ask you for the password and activate the devices to afterwards mount the file system contained in those encrypted devices. Systemd should be able to distinguish when an encrypted device is based on a network-based storage device and, thus, can only be initialized after the network is up. In some cases that detection failed (for example network block device based mounts, such as iSCSI and FCoE disks) and systemd got stuck before initializing the network waiting for the device to be available.

Recently, SLE and openSUSE Leap has incorporated support for specifying the _netdev option in the /etc/crypttab file. With such option, systemd will recognize the encrypted device as network-based and will only try to activate it after setting up the network. That’s analogous to the corresponding _netdev option in /etc/fstab that has been already there for quite some time and that can be used to defer when a device is mounted. For it to work properly, the _netdev option must be present in all the relevant entries of both crypttab and fstab.

And that’s exactly what YaST will do now in openSUSE Tumbleweed and upcoming releases of both SLE and openSUSE Leap. From now on, the _netdev option will be added fstab for all mount points depending (directly or indirectly) on the network. In addition, that option (and also the noauto and nofail ones) will be propagated from fstab to all the corresponding crypttab entries.

This should mark the end of a dark age of encrypted iSCSI and FCoE devices timing out during boot.

AutoYaST Support for Random and Pervasive Encryption

Back in October, we announced that YaST got support for new encryption methods like random or pervasive encryption. At that time, AutoYaST was out of scope because we wanted to have a stable (and tested) API first. Fortunately, the time has come and now AutoYaST supports these encryption mechanisms.

Starting in autoyast2 4.2.17, you can specify the encryption method using a crypt_method element, as shown in the example below. Possible values are luks1 (regular LUKS1 encryption), pervasive_luks2 (pervasive volume encryption), protected_swap (encryption with volatile protected key), secure_swap (encryption with volatile secure key) and random_swap (encryption with volatile random key).

    <drive>
     <type config:type="symbol">CT_DISK</type>
     <use>all</use>
     <partitions config:type="list">
      <partition>
       <size>20G</size>
       <mount>/</mount>
       <filesystem config:type="symbol">ext4</filesystem>
       <crypt_method config:type="symbol">luks1</crypt_method> <!-- default method if crypt_key is defined -->
       <crypt_key>S3CR3T</crypt_key>
      </partition>
      <partition>
       <size>1G</size>
       <mount>swap</mount>
       <crypt_method config:type="symbol">random_swap</crypt_method> <!-- set encryption method -->
      </partition>
     </partitions>
    </drive>

As we want AutoYaST to be as user-friendly as possible, it will try to help you if you do some mistake setting the encryption configuration as when in the screenshot below.

Finally, the old crypt_fs element is deprecated, although it stills works for backward-compatibility reasons. Basically, it is equivalent to setting crypt_method to luks1.

Improve Support for AutoYaST Guided Partitioning

When it comes to partitioning, we can categorize AutoYaST use cases into three different levels:

  • Automatic partitioning: the user does not care about the partitioning and trusts in AutoYaST to do the right thing.
  • Guided partitioning: the user would like to set some basic settings (use LVM, set an encryption password, etc.)
  • Expert partitioning: the user specifies how the layout should look, although a complete definition is not required.

To some extent, it is like using the regular installer where you can skip the partitioning screen and trust in YaST, use the Guided Proposal, or define the partitioning layout through the Expert Partitioner.

The second level (Guided partitioning) was introduced in AutoYaST with the release of SUSE Linux Enteprise 15 (and Leap 15.0) but it was not documented at all. Additionally, although it was working as designed at first sight, it was far from being that useful.

This sprint with invested quite some time improving the documentation (kudos to our awesome documentation team) and the behaviour. Now, if you want to set up an LVM system without having the specify all the details, you can use the following snippet in your profile:

<general>
  <storage>
    <lvm config:type="boolean">true</lvm>
  </storage>
</general>

If you are interested in the available options, you can check the documentation draft.

Highlights of YaST Development Sprint 85

September 26th, 2019 by

Refactored Network on its way to Tumbleweed

A few weeks ago we submitted the first round of changes to the network module to Tumbleweed. At that point, it was still using the old data model for most operations (except routing and DNS handling) and a lot of work remained to be done.

We have been working hard on improving the overall quality of this module and we will submit an updated (and much improved) version in the upcoming days. To summarize, here are some highlights:

  • Completed the new data model (support for TUN/TAP, bridges, bonding, VLANs, etc.).
  • New wireless configuration workflow.
  • Revamped support for interface renaming and driver assignment, including better udev rule handling.
  • Fixed /etc/hosts handling when switching from static to DHCP based configuration.
  • Many small fixes in several areas.

Are we done with the refactoring? No, we are still working on improving S390 support and fixing small issues, but most of the work is already done.

Of course, as soon as we finish, we will publish a blog entry with the gory details. But, as we know that you love screenshots, let us show you a handful of them.

Although we have not introduced big user interface changes, we have tried to improve small things, like properly displaying whether an interface belongs to a VLAN or hiding the "Hardware" tab for virtual interfaces.

DNS resolution not working during installation, or: openQA is different

When we got a bug report that DNS resolution was not working during installation (since SLE-15 apparently), a solution seemed straightforward: /run/netconfig/resolv.conf was missing because the /run directory was not mirrored (bind-mounted) into the target system. That’s a task that used to be done by yast-storage in times before SLE-15 and was for some unknown reason forgotten when we implemented yast-storage-ng. A one-line fix was easily done, tested, and submitted.

Or so it seemed.

A few days later we got reports from SLE openQA that tests started to fail with this patch. Nothing networking related but the installation did not finish because the 10 seconds countdown dialog (‘going to reboot’) at the very end of the installation was frozen. The UI didn’t accept any input whatsoever. But whatever we tried, the issue was not reproducible outside openQA. YaST logs from openQA showed that /run got mounted as planned and was cleanly unmounted at the end of the installation – before that frozen dialog. So no clue so far and the issue was set aside for a while. Until the same reports came in from Tumbleweed testing. It was clearly linked to this one-line patch. But how?

It stayed a mystery until a chat with an openQA expert shed some light on the issue. What we thought was happening was: openQA stopped the dialog (by pressing a button) and when it tried to go on the OK button did not respond anymore. What we learned does actually happen is this: openQA stops the dialog, then switches from X to the text console, collects logs, switches back to X, and then the UI does not respond anymore. So that was quite an essential point missing.

And with this it was easily reproduced outside openQA: the X logs showed that the X server lost all its input devices after the switch. And that was because someone had deleted the whole /run directory. The YaST logs didn’t contain a hint (of course not, would have been too easy) but grepping the sources found the place where YaST deleted the directory.

The code had been added after complaints that the installation left a cluttered /run directory – of course the installation did leave files there, since it was forgotten to bind-mount the directory. So once the mentioned patch bind-mounted it again the deletion code cleaned up not /run in the installation target system but the real /run as well – cutting off the X server from the outside world resulting in freezing the openQA test.

And the moral of the story is: probably none. But it highlights again that the automated test setup can have unexpected feedback on the test itself. Luckily in this case, as the issue would not have been noticed otherwise.

Computer, Enlarge the Device Graph

The partitioner module has a graphical view to help you see the relations in more complex storage setups:

If you turn the mouse scroll wheel up or down, we will zoom the view in or out. Now the direction matches the behavior in web browsers and online maps, previously we had it the wrong way around.

Update: during proofreading, a team mate told me: "A device graph? That’s not a device graph. THAT’s a device graph:"

YaST Squad Sprint 62

September 12th, 2018 by
  • Jenkins commenting in GitHub pull requests
  • Intel Rapid Start Technology for better sleeping
  • Consistent storage proposal in SLE-12
  • Partitioner: designing the UI for its full potential
  • Partitioner: entire disks as members of a software MD RAID
  • Partitioner: better explanation of unusual conditions
  • A sample of bug fixes

Improved Jenkins Integration

It happened quite often that our Jenkins job failed for some reason after merging a pull request at GitHub. And because the Jenkins is supposed to submit the changes to the build service it happened that the fixes from Git were not released in the RPM packages if nobody noticed the failure. That was a bit confusing because we closed a bug at Bugzilla but the fix was not available anywhere.

To avoid this we have added a wrapper script which runs the original Jenkins command (rake osc:sr in this case) and writes the result as a comment to the respective pull request at GitHub. If a submit request is created it additionally adds a link to it.

Since now you should keep the pull request page open after merging it and wait for the Jenkins status result. If it fails for some reason then try fixing it or ask for help on IRC or the YaST mailing list.

Note: This automation works only for the code branches which are in active development and for packages which have an Jenkins job assigned (most of the YaST packages have).

Intel Rapid Start Technology Support

The Intel Rapid Start Technology allows to use a fast disk (SSD) for suspend-to-RAM to save energy. The idea is that after a given time the contents of RAM will be moved to SSD so that the system can power itself off. When powered on, RAM will be read back. So it’s something like a dynamic changing of suspend-to-RAM to suspend-to-fast-disk.

What does this technology need from the installer? It needs its own dedicated partition where it can store the contents of RAM. To support this technology we added in this sprint the ability to create and recognize such a partition. It looks like this:

Consistent Storage Proposal (SLE-12-SP4 / yast2-storage-old)

We fixed a bug where the behavior was inconsistent if you switched the storage proposal between partition-based and LVM-based / encrypted LVM-based: bsc#1085169.

The behavior was pretty irritating: Initially, it would propose to create a separate “/home” partition, but when you changed the proposal parameters and simply kept that checkbox “[x] separate /home” checked, it would complain that with the current settings a separate “/home” was not possible.

The two code paths did the calculations differently: One accounted for the other partitions that were also proposed like “swap” and “boot” (or “PrEP” or “/efi-boot”), the other did not. We unified that as much as reasonably possible without breaking things, but since calculating when and how to use any boot partitions is quite complex in that old legacy storage stack, we did not go all the way; boot partitions are pretty small, so their size matters only in very pathological fringe cases. We try not to overengineer things, in particular not with the 4th service pack for a business product.

More details in the pull request with the fix.

The Partitioner looks to the future

We have blogged a lot about Storage-NG and the possibilities and features it will bring to the users. But a significant part of its power remain dormant under the surface because we decided to clone the user interface and the functionality of the classical YaST Partitioner for the deadline marked by SLE 15 (and Leap 15.0). But now we are finally able to start exposing those long-awaited features and to bring new ones for the current Tumbleweed and for the future SLE-15-SP1 and Leap 15.1.

The user interface of the Partitioner is already rather packed with functionality, but we want to avoid a too disruptive redesign. So it was time to some pen and paper sessions, trying to find and draw a good way to add exciting new stuff to the Partitioner, including all of the following:

  • Allow entire disks (no partition table) to be added as members of a software MD RAID.
  • Manage (create, modify and delete) partitions within a software MD RAID device.
  • Make possible to format an entire disk (no partition table) and/or define a mount point for it (just as we do with partitions or LVM logical volumes).
  • Manage Bcache so the user can set and configure which devices will be used to speed up others.

As usual, we consulted some UI experts in the process and the result is this first version of a document, which summarizes how to incorporate all that to the Partitioner, including some alternatives we are considering for the near future.

That document will become the cornerstone of future developments. Sometimes you need to spend a sprint doing other stuff (like researching and documenting) before you can go ahead with writing code.

Partitioner: full disks as members of a software MD RAID

The first of the features described in the previous document is already available for Tumbleweed users (or it will be as soon as the integration process concludes) and, thus, ready for the upcoming releases of SLE and Leap. Now the Partitioner offers full disks as “Available Devices” in the RAID screen, following the criteria and considerations explained at the document.

That brings even more ways of combining devices together (disks, partitions, software RAIDs, you name it) to create a storage setup. As a result, we decided it was important to explain the situation when some combinations are not possible right away, likely because they need some previous step. Which brings us to…

Partitioner: more specific errors when a device is in use

In general, most of the checks already present in the Partitioner were already able to correctly handle situations in which the disk was a direct member of an MD RAID or an LVM volume group. But the message about the device being in use was not informative enough.

Now the message includes the name of the device that is making the operation impossible (it’s usually one, but there are corner cases in which it can be more than one), so the user has some clue about how to fix it.

Partitioner: improved creation of partition table

One part of the Partitioner that was specially bad at explaining the current situation and the possible consequences was the workflow of “Create New Partition Table”, which also used to exhibit a behavior quite inconsistent with the rest of the Partitioner actions.

In SLE-15 and openSUSE Leap 15.0, the “Create New Partition Table” button immediately presents a form to select the partition table type in case the device supports more than one.

And after the user selects one type it always shows a warning about all kind of devices to be destroyed, no matter if some device is really affected or not.

Even better, if only one partition table type is possible, it still shows the form but with no question. So creating a partition table in a completely empty DASD device result in a misleading warning (nothing is going to be destroyed) on top of an empty wizard.

So the whole action was reimplemented to display the warning only if some devices are indeed going to be affected (including the list of affected devices) and to display that warning as soon as the user clicks the button (as any other Partitioner action).

As seen in the screenshot, the check handles correctly situations in which the disk as a whole (no partitions) is part of an MD RAID or an LVM setup. And, of course, there are no empty wizard steps in the case of DASD or nothing like that. Now the workflow works in the expected way on each situation.

In short, the Storage-NG Partitioner is moving away, step by step, from being a 1:1 clone of the historical Partitioner to offer more functionality and usability. And there are more improvements to come in that area.

Partitioner: Unmounting devices in advance

The Partitioner allows to extensively configure your system storage devices. You can perform a lot of different kind of actions, from changing the label of a file system to creating a complex configuration by using LVM or RAIDs. Each modification you perform is stored in memory, so the real system is not altered at all until you confirm to apply the changes as last step. But in some circumstances, the Partitioner could not be able to perform some of the required actions, and it would fail when trying to modify the real system. One action that sometimes might fail is unmounting a device. This action might fail for several reasons, but the most common is because the file system is busy. And moreover, sometimes there are actions that require the device to be unmounted, for example, for deleting a partition, so the Partitioner would try to automatically unmount it.

During this sprint, the Partitioner has recovered its ability of unmounting devices on the fly to avoid possible failures when applying the changes. Now, if you want to delete a currently mounted device (e.g., a LVM Logical Volume) you will be asked in advance to unmount it. If you accept, the Partitioner will try to unmount the device on the fly without waiting to apply all the changes. In case the unmount action fails, you will be informed about the problem and you might try to manually solve the problem before the Partitioner applies the changes in your system. Of course, you can also continue without unmounting the device and the Partitioner will try to automatically unmount it after accepting all the changes.

Another task that might require unmounting the device is resizing the filesystem. The Partitioner will ask you about unmounting the device when the filesystem does not support resizing while being mounted. And, even when the filesystem does support it, you still might be requested to unmount the device. For example when you want to extend a device by more than 50 GiB. That task might be quite slow and it is highly recommended to unmount the device to speed up the resizing time, otherwise it could take hours.

Bug Fixes

Of course, we continue fighting against bugs. Thus, from this sprint on, alongside other minor stuff, the system

Elfcloud.fi a small cloud storage that could

February 13th, 2014 by

I’ve been seeking cloud storage of my life for long time now. My needs are not much (but most of the time they are too big as I have learned) only space and if possible Linux/Mac OS X FUSE for using service.
Service being open source or not isn’t such big thing in this case. How data is stored (crypted or not) and how can I get them out there if I need them is what I treasure most.
I have tested SpiderOak, Wuala, Dropbox and box.net but non of them fits for my needs perfectly. As I want to use these services with Linux all of them have Linux clients and most of them have FUSE-filesystem. Lately I have been using Wuala but it has problem that FUSE stuff is written in Java (it works under openSUSE just fine!). With GUI it’s clean and like said works very well but then comes but if you don’t want to use GUI you are in little bit trouble. It’s supported but I haven’t got it working. Xvfb comes to rescue but still it’s not really a solution!
When I heard about Elfcloud.fi I though okay we have some storage provider in Finland no big deal. I just popped their web site and noticed that they are really open source friendly company. They have Github Python-library (Apache Version 2.0 licensed) and full API documentation which is dead simple JSON stuff also their pricing scheme ain’t that bad. I have mention that this is hardcore crypto cloud service. If you lose your key you can wave goodbye to you data. Best of all they gonna have marvelous FUSE implementation as I’m writing it currently (They also have C++ library available on request. Apache Version 2.0 licensed of course).
So if you want to keep your data on Europe and have it stored in same country that have also is also trusted by Google datacenters and Microsoft you can check out elfcloud.fi. It’s not for everyone for sure but those who are in same need place to store data without hustle this can be the stuff for you.

You can download python API RPM from here: http://download.opensuse.org/repositories/home:/illuusio:/elfcloud/