Home Home > Miscellaneous
Sign up | Login

Archive for the ‘Miscellaneous’ Category

Highlights of YaST development sprint 32

March 22nd, 2017 by

To make sure you didn’t missed us too much, in our latest blog post we summarized all the YaST-related projects worked during Hack Week 15. But after all the fun, it was time for… more fun! So let’s take a look to what the team has delivered on this first sprint after Hack Week 15.

Storage reimplementation: encrypted proposal without LVM

One of the known limitations of the current installer is that it’s only able to automatically propose an encrypted schema if LVM is used. For historical reasons, if you want to encrypt your root and/or home partitions but not to use LVM, you would need to use the expert partitioner… and hope for the best from the bootloader proposal.

But the new storage stack is here (well, almost here) to make all the old limitations vanish. With our testing ISO it’s already possible to set encryption with just one click for both partition-based and LVM-based proposals. The best possible partition schema is correctly created and everything is encrypted as the user would expect. We even have continuous tests in our internal openQA instance for it.

The part of the installer managing the bootloader installation is still not adapted, which means the resulting system would need some manual fixing of Grub before being able to boot… but that’s something for an upcoming sprint (likely the very next one).

Improved add-ons listing for SLE12-SP1

The dialog in SLES-12-SP1 for selecting the add-ons after registering the system was originally designed just for a small list of add-ons. Unfortunately (or fortunately, depending on how you look at it), the number of add-ons grew over the time and it exceeded the original limit for the text mode UI.

The equivalent screen in SLE-12-SP2 is not affected by the problem because it uses a different layout with scrollable list. But the SP1 dialog looks like this.

Broken add-ons list in SP1

If you look carefully at the screenshot you will see that the Web and Scripting Module is missing in the list and the Back, Next and Abort buttons at the bottom are also not displayed.

The fix decreased the size of the Details widget and allowed displaying more items in each column. Now there is even free space for three more add-ons.

Fixed addons list in SP1

Moreover the dialog is now dynamic and checks the current size of the screen. If there is enough free space then the list is displayed in one column so the labels are not truncated and the Details widget size is increased back to the original size.

Add-ons list in SP1 with enough space

Storage reimplementation: Btrfs subvolumes

The management of subvolumes is one of those features that make Btrfs rather unique and that need special handling when compared to more traditional file systems. That was indeed one of the several reasons to rewrite libstorage – Btrfs subvolumes never fully fitted the philosophy and data structures on the old (current) libstorage and yast2-storage.

In this sprint we introduced support for subvolumes in libstorage-ng from the ground up, taking into consideration all the specificities, use cases and scenarios found in the past. And, hopefully, in a way that is also prepared for whatever the future brings.

The new functionality is already working and tested and it’s included in the latest versions of libstorage-ng, but is still not used in the proposal or any other part yast2-storage. You will have to wait another sprint to see more visible results. At least if “more visible” means screenshots. Meanwhile, if you like images you can always enjoy the graphs generated from the internal structures managed by libstorage-ng.

Internal subvolumes representation in libstorage-ng

Storage reimplementation: system upgrade

The new storage stack has been able to install an openSUSE system for quite some time already. While we keep improving that area, the next challenge was to make the upgrade from a previous openSUSE version also possible using our testing ISO.

That implies scanning the hard disks looking for previous installations, allowing the user to select one, mounting the corresponding partitions or LVM volumes, performing the update of every package and doing some final tasks like updating the bootloader configuration.

Following the iterative approach encouraged by Scrum, we focused in the first three steps, which is something that a user (or openQA, for that matter) can test and verify. So now we are able to detect and list pre-existing systems and start the upgrade process on the selected one. And we have automated tests in openQA to ensure it works across all the combinations of partition-based vs LVM-based layout and UUID-based vs name-based fstab file.

Add-ons can define new system roles

YaST is pretty customizable when it comes to adapt/modify the installation workflow. Among other things, add-ons are allowed to adapt the workflow (adding/removing steps), define new proposals, etc. And starting now, they can also define new system roles.

Let’s see an example of adding a new mail server role:

<update>
  <system_roles>
    <insert_system_roles config:type="list">
      <insert_system_role>
        <system_roles config:type="list">
          <system_role>
            <id>mail_role</id>
            <software>
              <default_patterns>base Minimal mail_server</default_patterns>
            </software>
          </system_role>
        </system_roles>
      </insert_system_role>
    </insert_system_roles>
  </system_roles>
</update>

<!-- Don't forget to add the texts -->
<texts>
  <mail_role>
    <label>Mail Server</label>
  </mail_role>
  <mail_role_description>
    <label>• Software needed to set up a mail server
• No production ready yet!</label>
  </mail_role_description>
</texts>

And now let’s see how it looks:

A role added by an addon

Which leads us to the next section…

The list of roles becomes responsive in text mode

A really nice thing about YaST is that it’s able to run in textmode, so you don’t need a graphical interface to install or configure your system. As a consequence, YaST developers need to keep certain limitations in mind when working in the user interface.

Now that add-ons can add new system roles, we noticed a potential problem in the dialog selection screen: we eventually will get out of space if more than one system role is added. So we decided to improve how system roles are displayed to make them fit in a 80×25 mode (that is, only 25 lines of text). Let’s see the changes with some examples.

This is how the screen looks by default, with a reasonably small set of roles.

Default system roles list

If the system detects there is no space to present all the information in such a convenient way, it removes all the spaces so at least the information is all there, even if it looks a little bit packed.

Roles list with no extra space

If even that is not enough, the extra descriptions are omitted, which gives us way more room.

Compact roles list

If roles don’t fit even without the descriptions, the introductory text will be also omitted which means we can present up to eighteen (yes, 18!) roles in the screen.

Storage reimplementation: guided setup mock-up

As explained in several previous reports, we are collaborating closely with SUSE UX experts to design the revamped interfaces of the installer’s partitioning proposal and the expert partitioner. We already showed you the document we used as a base to discuss the partitioning proposal, including the conclusions, and the first very simple prototype of the so-called Guided Setup.

During this sprint, that collaborative effort focused on defining exactly how every step of that wizard should work and look like. The goal was to get some interface mock-ups to be used as starting point for the upcoming sprint. More than ever, a picture (well, four of them) is worth a thousand words.

First step of the guided partitioning setup mock-up

Second step of the guided partitioning setup mock-up

Third step of the guided partitioning setup mock-up

Fourth step of the guided partitioning setup mock-up

Prevent the installation of CaaSP if Btrfs snapshots are not possible

CaaSP is a single purpose system, and having snapshots enabled is essential. So there’s now a check in place that will simply prevent you from going on with the installation if snapshots are disabled (for example, if the disk is too small).

Blocked CaaSP installation

Storage reimplementation: better handling of /etc/fstab and /etc/cryptab

For the new storage stack, we refactored the classes to handle /etc/fstab. While this would normally not be anything to write much about, we included intelligent handling for existing comments based on this standalone GitHub project.

This means that existing comment blocks at the start and at the end of the file remain untouched, and comments before any content entry remain attached to that entry; i.e. when that entry is moved around in the file (e.g. because of mount dependencies), that comment is moving along with the entry it belongs to. While this is not 100% fool proof, it is much better than the usual strategy to simply discard such comments when the file is rewritten.

Quite some adaptations and bugfixes for CaaSP

As you already know from previous reports and other sources, a considerable part of SUSE’s development firepower is focused on building the upcoming CaaSP. As part of that heavy development process, the YaST team invested a significant part of the sprint adapting YaST for CaaSP and fixing bugs introduced by previous adaptations. A large collection of changes here and there that are hard to summarize here but that help CaaSP to be a couple of steps closer to the final goal.

Keep it rolling!

We have already planned our next sprint which will hopefully bring more features to the new storage stack, CaaSP-related improvements, a surprise about AutoYaST and more stuff. And, of course, it will be followed by its corresponding report.

So see you in three weeks. Stay tuned and have a lot of fun!

Fun things to do with driver updates

March 16th, 2017 by

Today: But what if I need a new kernel?

A driver update (DUD) can of course update a single driver. But if that’s not enough and you need a whole new kernel to run an installation?

There are two parts to solve:

  1. replace the kernel used during installation and
  2. get the new kernel installed

We’ll need two tools for this (both available in Tumbleweed or here: mksusecd and mkdud).

1. Replace the kernel used during installation

For this it’s important to know which kernel packages you’ll actually need. Typically it will be kernel-default and kernel-firmware. But older SUSE distributions (SLE 11 comes to mind) had the kernel packages split into kernel-default and kernel-default-base – you’ll need them both.

To make things confusing, modern SUSE distributions also have kernel-default-base – but it’s an alternative to kernel-default. In this case we don’t need it.

If unsure, check kernel-default. If it contains the actual kernel (e.g. /boot/vmlinuz) then you don’t need kernel-default-base.

On some architectures modules are also taken from xen-kmp-default. If that’s important for you, you can add this package to the kernel list as well.

In fact you can add any number of kernel packages or kmps you like.

In the past, sometimes a different kernel flavor was used. For example PowerPC had kernel-ppc64 for a while. Simply use the flavor you need.

It’s a good idea to gather all the kernel rpms into a single directory for easier use:

> mkdir k
> cp kernel-default.rpm kernel-firmware.rpm k
> cp kernel-default-base.rpm k    # only if needed
# add any kernel-related rpms you need

Then, take your SUSE installation iso and run

> mksusecd --create new.iso \
  --kernel k/* -- \
  original_dvd1.iso

Note that the --kernel option accepts a variable number of arguments, so you have to add an isolated -- to terminate the argument list properly.

The output could look like this:

> mksusecd --create new.iso \
  --kernel k/* -- \
  SLES-11-SP4-DVD-ppc64-GM-DVD1.iso
kernel version: 3.0.101-63-ppc64 --> 3.0.101-94-ppc64
CHRP bootable (ppc64)
building: 100%
calculating sha1...

The command above will actually get the list of required modules from the old installation iso. If you are missing some driver or the new kernel comes with some additional driver, the module will not be added to the new iso.

But there’s the --modules option. It will add the listed modules together with any implicitly required modules via module dependencies.

For example, let’s add the airport wifi-module to our PowerPC iso:

> mksusecd --create new.iso \
  --kernel k/* \
  --modules airport -- \
  SLES-11-SP4-DVD-ppc64-GM-DVD1.iso
kernel version: 3.0.101-63-ppc64 --> 3.0.101-94-ppc64
kernel modules added:
  airport, cfg80211, orinoco
CHRP bootable (ppc64)
building: 100%
calculating sha1...

As you can see, it automatically adds orinoco and cfg80211 as well.

2. Get the new kernel installed

This is relatively simple. A driver update can do this:

> mkdud --create foo.dud \
  --dist sle11 \
  --install repo \
  k/*

This creates a driver update for SLE 11 (which also applies to SP4) and the kernel rpms are installed via an auto-generated add-on repo (--install repo).

Now we have the driver update that installs our kernel packages. But how do we use it?

We integrate it into our iso above!

> mksusecd --create new.iso \
  --initrd foo.dud \
  --kernel k/* -- \
  SLES-11-SP4-DVD-ppc64-GM-DVD1.iso

mksusecd has an --initrd option that directly accepts driver updates and integrates them into the iso.

3. Can I have a choice?

Maybe you just want to test this new kernel or sometimes need the old one and sometimes the new one. Can you make an installation iso that lets you choose the kernel?

Oh yes! 🙂

> mksusecd --create new.iso \
  --add-entry 3.0.101-94 \
  --initrd foo.dud \
  --kernel k/* -- \
  SLES-11-SP4-DVD-ppc64-GM-DVD1.iso

This does not replace the old kernel but adds a new boot entry Installation - 3.0.101-94.

So you can install with old or the new kernel.

Fun things to do with driver updates

February 16th, 2017 by

Today: update the update process!

Yesterday a colleague asked me if it would be possible to apply a driver update (DUD) to the rescue system. He wanted to use a new btrfsprogs package.

My immediate reaction was: no, you can’t do it. But then, there’s no technical reason why it shouldn’t be possible – it actually nearly works. The updates are downloaded as usual – just not applied to the rescue system.

So I thought: “Why not make a driver update so driver updates work also for the rescue system?”

Here’s how I did it.

First, let’s find out how driver updates are usually applied. The code is here:

https://github.com/openSUSE/installation-images/blob/master/data/root/etc/inst_setup#L84-L87

We need just these three lines:

for i in /update/[0-9]*/inst-sys ; do
  [ -d "$i" ] && adddir "$i" /
done

linuxrc downloads the driver updates and stores them in an /update directory. One (numbered) subdirectory for each update.

It obviously uses some adddir script. So we’ll need it as well. Luckily, it’s not far away:

https://github.com/openSUSE/installation-images/blob/master/data/root/etc/adddir

Next, we’ll have to find the spot where the rescue system is set up. It’s done in this script:

https://github.com/openSUSE/installation-images/blob/master/data/initrd/scripts/prepare_rescue

Let’s do some copy-and-paste programming and insert the above code near the end of the script. It then might look like this

# driver update: add files to rescue system
if [ -d /mounts/initrd/update ] ; then
  cp -r /mounts/initrd/update /
  for i in /update/[0-9]*/inst-sys ; do
    [ -d "$i" ] && /mounts/initrd/scripts/adddir "$i" /
  done
fi

Some notes:

  • You have to know that prepare_rescue is run as the last thing before we exec to init. So everything is already in place, the left-over files from initrd are mounted at /mounts/initrd and will be removed at the end of the script.
  • This means we have to copy our updates into the new root directory, else they will be lost.
  • Also, we plan to make the adddir script available at /scripts/adddir by our driver update (see below).

Now let’s create the driver update:

mkdud --create dud_for_rescue.dud \
  --dist tw --dist leap42.1 --dist leap42.2 --dist sle12 \
  --name 'Apply DUD also to rescue system' \
  --exec 'cp adddir prepare_rescue /scripts' \
  adddir prepare_rescue

Here’s what this call does, line-by-line:

  • the fix works for all current SUSE distributions, so let’s support them
  • give the driver update some nice name
  • this command is run right after the driver update got loaded; we copy the scripts out of the driver update to their final location
  • add adddir and our modified prepare_rescue script

Here is the result: dud_for_rescue.dud.

Now, back to the original problem: how to use this to update a package in the rescue system? That’s easy:

mkdud --create new_btrfs.dud \
  --dist sle12 \
  dud_for_rescue.dud btrfsprogs.rpm

creates a driver update (for SLE12) that updates btrfsprogs also in the rescue system.

How we run our OpenStack cloud

January 23rd, 2017 by

This post it to document how we setup cloud.suse.de which is one of our many internal SUSE OpenStack Cloud deployments for use by R&D.

In 2016-06 we started the deployment with SOC6 on 4 nodes. 1 controller and 3 compute nodes that also served for ceph (distributed storage) with their 2nd HDD. Since the nodes are from 2012 they only have 1gbit network and spinning disks. Thus ceph only delivers ~50 MB/s which is sufficient for many use cases.

We did not deploy that cloud with HA, even though our product supports it. The two main reasons for that are

  • that it will use up two or three nodes instead of one for controller services, which is significant if you start out with only 4 (and grow to 6)
  • that it increases the complexity of setup, operations and debugging and thus might lead to decreased availability of the cloud service

Then we have a limited supply of vlans even though technically they are just numbers between 1 and 4095, in SUSE we do allocations to be able to switch together networks from further away. So we could not use vlan mode in neutron if we want to allow software defined network (=SDN) (we did not in old.cloud.suse.de and I did not hear complaints, but now I see a lot of people using SDN)
So we went with ovs+vxlan +dvr (open vSwitch + Virtual eXtensible LAN + Distributed Virtual Router) because that allows VMs to remain reachable even when the controller node reboots.
But then I found that they cannot use DNS during that time, because distributed virtual DNS was not yet implemented. And ovs has some annoying bugs are hard to debug and fix. So I built ugly workarounds that mostly hide^Wsolve the problems from our users’ point of view.
For the next cloud deployment, I will try to use linuxbridge+vlan or linuxbridge+vxlan mode.
And the uptime is pretty good. But it could be better with proper monitoring.

Because we needed to redeploy multiple times before we got all the details right and to document the setup, we scripted most of the deployment with qa_crowbarsetup (which is part of our CI) and extra files in https://github.com/SUSE-Cloud/automation/tree/production/scripts/productioncloud. The only part not in there are the passwords.

We use proper SSL certs from our internal SUSE CA.
For that we needed to install that root CA on all involved nodes.

We use kvm, because it is the most advanced and stable of the supported hypervisors. Xen might be a possible 2nd choice. We use two custom kvm patches to fix nested virt on our G3 Opteron CPUs.

Overall we use 3 vlans. One each for admin, public/floating, sdn/storage networks.
We increased the default /24 IP ranges because we needed more IPs in the fixed and public/floating networks

For authentication, we use our internal R&D LDAP server, but since it does not have information about user’s groups, I wrote a perl script to pull that information from the Novell/innerweb LDAP server and export it as json for use by the hybrid_json assignment backend I wrote.

In addition I wrote a cloud-stats.sh to email weekly reports about utilization of the cloud and another script to tell users about which instances they still have, but might have forgotten.

On the cloud user side, we and other people use one or more of

  • Salt-cloud
  • Nova boot
  • salt-ssh
  • terraform
  • heat

to script instance setup and administration.

Overall we are now hosting 70 instance VMs on 5 compute nodes that together have cost us below 20000€

Live USB improvements

August 14th, 2016 by

Tools to create multi distribution bootable USB stick got couple of new improvements and features.

live-usb-gui now offers choice of scripts to use, depending in your need you can either use live-fat-stick with vfat partitioned stick or live-grub-stick script which works with any partition format supported by grub2 including vfat, must be used if you have iso bigger than 4G.

(more…)

Tally ERP 9 on Linux

July 21st, 2016 by

Recently we implemented Tally ERP 9 solution for Antico Pumps. That itself is not interesting, the interesting part is they are using LTSP Fat client system on openSUSE. They have only one server from which all their client computers boot over the network, the clients do not have hard disk, client OS with all softwares they need including wine(Tally is Windows only software), as well as users’ data resides on the server. Once the client boots all the local resources are used so single low power server can be used to serve many clients.

Tally multiuser is served from a Samba share  on a NAS device, Tally folder is copied to samba share and path to Tally Data is changed so that it points there. Everything they need including printing and export(CSV) works from all clients. Same way Tally can be run on standalone computers. Neither Tally, Wine or openSUSE are modified for getting it working as it would under Windows environment.

Oh hell! It’s open source project

August 18th, 2015 by

This was supposed to be survival guide to open source and free software world but I realized I’m not that good citizen of open source world that I can give any advises to others. What I’m giving are hint’s what I have learn along the years. So why I’m not very good open source citizen? I read several projects mail lists but only topics that I like and make contributions but not with rage but when I feel like it. I answer few mails that I receive about open source in limited time frame that I have (which sometimes can be too long) and use many projects with out giving anything back. I prefer license to steal and freedom as value not as in beer.
(more…)

How To Get Superbly Hinted Fonts in openSUSE 13.2

May 10th, 2015 by

Hey geekos,

So, this has been kind of my obsession these past years using openSUSE – I couldn’t get the fonts to be gently hinted in the way I want them to be. Every patch or repo had an issue with displaying a specific group of fonts. Now I finally found a way to get my fonts to look like this:

Font hinting

(more…)

Ever wanted to be a Dj with open source touch?

April 29th, 2015 by

There are plenty of Dj software available on Internet. Most popular I think are Traktor and VirtualDJ. Those are no brainier to choose and don’t support Linux. Because I’m old fart and I started doing my dang long time a go with Technics vinyl-players (and still play my gigs with them). They work as they have always worked great but I though that I need new geeky Dj system with digital vinyls because many interesting release doesn’t do vinyls anymore and I don’t like CD-format. Summarizing all of that I wanted something that what is open source and I can still attach my digital vinyls to it (so it should work with Serato or Traktor vinyls).
(more…)

Linux audio library smackdown part4: LibAO

March 2nd, 2015 by

Last time I’ve took look at Simple Direct Layer and how to get audio out of it. If SDL still feels little bit too hard to cope with I think I have solutions for you: libAO. Besides being no brainier with API libAO provides huge list of supported operating systems.
There is so much audio systems supported that you won’t be dissapointed but as much as I like everyone use Roaraudio. I don’t see it’s happening really soon (sorry roar you had your time in fame) but supporting Roaraudio  doesn’t mean that libAO is obsolete. It’s far from being obsolete. Libao supports OSS, ALSA and Pulseaudio out of the box and only problem is license is GPL 2.0+ so it’s no-go for proprietary development.

History

LibAO is developed under Xiph umbrella. Xiph is the organization who brought you Ogg/Vorbis, FLAC, Theora and currently they are hammering together next generation video codec Daala. Opus-audio codec standard is also Xiph project. LibAO rised from Xiph’s need multi-platform audio output library for Vorbis-audio codec. In this point if you don’t have any glue what I just said in last sentences I think you should take your spoon and start shovelling about Open Source audio codecs.
Becaus of the history libAO only has output mode and doesn’t use any callbacks. It doesn’t have fancy Float32 mode (as much as I understood) but that doesn’t say it’s bad thing. It works as expected you just feed bytes and after while you hear them from your speakers.

What about API

Supported outputs: Alsa, Oss, Jack, Mac OS X, Windows
License: GNU General Public license 2.0+

As said libAO API is difficult to describe since there almost ain’t NAN of it. You initialize, ask output device, put in your mode settings and start feeding data. Pulseaudio simple is almost easy as this but it’s still more difficult if you compare it to libAO. LibAO doesn’t support recording so only output and there must be a way to use another device than default but it’s not very easy to find or I was too lazy to dig it out.

So who wants to use libAO? People in hurry and don’t mind GPL-license, someone with very very tiny need of just getting audio out and people how hate bloat.

Summary: So if you hate bloat and again license doesn’t make you unhappy please use this library. Still libAO has kind of same problem that ALSA has. It’s mature, usable and ready for hardcore torturing but is it sexy? No! Is fresh? No, No! Is something that will change it API every week or hour?

After this I have to choose what to bring next. I have FFmpeg, Gstreamer and VLC in row. If you have opinion about next framework let me know.