Home Home > Miscellaneous
Sign up | Login

Archive for the ‘Miscellaneous’ Category

Fun things to do with driver updates

February 16th, 2017 by

Today: update the update process!

Yesterday a colleague asked me if it would be possible to apply a driver update (DUD) to the rescue system. He wanted to use a new btrfsprogs package.

My immediate reaction was: no, you can’t do it. But then, there’s no technical reason why it shouldn’t be possible – it actually nearly works. The updates are downloaded as usual – just not applied to the rescue system.

So I thought: “Why not make a driver update so driver updates work also for the rescue system?”

Here’s how I did it.

First, let’s find out how driver updates are usually applied. The code is here:


We need just these three lines:

for i in /update/[0-9]*/inst-sys ; do
  [ -d "$i" ] && adddir "$i" /

linuxrc downloads the driver updates and stores them in an /update directory. One (numbered) subdirectory for each update.

It obviously uses some adddir script. So we’ll need it as well. Luckily, it’s not far away:


Next, we’ll have to find the spot where the rescue system is set up. It’s done in this script:


Let’s do some copy-and-paste programming and insert the above code near the end of the script. It then might look like this

# driver update: add files to rescue system
if [ -d /mounts/initrd/update ] ; then
  cp -r /mounts/initrd/update /
  for i in /update/[0-9]*/inst-sys ; do
    [ -d "$i" ] && /mounts/initrd/scripts/adddir "$i" /

Some notes:

  • You have to know that prepare_rescue is run as the last thing before we exec to init. So everything is already in place, the left-over files from initrd are mounted at /mounts/initrd and will be removed at the end of the script.
  • This means we have to copy our updates into the new root directory, else they will be lost.
  • Also, we plan to make the adddir script available at /scripts/adddir by our driver update (see below).

Now let’s create the driver update:

mkdud --create dud_for_rescue.dud \
  --dist tw --dist leap42.1 --dist leap42.2 --dist sle12 \
  --name 'Apply DUD also to rescue system' \
  --exec 'cp adddir prepare_rescue /scripts' \
  adddir prepare_rescue

Here’s what this call does, line-by-line:

  • the fix works for all current SUSE distributions, so let’s support them
  • give the driver update some nice name
  • this command is run right after the driver update got loaded; we copy the scripts out of the driver update to their final location
  • add adddir and our modified prepare_rescue script

Here is the result: dud_for_rescue.dud.

Now, back to the original problem: how to use this to update a package in the rescue system? That’s easy:

mkdud --create new_btrfs.dud \
  --dist sle12 \
  dud_for_rescue.dud btrfsprogs.rpm

creates a driver update (for SLE12) that updates btrfsprogs also in the rescue system.

How we run our OpenStack cloud

January 23rd, 2017 by

This post it to document how we setup cloud.suse.de which is one of our many internal SUSE OpenStack Cloud deployments for use by R&D.

In 2016-06 we started the deployment with SOC6 on 4 nodes. 1 controller and 3 compute nodes that also served for ceph (distributed storage) with their 2nd HDD. Since the nodes are from 2012 they only have 1gbit network and spinning disks. Thus ceph only delivers ~50 MB/s which is sufficient for many use cases.

We did not deploy that cloud with HA, even though our product supports it. The two main reasons for that are

  • that it will use up two or three nodes instead of one for controller services, which is significant if you start out with only 4 (and grow to 6)
  • that it increases the complexity of setup, operations and debugging and thus might lead to decreased availability of the cloud service

Then we have a limited supply of vlans even though technically they are just numbers between 1 and 4095, in SUSE we do allocations to be able to switch together networks from further away. So we could not use vlan mode in neutron if we want to allow software defined network (=SDN) (we did not in old.cloud.suse.de and I did not hear complaints, but now I see a lot of people using SDN)
So we went with ovs+vxlan +dvr (open vSwitch + Virtual eXtensible LAN + Distributed Virtual Router) because that allows VMs to remain reachable even when the controller node reboots.
But then I found that they cannot use DNS during that time, because distributed virtual DNS was not yet implemented. And ovs has some annoying bugs are hard to debug and fix. So I built ugly workarounds that mostly hide^Wsolve the problems from our users’ point of view.
For the next cloud deployment, I will try to use linuxbridge+vlan or linuxbridge+vxlan mode.
And the uptime is pretty good. But it could be better with proper monitoring.

Because we needed to redeploy multiple times before we got all the details right and to document the setup, we scripted most of the deployment with qa_crowbarsetup (which is part of our CI) and extra files in https://github.com/SUSE-Cloud/automation/tree/production/scripts/productioncloud. The only part not in there are the passwords.

We use proper SSL certs from our internal SUSE CA.
For that we needed to install that root CA on all involved nodes.

We use kvm, because it is the most advanced and stable of the supported hypervisors. Xen might be a possible 2nd choice. We use two custom kvm patches to fix nested virt on our G3 Opteron CPUs.

Overall we use 3 vlans. One each for admin, public/floating, sdn/storage networks.
We increased the default /24 IP ranges because we needed more IPs in the fixed and public/floating networks

For authentication, we use our internal R&D LDAP server, but since it does not have information about user’s groups, I wrote a perl script to pull that information from the Novell/innerweb LDAP server and export it as json for use by the hybrid_json assignment backend I wrote.

In addition I wrote a cloud-stats.sh to email weekly reports about utilization of the cloud and another script to tell users about which instances they still have, but might have forgotten.

On the cloud user side, we and other people use one or more of

  • Salt-cloud
  • Nova boot
  • salt-ssh
  • terraform
  • heat

to script instance setup and administration.

Overall we are now hosting 70 instance VMs on 5 compute nodes that together have cost us below 20000€

Live USB improvements

August 14th, 2016 by

Tools to create multi distribution bootable USB stick got couple of new improvements and features.

live-usb-gui now offers choice of scripts to use, depending in your need you can either use live-fat-stick with vfat partitioned stick or live-grub-stick script which works with any partition format supported by grub2 including vfat, must be used if you have iso bigger than 4G.


Tally ERP 9 on Linux

July 21st, 2016 by

Recently we implemented Tally ERP 9 solution for Antico Pumps. That itself is not interesting, the interesting part is they are using LTSP Fat client system on openSUSE. They have only one server from which all their client computers boot over the network, the clients do not have hard disk, client OS with all softwares they need including wine(Tally is Windows only software), as well as users’ data resides on the server. Once the client boots all the local resources are used so single low power server can be used to serve many clients.

Tally multiuser is served from a Samba share  on a NAS device, Tally folder is copied to samba share and path to Tally Data is changed so that it points there. Everything they need including printing and export(CSV) works from all clients. Same way Tally can be run on standalone computers. Neither Tally, Wine or openSUSE are modified for getting it working as it would under Windows environment.

Oh hell! It’s open source project

August 18th, 2015 by

This was supposed to be survival guide to open source and free software world but I realized I’m not that good citizen of open source world that I can give any advises to others. What I’m giving are hint’s what I have learn along the years. So why I’m not very good open source citizen? I read several projects mail lists but only topics that I like and make contributions but not with rage but when I feel like it. I answer few mails that I receive about open source in limited time frame that I have (which sometimes can be too long) and use many projects with out giving anything back. I prefer license to steal and freedom as value not as in beer.

How To Get Superbly Hinted Fonts in openSUSE 13.2

May 10th, 2015 by

Hey geekos,

So, this has been kind of my obsession these past years using openSUSE – I couldn’t get the fonts to be gently hinted in the way I want them to be. Every patch or repo had an issue with displaying a specific group of fonts. Now I finally found a way to get my fonts to look like this:

Font hinting


Ever wanted to be a Dj with open source touch?

April 29th, 2015 by

There are plenty of Dj software available on Internet. Most popular I think are Traktor and VirtualDJ. Those are no brainier to choose and don’t support Linux. Because I’m old fart and I started doing my dang long time a go with Technics vinyl-players (and still play my gigs with them). They work as they have always worked great but I though that I need new geeky Dj system with digital vinyls because many interesting release doesn’t do vinyls anymore and I don’t like CD-format. Summarizing all of that I wanted something that what is open source and I can still attach my digital vinyls to it (so it should work with Serato or Traktor vinyls).

Linux audio library smackdown part4: LibAO

March 2nd, 2015 by

Last time I’ve took look at Simple Direct Layer and how to get audio out of it. If SDL still feels little bit too hard to cope with I think I have solutions for you: libAO. Besides being no brainier with API libAO provides huge list of supported operating systems.
There is so much audio systems supported that you won’t be dissapointed but as much as I like everyone use Roaraudio. I don’t see it’s happening really soon (sorry roar you had your time in fame) but supporting Roaraudio  doesn’t mean that libAO is obsolete. It’s far from being obsolete. Libao supports OSS, ALSA and Pulseaudio out of the box and only problem is license is GPL 2.0+ so it’s no-go for proprietary development.


LibAO is developed under Xiph umbrella. Xiph is the organization who brought you Ogg/Vorbis, FLAC, Theora and currently they are hammering together next generation video codec Daala. Opus-audio codec standard is also Xiph project. LibAO rised from Xiph’s need multi-platform audio output library for Vorbis-audio codec. In this point if you don’t have any glue what I just said in last sentences I think you should take your spoon and start shovelling about Open Source audio codecs.
Becaus of the history libAO only has output mode and doesn’t use any callbacks. It doesn’t have fancy Float32 mode (as much as I understood) but that doesn’t say it’s bad thing. It works as expected you just feed bytes and after while you hear them from your speakers.

What about API

Supported outputs: Alsa, Oss, Jack, Mac OS X, Windows
License: GNU General Public license 2.0+

As said libAO API is difficult to describe since there almost ain’t NAN of it. You initialize, ask output device, put in your mode settings and start feeding data. Pulseaudio simple is almost easy as this but it’s still more difficult if you compare it to libAO. LibAO doesn’t support recording so only output and there must be a way to use another device than default but it’s not very easy to find or I was too lazy to dig it out.

So who wants to use libAO? People in hurry and don’t mind GPL-license, someone with very very tiny need of just getting audio out and people how hate bloat.

Summary: So if you hate bloat and again license doesn’t make you unhappy please use this library. Still libAO has kind of same problem that ALSA has. It’s mature, usable and ready for hardcore torturing but is it sexy? No! Is fresh? No, No! Is something that will change it API every week or hour?

After this I have to choose what to bring next. I have FFmpeg, Gstreamer and VLC in row. If you have opinion about next framework let me know.

Linux audio library smackdown part2: Pulseaudio

January 27th, 2015 by

Oh yes.. those were great times! Open Sound System was rocking my Linux based sound system and I was having a time of my life. Like we all know good doesn’t last long. Soon after happiness I find out that OSS couldn’t do recording and playing (full duplex) same time. Fiery same time this was working on Windows 98 very well. I was shocked how the heck my Linux box is so borked? Luckily times were changing and this time they formed only better. ALSA was about to take over OSS in Kernel version 2.6. Why I’m talking about ALSA and OSS when I should talk about Pulseaudio? Read further to find out why or go examples for Pulseaudio here: https://github.com/illuusio/linux-audio-example/tree/master/pulseaudio (more…)

It’s hoolihoolihooliday and what fun I can do?

December 16th, 2014 by

I know I’m old losah! Nowadays it’s all about pads, phones and mobile this and mobile that. Install some cool stuff from app store and start mangling. I was amused how much you can do with these small entertainment units but then comes small detail that is not that cool over 20% stuff that is downloaded from Apple store is games and those neat utilities are just 5%. I don’t know what education means but it’s after games so I think it’s good thing. What I’m trying to say stop playing for a while (it’s good thing to do. Don’t get me wrong), put pad/phone down and look around what kind of world we are living in 2014.

Nah Open Source/Free Software doesn’t matter anyway

In year 2014 wasn’t just year on Linux getting into you hands as Android. It was strange year in Linux land. Probably biggest Linux hater from Redmond that called Linux Cancer said ‘hey Free Software and Open Source are cool! We take cherrys like Apple and sell them to you in nice package and there nothing wrong with that (And I’m with them.. it’s what they can do if you buy it)’. Small IoT (Internet-Of-Things, Industrial Internet or choose your favorite term) ARM SOC boards are so Nineties. MIPS made proud comeback! Once it was SGI’s own bitch but now there is growing mass of small very cheap wireless boards started to flow in. All supporting Linux and most have also Android. Targeted to next big thing that you are doing when not playing.

In Finland where I’m located this have been year of economical rumble (read economical crash). Mass layoffs, industrial work places are getting rare and Nokia was sold to that huge firm from Redmond. Selling Nokia out was end of Finland’s mobile dreams as they now have new tablet and fear not also Jolla is launching their new shiny Sailfish tablet and got very good funding through indie-go-go. Ok nobody really wants this tablet because of Android but it’s good to see there is guys and girls that still think with punk attitude and believe that very small firm like Jolla can produce something that big players can’t.

Year 2014 also was big climate is changing year. USA and China make promise to cut their carbon dioxide emission till 2030. USA will do something and China says their peak will be 2030. Does this stop global warming probably not but at least now it’s official that climate is changing. How this is relevant with Linux? Most of these calculation are done in ‘super computers’ and those run tadaaaa… Linux. They don’t run Fedora or openSUSE because they are mostly very fast calculators but they run Linux kernel and something that as immortals never will know because most of them doesn’t calculate climate change things. They calculate nuclear war scenarios. Hello Dr. Falken shall we play a game? Linux++ and HP Machine anyone?

Firefox which is now 10 years old. They broke up with Google because as you can see they don’t need that search flow anymore. Firefox preset for searching in USA is now Yahoo, Russian Yandex and China Baidu. Mozilla also made bold moves with Firefox OS in 2014. More phones and if you need low end cheap phone so you should consider Firefox OS phone 25$ that is very cheap.

So what happened in Linux land

openSUSE made only one release 13.2 but then rumbled big changes in. openSUSE is also catching up popular rolling release method like Arch Linux with Tumbleweed but this year openSUSE upstream Factory also become a rolling release and then it get together with Tumbleweed. It’s good thing! Really things get tested and traditional releases are more stable. Hooray for that. There was and gigantic war against systemd and you can start sending me millions of hate mails for a start because I Like it! It’s modern and does everything that was wrong with Linux init. No more big tune this tune that complex learn curve up to your a** stuff very neat simple configuration. Ok it’s bloat.. but why don’t you start hating Linux kernel. It’s not very UNIX nether it’s doesn’t just one thing like BSD micro-kernel approach. But that is what I wanted to say about that. If you fork Debian just to have Sysvinit I think you are little bit over reacting but best luck to you I’m nothing but thumbs up get it rolling! At the end this feels like Pulseaudio hating campaign. Everybody hated it first but now nobody even notices that they are using it.

Also it becomes clear 2014 that Desktop Linux will never come. Time just passed by and people moved to other things and we have desktop Linux Google’s Chromebooks. In year 2014 It also becomes clear that nor Tizen or Ubuntu phone will be huge success at least it takes little bit longer. My hope is that X will die in great crash of meteorite or let me rephrase how about Wayland project will never take X-windows place and things never become sweet again. Canonical  shiny move with Mir display server will shake things up if it becomes successful. Time will tell how things end up will Mir be successor of X or should it be Wayland?

So what to give as present if you want to give Linux based device?

There is so many Linux gift list to choose from that I just link to them so you can choose from yourself.

This was how I see year 2014. Let’s meet interesting year 2015 which is just few weeks away and light will come back and darkness fade away.