Nah Open Source/Free Software doesn’t matter anyway

In year 2014 wasn’t just year on Linux getting into you hands as Android. It was strange year in Linux land. Probably biggest Linux hater from Redmond that called Linux Cancer said ‘hey Free Software and Open Source are cool! We take cherrys like Apple and sell them to you in nice package and there nothing wrong with that (And I’m with them.. it’s what they can do if you buy it)’. Small IoT (Internet-Of-Things, Industrial Internet or choose your favorite term) ARM SOC boards are so Nineties. MIPS made proud comeback! Once it was SGI’s own bitch but now there is growing mass of small very cheap wireless boards started to flow in. All supporting Linux and most have also Android. Targeted to next big thing that you are doing when not playing.

In Finland where I’m located this have been year of economical rumble (read economical crash). Mass layoffs, industrial work places are getting rare and Nokia was sold to that huge firm from Redmond. Selling Nokia out was end of Finland’s mobile dreams as they now have new tablet and fear not also Jolla is launching their new shiny Sailfish tablet and got very good funding through indie-go-go. Ok nobody really wants this tablet because of Android but it’s good to see there is guys and girls that still think with punk attitude and believe that very small firm like Jolla can produce something that big players can’t.

Year 2014 also was big climate is changing year. USA and China make promise to cut their carbon dioxide emission till 2030. USA will do something and China says their peak will be 2030. Does this stop global warming probably not but at least now it’s official that climate is changing. How this is relevant with Linux? Most of these calculation are done in ‘super computers’ and those run tadaaaa… Linux. They don’t run Fedora or openSUSE because they are mostly very fast calculators but they run Linux kernel and something that as immortals never will know because most of them doesn’t calculate climate change things. They calculate nuclear war scenarios. Hello Dr. Falken shall we play a game? Linux++ and HP Machine anyone?

Firefox which is now 10 years old. They broke up with Google because as you can see they don’t need that search flow anymore. Firefox preset for searching in USA is now Yahoo, Russian Yandex and China Baidu. Mozilla also made bold moves with Firefox OS in 2014. More phones and if you need low end cheap phone so you should consider Firefox OS phone 25$ that is very cheap.

So what happened in Linux land

openSUSE made only one release 13.2 but then rumbled big changes in. openSUSE is also catching up popular rolling release method like Arch Linux with Tumbleweed but this year openSUSE upstream Factory also become a rolling release and then it get together with Tumbleweed. It’s good thing! Really things get tested and traditional releases are more stable. Hooray for that. There was and gigantic war against systemd and you can start sending me millions of hate mails for a start because I Like it! It’s modern and does everything that was wrong with Linux init. No more big tune this tune that complex learn curve up to your a** stuff very neat simple configuration. Ok it’s bloat.. but why don’t you start hating Linux kernel. It’s not very UNIX nether it’s doesn’t just one thing like BSD micro-kernel approach. But that is what I wanted to say about that. If you fork Debian just to have Sysvinit I think you are little bit over reacting but best luck to you I’m nothing but thumbs up get it rolling! At the end this feels like Pulseaudio hating campaign. Everybody hated it first but now nobody even notices that they are using it.

Also it becomes clear 2014 that Desktop Linux will never come. Time just passed by and people moved to other things and we have desktop Linux Google’s Chromebooks. In year 2014 It also becomes clear that nor Tizen or Ubuntu phone will be huge success at least it takes little bit longer. My hope is that X will die in great crash of meteorite or let me rephrase how about Wayland project will never take X-windows place and things never become sweet again. Canonical  shiny move with Mir display server will shake things up if it becomes successful. Time will tell how things end up will Mir be successor of X or should it be Wayland?

So what to give as present if you want to give Linux based device?

There is so many Linux gift list to choose from that I just link to them so you can choose from yourself.

This was how I see year 2014. Let’s meet interesting year 2015 which is just few weeks away and light will come back and darkness fade away.

A bit of theory

Basically you have three possibilities how to make your boot fast.

1. Start things in a parallel.
2. Start things on demand.
3. Start less things.

… root of all evil …

“Premature optimization is the root of all evil.”

And I will modestly extend Donald Knuth’s sentence, that the blind optimization as well. So if we have to optimize something, we have to know what to do. Fortunately systemd comes with an excellent tool called systemd-analyze, which show us our boot in several ways.

The simple run of command prints the time we spent in a boot and in which phase.

# systemd-analyze
Startup finished in 8480ms (kernel) + 30873ms (userspace) = 39353ms

That was the default (minimal X system) 12.2 installation on my EEE 701 netbook, which is probably not suitable to work as nowadays cellsmarphone, because is pathetically slow. On the other hand is it a perfect playground, so let’s continue with an investigating.

The overall time is nice, but won’t help to know what’s going on. There are two more subcommands, blame and plot shows us more information about the boot. The first shows the services sorted by the start time. The ones boots so long are those we should kicked off as a first ones.

Let see what slow the boot down a most

$ systemd-analyze blame | head
 11385ms network.service
  5664ms SuSEfirewall2_init.service
  5575ms systemd-vconsole-setup.service
  3032ms ntp.service
  2840ms remount-rootfs.service
  2230ms postfix.service
  2021ms network-remotefs.service
  1925ms cpufreq.service
  1661ms SuSEfirewall2_setup.service
  1506ms xdm.service

And take look at the output of systemd-analyze plot command

You can see, that there is a long chain of SuSEfirewall2_init -> network -> network-remotefs -> SuSEfirewall2_setup tooks several dozen seconds to be finished. And nothing is wrong with that, but that is the server solution, not what I want to have on my tiny laptop.

Making a laptop boot twice more faster

So having the complex dependencies of several services in mind, I decided to mask some of them. Masking in systemd world means the service cannot be started using systemd, so it becomes invisible for it. I masked those

• network.service – will be replaced by NetworkManager, which is more suitable for laptops usage
• SuSEfirewall2_init and SuSEfirewall2_setup – even if it’s a security feature, a risc for laptop, which is mostly offline and running only sshd is pretty small.
• ntp.service, network-remotefs.service – those does not makes a sense on my laptop
• postfix.service – I do not want to send emails via /usr/bin/sendmail
• cpufreq.service – it is even not supported by my CPU (grep rc.cpufreq /var/log/messages)

Do not forget to install NetworkManager and the applet and change the /etc/sysconfig/network/config and reboot.

Now we have

$ systemd-analyze
Startup finished in 8528ms (kernel) + 11123ms (userspace) = 19652ms

Using an strace with systemd

Now we have a list of worse services

$ systemd-analyze blame | head -n 10
  5476ms xdm.service
  4172ms systemd-vconsole-setup.service
  3950ms systemd-modules-load.service
  2781ms remount-rootfs.service
  1848ms NetworkManager.service
  1439ms media.mount
  1426ms systemd-remount-api-vfs.service
  1419ms dev-hugepages.mount
  1411ms dev-mqueue.mount
  1371ms sys-kernel-debug.mount

and a proper boot chart

It shows us an another botleneck, which is the systemd-vconsole-setup.service, because it delay the sysinit.target, which is very early boot stage. In case like this, we can only use strace to know, what is taking too long. And debugging is pretty straightforward in systemd world. All we have to do is copy service file to /etc/systemd/system and change the ExecStart

ExecStart=/usr/bin/strace -f -tt -o /run/%N.strace /lib/systemd/systemd-vconsole-setup

and reboot. Then you will find the output in /run/systemd-vconsole-setup.strace with a timestamps. Looking there it’s obvious calling hwinfo --bios is extremely expensive in this stage. You can speedup the unit by setting the KBD_NUMLOCK to yes or no in /etc/sysconfig/keyboard, or you can try to mask it completely I did.

The next service needs to closer look was system-modules-load – then strace says that it spent 2(!) in init_module() for module microcode. I disabled it as well, even for CPUs needs it can’t be recommended.

Native systemd units

There is one tiny init script called purge-kernels, which starts for 300ms according blame. And in this particular case systemd alternative will be way more effective

$ cat /etc/systemd/system/purge-kernels.service
[Unit]
Description=Purge old kernels
After=local_fs.target
ConditionPathExists=/boot/do_pure_kernels

[Service]
Type=oneshot
ExecStart=/sbin/purge-kernels

because systemd only do one stat on the file and do not run it at all, so this service disappears from the blame at all.

The kernel time

There is one interesting thing about kernel time – 8 seconds spent there seems to be a lot to me. Simple ls on /boot gave me a pointer

$ ls -lh /boot/vmlinuz-* /boot/initrd-*
-rw-r--r-- 1 root root  14M Jul 24 11:03 /boot/initrd-3.4.4-1.1-desktop
-rw-r--r-- 1 root root 4.7M Jul 10 15:48 /boot/vmlinuz-3.4.4-1.1-desktop

The initrd is huge, around three times bigger than kernel? So let’s try to find what caused that. Every package can add it’s own setup script into /lib/mkinitrd/scripts/ thus let ask rpm whose did that

$ rpm -qf /lib/mkinitrd/scripts/setup-* | sort -u
cifs-utils-5.5-2.2.2.i586
cryptsetup-1.4.2-3.2.1.i586
device-mapper-1.02.63-26.1.1.i586
dmraid-1.0.0.rc16-18.2.1.i586
kpartx-0.4.9-3.1.1.i586
lvm2-2.02.84-26.1.1.i586
mdadm-3.2.5-3.3.2.i586
mkinitrd-2.7.0-62.2.1.i586
multipath-tools-0.4.9-3.1.1.i586
plymouth-scripts-0.8.5.1-1.3.1.noarch
splashy-0.3.13-35.1.1.i586

So I went through a list and try to uninstall things I do not need

• cifs-utils – if you do not have any windows disc to mount, you can remove, but no impact on initrd size
• cryptsetup – this is a popular service for laptops, but I do not have any luks device, so let skip that. It removes a half of Yast as well, so I saved 18M of space, but a little in initrd.
• device-mapper, dmraid, kpartx and lvm2 – cannot be easily removed as too much low-level stuff depends on it
• mdadm – no linux md devides, skip that
• mkinitrd – removal can reduce initrd to zero, but we would need own kernel

multipath-tools – no multipath device, let skip that

• plymouth-scripts – who would need the “fancy” boot when booting so fast? – reducing initrd to 8.9M
• splashy – the same – and reducing initrd to 6.6M

So the things intended to provide fancy boot actually bloats the system. Let’s measure the impact of those changes

$ systemd-analyze
2781ms (kernel) + 4999ms (userspace) = 7780ms

And that’s all folks …?

There are a lot of factors slowing our boot – reducing it to 8 seconds is not that bad. One have to go carefully through blame and plot output to see what delays his computer in start. I would say making NetworkManager default one at least when installing laptop pattern would be nice and simple change as well as continue on “systemdifization” of openSUSE.

There are few other tricks, which get us closer to the target time, but I’ll post them next day.