During the hackweek, I have decided to take a look onto a boot process to realize, how fast can be boot of the system. This metric is considered as a not important, especially in a geek community. But I am sure that this is an important part of user experience. Following text is mean to share my investigations with you – but you should be aware what you are doing, since I have been focused on reducing the boot time as much as possible.
A bit of theory
Basically you have three possibilities how to make your boot fast.
- Start things in a parallel.
- Start things on demand.
- Start less things.
… root of all evil …
“Premature optimization is the root of all evil.”
And I will modestly extend Donald Knuth’s sentence, that the blind optimization as well. So if we have to optimize something, we have to know what to do. Fortunately systemd comes with an excellent tool called systemd-analyze, which show us our boot in several ways.
The simple run of command prints the time we spent in a boot and in which phase.
# systemd-analyze
Startup finished in 8480ms (kernel) + 30873ms (userspace) = 39353ms
That was the default (minimal X system) 12.2 installation on my EEE 701 netbook, which is probably not suitable to work as nowadays cellsmarphone, because is pathetically slow. On the other hand is it a perfect playground, so let’s continue with an investigating.
The overall time is nice, but won’t help to know what’s going on. There are two more subcommands, blame and plot shows us more information about the boot. The first shows the services sorted by the start time. The ones boots so long are those we should kicked off as a first ones.
Let see what slow the boot down a most
$ systemd-analyze blame | head
11385ms network.service
5664ms SuSEfirewall2_init.service
5575ms systemd-vconsole-setup.service
3032ms ntp.service
2840ms remount-rootfs.service
2230ms postfix.service
2021ms network-remotefs.service
1925ms cpufreq.service
1661ms SuSEfirewall2_setup.service
1506ms xdm.service
And take look at the output of systemd-analyze plot command
You can see, that there is a long chain of SuSEfirewall2_init -> network -> network-remotefs -> SuSEfirewall2_setup tooks several dozen seconds to be finished. And nothing is wrong with that, but that is the server solution, not what I want to have on my tiny laptop.
Making a laptop boot twice more faster
So having the complex dependencies of several services in mind, I decided to mask some of them. Masking in systemd world means the service cannot be started using systemd, so it becomes invisible for it. I masked those
network.service – will be replaced by NetworkManager, which is more suitable for laptops usageSuSEfirewall2_init and SuSEfirewall2_setup – even if it’s a security feature, a risc for laptop, which is mostly offline and running only sshd is pretty small.ntp.service, network-remotefs.service – those does not makes a sense on my laptoppostfix.service – I do not want to send emails via /usr/bin/sendmailcpufreq.service – it is even not supported by my CPU (grep rc.cpufreq /var/log/messages)
Do not forget to install NetworkManager and the applet and change the /etc/sysconfig/network/config and reboot.
Now we have
$ systemd-analyze
Startup finished in 8528ms (kernel) + 11123ms (userspace) = 19652ms
Using an strace with systemd
Now we have a list of worse services
$ systemd-analyze blame | head -n 10
5476ms xdm.service
4172ms systemd-vconsole-setup.service
3950ms systemd-modules-load.service
2781ms remount-rootfs.service
1848ms NetworkManager.service
1439ms media.mount
1426ms systemd-remount-api-vfs.service
1419ms dev-hugepages.mount
1411ms dev-mqueue.mount
1371ms sys-kernel-debug.mount
and a proper boot chart
It shows us an another botleneck, which is the systemd-vconsole-setup.service, because it delay the sysinit.target, which is very early boot stage. In case like this, we can only use strace to know, what is taking too long. And debugging is pretty straightforward in systemd world. All we have to do is copy service file to /etc/systemd/system and change the ExecStart
ExecStart=/usr/bin/strace -f -tt -o /run/%N.strace /lib/systemd/systemd-vconsole-setup
and reboot. Then you will find the output in /run/systemd-vconsole-setup.strace with a timestamps. Looking there it’s obvious calling hwinfo --bios is extremely expensive in this stage. You can speedup the unit by setting the KBD_NUMLOCK to yes or no in /etc/sysconfig/keyboard, or you can try to mask it completely I did.
The next service needs to closer look was system-modules-load – then strace says that it spent 2(!) in init_module() for module microcode. I disabled it as well, even for CPUs needs it can’t be recommended.
Native systemd units
There is one tiny init script called purge-kernels, which starts for 300ms according blame. And in this particular case systemd alternative will be way more effective
$ cat /etc/systemd/system/purge-kernels.service
[Unit]
Description=Purge old kernels
After=local_fs.target
ConditionPathExists=/boot/do_pure_kernels
[Service]
Type=oneshot
ExecStart=/sbin/purge-kernels
because systemd only do one stat on the file and do not run it at all, so this service disappears from the blame at all.
The kernel time
There is one interesting thing about kernel time – 8 seconds spent there seems to be a lot to me. Simple ls on /boot gave me a pointer
$ ls -lh /boot/vmlinuz-* /boot/initrd-*
-rw-r--r-- 1 root root 14M Jul 24 11:03 /boot/initrd-3.4.4-1.1-desktop
-rw-r--r-- 1 root root 4.7M Jul 10 15:48 /boot/vmlinuz-3.4.4-1.1-desktop
The initrd is huge, around three times bigger than kernel? So let’s try to find what caused that. Every package can add it’s own setup script into /lib/mkinitrd/scripts/ thus let ask rpm whose did that
$ rpm -qf /lib/mkinitrd/scripts/setup-* | sort -u
cifs-utils-5.5-2.2.2.i586
cryptsetup-1.4.2-3.2.1.i586
device-mapper-1.02.63-26.1.1.i586
dmraid-1.0.0.rc16-18.2.1.i586
kpartx-0.4.9-3.1.1.i586
lvm2-2.02.84-26.1.1.i586
mdadm-3.2.5-3.3.2.i586
mkinitrd-2.7.0-62.2.1.i586
multipath-tools-0.4.9-3.1.1.i586
plymouth-scripts-0.8.5.1-1.3.1.noarch
splashy-0.3.13-35.1.1.i586
So I went through a list and try to uninstall things I do not need
cifs-utils – if you do not have any windows disc to mount, you can remove, but no impact on initrd sizecryptsetup – this is a popular service for laptops, but I do not have any luks device, so let skip that. It removes a half of Yast as well, so I saved 18M of space, but a little in initrd.device-mapper, dmraid, kpartx and lvm2 – cannot be easily removed as too much low-level stuff depends on itmdadm – no linux md devides, skip thatmkinitrd – removal can reduce initrd to zero, but we would need own kernelmultipath-tools – no multipath device, let skip that
plymouth-scripts – who would need the “fancy” boot when booting so fast? – reducing initrd to 8.9Msplashy – the same – and reducing initrd to 6.6M
So the things intended to provide fancy boot actually bloats the system. Let’s measure the impact of those changes
$ systemd-analyze
2781ms (kernel) + 4999ms (userspace) = 7780ms
And that’s all folks …?
There are a lot of factors slowing our boot – reducing it to 8 seconds is not that bad. One have to go carefully through blame and plot output to see what delays his computer in start. I would say making NetworkManager default one at least when installing laptop pattern would be nice and simple change as well as continue on “systemdifization” of openSUSE.
There are few other tricks, which get us closer to the target time, but I’ll post them next day.
I am really looking forward to the next Hackweek that we have in Novell – it will be in the week from 7-11 of June 2010.
