Home Home > 2012 > 07 > 26 > Optimizing a boot time, aka 2 second boot
Sign up | Login

Optimizing a boot time, aka 2 second boot

July 26th, 2012 by

During the hackweek, I have decided to take a look onto a boot process to realize, how fast can be boot of the system. This metric is considered as a not important, especially in a geek community. But I am sure that this is an important part of user experience. Following text is mean to share my investigations with you – but you should be aware what you are doing, since I have been focused on reducing the boot time as much as possible.

A bit of theory

Basically you have three possibilities how to make your boot fast.

  1. Start things in a parallel.
  2. Start things on demand.
  3. Start less things.

… root of all evil …

“Premature optimization is the root of all evil.”

And I will modestly extend Donald Knuth’s sentence, that the blind optimization as well. So if we have to optimize something, we have to know what to do. Fortunately systemd comes with an excellent tool called systemd-analyze, which show us our boot in several ways.

The simple run of command prints the time we spent in a boot and in which phase.

# systemd-analyze
Startup finished in 8480ms (kernel) + 30873ms (userspace) = 39353ms

That was the default (minimal X system) 12.2 installation on my EEE 701 netbook, which is probably not suitable to work as nowadays cellsmarphone, because is pathetically slow. On the other hand is it a perfect playground, so let’s continue with an investigating.

The overall time is nice, but won’t help to know what’s going on. There are two more subcommands, blame and plot shows us more information about the boot. The first shows the services sorted by the start time. The ones boots so long are those we should kicked off as a first ones.

Let see what slow the boot down a most

$ systemd-analyze blame | head
 11385ms network.service
  5664ms SuSEfirewall2_init.service
  5575ms systemd-vconsole-setup.service
  3032ms ntp.service
  2840ms remount-rootfs.service
  2230ms postfix.service
  2021ms network-remotefs.service
  1925ms cpufreq.service
  1661ms SuSEfirewall2_setup.service
  1506ms xdm.service

And take look at the output of systemd-analyze plot command

openSUSE 12.2 boot diagram

You can see, that there is a long chain of SuSEfirewall2_init -> network -> network-remotefs -> SuSEfirewall2_setup tooks several dozen seconds to be finished. And nothing is wrong with that, but that is the server solution, not what I want to have on my tiny laptop.

Making a laptop boot twice more faster

So having the complex dependencies of several services in mind, I decided to mask some of them. Masking in systemd world means the service cannot be started using systemd, so it becomes invisible for it. I masked those

  • network.service – will be replaced by NetworkManager, which is more suitable for laptops usage
  • SuSEfirewall2_init and SuSEfirewall2_setup – even if it’s a security feature, a risc for laptop, which is mostly offline and running only sshd is pretty small.
  • ntp.service, network-remotefs.service – those does not makes a sense on my laptop
  • postfix.service – I do not want to send emails via /usr/bin/sendmail
  • cpufreq.service – it is even not supported by my CPU (grep rc.cpufreq /var/log/messages)

Do not forget to install NetworkManager and the applet and change the /etc/sysconfig/network/config and reboot.

Now we have

$ systemd-analyze
Startup finished in 8528ms (kernel) + 11123ms (userspace) = 19652ms

Using an strace with systemd

Now we have a list of worse services

$ systemd-analyze blame | head -n 10
  5476ms xdm.service
  4172ms systemd-vconsole-setup.service
  3950ms systemd-modules-load.service
  2781ms remount-rootfs.service
  1848ms NetworkManager.service
  1439ms media.mount
  1426ms systemd-remount-api-vfs.service
  1419ms dev-hugepages.mount
  1411ms dev-mqueue.mount
  1371ms sys-kernel-debug.mount

and a proper boot chart

bootchart w/o network.service

It shows us an another botleneck, which is the systemd-vconsole-setup.service, because it delay the sysinit.target, which is very early boot stage. In case like this, we can only use strace to know, what is taking too long. And debugging is pretty straightforward in systemd world. All we have to do is copy service file to /etc/systemd/system and change the ExecStart

ExecStart=/usr/bin/strace -f -tt -o /run/%N.strace /lib/systemd/systemd-vconsole-setup

and reboot. Then you will find the output in /run/systemd-vconsole-setup.strace with a timestamps. Looking there it’s obvious calling hwinfo --bios is extremely expensive in this stage. You can speedup the unit by setting the KBD_NUMLOCK to yes or no in /etc/sysconfig/keyboard, or you can try to mask it completely I did.

The next service needs to closer look was system-modules-load – then strace says that it spent 2(!) in init_module() for module microcode. I disabled it as well, even for CPUs needs it can’t be recommended.

Native systemd units

There is one tiny init script called purge-kernels, which starts for 300ms according blame. And in this particular case systemd alternative will be way more effective

$ cat /etc/systemd/system/purge-kernels.service
[Unit]
Description=Purge old kernels
After=local_fs.target
ConditionPathExists=/boot/do_pure_kernels

[Service]
Type=oneshot
ExecStart=/sbin/purge-kernels

because systemd only do one stat on the file and do not run it at all, so this service disappears from the blame at all.

The kernel time

There is one interesting thing about kernel time – 8 seconds spent there seems to be a lot to me. Simple ls on /boot gave me a pointer

$ ls -lh /boot/vmlinuz-* /boot/initrd-*
-rw-r--r-- 1 root root  14M Jul 24 11:03 /boot/initrd-3.4.4-1.1-desktop
-rw-r--r-- 1 root root 4.7M Jul 10 15:48 /boot/vmlinuz-3.4.4-1.1-desktop

The initrd is huge, around three times bigger than kernel? So let’s try to find what caused that. Every package can add it’s own setup script into /lib/mkinitrd/scripts/ thus let ask rpm whose did that

$ rpm -qf /lib/mkinitrd/scripts/setup-* | sort -u
cifs-utils-5.5-2.2.2.i586
cryptsetup-1.4.2-3.2.1.i586
device-mapper-1.02.63-26.1.1.i586
dmraid-1.0.0.rc16-18.2.1.i586
kpartx-0.4.9-3.1.1.i586
lvm2-2.02.84-26.1.1.i586
mdadm-3.2.5-3.3.2.i586
mkinitrd-2.7.0-62.2.1.i586
multipath-tools-0.4.9-3.1.1.i586
plymouth-scripts-0.8.5.1-1.3.1.noarch
splashy-0.3.13-35.1.1.i586

So I went through a list and try to uninstall things I do not need

  • cifs-utils – if you do not have any windows disc to mount, you can remove, but no impact on initrd size
  • cryptsetup – this is a popular service for laptops, but I do not have any luks device, so let skip that. It removes a half of Yast as well, so I saved 18M of space, but a little in initrd.
  • device-mapper, dmraid, kpartx and lvm2 – cannot be easily removed as too much low-level stuff depends on it
  • mdadm – no linux md devides, skip that
  • mkinitrd – removal can reduce initrd to zero, but we would need own kernel
  • multipath-tools – no multipath device, let skip that

  • plymouth-scripts – who would need the “fancy” boot when booting so fast? – reducing initrd to 8.9M
  • splashy – the same – and reducing initrd to 6.6M

So the things intended to provide fancy boot actually bloats the system. Let’s measure the impact of those changes

$ systemd-analyze
2781ms (kernel) + 4999ms (userspace) = 7780ms

bootchart w/o network.service

And that’s all folks …?

There are a lot of factors slowing our boot – reducing it to 8 seconds is not that bad. One have to go carefully through blame and plot output to see what delays his computer in start. I would say making NetworkManager default one at least when installing laptop pattern would be nice and simple change as well as continue on “systemdifization” of openSUSE.

There are few other tricks, which get us closer to the target time, but I’ll post them next day.

Both comments and pings are currently closed.

12 Responses to “Optimizing a boot time, aka 2 second boot”

  1. Andreas Jaeger

    Great! Is there a way to do some of this in an automatic way? E.g. tell mkinitrd to only include cifs-utils if it is needed?

    It would be interesting how dracut handles this.

    • Michal Vyskocil

      @Andreas Jaeger: sure, I tested dracut as well. Frederic pointed me to some cool switches reduced a kernel time a bit.

      @Wolfgang: the trick is not run NM through /etc/init.d/network, but mask it and use native NetworkManager.service. Then no dependencies are included.

      @zoehneto: sure, I sent a patch to maintainer

      @Petr Uzel: unfortunately I did rpm -e –force. The workaround would be split mkinitrd setup macros to suspend-mkinitrd with Supplements: packageand(suspend:mkinitrd) -will send a request.

      @jospoortvliet: I see three things to do – 1.) purge-kernels (and continue with systemdification), 2.) enable NM.service with laptop pattern, 3.) split the suspend mkinitrd files to enable simple removal – or adapt dracut.

  2. Wolfgang

    So NetworkManager is finally working if network-remotefs is disabled? Nice, I haven’t tried that since 11.3 where it was a requirement to have network-remotefs enabled to be able to start a locally installed NetworkManager -> https://bugzilla.novell.com/show_bug.cgi?id=633923

  3. zoehneto

    Thanks for the great post. Could you please make sure that your purge-kernel.service file is integrated into the appropriate package so all openSUSE users can benefit from it.

  4. Petr Uzel

    Great post, thanks.

    +1 on pushing your purge-kernel.service to 12.2

    One more additional question: how to remove splashy without breaking suspend? Suspend seems to depend on it. TIA

    • Yeah, I bumped into the same. I’m OK with dumping any fancy boot if we can boot all openSUSE systems in 7 seconds, but then suspend should of course work…

      • Bruno Friedmann

        Well with plymouth entering the danse in 12.2, did we really need that splashy things ?
        Okay plymouth and branding are heavy on initrd cause we need all the graphics present before you unlock the crypted disk on you laptop (cause you have it, otherwise its no more your data :-)
        Should we adapt a minimalistic text prompt and sacrifice the beauty?
        Can we have both alternative, a branding quick boot and a branding nice my life?
        Both of that look like nice pretendant for a bof at Summit and OSC no ?

  5. BTW where’s the second blog :D

    I’m quite excited about these improvements, hope some of them can somehow make it into 12.2 or at least 12.3…

  6. SUSE

    2 second boot is great,

    but what about a 2 second startup of firefox, which is achievable in other distro but averaged 30 seconds in openSUSE.

  7. philomath

    Is it safe to just mask vconsole.service?

  8. tomtomme

    nice after switching to Network manager bootup is also twice as fast for me – but why is ifup so slow?

    systemd-analyze blame | head
    1859ms nfsserver.service
    1473ms SuSEfirewall2_setup.service
    811ms SuSEfirewall2_init.service
    569ms network-remotefs.service
    564ms xdm.service
    488ms systemd-logind.service
    430ms nmb.service
    328ms syslog.service
    302ms NetworkManager.service
    250ms nfs.service

    Startup finished in 3825ms (kernel) + 15064ms (userspace) = 18889ms

  9. Thomas Langkamp

    On another machine I wait very long for nfsserver to start (its a fast system with an SSD). This is when no nfs-client is online, and also when one is online, but I have configured them to not auto-mount. So there is no activity required of NFS at boot. Would it be possible to delay the start of the nfsserver.service to say 1 minute after the boot finished? How?

    systemd-analyze blame | head
    7136ms syslog.service
    7127ms systemd-logind.service
    5871ms nfsserver.service
    805ms md.service