Home Home
Sign up | Login

Deprecation notice: openSUSE Lizards user blog platform is deprecated, and will remain read only for the time being. Learn more...

Optimizing a boot time, aka 2 second boot

July 26th, 2012 by

During the hackweek, I have decided to take a look onto a boot process to realize, how fast can be boot of the system. This metric is considered as a not important, especially in a geek community. But I am sure that this is an important part of user experience. Following text is mean to share my investigations with you – but you should be aware what you are doing, since I have been focused on reducing the boot time as much as possible.

A bit of theory

Basically you have three possibilities how to make your boot fast.

  1. Start things in a parallel.
  2. Start things on demand.
  3. Start less things.

… root of all evil …

“Premature optimization is the root of all evil.”

And I will modestly extend Donald Knuth’s sentence, that the blind optimization as well. So if we have to optimize something, we have to know what to do. Fortunately systemd comes with an excellent tool called systemd-analyze, which show us our boot in several ways.

The simple run of command prints the time we spent in a boot and in which phase.

# systemd-analyze
Startup finished in 8480ms (kernel) + 30873ms (userspace) = 39353ms

That was the default (minimal X system) 12.2 installation on my EEE 701 netbook, which is probably not suitable to work as nowadays cellsmarphone, because is pathetically slow. On the other hand is it a perfect playground, so let’s continue with an investigating.

The overall time is nice, but won’t help to know what’s going on. There are two more subcommands, blame and plot shows us more information about the boot. The first shows the services sorted by the start time. The ones boots so long are those we should kicked off as a first ones.

Let see what slow the boot down a most

$ systemd-analyze blame | head
 11385ms network.service
  5664ms SuSEfirewall2_init.service
  5575ms systemd-vconsole-setup.service
  3032ms ntp.service
  2840ms remount-rootfs.service
  2230ms postfix.service
  2021ms network-remotefs.service
  1925ms cpufreq.service
  1661ms SuSEfirewall2_setup.service
  1506ms xdm.service

And take look at the output of systemd-analyze plot command

openSUSE 12.2 boot diagram

You can see, that there is a long chain of SuSEfirewall2_init -> network -> network-remotefs -> SuSEfirewall2_setup tooks several dozen seconds to be finished. And nothing is wrong with that, but that is the server solution, not what I want to have on my tiny laptop.

Making a laptop boot twice more faster

So having the complex dependencies of several services in mind, I decided to mask some of them. Masking in systemd world means the service cannot be started using systemd, so it becomes invisible for it. I masked those

  • network.service – will be replaced by NetworkManager, which is more suitable for laptops usage
  • SuSEfirewall2_init and SuSEfirewall2_setup – even if it’s a security feature, a risc for laptop, which is mostly offline and running only sshd is pretty small.
  • ntp.service, network-remotefs.service – those does not makes a sense on my laptop
  • postfix.service – I do not want to send emails via /usr/bin/sendmail
  • cpufreq.service – it is even not supported by my CPU (grep rc.cpufreq /var/log/messages)

Do not forget to install NetworkManager and the applet and change the /etc/sysconfig/network/config and reboot.

Now we have

$ systemd-analyze
Startup finished in 8528ms (kernel) + 11123ms (userspace) = 19652ms

Using an strace with systemd

Now we have a list of worse services

$ systemd-analyze blame | head -n 10
  5476ms xdm.service
  4172ms systemd-vconsole-setup.service
  3950ms systemd-modules-load.service
  2781ms remount-rootfs.service
  1848ms NetworkManager.service
  1439ms media.mount
  1426ms systemd-remount-api-vfs.service
  1419ms dev-hugepages.mount
  1411ms dev-mqueue.mount
  1371ms sys-kernel-debug.mount

and a proper boot chart

bootchart w/o network.service

It shows us an another botleneck, which is the systemd-vconsole-setup.service, because it delay the sysinit.target, which is very early boot stage. In case like this, we can only use strace to know, what is taking too long. And debugging is pretty straightforward in systemd world. All we have to do is copy service file to /etc/systemd/system and change the ExecStart

ExecStart=/usr/bin/strace -f -tt -o /run/%N.strace /lib/systemd/systemd-vconsole-setup

and reboot. Then you will find the output in /run/systemd-vconsole-setup.strace with a timestamps. Looking there it’s obvious calling hwinfo --bios is extremely expensive in this stage. You can speedup the unit by setting the KBD_NUMLOCK to yes or no in /etc/sysconfig/keyboard, or you can try to mask it completely I did.

The next service needs to closer look was system-modules-load – then strace says that it spent 2(!) in init_module() for module microcode. I disabled it as well, even for CPUs needs it can’t be recommended.

Native systemd units

There is one tiny init script called purge-kernels, which starts for 300ms according blame. And in this particular case systemd alternative will be way more effective

$ cat /etc/systemd/system/purge-kernels.service
[Unit]
Description=Purge old kernels
After=local_fs.target
ConditionPathExists=/boot/do_pure_kernels

[Service]
Type=oneshot
ExecStart=/sbin/purge-kernels

because systemd only do one stat on the file and do not run it at all, so this service disappears from the blame at all.

The kernel time

There is one interesting thing about kernel time – 8 seconds spent there seems to be a lot to me. Simple ls on /boot gave me a pointer

$ ls -lh /boot/vmlinuz-* /boot/initrd-*
-rw-r--r-- 1 root root  14M Jul 24 11:03 /boot/initrd-3.4.4-1.1-desktop
-rw-r--r-- 1 root root 4.7M Jul 10 15:48 /boot/vmlinuz-3.4.4-1.1-desktop

The initrd is huge, around three times bigger than kernel? So let’s try to find what caused that. Every package can add it’s own setup script into /lib/mkinitrd/scripts/ thus let ask rpm whose did that

$ rpm -qf /lib/mkinitrd/scripts/setup-* | sort -u
cifs-utils-5.5-2.2.2.i586
cryptsetup-1.4.2-3.2.1.i586
device-mapper-1.02.63-26.1.1.i586
dmraid-1.0.0.rc16-18.2.1.i586
kpartx-0.4.9-3.1.1.i586
lvm2-2.02.84-26.1.1.i586
mdadm-3.2.5-3.3.2.i586
mkinitrd-2.7.0-62.2.1.i586
multipath-tools-0.4.9-3.1.1.i586
plymouth-scripts-0.8.5.1-1.3.1.noarch
splashy-0.3.13-35.1.1.i586

So I went through a list and try to uninstall things I do not need

  • cifs-utils – if you do not have any windows disc to mount, you can remove, but no impact on initrd size
  • cryptsetup – this is a popular service for laptops, but I do not have any luks device, so let skip that. It removes a half of Yast as well, so I saved 18M of space, but a little in initrd.
  • device-mapper, dmraid, kpartx and lvm2 – cannot be easily removed as too much low-level stuff depends on it
  • mdadm – no linux md devides, skip that
  • mkinitrd – removal can reduce initrd to zero, but we would need own kernel

    • multipath-tools – no multipath device, let skip that

  • plymouth-scripts – who would need the “fancy” boot when booting so fast? – reducing initrd to 8.9M
  • splashy – the same – and reducing initrd to 6.6M

So the things intended to provide fancy boot actually bloats the system. Let’s measure the impact of those changes

$ systemd-analyze
2781ms (kernel) + 4999ms (userspace) = 7780ms

bootchart w/o network.service

And that’s all folks …?

There are a lot of factors slowing our boot – reducing it to 8 seconds is not that bad. One have to go carefully through blame and plot output to see what delays his computer in start. I would say making NetworkManager default one at least when installing laptop pattern would be nice and simple change as well as continue on “systemdifization” of openSUSE.

There are few other tricks, which get us closer to the target time, but I’ll post them next day.

Tags: , ,
Posted in Base System, Hackweek, Usability | 12 Comments »

Snapper and LVM thin-provisioned Snapshots

July 25th, 2012 by

SUSEs Hackweek 8 allowed me to implement support for LVM thin-provisioned snapshots in snapper. Since thin-provisioned snapshots themself are new I will shortly show their usage.

Unfortunately openSUSE 12.2 RC1 does not include LVM tools with thin-provisioning so you have to compile them on your own. First install the thin-provisioning-tools. Then install LVM with thin-provisioning enabled (configure option –with-thin=internal).

To setup LVM we first have to create a volume group either using the LVM tools or YaST. I assume it’s named test. Then we create a storage pool with 3GB space.

  # modprobe dm-thin-pool
  # lvcreate --thin test/pool --size 3G

Now we can create a thin-provisioned logical volume named thin with a size of 5GB. The size can be larger than the pool since data is only allocated from the pool when needed.

  # lvcreate --thin test/pool --virtualsize 5G --name thin

  # mkfs.ext4 /dev/test/thin
  # mkdir /thin
  # mount /dev/test/thin /thin

Finally we can create a snapshot from the logical volume.

  # lvcreate --snapshot --name thin-snap1 /dev/test/thin

  # mkdir /thin-snapshot
  # mount /dev/test/thin-snap1 /thin-snapshot

Space for the snapshot is also allocated from the pool when needed. The command lvs gives an overview of the allocated space.

  # lvs
  LV         VG   Attr     LSize Pool Origin Data%  Move Log Copy%  Convert
  pool       test twi-a-tz 3.00g               4.24
  thin       test Vwi-aotz 5.00g pool          2.54
  thin-snap1 test Vwi-a-tz 5.00g pool thin     2.54

After installing snapper version 0.0.12 or later we can create a config for the logical volume thin.

  # snapper -c thin create-config --fstype="lvm(ext4)" /thin

As a simple test we can create a new file and see that snapper detects its creation.

  # snapper -c thin create --command "touch /thin/lenny"

  # snapper -c thin list
  Type   | # | Pre # | Date                          | Cleanup | Description | Userdata
  -------+---+-------+-------------------------------+---------+-------------+---------
  single | 0 |       |                               |         | current     |
  pre    | 1 |       | Tue 24 Jul 2012 15:49:51 CEST |         |             |
  post   | 2 | 1     | Tue 24 Jul 2012 15:49:51 CEST |         |             |

  # snapper -c thin status 1..2
  +... /thin/lenny

So now you can use snapper even if you don’t trust btrfs. Feedback is welcomed.

Tags: ,
Posted in Base System, Hackweek, Systems Management | Comments Off on Snapper and LVM thin-provisioned Snapshots

[gsoc] osc2 client – summary of week 9

July 23rd, 2012 by

Hi,

here’s a small summary of the 9th (coding) week. Last week I worked
on the fetcher and cache manager code. In order to support all
features some of the existing classes had to be enhanced with some
more parameters.

Done:

  • cache manager code
  • BinaryList class supports view=cpio
  • RORemoteFile class supports lazy_open
    parameter (by default the file is opened lazily that is when a
    read request is issued) (for the fetcher code we use
    lazy_open=False)
  • minor changes in the httprequest module (AbstractHTTPRequest
    supports the same query parameter more than once)

    • The fetcher code is more or less done (not yet committed) and will be
    finished by friday evening (I’ve some exams this week…).

    Marcus

Posted in Build Service, GSOC | Comments Off on [gsoc] osc2 client – summary of week 9

We can do better

July 20th, 2012 by

Maybe it is just me, but lately it appears that there is a lot brewing on our lists. Generally I try to stay out of the fray, but of course we, as members of our community, are all in the middle of it in one way or another. With a very recent endless thread on the opensuse list fresh in memory, not that I read all or even most of the messages it generated, and the follow on thread of the original poster bidding his farewell to the list, supposedly because the poster didn’t like the responses, I feel compelled to share some of my own thoughts on the topic.

My feeling is that a good number of people that complain about the noise on our lists are also those that contribute to that noise at a good rate. Thus, I can only say that sometimes it is nice to exercise some restraint and not hit that “Send” button; who hasn’t had the “I shouldn’t have sent this” thought? The bottom line is, that it is almost impossible to write anything that does not step on somebody’s toes somewhere along the way and if everyone that feels the least bit uneasy about some comment would respond all the time we would really be in an endless loop. I am certain, this post will make someone uneasy, upset, angry or worse. If that’s the way you feel right now, I am sorry, I am not trying to make you angry or upset on purpose. Please accept my apology.

That said, who hasn’t been frustrated out of their mind by some perceived dumb software problem or other issue? Even worse when it is our own hurdle we cannot cross. In the end we just wanted to yell and scream and the result is too often a message with: pick your “Emotional state appropriate inflamatory subject…” and rant your question to the list. Everyone gets emotional, and frustration happens to be a very strong reaction. However, the question that should come to mind before hitting that “Send” button to a list of volunteers is this, “Will an emotional reaction with moaning, groaning, whining, and complaining, that hides the real problem, give me the feedback needed to resolve my issue?” The answer is simple; No it will not. A post charged with negative energy will, and there is plenty of proof on our lists, solicit emotional, mostly negative, responses. These do not contribute to resolving the problem at hand. However, once this storm is set in motion there is, for better or worse no stopping it and it just has to run its course, i.e. eventually people will be tired of feeding the storm that should have never happened and things will go back to “normal”, whatever that may be.

I believe that the people, volunteers, on the opensuse mailing lists are generally willing to help solve problems others encounter. Yes, there will be the occasional cynical remark here and there, but I do not believe these remarks are ushered in a mean spirited way, and in the end we do not really have to nit-pick everything to death. It is one thing to yell at someone because the product or service you purchased from them does not meet your expectations, it is another thing to go bananas on a list where answers are provided by people with generally good intentions that volunteer their time. As posters of questions we all have a responsibility to keep this in mind before we go down the emotionally charged road to the abyss of not getting our questions answered.

However, putting the onus completely on the question poster is a bit too easy, isn’t it. While we as helpers would all love to get the “perfect” problem description, that is completely factual and contains no mistakes in description and actions to reproduce the problem, we have to realize that this is just not going to happen. By the time an issue hits the list the person posting probably is charged up in one way or another and ready to get rid of some of that stored energy, some more than others. Thus, as helpers we have to develop a bit more tolerance and let things roll off our backs a bit more. As potential helper we can just ignore the ranting posters. If you have it within you to provide the answer to the hidden problem and can rise above the fray, fantastic! help and do a good deed. However, if you know the answer to the hidden problem but feel the urge to feed the emotional storm it may be best to just ignore the message and not respond. In the end a raging answer hides the kind deed of help just as the raging question hides the problem.

No we do not have to have boring and no fun lists, but in the end flame wars or endless bickering threads are not fun and having people leave the lists or even not using openSUSE because of silly things is just not helpful to anyone. If we as posters, seekers of answers and providers thereof can just tone it down a bit things will probably work better for everyone.

Tags: ,
Posted in Uncategorized | 2 Comments »

How to peek into remote isos

July 17th, 2012 by

When people want to provide a collection of files, they sometimes choose to do so by providing a .iso image file. But if you only want to look what files are in there or only need a few files, e.g. kernel and initrd for PXE-booting, you still had to download the whole thing to loop-mount it.

But you don’t have to anymore. Because modern web servers support delivering only parts of a file (using the “Range” header field), that allowed me to implement curlwwwfs that mounts remote HTTP directories into your local filesystem. And then you can use fuseiso on top to access the actual content within the .iso. All without root access.

This is how it works:

First you have to install the required packages (replace 12.2 with your version of openSUSE (or if you use a different Linux distribution, do git clone git://github.com/bmwiedemann/curlwwwfs and “make install” in there)):

zypper ar http://download.opensuse.org/repositories/home:/bmwiedemann/openSUSE_12.2/home:bmwiedemann.repo

zypper in curlwwwfs fuseiso

# Then you start it:

mkdir mnthttp mntiso

curlwwwfs http://zq1.de/bootcd mnthttp &

ls -la mnthttp/

fuseiso mnthttp/bmwinux-8.2-040808.iso mntiso

cat mntiso/isolinux/isolinux.cfg

# and later you clean it up with

fusermount -u mntiso

fusermount -u mnthttp

Tags:
Posted in Apache, Network | 1 Comment »

[gsoc] osc2 client – summary of week 8

July 16th, 2012 by

Hi,

here’s a small summary of the 8th (coding) week. The last days I
was working on getting build support into the osc2 library. I just
checked in a BuildInfo and Builder class (+ testcases). Here’s a
brief example how they can be used:

# example how to use the Builder class
builder = Builder(su_cmd=Builder.SUDO, root='/var/tmp/build-root')
builder.rpmlist = '/path/to/rpmlist'
builder.dist = '/path/to/buildconfig'
builder.without = 'feature1'
builder.without += 'feature2'
builder.arch = 'x86_64'
# run the build:
builder.run('/path/to/osc.spec')
# this executes:
sudo /usr/bin/build --arch x86_64 --dist /path/to/buildconfig \
   --root /var/tmp/build-root --rpmlist /path/to/rpmlist \
   --without feature1 --without feature2 /path/to/osc.spec

Basically Builder is just a wrapper around the build script
(all options will be passed to the build script; if an option contains
a “-” character like “vm-type” it can be set like this:
“binfo.vm_type = ‘xen'” (a “_” will be replaced with “-“)).

Additionally here’s a small example how to utilize the BuildInfo
class:

# fname is a path to a spec file
binfo = BuildInfo('openSUSE:Tools', 'osc', 'openSUSE_Factory', 'x86_64',
                  filename=fname)
# print preinstall packages
for bdep in binfo.preinstall():
    print bdep.get('name'), bdep.get('version')
# save binfo xml in a file
binfo.write_to('/path/to/file.xml')

The next thing on my TODO is the fetcher and cachemanager code.

Marcus

Posted in Build Service, GSOC | Comments Off on [gsoc] osc2 client – summary of week 8

Linux Kernel built with clang boots into openSUSE

July 14th, 2012 by

Whatch this:  http://youtu.be/Jp04DMXH2Rg

The kernel was compiled with Clang (C frontend of LLVM)  and boots into a running openSUSE desktop!

 

Posted in Architectures, Base System, Kernel, Toolchain | Comments Off on Linux Kernel built with clang boots into openSUSE

[gsoc] osc2 client – summary of week 7

July 11th, 2012 by

Hi,

here’s a small summary of the 7th (coding) week. Last week
I was really busy with university stuff (it was the second
last week in the lecture period so I had to recap quite
some stuff) and didn’t manage to work much on the GSoC project.
I’m going work off the TODO this weekend.

Marcus

Posted in Build Service, GSOC | Comments Off on [gsoc] osc2 client – summary of week 7

[gsoc] osc2 client – summary of week 6

July 3rd, 2012 by

Hi

here’s a small summary of the 6th (coding) week. Last week I
continued working on the build module and developed a concept
for the package fetcher (and discussed some parts
with darix:) ).
The main idea is to modularize the fetcher code, verify code etc.
Thus we have a fetcher class which takes care of retrieving the
packages. The fetcher class utilizes a “cache manager” which takes
care of storing the fetched packages on the filesystem. The goal is
that at some point in time one can exchange the “simple” cache manager
with a more “clever” cache manager (which for instance cleans up the
cache from time to time or only allows exactly one version of a
package in the cache etc.). In order to achieve this no code in the
fetcher has to be touched – instead it’s sufficient to pass a different
cache manager object to the fetcher.
Additionally the fetcher provides some hooks like pre, pre_fetch,
post_fetch and post. For instance a post_fetch hook can be used to
verify the just fetched package etc.

TODO:
– write testcases and implement the concept from above

If you have questions, suggestions etc. feel free to contact me:)

Marcus

Posted in Build Service, GSOC | Comments Off on [gsoc] osc2 client – summary of week 6

Repository GNOME:Contrib is dead

June 29th, 2012 by

An announcement for GNOME users in openSUSE: the repository GNOME:Contrib is now dead. This used to be the development branch of GNOME packages living in Contrib. Packages previously in this repository have all been pushed to Factory. If this is in the list of your subscribed repositories, please remove it now (using zypper rr <reponame> or from YaST, etc.), because the repository itself will be deleted from the download.opensuse.org server shortly.

Posted in Contrib, GNOME | Comments Off on Repository GNOME:Contrib is dead

« Older Entries
Newer Entries »