Home Home > Tag > Hackweek
Sign up | Login

Posts Tagged ‘Hackweek’

Optimizing a boot time, aka 2 second boot

July 26th, 2012 by

During the hackweek, I have decided to take a look onto a boot process to realize, how fast can be boot of the system. This metric is considered as a not important, especially in a geek community. But I am sure that this is an important part of user experience. Following text is mean to share my investigations with you – but you should be aware what you are doing, since I have been focused on reducing the boot time as much as possible.

A bit of theory

Basically you have three possibilities how to make your boot fast.

  1. Start things in a parallel.
  2. Start things on demand.
  3. Start less things.

… root of all evil …

“Premature optimization is the root of all evil.”

And I will modestly extend Donald Knuth’s sentence, that the blind optimization as well. So if we have to optimize something, we have to know what to do. Fortunately systemd comes with an excellent tool called systemd-analyze, which show us our boot in several ways.

The simple run of command prints the time we spent in a boot and in which phase.

# systemd-analyze
Startup finished in 8480ms (kernel) + 30873ms (userspace) = 39353ms

That was the default (minimal X system) 12.2 installation on my EEE 701 netbook, which is probably not suitable to work as nowadays cellsmarphone, because is pathetically slow. On the other hand is it a perfect playground, so let’s continue with an investigating.

The overall time is nice, but won’t help to know what’s going on. There are two more subcommands, blame and plot shows us more information about the boot. The first shows the services sorted by the start time. The ones boots so long are those we should kicked off as a first ones.

Let see what slow the boot down a most

$ systemd-analyze blame | head
 11385ms network.service
  5664ms SuSEfirewall2_init.service
  5575ms systemd-vconsole-setup.service
  3032ms ntp.service
  2840ms remount-rootfs.service
  2230ms postfix.service
  2021ms network-remotefs.service
  1925ms cpufreq.service
  1661ms SuSEfirewall2_setup.service
  1506ms xdm.service

And take look at the output of systemd-analyze plot command

openSUSE 12.2 boot diagram

You can see, that there is a long chain of SuSEfirewall2_init -> network -> network-remotefs -> SuSEfirewall2_setup tooks several dozen seconds to be finished. And nothing is wrong with that, but that is the server solution, not what I want to have on my tiny laptop.

Making a laptop boot twice more faster

So having the complex dependencies of several services in mind, I decided to mask some of them. Masking in systemd world means the service cannot be started using systemd, so it becomes invisible for it. I masked those

  • network.service – will be replaced by NetworkManager, which is more suitable for laptops usage
  • SuSEfirewall2_init and SuSEfirewall2_setup – even if it’s a security feature, a risc for laptop, which is mostly offline and running only sshd is pretty small.
  • ntp.service, network-remotefs.service – those does not makes a sense on my laptop
  • postfix.service – I do not want to send emails via /usr/bin/sendmail
  • cpufreq.service – it is even not supported by my CPU (grep rc.cpufreq /var/log/messages)

Do not forget to install NetworkManager and the applet and change the /etc/sysconfig/network/config and reboot.

Now we have

$ systemd-analyze
Startup finished in 8528ms (kernel) + 11123ms (userspace) = 19652ms

Using an strace with systemd

Now we have a list of worse services

$ systemd-analyze blame | head -n 10
  5476ms xdm.service
  4172ms systemd-vconsole-setup.service
  3950ms systemd-modules-load.service
  2781ms remount-rootfs.service
  1848ms NetworkManager.service
  1439ms media.mount
  1426ms systemd-remount-api-vfs.service
  1419ms dev-hugepages.mount
  1411ms dev-mqueue.mount
  1371ms sys-kernel-debug.mount

and a proper boot chart

bootchart w/o network.service

It shows us an another botleneck, which is the systemd-vconsole-setup.service, because it delay the sysinit.target, which is very early boot stage. In case like this, we can only use strace to know, what is taking too long. And debugging is pretty straightforward in systemd world. All we have to do is copy service file to /etc/systemd/system and change the ExecStart

ExecStart=/usr/bin/strace -f -tt -o /run/%N.strace /lib/systemd/systemd-vconsole-setup

and reboot. Then you will find the output in /run/systemd-vconsole-setup.strace with a timestamps. Looking there it’s obvious calling hwinfo --bios is extremely expensive in this stage. You can speedup the unit by setting the KBD_NUMLOCK to yes or no in /etc/sysconfig/keyboard, or you can try to mask it completely I did.

The next service needs to closer look was system-modules-load – then strace says that it spent 2(!) in init_module() for module microcode. I disabled it as well, even for CPUs needs it can’t be recommended.

Native systemd units

There is one tiny init script called purge-kernels, which starts for 300ms according blame. And in this particular case systemd alternative will be way more effective

$ cat /etc/systemd/system/purge-kernels.service
[Unit]
Description=Purge old kernels
After=local_fs.target
ConditionPathExists=/boot/do_pure_kernels

[Service]
Type=oneshot
ExecStart=/sbin/purge-kernels

because systemd only do one stat on the file and do not run it at all, so this service disappears from the blame at all.

The kernel time

There is one interesting thing about kernel time – 8 seconds spent there seems to be a lot to me. Simple ls on /boot gave me a pointer

$ ls -lh /boot/vmlinuz-* /boot/initrd-*
-rw-r--r-- 1 root root  14M Jul 24 11:03 /boot/initrd-3.4.4-1.1-desktop
-rw-r--r-- 1 root root 4.7M Jul 10 15:48 /boot/vmlinuz-3.4.4-1.1-desktop

The initrd is huge, around three times bigger than kernel? So let’s try to find what caused that. Every package can add it’s own setup script into /lib/mkinitrd/scripts/ thus let ask rpm whose did that

$ rpm -qf /lib/mkinitrd/scripts/setup-* | sort -u
cifs-utils-5.5-2.2.2.i586
cryptsetup-1.4.2-3.2.1.i586
device-mapper-1.02.63-26.1.1.i586
dmraid-1.0.0.rc16-18.2.1.i586
kpartx-0.4.9-3.1.1.i586
lvm2-2.02.84-26.1.1.i586
mdadm-3.2.5-3.3.2.i586
mkinitrd-2.7.0-62.2.1.i586
multipath-tools-0.4.9-3.1.1.i586
plymouth-scripts-0.8.5.1-1.3.1.noarch
splashy-0.3.13-35.1.1.i586

So I went through a list and try to uninstall things I do not need

  • cifs-utils – if you do not have any windows disc to mount, you can remove, but no impact on initrd size
  • cryptsetup – this is a popular service for laptops, but I do not have any luks device, so let skip that. It removes a half of Yast as well, so I saved 18M of space, but a little in initrd.
  • device-mapper, dmraid, kpartx and lvm2 – cannot be easily removed as too much low-level stuff depends on it
  • mdadm – no linux md devides, skip that
  • mkinitrd – removal can reduce initrd to zero, but we would need own kernel
  • multipath-tools – no multipath device, let skip that

  • plymouth-scripts – who would need the “fancy” boot when booting so fast? – reducing initrd to 8.9M
  • splashy – the same – and reducing initrd to 6.6M

So the things intended to provide fancy boot actually bloats the system. Let’s measure the impact of those changes

$ systemd-analyze
2781ms (kernel) + 4999ms (userspace) = 7780ms

bootchart w/o network.service

And that’s all folks …?

There are a lot of factors slowing our boot – reducing it to 8 seconds is not that bad. One have to go carefully through blame and plot output to see what delays his computer in start. I would say making NetworkManager default one at least when installing laptop pattern would be nice and simple change as well as continue on “systemdifization” of openSUSE.

There are few other tricks, which get us closer to the target time, but I’ll post them next day.

Boo

January 28th, 2011 by

Arts, buffer, check, clutter, cobbler, colorblind, concurrent, convert, cook, crash, dialog, dump, expect, file, folks, fortune, genius, global, hello, indent, less, links, meanwhile, mirror, screen, sparse, suck, tree, units, words. What do these ordinary English words have in common? They are also names of software projects, which becomes a problem if you want to recognize package names in text. I understand that in the old days, the name of a command or application was only relevant in the context of the computer it ran on, and file names had to be short. Some of these names have allowed for a variety of jokes. But why, in the age of portable programs, WWW and search engines, can’t people come up with less ambiguous names? I mean, it’s not hard to join two words, or, at a minimum, prefix a word with a vowel, like, uhm, a round fruit does. :)

Oh, and did I mention that we have over 160 packages with a 2-3 letter name? The one mentioned in the title is a programming language, btw.

Hackweek V: Local caching for CIFS network file system

June 14th, 2010 by

Hackweek

It’s that time of the year when SUSE/Novell developers use their Innovation Time-off to do a project of their interest/wish – called as Hackweek. Last week was Hackweek V. I worked on making the Common Internet File System (CIFS) cache aware, i.e. local caching for CIFS Network File System.

Linux FS-Cache

Caching can result in performance improvements in network filesystems where access to network and media is slow. The cache can indirectly improve performance of the network and the server by reduced network calls. Caching can be also viewed as a preparatory work for making disconnected operation (Offline) work with network filesystems.

The Linux Kernel recently added a generic caching facility (FS-Cache) that any network filesystem like NFS or CIFS or other service can use to cache data locally. FS-Cache supports a variety of cache backends i.e. different types of cache that have different trade-offs (like CacheFiles, CacheFS etc.) FS-Cache mediates between cache backends and the network filesystems. Some of the network filesystems such as NFS and AFS are already integrated with FS-Cache.

Making CIFS FS-Cache capable

To make any network filesystem FS-Cache aware, there are a few things to consider. Let’s consider them step by step (though not in detail):

* First, we need to define the network filesystem and it should be able to register/unregister with the FS-Cache interface.
* The network filesystem has to define the index hierarchy which could be used to locate a file object or discard a certain subset of all the files cached.
* We need to define the objects and the methods associated.
* All the indices in the index hierarchy and the data file need to be registered. This could be done by requesting a cookie for each index or data file. Upon successful registration, a corresponding cookie is returned.
* Functions to store and retrieve pages in the cache.
* Way to identify whether the cache for a file is valid or not.
* Function to release any in-memory representation for the network filesystem page.
* Way to invalidate a data file or index subtree and relinquish cookies.

Implementation

I wanted to get the prototype working within a week. So the way I have implemented it is rudimentary and has lot of room for improvement.

The index hierarchy is not very deep. It has three levels – Server, Share and Inode. The only way that I know of identifying files with CIFS is by ‘UniqueId’ which is supposed to be unique. However, some server do not ensure that the ‘UniqueId’ is always unique (for example when there is more than one filesystem in the exported share). The cache coherency is currently ensured by verifying the ‘LastWriteTime’ and size of the file. This is not a reliable way of detecting changes as some CIFS servers will not update the time until the filehandle is closed.

The rudimentary implementation is ready and the cumulative patch can be found here:

http://www.kernel.org/pub/linux/kernel/people/jays/patches/

[WARNING: The patch is lightly tested and of prototype quality.]

Here are some initial performance numbers with the patch:

Copying one big file of size ~150 MB.

$time cp /mnt/cifs/amuse.zip .
(Cache initialized)

real 1m18.603s
user 0m0.016s
sys 0m8.569s

$time cp /mnt/cifs/amuse.zip /
(Read from Cache)

real 0m28.055s
user 0m0.008s
sys 0m1.140s

Novell Hackweek Five

May 28th, 2010 by

Hackweek Five LogoI am really looking forward to the next Hackweek that we have in Novell – it will be in the week from 7-11 of June 2010.
In that week, Novell allows a whole lot of people to spend the full work time (and more ;-) to work on whatever free software they want. That is really a huge thing, because we’re talking about hundrets of engineers.
What everybody is working on is as said not at all prescribed, except that it should benefit the idea of free software. There is a list maintained of ideas which people have for Hackweek Five in order to find somebody joining the team or to pick the idea up at all.
The good thing now is that of course openFATE is used to maintain this list and thus it is open for the openSUSE Community to also add ideas, comment or vote on whats already there. This is of course no guarantee that the idea is going to be picked up but still. So everybody who thinks she has an idea that will inspire someone on Hackweek Five, feel free to add it to openFATE and talk about.
Of course it is also possible and appreciated to work on Hackweek projects also as non Novell employee :-)
Get in touch – it will be exciting!

HackWeek III Day 2 – Diary Of An Outsider

August 28th, 2008 by

So the fun and sun in Nurenberg continues :D

Again day 2 started off very much as day 1 – lots of discussions.  This is not a bad thing, and maybe Novell would consider sponsoring a ChatWeek (it could be condensed into 2 or 3 days), it is a fact that communication is the one major stumbling block that any company/community suffers from and Novell and openSUSE are no exception to this.

We started with general discussions and continuing some of the discussions we had over dinner last night.  After lunch we then continued our discussions on openSUSE and what/how things could be improved – highly productive and very inspiring to get internal and external community members to speak honestly and bluntly about their thoughts and ideas. 

Note to the community (yes, all of you!): If you have thoughts on what or how to help imporve anything within the openSUSE universe speak up!!  If you’re not sure who or how to do so, the key contact details (in no particular order) are Zonker, opensuse-project mailing list, #opensuse-project irc channel and of course the openSUSE Board.  One of the advantages of openSUSE is that you can pick things up and run with them, if you need any help just ask and people will rally around (sometimes it takes a bit of time but it does happen, honest).

After all the discussions I had just enough time to grab a drink before the first openSUSE Marketing Team irc meeting took place.  This was shorter than I would personally have like, but that was down to one very important reason – the HackWeek BBQ!

The BBQ was and is a great opportunity for people just to chat on all levels, and general have fun with some fine food and some fine Bavarian Beer – Pascal will certainly disagree with me on the last point, but then again he is kind of correct; being Belgian he is used to a much wider variety and quality of beers, just go to FOSDEM to find out ;-)  You can see a montage of the video that I took over here – yes I know it’s evil flash!

The party atmosphere went on long into the night and everyone managed to enjoy themselves. Thank you SUSE for the fine food and beer, and thank you Pascal for bringing some prized Belgian beers for comparison with the Bavarian ones.

Unifying Progress During Installation

August 27th, 2008 by

This might not be that useful, but it’s fun.

If you look at the openSUSE installer during the time it’s busy, you can see at least 4 types of progress bars shown:

  1. Disk preparation
  2. Image deployment
  3. Additional package installation
  4. Writing final configuration before reboot

My goal for this hackweek is to unify them as much as possible. So, now I have something to show. The prototype works on openSUSE 11.0 Alpha2 and unifies steps 1-3. Here is a screenshot:

Installer deploying images, switched to the details view

HackWeek III Day 1 – Diary Of An Outsider

August 26th, 2008 by

After my disasterous attempt to travel out to Nurenberg on Sunday – I managed to get my passport stolen at security of all places :( (I thought I had lost it, but nothing found yet).  I managed to travel out first thing on Monday.

Geekos and Tux

The timing was perfect, I managed to get to the SUSE offices in NUE just in time to drop my bags and meet up with Andreas for a little tour/familiarisation of the site.  This was great because we (openSUSE invited about 10 community members to NUE) got to see a load of the great folks that work on the project and see how they do it (including the ever elusive Beineri:

Stephan Binner

We then had a pleasant brunch, and afterwards sat down to start hacking on our respective projects.

So what is one doing for HackWeek? Well it isnt exactly hacking (well not initially although I may have to so I can get the end product out); I’m hoping to do some videos of HackWeek, both documentary style(ish) and some interviews.  Why do I say “hoping”? Well I’ve hardly filmed anything :( I have been way too busy discussing and speaking about openSUSE with people.

Day 1 was very much a ChatWeek moment rather than HackWeek :-)  The chatting took place all around the offices, and even in Meeting Rooms where we discussed many aspects of the project.  I continued chatting with fellow openSUSE peeps over dinner and well into the night:

Chatting over dinner

Chatting into the night

It isn’t all chatting though, some people did manage to get some coding done albeit in between chats :-)  By the way Andreas isn’t laughing at the code quality and Benji isn’t playing an advanced level of nethack – honest!

Actually hacking

I’m certainly hoping to be able to get some video recorded on Day 2, but these people at SUSE are just so damned social that it’s really hard!! :D