• You’re still carrying around your precious autoyast config files on an unencrypted usb stick?
• You have a customized installation disk that could reveal lots of personal details?
• You use ad blockers, private browser tabs, or even tor but still carry around your install or rescue disk unencrypted for everyone to see?
• You have your personal files and an openSUSE installation tree on the same partition just because you are lazy and can’t be bothered to tidy things up?
• A simple Linux install stick is just not geekish enough for you?

Not any longer!

mksusecd can now (well, once this pull request has been merged) create fully encrypted installation media (both UEFI and legacy BIOS bootable).

Everything (but the plain grub) is on a LUKS-encrypted partition. If you’re creating a customized boot image and add sensitive data via --boot or add an add-on repo or autoyast config or some secret driver update – this is all safe now!

You can get the latest mksusecd-1.54 already here to try it out! (Or visit software.opensuse.org and look for (at least) version 1.54 under ‘Show other versions’.

It’s as easy as

mksusecd --create crypto.img --crypto --password=xxx some_tumbleweed.iso

And then dd the image to your usb stick.

But if your Tumbleweed or SLE/Leap 15 install media are a bit old (well, as of now they are) check the ‘Crypto notes’ in mksusecd --help first! – You will need to add two extra options.

This is how the first screen looks then

I’m contacting you personally, as an openSUSE Board member.

You certainly already know that we want to have a kicking openSUSE conference next 1st-4th May 2015 at the Haag (NL).

Thus I’ve found that creating special workshop organized by development project could foster our beloved distribution.

oSC is the unique case in the year, where Geekos from all around the world meet together.
Let’s imagine you, meeting perhaps for the first time your fellows, having nice discussions, and hacking around the software you maintain.
There’s high level of chance to meet also your end users, and have constructive exchanges.

That’s why I invite you to propose a workshop directly to our event tools:
https://events.opensuse.org/conference/osc15/proposal

Having a workshop run like a mini-hack sprint, would help any of us in the distribution and the project.
Be it like learn people how to submit nice package, how to do maintenance, or how to do bug triage.

I feel confident that you will have the creative approach to resolve your own problematic.
The event place has small rooms for unattended sessions and they could be used to extend your workshop to get some more work done.

Some practical aspects:
oSC website : https://events.opensuse.org/conference/oSC15

And soon the travel support program for oSC 15 will be opened to handle your request about getting financial support for going to oSC.
https://connect.opensuse.org/travel-support/

If you have any questions, thoughts or ideas, don’t hesitate to ask on -project mailing list
or ping me by reply.

A final note about the why you should do it? Well beside being one of our “heroes” even if nobody need them
You and your co-maintainers will be able to explain your “job” on the project.
Don’t you want to inspire new comers, lead them directly to the right direction, and share the load.
Meeting you there, will also help our “marketing” force to light up a bit the work done in the shadow.

I really will appreciate your presence, afterwards, it’s you that create our distribution.
The time has come for you to be warmly thank.

I’m looking forward to see you there.

A bit of theory

Basically you have three possibilities how to make your boot fast.

1. Start things in a parallel.
2. Start things on demand.
3. Start less things.

… root of all evil …

“Premature optimization is the root of all evil.”

And I will modestly extend Donald Knuth’s sentence, that the blind optimization as well. So if we have to optimize something, we have to know what to do. Fortunately systemd comes with an excellent tool called systemd-analyze, which show us our boot in several ways.

The simple run of command prints the time we spent in a boot and in which phase.

# systemd-analyze
Startup finished in 8480ms (kernel) + 30873ms (userspace) = 39353ms

That was the default (minimal X system) 12.2 installation on my EEE 701 netbook, which is probably not suitable to work as nowadays cellsmarphone, because is pathetically slow. On the other hand is it a perfect playground, so let’s continue with an investigating.

The overall time is nice, but won’t help to know what’s going on. There are two more subcommands, blame and plot shows us more information about the boot. The first shows the services sorted by the start time. The ones boots so long are those we should kicked off as a first ones.

Let see what slow the boot down a most

$ systemd-analyze blame | head
 11385ms network.service
  5664ms SuSEfirewall2_init.service
  5575ms systemd-vconsole-setup.service
  3032ms ntp.service
  2840ms remount-rootfs.service
  2230ms postfix.service
  2021ms network-remotefs.service
  1925ms cpufreq.service
  1661ms SuSEfirewall2_setup.service
  1506ms xdm.service

And take look at the output of systemd-analyze plot command

You can see, that there is a long chain of SuSEfirewall2_init -> network -> network-remotefs -> SuSEfirewall2_setup tooks several dozen seconds to be finished. And nothing is wrong with that, but that is the server solution, not what I want to have on my tiny laptop.

Making a laptop boot twice more faster

So having the complex dependencies of several services in mind, I decided to mask some of them. Masking in systemd world means the service cannot be started using systemd, so it becomes invisible for it. I masked those

• network.service – will be replaced by NetworkManager, which is more suitable for laptops usage
• SuSEfirewall2_init and SuSEfirewall2_setup – even if it’s a security feature, a risc for laptop, which is mostly offline and running only sshd is pretty small.
• ntp.service, network-remotefs.service – those does not makes a sense on my laptop
• postfix.service – I do not want to send emails via /usr/bin/sendmail
• cpufreq.service – it is even not supported by my CPU (grep rc.cpufreq /var/log/messages)

Do not forget to install NetworkManager and the applet and change the /etc/sysconfig/network/config and reboot.

Now we have

$ systemd-analyze
Startup finished in 8528ms (kernel) + 11123ms (userspace) = 19652ms

Using an strace with systemd

Now we have a list of worse services

$ systemd-analyze blame | head -n 10
  5476ms xdm.service
  4172ms systemd-vconsole-setup.service
  3950ms systemd-modules-load.service
  2781ms remount-rootfs.service
  1848ms NetworkManager.service
  1439ms media.mount
  1426ms systemd-remount-api-vfs.service
  1419ms dev-hugepages.mount
  1411ms dev-mqueue.mount
  1371ms sys-kernel-debug.mount

and a proper boot chart

It shows us an another botleneck, which is the systemd-vconsole-setup.service, because it delay the sysinit.target, which is very early boot stage. In case like this, we can only use strace to know, what is taking too long. And debugging is pretty straightforward in systemd world. All we have to do is copy service file to /etc/systemd/system and change the ExecStart

ExecStart=/usr/bin/strace -f -tt -o /run/%N.strace /lib/systemd/systemd-vconsole-setup

and reboot. Then you will find the output in /run/systemd-vconsole-setup.strace with a timestamps. Looking there it’s obvious calling hwinfo --bios is extremely expensive in this stage. You can speedup the unit by setting the KBD_NUMLOCK to yes or no in /etc/sysconfig/keyboard, or you can try to mask it completely I did.

The next service needs to closer look was system-modules-load – then strace says that it spent 2(!) in init_module() for module microcode. I disabled it as well, even for CPUs needs it can’t be recommended.

Native systemd units

There is one tiny init script called purge-kernels, which starts for 300ms according blame. And in this particular case systemd alternative will be way more effective

$ cat /etc/systemd/system/purge-kernels.service
[Unit]
Description=Purge old kernels
After=local_fs.target
ConditionPathExists=/boot/do_pure_kernels

[Service]
Type=oneshot
ExecStart=/sbin/purge-kernels

because systemd only do one stat on the file and do not run it at all, so this service disappears from the blame at all.

The kernel time

There is one interesting thing about kernel time – 8 seconds spent there seems to be a lot to me. Simple ls on /boot gave me a pointer

$ ls -lh /boot/vmlinuz-* /boot/initrd-*
-rw-r--r-- 1 root root  14M Jul 24 11:03 /boot/initrd-3.4.4-1.1-desktop
-rw-r--r-- 1 root root 4.7M Jul 10 15:48 /boot/vmlinuz-3.4.4-1.1-desktop

The initrd is huge, around three times bigger than kernel? So let’s try to find what caused that. Every package can add it’s own setup script into /lib/mkinitrd/scripts/ thus let ask rpm whose did that

$ rpm -qf /lib/mkinitrd/scripts/setup-* | sort -u
cifs-utils-5.5-2.2.2.i586
cryptsetup-1.4.2-3.2.1.i586
device-mapper-1.02.63-26.1.1.i586
dmraid-1.0.0.rc16-18.2.1.i586
kpartx-0.4.9-3.1.1.i586
lvm2-2.02.84-26.1.1.i586
mdadm-3.2.5-3.3.2.i586
mkinitrd-2.7.0-62.2.1.i586
multipath-tools-0.4.9-3.1.1.i586
plymouth-scripts-0.8.5.1-1.3.1.noarch
splashy-0.3.13-35.1.1.i586

So I went through a list and try to uninstall things I do not need

• cifs-utils – if you do not have any windows disc to mount, you can remove, but no impact on initrd size
• cryptsetup – this is a popular service for laptops, but I do not have any luks device, so let skip that. It removes a half of Yast as well, so I saved 18M of space, but a little in initrd.
• device-mapper, dmraid, kpartx and lvm2 – cannot be easily removed as too much low-level stuff depends on it
• mdadm – no linux md devides, skip that
• mkinitrd – removal can reduce initrd to zero, but we would need own kernel

multipath-tools – no multipath device, let skip that

• plymouth-scripts – who would need the “fancy” boot when booting so fast? – reducing initrd to 8.9M
• splashy – the same – and reducing initrd to 6.6M

So the things intended to provide fancy boot actually bloats the system. Let’s measure the impact of those changes

$ systemd-analyze
2781ms (kernel) + 4999ms (userspace) = 7780ms

And that’s all folks …?

There are a lot of factors slowing our boot – reducing it to 8 seconds is not that bad. One have to go carefully through blame and plot output to see what delays his computer in start. I would say making NetworkManager default one at least when installing laptop pattern would be nice and simple change as well as continue on “systemdifization” of openSUSE.

There are few other tricks, which get us closer to the target time, but I’ll post them next day.

Oh, and did I mention that we have over 160 packages with a 2-3 letter name? The one mentioned in the title is a programming language, btw.

It’s that time of the year when SUSE/Novell developers use their Innovation Time-off to do a project of their interest/wish – called as Hackweek. Last week was Hackweek V. I worked on making the Common Internet File System (CIFS) cache aware, i.e. local caching for CIFS Network File System.

Linux FS-Cache

Caching can result in performance improvements in network filesystems where access to network and media is slow. The cache can indirectly improve performance of the network and the server by reduced network calls. Caching can be also viewed as a preparatory work for making disconnected operation (Offline) work with network filesystems.

The Linux Kernel recently added a generic caching facility (FS-Cache) that any network filesystem like NFS or CIFS or other service can use to cache data locally. FS-Cache supports a variety of cache backends i.e. different types of cache that have different trade-offs (like CacheFiles, CacheFS etc.) FS-Cache mediates between cache backends and the network filesystems. Some of the network filesystems such as NFS and AFS are already integrated with FS-Cache.

Making CIFS FS-Cache capable

To make any network filesystem FS-Cache aware, there are a few things to consider. Let’s consider them step by step (though not in detail):

* First, we need to define the network filesystem and it should be able to register/unregister with the FS-Cache interface.
* The network filesystem has to define the index hierarchy which could be used to locate a file object or discard a certain subset of all the files cached.
* We need to define the objects and the methods associated.
* All the indices in the index hierarchy and the data file need to be registered. This could be done by requesting a cookie for each index or data file. Upon successful registration, a corresponding cookie is returned.
* Functions to store and retrieve pages in the cache.
* Way to identify whether the cache for a file is valid or not.
* Function to release any in-memory representation for the network filesystem page.
* Way to invalidate a data file or index subtree and relinquish cookies.

Implementation

I wanted to get the prototype working within a week. So the way I have implemented it is rudimentary and has lot of room for improvement.

The index hierarchy is not very deep. It has three levels – Server, Share and Inode. The only way that I know of identifying files with CIFS is by ‘UniqueId’ which is supposed to be unique. However, some server do not ensure that the ‘UniqueId’ is always unique (for example when there is more than one filesystem in the exported share). The cache coherency is currently ensured by verifying the ‘LastWriteTime’ and size of the file. This is not a reliable way of detecting changes as some CIFS servers will not update the time until the filehandle is closed.

The rudimentary implementation is ready and the cumulative patch can be found here:

http://www.kernel.org/pub/linux/kernel/people/jays/patches/

[WARNING: The patch is lightly tested and of prototype quality.]

Here are some initial performance numbers with the patch:

Copying one big file of size ~150 MB.

$time cp /mnt/cifs/amuse.zip .
(Cache initialized)

real 1m18.603s
user 0m0.016s
sys 0m8.569s

$time cp /mnt/cifs/amuse.zip /
(Read from Cache)

real 0m28.055s
user 0m0.008s
sys 0m1.140s

Again day 2 started off very much as day 1 – lots of discussions.  This is not a bad thing, and maybe Novell would consider sponsoring a ChatWeek (it could be condensed into 2 or 3 days), it is a fact that communication is the one major stumbling block that any company/community suffers from and Novell and openSUSE are no exception to this.

We started with general discussions and continuing some of the discussions we had over dinner last night.  After lunch we then continued our discussions on openSUSE and what/how things could be improved – highly productive and very inspiring to get internal and external community members to speak honestly and bluntly about their thoughts and ideas. 

Note to the community (yes, all of you!): If you have thoughts on what or how to help imporve anything within the openSUSE universe speak up!!  If you’re not sure who or how to do so, the key contact details (in no particular order) are Zonker, opensuse-project mailing list, #opensuse-project irc channel and of course the openSUSE Board.  One of the advantages of openSUSE is that you can pick things up and run with them, if you need any help just ask and people will rally around (sometimes it takes a bit of time but it does happen, honest).

After all the discussions I had just enough time to grab a drink before the first openSUSE Marketing Team irc meeting took place.  This was shorter than I would personally have like, but that was down to one very important reason – the HackWeek BBQ!

The BBQ was and is a great opportunity for people just to chat on all levels, and general have fun with some fine food and some fine Bavarian Beer – Pascal will certainly disagree with me on the last point, but then again he is kind of correct; being Belgian he is used to a much wider variety and quality of beers, just go to FOSDEM to find out   You can see a montage of the video that I took over here – yes I know it’s evil flash!

The party atmosphere went on long into the night and everyone managed to enjoy themselves. Thank you SUSE for the fine food and beer, and thank you Pascal for bringing some prized Belgian beers for comparison with the Bavarian ones.

If you look at the openSUSE installer during the time it’s busy, you can see at least 4 types of progress bars shown:

1. Disk preparation
2. Image deployment
3. Additional package installation
4. Writing final configuration before reboot

My goal for this hackweek is to unify them as much as possible. So, now I have something to show. The prototype works on openSUSE 11.0 Alpha2 and unifies steps 1-3. Here is a screenshot:

The timing was perfect, I managed to get to the SUSE offices in NUE just in time to drop my bags and meet up with Andreas for a little tour/familiarisation of the site.  This was great because we (openSUSE invited about 10 community members to NUE) got to see a load of the great folks that work on the project and see how they do it (including the ever elusive Beineri:

We then had a pleasant brunch, and afterwards sat down to start hacking on our respective projects.

So what is one doing for HackWeek? Well it isnt exactly hacking (well not initially although I may have to so I can get the end product out); I’m hoping to do some videos of HackWeek, both documentary style(ish) and some interviews.  Why do I say “hoping”? Well I’ve hardly filmed anything I have been way too busy discussing and speaking about openSUSE with people.

Day 1 was very much a ChatWeek moment rather than HackWeek   The chatting took place all around the offices, and even in Meeting Rooms where we discussed many aspects of the project.  I continued chatting with fellow openSUSE peeps over dinner and well into the night:

It isn’t all chatting though, some people did manage to get some coding done albeit in between chats   By the way Andreas isn’t laughing at the code quality and Benji isn’t playing an advanced level of nethack – honest!

I’m certainly hoping to be able to get some video recorded on Day 2, but these people at SUSE are just so damned social that it’s really hard!!