There are various ways to build a live cd and since 11.2 Milestone 1 there is a new one: clicfs. I’ll try to explain:

The challenge with a live cd is the size of the CD and what you put on it, 700MB is not enough for a typical desktop experience. So most (if not all) distributions use compression to squeeze about 2GB on a CD. There are various compression file systems, most famous is squashfs – which is since 4.0 even in the kernel mainline (>= 2.6.29) and uses gzip compression, even though there exist patches to make it use lzma. These lzma patches are not very often refreshed and not officially supported by the squashfs authors. This might actually change soon as kernel 2.6.30 has lzma decompression built in – but it’s not there yet.. There is also cloop, which I think is exclusive to Knoppix. It’s also gzip compressed and has a different semantic than squashfs. More to that later.

But however you compress, it comes with a catch: your compression will create a read only file system. But you need to write to it, not to all places, but in various (/var/run, /var/tmp, /tmp/, often in /etc, surely in /home). For a long time, the live cds existent created one large tmpfs and symlinked all the places and files that needed writing. But this is pretty unflexible and also takes more memory than really necessary. The new solution was first unionfs, which has a long history with many ups and downs and then later aufs. Aufs is Another unionfs and is the reason for one of the downs of unionfs. With a union you can generate a file system that is actually a map of two: the read only part and a read write part. But neither of them has good short term prospect of getting in the kernel mainline (aufs is trying hard at the moment, but still my personal guess is: 11.2 will pass without it being in mainline).

Some distributions have no problem with using large kernel patches, but for openSUSE it became more and more of a problem. Every kernel update broke it and the way aufs works is not easy to adopt to kernel changes – which is the reason the aufs author is pushing into mainline. Let’s wish him luck.

So I looked around and tried unionfs-fuse and deltafs. Both are fuse file systems avoiding the need of large kernel patches to create a compressed read write file system. Both had their issues, deltafs is just a prototype at the moment and unionfs-fuse is still pretty young too (unionfs-fuse fixed most of my issues meanwhile, so possibly it will come back to me 🙂

Then I looked closer at what Fedora(10) does: they do it completely different than most distributions I checked so far. They use a huge ext3 file system with their read only content, compress it in squashfs and then use a device-mapper snapshot on top of it to make it read write into a device-mapper copy-on-write file. Very interesting, in theory very slow – not slower than 11.1 in practise. So kiwi supports that mode since v3.29 (called “dmsquash”). But as the 11.2 kernel OOPSed with such loopback mounted file systems (fixed meanwhile) I kept on looking for alternatives. And as unionfs-fuse was so close and has also support for cow, I thought if I can’t do my own fuse filesystem that combines the best of all. Actually I thought out loud while eating doener with Michael and Jan and so I hacked doenerfs as prototype in just an afternoon.

The idea is very simple: Instead of using device-mapper with squashfs snapshot and copy-on-write file I put the ext3 file system in another file system just made to compress it: doenerfs. That filesystem uses xz libraries. So you can mount the compressed image and then mount the ext3 file system loop and get a read write file system. Of course it comes with a catch: the writes don’t go anywhere but in the memory of the fuse driver. The more you write, the bigger the process gets. But in the end it doesn’t matter what uses your RAM, tmpfs or fuse mounts.

Meanwhile I extended the code quite a bit in adding some possibly unique features and renamed it to clicfs: Compressed Loop Image Container. The 11.2 Milestone 1’s Live CDs use it’s first version.

As the file system is explicitly made to compress another file system, I only need to support one file with a fixed name and with a known size and all that – a lot of complexity of other file systems is gone. So I could spend most time developing the interesting parts 😉

The fuse driver has only 3 options and each of them marks a feature:

• -m adds sparse blocks at the end of file. The actual ext3 file system has 0 blocks free, the sparse blocks are appended during boot of the livecd. After that the live cd will resize the file system to match the gained room. And only blocks actually written to will end up in memory. Sparse blocks are only marked as not yet existent. Current live cds hard code 470MB, but it’s just some /proc/meminfo left to do and then it will have free space depending on RAM amount. So far the free space was limited by either what the ext3 image left free (fedora) or how much is in the tmpfs (aufs solution)
• -l logfile will create a log file with the access pattern. You can pass this log to mkclicfs to group the ext3 blocks in order in the compressed image. I use this while building the livecds to avoid seeks on the CD.
• -c cowfile puts the blocks that are written to in an extra file instead of RAM. This makes it possible to store them on an usb stick and load them at later boots. This is the newest and least tested feature.

The sources are at git.opensuse.org.