Now I wanted to document some neat tricks and details.

When you have the hex-encoded sha256sum of a small file – for this example let’s use the GPLv3.txt on our media –
sha256sum /ipns/opensuse.zq1.de/tumbleweed/repo/oss/GPLv3.txt
8ceb4b9ee5adedde47b31e975c1d90c73ad27b6b165a1dcd80c7c545eb65b90

Then you can use the hash to address content directly by prefixing it with /ipfs/f01551220 so it becomes

/ipfs/f015512208ceb4b9ee5adedde47b31e975c1d90c73ad27b6b165a1dcd80c7c545eb65b903

In theory this also works with SHA1 and the /ipfs/f01551114 prefix, but then you risk experiencing non-unique content like
/ipfs/f0155111438762cf7f55934b34d179ae6a4c80cadccbb7f0a

And dont even think about using MD5.

For this trick to work, the file needs to be added with ipfs add --raw-leaves and it must be a single chunk – by default 256kB or smaller, but if you do the adding, you can also use larger chunks.

Here is a decoding of the different parts of the prefix:
/ipfs/ is the common path for IPFS-addressed content
f is the multibase prefix for hex-encoded data
01 is for the CID version 1
55 is for raw binary
12 is for sha2-256 (the default hash in IPFS)
20 is for 32 byte = 256 bit length of hash

And finally, you can also access this content via the various IPFS-web-gateways:
https://ipfs.io/ipfs/f015512208ceb4b9ee5adedde47b31e975c1d90c73ad27b6b165a1dcd80c7c545eb65b903

You can also do the same trick with other multibase encodings of the same data – e.g. base2

Base2 looks pretty geeky, but so far I have not found practical applications.

To get to the root cause of the issue, we enabled remote debugging with

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=ci.suse.de -Dcom.sun.management.jmxremote.password.file=/var/lib/jenkins/jmxremote.password

and attached visualvm to see what it was doing.
This showed the number of threads and memory usage in a sawtooth pattern. Every time the garbage collector ran, it dropped 500-1000 threads.

Today we noticed that every time it threw these java.lang.OutOfMemoryError: unable to create new native thread errors, the maximum number of threads was 2018… suspiciously close to 2048. Looking for the same time in journalctl showed
kernel: cgroup: fork rejected by pids controller in /system.slice/jenkins.service

So it was systemd refusing java’s request for a new thread and jenkins not handling that gracefully in all cases.
That was easily avoided with a
TasksMax=8192

Now the new peak was at 4890 live threads and jenkins served all Geekos happily ever after.

Our filesystem repo already has the go-ipfs client.

You use it with
ipfs daemon --init

And then you can add my Tumbleweed mirror with
zypper ar http://127.0.0.1:8080/ipns/opensuse.zq1.de./tumbleweed/repo/oss/ ipfs-oss

You can also browse the content online at
http://opensuse.zq1.de./tumbleweed/repo/oss/ . During my testing I found that the results are sometimes inappropriately cached on the Cloudflare CDN, so if you used it under this URL without the ipfs client, this might throw signature errors in zypper.

On the server side, the mirror is updated using the syncopensuse script from
https://github.com/bmwiedemann/opensusearchive and consistency of the repo is verified with checkrepo

When a complete repo was synced, dynaname updates a DNS entry to point to the new head:

> host -t txt _dnslink.opensuse.zq1.de.
_dnslink.opensuse.zq1.de is an alias for tumbleweedipfs.d.zq1.de.
tumbleweedipfs.d.zq1.de descriptive text “Last update: 2019-04-03 12:23:43 UTC”
tumbleweedipfs.d.zq1.de descriptive text “dnslink=/ipfs/QmSXEVuU5z23rDxMyFYDhSAUaGRUPswuSXD3aVsBEzucjE”

If you got spare bandwidth and 300 GB disk on some public server, you could also host a mirror of today’s version, simply by doing ipfs pin add QmSXEVuU5z23rDxMyFYDhSAUaGRUPswuSXD3aVsBEzucjE

This is a permalink: http://127.0.0.1:8080/ipfs/QmSXEVuU5z23rDxMyFYDhSAUaGRUPswuSXD3aVsBEzucjE also browsable via any public IPFS gateway. This means, it will always remain on the 20190401 version of Tumbleweed and no changes in content are possible – similar to how a git commit ID always refers to the same data.

So why did I create this IPFS mirror? That is related to my work on reproducible builds for openSUSE. There it regularly happened that published Tumbleweed binaries were built with libraries, compilers and toolchains that were no longer available in current Tumbleweed. This prevented me from verifying that the published binaries were indeed built correctly without manipulation on the OBS build workers.

Now, with this archive of rpms easily available, it was possible to verify many more Tumbleweed packages than before. And most importantly, it remains possible to independently verify even after Tumbleweed moves on to newer versions. This data is going to stay available as long as anyone pins it on a reachable server. I’m going to pin it as long as it remains relevant to me, so probably a bit until after the next full Tumbleweed rebuild – maybe 6 to 12 months.

Thus, it now is even less easy to sneak in binary backdoors during our package build process.

You should be aware that if you follow the below steps you will punch a hole through your firewall, so be sure and consider the security issues associated with that hole.  Many organizations require security beyond a simple login and password when connectivity is allowed from outside the firewall.  In some organizations following the below instructions without authorization from your IT security team could be a firing offense.

In theory this functionality is relatively basic, but there are lots of resources on the web that only serve to complicate the matter.  The below instructions were followed in 2015 with current SSH to create an actual working reverse tunnel.

The assumed situation is you have:

– Windows 7 or 8.1 PC behind a firewall you want to remote desktop to (the target PC)
– A openSUSE server in the cloud that you are able to ssh into and open appropriate ports and firewall holes
– A client PC from which you want to originate Remote Desktop sessions

The instructions here borrow heavily from the below blog post, but I was unable to get the tunnels to work by following the steps described at that site:

Creating persistent SSH tunnels in Windows using autossh

What has worked for me so far is:

1. On the openSUSE server
   1. Ensure you have a user account “autossh” (or whatever you want to call it).  This will be used exclusively for reverse tunnels.
   2. Ensure you have a normal user account you can use with scp with to copy files from the target PC to the openSUSE server.
2. On the target (destination) PC:
   1. Ensure you have remote desktop setup.  Machines on the local LAN should be able to remote desktop into the PC prior to starting this procedure.
   2. Download Cygwin (http://www.cygwin.com/)
   3. Install Cygwin, selecting the autossh and openssh packages.
   4. Start the Cygwin shell (Start -> Programs -> Cygwin).
   5. Generate a public/private key pair.
      1. At the command line, run: ssh-keygen
      2. Accept the default file locations
      3. Use an empty passphrase
   6. Copy your newly-created public key to the SSH server.
      1. scp .ssh/id_rsa.pub user_account@ssh.host.name:id_rsa.pub
3. Add your public key to your list of authorized keys on the server.
   1. Login to your SSH server as your normal user_account
   2.  mv id_rsa.pub  /tmp
   3. su –  # become root
   4. Ensure /home/autossh/.ssh exists
      1. # ls -ld /home/autossh/.ssh
         drwxr-xr-x 2 autossh users 4096 Apr 24 2015 /home/autossh/.ssh
      2. If not: mkdir /home/autossh/.ssh; chown autossh.users /home/autossh/.ssh; chmod 755 /home/autossh/.ssh
   5. cat /tmp/id_rsa.pub >>  /home/autossh/.ssh/authorized_keys
4. Tweak the sshd_config on the server
   1. By default openSUSE enables “AllowTcpForwarding” and “TCPKeepAlive”. Verify they are either commented out or set to “yes” in /etc/ssh/sshd_config
   2. Set “GatewayPorts yes” and “ClientAliveInterval 300” in /etc/ssh/sshd_config.  Also make sure they are not commented out.
   3. restart sshd to get the config values to be re-read:    sudo systemctl restart sshd.service
5. Test your SSH key.
   1. Logout of your SSH sever.
   2. Login to your SSH server again, but as autossh this time. This time, your key will be used for authentication and you won’t be challenged for your login credentials. If you are not logged in automatically, review the previous steps. Or contact your server administrator.
      1. ssh autossh@@ssh.host.name
   3. Logout of your SSH server.
   4. Exit of the Cygwin shell.
6. Test your SSH Tunnel capability
   1. Open a cmd prompt on your target PC as administrator
      1. start -> run -> cmd -> right click on “cmd” -> left click “run as administrator”
   2. C:\cygwin\bin\ssh -N -R 4489:localhost:3389 autossh@ssh.host.name
      1. Note that this should open an alternate port (4489) or your openSUSE server in the cloud.  openSUSE uses 3389 by default, so you need to use an alternate port on the openSUSE server end.
      2. Any connections to the alternate port should be funneled through the SSH tunnel back to the windows 7 PC on port 3389
   3. From a 3rd computer open a remote desktop connection to ssh.host.name:4489
      1. Note that remote desktop uses :4489 after the server name to designate an alternate port.
      2. If it acts like you’re not connecting at all, in all likelihood you’re not.  You probably have a firewall in place on the openSUSE server.
         1. Open port 4489 in your opensuse server firewall
            1. https://doc.opensuse.org/documentation/html/openSUSE_122/opensuse-security/cha.security.firewall.html#sec.security.firewall.SuSE.yast
            2. Or   “sudo /sbin/yast -> security and Users -> Firewall -> Allowed Services -> Advanced -> add 4489 to list of TCP Ports -> OK -> next -> finish -> quit
      3. retry remote desktop connection
   4. Once it works, from the Windows 7 command prompt kill the the ssh connection to your openSUSE server (contrl-C)
7. Test your AutoSSH Tunnel capability
   1. From the CMD prompt running as administrator
      1. C:\cygwin\bin\autossh -M 20000 -N -R 4489:localhost:3389 autossh@ssh.host.name
         1. Note that-M opens a monitoring port (I’m not sure how to leverage that)  The monitoring port is opened on the server (ssh.host.name) so if you have multiple autossh commands pointed at the same server, each should use a unique monitoring port as well as a unique tunnel port (4489 in the above.)
   2. From a 3rd computer open a remote desktop connection to ssh.host.name:4489
      1. Make sure you terminate your remote desktop session from the 3rd computer when done testing
   3. If it worked, from the Windows 7 command prompt kill the autossh command (contrl-C)
   4. exit out of your cmd window

• At this point you can manually invoke autossh to setup a semi-persistent tunnel

Setup the autossh feature as a Windows service

Note:

• the below uses cygrunsrv to install the Windows service.  If you experience problems cygrunsrv -L, cyrunsrv -LV, and cygrunsrv -R <service>    may all be useful for diagnosing the problem.  The first 2 commands list the installed services, and -R removes installed services.
• Logs for cygrunsrv default to C:\cygwin\var\log\AutoSSH.log

1. Install autossh as a Windows service
   1. Open a cmd prompt on your target PC as administrator
      1. start -> run -> cmd -> right click on “cmd” -> left click “run as administrator”
   2. cd C:\cygwin\bin
   3. cygrunsrv -I AutoSSH -p /bin/autossh -a “-M 20000 -N -R 4489:localhost:3389 autossh@ssh.host.name” -e AUTOSSH_NTSERVICE=yes
      1. If you get an error with this command, manually type the ” marks.  They may not be handled properly with cut&paste.
      2. Be very careful with the above.  A misbehaving service can be hard to remove in Windows.  It may require safe mode if the service won’t accept stop commands.
   4. Tweak Windows service settings.
      1. Open the Services management console (Administrative Tools -> Services).
      2. Edit the properties of the AutoSSH service.
      3. In the “Log On” tab, select the “This account” radio button and set the service to run as your current user.  This is very important to do before starting the service in order for the ssh certificate to be used.
      4. Change the startup mode to “Automatic (Delayed Start)”
      5. Start the service.
2. Test your tunnel as described in 6.2 above
   1. Be sure to test after rebooting your target Windows 7 or 8.1 PC.
   2. I have had it working for 6months and used it a lot.  I’ve seen network drops.  Target PC reboots.   openSUSE server reboots.  The tunnel just keeps working.

If all went well, congratulations you now have a persistent tunnel

You should be aware you have just punched a hole through your firewall, so be sure and consider the security issues associated with that hole.  Many organizations require security beyond a simple login and password when connectivity is provided from outside the firewall.

On the second day we learned that the infra team manages servers with puppet, about jenkins-job-builder (jjb) that creates around 4000 jobs from yaml templates. We learned about nodepool that keeps some VMs ready so that jobs in need will not have to wait for them to boot. 180-800 instances is quite an impressive number.
And then we spent three days on discussing and hacking things, the topics and outcomes of which you can find in the etherpad linked from the wiki page.
I got my first infra patch merged, and a SUSE Cloud CI account setup, so that in the future we can test devstack+tempest on openSUSE and have it comment in Gerrit. And maybe some day we can even have a test to deploy crowbar+openstack from git (including the patch from an open review) to provide useful feedback, but for that we might first want to move crowbar (which is consisting of dozens of repos – one for each module) to stackforge – which is the openstack-provided Gerrit hosting.

see also: pleia2’s post

Overall for me it was a nice experience to work together with all these smart people and we certainly had a lot of fun

It took a bit but I am happy to report that all openSUSE 13.1 images in Amazon EC2, Google Compute Engine and Microsoft Azure public cloud environments have been refreshed. After the latest round of the GNU-TLS and OpenSSL fixes the security was, as usual, extremely efficient in providing fixed packages and these have been available in all cloud images via zypper up since last Friday. As of today the base images available in the public cloud frameworks contain the fixes by default.

In Amazon the new images are as follows:

• ap-northeast-1: ami-79296078
• ap-southeast-1: ami-84a7fbd6
• ap-southeast-2: ami-41cbae7b
• eu-west-1: ami-b56aa4c2
• sa-east-1: ami-bffb54a2
• us-east-1: ami-5e708d36
• us-west-1: ami-16f2f553
• us-west-2: ami-b7097487

In Google compute engine the image name is: opensuse-13-1-v20140609

The old image (opensuse131-v20140417) has been deprecated. To access the image you will need to add –image=opensuse-cloud/global/images/opensuse-13-1-v20140609 as the openSUSE images are not yet fully integrated into the GCE framework. Still working on that part with Google. This image also has upgrades to the google-cloud-sdk package and enable the bq (big-query) command. The gcloud command is still a bit rough around the edges, but the gcutil command should work as expected. Eventually gcutil is going to be deprecated by Google thus there is work to be done to fix the integration issues with the gcloud command. If anyone has time to work on that please send submit request to the google-cloud-sdk package in the Cloud:Tools project in OBS. Unfortunately Google still hasn’t posted the source anywhere for open collaboration . They’ll get there eventually. I will try and push any changes upstream.

In Azure just search for openSUSE in the Gallery, it’s more of a point an click thing

And that’s a wrap. Not certain we will be able to improve on the speed of such fire drill updates, but we’ll try to keep refreshing images as quickly as time allows when critical vulnerabilities in the core libraries get exposed.

Have a lot of fun….

Patching

Being this old means that my first OS was MS-DOS (or was it AMIGA or can I count C-64 in?). It was MS-DOS version 3 and it was pain to use but 4DOS that was something! It had all the goodies that I’m enjoying in Unix Shell. I also have seen time before rise of Git or Kernel Bitbucket episode. It was those sweet times when Linux kernel was all about patches and there where no broadband connections for homes or 2/3/4/5G wireless Internet.
Patch is small text or binary file that tells what have changed in that particular file. So if two persons are working with same file and they don’t use version control (which they should use) then they should use patches. I have been long in enough in business and been university life (in specially ICT area) that I know most students tends to send whole changed file and you have to guess what have changed. That annoying and stupid many ways but most of the people argues that it’s safer to send whole file to make sure all changes are there. So I have to figure out what they have changed and what have I. I know how to make it but please please all of you should learn it and that end of that rant.
Nowadays most people favor GIT, Mercurial, Subversion or ancient but working CVS (Which was de-facto factory standard before Subversion). There is plenty more version control options to choose from but I think those are most popular.
They all work the same way deep down (especially GIT) than sending patch by mail. There just data base of file changes and If you do all of them you get solid file.

So what all of this means?

There is still one place where patches are feeling strong and that place is Linux kernel patches. Why to make kernel patch? You want to add new kernel feature but can’t get it in Linus Torvalds tree? Then you start to deliver it as a patch or patch-set.
There is many healthy patch-sets like openSUSE kernel patch-set if I have to mention one. Feeling utterly confused so it time to read this and this then you should understand little bit more and I can rest my hands for a while from writing.

Couple of patches BFS and BFQ

Ok I have wrapped some patches to fit with openSUSE kernel (For you to test with). BFQ is Budget Fair Queueing so it’s more robust than kernel default CFQ . CFQ is tuned for common rotating HDD and it’s behaving badly with SSDs. BFQ in other words works business as usual with SSD. What are I/O Schedulers? You got to read little bit here and here to understand about it.
So what is BFS? It’s more than I/O scheduler it’s for scheduling task and stuff like that. So both of them makes your desktop fly or they don’t. They are not part official kernel though both of them want to get in sooner than later.

Applying patch

Just download patches to your computer with (and remember they are for Linux kernel version 3.14 [they won’t work with version 3.something or up coming 3.15 they only work with version 3.14]).

wget https://raw.githubusercontent.com/illuusio/misc-patches/master/linux/opensuse/BFQ/3.14/0001-Budget-Fair-Queueing-I-O-Scheduler.patch
wget https://github.com/illuusio/misc-patches/raw/master/linux/opensuse/BFS/3.14/0002-The-Brain-Fuck-Scheduler-v0.447-by-Con-Kolivas.patch

and apply them again in root where you cloned git in first part

patch -p1 -i 0001-Budget-Fair-Queueing-I-O-Scheduler.patch
patch -p1 -i 0002-The-Brain-Fuck-Scheduler-v0.447-by-Con-Kolivas.patch

and then get config

wget https://raw.githubusercontent.com/illuusio/misc-patches/master/linux/opensuse/config-3.14-bfsbfq
cp config-3.14-bfsbfq .config
make oldconfig

then compile it like last time and get it working. Then you are again on your on in you journey to deeper knowledge.

Basics

Understanding how Linux boots read about Linux startup. Basics are like this: you have file name bzImage (vmlinuz) that is bzip2 compressed kernel image and then you have Linux kernel modules. You have to understand that everything can be compiled into Linux kernel (ok nearly nowadays) that is why Linux is called monolithic kernel. Most of the cases that is not how kernel is compiled. You have small boot kernel and then there is plenty of modules besides this bzImage-thing. What are these modules hmm… it’s again bit complicated.
Linux Kernel modules are extensions to Linux kernel. They can be new drivers for webcam, new filesystem or they can be bands album like here. Modules are way to get Linux kernel to minimum size and boot ultra fast. After kernel is kicked alive modules are loaded automaticly by user-space tool udev.

How to get you shiny new Linux kernel working?

Ok let’s do it. Never remove your working binary openSUSE RPM kernel or it’s modules (if you do reinstall them right away back)! If you remove kernel modules it’s most likely that your machine won’ẗ boot. Now you know what not to do.. so let’s see what you can do? You compiled kernel with tutorial that I wrote last time? Then you should have very fancy openSUSE kernel. I assume you are at commandline on same directory where your openSUSE kernel GIT is compiled. Look at directory ‘arch/x86_64/boot/’ (or arch/x86/boot/) with

ls arch/x86_64/boot/

there should be bzImage. Let’s copy it to correct location (after this only root can do these or with sudo-command) and version number you can get from Makefile current is 3.14.1 (but it changes rapidly so check first three line from Makefile if you are unsure) and suffix ‘-desktop’ comes from ‘.config’ file.

cp arch/x86_64/boot/bzImage /boot/vmlinuz-3.14.1-desktop
cp System.map /boot/System.map-3.14.1-desktop

Ok we have kernel in correct place then we have to install modules. Now I have learned in rough way not to use ‘make install’ so we do it like this (you can use any temp directory you like I tend to use ‘/tmp/kernel-install’).

make INSTALL_MOD_PATH=/tmp/kernel-install modules_install
cd /tmp/kernel-install/lib/modules
cp -r 3.14.1-desktop /lib/modules

There is also firmwares but they should be installed by default and remember if you need some outside kernel module they are not build in your new shiny kernel. So if you have some fancy pantsy WIFI or Radeon catalyst you have to compile it afterwards (we see that next time) or you use you searching skills to find out.

cd /boot
mkinitrd -k vmlinuz-3.14.1-desktop -i initrd-3.14.1-desktop
grub2-mkconfig > grub2/grub.cfg 

And I assume you have grub2. If not don’t have you have to figure out this by yourself with YaST. GRUB1 config is /boot/grub/menu.lst if you want to do it by hand. Next time you boot it will try to first boot your new kernel you can choose old kernel from start up menu but that is up to you to discover how. If you end up system that is not booting don’t blame me there is ways to recover but it’s not this blogs agenda (my kind recommendation to have openSUSE live cd around).

What is initrd?

Initrd is for booting. Initrd will be loaded before your harddrive is. There is tools that needed for booting after harddrive is mounted Initrd will step down and kernel start using harddrive. OBS! after 13.1 initrd system will be done with new tool not any more with good old mkinitrd.

Ok now you can play with your new Linux Kernel. There can be new tricks or something can be broken. Have again time and patience as long as you have your old kernel and modules you can boot again.

Kernel? eh what?

Imagine a car where you can change motor as many times you like. You can tune your motor as much as you can and you can run it in your car again with Linux you can do that. Your ride with new kernel can be good or really bad (or something between).
As you might know Linux is just a name for Linux-kernel as Google Android is just Google’s forked Linux kernel. What happens after kernel booting is not that important anymore.

User-space and kernel-space

Nearly every modern operating system separate kernel-space and user-space. Your applications like browser works in user-space and your USB-stick operates in kernel-space. In normal life you never have to cross kernel space but If you do you should really find what are: /dev, /proc and /sys directories in Linux. Those directories contains kernel stuff what is in there you have to check from Internet or how /sys works from this Wikipedia article.

openSUSE kernel

Official linux kernel can be found on https://www.kernel.org/ and kernel newbies can go here (Really give time to that site if you don’t know what you are doing). You have to make clear to yourself there is no official binaries for Linux kernel. If you compile and start using yours it’s as official as anyone else. Distributions have their own official kernel binaries and that that.
So you can just pull kernel from git version control down and start compiling right a way. It’s not that easy you need to have config file and doing that is hardest part. There is preset configs in Linux but I assume you are using openSUSE version of Linux kernel.
That means what you ask from me? You have to understand there is official and only official Linus Torvals kernel GIT and then there is hundreds versions of kernel that contains some random stuff that is not allowed in mainline Linus Torvalds version.
Mainly every distribution have their own version of Kernel and openSUSE is not an exception of this. openSUSE Linux kernel can be pulled from git: http://kernel.opensuse.org/cgit/kernel/.

WTF? ROLF? I’ll install something that only has binaries

Yes please do so there is kernel binary running in your openSUSE currently and you don’t have to change that. Compiling kernel is not for people in hurry nor for people that doesn’t have adventurous mind or some urge to do it. It takes time to learn and years to master really. In that time frame you just have to admit that sometimes your new compiled kernel doesn’t event boot. So we start and we need some tools (install them as root)

zypper install gcc make git

so then we clone openSUSE kernel stable branch (see this page for more)

git clone -b stable git://kernel.opensuse.org/kernel.git

You need couple of gigabytes free space just for pull and one gigabyte more for the build. So now you have openSUSE Linux kernel. If you like to compile it biw then we get config file. We get that openSUSE uses for desktop kernel from x86_64. Attention! if you are using x86 machine it won’t boot 64-bit kernel! So use this file here if you have 32-bit machine.. After that this is how you do it:

wget http://kernel.opensuse.org/cgit/kernel-source/plain/config/x86_64/desktop?h=stable
mv desktop\?h\=stable .config
make oldconfig
make bzImage
make modules
or
make bzImage modules

and again it takes time if you have for example 4 cpu you can allways compile with all of them

make bzImage modules -j4

I won’t tell you how to get it running until next time. You can tune config with

make menuconfig

Test compiling with different kernel config options make it non workable and start again. You can test different x86_64/x32 architectures and remove modules or compile everything in kernel (After all linux is monolithic kernel). So happy compiling times and remember to have fun..

Well, that’s the announcement, but would make for a rather short blog. Thus, let me talk a bit about how this all works and speculate a bit why we’ve not been all that good with getting stuff out into the public cloud.

Let me start with the speculation part, i.e. hindrances in getting openSUSE images published. In general to get anything into a public cloud one has to have an account. This implies that you hand over your credit card number to the cloud provider and they charge you for the resources you use. Resources in the public cloud are anything and everything that has something to do with data. Compute resources, i.e. the size of an instance w.r.t. memory and number of CPUs are priced differently. Sending data across the network to and from your instances incurs network charges and of course storing stuff in the cloud is not free either. Thus, while anyone can put an image into the cloud and publish it, this service costs the person money, granted not necessarily a lot, but it is a monthly recurring out of pocket expense.

Then there always appears to be the “official” apprehension, meaning if person X publishes an openSUSE image from her/his account what makes it “official”. Well first we have the problem that the “official” stamp is really just an imaginary hurdle. An image that gets published by me is no more or less “official” than any other images. I am after all not the release manager or have any of my fingers in the openSUSE release in any way. I do have access to the SUSE accounts and can publish from there and I guess that makes the images “official”. But please do not get any ideas about “official” images, they do not exist.

Last but not least there is a technical hurdle. Building images in OBS is not necessarily for the faint of heart. Additionally there is a bunch of other stuff that goes along with cloud images. Once you have one it still has to get into the cloud of choice, which requires tools etc.

That’s enough speculation as to why or why not it may have taken us a bit longer than others, and just for the record we did have openSUSE 12.1 and openSUSE 12.2 images in Amazon. With that lets talk about what is going on.

We have a project in OBS now, actually it has been there for a while, Cloud:Images that is intended to be used to build openSUSE cloud images. The GCE image that is public and the Amazon image that is public both came from this project. The Azure image that is currently public is one built with SUSE Studio but will at some point also stem from the Cloud:Images OBS project.

Each cloud framework has it’s own set of tools. The tools are separated into two categories, initialization tools and command line tools. The initialization tools are tools that reside inside the image and these are generally services that interact with the cloud framework. For example cloud-init is such an initialization tool and it is used in OpenStack images, Amazon images, and Windows Azure images. The command line tools let you interact with the cloud framework to start and stop instances for example. All these tools get built in the Cloud:Tools project in OBS. From there you can install the command line tools into your system and interact with the cloud framework they support. I am also trying to get all these tools into openSUSE:Factory to make things a bit easier for image building and cloud interaction come 13.2.

With this lets take a brief closer look at each framework, in alphabetical order no favoritism here.

Amazon EC2

An openSUSE 13.1 image is available in all regions, the AMI (Amazon Machine Image) IDs are as follows:

sa-east-1 => ami-2101a23c
ap-northeast-1 => ami-bde999bc
ap-southeast-2 => ami-b165fc8b
ap-southeast-1 => ami-e2e7b6b0
eu-west-1 => ami-7110ec06
us-west-1 => ami-44ae9101
us-west-2 => ami-f0402ec0
us-east-1 => ami-ff0e0696

These images use cloud-init as opposed to the “suse-ami-tools” that has been used previously and is no longer available in OBS. The cloud-init package is developed in launchpad and was started by the Canonical folks. Unfortunately to contribute you have to sign the Canonical Contributor Agreement (CCA). If you do not want to sign it or cannot sign it for company reasons you can still send stuff to the package and I’ll try and get the stuff integrated upstream. For the interaction with Amazon we have the aws-cli package. The “aws” command line client supersedes all the ec2-*-tools and is an integrated package that can interact with all Amazon services, not just EC2. It is well documented fully open source and hosted on github. The aws-cli package replaces the previously maintained ec2-api-tools package which I have removed from OBS.

Google Compute Engine

In GCE things work by name and the openSUSE 13.1 image is named opensuse131-20140227 and is available in all regions. Google images use a number of tools for initialization, google-daemon and google-startup-scripts. All the Google specific tools are in the Cloud:Tools project. Interaction with GCE is handled with two commands, gcutil and gsutil, both provided by the google-cloud-sdk package. As the name suggests google-cloud-sdk has the goal to unify the various Google tools, same basic idea as aws-cli, and Google is working on the unification. Unfortunately they have decided to do this on their own and there is no public project for google-cloud-sdk which makes contributing a bit difficult to say the least. The gsutil code is hosted on github, thus at least contributing to gsutil is straight forward. Both utilities, gsutil for storage and gcutil for interacting with GCE are well documented.

In GCE we also were able to stand up openSUSE mirrors. These have been integrated into our mirrorbrain infrastructure and are already being used quite heavily. The infrastructure team is taking care of the monitoring and maintenance and that deserves a big THANK YOU from my side. The nice thing about hosting the mirrors in GCE is that when you run an openSUSE instance in GCE you will not have to pay for network charges to pull your updated packages and things are really fast as the update server is located in the same data center as your instance.

Windows Azure

As mentioned previously the current image we have in Azure is based on a build from SUSE Studio. It does not yet contain cloud-init and only has WALinuxAgent integrated. This implies that processing of user data is not possible in the image. User data processing requires cloud-init and I just put the finishing touches on cloud-init this week. Anyway, the image in Azure works just fine, and I have no time line when we might replace it with an image that contains cloud-init in addition to WALinuxAgent.

Interacting with Azure is a bit more cumbersome than with the other cloud frameworks. Well, let me qualify this with, if you want packages. The Azure command line tools are implemented using nodejs and are integrated into the npm nodejs package system. Thus, you can use npm to install everything you need. The nodejs implementation provides a bit of a problem in that we hardly have a nodejs infrastructure in the project. I have started packaging the dependencies, but there is a large number and thus this will take a while. Who would ever implement….. but that’s a different topic.

That’s where we are today. There is plenty of work left to do. For example we should unify the “generic” OpenStack image in Cloud:Images with the HP specific one, the HP cloud is based on OpenStack, and also get an openSUSE image published in the HP cloud. There’s tons of packaging left to do for nodejs modules to support the azure-cli tool. It would be great if we could have openSUSE mirrors in EC2 and Azure to avoid network charges for those using openSUSE images in those clouds. This requires discussions with Amazon and Microsoft, basically we need to be able to run those services for free, which implies that both would become sponsors of our project just like Google has become a sponsor of our project by letting us run the openSUSE mirrors in GCE.

So if you are interested in cloud and public cloud stuff get involved, there is plenty of work and lots of opportunities. If you just want to use the images in the public cloud go ahead, that’s why they are there. If you want to build on the images we have in OBS and customize them in your own project feel free and use them as you see fit.