Our filesystem repo already has the go-ipfs client.

You use it with
ipfs daemon --init

And then you can add my Tumbleweed mirror with
zypper ar http://127.0.0.1:8080/ipns/opensuse.zq1.de./tumbleweed/repo/oss/ ipfs-oss

You can also browse the content online at
http://opensuse.zq1.de./tumbleweed/repo/oss/ . During my testing I found that the results are sometimes inappropriately cached on the Cloudflare CDN, so if you used it under this URL without the ipfs client, this might throw signature errors in zypper.

On the server side, the mirror is updated using the syncopensuse script from
https://github.com/bmwiedemann/opensusearchive and consistency of the repo is verified with checkrepo

When a complete repo was synced, dynaname updates a DNS entry to point to the new head:

> host -t txt _dnslink.opensuse.zq1.de.
_dnslink.opensuse.zq1.de is an alias for tumbleweedipfs.d.zq1.de.
tumbleweedipfs.d.zq1.de descriptive text “Last update: 2019-04-03 12:23:43 UTC”
tumbleweedipfs.d.zq1.de descriptive text “dnslink=/ipfs/QmSXEVuU5z23rDxMyFYDhSAUaGRUPswuSXD3aVsBEzucjE”

If you got spare bandwidth and 300 GB disk on some public server, you could also host a mirror of today’s version, simply by doing ipfs pin add QmSXEVuU5z23rDxMyFYDhSAUaGRUPswuSXD3aVsBEzucjE

This is a permalink: http://127.0.0.1:8080/ipfs/QmSXEVuU5z23rDxMyFYDhSAUaGRUPswuSXD3aVsBEzucjE also browsable via any public IPFS gateway. This means, it will always remain on the 20190401 version of Tumbleweed and no changes in content are possible – similar to how a git commit ID always refers to the same data.

So why did I create this IPFS mirror? That is related to my work on reproducible builds for openSUSE. There it regularly happened that published Tumbleweed binaries were built with libraries, compilers and toolchains that were no longer available in current Tumbleweed. This prevented me from verifying that the published binaries were indeed built correctly without manipulation on the OBS build workers.

Now, with this archive of rpms easily available, it was possible to verify many more Tumbleweed packages than before. And most importantly, it remains possible to independently verify even after Tumbleweed moves on to newer versions. This data is going to stay available as long as anyone pins it on a reachable server. I’m going to pin it as long as it remains relevant to me, so probably a bit until after the next full Tumbleweed rebuild – maybe 6 to 12 months.

Thus, it now is even less easy to sneak in binary backdoors during our package build process.

Because we were around 50 participants, introductions and announcements were the only things done in the big group. All actual work happened in 5-10 smaller circles.

We had participants from large companies like Google (with bazel), MicroSoft and Huawei, but also from many distributions and open source projects. Even MirageOS as non-Linux OS.

We did knowledge-sharing, refine definitions of terms, evolve concepts like “rebuilders” for verifying builds and allow users to better trust software they install, and such.

I learned about the undocumented DB dump (153 MB) and DB schema

And we had some hacking time, too, so there is now
a jenkins job that renders the list of unreproducible openSUSE Factory packages.

Also, my maintainer tool now has added support for the Alpine Linux distribution, thanks to help by one of its maintainers: Natanael Copa.
This is meant to help all cross-distro collaboration, not just for reproducible builds.

There is still work to be done to make better use of Mitre CPE to map package names across distributions.

I think, one major benefit of the summit was all the networking and talking going on, so that we have an easier time working with each other over the internet in the future.

Thus, I started to use a simpler alternative image building method, named altimagebuild when I built our first working Raspberry Pi images in 2013 and now I re-used that to build x86_64 VM images at
https://build.opensuse.org/package/show/home:bmwiedemann/altimagebuild
after I found out that it even works in OBS, including publishing the result to our mirror infrastructure.
It is still in rpm format because of how it is produced, so you have to use unrpm to get to the image file.

This method uses 3 parts.

• a .spec file that lists packages to be pulled into the image
• a mkrootfs.sh that converts the build system into the future root filesystem you want
• a mkimage.sh that converts the rootfs into a filesystem image

The good thing about it is that you do not need specialized external tools, because everything is hard-coded in the scripts.
And the bad thing about it is that everything is hard-coded in the scripts, so it is hard to share general improvements over a wider range of images.

In the current version, it builds cloud-enabled partitionless images (which is nice for VMs because you can just use resize2fs to get a larger filesystem and if you later want to access your VM’s data from outside, you simply use mount -o loop)
But it can build anything you want.

To make your own build, do osc checkout home:bmwiedemann/altimagebuild && cd $_ && osc build openSUSE_Leap_42.2

So what images would you want to build?

DAPS lets you publish your DocBook 4 or Docbook 5 XML sources in various output formats such as HTML, PDF, ePUB, man pages or ASCII with a single command. It is perfectly suited for large documentation projects by providing profiling support and packaging tools. DAPS supports authors by providing linkchecker, validator, spellchecker, and editor macros. DAPS exclusively runs on Linux.

Download & Installation

For download and installation instructions refer to https://github.com/openSUSE/daps/blob/master/INSTALL.adoc
Highlights of the DAPS 2.0 release include:

• fully supports DocBook 5 (production ready)
• daps_autobuild for automatically building and releasing books from different sources
• support for EPUB 3 and Amazon .mobi format
• default HTML output is XHTML, also supports HTML5
• now supports XSLT processor saxon6 (in addition to xsltproc)
• improved “scriptability”
• properly handles CSS, JavaScript and images for HTML and EPUB builds (via a “static/” directory in the respective stylesheet folder)
• added support for JPG images
• supports all DocBook profiling attributes
• improved performance by only loading makefiles that are needed for the given subcommand
• added a comprehensive test suite to ensure better code quality when releasing
• tested on Debian Wheezy, Fedora 20/21 openSUSE 13.x, SLE 12, and Ubuntu 14.10.

Please note that this DAPS release does not support webhelp. It is planned to re-add webhelp support with DAPS 2.1.

For a complete Changelog refer to https://github.com/openSUSE/daps/blob/master/ChangeLog

Support

If you have got questions regarding DAPS, please use the discussion forum at https://sourceforge.net/p/daps/discussion/General/. We will do our best to help.

Bug Reports

To report bugs or file enhancement issues, use the issue Tracker at https://github.com/openSUSE/daps/issues.

The DAPS Project

DAPS is developed by the SUSE Linux documentation team and used to generate the product documentation for all SUSE Linux products. However, it is not exclusively tailored for SUSE documentation, but supports every documentation written in DocBook.
DAPS has been tested on Debian Wheezy, Fedora 20/21 openSUSE 13.x, SLE 12, and Ubuntu 14.10.

The DAPS project moved from SourceForge to GitHub and is now available at https://opensuse.github.io/daps/.

How it should work

Legally, the Linux kernel GPLv2 license leaves proprietary binary drivers in a bit of a tangle. While some claim it should be OK, others say not – and that is what most distributions currently assume: one can not ship a Linux distribution with proprietary, binary drivers. NVIDIA has agreed that our users can grab their drivers from the official NVIDIA servers. They are packaged by SUSE engineers however. They take care of both SLE and openSUSE proprietary driver packaging for NVIDIA hardware, and have contacts at NVIDIA who get the drivers up on their ftp mirrors.

The packages are build on a dedicated system, but the package spec (skeleton for building) is in OBS – for example, G03 is here. Anybody could use these to build the nvidia driver locally (the nvidia binary driver is grabbed from the nvidia driver during building). The command sequence for local building can be found in the README.

Once the packages are build, they are send to NVIDIA, which signs the packages with their key and generates and signs repositories, making them available to the public.

To note is the fact that this is all manual and takes a while on NVIDIA’s side (and ours). This of course is part of the reason why we don’t offer NVIDIA packages for Factory, our fast-rolling development repository.

What happened

What occurred for 13.1 is a typical case of “everything went wrong”.

Up until a few weeks before the release, the driver did not build against the Linux Kernel version 3.11. NVIDIA warned against the use of a patch for this problem created by third parties so the driver team did not have a running build. Due to a holiday, it took a while to get the packages build and pushed to NVIDIA. Unfortunately, by the time this was done, the holiday period at NVIDIA blocked progress for another week. Once the package was signed, it took the webteam at NVIDIA another week to get the repository published. It all added up to almost a month and brought quite some inconvenience for users eager to use their latest NVIDIA cards with the latest openSUSE.

About a week later, the openSUSE 13.1 NVIDIA drivers disappeared again from the nvidia servers for a day or so. We don’t know exactly what happened there, it might have simply been a server issue.

What we will do

The first and most obvious thing to change would be to improve coordination. The developers taking care of the NVIDIA packages should be made aware as early as possible about the release planning and a replacement in case for holidays should be available. We noted this in our 13.1 release report and will make sure that there will be a task in our task tracker for this for the next release. We also need to talk to NVIDIA about this to make sure that there, too, somebody can fill in for SUSE’s contacts.

But there are also thoughts on how to improve further. Some of the current ideas:

• To get drivers for Factory the process needs more automation as the driver may break on kernel or X changes
• We could try to open the process and work with community members to secure the process in case the SUSE folks are unavailalbe
• It would be nice if we could make NVidia to see benefit of working more on the driver in openSUSE, e.g. by doing some testing
• We could write some scripts that check regularly whether repo and packages are still available and in a valid state (meta data matches available RPMs)

We’ll have to see which of these we can implement, when and how. But rest assured that we will do what we can to prevent this in the future!

And the stats!

After a bit of a hiatus, we’re back with the numbers. Development has slowed down around new year and isn’t back to speed yet so the 10th spot is shared by quite a big group of people…

update: new image with kernel-3.6 and minimal X11/icewm http://www.zq1.de/~bernhard/linux/opensuse/raspberrypi-opensuse-20130911x.img.xz (103MB)

We got a new armv6 based image for the Raspberry Pi.
This one is only 82MB compressed, so pretty minimalistic.
http://www.zq1.de/~bernhard/linux/opensuse/raspberrypi-opensuse-20130907.img.xz

The exciting new thing is that this was created using an alternative image building automatism which I wrote from scratch in three hours this morning.
The scripts can be found at
https://build.opensuse.org/package/show/devel:ARM:Factory:Contrib:RaspberryPi/altimagebuild
and are also embedded within the image under /home/abuild/rpmbuild/SOURCES/

This means that everyone can now easily build his own images the way he likes and even branch and do submit requests for changes that are useful for others.
The way to use this is simple.
If you have 6GB RAM, you can speed things up with export OSC_BUILD_ROOT=/dev/shm/arm before you do
osc co devel:ARM:Factory:Contrib:RaspberryPi altimagebuild
cd devel:ARM:Factory:Contrib:RaspberryPi/altimagebuild
bash -x main.sh


This pseudo-package does not easily build within OBS or osc alone because it needs root permissions for some of the steps (chroot, mknod, mount), which could only be workarounded with User-Mode-Linux or patching osc.
The build consists of three steps that can be seen in main.sh:

1. First, osc build is used to pull in required packages and setup the armv6 rootfs.
2. Second, mkrootfs.sh modifies a copy of the rootfs under .root to contain all required configs
3. And finally, mkimage.sh takes the .root dir and creates a .img from it that can be booted

This can build an image from scatch in three minutes. And my Raspberry Pi booted successfully with it within 55 seconds.

There are some remaining open issues:

• the repo key is initially untrusted
• still uses old 3.1 kernel – solved
• build scripts have no error handling

Compared to the old image, this one has some advantages:

• It is easier to resize because the root partition is the last one
• Compressed image is much smaller
• Reproducible image build, so easy to customize
• It is armv6 with floating point support, so could be faster
• We have over 5200 successfully built packages from openSUSE:Factory:ARM
  so for example you can install a minimalistic graphical environment with zypper install xauth twm xorg-x11-server xinit and start it with startx

So if you wanted to play with openSUSE on RPi, you can do so right now and have a lot of fun.

The kernel was compiled with Clang (C frontend of LLVM)  and boots into a running openSUSE desktop!

Another news is that I had fixed the bugs in Virtual Machine builds (build script) when using them with some architectures like PowerPC 32bit and SH4. So now also the combination of using for example KVM (XEN should also work) in a worker together with ARM, MIPS, PowerPC and SH4 is working. The appropriate fixes are in one of the next build script releases (if not even released already now with OBS 2.1, I have to check that). You can select architecture “sh4” with OBS 2.1 and also start a scheduler with “sh4”.

With the use of the QEMU User Mode, you can build also accelerated native cross toolchains for your host architecture so time critical parts like the compiler can run without the emulator. This works with .deb as well as with .rpm based backages. The MeeGo Project as well as the openSUSE Port to ARM uses this technique to provide an optimum between compatibility and performance. It means you can mix natively build packages and use cross toolchains on it. The “CBinstall:” feature helps you to use native or cross builds automatically depending on if your build host is a native machine or a x86 machine with cross build. In summary, we have the current classics of linux embedded archs together now in OBS: ARM, x86, MIPS 32, PowerPC 32 and SH4.

I have uploaded the fixed QEMU package to the OBS project openSUSE:Tools:Unstable inside the package “qemu-devel” after some more testing. I have of course also a OBS meta prjconf file working with Debian Sid. The SH4 port of Debian Sid you can find at Debian Ports Site.

And last but not least I would like to thank Riku Voipio of the Debian Project, QEMU project and MeeGo project and other major contributors during the QEMU 0.13 development cycle for the restless work on QEMU user mode improvements. In case of KVM, QEMU is used even twice, with QEMU-KVM as well as QEMU User Mode. I am sure I had forgotten other important people, so thanks to them also.

Interesting is the fact that we adapted the cross build system for OBS to MeeGo, first developed for use in Maemo and openSUSE @ ARM. An improved version for the standard MeeGo releases, and for the MeeGo weekly snapshots is used in the MeeGo OBS System to build all ARM releases of MeeGo (the cross toolchain will later get part of the MeeGo SDK @ ARM), thanks to Jan-Simon Möller (In the openSUSE ML, the issue of reactivating openSUSE:Factory ARM builds were brought up. So it might be a good variant to backport Jan-Simons new solution back into openSUSE @ ARM for that purpose). All the MeeGo related OBS installations will move sooner or later to OBS 2.1.

But now to the most recent work, Access Control support. A preview was shipped with OBS 1.8. Now an own OBS version, 2.1, will be dedicated to the introduction of this single new feature into the OBS mainline: Access Control (or abbrevated ACL for Access Control Lists). ACL means that there is control by the user on a per project or per package basis to protect information, source and binaries from the read access of other users in an OBS system and to hide projects or packages.

What is the intended audience of ACL? ACL is intended for installations of OBS that require protection of projects or packages during work. This can be but is not limited to commercial installations of OBS, or semi public installations of OBS.

How does ACL work? ACL sits on top of two features introduced with OBS 2.0: Role and Permission Management as well as freely definable user groups. ACL uses 4 specifically defined permissions (‘source_access’ for read access to sources, ‘private_view’ for viewing package and project information, ‘download_binaries’ for read access to binaries and ‘access’ permission to protect and hide everything and all from read access and viewing) on a user or group in the Role and Permission management. Also, the preexisting roles “maintainer”, “reader” and “downloader” had been modified with specific predifined permissions (which can at any time changed with the role and permission editor dynamically). And last but not least 4 new flags (namely ‘sourceaccess’ to signal a project/package has read protected source code, ‘binarydownload’ to signal it has read protected packages, ‘privacy’ to signal information/logfiles or status cannot be read and ‘access’ to hide and protect a project or package completely in all possible OBS API calls) had been added to the project and package descriptions to signal that some information is only readable by specific users or groups, or that information is hidden.

How do I use ACL? There are 4 steps to use ACL (a part of them a optional and can only be performed by the Administator of an OBS instance). Step one is to assign the listed permissions to a role, user or group (this step can be done only by the admin, and is not needed for the predefined roles “maintainer”, “reader” and “downloader”). Step two is to add a group for special users to projects which are intended to be run with ACL (this operations can only be performed by the admin). Step three is to protect a project with appropriate protection flags at project creation by adding them to the project meta. Step four is to add other users or groups with one of the new predefined roles that has ACL permissions added to the project meta.

What information can be protected by ACL? The protected information is grouped into 4 categories. Category 1 (flag ‘sourceaccess’) is source code. Category 2 (flag ‘binarydownload’) is binary packages or logfiles or builds. Category 3 (flag ‘privacy’) is project or package information like build status. Category 4 (flag ‘access’) is all viewable or accessable information to any project or package (full blocking of all access and information).

Example of a project configuration using ACL:

<user userid="MartinMohring" role="maintainer" />
<!-- grant user full write and read access -->

<group groupid="MeeGo-Reviewer" role="maintainer" />
<!-- grant group full write and read access -->

<group groupid="MeeGo-Developers" role="reader" />
<!-- grant group full source read access -->

<group groupid="MeeGo-BetaTesters" role="downloader" />
<!-- grant group access to packages/images -->

  <sourceaccess>
    <disable/>
  </sourceaccess>
  <!-- disable read access - unless granted explictely.
          This flag will not accept arch or repository arguments. -->

  <binarydownload>
    <disable/>
  </binarydownload>
  <!-- disable access - unless granted explictely -
          to packages/image and logfiles -->

  <access>
    <disable/>
  </access>
  <!-- disable access - unless granted explictely-,
          project will not visible or found via search,
          nor will any source or binary or logfile be accessable.
          This flag will not accept arch or repository arguments. -->

  <privacy>
    <enable/>
  </privacy>
  <!-- project will not visible.
          This flag will not accept arch or repository arguments. -->

What is the current status of the ACL implementation? The current status is that the complete API of the OBS git master had been instrumented with ACL code, critical portions of the API controllers had been code inspected and a big portion of these API calls now have a testcase in the OBS testsuite. Work is ongoing to make ACL as secure as possible. A code drop of current git master is under test in some bigger OBS systems, most notably the openSUSE Buildsystem. You can find snapshots of this codebase as usual in the OBS project openSUSE:Tools:Unstable. Adrian Schröter updates these “Alpha Snapshots” relatively often, on a 1-2 weekly basis, and runs the testsuite on git master daily. Thanks to Jan-Simon Möller for putting in many of the testcases into the testsuite for the ACL checks. On OBS Testing in general, read also Development and Test.

What is next? Code is tested and debugged against granting unwanted access due to some concepts inside OBS that are “working against ACL”, like project or package links, aggregates or kiwi imaging. We will inform you interested user of course about beta releases and an official 2.1 release.

Stay tuned.

The topic of this week was to speed up the build. As I mentioned last week, the
prototype in Base:build:arm is compiling in an emulated environment and thus is
quite slow.

I prepared a cross-toolchain from the sources of binutils, gcc44 and glibc from
openSUSE:Factory. Now its time to integrate this into the build process.
This will improve the performace of the compilation itself. But other areas
in the build process can also be improved, just think of the bash.

Some patches for the Build Service were also commited to svn.

In progress / todo:
* prepare / submit fixes for rpm
* more work on cross-toolchain