I’ve blogged already about the work done by Coolo and Michael on reducing the size of factory updates.  Looking into some  of the packages that did not get filtered out due to timestamps, I decided to use that as my hackweek project – and learn some more about the openSUSE Build Service.

Coolo told me that I could create a small project in OBS with packages I wanted to look at and with my testing version of build-compare.  So, I created a project and set it up to build against openSUSE:Factory and linked some packages from factory into the project.

Here are some of the packages and problems  I looked at with their solutions – and each of the package stands for a larger number of packages (or was a package that is so large that it made sense to look at it):

emacs-vm, psgml

Emacs Lisp files contain a timestamp:

compare /.build.oldpackages/psgml-1.3.2-241.1.noarch.rpm /usr/src/packages/RPMS/noarch/psgml-1.3.2-241.2.noarch.rpm
/usr/share/emacs/site-lisp/psgml/ke-docbook.elc differs ( Emacs/XEmacs v20 byte-compiled Lisp data)
--- /tmp/tmp.8RceRNRkv1 2009-07-18 14:26:04.000000000 +0000
+++ /tmp/tmp.o9McQ4uZGD 2009-07-18 14:26:04.000000000 +0000
@@ -1,7 +1,7 @@
 00000000  3b 45 4c 43 14 00 00 00  0a 3b 3b 3b 20 43 6f 6d  |;ELC.....;;; Com|
 00000010  70 69 6c 65 64 20 62 79  20 61 62 75 69 6c 64 40  |piled by abuild@|
-00000020  62 75 69 6c 64 31 36 20  6f 6e 20 53 75 6e 20 4a  |build16 on Sun J|
-00000030  75 6e 20 32 31 20 30 35  3a 35 38 3a 35 30 20 32  |un 21 05:58:50 2|
+00000020  62 75 69 6c 64 32 30 20  6f 6e 20 53 61 74 20 4a  |build20 on Sat J|
+00000030  75 6c 20 31 38 20 31 34  3a 32 35 3a 35 36 20 32  |ul 18 14:25:56 2|
 00000040  30 30 39 0a 3b 3b 3b 20  66 72 6f 6d 20 66 69 6c  |009.;;; from fil|
 00000050  65 20 2f 75 73 72 2f 73  72 63 2f 70 61 63 6b 61  |e /usr/src/packa|
 00000060  67 65 73 2f 42 55 49 4c  44 2f 70 73 67 6d 6c 2d  |ges/BUILD/psgml-|

Now build-compare ignores that timestamp line and only looks at the rest.

gnome-bluetooth

gnome-bluetooth had a diff in the debuginfo package due to dbus-1-glib marshaller generating temporary files that had in a comment the name of the temporary file:

compare /.build.oldpackages/gnome-bluetooth-debugsource-2.27.5-1.18.x86_64.rpm /usr/src/packages/RPMS/x86_64/gnome-bluetooth-debugsource-2.27.5-1.19.x86_64.rpm
/usr/src/debug/gnome-bluetooth-2.27.5/common/bluetooth-agent-glue.h differs ( ASCII C program text, with very long lines)
--- old//usr/src/debug/gnome-bluetooth-2.27.5/common/bluetooth-agent-glue.h     2009-07-18 14:36:34.000000000 +0000
+++ new//usr/src/debug/gnome-bluetooth-2.27.5/common/bluetooth-agent-glue.h     2009-07-20 12:49:25.000000000 +0000
@@ -53,7 +53,7 @@
 #endif /* !G_ENABLE_DEBUG */

-/* NONE:BOXED,UINT,UCHAR,POINTER (/tmp/dbus-binding-tool-c-marshallers.T26RXU:1) */
+/* NONE:BOXED,UINT,UCHAR,POINTER (/tmp/dbus-binding-tool-c-marshallers.UU2NXU:1) */
 extern void dbus_glib_marshal_bluetooth_agent_VOID__BOXED_UINT_UCHAR_POINTER (GClosure     *closure,
                                                                               GValue       *return_value,
                                                                               guint         n_param_values,

So, instead of changing build-compare, I fixed dbus-1-glib and that fixed this class of problems.

.a files and jar files

ar archives (.a files) and jar files both store the timestamp of the files in the archive.  The build-compare script had some code that unpacked and then compared the files but that one was broken for both so I fixed these.  For jar files, I also had to find a portable way to unpack them.

Then I’ve been confused why the Java program “jar” failed to run during the compare.  Rudi helped debug it and we found that the build script did not mount the proc filesystem for the invocation of the compare step.  After mounting it, everything went fine.

xorg-x11-fonts, ghostscript library

The fonts.dir, fonts.scale and encondings.dir files are not sorted so they are dependend on the order the files lay in the directory.  Now build-compare compares the sorted files.  With xorg-x11-fonts there’s still sometimes a problem where two fonts have an identical font name – and only one of them gets mentioned in fonts.dir.  This is now a bug against the xorg-x11-fonts package.

texlive

After Werner fixed one occurence where a temporary file was packaged, I fixed another one – and then tried to handle the tex format files but was not really successfull.  This is bad since it’s a 266 MB package.  Right now my best idea is to wait for Richard to finish his rpm build number support so that we can filter out not only complete packages with subpackages (as we do right now) but also single subpackages.  Then it would be possible to only publish the changed subpackage – and we could put all those files that change and are not easy to compare in a separate smaller package so that only this one gets published.

Perl and Java Packages

Perl writes a timestamp in one of its files and javadoc produces also a timesteamp every time it’s run.  So, I added a couple of perl and java packages to my repository for testing the changes I did to the build-compare package.

Build Service Help

With having setup my packages in the build service, it did the following quite often:

• Change the build-compare package. I’m using the osc command line client, so this was a “osc ci -m ‘xxx'” command.
• Wait for the build to finish
• rebuild either the complete project (only seldomly done) or just those packages that my change would effect (“osc rebuild home:a_jaeger:branches:openSUSE:Factory somepackage”)
• Wait for the builds to finish – that gave me time to do other work or discuss with others
• Check that the Live Build Log of the package at the end contains the success line:
Retried build at Wed Jul 22 10:36:33 2009 returned same result, skipped
• If there was no success, figure out why and start again.

I setup the OBS project to use only one architecture (x86-64).  Initially I did not use debuginfo packages but enabled them later to check also the dbus-1-glib marshalling.  I also changed the project config to not publish the resulting rpms, there’s really no sense in that.  With just building what I needed, I also got faster turn-around times.

The build service did the building and comparing for me.  My job was reduced to the bare essential – and some time waiting for builds to finish.  With the old SUSE autobuild, I would have checked in build-compare and rebuild the whole distribution several times until I got everything right (and that would take weeks) – or build packages by hand and compare them manually.  So, the OBS way helped me really good with this project.

How can you Reduce Updates in Your Build Service Project?

AFAIK the build-compare package is used whenever it’s available, so if you build against openSUSE 11.1 which does not have the package, you can still make use of it with linking the package from openSUSE:Factory (or better aggregate it) – and the amount of published package changes gets reduced.  This is interesting for larger projects like the GNOME and KDE unstable projects.

Next Factory Update Will be Large

Note that the next update will be really large since Richard’s change for rpm went in so that debuginformation packages for each subpackage got in – and build-compare notices that there are a different number of subpackages and therefore will mark the new package as different.  Also the newly submitted gcc 4.4.1 generates slightly different code.  Right now only 17 per cent of the packages got filtered out.  Without these changes I would expect more than 70 per cent to get filtered out.

Both comments and pings are currently closed.

• Tags:No tags available
• Category: Build Service · Factory · Hackweek