Home Home > Build-service
Sign up | Login

Deprecation notice: openSUSE Lizards user blog platform is deprecated, and will remain read only for the time being. Learn more...

Archive for the ‘Build Service’ Category

« Older Entries
Newer Entries »

Policy proposal for Factory: Make source of tar balls trackable

March 21st, 2011 by

I like to suggest a general policy for openSUSE:Factory project to document from where a tar ball (or any other file from upstream) is comming from. Why that ? It makes it much easier to review version updates and it guarantees that no one can inject some mal code via a modifed tar ball.

So far I added the source services “download_url” and “tar_scm” to our OBS instance, which downloads the files and stores them as files via a commit. Some people use them already, some others don’t like them because they store the files with _service: prefix.

In last hackweek, I added another way to handle this, which I would like to request as setup and policy for openSUSE:Factory project. You can add a project wide source service, for example the new “download_files” service. That would mean that no needs to add a _service file to the sources anymore. It is enough to add an URL to the spec file Source: tags. The service will automatically download it from there.

But that does mean we still have have _service:download_files:osc-0.1.tar.bz2 file names ? Not when we also add the new “trylocal” parameter and use latest osc versions. This parameter will let act osc to execute the services, but name the files without prefix and commit them together with the other files.

Where is the advantage then ? The server is still validating that this is an identical file. It downloads it again and compares it. In case it is the same file, nothing will happen.

What will happen, when the file differes ? We basically have two options, either we can let the service mark the source as broken or we would store the file with _service: prefix again.

The later mode has the advantage that you can still do version upgrades via slow connections and let the server download the files.

Please find some more details about new possibilities with the source services here.

An example setup for this can be tested via

osc bco home:adrianSuSE:FactoryTest bc

and do for example a version downgrade to 1.05 version to see how it works. Please note that you need the osc from openSUSE:Tools:Unstable project for this.

We can also apply the still suse-internal spec formater and validator scripts via this way later one.

Another advantage of this setup would be the new “update_source” service, which could run in some openSUSE:Factory:AutoUpdate project and tries automatic version upgrades when upstream releases a new version. They could be reviewed and just picked (directly or with additional manual fixes).

Posted in Build Service, Distribution, Factory | 1 Comment »

How to work with OBS via slow connections

February 9th, 2011 by

The openSUSE Build Service Books have a new chapter called HOWTOs. These will be written for asked questions, the first one existing is how to work best with OBS if you have limited bandwidth. There are multiple methods to avoid large downloads or even uploads of large files (like tar balls) and let just the server do the work…

Posted in Build Service | Comments Off on How to work with OBS via slow connections

new osc feature to edit a request

January 30th, 2011 by

Hi,

I just pushed a new osc feature to git master which allows you to edit a submit action. Use case: suppose you review a request (which has at least one submit action) and you find a small typo (for instance in the spec file) but except the typo everything is fine. So instead of declining the request you can fix the typo, create a new request (which contains the fix + the original changes), accept the newly created request and supersede the original request (that’s basically what osc does behind the scenes).

Example:

# request with id 80 needs a small fix
marcus@linux:~> osc rq show 80 –edit
Request: #80

submit:       home:Admin/foo  -> home:foobar/dest
delete:       home:foobar/xxx

Message:
deletes package xxx and fixes dest.

State:   new        2011-01-30T15:04:03 Admin
Comment: <no comment>
A    /tmp/osc_editsrr2iDcI/test.spec
A    /tmp/osc_editsrr2iDcI/src.tar.bz2
At revision 1.
Checked out package ‘foo.home_Admin’ to /tmp/osc_editsrr2iDcI. Started a new shell (/bin/bash).
Please fix the package and close the shell afterwards.
marcus@linux:/tmp/osc_editsrr2iDcI> # fix it and commit changes
marcus@linux:/tmp/osc_editsrr2iDcI> exit
exit
Request: #None

submit:       home:Admin:branches:REQUEST_80/foo.home_Admin(cleanup) -> home:foobar/dest
delete:        home:foobar/xxx

Message:
<no message>
d(i)ff/(a)ccept/(b)uildstatus/(e)dit/(s)kip/(c)ancel > a -m “accepted request and applied small fix”
Supersede original request? (y|N) y
marcus@linux:~>

By the way you can also do it manually (osc rq clone <id>; osc co <clone project>; fix package(s) and commit changes; create a new request, accept it and supersede original request).

Tags:
Posted in Build Service | Comments Off on new osc feature to edit a request

Kick off for GNOME:Ayatana Project…

December 29th, 2010 by

“Follow effective action with quiet reflection. From the quiet reflection will come even more effective action.” // Peter F. Drucker

This has been one of the guidelines in my life for quite some time… It started as a curiosity a long time ago with Notify OSD and evolved to full project in openSUSE. It is important to acknowledge at this point the motivation provided by the openSUSE GNOME Team from which I’ve been getting plenty of guidance and help, namely from Vincent Untz (vuntz) and Dominique Leuenberger (Dimstar). Thanks to them, we have now a GNOME:Ayatana Project on OBS (openSUSE Build Service), currently being populated with the support libraries for Ayatana’s Unity and Indicators.

Susan Linton has made a small article for Linux Journal about this project in the past. Though some people pointed to me that it was advertising and excelling Ubuntu… I would like to leave a statement… We’re not taking a hike on Ubuntu visibility, and it isn’t bad at all, on the contrary… In fact it will help Ubuntu, us and many others… specially if some Ubuntu patches are accepted faster by upstream. I hope other RPM distributions will follow the way we, openSUSE, proudly seem to pioneering! From my personal point of view… a distribution ‘distributes’… and despite this software isn’t attractive for some openSUSE users, I’m happy it is available (totally or partially) for all those who want to test it… Wait… you don’t even need to install Ubuntu or changing the platform you run!

Due to several reasons, being the most important of them versioning, this repository will start on the next release of openSUSE in March 16th (World day of Conscience, interesting point). This is also interesting as if YOU are willing to improve a package or submit a package you can now do it to this repository.

This goes with a very huge cookie for Dimstar and Vuntz for taking care of this repository and making sure that everything will comply with the openSUSE Guidelines. You are my personal heroes.

It has been quite an interesting experience to be with openSUSE GNOME team which is full of knowledge and helpful in many ways. I can’t also forget to mention that last week Luis Medinas has taken tutorship of a Portuguese contributor to openSUSE GNOME, João Matias and will provide him the necessary help to integrate him on the workflow of the GNOME Team. My personal thanks to Luis for stepping into this task, which from my personal point of view is very important.

Regarding to Ayatana it is worth to mention that Dimstar provided some valuable help in fixing the dbusmenu package and taking care of the necessary patch submission in GTK and gdk-pixbuf to allow dbusmenu to build with introspection support and generate properly the Vala files required for other packages. This handicap beaten… we’re on the good road for better functionality. This patches also allowed to correct some behavior in some indicators, one fine example of this the ‘Me Menu’ which now displays correctly a  ‘dot’ on the selected status as the screenshot bellow shows:

In the last days, despite it’s Christmas season and soon new year…. I’ve been also working on providing additional extensions to enable some functionality on some indicators. This was the case of indicator-sound, which provides an alternative sound gadget that offers extended functionality with multimedia players. A fine example of this is Novell’s Banshee player which has astonishing out of the box implementation with the sound indicator from Ayatana by a small extension that can be enabled. I’m still wondering why so many people toss heavy critics at this indicator calling it ‘mac styled’… while to be honest I doubt such people have even seen OSX sound applet, which is more or less a direct copy of the one present in GNOME, the vertical switch. Interesting view nevertheless.

Indicator-sound has also been fixed and no longer requires the nasty hack in the previous package. Since I’m not an Ubuntu user, neither I have extensive experience on their Desktop, I’m not sure if the functionality present so far in the indicators is the one offered by Ubuntu. I plan to run a Open Beta on the Ayatana software repository during the last Milestone of Factory to all Factory users to collect more data and improve the packages, at least the indicators, as in the present since I have ATI hardware I don’t have FireGL enabled on Factory, so I can’t really push much on Unity and test it for the time being.

Another subject of plugins was Xchat which is GNOME’s premier IRC client. Novell’s Evolution also got it’s plugin which is found to be partially working. I haven’t tracked yet if there are patch submissions to Evolution from Ubuntu to enable indicator functionality. That’s for sure one of the next steps, and since Evolution is a bit of ‘in-house’, would be nice to have them approved upstream if submitted already, as it would serve openSUSE as well.

Most of this applications, evolution, xchat, gwibber, empathy, pidgin are supported by indicator messages which collects information from several messaging services and places them on a single indicator in a cascade style experience for the user. I personally find it weird as I’m not used to this, but seems nice. Unfortunalty some applications seem rely on patches to be fully supported, like empathy. I hope upstream accepts the patches from Ubuntu (if submitted) and we can also benefit from such changes in our side.

Basically to sum up everything in a short review…

* Indicators are working fully or partially depending on patch level on some applications. If upstream starts accepting Ubuntu patches (some shouldn’t be much of a problem), it will work out for Ayatana software in openSUSE as well.
* Unity – Though it builds already with some compiz packaged from git sources (to include glib mainloop patching), I have no way of testing it, neither I have done integration on it. Unfortunately both my systems have ATI hardware and I have no way of testing it on Factory which seems to be too much bleeding edge for ATI to keep up. It’s stalled a bit, but in the worst scenario will be available a few weeks after the next official release of openSUSE.
* During this wait time… we might see a new release of compiz with the patches upstreamed by Canonical, which will for sure help us also a lot. No need for hurry in Unity at this time.

My personal experience with this project has given me lots of knowledge about OBS, hacking Makefiles and configure scripts… debug skills… and specially I ended up loving the way openSUSE is built and how it works. I would take also this opportunity to make a small statement… all the development so far has been deployed over GNOME 2.32. Much of the software packaged already supports GTK3 and should be easy to migrate it to GNOME3. At the moment, since the next openSUSE release is still based on GNOME 2.32, I’m not testing all this software on GNOME3. It might take a few days/weeks to have it available for GNOME3 after it’s release, though from a personal perspective, what I’ve seen on GNOME3 seems to be overkill! It’s damn nice and I have no doubt GNOME3 will succeed as the ultimate Free Desktop.

I would also like to mention that I’m not testing this indicators with KDE. In case someone wants to do this, please feel free to nag me on IRC (#opensuse-gnome @ Freenode) and leave feedback. I’m focused only on GNOME2 and GNOME3 deployment of this software packages, though I will help in whatever way I can if someone wants to work them out for KDE.

I’m using a patched version of Metacity (patches were submitted upstream) which improves the display of buttons by improving overlaying of images (I think). Look at the sharp corners of the theme in the pic bellow. As always, Faenza Icon theme with Canonical’s light theme Radiance (not hacked). This is another change I hope that goes upstreamed soon.

My sincere congratulations to everyone working on the awesome GNOME3, I’m sure it will be a success and make the delights of many! My faith points that GNOME3 will change Desktop user experience forever!

Have a lot of fun…!

Nelson Marques

Tags: , , , , , , , , ,
Posted in Build Service, Desktop, Distribution, Factory, GNOME | 16 Comments »

osc 0.130.1 (bugfix release)

December 18th, 2010 by

Hi,

I just released a new osc version: 0.130.1 (bugfix release).
The following issues were fixed:

  • don’t crash if a file marked as ‘A’ does not exist (bnc#658664)
  • fixed proxy handling (bnc#657958)
  • fixed repairwc (bnc#657838)
  • fixed build for python2.4

The new version is available in the openSUSE:Tools repo.

Posted in Build Service | Comments Off on osc 0.130.1 (bugfix release)

How we use our power

December 16th, 2010 by

I had a side project the last two weeks: Make the build service more fun to use.

No matter how much fun you have creating packages, if they don’t build, there is little point in using a Service that has Build in its name, no? So one of the major goals of the service is actually to help those that want to build packages as good as possible. But there is a problem:

Let me quote from the landing page of build.opensuse.org: “The openSUSE Build Service hosts 16,414 projects, with 107,691 packages, in 26,259 repositories and is used by 25,967 confirmed users.”. That are quite some high numbers – especially in the relation to the ~25 servers we have for actually building.

If you look at the build statistics of the last month (and this is just i586, x86_64 has around the same), you notice that there is not much purple in the “Busy workers / Idle workers” graphic:

OBS load

Every 2nd weekend or so we have some pause where the servers actually idle around, the rest of the time they are usually under full load and it’s not exceptional that we have over 20,000 build jobs for said 25 servers at the same time. So if Sue comes wants to build her packages at that time, she competes with quite some other packages and she gets frustrated to still see “scheduled” when she goes away. So we use some algorithms in the so called dispatcher to distribute the power to the right packages.

Over years of its existence the dispatcher used the algorithm I would dub “Randomness with exceptions” – it would if the job’s filename matches a regexp and if so, preferred it, otherwise picked a random job. Such algorithms create some fairness if you have 28.000 users all active at the same time, because there is usually not a really good balance between those.

But with 2.0 this changed: we got load and priorities. A script of mine parses the logs of download.opensuse.org and counts how many users are for the repositories. From that I calculate priorities, so that repositories of interest to people get more build power than others. E.g. KDE:Release:45 for 11.3 is downloaded 65 more often than for 11.1, so the 11.3 packages should see more attention. For that the build service calculates how many workers were busy for the repositories and then allows a factor to lower that load while picking the next repository to build for. This is much more complex than “pick a random one”, but it lead to faster return times for those projects that see actual downloads (and new projects as they have no load registered). To give some fun for those actually working on their packages, we also lower the registered load if we see commits.

But there was one problem left: with so many projects registered you also have quite some that aren’t interesting at all. They are not downloaded and very often not even their maintainers care, e.g. for some testing subproject they created in 2009 and then forgot about it. But those repositories build against the often changing openSUSE:Factory and see rebuilds because of that. Those repositories had a very low load because they have few packages, so they are often preferred over projects that have a lot of packages.

To free up more power to recent changes, we now experimented with various ways. It turned out to be useful to look at the relation between since last source change in the project and time since the jobs are “scheduled”. From this we calculate a staleness penalty – when the job is freshly scheduled, it’s basically one chance for a worker for 2 months since the last commit to the project. But this chance rises quickly, the penalty gets smaller the longer the job is in scheduled. Even those projects that have no source commits are valid “customers” and deserve to be rebuild against the latest gcc from openSUSE:Factory.

So what does this mean to you as user of the OpenSUSE Build Service?

  • Don’t add repositories unless you really plan to use it. I know that clicking the checkbox to also add all SLE versions is easily done, but remember this is build power and disk space on various mirrors you’ll be using
  • If you really care for your project, it’s a good idea to fix build failures from time to time. You’ll get more build power in return
  • If you plan to do a larger update of some package in your project and want to test the resulting packages building against it, it’s a good idea to disable all other repositories while you do so. The fewer build jobs you create, the lower is your load, the higher are your chances to get more build power
  • Make your repository popular by telling the world about it. More users means more build power
  • And last but not least: don’t get frustrated, remember there are almost 26000 other users

Posted in Build Service | 3 Comments »

Happy Birthday Scribus

December 9th, 2010 by

I rarely blog and even this one is merely  a link to another one, but: http://rants.scribus.net/2010/12/08/happy-birthday-scribus/ is worth a look. So, where is the connection to openSUSE ? Well, way back when, SuSE 9.0 was the first distro to really promote Scribus. 🙂

You can have the latest Scribus rpms for many distros, thanks to the awesome Build Service.

Enjoy!

Tags: , , , ,
Posted in Build Service, Desktop, Miscellaneous | Comments Off on Happy Birthday Scribus

osc 0.130

December 6th, 2010 by

Hi,

we just released a new osc version: 0.130.

The main changes/features are:

  • – new “revert” command to restore the original working copy file (without
    downloading it)
  • – rewrote “diff” logic
  • – added new “–http-full-debug” option, “–http-debug” filters the “Authentication” and “Set-Cookie” header
  • – added new “–disabled-cpio-bulk-download” option: disable downloading packages as cpio archive from api
  • – added new “repairwc” command which tries to repair an inconsistent working copy
  • – workaround for broken urllib2 in python 2.6.5: wrong credentials lead to an infinite recursion
  • – support –interactive-review option when running “osc rq list <project>”
  • – improved “osc rq show <id> –interactive-review”
  • – do_config: added new options –stdin, –prompt, –no-echo:
    –stdin: read value from stdin
    –prompt: prompt for a value
    –no-echo: prompt for a value but don’t echo entered characters (for instance to enter a passwd)
  • – added template support for a submitrequest accept/decline message
  • – lots of internal rewrites (new working copy handling etc.)
  • – support added for osc search ‘perl(Foo::Bar)’
  • – New “service” command to run source services locally or trigger a re-run on the server.
  • – setlinkrev is setting now the revision to xsrcmd5 by default to avoid later breakage on indirect links by default.
  • Features which requires OBS 2.1:
    support reliable diff for an accepted request

NOTE:
 Due to the rewrite of the working copy handling osc might fail with the following error:
Your working copy ‘.’ is in an inconsistent state.
Please run ‘osc repairwc .’ (Note this might _remove_
files from the .osc/ dir). Please check the state
of the working copy afterwards (via ‘osc status .’)

Simply run “osc repairwc” which might fetch files from the api or delete some files from the storedir (.osc/). It won’t touch locally modified files. For more information see section “WORKING COPY INCONSISTENT” in the README.

The new packages should be available soon in the openSUSE:Tools repo.

Thanks to everyone who contributed to osc!

Posted in Build Service | 1 Comment »

New Package for packager: whohas

November 16th, 2010 by

Sometimes a packager asked himself, who has already packaged this Software? Maybe the Packagingfiles can help me to fix a error? Or maybe an other packager has a written a patch that i can use for my situation?
Philipp L. Wesche knows this situation, and he wrote a program, that allows to view in other Distributions and Repositories, who has a specific Software packaged. The commandline tool “whohas” supports Arch, Debian, Fedora, Gentoo, Mandriva, openSUSE, Slackware, linuxpackages.net, Source Mage, Ubuntu, FreeBSD, NetBSD, OpenBSD, Fink, MacPorts and Cygwin. Philipp wrote this tool in Perl and was designed to help package maintainers find ebuilds, pkgbuilds and similar package definitions to learn from.

The Tutorial from the Autor can found at: http://www.philippwesche.org/200811/whohas/intro.html

You can download this tool on: http://download.opensuse.org/repositories/openSUSE:/Factory:/Contrib

Tags: , ,
Posted in Build Service, Packaging | 1 Comment »

OBS 2.1: Status of SuperH (sh4) support with QEMU

October 24th, 2010 by

With established ARM support in OBS the as well as emulated MIPS and PowerPC is getting more mature, the last big embedded architecture not working in OBS with QEMU user mode was SH4. QEMU developers community had done a lot of work in improving QEMU user mode during the last months, so I can proudly present with currently only a few patches to QEMU git master OBS builds working with the SH4 port of Debian Sid. The new QEMU 0.13 released recently is a big milestone for this.

Another news is that I had fixed the bugs in Virtual Machine builds (build script) when using them with some architectures like PowerPC 32bit and SH4. So now also the combination of using for example KVM (XEN should also work) in a worker together with ARM, MIPS, PowerPC and SH4 is working. The appropriate fixes are in one of the next build script releases (if not even released already now with OBS 2.1, I have to check that). You can select architecture “sh4” with OBS 2.1 and also start a scheduler with “sh4”.

With the use of the QEMU User Mode, you can build also accelerated native cross toolchains for your host architecture so time critical parts like the compiler can run without the emulator. This works with .deb as well as with .rpm based backages. The MeeGo Project as well as the openSUSE Port to ARM uses this technique to provide an optimum between compatibility and performance. It means you can mix natively build packages and use cross toolchains on it. The “CBinstall:” feature helps you to use native or cross builds automatically depending on if your build host is a native machine or a x86 machine with cross build. In summary, we have the current classics of linux embedded archs together now in OBS: ARM, x86, MIPS 32, PowerPC 32 and SH4.

I have uploaded the fixed QEMU package to the OBS project openSUSE:Tools:Unstable inside the package “qemu-devel” after some more testing. I have of course also a OBS meta prjconf file working with Debian Sid. The SH4 port of Debian Sid you can find at Debian Ports Site.

And last but not least I would like to thank Riku Voipio of the Debian Project, QEMU project and MeeGo project and other major contributors during the QEMU 0.13 development cycle for the restless work on QEMU user mode improvements. In case of KVM, QEMU is used even twice, with QEMU-KVM as well as QEMU User Mode. I am sure I had forgotten other important people, so thanks to them also.

Posted in Build Service, Distribution, Packaging, Toolchain, Virtualization | 3 Comments »

« Older Entries
Newer Entries »