Home Home > Build-service
Sign up | Login

Archive for the ‘Build Service’ Category

osc: speedup update of a project working copy

June 26th, 2014 by

Hi,

recently, I pushed a commit that speeds up the update of an osc project
working copy, if most of the packages in the working copy are already up to
date (that is no update has to be performed).
The following table shows the improvements of the new code (in terms of
wall-clock time). Both project working copies were already up to date
and the packages in the home:Marcus_H project were unexpanded.

project       # number of packages  #    old code # new code
home:Marcus_H                   66        51.135s    10.653s
d:l:r:e                       1245     7:07.07min    17.804s

(the numbers for the devel:languages:ruby:extensions (d:l:r:e) project
were kindly provided by darix – thanks!).

Technically, we just reduced the number of http requests for packages
that are already up to date by using the backend’s getprojectsourceinfo
call (/source/project?view=info&package=pkg_1…&package=pkg_n).
Note: currently, such a reduction is not possible for packages that have
a _service file, because a small change in the backend is needed (see [1]).
Consequently, there are no time improvements for such packages.

If you want to test the new code, use the osc package from the
devel:tools:scm repo (http://download.opensuse.org/repositories/devel:/tools:/scm/).
Feedback is always welcome! :)

Next, my plan is to improve the speed of an update of a single package
working copy (again by reducing the number of http requests).

[1] http://lists.opensuse.org/opensuse-buildservice/2014-06/msg00067.html

First fruits – update on openQA and Staging work

February 19th, 2014 by

In our previous summary, we talked about some basic research and some ground work to build on. This time we have some first exciting results!

openQA

Last week we rearranged the repository a little bit, creating a new branch called "devel" where all the exciting (and not so exciting) changes are taking place. Our little Factory ;-)

The main difference between this branch as master is that, as you could read in the previous blog, the devel branch is openQA build on Mojolicious, a nice web development framework. And having a proper web framework is starting to show its benefits: we have openID login available! Unfortunately the current openSUSE openID provider is a little bit weird, so it doesn’t play well with our tool yet, but some others are working and openSUSE accounts will be the next step. Having working user accounts is necessary to be able to start defining permissions and to make openQA truly multiuser. And to be able to deploy the new version on a public server!

The other main focus of this week has been internal polishing. We have revamped the database layout and changed the way in which the different openQA components communicate with each other. The openQA functionality is spread out over several parts: the workers are responsible of actually executing the tests in virtual machines reporting the result after every execution; some client utilities are used to load new ISO images and similar tasks and, finally, we have the one openQA Server to rule them all. Until now, the communication between the server and the other components was done using JSON-RPC (a lightweight alternative to XML-RPC). We have dropped JSON-RPC in favor of a REST-like API with just JSON over plain HTTP. This change allowed us to implement exactly the same functionality in a way that is simpler, perfectly supported by any web framework, natively spoken by browsers and easier to authenticate (using, for example, plain HTTP authentication or openID). This is also the first step to future integration with other services (think OBS, as the ultimate goal is to use openQA to test staging projects).

But, who tests the tester? openQA is growing and changing quite fast so we have continued with the task of creating a good testing infrastructure to tests openQA itself to make sure that all our changes do not result in breakage. We only have a few tests right now, but we now have a solid foundation to write more and more tests.

Staging and package manipulation

In the last blog post we told you we were investigating a code test suite to test the abilities of a osc plugin we are writing. osc is the commandline tool for handling the Open Build Service, and this plugin is meant to help with the administration of staging projects. We’ve been thinking about how to move forward with the testing part as we want to make sure the functionality works as advertised. More important, we write tests to make sure that our additions and changes do not break existing functionality. We have started merging functionality from various scripts handling staging thingies and rings we had into this plugin. This is partially done so we can do basic staging project tasks! We can take a request (be it submit or delete) and put it to test in a staging project. We can also move packages between staging projects and we have a simple YAML in project descriptions to indicate what packages and what requests are there.

We're all green!
Coolo already started using the plugin for some tasks, so you can see pretty titles and metadata in staging projects descriptions. Not impressive enough? Let me provide you a good headline:

Thanks to staging projects, no single regression have been introduced this year in the core of Factory

You can enjoy a more detailed description in this announcement written by coolo to the Factory mailing list and have some visual joy in the included screenshot (genuine pr0n for QA engineers).

Last but not least, we also did some cleanup of the sources in the repo and of course we added more tests (as functionality grows). And there has been work on other parts of the plugin, like taking rings into account.

Result

We already have some useful functionality which we are going to expand and build on. Not that much to show yet, but we are progressing towards our goal. You can follow our progress either in the way of tasks (openQA and staging projects) or just follow our commit history on github (again for both openQA and staging plugin).

We are very much looking forward to feedback and thoughts – these changes aim to make Factory development easier and thus we need to hear from all you!

Some news from the trenches

February 7th, 2014 by

As you might know, we are focusing our development efforts in two fronts, namely openQA and staging projects. As we just started we don’t have fireworks for you (yet) but we did some solid ground work that we are going to build upon.

Working on openQA

We are organizing our daily job in openQA into highly focused sprints of two weeks. The focus of the first sprint was clear: cleanup the current codebase to empower future development and lower the entry barrier for casual contributors, which can be translated as “cleaning our own mess”. We created some tasks in progress, grouped in a version with a surprising and catchy name: Sprint 01.

Got my mojo workin’

Up to now, openQA web interface was written using just a bunch of custom CGI scripts and some configuration directives in Apache. We missed a convenient way to access to all the bell and whistles of modern web development and some tools to make the code more readable, reliable and easier to extend and test. In short, we missed a proper web development framework. We evaluated the most popular Perl-based alternatives and we finally decided to go for Mojolicious for several reasons:

  • It provides all the functionality we demand from a web framework while being lightweight and unbloated.
  • It’s stated as a “real time framework” which, buzzwords apart, means that is designed from the ground to fully support Comet (long-polling), EventSource and WebSockets. Very handy technologies for implementing some features in openQA.
  • It really “feels” very close to Sinatra, which makes Ruby on Rails developers feel like at home. And we have quite some Rails developers hanging around, don’t we? Just think about OBS, software.o.o, WebYast, progress.o.o, OSEM, the TSP app…
  • Mojolicious motto is “web development can be fun again”. Who could resist to that?

We’ve now reached the end of the sprint and we already have something that looks exactly the same than what we had before, but using Mojolicious internally. We are very happy with the framework and we are pretty confident that future development of openQA will be easier and faster than ever. OpenQA has mojo!

The database layer in openQA

Another part that we worked on during first sprint was the database layer. The user interface part of openQA use a SQLite database to store the jobs and workers registered in the system. The connection between the code and the database was expressed directly in SQL using a simple API.

We have replaced this layer with another equivalent that uses an ORM (Object-relational mapping) in Perl (DBIx::Class). Every data model in openQA is now a true object that can be created, copied and moved between the different layers of the application. Quite handy.

To make sure we don’t forget anything, we created a bunch of tests covering the whole functionality of the original code, running this test suite after each step of the migration. In this way we have achieved two goals: we now have a simple way to share and update information through the whole system and we can migrate very easily to a different database engine (something that we plan to do in the future).

What to do with staging

Over the time, coolo had accumulated quite some scripts that helped him with Factory. Most of them are actually related to something we are doing right now: the staging projects. So in the end we basically migrated all relevant ones to github and one by one we are merging their functionality to staging plugin. We also experimented with test frameworks that we could use to test the plugin itself, selected few and we even have a first test! The final plan is to have the whole plugin functionality covered with a proper test suite, so we will know when something breaks. Currently, there is a lot of mess in our repo and the plugin itself need big cleanup, but we are working on it.

Contributions are welcome

If you want to help but wonder where to start, we identified tasks that are good to dive into the topic and named them “easy hacks”: mostly self contained tasks we expect to have little effort but we lack the time to do right now. Just jump over the list for openQA or staging projects.

For grabbing the code related to staging project, you only need to clone the already mentioned repository. The openQA code is spread in several repositories (one, two, three and four), but setting up your own instance to play and hack is a piece of cake using the packages available in OBS (built automatically for every git push).

If you simply want to see what we are doing in more detail, take a look at progress.o.o, we have both openQA and Staging projects there.

We are having a lot of fun, and we encourage you to join us!

spec-cleaner: hide all your precious cruft!

January 31st, 2014 by

As we stated in our communication over the time, our team’s main focus for foreseeable future is Factory and how to manage all those contributions. Goal is not to increase the number of SRs that is coming to Factory, but to make sure we can process more and to make sure we see even well hidden consequences to make sure that Factory is “stable” and “usable”.

sprayg

Not really part of our current sprints, but something that will hopefully help us is spec-cleaner that Tomáš Chvátal and Tomáš Čech were working on lately during their free time/hackweek. What is it trying to address? Currently, there are some packaging guidelines, but when you write a spec file for your software, you still have plenty of choices. How do you order all the information in the header? Do you use curly brackets around macros? Do you use macros? Which ones do you use and which not? Do you use binaries in dependencies? Package config? Perl symbols? Package names? There is format_spec_file obs service that tries to unify a little bit the coding style but leaves quite some of the stuff up to you. Not necessarily a bad thing, but if you have to compare changes and review packages that are using completely different coding styles the process becomes harder and slower.

spec-cleaner is format_spec_file taken to another level. It tries to unify coding style as much as it can. It uses consistent conventions, makes most of the decisions mentioned previously for you and if you already decided for one way in the past, it will try to convert your spec file to follow the conventions that it specifies. It’s not enforcing anything, it’s standalone script and therefore you don’t have to be worried that you spec file will be out of your control. You can run it, verify the result (actually, you should verify the results as there might still be some bugs) and commit it to OBS. If we all do it, our packages will all look more alike and it will be easier to read and review them.

How to try it? How to help? Well, code is on GitHub and packages are in OBS. You may have a version of it in your distribution, but that one is heavily outdated (even the 13.1 version), so add openSUSE:Tools repo and try the version from there.

zypper ar -f obs://openSUSE:Tools/openSUSE_13.1 openSUSE-Tools
zypper in spec-cleaner

You can then go to some local checkout and try what changes does it propose for your spec file. Easiest way is to just let it do stuff by calling it and taking a look at changes afterwards.

spec-cleaner -p -i *.spec
osc diff

If it works, great, we will have more unified spec files. If it doesn’t, file a bug ;-)

Magic: It is now possible to use MS Silverlight based websites via pipelight

September 13th, 2013 by

It has long been a challenge to use MS Silverlight based websites on linux systems. Especially in The Netherlands this is a big hurdle as many (>80%) of the secondary school websites that pupils must use to communicate with their school (for homework, marks, etc) are equipped with Silverlight. Yes, really… :(

Fortunately at the end of August 2013 I discovered pipelight, a very smart idea to use MS Silverlight based website natively on Linux. The problem was however to find a working pipelight package for openSUSE. As there was none, I decided to build one myself using the incredible openSUSE Build Service. It was quite a quest to obtain a working package, but due to very good cooperation with the pipelight developers, I’m now able to present a working pipelight package to the openSUSE community. Oh, and while working on the package I reported a bug via the bug report system, that was solved and published via an rpm package within 1 hour after reporting it (that was during out of office hours). Indeed within 1 hour after reporting the problem it was; accepted, investigated, analysed, fixed, tested, handed over to me, packaged, tested and published! The amazing world of Open Source Software!

Pipelight works okay for the following sites (among many others): arte, LOVEFiLM, Netflix, Magister based NL schoolwebsites, WATCHEVER, etc. View the complete list on the pipelight website.

The installation instructions are on the pipelight website. Be aware though, that pipelight requires the wine package that is provided via the home:rbos:pipelight repository. With any other wine package, pipelight will (very likely) not work. If you rely on your currently installed wine package and installed MS applications and are unsure that the wine package provided via the home:rbos:pipelight repository will leave your currently in use MS applications untouched: don’t install pipelight (or only after making very good backups). You can always start by installing pipelight in a virtual machine.

Have fun with pipelight.

Keeping Factory in shape

June 13th, 2013 by

Michal Hrušecký has been helping out on maintaining Factory in shape and shares his experiences.

Factory is development version of openSUSE and it is where the next openSUSE is taking form. Hundreds of packagers send packages into Factory to be integrated as a part of the new release and many more use Factory for testing or for their daily work. Thus it is really important to keep Factory rolling and usable. Everybody knows that Coolo is the Factory master and he does everything to make next openSUSE be the best ever. But keeping factory in shape is really complicated and stressing task. There are dozens of request everyday and each one of them can potentially break something. So Factory can always use a pair of extra hands and for some time I have been one of them. I’d like to give you some insight in what we do, working on keeping Factory building and working. (more…)

openQA in openSUSE

June 6th, 2013 by

factory-testedToday, we’ve got for you an introduction of the teams’ work on openQA by Alberto Planas Domínguez.

The last 12.3 release was important for the openSUSE team for a number of reasons. One reason is that we wanted to integrate QA (Quality Assurance) into the release process in an early stage. You might remember that this release had UEFI and Secure Boot support coming and everybody had read the scary reports about badly broken machines that can only be fixed replacing the firmware. Obviously openSUSE can’t allow such things to happen to our user base, so we wanted to do more testing. (more…)

Tidy home, tidy Build Service

April 6th, 2013 by

Anyone using the Open Build Service in the last couple of weeks will notice how hopelessly overloaded it is.  I blame the ARM lads ;).  But there is something that we should all do as responsible community members: delete or disable your old stuff.

I’ve just spent 15 minutes going through my Open Build Service home:wstephenson project, deleting unfinished works in progress, finished branches that were never deleted, inherited repositories in branched projects that aren’t relevant to the tweaks I’m making, and I think I’ve saved about 100 repositories for the OBS scheduler to recalculate and look at publishing.

Time for you to spring clean too?

wxRuby is now on BuildService

March 22nd, 2013 by

I am happy to announce that i succeded in compiling wxRuby 2.0.1 on my Buildservice account and it is available to be installed in just one click for openSUSE 12.2 and 12.3.

wxRuby is an old but working library based on wxWidgets toolkit, till some day ago the dependency from SWIG 1.3.38 and some small errors raised during the manual compilation, made the use of this library the worst nightmare for beginners who was looking for a fast approach to GUI based programming in Ruby.

After some day spent to investigate about a possible upgrade of the SWIG dependency to the current 2.0 version, i produced some patches to fix this and the other annoying compiling errors, and finally, thanks to the Buildservice infrastructure, a wxRuby RPM compiled from sources with the relative patches are now availables for all openSUSE users!

As far i googled this should be the first distro to have a precompiled and working wxruby gem among its repositories (being compiled from sources the gem is generated for 32 and 64 bits architecture from Buildservice itself), so Rubyists take a look on software.opensuse.org, select the package coming from my home project account and enjoy!

openSUSE 12.3 Image available for ARM64 (AArch64)

March 5th, 2013 by

Howdy,

the openSUSE on ARM team was quite busy the last few weeks with getting openSUSE 12.3 for AArch64 (ARM64, also called ARMv8) ready. At the time of this post, we have finished around 4100 packages (out of ~ 6000) of openSUSE 12.3 built for AArch64, the ARM 64bit platform. With those successfully built packages, we’re also able to build a regular openSUSE image for you to try and run in the ARMv8 System emulator (ARMv8 Foundation Model).

This is a huge achievement and milestone for us, thanks to lots of helpful hands within openSUSE. Just to put this into perspective: This is not a minimal system with a couple of toolchain packages. It is also not an embedded variant of a Linux environment. No, this is the full featured, standard openSUSE distribution as you’re used to, ported to AArch64, up and running. We have built it based on (slightly newer versions of) standard openSUSE 12.3 packages, and the changes are mostly already merged back into openSUSE Factory. For all we know it’s also more successful package builds than any other Linux distribution has on AArch64! If you’d like to see the status yourself, please check out the OBS repository we created for this.

As an open distribution, it is important to make contributions easy and we worked hard to enable others to participate in our effort. We extended OBS to automatically spawn a Foundation Model virtual machine when you want to build for aarch64. This works remotely on the OBS server as well as locally using osc build. More information on this is available on the respective wiki page.

So, dive right into it:  Get the image and start with openSUSE on AArch64 by following our wiki page: https://en.opensuse.org/Portal:ARM/AArch64.