Home Home > Quality-assurance
Sign up | Login

Deprecation notice: openSUSE Lizards user blog platform is deprecated, and will remain read only for the time being. Learn more...

Archive for the ‘Quality Assurance’ Category

CLI to upload image to openstack cloud

April 18th, 2012 by

I work on automatic testing of one of our products that creates other projects.
And because there is a lot of clouds everywhere I want to use them too. We
have internally an OpenStack cloud (still Diablo release). So I need to solve
automatic uploading of images built in the Build Service. Below I describe my working version.

(more…)

Where our bugzilla needs improvement

April 15th, 2011 by

update 2012-02-07: Success: after bnc#732504 you can now open Advanced Search and find RESOLVED/DUPLICATE bugs by default. When bugs are finally solved and fixes released, those can then be moved to CLOSED state to no more appear in search results.

 

bugzilla.novell.com aka bnc is the central tool to track bugs of openSUSE.

It has a guided bug submission form that helps & encourages reporters to search for existing reports on an issue.

However, the integrated search function only shows open bugs. In principle that is nice, but bugs marked as duplicates of open bugs do not count as open. Also it is common that developers mark a bug as RESOLVED/FIXED as soon as they uploaded a patch into the OBS devel-project. Such a patch then needs some days until it gets into Factory or the Update repos where normal users would benefit from the fix. During that time, there will still be people with all regular updates installed hitting the bug and not finding it on bugzilla because it is no more marked open. Of course, one could also search for RESOLVED bugs, but this brings up a huge list of issues that are long solved, but never were marked CLOSED (e.g. openSUSE-11.3 has 453 CLOSED vs 2727 RESOLVED).

This wastes time of reporters writing a duplicate bug-report and wastes time of developers having to figure out that it really is a dup. This is probably why other projects recommend setting bugs to ASSIGNED/FIXED until the patch is released. Unfortunately bnc lacks this state.

Also bugzilla has an UNCONFIRMED state for new bugs that need triaging to get into the NEW state (named “Reproducible but not assigned” in other projects). Such a state is unavailable on bnc, so that people can not tell apart bugs that have been forgotten from bugs that are known to exist, but just wait for a developer to fix.

There was also little integration between OBS and bnc. I have therefore written scripts to update bugzilla entries with links to submit requests on OBS mentioning e.g. bnc#685133 (<=see link for how it looks like). But it still could be made a lot more useful.

Overall, there is some room for improvement to make people’s lives easier, and allow us having a lot of fun…

openQA is testing for you

January 20th, 2011 by

You might have read the announcement of the factory-tested repo (or not, because it did not go on planet.o.o). There have been many additions to the testing during the last eight weeks.

openQA.o.o is a machine running automated software tests all the time to decide if factory-tested can be updated. If you think, this was boring, you might find out that it is not, since every testrun generates a video of approx 4 minutes that can be conveniently viewed in firefox.

You can view those test results on the openQA page.

And those are the updates:

  • I have added testing of development repos, so that it is now possible to detect bugs before they even get into Factory (most of those tests have “devel” at the end of their name). In fact, this has already found two such bugs.
  • The test system’s hardware was upgraded with a sponsored SSD and the software adapted to allow running multiple tests in parallel (using make -j3)
  • I have updated the web-interface to use the nice Bento theme
  • And I found yet another cool use-case for automated testing: it allows to bisect bugs. I did a binary search in old results for a live-installer bug which made it really easy to find the causing commit. However, seeing how long such a bug went unnoticed, made me think, that it would be nice if more people would check the results for anomalies, since not all of those can easily be auto-detected. As this just needs a firefox and little time, this is pretty easy to do. Much easier than to actually setup a test-system in any case.
  • An IRC notification bot was added, keeping #opensuse-openqa updated with new test results
  • Then, I am looking for additional automatic test-scripts, that also work standalone on an openSUSE install. It is especially important to test high-impact things. Things that break often or block many users if they break. And it would be nice to have a small result range such as OK|fail|unknown or it should at least be mappable to such.
  • Jürgen Weigert has packaged sikuli, so it is easy to create GUI tests with it, that could then be included in the test-suite.

Then there are still some things on the ToDo list…

  • Hermes integration to get email alerts and RDF feeds for newsreaders
  • Have more test variants auto-scheduled (-live and -dup)
  • Find a way to have Add-On-Repos with custom priority (otherwise Factory packages might be used, as happened for kernel-default-2.6.37-rc7) – using linuxrc’s driverupdate feature could be one way.

your input is highly appreciated

Announcing factory-tested

November 29th, 2010 by

Good news everyone! There is something new that is better than Factory and Milestones.

You might have heard users complaining about openSUSE Milestones being alpha quality and RCs/GMs being beta quality only.

You might also have heard developers complaining that people only start testing RCs, which is too late to fix most bugs (some users don’t know that).

You might also have heard geeks calling for rolling releases to always have bleeding edge software.

Luckily, all of the above can be helped.

I have been thinking about this situation and how to improve it.
The main problem is, that Factory is sometimes broken. So how can we make testing new software safer, even between milestone-releases (which are manual factory-snaphots)?

There is now a factory-tested repository that only gets updated when tests show that factory meets the following minimum requirements:

  • The system can be installed,
  • Zypper can install software,
  • And X11 can be started – at least on KVM

While this can not save users from all major bugs, it is already a big improvement to current Factory which gets released whenever it builds. So, why don’t you give it a try?

For more info on test-results (including videos), you can also have a look at the openQA page.

BugDay

November 22nd, 2010 by

At the last openSUSE project meeting and after the discussion about “zombie” bugs on the opensuse-project mailing list, a small team of volunteers agreed to organize a Bug Day on Saturday, November 27th. What is a Bug Day? This is a day when many people from the community help to triage bugs in Bugzilla. It is a good and easy way to get involved in the openSUSE project!

Here is what you need to participate:
– a recent version of openSUSE (11.3 or a milestone of 11.4). It’s okay to run openSUSE in a virtual machine.
– an IRC client to interact with the other participants
– good mood 🙂

A small team will organize the event by providing lists of bugs, and will be available to guide new contributors if needed. So it will be easy to help!

For this specific Bug Day, we will focus on the “zombie” bug reports: those are reports against old versions of openSUSE (openSUSE 10.x and 11.0). As some reports might still be valid, we don’t want to close all of them automatically. We will therefore check all those reports to see if they are still valid in the latest version of openSUSE (11.3 or a milestone of 11.4). The goal is to close those bug reports if possible, or, if they are still valid, to move them to a current version of openSUSE so that they’re not lost in limbo. So during a full day, people come on irc and help each other triage bugs.

Please note that this is only for openSUSE bugs (living in bugzilla.novell.com), but a solution for some bugs might be to forward them upstream.

Come on #opensuse-bug (freenode) on a Saturday 27.11.2010, we’ll be glad to have you join the fun! 😉

Checking EPUBs

October 3rd, 2010 by

EPUBs are getting more and more important thesedays. If you believe the essays from well-informed magazines, they will develop into a standard for book and text consumption as MP3 did for audio.

(more…)

OBS 2.1: ACL Feature and Status

August 15th, 2010 by

One and a half year is now gone since I posted about my work for ARM support in the OBS and the work for a port of openSUSE to ARM. Lots of things had happened in the meantime that are related, from my limited view most notably Nokia and Intel joining Moblin and Maemo to MeeGo (MeeGo is currently working on a number of Atom and ARM based devices), chosing to use OBS as build system and last but not least myself joining The Linuxfoundation (you will be not surprised to hear that I work at LF on OBS). In the meantime there had also been a major new OBS release 1.8/2.0 with a bunch of new features.

Interesting is the fact that we adapted the cross build system for OBS to MeeGo, first developed for use in Maemo and openSUSE @ ARM. An improved version for the standard MeeGo releases, and for the MeeGo weekly snapshots is used in the MeeGo OBS System to build all ARM releases of MeeGo (the cross toolchain will later get part of the MeeGo SDK @ ARM), thanks to Jan-Simon Möller (In the openSUSE ML, the issue of reactivating openSUSE:Factory ARM builds were brought up. So it might be a good variant to backport Jan-Simons new solution back into openSUSE @ ARM for that purpose). All the MeeGo related OBS installations will move sooner or later to OBS 2.1.

But now to the most recent work, Access Control support. A preview was shipped with OBS 1.8. Now an own OBS version, 2.1, will be dedicated to the introduction of this single new feature into the OBS mainline: Access Control (or abbrevated ACL for Access Control Lists). ACL means that there is control by the user on a per project or per package basis to protect information, source and binaries from the read access of other users in an OBS system and to hide projects or packages.

What is the intended audience of ACL? ACL is intended for installations of OBS that require protection of projects or packages during work. This can be but is not limited to commercial installations of OBS, or semi public installations of OBS.

How does ACL work? ACL sits on top of two features introduced with OBS 2.0: Role and Permission Management as well as freely definable user groups. ACL uses 4 specifically defined permissions (‘source_access’ for read access to sources, ‘private_view’ for viewing package and project information, ‘download_binaries’ for read access to binaries and ‘access’ permission to protect and hide everything and all from read access and viewing) on a user or group in the Role and Permission management. Also, the preexisting roles “maintainer”, “reader” and “downloader” had been modified with specific predifined permissions (which can at any time changed with the role and permission editor dynamically). And last but not least 4 new flags (namely ‘sourceaccess’ to signal a project/package has read protected source code, ‘binarydownload’ to signal it has read protected packages, ‘privacy’ to signal information/logfiles or status cannot be read and ‘access’ to hide and protect a project or package completely in all possible OBS API calls) had been added to the project and package descriptions to signal that some information is only readable by specific users or groups, or that information is hidden.

How do I use ACL? There are 4 steps to use ACL (a part of them a optional and can only be performed by the Administator of an OBS instance). Step one is to assign the listed permissions to a role, user or group (this step can be done only by the admin, and is not needed for the predefined roles “maintainer”, “reader” and “downloader”). Step two is to add a group for special users to projects which are intended to be run with ACL (this operations can only be performed by the admin). Step three is to protect a project with appropriate protection flags at project creation by adding them to the project meta. Step four is to add other users or groups with one of the new predefined roles that has ACL permissions added to the project meta.

What information can be protected by ACL? The protected information is grouped into 4 categories. Category 1 (flag ‘sourceaccess’) is source code. Category 2 (flag ‘binarydownload’) is binary packages or logfiles or builds. Category 3 (flag ‘privacy’) is project or package information like build status. Category 4 (flag ‘access’) is all viewable or accessable information to any project or package (full blocking of all access and information).

Example of a project configuration using ACL:

<user userid="MartinMohring" role="maintainer" />
<!-- grant user full write and read access -->

<group groupid="MeeGo-Reviewer" role="maintainer" />
<!-- grant group full write and read access -->

<group groupid="MeeGo-Developers" role="reader" />
<!-- grant group full source read access -->

<group groupid="MeeGo-BetaTesters" role="downloader" />
<!-- grant group access to packages/images -->

  <sourceaccess>
    <disable/>
  </sourceaccess>
  <!-- disable read access - unless granted explictely.
          This flag will not accept arch or repository arguments. -->

  <binarydownload>
    <disable/>
  </binarydownload>
  <!-- disable access - unless granted explictely -
          to packages/image and logfiles -->

  <access>
    <disable/>
  </access>
  <!-- disable access - unless granted explictely-,
          project will not visible or found via search,
          nor will any source or binary or logfile be accessable.
          This flag will not accept arch or repository arguments. -->

  <privacy>
    <enable/>
  </privacy>
  <!-- project will not visible.
          This flag will not accept arch or repository arguments. -->

What is the current status of the ACL implementation? The current status is that the complete API of the OBS git master had been instrumented with ACL code, critical portions of the API controllers had been code inspected and a big portion of these API calls now have a testcase in the OBS testsuite. Work is ongoing to make ACL as secure as possible. A code drop of current git master is under test in some bigger OBS systems, most notably the openSUSE Buildsystem. You can find snapshots of this codebase as usual in the OBS project openSUSE:Tools:Unstable. Adrian Schröter updates these “Alpha Snapshots” relatively often, on a 1-2 weekly basis, and runs the testsuite on git master daily. Thanks to Jan-Simon Möller for putting in many of the testcases into the testsuite for the ACL checks. On OBS Testing in general, read also Development and Test.

What is next? Code is tested and debugged against granting unwanted access due to some concepts inside OBS that are “working against ACL”, like project or package links, aggregates or kiwi imaging. We will inform you interested user of course about beta releases and an official 2.1 release.

Stay tuned.

Bugs will not get fixed by themselves

August 5th, 2010 by

I received an email from a user who switched from openSUSE to Ubuntu since his Wireless netcard did not work. It worked with openSUSE 11.2 initially but after an online update it failed.  He hoped that openSUSE 11.3 worked, tested it, it failed – and he gave up and wrote a frustrated email.

I was frustrated reading this since we should have been able to help this user if he contacted us in time.

Such a regression is bad but if nobody reports the regression, then it will not get fixed at all. The openSUSE project takes fixes from the upstream projects and also adds fixes ourselves and sends them upstream. Those fixes work on the system of the developer – or the systems of the upstream developers – but nobody has access to every single hardware that a chip supports, so regressions might happen. In the past I’ve seen that such regressions that are reported with a pointer to the exact version that failed, are often fixed quite fast.

(more…)

Improve Software Quality

June 12th, 2010 by

Today I and some hundred others on LinuxTag in Berlin attended a keynote by Mark Shuttleworth, the “head dreamer” of the ubuntu Linux distribution.

He had pretty few slides with hardly any words on it. The headlines were “Cadence”, “Quality” and “Design”. As I have been working towards openSUSE quality, this interested me most – so this is what this text is going to be about. His thoughts were mostly what I was thinking anyway.

Quality works like this: if Factory (equivalent of Debian/sid or whatever it is called in ubuntu) breaks horribly every other week, few people will feel inclined to keep running on Factory or even on factory-snapshot. Mark spoke about a difference in number of testers of “one or two orders of magnitude” (which would mean a factor of 10-100 in mathematical terms). And having fewer testers during development can only mean lower quality for the final release.

This is why efforts should be taken to ensure a working Factory. Automated testsuites and reviews were specifically mentioned, among others. Some packages like gcc include own test-suites, but most do not. When asked how that was handled in ubuntu, he described an approach with scripts using computer vision, recognizing/clicking buttons which most likely referred to sikuli which has many similarities to what I did, just with a different focus (more user-friendliness) and quite some sophistication. Also for text-mode there is still good old “expect”.

Another thing that was not mentioned there, but which I was always thinking of as an odd shortcoming of openSUSE is that Debian has a “testing” distribution which is not like factory-snapshot. Debian/testing contains only packages migrated from sid after a certain time with no bugs above a certain criticality threshold. Of course, this needs a way to track automatically which bug applies to which package (something that ubuntu does in launchpad).

Launchpad is also nicely integrated with bazaar, so that adding “(LP: #12345)” into a changelog will cause the bug to have a link to the proposed fix and later be automatically closed, once the branch was merged into the main trunk.

This is not only cool, but plain useful. It can save a lot of work and frustration. I think Adrian Schröter will be working into that direction with better integration of buildservice into bugzilla and other openSUSE tools.

There is also a big human component. Especially with testers which are non-technicians. It is not so much about motivating them, but about not demotivating them (which can happen surprisingly easy in some cases). I have seen crash messages along the lines of “xxx has crashed. This is in no way your fault. You could help us by …” which is a very good thing in this regard.

Yet another related note: there are openSUSE-Promo and openSUSE-Biarch DVD iso variants which are so well hidden, that few people even know about their existence, so that there can not be much public testing. However, those are the variants that are given away to hordes of interested people at openSUSE booths around the world, but those are also the variants that are least tested. I still have bad memories of the 11.1 or 11.0 one which was degraded to coaster after three unsuccessful tries on different machines. If bandwidth or mirror-space is a problem, offering them only via bittorrent could be a solution.

Overall, still some way to go, but IMHO we are moving into the right direction.

Road to 11.3 : when pattern are not your friend, pre selection can be a trap

June 10th, 2010 by

So it’s time to take some hours to test our future version.

Today I start a fresh M7/Factory install : booting from pxe. The test case is build quickly a minimal server text mode.

Just uncheck the auto configuration, we are after all linux admin. Choose your partition keyboard, language (en recommanded for server) etc … normal.

Just before starting the install check software :  click on installation resume . You will discover that base-system-pattern would like to install a kernel-desktop, wtf why we want a server !

So there’s a new ticket about that : https://bugzilla.novell.com/show_bug.cgi?id=613216

I express the fact that it would be nice to have a new pattern selected when we choose minimal install server text mode.

And you what about your opinion about pre-selection or having a base-system-server pattern … Please comment, & vote on bugzilla

A pattern guru wanted to build a patch for that.