Home Home > Quality-assurance
Sign up | Login

Deprecation notice: openSUSE Lizards user blog platform is deprecated, and will remain read only for the time being. Learn more...

Archive for the ‘Quality Assurance’ Category

automated openSUSE testing

May 25th, 2010 by

Testing is an important task. But testing daily openSUSE-Factory snapshots would mean testing the same things every day. This would be pretty tiresome to people.
And there is a lot of software to test, including software unknown to most testers or new versions of known software, so how should the tester know if the results were the intended results?
My answer is: leave as much as possible to computers. Computers do not get tired. Computers do not stop testing something after a dozen identical results. Computers do not forget.

The following assumes that you have read my text on making openSUSE install videos.

So far, I am rather satisfied with my automated installations.
At the end of those, I added some basic application testing, which already showed in MS7
openSUSE-KDE-LiveCD-x86_64-Build0625a.ogv dated 2010-05-21 16:08
an issue filed 28 hours later at bnc bug 608087

Only that it currently still needs a human to look at the results.
I was thinking to improve upon that by scanning (rectangular) parts of the screenshot for known good or bad images. If either is found, the test could be automatically marked as passed or failed.
On unknown images, a human would still need to decide which part of the image is relevant and if it is good or bad. This decision can then be used to avoid human interaction (hard work) in further runs of that test.
If we push this further, it could be similar to nagios for network monitoring. Telling when something breaks and telling when something is back working. It could have an overview page about automated test status, giving totals e.g. “50 working, 10 unknown, 3 failing”. With links to more details.

The advantage in adding the application tests after the install test is that the system starts out in a clean, reproducible way. One disadvantage I see is that a newly failing test could prevent following tests to work.

I have also been working to enable others to run my isotovideo script. For that I have cleaned up my code so that it no more contains paths from my system. The other thing is that I documented how to get it working at http://www3.zq1.de/bernhard/git/autoinst/INSTALL

MS7 installation videos:

openSUSE-KDE-LiveCD-x86_64-Build0625c.ogv
openSUSE-KDE-LiveCD-i686-Build0625a.ogv
openSUSE-GNOME-LiveCD-x86_64-Build0625b.ogv
openSUSE-NET-x86_64-Build0623b.ogv
openSUSE-NET-i586-Build0623b.ogv
openSUSE-DVD-Build0625-x86_64b.ogv
openSUSE-DVD-Build0625-i586b.ogv

making of the openSUSE install video

April 29th, 2010 by

Videos of software can have many good uses. You can have tutorials or a quick-start how-to illustrated by a video that shows the steps you need to go.

However, creating such videos takes some effort. Especially hard to make is a video of an operating-system installation, which includes a bootloader and other low-level software where you can not run capture-software on the OS.

Being a member of the Testing Core Team, I wanted to automate the typical openSUSE installation workflow up to the point where the installed system first started, because if installation media would not get to that point, it could prevent testers to try the remaining software.

To do this automation, I employed KVM (the much fancied Kernel Virtual Machine). As it is a variant of QEMU, it also has all those many useful options to adjust its behaviour to ones liking. One of those options allows KVM to be remote-controlled via the management-console. You can (among others) send key-presses, move/click the mouse and capture screenshots.

Putting all this together, I got a series of many images. So the next logical step was to run ffmpeg2theora on those images to create with free tools and free codecs a .ogv video file that has an even smaller size and better quality than mpeg[124]. Plus it can be easily viewed in firefox&co at high speed.

The results can be seen on http://www3.zq1.de/bernhard/mirror/opensuse/video/

This is still work in progress. KVM might have issues. e.g today I found, that I could speed up the network from 290kB/s to over 30MB/s by changing the emulated network device model from e1000 to virtio. With this, a network install from a fresh proxy cache took only 34 minutes instead of 95.

There is some more info in  http://lists.opensuse.org/opensuse-factory/2010-04/msg00439.html

(note that my initial test-runs were flawed because they used old MS5 packages and thus hit well known bugs in those)

What do you think: What can I improve? Can this technique be put to other good uses?

Call for testing: unzip feature

April 7th, 2010 by

Hello Planet!

Have you ever faced a bug like this bnc#540598 ?

When you create  zip archive with non-English filenames and try to unpack it on openSUSE, filenames within archive become unreadable. It can irritate, isn’t it?

It seems as if we found a solution for Russian language. We tested it and it works for us.

It would be helpful if some of you could test your local language. And check whether core functionality still works 😉

Here is a list of  languages that are potentially affected by this bug: Ukrainian,Belorussian, Bulgarian, Croatian, Czech, Estonian, French, German,Hungarian, Italian, Lithuanian, Latvian, Polish, Slovak, Spanish,Slovenian, Swedish.

So it is worth to test them in the first place.

The reproducer is pretty simple:

  • create zip archive on windows with file named in you local language
  • transfer archive to openSUSE system
  • unpack it
  • see if filenames are readable

What needs to be tested:

  • if this bug applicable to you language
  • if core functionality of unzip still works

Please share your experience by commenting on bug.

Package to test located in Lazy_Kent home project

Thanks in advance

Quick tip: Novell Bugzilla and Klipper

December 22nd, 2009 by

Klipper from KDE has a set of predefined actions triggered by content of the clipboard. So when you have it enabled, it show up the popup menu with all available browsers when you copy the URL. In my job I open many bugs and the bug numbers have the well defined format bnc#number. So why don’t define automatic action triggered by this regular expression?

Because Klipper expects only oneliner as an action, I wrote the short bash script obug.sh which expects one argument with a bugzilla string.

#!/bin/bash

[ -z "${1}" ] && {
  echo "bug number required"
  exit 1
}

regex='^(bnc|bug)?#[0-9]+$'
[[ "${1}" =~ ${regex} ]] || {
  echo "${1} did not match ${regex}"
  exit 2
}

firefox https://bugzilla.novell.com/show_bug.cgi?id=${1#*#}

It’s obvious how it works – it checks if the argument exists and if matches the pattern and then opens a firefox (it’d be a xdg-open, but I use firefox for Novell Bugzilla). There should be only one strange expression – ${1#*#}. It is a Shell Parameter Expansion. This expression removes all characters from the beginning of the string including first #, so it convert bnc#1234 to 1234.

Then I defined a new action in Klipper triggered by ^(bnc|bug)?#[0-9]+$, which calls obug.sh %s, where %s is the content of clipboard.
Klipper config

And after that all I selected the bnc#1234 by mouse and then the following popup appeared.

One disadvantage of this method is that Klipper opens popup window only near its status icon in systemtray. It would be more useful to show it near current mouse position.

openSUSE Wiki Changes

December 19th, 2009 by

There was a lot of dicussion in the openSUSE project about the project wiki which was suffering from something all successful projects in some point of time experience: There is a huge amount of documentation in the Wiki, however it seems a bit unstructured, sometimes outdated or not really maintained.

The brave openSUSE wiki team stepped up to change that. The decision is to set up a new wiki with a well selected set of extensions and now the content of the old wiki is going to be transfered to the new wiki. Of course there will be a rigorous quality check of the articles, as well as a new thought through structure. The goal of this huge amount of work is to provide a outstanding good and well consumable source of information for all people in and interested in the openSUSE project. That is a high goal and I admire the energy and dedication of the wiki team.

The new wiki is now in place. So if you also want to help, either with the motivation of a developer telling how things work, or from the upstream perspective using the openSUSE vehicle to push the project or simply because you want to help openSUSE to become even better, first read the Transition Guidelines and subscribe on the wiki mailinglist, since most of the coordination happens there.
There is also a Forum Thread going on around that.

Please help to make this a success – thanks 🙂

openSUSE 11.2 is out

November 12th, 2009 by

The openSUSE Project is pleased to announce the release of openSUSE 11.2. After a couple of month good work towards the 11.2 we’re enjoying a very nice distribution which I already like very much. It is running on most of my machines for a few weeks now. I have already seen some SUSE Linux distros going gold over the time I spent with SUSE and my personal gut feeling tells me this is one of the more remarkable versions.

As usual it comes with tons of new up to date software and also the installation runs smoothly, please read the announcement for all the details, but what for me the most remarkable with 11.2 is that it is a real community openSUSE distro.

There is so much effort visible in 11.2 which was achieved through our growing community rather than just the SUSE people. We had a lots of requests in openFATE suggesting features, we discussed some of them quite heated, others were no-brainer. We again had lots of testers who hammered the alphas and betas and reported big and small bugs. On the openSUSE Conference many discussions about the upcoming distribution took place which were inspiring. We were able to utilize the powerful openSUSE Buildservice to build the distro together with all packagers very effectively. That improved the quality of our packages again. Another very visible thing for me personally is the desktop artwork which was done in best cooperation with upstream – and it looks so great that I hesitate to start applications which cover the desktop all day 😉

It is really exciting to see how things come together on the way to community distribution, and how far we got with openSUSE 11.2. I am happy about that and I am proud to be part of this and like to say thank you for every little bit you might have contributed. I believe that the message that openSUSE is your community distribution has arrived.

Of course openSUSE continues to be open for your ideas, the distribution can be the vehicle to power up ideas from a little application to huge software projects. The openSUSE project is the powerful community behind which helps to make ideas reality. And all that based on the principles of free software! I am really happy today and very excited about what future will bring 🙂

I hope to see you on the release event here in Nürnberg soon 🙂

Making technology Previews succeed – OSC 09 Unconference session notes

October 9th, 2009 by

I did an Unconference at openSUSE Conference 2009 titled: Roads Less Travelled – Making Technology Previews succeed“.  More number of people than I had expected participated in the unconference session. I wanted to make the discussion notes (rough) available to a wider audience so that we could act on some of those:

Often Technology Previews are not solving their purpose. The objective of this session was to discuss, find out how we could get better feedback on Technology Previews and make them better.

The discussion is focused primarily on these areas:

  • Advertise the feature through proper channels, places
  • Make it easy enough for users to try out and provide feedback
  • Make it less risk-prone
  • What stops users from trying out?
  • Provide better documentation?

Key discussion points, suggestions:

  • Announcement in opensuse.org main page/wiki could grab the attention of community members who could help test Technology Previews.
  • Reduce hassles in providing feedback. For e.g Perhaps facility without authentication/Single-signon?
  • Easy ways/methods to provide feedback/input
  • Bug/Issue reporting made easy, command line tools?
  • Text area to provide feedback (as opposed a authentication based system).
  • Create a dedicated page for preview for e.g. previews.opensuse.org
  • irc channel for previews? (Discussion on all TPs, User testing one TP might get interested in another)
  • Announcement in openSUSE Weekly news could help
  • More Blogs, Articles, Whitepapers etc.. (blog entries should have provision for giving comments)
  • Perhaps, try to get some help from documentation team?
  • Provide instructions, mechanisms to safely try out without breaking things.
  • Additional information about new technology while the community tries to use the old technology For e.g. while a user tries to do nfsv3 mount providing an informational message that NFSv4 is available and can be used
  • Suggest using a VM
  • Caution about what might break and what might not (Make community feel less riskier to try out).

Please feel free to comment on what might work and if you have any more suggestion.

The short presentation I used to introduce the topic can be found here: http://files.opensuse.org/opensuse/en/d/de/Roads_Less_Travelled.pdf

openSUSE Core Test Team Established

September 11th, 2009 by

Our call to apply for the openSUSE Core Test Team end of July attract a great deal of attention. So we got applications from nearly 50 members out of the openSUSE community, much more than expected.

Caused by that large number of offers to move testing of openSUSE forward, we decided to increase the number of members for the new openSUSE Core Test Team from 15 to 25. A number that enables even more areas to be covered by the openSUSE Core Test Team.

Six weeks later – with the release of Milestone 7 of openSUSE 11.2 – the team is formed and already started his work to ensure that openSUSE will also be a rock-solid Linux distro in the future. First discussion about prioritizing tasks is already started and members of the team are planing their first real time meeting on IRC.
(more…)

1-Click Bug Reporting?

August 23rd, 2009 by

First off, let me blow the dust off of my Lizards blog account 😉

Now that Bug iconwork and school has started to settle down, I’ve gotten back in to testing openSUSE’s newest version, 11.2. One of the things that annoyed me a little, though, was having to open Firefox to report or search for a bug in Bugzilla. So with a little inspiration from the Windows 7 beta’s links to “report a problem” everywhere, I created a desktop icon and panel launcher to automatically launch Firefox and open the “enter new bug” page on our Bugzilla.

You can download this icon here (right-click and Save Link As). Perhaps this is something that should be included with the pre-release versions of the OS starting in 11.3? What do ya’ll think?

Update: Due to a suggestion by Pavol Rusnak, it’s now in openFATE – #307492

Hackweek IV: Novell Bugzilla access from command line

July 29th, 2009 by

During a last Hackweek I decided to work on some tool which will helps users with submitting a new bug. My idea is to create something like reportbug from Debian, but targeted to SUSE. So the first step was find the way how to communicate with a bugzilla (especially to bugzilla.novell.com – bnc). Fortunately I found several libraries, which should be used for it:

perl-SUSE-BugzillaClient is written by Thomas Schmidt especially for bnc, so it is able to login using iChain. It also provides few functions for bugzilla communication, but unfortunately only for querying. Pybugz is written by Alastair Tse for Gentoo development in Python. But it is not able to login through iChain and I did not extended it for bnc, because it uses a plain GET/POST communication with parsing of HTML using regexes.

The reason why I have used the python-bugzilla written by Will Woods from RedHat is simple. It uses XMLRPC API, which bugzilla provides. And it’s also written with a flexibility in mind, so writing of another type of Bugzilla was easy. Just some subclassing and reimplementing of few methods and voila, the NovellBugzilla type is here. Because it uses iChain, it was necessary to reimplement login/logout methods – I spend a lot of time to understand how it works. Osc uses a HTTP authentication, which should be supported too, but even if it works with api.opensuse.org, it did not work with bnc. So I used a Firefox and LiveHTTP headers extension to track a communication and implemented the login according it.

The useful advantage of Novell bugzilla is that I extended a method readconfig, which reads a config file. Novell Bugzilla can read the username/password from ~/.oscrc, so you don’t need to have a same password in another config file, if you use osc.

So on the end those few lines of Python code

import bugzilla

bnc = bugzilla.NovellBugzilla(url="https://bugzilla.novell.com", user="mvyskocil", password="XXXXX")
kwargs = {
    'product': 'openSUSE 11.2',
    'severity': 'Normal',
    'cc': [],
    'rep_platform': 'x86-64',
    'component': 'Java',
    'summary': '[Java:packages/ant]: Cannot find a Java virtual machine',
    'version': 'unspecified',
    'assigned_to': 'mvyskocil@novell.com',
    'op_sys': 'Linux',
    'description': 'This is a testing bug report\n'
    }
bug = bnc.createbug(**kwargs)
print bug.bug_id

Have created this bug#525549. Even if Python is readable for regular programmers, the **kwargs should be expressed. It maps the dictionary to named function arguments (called keyword arguments – kwargs – in a Python world), so both calls in following snippet are equivalent.

d = {'name' : 'value')
call(**d)
call(name='value')

The python-bugzilla comes also with a simple command line tool called bugzilla, so this tool should be used also from shell scripts. It is sometimes little bit raw. For example listing of all products returns an output as a Python dictionary

bugzilla --bztype NovellBugzilla --user mvyskocil --password xxxxxx info -p
...
{'description': 'openSUSE 11.1', 'internals': {'disallownew': 0, 'classification_id': 7340, 'name': 'openSUSE 11.1', 
'votestoconfirm': 0, 'milestone_required': 0, 'id': 651, 'votesperuser': 20, 'maxvotesperbug': 5, 'defaultmilestone': '---',
'milestoneurl': '', 'description': 'openSUSE 11.1'}, 'name': 'openSUSE 11.1', 'id': 651} 
{'description': 'openSUSE 11.2', 'internals': {'disallownew': 0, 'classification_id': 7340, 'name': 'openSUSE 11.2',
'votestoconfirm': 0, 'milestone_required': 0, 'id': 755, 'votesperuser': 20, 'maxvotesperbug': 5, 'defaultmilestone': '---',
'milestoneurl': '', 'description': 'openSUSE 11.2'}, 'name': 'openSUSE 11.2', 'id': 755}
...

which might be hard to parse in other language. But it provides a lot of functionality, so for other informations about usage, please use man bugzilla.