Hi,

as I already I wrote in an earlier post the last week I worked on implementing osc2’s “request”
command. In order to create a new submit request the user has to run
osc request create --submit api://src_project/src_package api://tgt_project
What’s still needed is a way to specify a revision of the src_package. It isn’t possible
to introduce a “–revision ” option because it’s possible to specify multiple submit
actions for a request (so we can’t distinguish to which –submit action a revision
belongs to).
Finally I implemented darix’ cool idea: simply use a svn-like syntax:
osc request create --submit api://src_project/src_package@rev api://tgt_project

Marcus

Hi,

here’s a small summary of the 12th (coding) week. The last week I continued
the work on the new user interface. As a result the “request” command is more
or less implemented. Yesterday I pushed the current code and it’s also possible
to play around with it:

• checkout the code [1]
• cd osc2/osc/cli
• export PYTHONPATH=/path/to/osc2
• python cli.py -h (for more information)

Example:
In order to list a request it is sufficient to run
“python cli.py request api://your_project” (Note: all https requests are
_insecure_ – so far the certificate etc. is not checked).

Some implementation details:
In my last mail/post I described how a new (sub)command can be specified
so I’ll leave out the details here.
In the following I’ll shortly explain how the arguments specified by the
user at the commandline are passed to a function (which does the actual
work – like listing the requests).

class RequestList(CommandDescription, Request):
    """
    ...
    """
    cmd = 'list'
    args = 'api://project?/package?'
    opt_user = Option('U', 'user', 'list only requests for USER')
    func = call(request.list)

As we can see the project and package parameters are optional. After
the parsing process a so called “info” object is returned which encapsulates
the specified arguments. Assume the user runs “request list api://test_prj”
then the info object has the following attributes:

    info.apiurl = the default apiurl
    info.project = 'test_prj'
    info.package = None

The question is how can we pass this data to the request.list function?
A simple (and naive) approach would be to simply pass the “info” object
to the request.list function that is “list” has the following signature
“def list(info)”. As a consequence inside the method we always have to
use “info.project”, “info.package” etc. which is a bit awkward – at least
for parameters which are quite frequently used in the method definition.

So it would be nice if there’s a way to pass frequently used parameters
directly to the method (that is they’re declared as formal parameters
in the method definition) and infrequently used parameters can still be
accessed via the info object. Exactly like this it is currently
implementend in osc2.

So the following signatures would be correct for our example:

    def list(project, package, info)
    def list(project, package, user)
    def list(project, package, user, info)
    def list(project, info)
    def list(info, project)
    def list(project)  # using the info object is not mandatory
    def list()  # quite useless...
    ...

Invalid signatures:

    def list(prj, pkg, info)
    def list(foo, info)
    ...

So it is up to the developer how to define the signature of the
request.list function – it is not completely dictated by osc:)

Marcus

[1] https://github.com/openSUSE/osc2

Hi,

here’s a small summary of the 11th (coding) week. Last week I worked on
implementing the new commandline interface. While doing so I faced several
“issues”:

• How to combine argparse and our oscargs url-like syntax?
  Basically we have to run our oscargs parser on the result which is
  returned by argparse.ArgumentParser’s parse_args method. The problem is
  that both parsers have a different “syntax” that is using a naive approach
  will lead to some redundancies (we specify the ui twice: one time for
  argparse and one time for oscargs). In order to avoid this we need some
  “interface” to which the oscargs syntax is passed and which configures
  the argparse parser accordingly.
• How to support custom commands?
  We also have to provide an “easy” way to specify custom commands.
  Additionally it might be handy if existing commands can be enhanced
  (either by adding additional options etc. or by adding a new subcommand
  etc.). The best would be if the user simply drop his/her plugins in a
  specific directory and osc will scan this directory and use the new
  plugins/commands.
• Specifying the ui programmatically is somehow confusing/cluttered. It would
  be much better if the ui can be specified in a more “declarative” way
  without the syntactic “overhead” (well that’s a bit exaggerated) which
  is needed to configure the parser. Additionally it would be nice to have
  a convenient way to specify a multi line description for a command
  (hardcoding the str into the source makes the code “harder” to read).

Finally I ended up with a small DSL which can be used to specify the
ui in a “declarative” way (the initial idea + syntax is taken from the
django framework (see [1])).

Example:
Assume we want to implement a request command which consists (for the
sake of simplicity) of 2 subcommands “list” and “accept”. This can be
specified like the following:


# file: osc/cli/request/ui.py

     """
     cmd = 'accept'
     args = 'api://reqid'
     func = request_accept


In order to add the request command it is sufficient to add an

     import osc.cli.request.ui

statement to the main cli module. This produces the following output:

marcus@linux:~/osc2/osc/cli> python cli.py request -h
usage: cli.py request [-h] {list,accept} ...

Show and modify requests.

positional arguments:
  {list,accept}
    list         List requests.
    accept       Accept a specific request.

optional arguments:
  -h, --help     show this help message and exit
marcus@linux:~/osc2/osc/cli>

and

marcus@linux:~/osc2/osc/cli> python cli.py request list -h
usage: cli.py request list [-h]
                           [-s {new,review,accepted,revoked,declined,superseded}]
                           [-G GROUP] [-U USER]
                           api://project?/package?

List requests.

    By default open requests for a specific project or package will be
    listed.

    Examples:
    osc request list api://
    osc request list api://project
    osc request list api://project/package

positional arguments:
  api://project?/package?

optional arguments:
  -h, --help            show this help message and exit
  -s {new,review,accepted,revoked,declined,superseded}, --state {new,review,accepted,revoked,declined,superseded}
                        list only requests with state STATE
  -G GROUP, --group GROUP
                        list only requests for GROUP
  -U USER, --user USER  list only requests for USER
marcus@linux:~/osc2/osc/cli>

How does it work?
First of all each class which defines a command or subcommand has to inherit
from class “CommandDescription”. If a subcommand is to be defined it also
has to inherit from the “parent” command (that is in our example “RequestList”
and “RequestAccept” inherit from class “Request” (which in turn inherits from
class “OscCommand” (from this class all toplevel commands have to inherit))).
In short: with the help of the inheritance hierarchy it is possible to define
a command <- subcommand hierarchy.

Note: actually the classes "RequestList" and "RequestAccept" only inherit
from "CommandDescription". The "parent" command base class is only needed
for a "marking" purpose (it is filtered out with the help of a metaclass
when the concrete class is "constructed" – I'll leave out the details for
now and may write a dedicated blogpost about it).

Now the remaining task is to define and implement the commands (note: we will
definitely not finish the project on the "suggested pencils down" date and
use the week until the "firm pencils down" date for coding…).

Marcus

[1] https://docs.djangoproject.com/en/1.4/topics/db/models

Here’s a small example how to modify an existing command:

# plugins/myrequestaccept.py
from osc.cli.description import Option
import osc.cli.request.ui

class MyRequestAccept(osc.cli.request.ui.RequestAccept):
     # add a new option
     opt_foo = Option('f', 'foo', help='foo option')


This leads to

marcus@linux:~/osc2/osc/cli> python cli.py request accept -h
usage: cli.py request accept [-h] [-f FOO] api://reqid

positional arguments:
  api://reqid

optional arguments:
  -h, --help         show this help message and exit
  -f FOO, --foo FOO  foo option
marcus@linux:~/osc2/osc/cli>

Hi,

here’s a small summary of the 9th (coding) week. Last week I worked
on the fetcher and cache manager code. In order to support all
features some of the existing classes had to be enhanced with some
more parameters.

Done:

• cache manager code
• BinaryList class supports view=cpio
• RORemoteFile class supports lazy_open
  parameter (by default the file is opened lazily that is when a
  read request is issued) (for the fetcher code we use
  lazy_open=False)
• minor changes in the httprequest module (AbstractHTTPRequest
  supports the same query parameter more than once)

The fetcher code is more or less done (not yet committed) and will be
finished by friday evening (I’ve some exams this week…).

Marcus

Hi,

here’s a small summary of the 8th (coding) week. The last days I
was working on getting build support into the osc2 library. I just
checked in a BuildInfo and Builder class (+ testcases). Here’s a
brief example how they can be used:

# example how to use the Builder class
builder = Builder(su_cmd=Builder.SUDO, root='/var/tmp/build-root')
builder.rpmlist = '/path/to/rpmlist'
builder.dist = '/path/to/buildconfig'
builder.without = 'feature1'
builder.without += 'feature2'
builder.arch = 'x86_64'
# run the build:
builder.run('/path/to/osc.spec')
# this executes:
sudo /usr/bin/build --arch x86_64 --dist /path/to/buildconfig \
   --root /var/tmp/build-root --rpmlist /path/to/rpmlist \
   --without feature1 --without feature2 /path/to/osc.spec


Basically Builder is just a wrapper around the build script
(all options will be passed to the build script; if an option contains
a “-” character like “vm-type” it can be set like this:
“binfo.vm_type = ‘xen'” (a “_” will be replaced with “-“)).

Additionally here’s a small example how to utilize the BuildInfo
class:

# fname is a path to a spec file
binfo = BuildInfo('openSUSE:Tools', 'osc', 'openSUSE_Factory', 'x86_64',
                  filename=fname)
# print preinstall packages
for bdep in binfo.preinstall():
    print bdep.get('name'), bdep.get('version')
# save binfo xml in a file
binfo.write_to('/path/to/file.xml')


The next thing on my TODO is the fetcher and cachemanager code.

Marcus

Hi,

here’s a small summary of the 7th (coding) week. Last week
I was really busy with university stuff (it was the second
last week in the lecture period so I had to recap quite
some stuff) and didn’t manage to work much on the GSoC project.
I’m going work off the TODO this weekend.

Marcus

Hi

here’s a small summary of the 6th (coding) week. Last week I
continued working on the build module and developed a concept
for the package fetcher (and discussed some parts
with darix:) ).
The main idea is to modularize the fetcher code, verify code etc.
Thus we have a fetcher class which takes care of retrieving the
packages. The fetcher class utilizes a “cache manager” which takes
care of storing the fetched packages on the filesystem. The goal is
that at some point in time one can exchange the “simple” cache manager
with a more “clever” cache manager (which for instance cleans up the
cache from time to time or only allows exactly one version of a
package in the cache etc.). In order to achieve this no code in the
fetcher has to be touched – instead it’s sufficient to pass a different
cache manager object to the fetcher.
Additionally the fetcher provides some hooks like pre, pre_fetch,
post_fetch and post. For instance a post_fetch hook can be used to
verify the just fetched package etc.

TODO:
– write testcases and implement the concept from above

If you have questions, suggestions etc. feel free to contact me:)

Marcus

Hi,

here’s a small summary of the 5th (coding) week. Last week I spent
most of my time with working on the cpio module. Finally I ended
up with a complete rewrite of osc’s “old” cpio module. The
cpio implementation details are taken from the cpio 2.11 package.
Currently only the “new ascii” format is supported and we can
only handle regular files (this is sufficient for our needs). Here’s
a small example how to use it:

# cpio_open is just a convenience method for CpioArchive(filename=fname)
with cpio_open(fname) as archive:
    for archive_file in archive:
        # print filename
        print archive_file.hdr.name
        # print contents
        print archive_file.read()

It’s also possible to pass a file-like object to a CpioArchive
instance (in this case we do not need to use the with statement).
Also it can be “easily” enhanced to support different cpio formats
(one just have to write the specific ArchiveReader and ArchiveWriter
classes:) ). The code is available at [1] and the testcases at [2].

Todo for this week:
* finish the build module
* start with working on the package fetching code

Marcus

[1] https://github.com/openSUSE/osc2/blob/master/osc/util/cpio.py
[2] https://github.com/openSUSE/osc2/blob/master/test/util/test_cpio.py

Hi,

here’s a small summary of the 3rd and 4th (coding) week . First of all I
wasn’t able to do lots of work in week 3 and 4 so I’m still working on
the new build module for the osc2 library.
The initial plan was to copy/reuse some of the existing modules from the
(old) “osc” like the cpio [1] and packagequery modules. But I decided to
refactor/rewrite cpio module for the following reasons:

• “save” disc space:
  In our scenario we retrieve a cpio archive from the api (which contains
  binary packages for example). The old cpio module expects a filename
  in order to “unpack” the archive – that is the file has to be stored on
  disc first. Consequently approximately 2 * of free disc
  space is needed.
  The new idea is that we pass a file-like object (in our case an object
  which inherits from “AbstractHTTPResponse”) to the cpio module and
  unpack the archive “on the fly” (without storing the http response
  on disc first).
• have testcases:
  The old cpio module has no testcases (because some time ago I didn’t
  follow the TDD approach;) ). For nearly all modules in osc2 there exist
  testcases (white-box tests) thus it would be nice if we have some
  testcases for the cpio module, too (theoretically we could add some
  black-box tests (the current methods aren’t really testable thus
  white-box tests aren’t possible)).
• have a nice pythonic interface:
  The new interface will look like this:
  
from cpio import cpio_open
# let "f" be a file-like object (for instance a http response)
with cpio_open(fobj=f) as cpio_archive:
    for a_file in cpio_archive:
        # store file (with correct permissions etc.) in os.curdir
        a_file.write(os.curdir)
        # alternatively it's also possible to read some data (instead of
        # writing it to disc) via a_file.read(len)

  We will also support a plain filename in cpio_open.

Currently the cpio module will only support the “new ascii” (ascii SVR4
no CRC) format and regular files (that’s sufficient for our needs). But
it will be possible to simply pass in a class for a different format
(that is no code has to be altered in order to support a new format).

Finally this will be finished by the end of this week.
If you have any questions or suggestions please tell me:)

Marcus

[1] https://github.com/openSUSE/osc/blob/master/osc/util/cpio.py