OBS source repository moved

The OBS git repos have been moved to github.com as announced earlier on build service mailing list. Please read this mail for details.

OBS 2.2.72 released, switch to apache and SSL as default

We just released OBS 2.2.72, another alpha release for OBS 2.3. The most significant change to former alpha snapshots is the switch to apache. We do so for a number of reasons:

  1. Getting a maintained base again for our default httpd
  2. Using mod_rails (passenger) is more flexible then the static setup before
  3. Optimizations are possible via additional modules (more about that later)

We think that some bugs like the invalid occassional empty replies by the api server are solved via this switch as well.

The OBS 2.2.72 appliance is already comming with apache setup. Please note that we use also SSL by default, a default SSL CA is created on bootup and becomes part of your data partition.

In case you use the packages outside of the appliance, you need to do some configuration steps. But don’t worry, a apache vhost file comes with the packages already and only a few steps need to be done, the README files should reflect this already.

lighttpd setups should still work, but please note that lighttpd is running with an own user id (“lighttpd”) and the default is now the generic “wwwrun” user for some directories.


Policy proposal for Factory: Make source of tar balls trackable

I like to suggest a general policy for openSUSE:Factory project to document from where a tar ball (or any other file from upstream) is comming from. Why that ? It makes it much easier to review version updates and it guarantees that no one can inject some mal code via a modifed tar ball.

So far I added the source services “download_url” and “tar_scm” to our OBS instance, which downloads the files and stores them as files via a commit. Some people use them already, some others don’t like them because they store the files with _service: prefix.

In last hackweek, I added another way to handle this, which I would like to request as setup and policy for openSUSE:Factory project. You can add a project wide source service, for example the new “download_files” service. That would mean that no needs to add a _service file to the sources anymore. It is enough to add an URL to the spec file Source: tags. The service will automatically download it from there.

But that does mean we still have have _service:download_files:osc-0.1.tar.bz2 file names ? Not when we also add the new “trylocal” parameter and use latest osc versions. This parameter will let act osc to execute the services, but name the files without prefix and commit them together with the other files.

Where is the advantage then ? The server is still validating that this is an identical file. It downloads it again and compares it. In case it is the same file, nothing will happen.

What will happen, when the file differes ? We basically have two options, either we can let the service mark the source as broken or we would store the file with _service: prefix again.

The later mode has the advantage that you can still do version upgrades via slow connections and let the server download the files.

Please find some more details about new possibilities with the source services here.

An example setup for this can be tested via

osc bco home:adrianSuSE:FactoryTest bc

and do for example a version downgrade to 1.05 version to see how it works. Please note that you need the osc from openSUSE:Tools:Unstable project for this.

We can also apply the still suse-internal spec formater and validator scripts via this way later one.

Another advantage of this setup would be the new “update_source” service, which could run in some openSUSE:Factory:AutoUpdate project and tries automatic version upgrades when upstream releases a new version. They could be reviewed and just picked (directly or with additional manual fixes).

How to work with OBS via slow connections

The openSUSE Build Service Books have a new chapter called HOWTOs. These will be written for asked questions, the first one existing is how to work best with OBS if you have limited bandwidth. There are multiple methods to avoid large downloads or even uploads of large files (like tar balls) and let just the server do the work…

Fotostream from openSUSE Conference 2010

Yet another foto stream from the openSUSE conference. You see the desktop leads from KDE and Gnome (Cornelius Schumacher and Vincent Untz) giving a talk about the past and future of the free desktop, Stephan Kulow about the future of the distribution, Bernhard Wiedemann about QA testing and so on.

Most important may be the presentation of the openSUSE board (mainly by Pascal Bleser) how they plan to found an independent foundation for openSUSE as non-profit organization. An important rule of that foundation is that it is independent of any company (no majority of Novell here) but can handle sponsoring, partnering and trademark questions.

We had also very filled rooms during the OBS talks, but I was unable to take pictures at that point of time unfortunately 😉

OBS Development Team Member Job Position

SUSE GmbH has currently a job position open for an OBS Developer. Find details on the job position page at Novell.

OBS is used in the openSUSE project, but also internally at Novell and at plenty other places and companies.

The downside will be of course that you will have to work together with people like me 😉

Some LinuxTag 2010 impressions

LinuxTag 2010 has ended, openSUSE had a booth in the community area and we had a number talks. We also released OBS 2.0 on LinuxTag. You know this of course already, but here are some impressions.

openSUSE booth was very well visited. Various workshops and activities created several times actually a big swarm around it. Many people were interessted about OBS in special and I hope we won some more OBS users and developers.

Hennes and mine talk about “how to escape the free software hell” was provocant enough to get quite some people into our room directly after the keynote. I hope we were able to show off the coolness of OBS there.

Read the comments in the picture gallery for some background information.

OBS reports scheduling state now

After getting asked daily why package X or project Y is in a certain state, why it is not yet published or if a “trigger rebuild” button click is needed, I finally got around and implemented the last planned feature for OBS 1.7.

Each repository has a scheduler state now for each architecture. This state tells you in which state the scheduler saw it the last time he looked at it. In addition to that, there is also a “dirty” flag. This gets set, when something happened what requires a re-calculation by the scheduler, but he has not done this yet. So no need to hit the “trigger rebuild” button in this case, it won’t help 😉

A typical state round trip is the following, it can stop at any point and restart from a higher listed state again if this is needed:

unknown: a repo has just been created for this architecture. The scheduler has not seen this yet. This is currently the case also in most repos, just because the scheduler with this new code has not looked at all existing repos yet. But you will see this only in very rare cases in future.
scheduling: the scheduler is looking at this repo right now. The package states will get updated soon (from 1 second up to 2 minutes, depending on the project complexity)

blocked: packages from other repos need to build first until any of the packages from this repo can get build.
building: packages are scheduled for building, this means they are in “build”, “scheduled” or “finished” state.

unpublished: all packages got built, publishing is disabled, so nothing is to do here anymore.
finished: all packages which require a build have been built. This repo is enabled for publishing, but this has not yet happened.
publishing: The publisher is currently processing this repository.
published: the latest built packages have been uploaded to the stage server and are available via download.opensuse.org

Future plans for this are to add this state list in the web interface somehow, so everybody gets an explanation in the context.

Another thing what I want to add is a button beside the “unpublished” state, to force a publishing. So by default nothing gets published, but the project owner can force it when he things it makes sense. However, this is now also possible to achieve by swithing the publish flag and revert it after some time. The OBS is now even publishing when the build has already finished (unlike before).

openSUSE Build Service 1.7 Beta 1

The openSUSE Build Service (OBS) team just announced the Beta 1 version of the upcomming 1.7 release. Most of the features are already accessable in the Build Service instance which is used by the opensuse.org project.

Please find a more detailed announcement here.

The final release is planed early next year.

OBS Attribute System (not only for maintenance!)

People who follow the openSUSE Build Service (OBS) developments might know it already, we work on an attribute system for OBS. But what it is good for at all ?

Our current driver is to enable every OBS user to do maintenance for packages in the maintained products (which are currently openSUSE 11.0, 11.1 and a few days 11.2). The maintenance concept itself is described in a very first draft here

However, the attribute system is way more powerful and can be used to store all kind of informations, attached to projects, source packages or even binary sub packages. The important thing here is that the attribute types have own permission rules. So it is for example possible to edit data in projects like openSUSE:11.1 or Fedora:9 which are usually read only.

A simple example is the OBS:Screenshot attribute, as you might guess you can attach references to screenshots to it. Every maintainer or bugowner has write access to it, this means if you are the bugowner of a package, you store this kind of informations not only in your projects, but also in the openSUSE:11.X project packages.

There is also the openSUSE:Playground attribute type created, just for you, when you like to play with this. Btw, the current available attribute types can be requested via “osc meta prj OBS”. And when you use the osc 0.123svn from svn trunk or openSUSE:Tools:Unstable Project, you can even check single attributes in different ways or create them.

For example:

osc meta attribute openSUSE:11.2 # Shows the attributes of the openSUSE:11.2 project
osc meta attribute home:adrianSuSE --attribute openSUSE:Playground --create # just creates the attribute in my home project
osc meta attribute home:adrianSuSE zphoto # returns empty, since the package hasn't the attribute.
osc meta attribute home:adrianSuSE zphoto --attribute-project # returns with attribute, since it falls back to the project

# stores two values (World Domination and fast) inside of the attribute:
osc meta attribute home:adrianSuSE --attribute openSUSE:Playground --set "World Domination,fast"
osc meta attribute home:adrianSuSE # shows all attributes in my home

osc search --attribute openSUSE:Playground # finds all packages in all projects with the openSUSE:Playground attribute
osc search --package zphoto --attribute openSUSE:Playground # finds all zphoto packages in all project with the openSUSE:Playground attribute

Okay, Okay, all that sounds not horrible sexy when you read it first. But imaging the possibilities. Each team or use case can get their own attributes. They decide what to store in which package, independend if they can modify the sources of project or not. So a team can easily mark packages for any kind of purpose (to fix bugreport 1234, to complete their product Z, to show the state of the packages on web page X, …).

The “osc mbranch” command from the maintenance concept shows also the power of this. You do not need to know where all instances of your package, just tell the server that you need to work on it and the server collects them all.

Please note that the API for the attribute system still might change until OBS 1.7 gets released, we may even need to remove the attributes (even though this is not planned). However, the version running at opensuse.org should be ready to play with this system. And I _really_ would like to hear any kind of feedback, ideas or requests. Can you please comment here, what you can imaging, what else you can use this system for ?

Thanks a lot !

PS: New attribute types can be defined only by the administrator atm, but I am really happy to create any kind of attributes for you, even though you just want to play with it!