Login

Or, How I learned to stop worrying ...

Rationale

After running across a portability / build problem with a distutils package recently, I realized that the testing we are doing for Repoze may not have adequately covered our needs to keep the various Repoze components running against all supported platforms (basically, anything where packages with "C" extensions can be built easily).

After casting around a bit, I decided to bite the bullet and try out Buildbot, the same tool used to run the Zope unit tests.

Setting Up the Master

First, I decided to set up the buildbot "master", running on the repoze.org server. This process tracks all pending changes which require builds, and doles the builds out to one or more designated "buildslaves".

Because buildbot runs as one or more long-running daemon processes, I decided to create a new account for it, to enforce a bit of privilege isolation:

  $ sudo /usr/sbin/groupadd buildbot

  $ sudo /usr/sbin/useradd -g buildbot -c "Buildbot" -m -s /bin/bash \

    buildbot

Then, I set up the new user's home directory as a virtualenv.

  $ sudo su - buildbot

  $ /usr/bin/virtualenv ~

I then installed buildbot and its setuptools-compatible dependencies (note that those dependencies aren't spelled out in its setup.py, but are document in its INSTALL.txt.

  $ bin/easy_install buildbot zope.interface

The remaining dependency is Twisted, which can't be installed via easy_install (its setup is complex, and hasn't yet adapted to the setuptools regime). So, I installed it the old-fashined distutils way:

  $ mkdir src

  $ cd src

  $ wget http://tmrc.mit.edu/mirror/twisted/Twisted/2.5/Twisted-2.5.0.tar.bz2

  $ tar xjf Twisted-2.5.0.tar.bz2

  $ cd Twisted-2.5.0

  $ ../../bin/python setup.py install

  ...

Now, I could verify that the buildbot application was working:

  $ bin/buildbot --version

  Buildbot version: 0.7.6

  Twisted version: 2.5.0

I then created and initiailzed the buildbot "master" work area, and set up a configuration (lots of iterative tweakage elided here):

  $ mkdir -p var/masters/repoze

  $ bin/buildbot create-master var/masters/repoze

  $ cp var/masters/repoze/master.cfg{.sample,}

  $ vim var/masters/repoze/master.cfg

  $ bin/buildbot start var/masters/repoze

Setting up a "buildslave"

Next, I could work on setting up the first "buildslave":

  $ mkdir -p var/slaves/repoze

  $ bin/buildbot create-slave var/slaves/repoze repoze.org:9980 \

    laguna 

I used the slavename / password which I had created in the master.cfg file. Then, I edited the two "info" files identifying me and my host, and then started the buildbot "slave" daemon:

  $ vim slave/repoze/info/{admin,host}

  $ bin/buildbot start slave/repoze

It connected, and I could see it on the status page.

Now, I tried a "forced" build, which worked (after more iterative tweakage). The "builder recipe" I'm using does the following whenever it is triggered, either manually or when its "change source" notices a change:

1. checks out the branch containing the change (e.g., 'repoze.kiss/trunk');
2. runs 'bin/python setup.py test' in the checkout.

The "waterfall" page shows the build status.

I then set up another pair of buildslaves, one on my Ubuntu Gutsy laptop and one on a Centos4 server, and ensured that the builds worked with them. Carlos de la Guardia has since volunteered an Ubunty Gutsy server, which is great, as my laptop may or may not be online when a change is triggered.

Conclusions

First, the product does what it is designed for: it handles the complex process of dispatching asynchronous builds to the various platforms based on changes to the repository.

Second, the configuration format is an execfiletoo powerful: I would prefer a purely declarative configuration language. Many users, however, won't need to write any really imperative code in this file, which mitigates the risk somewhat. I chose to add some code here, rather than move it out to an importable module, largely to deal with having a single "scheduler" handle changes to any of the supported branches, and to arrange that each of those branches build in a separate subdirectory.

Third, the code is definitely "frameworky", with lots of base classes and policy objects. It succeeds in hiding a lot of that from the user, however, which covers a lot of sins.

Overall, I like it for what it does, and wouldn't hesitate to use it in other settings.

Tres.