Release System Architecture

/Future

WARNING: these docs are out-of-date. Stage 2/3 are now different, though the pipeline concept remains the same

The architecture of this system is based on our experience with our previous debian-building infrastructure. The new system is designed to be a pipeline, where each stage produces an artifact that is passed to the next stage. Each stage can be resumed independently.

Stage 1: rosrun release create.py

This is the end-user script that produces the actual release.

• Input:
  • Parameters: stack name, stack version, distribution name
  • Source tree (from SCM)
• Output:

  • stack_name-stack_version.tar.bz2

  • stack_name-stack_version.tar.yaml: metadata from Debian control file

  • SVN tags

  • updated .rosdistro

  • Triggers stage 2 Hudson job

The tar.bz2 and .yaml file are uploaded to the release repository in the download directory.

Stage 2: Hudson Source deb builder

http://build.willowgarage.com/job/debbuild-sourcedeb/

This stage downloads the tarball and control metadata from the release repository and builds source debs. This stage is generally very fast and a source deb is essentially a tarball with extra metadata. Our method of generating this source debs essentially wraps the previous tarball.

Supported Ubuntu platforms: intersection of platforms that debbuilder supports and the platforms the stack declares rosdeps for.

• Input:
  • Parameters: stack name, stack version, distribution name

  • stack_name-stack_version.tar.bz2

  • stack_name-stack_version.tar.yaml

• Output:
  • For each supported Ubuntu platform:

    • ros-distro-stack-name_stack-version-0~platform.dsc

    • ros-distro-stack-name_stack-version-0~platform.tar.gz

The .dsc and .tar.gz files are uploaded to the release repository.

Stage 3: Binary deb builder

Parallelized, e.g.:

• http://build.willowgarage.com/view/debbuild/job/debbuild-build-debs-unstable-lucid-amd64/

This stage uses the source debs to build the actual debian packages for binary installation and uploads them to a shadow repository.

If the building of the debian package succeeds, this stage will upload the new debian package and invalidate any downstream dependencies by removing them from the shadow repository.

If the building of the debian package fails, the action is to do nothing. As we do not version-lock the debian packages in the shadow repository, other stacks will continue to use the last successful version of the stack.

Because downstream dependencies are removed on success, it means that future builds may not have all the debian packages they require for building. Thus, stage 3 also builds any missing upstream dependencies.

• Input:
  • Parameters: stack name, stack version, distribution name, platform name
  • Source deb files
• Output:
  • Debian packages for stack dependencies that have not been built
  • Debian package for stack
  • Invalidator: removal of downstream debian packages

Stage 4: Deb Version Stamper

rosdeb stamp_versions.py <distro> <os> <arch>

• From: ros-shadow

  • http://code.ros.org/packages/ros-shadow/ubuntu/

• To: ros-shadow-fixed

  • http://code.ros.org/packages/ros-shadow-fixed/ubuntu/

This stage is triggered manually when the distribution maintainer determines that it is time to produce new debs and that the current system is stable.

As debians have already been produced by stage 3, this stage simply takes debian packages from the shadow repository and rewrites the dependencies to be version-locked.

• Input: non-version-locked debian packages from shadow repository
• Output: version-locked debian packages uploaded to reprepro repository

Software System

ros_release stack

• rosdeb: script and libraries for creating debian packages, plus some utilities for interacting with SVN, pulling data out of ROS packages, and pulling data off the ROS wiki

• hudson: library for managing Hudson jobs, plus scripts for setting up environments within Hudson jobs

• roslib2: in-development libraries for the roslib package. Essentially newer version of the distro.py and vcs modules that assist with .rosdistro file and version control systems, respectively.

release package

The release package in the release repository contains the release scripts that the user uses, plus some support libraries (Python release module) and support scripts. The create.py script has a hidden option that is used to (a) download tarballs from wgs32 to populate the repo and (b) create control files for them.

It also contains scripts for:

• updating SVN externals in wg-ros-pkg
• generating rosinstall files from rosdistro files

Platform configuration

There are two dictionaries that essentially define the build configuration:

• rosdeb/src/rosdeb/core.py: _ubuntu_map does Ubuntu version mapping. It doesn't do builds, but it is essential data.

• rosdeb/src/rosdeb/targets.py: os_platform maps a ROS release name (e.g. 'cturtle') to a list of platforms. This is the actual build configuration.

There also needs to exist the corresponding Hudson jobs for building the debians on that platform. Our scripts assume that the following job exists if it is in rosdeb.targets:

debbuild-build-debs-RELEASE-PLATFORM-amd64
debbuild-build-debs-RELEASE-PLATFORM-i386

These jobs are actually parameterized and copies of the base debbuild-build-debs-RELEASE job. The role in naming them is to allow parallelization in Hudson.

There is a script that will automatically assert these jobs:

rosrun rosdeb update_hudson.py <release-name>