Regression Tests
To catch regressions in our code base, we constantly run thousands of tests on all stacks in the ROS distributions. The tests range from small unit tests, to large simulation based integration tests. Our Hudson server manages and schedules all tests. Below you find more details on each class of test that we run.
Contents
Development Tests
The development tests will build and test the development branch of each released stack. The development branch is specified in the rosdistro file. For boxturtle this often is branches/stackname-1.0 and for cturtle this often is trunk. Every released stack has its own set of +- 20 corresponding development tests, covering each ros-distro / ubuntu-distro /architecture combination. For example, there are six cturtle tests for each stack: lucid 64, lucid 32, karmic 64, karmic 32, jaunty 64 and jaunty 32.
What gets tested The test downloads the development branch of a stack from svn or hg, and installs all dependencies of this stack from Debian packages. It then builds and tests all packages in the stack.
When do tests run Every time you commit to a development branch, one of the tests of the corresponding ros-distro will get triggered (for cturtle, the lucid 64 test will always get triggered first). Once this test succeeds, the other tests of the corresponding ros-distro will get triggered.
Duration of Tests One test typically takes about 3-7 minutes.
Prerelease Tests
Each stack can have a prerelease test. These jobs are dynamically generated on request at http://code.ros.org/prerelease/
What gets tested The test downloads the development branch of a stack from svn or hg, and installs all dependencies of this stack from Debian packages. It then builds and tests all packages in the stack. Next, it downloads all stacks that depend on the tested stack from source, and builds/tests them.
When do tests run When requested at http://code.ros.org/prerelease/
Duration of Tests Low level stack will take about an hour to test, while high level stacks can be finished in 5 minutes. The test duration mainly depends on how many stacks depends on the stack that gets tested.
Released Tests
Each released stack has its own set of +- 6 corresponding Hudson released tests, covering each ros-distro / ubuntu-distro /architecture combination. For example, there are six cturtle tests for each stack: lucid 64, lucid 32, karmic 64, karmic 32, jaunty 64 and jaunty 32.
What gets tested The test downloads the release tag of a stack from svn or hg, and installs all dependencies of this stack from Debian packages. It then builds and tests all packages in the stack. Next, it downloads all stacks that depend on the tested stack from source, and builds/tests them.
When do tests run Every time a stack gets released, one of the tests of the corresponding ros-distro will get triggered (for cturtle, the lucid 64 test will always get triggered first). Once this test succeeds, the other tests of the corresponding ros-distro will get triggered.
Duration of Tests Low level stack will take about an hour to test, while high level stacks can be finished in 5 minutes. The test duration mainly depends on how many stacks depends on the stack that gets tested.
Gazebo Tests
The Gazebo tests are high level integration tests. They exercise large parts of our software base in ways that individual stack tests don't, and typically discover problems such as race conditions, deadlocks, timing issues, etc. There are about a dozen Gazebo tests at this point, covering door opening, autonomous recharging, navigation, calibration, arm planning, etc.
What gets tested The test downloads an entire ros distribution from source, and overlays the gazebo test stack. It then builds and tests all packages in the Gazebo test stack.
When do tests run Every time a stack gets released, the Gazebo tests will get triggered. There are also a few 'manual' gazebo tests used by the developers of the Gazebo test stack.
Duration of Tests One test typically takes about 30-60 minutes.
Ros Stack Tests
The ROS stack tests cover both the released and unreleased stacks, plus code that lives in sandbox.
What gets tested The 'released' tests download an entire ros distribution from source, and build/test all released versions of stacks. The 'unreleased' tests download an entire ros distribution from source, overlay all unreleased code from source, and build/test all stacks.
When do tests run The 'released' tests get triggered every time a stack gets released. The 'unreleased' tests are scheduled to run at 3AM every day.
Duration of Tests One test typically takes about 1-2 hours.
Ros Only Tests
What gets tested The tests download only the ros stack from source, and build/test this stack for six different distributions and architectures.
When do tests run The are scheduled to run at 2AM every day.
Duration of Tests One test typically takes about 30-60 minutes.