[REP Index] [REP Source]

REP:122
Title:Filesystem Hierarchy Standard layout changes for ROS
Author:Ken Conley, Troy Straszheim, Morten Kjaergaard, Ethan Rublee, Tully Foote, Brian Gerkey, Jack O'Quin, Dirk Thomas
Status:Draft
Type:Standards Track
Content-Type:text/x-rst
Created:8-Feb-2012
ROS-Version:Groovy
Post-History:8-Feb-2012, 17-Feb-2012, 21-Sep-2012

Abstract

This REP describes a new ROS directory hierarchy for installed files based on the Filesystem Hierarchy Standard (FHS) [1]. It discusses the implications of those changes, and their effects on ROS code being converted to that standard.

To allow installation of multiple ROS distributions on a single machine, this layout is normally created within a distribution-specific install prefix. But, this specification supports installs within /usr, as well.

This REP has evolved since the original Fuerte draft. The "Fuerte Differences" section explains differences between that preliminary implementation and the current specification.

The majority of this specification is due to the hard work of Troy Straszheim and Morten Kjaergaard on the catkin build system, which is the reference implementation on which this REP is based. This REP does not document catkin itself and is intended to be implementation-neutral.

Specification

This specification defines changes to the filesystem layout, associated environment variable updates, and necessary modifications to ROS itself.

This document mostly avoids details of the catkin build system [10].

Filesystem layout

Install prefix

For clarity, the install prefix is referred to as /opt/ros/<distro>. The install prefix does not have to follow this exact pattern, but it is the convention for ROS binary distributions.

This layout is also intended to enable installations within the standard /usr hierarchy.

Directory layout

The layout within the install prefix is:

bin/
etc/
include/
lib/
share/
stacks/

This generally follows FHS conventions, with the addition of the stacks directory for backwards compatibility with rosmake packages. The install also contains setup files to easily configure environment variables for this prefix.

These subdirectories are described in greater detail below.

bin/

bin is used to store top-level executables, i.e. executables that should be included in every ROS user's PATH. This replaces the old ROS_ROOT/bin location that stored core ROS binaries, like rostopic and rospack.

ROS packages that are converted to use this new layout may install into bin, but should do so sparingly as it is included in the user's PATH. Developers should also beware of name collisions in thise executables. In general, try to name executables so that their origin is obvious, perhaps using the ROS package name.

In the case of a special install prefix, PATH must include this location, e.g.:

PATH=/opt/ros/<distro>/bin:$PATH

etc/

This location is used to install global configuration files for a particular ROS distribution. Within the context of ROS, we introduce etc/ros, where ROS-specific system configuration files are stored.

Additionally, there is an etc/catkin and etc/langs that are used by the catkin build system. Although etc/langs configures the active ROS message generators, its implementation is specific to message generation using catkin, not specified here.

include/

ROS packages using the new install layout must install all header files to the include directory. Assuming a ROS package uses the namespace <ros-package-name>, its include files are stored in:

include/<ros-package-name>/

The specific include directories for a package are located using pkg-config or CMake find_package().

lib/

ROS packages using the new install layout must install library files to the lib directory.

Python modules are installed to lib/pythonX.Y, where X.Y is the Python version. This directory follows the standard Python specification for the target platform, which may be lib/pythonX.Y/dist-packages, or lib/pythonX.Y/site-packages, or somewhere else.

In the case of a special install prefix, PKG_CONFIG_PATH must enable locating libraries and include directories via pkg-config. LD_LIBRARY_PATH must be also set so that executables can load shared libraries. We use the settings:

PKG_CONFIG_PATH=/opt/ros/<distro>/lib/pkgconfig:$PKG_CONFIG_PATH
LD_LIBRARY_PATH=/opt/ros/<distro>/lib:$LD_LIBRARY_PATH

PYTHONPATH must also be set in a special install prefix and is platform dependent. For example, on Ubuntu Precise with Python 2.7, the setting is:

PYTHONPATH=/opt/ros/<distro>/python2.7/dist-packages:$PYTHONPATH

lib/<ros-package-name>/

All package-relative executables and scripts are installed in lib/<ros-package-name>/ subdirectory.

share/

All architecture-independent package-relative assets are explicitly installed to share/<ros-package-name>. These include:

  • package definition files (package.xml)
  • launch files
  • other explicitly installed package-relative data such as bag files, or parameter definitions

stacks/

This is the installation location for rosmake-based stacks. It is included in the ROS_PACKAGE_PATH, but has lower precedence than packages following the new REP 122 layout. Converted packages do not use this folder.

Environment variables

ROS_ROOT

For backwards compatibility with rosmake build files, ROS_ROOT must exist within the new layout. However, its scope has been narrowed to finding legacy rosbuild and mk resources. Other uses, such as finding ROS binaries, are no longer supported. Thus, ROS_ROOT is deprecated. Although there are no plans to remove it, avoid using it for any new features.

For a full specification and rationale, see REP 123 [9], which also proposes new ROS filesystem environment variables based on the REP 122 layout.

Changes to ROS

PYTHONPATH, roslib.load_manifest()

Python modules now install into a dist-packages/site-packages in the lib directory in the install prefix. This installation uses the standard Python distutil/setuptools toolchain.

The Python install directory is added to the PYTHONPATH. Thus, these modules are immediately accessible via Python import and do not require a roslib.load_manifest().

In order to prevent unnecessary accumulation to the Python sys.path, roslib.load_manifest() does not recurse through packages built with catkin.

Python modules and scripts that are part of the FHS install layout must not use load_manifest as it does not work properly within an out-of-source build.

Motivation

The ROS Groovy release migrates many low-level libraries to use a standard Filesystem Hiearchy Standard-like layout [2] contained in a ROS-distribution-specific install prefix (e.g. /opt/ros/groovy/). The goals of this conversion are many:

  • Make ROS libraries more "normal"
  • Enable the ROS toolchain to use existing libraries for packaging and release (e.g. git-buildpackage)
  • Enable ROS messages, libraries and tools to be easily integrated with other build systems via pkg-config and CMake's find_package().
  • Enable ROS binary releases to be accepted into Debian, Ubuntu, and other Linux distributions.
  • Preserve the ability to install multiple ROS distributions on the same machine.
  • Eventually eliminate the need for ROS-specific filesystem environment variables if the installation is to a standard prefix, such as /usr.

The final goal should be possible with converted stacks, but has not yet been verified. The main focus has been on preserving the ability to install multiple ROS distributions on the same machine, as robots frequently are shared, expensive resources.

Compatibility with the Filesystem Hiearchy Standard has been a long-term goal for ROS. ROS began as an in-source, package-relative system for finding, developing, building, and distributing code. Multiple attempts have been made to transition towards an FHS-compatible layout for ROS, including the withdrawn REP 102 [5] and a prototype "rosbuild2" system that was deployed in ROS Electric. Experiences with those previous efforts have enabled us to accomplish this goal with minimal compatibility issues.

One major goal these changes have enabled is much better integration with external libraries, such as PCL. The standalone version of PCL 1.x can now build against the required ROS messages using CMake, although with significant modifications. The intent is for ROS to use the standalone PCL 2.0 libraries without modification. Hopefully, many more libraries will be able to transition to standalone implementations.

Rationale

The rationale for using the Filesystem Hierarchy Standard is simple: it makes ROS releases conform to generally-accepted open source practice. Most of this REP describes straightforward translations to that standard.

Backwards Compatibility

/opt/ros/<distro>/stacks

There are no known regressions related to the new layout for building rosmake stacks that are specific to this specification. We have been able to build legacy stacks on top of the REP 122-compliant stacks by using the ROS manifest "export" mechanism with the output of pkg-config. No changes were necessary to the rosmake toolchain based on this REP. Legacy stacks still use the same installation procedure as before.

We have put together a migration guide [6] that generally notes issues with increased link/include strictness that are separate from this REP.

rosmake

rosmake will recurse into packages that are installed in share. As this location does not contain Makefile files, it will treat this as not buildable.

roslaunch remote, <machine> tag incompatibility

The changes in this specification break compatibility with roslaunch remote launches as the environment configuration is very different. The non-backwards-compatible changes required in roslaunch are specified separately in REP 124 [7].

ROS_ROOT

As described above, ROS_ROOT preserves backwards-compatibility for mk and rosbuild files. For example,:

include($ENV{ROS_ROOT}/core/rosbuild/rosbuild.cmake)

resolves to the same path.

roslib.load_manifest()

The Python environment bootstrapper, roslib.load_manifest(), is safe to use in rosmake-based packages. However, it is not necessary for importing Python modules that use the FHS install layout.

As noted above, roslib.load_manifest() should not be used in converted packages.

rosinstall

Changes to both ROS_ROOT and environment setup have created incompatibilities with the rosinstall tool. There are also separate issues with rosinstall and catkin compatibility when building completely from source that are separate from the issues of install layout changes. These are resolvable and are handled by the affected scripts.

Package contents

The share/ does not preserve the full contents of ROS packages within the installed layout. This affects package-relative workflows, like using``rosed`` to quickly view header file information. There are two categories of contents no longer locatable by package-relative tools:

  1. Resources that are installed in a different location (e.g. C++ header files in include)
  2. Sources resources (e.g. C++ .cpp files) that are no longer installed due to explicit installation targets.

roscreate-stack

With Groovy, stacks and packages no longer have the same container relationship as before. It is no longer possible to determine the packages that a stack contains in the install layout. So, the share/ layout does not preserve any stack-package relationship.

The tool most directly affected is roscreate-stack. Although convenient, it has little utility moving forward, so we made the difficult decision to delete it.

Fuerte Differences

A reference implementation was initially deployed in the ROS Fuerte release. There have been some changes since then, which will be incorporated in the ROS Groovy release.

This section notes differences between the Fuerte implementation and the current specification.

buildspace

In Fuerte the FHS layout was only applied during the installation step. The build folder was not a supported environment to run the compiled binaries and generated code. While some simple applications worked features like i.e. discovering plugins were not supported in the build folder. Therefore the make install step was always necessary before developers could run their modified code. This made it less convenient and increased the round-trip time during development substantially.

In Groovy the build folder contains a subdirectory or sibling folder devel which resembles the same structure than an installation. The content of that folder also follows FHS conventions. It enables to run any application from devel space as well as if it would be installed.

catkin tag in manifest

Fuerte package manifests [3] temporary had a <catkin/> tag to indicate that they are built using the catkin build system. That tag was a performance optimization for tools like rosmake and the Python roslib.load_manifest() API. It was not required, but was recommended.

For compatibility with newer versions of rospkg (released for Groovy) the <catkin/> tag must be removed from Fuerte manifests.

Catkin packages in Groovy replace the traditional manifest.xml with a newly-defined package.xml, defined in REP 127 [11].

share/

ROS Fuerte used share to install all ROS manifest and package-relative resources. That included package-relative binaries, which is contrary to normal FHS standards as share/ is intended for "read-only architecture independent data files" [8].

As documented above, some archtecture-independent files (like launch and parameter files) are still installed to share/<ros-package-name>/. But, some that landed there in Fuerte are now omitted or stored somewhere else:

  • manifest.xml is no longer used
  • stack.xml is no longer used
  • package-relative-executables (both binary and script) moved to lib/<ros-package-name>/ instead.

References

[1]Wikipedia: Filesystem Hiearchy Standard (http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard)
[2]ROS filesystem environment variables (http://ros.org/doc/independent/api/rospkg/html/environment.html)
[3]ROS Manifest XML Tags Reference (http://ros.org/doc/independent/api/rospkg/html/manifest_xml.html)
[4]REP 109: Unary Stacks (http://ros.org/reps/rep-0109.html)
[5]REP 102: ROS Install Target (http://ros.org/reps/rep-0102.html)
[6]ROS Fuerte Migration Guide (http://ros.org/wiki/fuerte/Migration)
[7]REP 124: Changes to roslaunch and rosrun for REP 122 and catkin build system (http://ros.org/reps/rep-0124.html)
[8]Filesystem Hierarchy Standard 2.3 (http://www.pathname.com/fhs/pub/fhs-2.3.pdf)
[9]REP 123: ROS_ETC_DIR, ROS_DISTRO environment variables and ROS_ROOT changes (http://ros.org/reps/rep-0123.html)
[10]Catkin build system documentation (http://ros.org/wiki/catkin)
[11]REP 127: Specification of package manifest format (http://ros.org/reps/rep-0127.html)