object_recognition: object_recognition_core | object_recognition_msgs | object_recognition_server

Package Summary

Uses opencv to object recognition.

• Author: Alexander Shishkov, Victor Eruhimov
• License: BSD
• Repository: wg-ros-pkg
• Source: svn https://code.ros.org/svn/wg-ros-pkg/branches/trunk_diamondback/stacks/object_recognition/textured_object_detection

Textured_object_detection guide

Training base format

Training base consists of a set of folders with images and PCD files for every object:

• /object1
  • 1.png - first image for object1.
  • 1.pcd - corresponding point cloud.
  • 2.png
  • 2.pcd
  • ...
  • info.txt - camera parameters
• /object2
  • ...
• ...
• /objectn
• info.txt - camera parameters for test image
• config.txt - configuration file

The format of config.txt file:

object_folder_name1 images_in_object_folder

object_folder_name2 images_in_object_folder

...

You can find examples of training sets in http://vault.willowgarage.com/wgdata1/vol1/binpick_bags/train/.

Training base creation

This package is capable of creating a training base from a set of bag files. Each bag file should contain an object on a turntable recorded with a stereo camera. Training base creation is currently available only in a manual mode (due to some failures in automatic table plane segmentation). A tool for automatic processing of bag files is in progress .

Initialization

First you should convert topics from your bag file to files on HDD. There is a tool is for doing it (sources in image_subscriber.cpp).

You can use "image_sub.launch" to run it:

<launch>
  <node pkg="textured_object_detection" type="is" name="is" output="screen">
      <remap from="ci_left_in" to="/narrow_stereo/left/camera_info"/>
      <remap from="ci_right_in" to="/narrow_stereo/right/camera_info"/>
      
      <remap from="im_left_in" to="/narrow_stereo/left/image_raw"/>
      <remap from="im_right_in" to="/narrow_stereo/right/image_raw"/>
      
      <remap from="imt_left_in" to="/narrow_stereo_textured/left/image_raw"/>
      <remap from="imt_right_in" to="/narrow_stereo_textured/right/image_raw"/>
      
      <remap from="dir" to="/home/alex/temp/test" />
  </node>
  <param name="have_textured" value="true" />
</launch>

You should set the corresponding names of topics; dir - path, where directories with corresponding files will be created. If your bag file does not contain textured light images, you need to set have_textured parameter to false.

Then you should launch this file:

roslaunch textured_object_detection image_sub.launch

and you should run in parallel rosbag play for your bag file:

rosbag play 9403_All.bag

After playing the bag file a folder (see dir parameter) will contain the next files and subfolders:

• left - images from left camera (images will be named "#.png")
• right - images from right camera

• left_tex - textured images from left camera (this subfolder exists only if parameter have_textured equals true)

• right_tex - textured images from right camera
• left_info.txt - camera parameters for left_camera
• right_info.txt - camera parameters for right_camera

Sometimes in order to avoid train/test overlap, we use odd frames for training set, and even -- for the test set. For this purpose you can use half_script.py.

Launch command:

half_script.py <srcdir> <dstdir> <have_textured>

The script copies all images from left and right (and left_tex and right_tex if <have_textured> > 0) subfolders with odd indexes to corresponding subfolders in <dstdir>. Images from the left camera (from left subfolder) with even indexes are copied to test subfolder of <dstdir>. Also it copies left_info.txt and right_info.txt to <dstdir>. If you don't set the third parameter:

half_script.py <srcdir> <dstdir>

the script initializes <have_textured> parameter with 1. Also there are some restrictions:

• the script skips first 50 images with even indices (because in our bag files the first 100 images are usually the same

• the script writes not more than 200 images to the test subfolder

Training base pruning

For pruning samples in the train base you should use crop binary file from textured_object_detection package (source in crop_object.cpp). This tool iteratively puts into the training base only those samples that are recognized poorly.

Here is an example of a launch command:

roscd textured_object_detection
bin/crop <srcdir> <object> <dstdir> -prosilica camera/transform.xml -config camera/prosilica1.config -skip-frames 15

<srcdir> must contain next files and subfolders:

• left - images from left camera (images must be named "#.png")
• right - images from right camera
• left_info.txt - camera parameters for left_camera
• right_info.txt - camera parameters for right_camera

If <srcdir> contains left_tex and right_tex subfolders, images from them will be used for disparity calculation. If you use prosilica, raw should also contain:

• pro - images from prosilica camera
• pro_info.txt - calibration info for prosilica

After launching this tool you will see a window like this: crop_screenshot.png

Then you should select an object on left side of the window. This is the only step that you have to do manually, because we have problems in automatic object localization. You are supposed to select a contour around an object (as tight as possible) with left mouse clicks. If the distance between the first and the last clicked points is less than 20 px, the contour will be closed. After contour closing you should press 's' key on your keyboard for saving this object. When the next unrecognized image is found, it is visualized in the window. And you are supposed to select a contour again.

Example of a selected object: selected_object.png

After the completion of the program <dstdir> will be contain subfolder with <object> name. After that you should copy this subfolder to your training folder and add to config.txt (in training folder) string from config.txt (in <dstdir>).

If the CHESSBOARD macro is turned on during compilation, the training will try to find a chessboard in each image and register poses of all subsequent frames relative to the first frame. The resulting pose during recognition will also be relative to the first frame. In order to help control accuracy of pose registration point clouds from different frames are visualized in rviz (fixed and target frames should be set to /pgraph, marker /tod_train/points should be turned on).

Recognition

rosrun textured_object_detection rs <training_base_folder_path> <test_image>
rosrun textured_object_detection rs <training_base_folder_path> <test_image> <test_camera_info>

<training_base_folder_path> - path to a training folder. Also you should select camera parameters of a camera that a test image was taken with. If you don't select it manually (see <test_camera_info> parameter) the info.txt file in the root of <training_base_folder_path> will be used.

rosrun textured_object_detection rt <training_base_folder_path> <test_base_folder_path> <views_count_in_every_object>

It is used for testing recognition algorithm quality.

rosrun textured_object_detection rtc <training_base_folder_path> <images_path> <views_count>
rosrun textured_object_detection rtc <training_base_folder_path> <images_path> <views_count> <test_camera_info>

Run recognize method on every image in images_path folder from 1.png to <views_count>.png.

Running as ros node

Package contains ros node (it's source code in node.cpp file) for object recognition. For running it you should use narrow.launch file:

<launch>
  <node pkg="textured_object_detection" type="tod" name="tod" output="screen">
    <remap from="image_in" to="/narrow_stereo/left/image_raw"/>
  </node>
  <param name="training_base_dir" value="/home/alex/tod_bases/train/" />
</launch>

You should set training_base_dir parameter to your training base directory and image_in parameter to your image topic.

Then you should run it with command:

roslaunch textured_object_detection narrow.launch

The node publishes the point cloud in the test camera coordinate system in the message /tod/points.