Overview

The srs_mixed_reality server is an component of the SRS Project for elderly care robot named Care-O-Bot. Using the information from the map server, Household database and SRS Knowledge Database it provides augmented reality stream for the UI_PRI user interface.

System architecture

The Mixed reality server consists of 4 nodes:

MRS

This node streams as a standard MJPEG/HTTP videostream the map information and augmented reality – household objects such a furniture and graspable objects. It also provides the functionality to stream selected ROS image topics.

The MRS node subscribes to /control_mrs topic which feeds object information for the augmented map drawing. The topic is published by the ControlMRS node.

ControlMRS

This node generates the combined information for the augmented reality using service calls to the SRS Household Database (info about the object size, grasping possibility and icon) and SRS Knowledge Database (info about name, class and pose).

The published topic (/control_mrs) structure is:

string command # Can be ADD or REMOVE (add or remove object from augmented map)

uint32 virtual_obj_number # the unique number of the object on the virtual map

uint32 virtual_obj_count  # total number of the objects drawn on current frame of the virtual map

string topic # name of the image topic which will be augmented

int32 id # ID of the object in HH DB

string label # Text to label the drawn object

int32 type # Possible types:
# 1 - CONTOUR_RECTANGLE; (only for augmentation level 2)
# 2 - CONTOUR_ELIPSE; (only for </nowiki>augmentation level 2)
# 3 - USE IMAGE (only for augmentation level 3)

float32 x # Center of the drawn object on x axis

float32 y # Center of the drawn object on y axis

float32 width # Width of the object bounding box

float32 height # Height of the object bounding box

float32 angle # Rotation of the object in Degrees

string clr 
# clr - Color of the object representation shape in HTML color – for exmple #FF0000 red
# (valid only for augmentation level 2)

sensor_msgs/Image image # topview image icon for the object (only for augmentation level 3)

RealityPublusher

This node provide information to the UI about the objects located on the augmented map.

The published topic - /map_active_areas consists of ros header and array of data which conatains all the objects presented on the virtual map.

Header header

uint32[] object_id # IDs of the object in Household Database

string[] object_name # Object names – eg. Table, Fridge and etc.

string[] object_base_shape # Object base shapes (currently rectangle or ellipse)

uint32[] object_type # Object types such as furniture, unknown.

uint32[] x1 # Top left corners of object bounding box (x axis) in pixel coordinates

uint32[] y1 # Top left corners of object bounding box (y axis) in pixel coordinates

uint32[] x2 # Bottom right corners of object bounding box (x axis) in pixels

uint32[] y2 # Bottom right corners of object bounding box (y axis) in pixels

uint32[] angle # Bounding box rotation angles in degrees

bool[] graspable # Grasping possibility 

MapTF

This node converts the coordinates, and sizes from ROS metric to pixel coordinates on virtual map and vice versa. It is provided as a ROS service srs_mixed_reality_server/MapTransform:

uint32 transform_type 
# Transform type can be:
# 1 - transform position from metric to map pixel coordinates
# 2 - transform size from metric to map pixel size
# 3 - transform position from pixel to metric coordinates
# 4 - transform size from pixels on map to metric
geometry_msgs/Point world_coordinates (input point in metric - types 1 and 2 or pixels - types 3 and 4)
---

geometry_msgs/Point map_coordinates (output point in pixels - types 1 and 2 or metric - types 3 and 4)

It also publishes the position of the robot in pixel map coordinates on topic /maptf.

Installation

These SRS package is not open to the public. If you want to run the packages, please write to goa@ipa.fhg.de, fmw@ipa.fhg.de. You can then fetch the source from the github.

rosmake srs_mixed_reality_server

Configuration

The node can be configured via its launch file – MRS.launch. You can customize the virtual map size and the augmentation level – 0 draw map only, 1 draw map and graphical representation of the objects (rectangles and ellipses) and 2 draw map with image icons.

<param name="mapWidth"           value="1024" type="int"/>
<param name="mapHeight"          value="1024" type="int"/>
<param name="augmentationLevel"  value="2"    type="int"/>

To start Mixed Reality Server

roslaunch srs_mixed_reality_server MRS.launch

Dependencies

In order to generate augmented map you need following components compiled and running:

roscore

Care-O-bot simulation in gazebo or the real robot

cob_2dnav (Care-O-Bot navigation stack)

srs_knowledge (SRS Knowledge Database)

srs_object_database (The Household Objects Database)

rosbridge (to communicate to the UI_PRI)

Testing

You can test the output result using the UI_PRI on iPad or using a standard websockets compliant browser – as Google Chrome.

To view a snapshot - please open: 
http://127.0.0.1:8080/snapshot?topic=/map 

To view the live stream - please open: 
http://127.0.0.1:8080/stream?topic=/map

You can monitor any ROS image topic for example:
http://127.0.0.1:8080/stream?topic=/stereo/right/image_raw 

Note: replace 127.0.0.1 with the IP of your ROS server where you run the MRS.