Petra Kocmanova

Fusion of thermal imaging and CCD camera-based data for stereovision visual telepresence

The aim of this paper is to describe the fusion of data acquired from a CCD camera and a thermal imager. The fusion is realized by means of spatial data from a TOF camera to ensure “natural” representation of a robots environment; thus, the thermal and color-related data are comprised in one stereo image presented to a binocular, head-mounted display. The data acquisition is performed using a sensor head, which is placed on an Orpheus-X3 robot; both the head and the robot were developed by our working group. The head contains five matrix sensors: a pair of CCD cameras, a pair of thermal imagers, and one TOF camera. After the geometrical calibration of each sensor, the positions of the sensors in 6DOFs are computed. The corresponding data from the CCD camera and the thermal imager are subsequently determined via homogeneous and perspective transformations. The result consists in an image containing aligned data from the CCD camera and the thermal imager for each eye.

Robot mapping with range camera, CCD cameras and thermal imagers.

This paper describes a multispectral data fusion system for real-time telepresence and digital mapping applicable in reconnaissance robotics. The system was developed by our team for CASSANDRA, a robotic group comprising several robots with different features controllable from a single operator station. Special emphasis is put on cooperative mapping and problems arising from the fact that each robot has distinct sensors and computing capabilities; at the same time, however, it is important to stress that all the robots can supply valuable data to the global map. The system is currently able to process distance data, color data, and thermal information, but it is also prepared for extension such that other parts of the electromagnetic spectrum (e.g., the near infrared region) or mission-specific (e.g., radiation) data can be easily added to the map. The fusion quality and map features were studied both theoretically and via field experiments. The data acquisition is performed using a sensor head placed on an Orpheus-X3 robot; both the head and the robot were developed by our working group. The head contains five matrix sensors: a pair of CCD cameras, a pair of thermal imagers, and one TOF camera.

Color and Thermal Image Fusion for Augmented Reality in Rescue Robotics

At the beginning of this article, the authors address the main problems of todays remotely-operated reconnaissance robots. The reconnaissance robots Orpheus-AC, Orpheus-AC2 and Orpheus-Explorer, made in the Department of Control and Instrumentation (DCI), are then shortly described. Since all the described robotic systems use visual telepresence as the main control technique, visual information from the robots surroundings is essential for the operator. For this reason, the authors make a fusion of data from a Charge-Coupled Device (CCD) color camera, and a thermovision camera to provide the operator with data in all visibility conditions, such as complete darkness, fog, smoke, etc.

Breath Analysis Using a Time-of-Flight Camera and Pressure Belts.

The proper way of breathing is important for everyone. Healthy people often do not follow respiration until breathing problems start-during stress or during sport activity in physiological cases. More serious cases are stroke, injury, or surgery of the chest and others. So, learning to breathe correctly and/or breathing diagnosis is considerable for many reasons. Two novel methods of breath analysis suitable for diagnostics and rehabilitation are presented. The first technique utilizes pressure belts fastened to the patients belly and chest, and the second method relies on a SwissRanger SR-4000 time-of-flight camera. The measurement principles are described together with the advantages and disadvantages of the applied techniques. The SwissRanger camera depth calibration is proposed to facilitate better results during the breath analysis. The methods are tested on a group of students to provide a comparison of their individual performances. As it was demonstrated, presented methods proved to work reliably. The method based on time-of-flight camera seems to be more suitable for diagnosis, while the method based on pressure belts is more suitable for rehabilitation and biofeedback applications.

Multispectral data fusion for robotic reconnaissance and mapping

The aim of the paper is to describe the data-fusion from optical sensors for mobile robotics reconnaissance and mapping. Data are acquired by stereo pair of CCD cameras, stereo pair of thermal imagers, and TOF (time-of-flight) camera. The fusion is realized by means of spatial data from a TOF camera to ensure “natural” representation of a robots environment; thus, the thermal and CCD camera data are comprised in one stereo image presented to a binocular, head-mounted display. The data acquisition is performed using a sensor head, which is placed on an Orpheus-X3 robot; both the head and the robot were developed by our working group. After the geometrical calibration of each sensor, the positions of the sensors in 6DOFs are computed. The corresponding data from the CCD camera and the thermal imager are subsequently determined via homogeneous and perspective transformations. The result consists in an image containing aligned data from the CCD camera and the thermal imager for each eye. TOF camera calibration and its impact to the precision of fusion is described. Although the fusion is used for two different tasks - automatic environment mapping and visual telepresence, the utilised calibration and fusion algorithms are, in principle, the same.

Multi-robot system for disaster area exploration

CASSANDRA robotic system developed at LTR s.r.o. company and Brno University of Technology is described. The system contains an operator’s station controlled with one operator and a couple of robots – small and big ground robots, flying robots (quadrocopters), and mapping robot. The robots are primarily controlled by the operator with an advanced user interface with visual telepresence and augmented reality. Nevertheless, the robots include the possibility of semiautonomous operation based on self-localisation. The user interface consists of a computer, joypad, head-mounted display with inertial head-tracker, communication device, and Cassandra software developed by our team in Microsoft .NET. Orpheus class robots are described in the text. The robots are made to be reliable and to be able to work in extreme conditions, they are tested by a series of MIL-STD military tests for environmental parameters, EMC, vibrations and shocks, contamination/decontamination, etc. Orpheus-X3 is a general US&R robot with enhanced victim search capabilities, Orpheus-HOPE is made for water contamination measurements, Orpheus-AC2 is a ruggedized version for environmental parameter measurement. Two flying drones developed completely by our team are described, as well as EnvMap mapping robot for realtime construction of spatial digital maps with texture mapping. All the robots can be controlled with the help of visual telepresence and augmented reality – that makes robot control much more intuitive, and lets the rescuer concentrate on the mission itself. The control station may be used as a self-containing wearable system. The fusion system with multispectral measurement containing tricolor cameras, thermal imagers and TOF camera is described.

3D proximity laser scanner calibration

This paper deals with the calibration of scanner Velodyne HDL-64E that is used in field robotic applications. The aim of calibration is to improve the performance in advanced mapping algorithms such as robot evidence grids or octree. The proposed novel calibration method is founded on conditional adjustment for correlated measurements and using close range photogrammetry. The proposed calibration significantly improves scanner accuracy.

Multispectral Stereoscopic Robotic Head Calibration and Evaluation

The aim of the paper is to describe the data-fusion from optical sensors for mobile robotics reconnaissance and mapping. Data are acquired by stereo pair of CCD cameras, stereo pair of thermal imagers, and TOF (time-of-flight) range camera.

Effective Calibration and Evaluation of Multi-Camera Robotic Head

The paper deals with appropriate calibration of multispectral vision systems and evaluation of the calibration and data-fusion quality in real-world indoor and outdoor conditions. Checkerboard calibration pattern developed by our team for multispectral calibration of intrinsic and extrinsic parameters is described in detail. The circular object for multispectral fusion evaluation is described as well. The objects were used by our team for calibration and evaluation of advanced visual system of Orpheus-X3 robot that is taken as a demonstrator, but their use is much wider, and authors suggest to use them as testbed for visual measurement systems of mobile robots. To make the calibration easy and straightforward, the authors developed MultiSensCalib program in Matlab, containing all the described techniques. The software is provided as publicly available, including source code and testing images.

Creating Three-Dimensional Computer Models Using Robotic Manipulator and Laser Scanners

Abstract This paper presents scanning system providing three-dimensional models. Use of laser scanner mounted on robotic manipulator provides very flexible device capable of building models of both tiny detailed structures and large object. It could be used in many various applications, especially in health care, where it brings lot of advantages comparing to present scanning systems. Mechanical constitution, interfacing approaches and operating principles of this device are described.