Ioana Gheta

Data association in a world model for autonomous systems

This contribution introduces a three pillar information storage and management system for modeling the environment of autonomous systems. The main characteristics is the separation of prior knowledge, environment model and sensor information. In the center of the system is the environment model, which provides the autonomous system with information about the current state of the environment. It consists of instances with attributes and relations as virtual substitutes of entities (persons and objects) of the real world. Important features are the representation of uncertain information by means of Degree-of-Belief (DoB) distributions, the information exchange between the three pillars as well as creation, deletion and update of instances, attributes and relations in the environment model. In this work, a Bayesian method for fusing new observations to the environment model is introduced. For this purpose, a Bayesian data association method is derived. The main question answered here is the observation-to-instance mapping and the decision mechanisms for creating a new instance or updating already existing instances in the environment model.

7 – Bayesian methods for image fusion

The Bayesian fusion methodology bases upon a solid mathematical theory, provides a rich ensemble of methods and allows an intuitive interpretation of the fusion process. It is applicable independently of the goal pursued by image fusion, at different abstraction levels, and also if different kinds of image data have to be fused. It allows transformation, fusion, and focusing, i.e. it fulfils the basic requirements that a reasonable fusion methodology has to satisfy.

Robust Depth Estimation by Fusion of Stereo and Focus Series Acquired with a Camera Array

In order to obtain depth information from intensity sensors without auxiliary means, an image series is needed, gathered by varying at least the geometrical or focus position of the camera. Each of the fusion methods imposes certain constraints on the observed scene. With the aim of alleviating these restrictions, this contribution presents an algorithm to fuse combined stereo and focus series using energy functionals. A camera array is employed to record the series of images simultaneously

Three pillar information management system for modeling the environment of autonomous systems

This contribution is about an information management and storage system for modeling the environment of autonomous systems. The three pillars of the system consist of prior knowledge, environment model and sensory information. The main pillar is the environment model, which supplies the autonomous system with relevant information about its current environment. For this purpose, an abstract representation of the real world is created, where instances with attributes and relations serve as virtual substitutes of entities (persons and objects) of the real world. The environment model is created based on sensory information about the real world. The gathered sensory information is typically uncertain in a stochastic sense and is represented in the environment model by means of Degree-of-Belief (DoB) distributions. The prior knowledge contains all relevant background knowledge (e. g., concepts organized in ontologies) for creating and maintaining the environment model. The concept of the three pillar information system has previously been published. Therefore this contribution focuses on further central properties of the system. Furthermore, the development status and possible applications as well as evaluation scenarios are discussed.

A novel region-based approach for the fusion of combined stereo and spectral series

This contribution proposes a novel approach for image fusion of combined stereo and spectral series acquired simultaneously with a camera array. To this purpose, nine cameras are equipped with spectral filters (50 nm spectral bandwidth) such that the visible and near infrared parts of the spectrum (400-900 nm) are observed. The resulting image series is fused in order to obtain two types of information: the 3D shape of the scene and its spectral properties. For the registration of the images, a novel region based registration approach which evaluates the gray value invariant features (e.g. edges) of regions in segmented images is proposed. The registration problem is formulated by means of energy functionals. The data term of our functional compares features of a region in one image with features of an area in another image, such that an additional independency of the form and size of the regions in the segmented images is obtained. As regularization, a smoothness term is proposed, which models the fact that disparity discontinuities should only occur at edges in the images. In order to minimize the energy functional, we use graph cuts. The minimization is carried out simultaneously over all image pairs in the series. Even though the approach is region based, a label (e.g. disparity) is assigned to each pixel. The result of the minimization approach consists of a disparity map. By means of calibration, we use the disparity map to compute a depth map. Once pixel depths are determined, the images can be warped to a common view, such that a pure spectral series is obtained. This can be used to classify different materials of the objects in the scene based on real spectral information, which cannot be acquired with a common RGB camera.

Fusion kombinierter Stereo- und Fokusserien zur 3D-Rekonstruktion (Fusion of Combined Stereo and Focus Series for 3D Reconstruction)

Um 3D-Information aus Bildern auch für Szenen mit schwacher oder periodischer Struktur mit hoher Robustheit zu erzeugen, werden kombinierte Stereo- und Fokusserien fusioniert. Dazu wird eine Kombination von depth from stereo und depth from (de)focus verwendet. Zur Analyse des Problems wird ein Signalmodell vorgestellt, das die Abbildung durch ein Kamera-Array bei Defokussierung strahlenoptisch beschreibt. Zur Formulierung der Fusionsaufgabe dient ein Energiefunktional, das mittels numerischer Standardverfahren minimiert wird. Die erzielten Ergebnisse der Fusionsmethode an unterschiedlichen Szenen zeigen eine erhebliche Verbesserung der Tiefenschätzung gegenüber Standardverfahren. Combined stereo and focus image series are fused for obtaining robust 3D information. The combination of depth from stereo and depth from (de)focus leads to good results also when dealing with scenes presenting periodical or weak structure. For analyzing the problem in ray optics, a signal model is presented which describes the imaging of a camera array taking into consideration a defocussed projection. The problem is formulated by using energy functionals which are minimized by means of standard algorithms. The results obtained by applying the method to several scenes show considerably improved results compared to the use of standard algorithms.

Flächenbasierte Registrierung kombinierter Stereo- und Spektralserien

Zusammenfassung Kamera-Arrays und kombinierte Bildserien werden verwendet, um zeitsparend heterogene Informationen über eine Szene zu erfassen. Als Beispiel beinhalten kombinierte Stereo- und Spektralserien sowohl räumliche als auch spektrale Information über eine Szene. Als Grundlage für eine Auswertung des Steroeffekts in solchen kombinierten Bildserien wird im vorliegenden Beitrag ein neuartiges flächenbasiertes Registrierungsverfahren dargestellt. Dabei werden die Bilder der Serie zunächst segmentiert und anschließend für die erhaltenen Regionen Merkmale extrahiert. Anhand dieser Merkmale, die im Wesentlichen Kanten in den Bildern beschreiben, werden Korrespondenzen zwischen Regionen bestimmt und daraus Tiefenkarten berechnet. Das Registrierungsproblem wird durch Energiefunktionale modelliert und mittels eines modifizierten Graph-Cuts-Verfahrens minimiert. Beispiele veranschaulichen die Vorgehensweise. Abstract The fast acquisition of heterogeneous information about a scene can be approached by image series with more than one varied parameter, also referred to as combined image series acquired with camera arrays. Examples of such series are combined stereo and spectral series which contain both spatial and spectral information about the scene. We propose in this contribution a new region-based registration method for evaluating the stereo effect in such combined image series. The images are first segmented and then features of the resulting regions are extracted. The features mainly describe edges of the regions and are used to determine correspondences between one or several regions. Energy functionals are employed to model the registration problem. The solution is found by minimizing the energy functional by means of a modified graph-cuts algorithm. Examples are presented to visualize the methods proposed.

3D Fusion of Stereo and Spectral Series Acquired With Camera Arrays

One of the main requirements of industrial visual inspection is that the information acquisition is accomplished in real-time. Camera arrays are a promising solution since they offer the possibility of simultaneous image acquisition. Moreover, the acquisition parameters of the different cameras can be varied. Due to dropping prices of industrial cameras, a large number of cameras can be employed in a camera array for automated visual inspection. The advantages offered by camera arrays come with a price. Simultaneously triggering the cameras results in obtaining image series that contain a stereo effect. If more acquisition parameters (e. g., focus, different spectral filters) are varied, the obtained image series are combined image series, i. e., the images differ in more than one effect. For example, if the cameras are equipped with spectral filters, the obtained image series are combined stereo and spectral series, i. e., the images differ due to both the stereo effect and the acquisition in different parts of the spectrum. However, the main advantage of such image series is that they contain different types of information gained simultaneously: in this case, it is spatial information due to the stereo effect and spectral information due to the use of spectral filters. The challenge consists in fusing the image series, since the different types of information in the combined image series cannot be evaluated separately. The present chapter deals with different methods of fusing combined stereo and spectral images in order to obtain both spatial and spectral information. For obtaining the spatial information, region based image registration methods for the exploitation of the stereo effect are presented. The problem is modeled with energy functionals, which are minimized by state-of-the-art methods, e. g., dynamic programming or graph cuts. With the help of the obtained spatial information (in form of depth maps), the spectral information can be extracted and further employed, e. g., for material classification or an improved object detection.

Sensoreinsatzplanung und Informationsfusion zur Umgebungsexploration

Zusammenfassung Die Wahrnehmung und Modellierung einer dynamischen Umwelt stellen Schlüsselkomponenten intelligenter Systeme dar. In diesem Beitrag wird dazu einerseits eine Methodik vorgestellt, um aus verfügbaren Eingangsdaten eine optimale Auswahl zu treffen. Andererseits wird ein objektorientiertes Umweltmodell vorgeschlagen, das eine laufende Fusion vorhandenen Wissens mit neuen Sensorinformationen gestattet. Sämtliche Verfahren basieren auf der Bayes´schen Statistik in einer objektiven “Degree of Belief”-Interpretation. Einsatzgebiete werden am Beispiel humanoider Roboter und autonomer Fahrzeuge aufgezeigt. Abstract The abilities to sense and model a dynamic environment are key components of intelligent systems. In this contribution, firstly a methodology is presented to make an ideal selection of the input data available. Then, an object oriented environment model is proposed which allows a continuous fusion of existing knowledge with new sensor information. All methods are based on Bayesian statistics in an objective “degree of belief” interpretation. Application areas are demonstrated by the examples of humanoid robots and autonomous vehicles.

Fusion of combined stereo and spectral series for obtaining 3D information

This contribution presents a fusion method for spectral series with the main purpose of obtaining 3D information. The image series to be fused are combined stereo and spectral series gained with a camera array. Therefore, in order to register them, features that are invariant with respect to the varying gray values in the spectral images are extracted. The proposed approach is region based and uses characteristics like size, position and shape for registration. The regions are identified using the watershed transformation. The fusion problem is modeled using energy functionals that are to be optimized. They take into consideration the size, position, shape and correlation of the regions. Using the implemented algorithm, several scenes have been reconstructed. The experimental results show that the proposed method delivers reliable and accurate dense depth maps of combined stereo and spectral series.