Cosmin D. Pantilie

SORT-SGM: Subpixel Optimized Real-Time Semiglobal Matching for Intelligent Vehicles

The suitability of stereo algorithms for intelligent vehicle applications is conditioned by their ability to compute dense accurate disparity maps in real time. In this paper, an original stereo reconstruction system that is designed for automotive applications is presented. The system is based on the semiglobal matching algorithm (SGM), which is widely known for its high quality and potential for real-time implementation. Several improvements that target the matching, disparity optimization, and disparity refinement steps are proposed. Pixel-level matching uses the census transform because of its invariance to intensity differences due to camera bias or gain that affects the images. The huge memory bandwidth requirements for the SGM disparity optimization step are reduced through a new integration strategy. At the subpixel level, accuracy is increased by devising a new methodology for generating dedicated subpixel interpolation functions. Using this methodology, two novel subpixel interpolation functions for the SGM algorithm are implemented and evaluated. The proposed algorithm has been implemented on a graphics processing unit in the Compute Unified Device Architecture (CUDA). The result is an increased speed and accuracy algorithm profiled for complex real-traffic scenarios. The proposed algorithm has been evaluated at a large scale, and evidence that was collected from both standard benchmarks and real-world images confirm the findings and show a significant improvement over existing solutions.

Real-time semi-global dense stereo solution with improved sub-pixel accuracy

In this work we focus on creating a real-time dense stereo reconstruction system with accurate sub-pixel estimation. We selected the Semi-Global Matching method as the basis of our system due to its high quality and possible real-time implementations. In our solution we use the Census transform as the matching metric because our results show that it can reduce the matching errors for traffic images compared to classical solutions. We also propose several modifications to the original Semi-Global algorithm to improve the sub-pixel accuracy and the execution time. One of these proposals is the reduction in the number of optimization directions without affecting the results. The second modification is a correction of the energy function to reduce the spread of depth values. Besides these improvements, the paper also introduces a new aggregation method used to reduce the spread of sub-pixel values. Finally we propose a new method to generate sub-pixel interpolation functions based on real-world data. The result of these enhancements is a significant improvement in sub-pixel accuracy. The system was implemented and evaluated on a current generation GPU with a running time of 19ms for image having the resolution 512×383.

Real-time obstacle detection in complex scenarios using dense stereo vision and optical flow

Mobile robots as well as tomorrows intelligent vehicles acting in complex dynamic environments must be able to detect both static and moving obstacles. In intersections or crowded urban areas this task proves to be highly demanding. Stereo vision has been extensively used for this task, as it provides a large amount of data. Since it does not reveal any motion information, static and dynamic objects immediately next to each other, or closely positioned obstacles moving in different directions are often merged into a single obstacle. In this paper we address these problems through a powerful fusion between 3D position information delivered by the stereo sensor and 3D motion information, derived from optical flow, in a depth-adaptive occupancy grid. The proposed model is presented and then applied for determining obstacle localization, orientation and speed.

Real-time obstacle detection using dense stereo vision and dense optical flow

Accurate detection of moving obstacles from a moving vehicle is at the core of safe autonomous driving research. Stereo vision based sensors have been extensively used for this task as they are passive and provide a large amount 3D and 2D data. However, since no motion information is revealed, in intersections or crowded urban areas, static and dynamic objects immediately next to each other, or closely positioned obstacles moving in different directions are often merged into a single obstacle leading to dangerous misinterpretations. In this paper we address these problems through a powerful fusion between dense stereo vision and dense optical flow in a depth-adaptive occupancy grid framework. The proposed fusion model is presented and then applied for obstacle detection in an intersection assistance system.

Particle Grid Tracking System Stereovision Based Obstacle Perception in Driving Environments

This paper presents an occupancy grid tracking system based on particles, and the use of this system for dynamic obstacle detection in driving environments. The particles will have a dual nature they will denote hypotheses, as in the particle filtering algorithms, but they will also be the building blocks of our modeled world. The particles have position and speed, and they can migrate in the grid from cell to cell depending on their motion model and motion parameters, but they will also be created and destroyed using a weighting-resampling mechanism specific to particle filter algorithms. An obstacle grid derived from processing a stereovision-generated elevation map is used as measurement information, and the measurement model takes into account the uncertainties of the stereo reconstruction. The dynamic occupancy grid is used for improving the quality of the stereovision-based reconstruction as oriented cuboids. The resulted system is a flexible, real-time tracking solution for dynamic unstructured driving environments, and a useful tool for extracting intermediate dynamic information that can considerably improve object detection and tracking.

Statistical method for sub-pixel interpolation function estimation

Depth accuracy is one of the most important characteristics for sensors used in distance estimation. Stereo-vision systems employ sub-pixel interpolation to achieve such accuracy. Literature in this domain is usually dedicated to simple window based stereo solutions. There are currently several new stereo algorithms developed to counter pixel level errors, but they neglect sub-pixel results. We propose the use of function fitting to generate interpolation functions optimized for each algorithm type. Dedicated interpolation functions require the mathematical model of the algorithm. In the proposed methodology of generating the interpolation function the explicit model of the stereo algorithm is replaced by modeling the data distribution resulted from a pre-defined input. Several transformations are also proposed to reduce the dimensionality of the fitting data without loosing any information. The most accurate match for the fitting data-set was a sinusoidal function, a novel shape for sub-pixel interpolation. The function shows a significant improvement compared to legacy solutions, by reducing the error magnitude by several factor for both synthetic and real scenarios. sf]Y

Real-time semi-global matching using segmentation and plane fitting for improved accuracy on the GPU

The suitability of stereo algorithms for intelligent vehicle applications is conditioned by their ability to compute dense, accurate disparity maps in real time. In this paper a realtime stereo reconstruction system designed for automotive applications is presented. The system is based on the Semi-Global Matching algorithm, widely known for its high quality and potential for real-time implementation. The accuracy of this algorithm is increased by a new segmentation and plane fitting based approach which is used for refining the estimated disparities. The main contribution is scalable solution for obtaining a significant enhancement of the matching quality in textureless or otherwise difficult areas that also maintains the algorithms real-time processing capabilities. In this sense, a pixel-centered approach is proposed and compared with two existing plane fitting strategies. The different approaches are also analyzed from the perspective of matching quality and parallelization cost on GPU architectures. Results on the Middlebury benchmark and real-world images confirm the strength of the new method. The complete stereo reconstruction system was implemented and evaluated on a current generation GPU.

Optimizing the Census Transform on CUDA enabled GPUs

The Census Transform is one of the most widely used matching metrics in problems that involve correspondence search such as stereo reconstruction and optical flow. Graphic processing units (GPUs) have become popular platforms for such computation intensive applications that expose a high degree of data parallelism. Their evolution as a platform for general purpose computing by continuously adding new hardware features has improved performance for many applications but it has also expanded the set of possible implementations choices up to the point where guidelines alone are not sufficient for optimum performance. What is the best implementation in the case of the Census Transform? This paper will answer that question by benchmarking all major possible implementations. Its aim is to provide an optimal implementation of the Census Transform on a current generation graphics processing unit using the Compute Unified Device Architecture (CUDA). The results have value reaching far beyond the Census Transform and provide insight for applications where non-separable 2D convolutions are present.

Real-Time Image Rectification and Stereo Reconstruction System on the GPU

The increase in computational power of consumer graphic cards has successfully motivated adaptation of stereo algorithms to this kind of hardware. In order to solve the stereo correspondence problem efficiently, the images need to be rectified and lens distortions need to be removed. This paper presents an efficient two step solution for rectifying and correcting lens distortions in images captured using a pair of stereo cameras. The first step consists of a one-time, off-line calculation of a look-up table, based on the calibration parameters, for each of the two cameras. The second step computes the final pixel intensities based on the pre-calculated mappings stored in the look-up table. The GPU implementation proposed makes use of the inherent parallelism in a cost-effective manner, making the method suitable for rectifying high resolution images in real-time. Results are compared against an optimized CPU-based implementation, written in assembly language using MMX instructions, for reference. The complete stereo reconstruction system was implemented and evaluated on a current generation GPU and offers a running time of 11ms for images with resolution 512x383.

Statistical methods for automatic segmentation of elastographic images

Elastography is a new ultrasonic method for measuring tissuespsila elasticity. Besides many and widely acknowledged benefits, the method suffers severe limitations due to the high motion sensitivity and inter-operator dependency to the point where it provides only qualitative information, not having, until now, any real quantification means. In this paper we present an automatic segmentation method for elastographic images based on statistical techniques. First a probabilistic model is built for every pixel in the image, derived by processing a video sequence instead of a single image. The built image contains the values with highest probability for each pixel. Next a DAEM (Deterministic Annealing Expectation Maximization) method is used for automatic image segmentation. Finally a numerical quantification of tissue elasticity is provided based on the segmentation.