Stefan K. Gehrig

A Real-Time Low-Power Stereo Vision Engine Using Semi-Global Matching

Many real-time stereo vision systems are available on low-power platforms. They all either use a local correlation-like stereo engine or perform dynamic programming variants on a scan-line. However, when looking at high-performance global stereo methods as listed in the upper third of the Middlebury database, the low-power real-time implementations for these methods are still missing. We propose a real-time implementation of the semi-global matching algorithm with algorithmic extensions for automotive applications on a reconfigurable hardware platform resulting in a low power consumption of under 3W. The algorithm runs at 25Hz processing image pairs of size 750x480 pixels and computing stereo on a 680x400 image part with up to a maximum of 128 disparities.

6D-vision: fusion of stereo and motion for robust environment perception

Obstacle avoidance is one of the most important challenges for mobile robots as well as future vision based driver assistance systems. This task requires a precise extraction of depth and the robust and fast detection of moving objects. In order to reach these goals, this paper considers vision as a process in space and time. It presents a powerful fusion of depth and motion information for image sequences taken from a moving observer. 3D-position and 3D-motion for a large number of image points are estimated simultaneously by means of Kalman-Filters. There is no need of prior error-prone segmentation. Thus, one gets a rich 6D representation that allows the detection of moving obstacles even in the presence of partial occlusion of foreground or background.

Pushing the limits of stereo using variational stereo estimation

We examine high accuracy stereo estimation for binocular sequences that where obtained from a mobile platform. The ultimate goal is to improve the range of stereo systems without altering the setup. Based on a well-known variational optical flow model, we introduce a novel stereo model that features a second-order regularization, which both allows sub-pixel accurate solutions and piecewise planar disparity maps. The model incorporates a robust fidelity term to account for adverse illumination conditions that frequently arise in real-world scenes. Using several sequences that were taken from a mobile platform we show the robustness and accuracy of the proposed model.

Collision Avoidance for Vehicle-Following Systems

The vehicle-following concept has been widely used in several intelligent-vehicle applications. Adaptive cruise control systems, platooning systems, and systems for stop-and-go traffic employ this concept: The ego vehicle follows a leader vehicle at a certain distance. The vehicle-following concept comes to its limitations when obstacles interfere with the path between the ego vehicle and the leader vehicle. We call such situations dynamic driving situations. This paper introduces a planning and decision component to generalize vehicle following to situations with nonautomated interfering vehicles in mixed traffic. As a demonstrator, we employ a car that is able to navigate autonomously through regular traffic that is longitudinally and laterally guided by actuators controlled by a computer. This paper focuses on and limits itself to lateral control for collision avoidance. Previously, this autonomous-driving capability was purely based on the vehicle-following concept using vision. The path of the leader vehicle was tracked. To extend this capability to dynamic driving situations, a dynamic path-planning component is introduced. Several driving situations are identified that necessitate responses to more than the leader vehicle. We borrow an idea from robotics to solve the problem. Treat the path of the leader vehicle as an elastic band that is subjected to repelling forces of obstacles in the surroundings. This elastic-band framework offers the necessary features to cover dynamic driving situations. Simulation results show the power of this approach. Real-world results obtained with our demonstrator validate the simulation results

Fast detection of moving objects in complex scenarios

More than one third of all traffic accidents with injuries occur in urban areas, especially at intersections. A suitable driver assistance system for such complex situations requires the understanding of the scene, in particular a reliable detection of other moving traffic participants. This contribution shows how a robust and fast detection of relevant moving objects is obtained by a smart combination of stereo vision and motion analysis. This approach, called 6D Vision, estimates location and motion of pixels simultaneously which enables the detection of moving objects on a pixel level. Using a Kalman filter attached to each tracked pixel, the algorithm propagates the current interpretation to the next image. In addition, a Kalman filter based ego-motion compensation is described that takes advantage of the 6D information. This precise information enables us to discriminate between static and moving objects exactly and to obtain a better prediction. This speeds up tracking and a real-time implementation is achieved. Examples of critical situations in urban areas exhibit the potential of the 6D Vision concept which can also be extended to robotics applications.

Exploiting the Power of Stereo Confidences

Applications based on stereo vision are becoming increasingly common, ranging from gaming over robotics to driver assistance. While stereo algorithms have been investigated heavily both on the pixel and the application level, far less attention has been dedicated to the use of stereo confidence cues. Mostly, a threshold is applied to the confidence values for further processing, which is essentially a sparsified disparity map. This is straightforward but it does not take full advantage of the available information. In this paper, we make full use of the stereo confidence cues by propagating all confidence values along with the measured disparities in a Bayesian manner. Before using this information, a mapping from confidence values to disparity outlier probability rate is performed based on gathered disparity statistics from labeled video data. We present an extension of the so called Stixel World, a generic 3D intermediate representation that can serve as input for many of the applications mentioned above. This scheme is modified to directly exploit stereo confidence cues in the underlying sensor model during a maximum a posteriori estimation process. The effectiveness of this step is verified in an in-depth evaluation on a large real-world traffic data base of which parts are made publicly available. We show that using stereo confidence cues allows both reducing the number of false object detections by a factor of six while keeping the detection rate at a near constant level.

Real-time Semi-Global Matching on the CPU

Among the top-performing stereo algorithms on the Middlebury Stereo Database, Semi-Global Matching (SGM) is commonly regarded as the most efficient algorithm. Consequently, real-time implementations of the algorithm for graphics hardware (GPU) and reconfigurable hardware (FPGA) exist. However, the computation time on general purpose PCs is still more than a second.

Performance Evaluation of Stereo Algorithms for Automotive Applications

The accuracy of stereo algorithms is commonly assessed by comparing the results against the Middlebury database. However, no equivalent data for automotive or robotics applications exist and these are difficult to obtain. We introduce a performance evaluation scheme and metrics for stereo algorithms at three different levels. This evaluation can be reproduced with comparably low effort and has very few prerequisites. First, the disparity images are evaluated on a pixel level. The second level evaluates the disparity data roughly column by column, and the third level performs an evaluation on an object level. We compare three real-time capable stereo algorithms with these methods and the results show that a global stereo method, semi-global matching, yields the best performance using our metrics that incorporate both accuracy and robustness.

Improving Stereo Sub-Pixel Accuracy for Long Range Stereo

Dense stereo algorithms are able to estimate disparities at all pixels including untextured regions. Typically these disparities are evaluated at integer disparity steps. A subsequent sub-pixel interpolation often fails to propagate smoothness constraints on a sub-pixel level. The determination of sub-pixel accurate disparities is an active field of research, however, most sub-pixel estimation algorithms focus on textured image areas in order to show their precision. We propose to increase the sub-pixel accuracy in low- textured regions in three possible ways: First, we present an analysis that shows the benefit of evaluating the disparity space at fractional disparities. Second, we introduce a new disparity smoothing algorithm that preserves depth discontinuities and enforces smoothness on a sub-pixel level. Third, we present a novel stereo constraint (gravitational constraint) that assumes sorted disparity values in vertical direction and guides global algorithms to reduce false matches, especially in low-textured regions. Our goal in this work is to obtain an accurate 3D reconstruction. Large- scale 3D reconstruction will benefit heavily from these sub- pixel refinements, especially with a multi-baseline extension. Results based on semi-global matching , obtained with the above mentioned algorithmic extensions are shown for the Middlebury stereo ground truth data sets. The presented improvements, called ImproveSubPix, turn out to be one of the top-performing algorithms when evaluating the set on a sub-pixel level while being computationally efficient. Additional results are presented for urban scenes. The three improvements are independent of the underlying type of stereo algorithm and can also be applied to sparse stereo algorithms.

Stereo matching in the presence of sub-pixel calibration errors

Stereo matching commonly requires rectified images that are computed from calibrated cameras. Since all underlying parametric camera models are only approximations, calibration and rectification will never be perfect. Additionally, it is very hard to keep the calibration perfectly stable in application scenarios with large temperature changes and vibrations. We show that even small calibration errors of a quarter of a pixel are severely amplified on certain structures. We discuss a robotics and a driver assistance example where sub-pixel calibration errors cause severe problems. We propose a filter solution based on signal theory that removes critical structures and makes stereo algorithms less sensitive to calibration errors. Our approach does not aim to correct decalibration, but rather to avoid amplifications and mismatches. Experiments on ten stereo pairs with ground truth and simulated decalibrations as well as images from robotics and driver assistance scenarios demonstrate the success and limitations of our solution that can be combined with any stereo method.