Christian Zinner

A fast stereo matching algorithm suitable for embedded real-time systems

In this paper, the challenge of fast stereo matching for embedded systems is tackled. Limited resources, e.g. memory and processing power, and most importantly real-time capability on embedded systems for robotic applications, do not permit the use of most sophisticated stereo matching approaches. The strengths and weaknesses of different matching approaches have been analyzed and a well-suited solution has been found in a Census-based stereo matching algorithm. The novelty of the algorithm used is the explicit adaption and optimization of the well-known Census transform in respect to embedded real-time systems in software. The most important change in comparison with the classic Census transform is the usage of a sparse Census mask which halves the processing time with nearly unchanged matching quality. This is due the fact that large sparse Census masks perform better than small dense masks with the same processing effort. The evidence of this assumption is given by the results of experiments with different mask sizes. Another contribution of this work is the presentation of a complete stereo matching system with its correlation-based core algorithm, the detailed analysis and evaluation of the results, and the optimized high speed realization on different embedded and PC platforms. The algorithm handles difficult areas for stereo matching, such as areas with low texture, very well in comparison to state-of-the-art real-time methods. It can successfully eliminate false positives to provide reliable 3D data. The system is robust, easy to parameterize and offers high flexibility. It also achieves high performance on several, including resource-limited, systems without losing the good quality of stereo matching. A detailed performance analysis of the algorithm is given for optimized reference implementations on various commercial of the shelf (COTS) platforms, e.g. a PC, a DSP and a GPU, reaching a frame rate of up to 75 fps for 640x480 images and 50 disparities. The matching quality and processing time is compared to other algorithms on the Middlebury stereo evaluation website reaching a middle quality and top performance rank. Additional evaluation is done by comparing the results with a very fast and well-known sum of absolute differences algorithm using several Middlebury datasets and real-world scenarios.

An Optimized Software-Based Implementation of a Census-Based Stereo Matching Algorithm

This paper presents S 3 E , a software implementation of a high-quality dense stereo matching algorithm. The algorithm is based on a Census transform with a large mask size. The strength of the system lies in the flexibility in terms of image dimensions, disparity levels, and frame rates. The program runs on standard PC hardware utilizing various SSE instructions. We describe the performance optimization techniques that had a considerably high impact on the run-time performance. Compared to a generic version of the source code, a speedup factor of 112 could be achieved. On input images of 320×240 and a disparity range of 30, S 3 E achieves 42fps on an Intel Core 2 Duo CPU running at 2GHz.

ROS-DMA: A DMA Double Buffering Method for Embedded Image Processing with Resource Optimized Slicing

Image processing on a Digital Signal Processor (DSP) often requires image data to be stored in external memory, because the amount of fast on-chip memory is usually very limited. Processing images in external memory causes significant performance drawbacks. This paper presents a double buffering method using Direct Memory Access (DMA), called Resource Optimized Slicing (ROS-DMA), which is intended to be used instead of a Level 2 (L2) data cache. The idea of ROS-DMA is to transfer image slices into small intermediate buffers of fast internal memory, where the processing can be completed utilizing the full processing power. Use of DMA enables the data transfers and the processing to be accomplished in parallel. The proposed method has the advantage of a modular implementation, making it easy to re-use components for various image processing operations. The sequence of transfers is organized in such a way that use of processor resources is optimized to achieve the shortest possible execution time. ROS-DMA can yield substantially better performance compared to using L2 cache. Furthermore, we expect that with ROS-DMA it will be easier to obtain reliable and tight Worst Case Execution Times (WCETs). Test runs achieved up to six times faster execution with ROS-DMA compared to using the L2 cache on a C6416 DSP from Texas Instruments.

Performance evaluation of a census-based stereo matching algorithm on embedded and multi-core hardware

In this paper a performance evaluation of a fast stereo matching algorithm on embedded and multi-core hardware is given. The algorithm is based on the census transform and adapted to the needs of real-time capability and embedded systems. The main focus lies on high frame rates and low memory consumption. On a standard PC with 4 cores a frame rate of 19 fps, on a GPU a frame rate of 39 fps and on a DSP a frame rate of 3.8 fps for 640 times 480 images with a disparity range of 50 was reached.

Accurate 3D-vision-based obstacle detection for an autonomous train

In this paper we present a 3D-vision based obstacle detection system for an autonomously operating train in open terrain environments. The system produces dense depth data in real-time from a stereo camera system with a baseline of 1.4m to fulfill accuracy requirements for reliable obstacle detection 80m ahead. On an existing high speed stereo engine, several modifications have been applied to significantly improve the overall performance of the system. Hierarchical stereo matching and slanted correlation masks increased the quality of the depth data in a way that the obstacle detection rate increased from 89.4% to 97.75% while the false positive detection rate could be kept as low as 0.25%. The evaluation results have been obtained from extensive real-world test data. An additional stereo matching speed-up of factor 2.15 was achieved and the overall latency of obstacle detection is considerably faster than 300ms.

A miniature embedded stereo vision system for automotive applications

Dependable 3D perception modules are essential for safe operation of autonomous systems. Therefore, we present a highly compact stereo vision system that gets along without a dedicated processing platform, having the DSPs integrated in the cameras. To enable the computation of dense and accurate depth maps, we implement a Sparse Census Transform, reducing the complexity of the stereo matching procedure by a factor of four while still ensuring highly accurate results. Besides the detection of false positives, wrong matches are highly reduced due to the computation and analysis of a dedicated confidence value. Furthermore, the algorithm allows for the computation of camera images with up to 16 bit camera resolution, leading just to minor increases in computational time.

Benchmarks of Low-Level Vision Algorithms for DSP, FPGA, and Mobile PC Processors

We present recent results of a performance benchmark of selected low-level vision algorithms implemented on different high-speed embedded platforms. The algorithms were implemented on a digital signal processor (DSP) (Texas Instruments TMS320C6414), a field-programmable gate array (FPGA) (Altera Stratix-I and II families) as well as on a mobile PC processor (Intel Mobile Core 2 Duo T7200). These implementations are evaluated, compared, and discussed in detail. The DSP and the mobile PC implementations, both making heavy use of processor-specific acceleration techniques (intrinsics and resource optimized slicing direct memory access on DSPs or Intel integrated performance primitives Library on mobile PC processors), outperform the FPGA implementations, but at the cost of spending all its resources to these tasks. FPGAs, however, are very well suited to algorithms that benefit from parallel execution.

Pfelib: a performance primitives library for embedded vision

This paper presents our work on PfeLib—a high performance software library for image processing and computer vision algorithms for an embedded system. The main target platform for PfeLib is the TMS320C6000 series of digital signal processors (DSPs) from Texas instruments. PfeLib contains several new approaches for problems that are typical when developing software for embedded systems. We propose a method for image data transfer from a development host (PC) to an embedded system for test and verification. This enables step-by-step performance optimizations directly on the target platform. An optimization procedure is described that illustrates our approach for obtaining the best possible DSP performance with a reasonable development effort. Speedup improvement factors of up to 16 were achieved. Also, the problem of the limited on-chip memory on DSPs is addressed by a novel double buffering method using direct memory access (DMA), called resource optimized slicing (ROS-DMA). ROS-DMA is intended to be used instead of L2 cache and it is a core component of PfeLib—it achieves up to six times faster image processing as compared to using L2 cache.

Reliable Left Luggage Detection Using Stereo Depth and Intensity Cues

Reliable and timely detection of abandoned items in public places still represents an unsolved problem for automated visual surveillance. Typical surveilled scenarios are associated with high visual ambiguity such as shadows, occlusions, illumination changes and substantial clutter consisting of a mixture of dynamic and stationary objects. Motivated by these challenges we propose a reliable left item detection approach based on the combination of intensity and depth data from a passive stereo setup. The employed in-house developed stereo system consists of low-cost sensors and it is capable to perform detection in environments of up to 10m x 10m in size. The proposed algorithm is tested on a set of indoor sequences and compared to manually annotated ground truth data. Obtained results show that many failure modes of intensity-based approaches are absent and even small-sized objects such as a handbag can be reliably detected when left behind in a scene. The presented results display a very promising approach, which can robustly detect left luggage in dynamic environments at a close to real-time computational speed.

An optimized Silicon Retina stereo matching algorithm using time-space correlation

This paper presents an optimized implementation of a Silicon Retina based stereo matching algorithm using time-space correlation. The algorithm combines an event-based time correlation approach with a census transform based matching method on grayscale images that are generated from the sensor output. The data processing part of the system is optimized for an Intel i7 mobile architecture and a C64x+ multi-core digital signal processor (DSP). Both platforms use an additional C64x+ single-core DSP system for acquisition and pre-processing of sensor data. We focus on the performance optimization techniques that had a major impact on the run-time performance of both processor architectures used.