W. James MacLean

Video-rate stereo depth measurement on programmable hardware

This paper describes the implementation of a stereo depth measurement algorithm in hardware on field programmable gate arrays (FPGAs). This system generates 8 bit sub-pixel disparities on 256 by 360 pixel images at video rate (30 frames/sec). The algorithm implemented is a multi-resolution, multi-orientation phase-based technique called local weighted phase-correlation (Fleet, 1994). Hardware implementation speeds up the performance more than 300 times that of the same algorithm running in software. In this paper, we describe the programmable hardware platform, the base stereo vision algorithm and the design of the hardware. We include various trade-offs required to make the hardware small enough to fit on our system and fast enough to work at video rate. We also show sample outputs from the functioning hardware. Although this paper is specifically focused on phase-based stereo vision FPGA realizations, most of the design issues are common to other DSP and vision applications.

Reconfigurable hardware implementation of a phase-correlation stereoalgorithm

This paper describes the implementation of a stereo-vision system using Field Programmable Gate Arrays (FPGAs). Reconfigurable hardware, including FPGAs, is an attractive platform for implementing vision algorithms due to its ability to exploit parallelism often found in these algorithms, and due to the speed with which applications can be developed as compared to hardware. The system outputs 8-bit, subpixel disparity estimates for 256× 360 pixel images at 30,fps. A local-weighted phase correlation algorithm for stereo disparity [Fleet, D. J.: {Int. Conf. Syst. Man Cybernetics 1:48–54 (1994)] is implemented. Despite the complexity of performing correlations on multiscale, multiorientation phase data, the system runs as much as 300 times faster in hardware than its software implementation. This paper describes the hardware platform used, the algorithm, and the issues encountered during its hardware implementation. Of particular interest is the implementation of multiscale, steerable filters, which are widely used in computer vision algorithms. Several trade-offs (reducing the number of filter orientations from three to two, using fixed-point computation, changing the location of one localized low-pass filter, and using L1 instead of L2 norms) were required to both fit the design into the available hardware and to achieve video-rate processing. Finally, results from the system are given both for synthetic data sets as well as several standard stereo-pair test images.

Fast pattern recognition using gradient-descent search in an image pyramid

A new technique for fast pattern recognition using normalized grey-scale correlation (NGC) is described. While NGC has traditionally been slow due to computational intensity issues, the introduction of both a pyramid structure and a local estimate of the correlation surface gradient allows for recognition in 10-50 ms using modest microcomputer hardware. The algorithm is designed to analyze the target off-line prior to starting the search. Issues surrounding determining an appropriate depth for the pyramid representation and performing sub-pixel localization of the target instance are discussed. The speed and robustness of the method makes it attractive for industrial applications.

Fast pattern recognition using normalized grey-scale correlation in a pyramid image representation

The ability to quickly locate one or more instances of a model in a grey scale image is of importance to industry. The recognition/localization must be fast and accurate. In this paper we present an algorithm which incorporates normalized correlation into a pyramid image representation structure to perform fast recognition and localization. The algorithm employs an estimate of the gradient of the correlation surface to perform a steepest descent search. Test results are given detailing search time by target size, effect of rotation and scale changes on performance, and accuracy of the subpixel localization algorithm used in the algorithm. Finally, results are given for searches on real images with perspective distortion and the addition of Gaussian noise.

Leveraging cost matrix structure for hardware implementation of stereo disparity computation using dynamic programming

Dynamic programming is a powerful method for solving energy minimisation problems in computer vision, for example stereo disparity computations. While it may be desirable to implement this algorithm in hardware to achieve frame-rate processing, a nai@?ve implementation may fail to meet timing requirements. In this paper, the structure of the cost matrix is examined to provide improved methods of hardware implementation. It is noted that by computing cost matrix entries along anti-diagonals instead of rows, the cost matrix entries can be computed in a pipelined architecture. Further, if only a subset of the cost matrix needs to be considered, for example by placing limits on the disparity range (include neglecting negative disparities by assuming rectified images), the resources required to compute the cost matrix in parallel can be reduced. Boundary conditions required to allow computing a subset of the cost matrix are detailed. Finally, a hardware solution of Coxs maximum-likelihood, dynamic programming stereo disparity algorithm is implemented to demonstrate the performance achieved. The design provides high frame rate (>123fps) estimates for a large disparity range (e.g. 128 pixels), for image sizes of 640x480 pixels, and can be simply extended to work well over 200fps.

Integrating region and boundary information for spatiallycoherent object tracking

Abstract The problem of segmenting image sequences based on 2D motion has been under study for many years now. Most early approaches were either region-based, doing some sort of robust motion estimation, or boundary-based, preferring instead to track the bounding contours of the moving image region. In this paper, we explore an approach based on a synergy between these two previous approaches. For example, while motion constraints are often in violation of their underlying assumptions at region boundaries, image edges are a rich source of information. The approach we propose uses feed-forward to use region-based information to propagate boundary estimates, feedback to use boundaries to improve motion estimation, and finally uses motion-based warping to compare image appearance between frames in order to provide additional information for the boundary estimation process. We show results from an implementation in which a hierarchical, layered-motion estimation using parametric models is coupled with a distance-transform based active contour. The system is shown to provide stable and accurate segmentation in sequences with background motion, and multiple moving objects. Quantitative measures are proposed and reported for these sequences. Finally, a modification is detailed which allows the system to incorporate a Condensation algorithm tracker, but without requiring off-line learning in advance.

A real-time large disparity range stereo-system using FPGAs

In this paper, we discuss the design and implementation of a Field-Programmable Gate Array (FPGA) based stereo depth measurement system that is capable of handling a very large disparity range. The system performs rectification of the input video stream and a left-right consistency check to improve the accuracy of the results and generates subpixel disparities at 30 frames/second on 480 × 640 images. The system is based on the Local Weighted Phase-Correlation algorithm [9] which estimates disparity using a multi-scale and multi-orientation approach. Though FPGAs are ideal devices to exploit the inherent parallelism in many computer vision algorithms, they have a finite resource capacity which poses a challenge when adapting a system to deal with large image sizes or disparity ranges. In this work, we take advantage of the temporal information available in a video sequence to design a novel architecture for the correlation unit to achieve correlation over a large range while keeping the resource utilisation very low as compared to a naive approach of designing a correlation unit in hardware.

Evaluating multiview reconstruction

We survey the state of evaluation in current multiview reconstruction algorithms, with a particular focus on uncalibrated reconstruction from video sequences. We introduce a new evaluation framework, with high quality ground truth, as a vehicle for accelerating research in the area. Our source code is also freely available under the GNU General Public License, (GPL); a first for complete end-to-end reconstruction systems.

Introduction: Special issue on spatial coherence for visual motion analysis

The analysis of visual motion is fundamental in computer vision and image understanding. To date we do not have techniques for generating simple descriptions of commonplace image sequences, including the identification and representation of moving objects. Recent research has demonstrated successful methods for egomotion estimation under challenging conditions, as well as the recovery of scene structure when objects in the field of view are sufficiently large and textured and viewed against an otherwise rigid scene. Such techniques can now support a range of interesting tasks including navigation and localization. Nevertheless, we still have not developed the tools to reliably determine what parts of the image correspond to coherently moving objects in situations where there are multiple moving objects, non-rigidity, and significant appearance variations. As a result, the computation of stable descriptions of image sequences in terms of coherent, moving objects remains an elusive goal. This special issue grew out of the International Workshop on Spatial Coherence for Visual Motion Analysis, held in conjunction with ECCV 2004 in Prague. There were 33 participants in attendance to enjoy the eleven oral presentations given at the workshop. The best papers from the workshop, along with a large number of new submissions, were considered for this issue. We include five papers here, presenting new ideas on visual motion analysis: