Jiangbo Lu

Cross-Based Local Stereo Matching Using Orthogonal Integral Images

We propose an area-based local stereo matching algorithm for accurate disparity estimation across all image regions. A well-known challenge to local stereo methods is to decide an appropriate support window for the pixel under consideration, adapting the window shape or the pixelwise support weight to the underlying scene structures. Our stereo method tackles this problem with two key contributions. First, for each anchor pixel an upright cross local support skeleton is adaptively constructed, with four varying arm lengths decided on color similarity and connectivity constraints. Second, given the local cross-decision results, we dynamically construct a shape-adaptive full support region on the fly, merging horizontal segments of the crosses in the vertical neighborhood. Approximating image structures accurately, the proposed method is among the best performing local stereo methods according to the benchmark Middlebury stereo evaluation. Additionally, it reduces memory consumption significantly thanks to our compact local cross representation. To accelerate matching cost aggregation performed in an arbitrarily shaped 2-D region, we also propose an orthogonal integral image technique, yielding a speedup factor of 5-15 over the straightforward integration.

An Epipolar Geometry-Based Fast Disparity Estimation Algorithm for Multiview Image and Video Coding

Effectively coding multiview visual content is an indispensable research topic because multiview image and video that provide greatly enhanced viewing experiences often contain huge amounts of data. Generally, conventional hybrid predictive-coding methodologies are adopted to address the compression by exploiting the temporal and interviewpoint redundancy existing in a multiview image or video sequences. However, their key yet time-consuming component, motion estimation (ME), is usually not efficient in interviewpoint prediction or disparity estimation (DE), because interviewpoint disparity is completely different from temporal motion existing in the conventional video. Targeting a generic fast DE framework for interviewpoint prediction, we propose a novel DE technique in this paper to accelerate the disparity search by employing epipolar geometry. Theoretical analysis, optimal disparity vector distribution histograms, and experimental results show that the proposed epipolar geometry-based DE can greatly reduce search region and effectively track large and irregular disparity, which is typical in convergent multiview camera setups. Compared with the existing state-of-the-art fast ME approaches, our proposed DE can obtain a similar coding efficiency while achieving a significant speedup for interviewpoint prediction and coding. Moreover, a robustness study shows that the proposed DE algorithm is insensitive to the epipolar geometry estimation noise. Hence, its wide application for multiview image and video coding is promising

Real-Time and Accurate Stereo: A Scalable Approach With Bitwise Fast Voting on CUDA

This paper proposes a real-time design for accurate stereo matching on compute unified device architecture (CUDA). We present a leading local algorithm and then accelerate it by parallel computing. High matching accuracy is achieved by cost aggregation over shape-adaptive support regions and disparity refinement using reliable initial estimates. A novel sample-and-restore scheme is proposed to make the algorithm scalable, capable of attaining several times speedup at the expense of minor accuracy degradation. The refinement and the restoration are jointly realized by a local voting method. To accelerate the voting on CUDA, a graphics processing unit (GPU)-oriented bitwise fast voting method is proposed, faster than the traditional histogram-based approach with two orders of magnitude. The whole algorithm is parallelized on CUDA at a fine granularity, efficiently exploiting the computing resources of GPUs. Our design is among the fastest stereo matching methods on GPUs. Evaluated in the Middlebury stereo benchmark, the proposed design produces the most accurate results among the real-time methods. The advantages of speed, accuracy, and desirable scalability advocate our design for practical applications such as robotics systems and multiview teleconferencing.

Anisotropic local high-confidence voting for accurate stereo correspondence

We present a local area-based, discontinuity-preserving stereo matching algorithm that achieves high quality results near depth discontinuities as well as in homogeneous regions. To address the well-known challenge of defining appropriate support windows for local stereo methods, we use the anisotropic Local Polynomial Approximation (LPA) - Intersection of Confidence Intervals (ICI) technique. It can adaptively select a nearoptimal anisotropic local neighborhood for each pixel in the image. Leveraging this robust pixel-wise shape-adaptive support window, the proposed stereo method performs a novel matching cost aggregation step and an effective disparity refinement scheme entirely within a local high-confidence voting framework. Evaluation using the benchmark Middlebury stereo database shows that our method outperforms other local stereo methods, and it is even better than some algorithms using advanced but computationally complicated global optimization techniques.

Real-time stereo matching: A cross-based local approach

We propose an area-based local stereo matching algorithm that yields accurate disparity estimates, while achieving the real-time speed completely on the graphics processing unit (GPU). For a local stereo method, the key challenge is to decide an appropriate support window for the pixel under consideration. Our stereo method starts with computing an upright local cross adaptively for each anchor pixel, which defines a per-pixel support skeleton. Next, based on this compact local cross representation, we aggregate the matching costs in a shape adaptive full support region using two orthogonal integration steps. Approximating scene structures accurately, the proposed method is among the best-performing real-time stereo methods according to the benchmark Middlebury stereo evaluation. Additionally, our method is very easy to implement, memory efficient, and hence it is promising for many practical applications.

Real-time stereo-based view synthesis algorithms: A unified framework and evaluation on commodity GPUs

Novel view synthesis based on dense stereo correspondence is an active research problem. Despite that many algorithms have been proposed recently, this flourishing, cross-area research field still remains relatively less structured than its front-end constituent part, stereo correspondence. Moreover, so far little work has been done to assess different stereo-based view synthesis algorithms, particularly when real-time execution is enforced as a hard application constraint. In this paper, we first propose a unified framework that seamlessly connects stereo correspondence and view synthesis. The proposed framework dissects the typical algorithms into a common set of individual functional modules, allowing the comparison of various design decisions. Aligned with this algorithmic framework, we have developed a flexible GPU-accelerated software model, which contains optimized implementations of several recent real-time algorithms, specifically focusing on local cost aggregation and image warping modules. Based on this common software model running on graphics hardware, we evaluate the relative performance of various design combinations in terms of both view synthesis quality and real-time processing speed. This comparative evaluation leads to a number of observations, and hence offers useful guides to the future design of real-time stereo-based view synthesis algorithms.

Stream-Centric Stereo Matching and View Synthesis: A High-Speed Approach on GPUs

In this paper, we propose a real-time image-based rendering (IBR) system. It is specifically designed for photorealistic view synthesis at high-speed on the graphics processing unit (GPU). We steer the proposed IBR system design with two high-level ideas. First, for cost-effective IBR, as long as the synthesized views look visually plausible, the estimated disparity and occlusion need not be correct. Hence, we jointly optimize stereo matching and view synthesis for a favorable end-to-end performance. Second, for great real-time acceleration on GPUs, all functional modules need be shaped at an early design stage, fitting the massively parallel streaming architecture of GPUs. Based on these two guidelines, we first propose a stream-centric local stereo matching algorithm. The key idea is to construct a versatile set of variable support patterns in a highly efficient manner, and then an optimal local support pattern is selected to approximate varying image structures adaptively. Next, a low-complexity adaptive view synthesis technique is proposed. It efficiently tackles visual artifacts in synthesized images, using a novel photometric outlier detection and handling scheme. We evaluated both the disparity estimation accuracy and novel view synthesis quality of the proposed approach, based on the benchmark Middlebury stereo datasets. The experiments show that our local stereo method produces consistently reliable disparity estimates for both homogeneous regions and depth discontinuities, outperforming several previous GPU-based local methods. More importantly, visually plausible intermediate views are generated by our IBR approach at high-speed on the GPU. With stereo matching and view synthesis completely running on an NVIDIA GeForce 8800 GT graphics card, the proposed IBR system reaches about 100 f/s for 450times375 stereo images with 60 disparity levels.

Scalable stereo matching with Locally Adaptive Polygon Approximation

We present a scalable stereo matching algorithm based on a Locally Adaptive Polygon Approximation (LAPA) technique. For accurate local stereo matching, pixel-wise adaptive polygon-based support windows are constructed to approximate spatially varying image structures. Central to building these pixel-wise polygons is a fast algorithm that adaptively decides a set of directional scales, utilizing intensity and spatial information. Thanks to the locally adaptive support window, the proposed method achieves high stereo reconstruction quality both in depth-discontinuity regions and homogenous regions. Moreover, our LAPA-based method offers flexible scalability in terms of quality-complexity trade-off. As a specific instantiation favoring high-quality stereo estimation, our 8-direction stereo method outperforms most of the other local stereo methods and even some global optimization techniques. Another low-complexity alternative is also presented, achieving a significant speedup of up to a factor 20 with graceful accuracy degradation. Within a unified LAPA framework, our stereo method hence facilitates more flexibility in conciliating different algorithm design needs with processing performance issues.

Robust stereo matching with fast Normalized Cross-Correlation over shape-adaptive regions

Normalized Cross-Correlation (NCC) is a common matching technique to tolerate radiometric differences between stereo images. However, traditional rectangle-based NCC tends to blur the depth discontinuities. This paper proposes an efficient stereo algorithm with NCC over shape-adaptive matching regions, producing depth-discontinuity preserving disparity maps while remaining the advantage of robustness to radiometric differences. To alleviate the computational intensity, we propose an acceleration algorithm using an orthogonal integral image technique, achieving a speedup factor of 10∼27. In addition, a voting scheme on reliable estimates is applied to refine the initial estimates. Experiments show that, besides the robustness, the proposed method obtains accurate disparity maps at fast speed. Our method highly ranks among the local approaches in the Middlebury stereo benchmark.

High-Speed Stream-Centric Dense Stereo and View Synthesis on Graphics Hardware

This paper presents an efficient image-based rendering system capable of performing online stereo matching and view synthesis at high speed, completely on the graphics processing unit (GPU). Given two rectified stereo images, our algorithm first extracts the disparity map with a stream-centric dense depth estimation approach. For high-quality view synthesis, multi-label masks are then automatically generated to postprocess occlusions and ambiguously estimated regions adaptively. To allow even faster interactive view generation, an alternative forward warping method is also integrated. The experiments show that photorealistic intermediate views of high image quality are yielded by our algorithm. The optimized implementation also provides the state-of-the-art stereo analysis and view synthesis speed, achieving over 47 fps with 450x375 stereo images and 60 disparity levels on an Nvidia GeForce 7900 graphics card.