Guilherme P. Fickel

Stereo Matching and View Interpolation Based on Image Domain Triangulation

This paper presents a new approach for stereo matching and view interpolation problems based on triangular tessellations suitable for a linear array of rectified cameras. The domain of the reference image is initially partitioned into triangular regions using edge and scale information, aiming to place vertices along image edges and increase the number of triangles in textured regions. A region-based matching algorithm is then used to find an initial disparity for each triangle, and a refinement stage is applied to change the disparity at the vertices of the triangles, generating a piecewise linear disparity map. A simple post-processing procedure is applied to connect triangles with similar disparities generating a full 3D mesh related to each camera (view), which are used to generate new synthesized views along the linear camera array. With the proposed framework, view interpolation reduces to the trivial task of rendering polygonal meshes, which can be done very fast, particularly when GPUs are employed. Furthermore, the generated views are hole-free, unlike most point-based view interpolation schemes that require some kind of post-processing procedures to fill holes.

Selective hole-filling for depth-image based rendering

One of the biggest challenges in view interpolation is to fill the regions without projective information in the synthesized view. In this paper, we present a new approach that identifies and corrects different types of missing information. In the first stage, we propose a fast solution to tackle the problems of cracks and ghost, common artifacts in the view interpolation process. Then, we complete larger holes by exploring the disparity map as an additional cue to select the best patch in a patch-based inpainting procedure. Our experimental results indicate that we were able to outperform current state of the art hole filling techniques for view interpolation.

Bilayer video segmentation for videoconferencing applications

This paper presents a new bilayer video segmentation algorithm focusing on videoconferencing application. A face tracking algorithm is used to guide a generic Ω-shaped template of the head and shoulders. A region of interest (ROI) is created around the generic template, and an energy function based on edge, color and motion cues is used to define the boundary between the person and the background. Our experimental results indicate that the silhouettes can be effectively extracted in common videoconferencing scenarios.

An evaluation of stereo matching methods for view interpolation

Stereo matching has a long history in image processing and computer vision. In fact, there are inumerous approaches reported in the literature, and quantitative evaluation is usually performed by comparing the obtained disparity maps with ground truth data (using the MSE, for instance). One important application of stereo matching is view interpolation, where it is desired to produce a new synthetic view from (at least) a pair of images and the corresponding disparity maps. In view interpolation, evaluation is mostly qualitative (visual quality of the synthesized image), and quantitative approaches compute objective similarity metrics between the synthesized image and the actual image at the same position (e.g. PSNR). The main goal of this paper is to evaluate the impact of several different stereo matching algorithms in a view interpolation context, relating the quality of the disparity maps with the quality of the corresponding synthesized views using standardized datasets. In this paper, experiments using the MPEG reference software for view interpolation and more than twenty datasets are presented and discussed. Our results indicate that the use of the common percentage of bad pixels as a metric for stereo matching methods does not translate well to the quality of view interpolation.

Stereo matching based on image triangulation for view synthesis

In this paper we propose a new disparity map estimation algorithm from multiple rectified images. The reference image is initially segmented into triangular regions, and each triangle is assigned to an initial disparity, leading to a piece-wise constant disparity map. Then, a refinement step is applied, where the disparity of the triangle vertices is adjusted aiming to impose spatial consistency to the disparity map, smoothing the map within the objects and keeping the discontinuities between them. The final result is a piece-wise linear depth map on a triangular domain that can be explored for view synthesis.

3D Motion Tracking Based on Probabilistic Volumetric Reconstruction and Optical Flow

This paper proposes a method for motion tracking of objects without a pre-defined shape, the main aspect of this method is the use of a probabilistic volumetric reconstruction that incorporates motion information. First, a volumetric reconstruction of the objects of interest is obtained by the 3D Probabilistic Occupancy Grid method, which was recently proposed for to be applied in environments sensed by multiple cameras. Then, we originally propose to add Optical Flow information to this reconstruction. Next, a method similar to the Expectation-Maximization (EM) algorithm is used to identify and track the body parts of objects of interest. It was noted that the proposed information of velocity vector fields are a good option to improve the perception of motion in 3D reconstruction, providing the best results in the tracking.

Disparity map estimation and view synthesis using temporally adaptive triangular meshes

Abstract This paper presents a new method for spatio-temporally coherent disparity map estimation and view interpolation for multiview linear camera arrays based on 2D domain triangulation. In the first frame of the sequence, a 3D mesh is computed for each camera, leading to a spatially coherent view interpolation. For the remaining frames of the sequence, a new scheme is proposed to update the 3D mesh dynamically, by moving, deleting and inserting vertices based on the optical flow and the previously computed disparity map. With this approach it is possible to relate triangles of the mesh across time, and a combination of Hidden Markov Models (HMMs) applied to time-persistent triangles with the Kalman Filter applied to the vertices produces temporally coherent disparity maps and interpolated views. Experimental results indicate that our approach was able to generate visually coherent in-between interpolated views for challenging, real-world videos with natural lighting and camera movement. Also, quantitative evaluations using objective video quality metrics show that our interpolated videos are typically better than competitive approaches.

Multiview image and video interpolation using weighted vector median filters

In Depth Image-Based Rendering (DIBR), interpolated views generated using one or two cameras usually present artifacts and holes due to occlusions and/or inconsistencies in the input disparity maps. In this paper we propose a multiple (3 or more) camera view interpolation technique that is able to combine redundant projections in a single interpolated view by using Weighted Vector Median Filters (WVMFs). By expressing the weights of the WVMF using both the distance from each reference view to the synthetic view and a measure of consonance of each projection to the others, we achieve a high quality view interpolation without holes and common visual artifacts, such as cracks and ghost effects. Additionally, we present an extension to multiview video sequences by imposing temporal coherence in the estimated disparity maps.

Visual Tracking Based on 3D Probabilistic Reconstruction

This paper presents an approach to the 3D visual tracking problem into multi-camera environments. This proposal executes the markerless visual tracking observing the environment through a model based in a volumetric reconstruction technique, called 3D Probabilistic Occupancy Grids, which is still seldom used for this purpose. The target is tracked by the use of Expectation-Maximization algorithm with an object representation model constructed with Gaussians blobs representing the object body parts.

Using Computer Vision for 3D Probabilistic Reconstruction and Motion Tracking

This paper presents an approach to the 3D visual tracking problem into multi-camera environments. This proposal executes the markerless visual tracking observing the environment through a model based in a volumetric reconstruction technique, called 3D Probabilistic Occupancy Grids, which is still seldom used for this purpose. The target is tracked by the use of Expectation-Maximization algorithm with an object representation model constructed with Gaussians blobs representing the object body parts.