Chia-Ming Cheng

Improved novel view synthesis from depth image with large baseline

In this paper, a new algorithm is developed for recovering the large disocclusion regions in depth image based rendering (DIBR) systems on 3DTV. For the DIBR systems, undesirable artifacts occur in the disocclusion regions by using the conventional view synthesis techniques especially with large baseline. Three techniques are proposed to improve the view synthesis results. The first is the preprocessing of the depth image by using the bilateral filter, which helps to sharpen the discontinuous depth changes as well as to smooth the neighboring depth of similar color, thus restraining noises from appearing on the warped images. Secondly, on the warped image of a new viewpoint, we fill the disocclusion regions on the depth image with the background depth levels to preserve the depth structure. For the color image, we propose the depth-guided exemplar-based image inpainting that combines the structural strengths of the color gradient to preserve the image structure in the restored regions. Finally, a trilateral filter, which simultaneous combines the spatial location, the color intensity, and the depth information to determine the weighting, is applied to enhance the image synthesis results. Experimental results are shown to demonstrate the superior performance of the proposed novel view synthesis algorithm compared to the traditional methods.

Spatio-Temporally Consistent Novel View Synthesis Algorithm From Video-Plus-Depth Sequences for Autostereoscopic Displays

In this paper, we propose a novel algorithm to generate multiple virtual views from a video-plus-depth sequence for modern autostereoscopic displays. To synthesize realistic content in the disocclusion regions at the virtual views is the main challenging problem for this task. Spatial coherence and temporal consistency are the two key factors to produce perceptually satisfactory virtual images. The proposed algorithm employs the spatio-temporal consistency constraint to handle the uncertain pixels in the disocclusion regions. On the one hand, regarding the spatial coherence, we incorporate the intensity gradient strength with the depth information to determine the filling priority for inpainting the disocclusion regions, so that the continuity of image structures can be preserved. On the other hand, the temporal consistency is enforced by estimating the intensities in the disocclusion regions across the adjacent frames with an optimization process. We propose an iterative re-weighted framework to jointly consider intensity and depth consistency in the adjacent frames, which not only imposes temporal consistency but also reduces noise disturbance. Finally, for accelerating the multi-view synthesis process, we apply the proposed view synthesis algorithm to generate the intensity and depth maps at the leftmost and rightmost viewpoints, so that the intermediate views are efficiently interpolated through image warping according to the associated depth maps between the two synthesized images and their corresponding symmetric depths. In the experimental validation, we perform quantitative evaluation on synthetic data as well as subjective assessment on real video data with comparison to some representative methods to demonstrate the superior performance of the proposed algorithm.

A consensus sampling technique for fast and robust model fitting

In this paper, a new algorithm is proposed to improve the efficiency and robustness of random sampling consensus (RANSAC) without prior information about the error scale. Three techniques are developed in an iterative hypothesis-and-evaluation framework. Firstly, we propose a consensus sampling technique to increase the probability of sampling inliers by exploiting the feedback information obtained from the evaluation procedure. Secondly, the preemptive multiple K-th order approximation (PMKA) is developed for efficient model evaluation with unknown error scale. Furthermore, we propose a coarse-to-fine strategy for the robust standard deviation estimation to determine the unknown error scale. Experimental results of the fundamental matrix computation on both simulated and real data are shown to demonstrate the superiority of the proposed algorithm over the previous methods.

An integrated approach to 3D face model reconstruction from video

We present an integrated system for reconstruction an individualized 3D head model from a video sequence. Our reconstruction algorithm is based on the adaptation of a generic 3D head model. 3D geometric constraints on the head model are computed from the robust bundle adjustment algorithm and the structure from silhouette method. These 3D constraints are integrated to adapt the generic head model via radial basis function interpolation. Then the texture map of the reconstructed 3D head model is obtained by integrating all the images in the sequence through appropriate weighting. The proposed face model reconstruction method has the advantages of efficient computation as well as robustness against noises and outliers.

Robust Fundamental Matrix Estimation with Accurate Outlier Detection

The estimation of fundamental matrix from two-view images has been an important topic of research in 3D computer vision. In this paper, we present an improved robust algorithm for fundamental matrix estimation via modification of the RANSAC algorithm. The proposed algorithm is based on constructing a voting array for all the point correspondence pairs to record the consistency votes for each point correspondence from a number of the fundamental matrix estimations determined from randomly selected subsets of correspondence pairs to facilitate the identification of outliers. The boundary between the inliers and outliers in the sorted voting array are determined through a hypothesis testing procedure. With this strategy, we can accurately determine the outliers from all pairs of point correspondences, thus leading to accurate and robust fundamental matrix estimation under noisy feature correspondences. Through experimental comparison with previous methods on simulated and real image data, we show the proposed algorithm in general outperforms other best-performed methods to date.

Spatio-temporally Consistent Multi-view Video Synthesis for Autostereoscopic Displays

In this paper, we propose a novel algorithm to generate multiple virtual views from a video-plus-depth sequence for modern autostereoscopic displays. To synthesize realistic content in the disocclusion regions from the virtual views is the main challenging problem in this task. In order to produce perceptually satisfactory images, our proposed algorithm takes advantage of spatial coherence and temporal consistency to handle the uncertain pixels in the disocclusion regions. On the one hand, regarding the spatial coherence, we incorporate the intensity gradient strength with the depth information to determine the filling priority for inpainting the disocclusion regions, so that the continuity of image structures can be preserved. On the other hand, the temporal consistency is enforced by considering the intensities in the disocclusion regions across the adjacent frames through an optimization process. We propose an iterative re-weighted framework to jointly consider intensity and depth consistency in the adjacent frames, which not only imposes temporal consistency but also reduces noise disturbance. Finally, for accelerating the multi-view synthesis process, we apply the proposed view synthesis algorithm to generate the images plus depth at the leftmost and rightmost viewpoints, so that the intermediate views are efficiently interpolated through image warping according to the associated depth maps between the two views. In the experimental validation, we perform quantitative evaluation on synthetic data as well as subjective assessment on real video data with comparisons to some previous representative methods to demonstrate the superior performance of the proposed method.

Geodesic tree-based dynamic programming for fast stereo reconstruction

In this paper, we present a novel tree-based dynamic programming (TDP) algorithm for efficient stereo reconstruction. We employ the geodesic distance transformation for tree construction, which results in sound image over-segmentation and can be easily parallelized on graphic processing unit (GPU). Instead of building a single tree to convey message in dynamic programming (DP), we construct multiple trees according to the image geodesic distance to allow for parallel message passing in DP. In addition to efficiency improvement, the proposed algorithm provides visually sound stereo reconstruction results. Compared with previous related approaches, our experimental results demonstrate superior performance of the proposed algorithm in terms of efficiency and accuracy.

A novel structure-from-motion strategy for refining depth map estimation and multi-view synthesis in 3DTV

The video-plus-depth format has been widely used for representing the 3D scene due to its main advantage of compatibility to image format. In practice, the depth inconsistency may lead to unsatisfactory view synthesis results. In this paper, we propose a new structure-from-motion (SfM) technique, called locally temporal bundle adjustment (LTBA), to handle the dynamic scenes as well as the static camera motion, which violates the conventional structure from motion assumption. By integrating the camera information, depth map, and video temporally, we develop a geometric quadrilateral filter to reduce noise in the depth map and enhance the spatio-temporal consistency to improve the quality of depth maps. We show the improved quality of dynamic depth maps by using the proposed algorithm through experiments on real video-plus-depth sequences.

Technical Section: Image-based three-dimensional model reconstruction for Chinese treasure-Jadeite Cabbage with Insects

This paper presents a novel 3D reconstruction system for the famous Chinese treasure, Jadeite Cabbage with Insects, from uncalibrated image sequences. There are two major challenges for this 3D model reconstruction problem. The first is the difficult image registration problem due to the semi-diaphaneity and the highly specular property of jadeite materials. Secondly, the unknown camera information, including the intrinsic (calibration) and extrinsic (position and orientation) parameters, to be recovered from the uncalibrated image sequences makes the 3D reconstruction problem very challenging. The proposed 3D modeling process first recovers the camera information as well as sparse 3D structure by using a robust structure from motion algorithm. Then an approximate 3D object model is recovered from the silhouettes at the corresponding multiple views by using the visual hull technique. The final process refines the 3D model by further integrating the 3D information extracted from dense correspondences between image patches of different views. In the proposed 3D reconstruction system, we successfully combine the structure from motion and visual hull techniques to accomplish this challenging task of reconstructing an accurate 3D model for jadeite object from uncalibrated multi-view images. Finally, we assess the 3D reconstruction results for the Chinese jadeite treasure and simulated data by using the proposed 3D reconstruction system.

Robust 3D object pose estimation from a single 2D image

In this paper, we propose a robust algorithm for 3D object pose estimation from a single 2D image. The proposed pose estimation algorithm is based on modifying the traditional image projection error function to a sum of squared image projection errors weighted by their associated distances. By using an Euler angle representation, we formulate the energy minimization for the pose estimation problem as searching a global minimum solution. Based on this framework, the proposed algorithm employs robust techniques to detect outliers in a coarse-to-fine fashion, thus providing very robust pose estimation. Our experiments show that the algorithm outperforms previous methods under noisy conditions.