Zhenyu Zhu

Real-Time Depth Image Acquisition and Restoration for Image Based Rendering and Processing Systems

Depth information is an important ingredient in image-based rendering (IBR) systems. Traditional depth acquisition is mainly based on computer vision or depth sensing devices. With the advent of electronics, low-cost and high-speed depth acquisition devices, such as the recently launched Microsoft Kinect, are getting increasingly popular. A comprehensive review of these important and emerging problems and their solutions are thus highly desirable. This paper aims to 1) review and summarize the various approaches to depth acquisition and highlight their advantages and disadvantages, 2) review problems arising from calibration and imperfections of these devices and state-of-the-art solutions, and 3) propose a surface-normal-based joint-bilateral filtering method for fast spatial-only restoration of missing depth data and a confidence-based IBR algorithm for reducing artifacts under depth uncertainties. For the latter, we propose a confidence measure based on color-depth, spatial and restoration information. A joint color-depth Bayesian matting approach is proposed for refining the depth discontinuities and the alpha matte for rendering. Improved rendering results are obtained compared with rendering using conventional restored depth maps. Possible future work and research directions are also briefly outlined.

A Multi-Camera Approach to Image-Based Rendering and 3-D/Multiview Display of Ancient Chinese Artifacts

This paper proposes an image-based approach for the capturing, rendering and display of ancient Chinese artifacts for cultural heritage preservation. A multiple-camera circular array is proposed to record images of the artifacts, which forms a simplified circular light field (SCLF). A systematic image-based approach and associate algorithms such as segmentation, depth estimation and shape morphing are developed for rendering new views of the Chinese artifacts. An object-based compression scheme is also proposed to reduce the data size for storage and transmission of the texture, depth maps and alpha maps associated with the object-based circular light field. Spatial redundancies among the various images are exploited to improve the coding performance, while avoiding excessive complexity in selective decoding of the light field to support fast rendering speed. To allow the Chinese artifacts to be viewed over the internet, scalable prioritized transmission and rendering schemes of the SCLF with low latency were also developed. The multiple views so synthesized enable the ancient artifacts to be displayed in 3-D/multi-view displays. Several collections from the University Museum and Art Gallery at The University of Hong Kong were captured and excellent rendering results are obtained.

Object-Based Rendering and 3-D Reconstruction Using a Moveable Image-Based System

This paper proposes a movable image-based rendering (M-IBR) system for improving the viewing freedom and environmental modeling capability of conventional static IBR systems. The system supports object-based rendering and 3-D reconstruction capability and consists of three main components. 1) An improved video stabilization method to reduce the shaky motion frequently encountered in movable IBR systems. It employs local polynomial regression (LPR) to automatically select an appropriate bandwidth for smoothing the estimated motion. 2) A novel view synthesis algorithm using a new segmentation and mutual-information (MI)-based algorithm for dense depth map estimation, which relies on segmentation, LPRbased depth map smoothing, and MI-based matching algorithm to iteratively estimate the depth map. The method is very flexible and both semiautomatic and automatic segmentation methods can be employed. They rank fourth and sixth, respectively, in the Middlebury comparison of existing depth estimation methods. This allows high-quality renderings of outdoor scenes with improved mobility/freedom to be obtained. 3) A new 3-D reconstruction algorithm that utilizes the sequential structure-from-motion technique and the dense depth maps estimated previously. It relies on a new iterative point cloud refinement algorithm based on Kalman filter for outlier removal and the segmentation-MI-based algorithm to further refine the correspondences and the projection matrices. The mobility of our system allows us to recover more conveniently 3-D model of static objects from the improved point cloud using a new robust radial basis function-based modeling algorithm to further suppress possible outliers and generate smooth 3-D meshes of objects. Experimental results show that the proposed 3-D reconstruction algorithm significantly reduces the adverse effect of the outliers and produces high-quality renderings using shadow light field and the model reconstructed.

A new two-stage method for restoration of images corrupted by Gaussian and impulse noises using local polynomial regression and edge preserving regularization

This paper proposes a new two-stage method for restoring image corrupted by additive impulsive and Gaussian noise based on local polynomial regression (LPR) and edge preserving regularization. In LPR, the observations are modeled locally by a polynomial using least-squares criterion with a kernel controlled by a certain bandwidth matrix. A refined intersection confidence intervals (RICI) adaptive scale selector for symmetric kernel is applied in LPR to achieve a better bias-variance tradeoff. The method is further extended to steering kernel with local orientation to adapt better to local characteristics of images. The resulting steering-kernel-based LPR with RICI method (SK-LPR-RICI) is applied to smooth images contaminated with Gaussian noise. Furthermore, to remove the impulsive noise in images, an edge-preserving regularization method is employed prior to SK-LPR-RICI and it gives rise to a two-stage method for suppressing both additive impulsive and Gaussian noises. Simulation results show that the proposed method performs satisfactorily and the SK-LPR-RICI method significantly improves the performance after edge-preservation regularization in suppressing the impulsive noise.

An approach to 2D-To-3D conversion for multiview displays

This paper proposes an approach for 2D-to-3D conversion for multiview displays. It employs an object-based approach where objects at large depth differences are first segmented by semi-automatic tools. Appropriate depth values are assigned to these objects and the missing image pixels at the background are filled in by inpainting techniques so that different views of the image can be synthesized. This approach helps to shorten the process of 2D-to-3D conversion and its performance is satisfactory for images and short video clips.

A movable image-based rendering system and its application to multiview audio-visual conferencing

Image-based rendering (IBR) is an emerging technology for rendering photo-realistic views of scenes from a collection of densely sampled images or videos. It provides a framework for developing revolutionary virtual reality and immersive viewing systems. This paper studies the design of a movable image-based rendering system based on a class of dynamic representations called plenoptic videos. It is constructed by mounting a linear array of 8 video cameras on an electrically controllable wheel chair with its motion being controllable manually or remotely through wireless LAN by means of additional hardware circuitry. We also developed a real-time object tracking algorithm and utilize the motion information computed to adjust continuously the azimuth or rotation angle of the movable IBR system in order to cope with a given moving object. Due to the motion of the wheel chair, videos may appear shaky and video stabilization technique is proposed to overcome this problem. The system can be used in a multiview audio-visual conferencing via a multiview TV display. Through this pilot study, we hope to develop a framework for designing movable IBR systems with improved viewing freedom and ability to cope with moving object in large environment.

Image-based rendering of ancient Chinese artifacts for multi-view displays — a multi-camera approach

Image-based rendering (IBR) is an emerging and promising technology for photo-realistic rendering of scenes and objects from a collection of densely sampled images and videos. This paper proposes an image-based approach to the rendering and multi-view display of ancient Chinese artifacts for cultural heritage preservation. A multiple-camera circular array was constructed to record images of the artifacts. Novel techniques for segmenting and rendering new views of the artifacts from the sampled images are developed. The multiple views so synthesized enable the ancient artifacts to be displayed in modern multi-view displays and conventional stereo systems. Several collections from the University Museum and Art Gallery at the University of Hong Kong are captured and excellent rendering results are obtained.

Image-based compression, prioritized transmission and progressive rendering of circular light fields (CLFS) for ancient Chinese artifacts

This paper proposes an efficient algorithm for the compression, prioritized transmission and progressive rendering of circular light field (CLF) for ancient Chinese artifacts. It employs wavelet coder to achieve spatial scalability and divide the compressed data into a lower resolution base layer and an additional enhancement layer. The enhancement layer is coded as in JPEG2000 into packets where the base-layer is coded using disparity compensation prediction (DCP). The frame structure is designed to provide efficient access to the compressed data in order to support selective transmission and decoding. The depth and alpha maps are coded analogously. A prioritized transmission scheme which support interactive progressive rendering is also proposed to further reduce the latency and response time of rendering.

Superpixel-based color–depth restoration and dynamic environment modeling for Kinect-assisted image-based rendering systems

Depth information is an important ingredient in many multiview applications including image-based rendering (IBR). With the advent of electronics, low-cost and high-speed depth cameras, such as the Microsoft Kinect, are getting increasingly popular. In this paper, we propose a superpixel-based joint color–depth restoration approach for Kinect depth camera and study its application to view synthesis in IBR systems. Thus, an edge-based matching method is proposed to reduce the color–depth registration errors. Then the Kinect depth map is restored based on probabilistic color–depth superpixels, probabilistic local polynomial regression and joint color–depth matting. The proposed restoration algorithm does not only inpaint the missing data, but also correct and refine the depth map to provide better color–depth consistency. Last but not the least, a dynamic background modeling scheme is proposed to address the disocclusion problem in the view synthesis for dynamic environment. The experimental results show the effectiveness of the proposed algorithm and system.

The Design and Construction of a Movable Image-Based Rendering System and Its Application to Multiview Conferencing

Image-based rendering (IBR) is an promising technology for rendering photo-realistic views of scenes from a collection of densely sampled images or videos. It provides a framework for developing revolutionary virtual reality and immersive viewing systems. While there has been considerable progress recently in the capturing, storage and transmission of image-based representations, most multiple camera systems are designed to be stationary and hence their ability to cope with moving objects and dynamic environment is somewhat limited. This paper studies the design and construction of a movable image-based rendering system based on a class of dynamic representations called plenoptic videos, its associated video processing algorithms and an application to multiview audio-visual conferencing. It is constructed by mounting a linear array of 8 video cameras on an electrically controllable wheel chair and its motion is controllable manually or remotely through wireless LAN by means of additional hardware circuitry. We also developed a real-time object tracking algorithm and utilize the motion information computed to adjust continuously the azimuth or rotation angle of the movable IBR system in order to cope with a given moving object in a large environment. Due to imperfection in tracking and mechanical vibration encountered in movable systems, the videos may appear very shaky and a new video stabilization technique is proposed to overcome this problem. The usefulness of the system is illustrated by means of a multiview conferencing application using a multiview TV display. Through this pilot study, we hope to disseminate useful experience for the design and construction of movable IBR systems with improved viewing freedom and ability to cope with moving object in a large environment.