Zhebin Zhang

Visual pertinent 2D-to-3D video conversion by multi-cue fusion

We describe an approach to2D-to-3D video conversion for the stereoscopic display. Targeting the problem of synthesizing the frames of a virtual ‘right view’ from the original monocular 2D video, we generate the stereoscopic video in steps as following. (1) A 2.5D depth map is first estimated in a multi-cue fusion manner by leveraging motion cues and photometric cues in video frames with a depth prior of spatial and temporal smoothness. (2)The depth map is converted to a disparity map with considering both the displaying device size and humans stereoscopic visual perception constraints. (3)We fix the original 2D frames as the ‘left view’ ones, and warp them to “virtually viewed” right ones according to the predicted disparity value. The main contribution of this method is to combine motion and photometric cues together to estimate depth map. In the experiments, we apply our method to converting several movie clips of well-known films into stereoscopic 3D video and get good results1.

Interactive Stereoscopic Video Conversion

This paper presents a system of converting conventional monocular videos to stereoscopic ones. In the system, an input monocular video is firstly segmented into shots so as to reduce operations on similar frames. An automatic depth estimation method is proposed to compute the depth maps of the video frames utilizing three monocular depth cues - depth-from-defocus, aerial perspective, and motion. Foreground/background objects can be interactively segmented on selected key frames and their depth values can be adjusted by users. Such results are propagated from key frames to nonkey frames within each video shot. Equipped with a depth-to-disparity conversion module, the system synthesizes the counterpart (either left or right) view for stereoscopic display by warping the original frames according to their disparity maps. The quality of converted videos is evaluated by human mean opinion scores, and experiment results demonstrate that the proposed conversion method achieves encouraging performance.

An interactive system of stereoscopic video conversion

With the recent booming of 3DTV industry, more and more stereoscopic videos are demanded by the market. This paper presents a system of converting conventional monocular videos to stereoscopic ones. In this system, an input video is firstly segmented into shots to reduce operations on similar frames. Then, automatic depth estimation and interactive image segmentation are integrated to obtain depth maps and foreground/background segments on selected key frames. Within each video shot, such results are propagated from key frames to non-key frames. Combined with a depth-to-disparity conversion method, the system synthesizes the counterpart (either left or right) view for stereoscopic display by warping the original frame according to disparity maps. For evaluation, we use human labeled depth map as the reference and compute both the mean opinion score (MOS) and Peak signal-to-noise ratio (PSNR) to valuate the converted video quality. Experiment results demonstrate that the proposed conversion system and methods achieves encouraging performance.

Stereoscopic learning for disparity estimation

In this paper, we propose a learning based approach to estimating pixel disparities from the motion information extracted out of input monoscopic video sequences. We represent each video frame with superpixels, and extract the motion features from the superpixels and the frame boundary. These motion features account for the motion pattern of the superpixel as well as camera motion. In the learning phase, given a pair of stereoscopic video sequences, we employ a state-of-the-art stereo matching method to compute the disparity map of each frame as ground truth. Then a multi-label SVM is trained from the estimated disparities and the corresponding motion features. In the testing phase, we use the learned SVM to predict the disparity for each superpixel in a monoscopic video sequence. Experiment results show that the proposed method achieves low error rate in disparity estimation.

A Compact Stereoscopic Video Representation for 3D Video Generation and Coding

We propose a novel compact representation for stereoscopic videos - a 2D video and its depth cues. Depth cues are derived from an interactive labeling process during 2D-to-3D video conversion, they are contour points of foreground objects and a background geometric model. By using such cues and image features of 2D video frames, depth maps of the frames can be recovered. Compared with traditional 3D video representation, the proposed one is more compact. We also design algorithms to encode and decode the depth cues. The representation benefits both 3D video generation and coding. Experimental results demonstrate that the bit rate can be saved about 10%-50% in coding 3D videos compared with multi-view video coding and 2D+depth methods. A system coupling 2D-to-3D video conversion and coding (CVCC) is proposed to verify advantages of the representation.

A virtual view genetation method for free-viewpoint video system

Generating virtual views is one of the crucial problems for a free-viewpoint video system. In this paper, we solve the problem by using a method based on the technique of view morphing, and construct a multi-camera based experimental free-viewpoint video platform to validate its feasibility. This method requires no setting up of dense cameras, therefore the system is easy to be established and controlled in practice, and the mount of data for processing and transmitting can also be reduced. Experimental results show that the effect of this method can satisfy the requirement of view generation for the system.

A Compact Representation for Compressing Converted Stereo Videos

We propose a novel representation for stereo videos namely 2D-plus-depth-cue. This representation is able to encode stereo videos compactly by leveraging the by-product of a stereo video conversion process. Specifically, the depth cues are derived from an interactive labeling process during 2D-to-stereo video conversion-they are contour points of image regions and their corresponding depth models, and so forth. Using such cues and the image features of 2D video frames, the scene depth can be reliably recovered. Experimental results demonstrate that the bit rate can be saved about 10%-50% in coding a stereo video compared with multiview video coding and the 2D-plus-depth methods. In addition, since the objects are segmented in the conversion process, it is convenient to adopt the region-of-interest (ROI) coding in the proposed stereo video coding system. Experimental results show that using ROI coding, the bit rate is reduced by 30%-40% or the video quality is increased by 1.5-4 dB with the fixed bit rate.

An interactive method for curve extraction

We introduce a curve process framework to solve the challenging problem of curve extraction from “non-traceable” curve groups. We propose a comprehensive curve model, which consists of the geometric, photometric and topological sub-models. Two typical categories of the non-traceable curve groups are considered. First, for the interlaced curves with complex structures, we show how to use the proposed curve model especially the topological sub-model to extract curves from the group. Second, for the non-interlaced but over-dense or faint curves we leverage the curve group pattern priors in addition, and extract the whole pattern in a global optimization. Applications and experiments demonstrate the competence of our models and methods.

Segmentation-based dense stereo matching algorithm for virtual view synthesis

This paper presents a new stereo matching algorithm to compute dense disparity map for virtual view synthesis. The reference view is segmented using mean-shift segmentation method. An adaptive support-weights window-based approach is adopted to obtain the initial disparity map per pixel. We utilize cross-checking technique to filter out reliable correspondences and detect occlusion regions. The disparity in each segment is represented as a 3D planar plane by a robust plane fitting process. With the plane model of each segment, the disparity of unreliable points and occlusion regions can be refined. Finally, pixel domain matches are defined by computed disparity map, and virtual view can be synthesized by interpolating. Experimental results with the Middlebury stereo testing images show that our stereo matching algorithm gives a good performance.