Zhang Zhao-yang

Stereo video coding based on frame estimation and interpolation

The paper proposes a stereo video coding system. To ensure compatibility with monoscopic transmission, one of the view sequences is coded and transmitted conforming to the MPEG standard, referred to as the reference stream, and the other view stream is referred to as target stream. Only a few frames of the latter are coded and transmitted, while the rest are skipped and reconstructed at the decoder using a novel stereoscopic frame compensation and interpolation technique, termed SFEI BLCF. In disparity estimation, smooth and accurate disparity fields are obtained by using hierarchical Markov random field (MRF) and Gibbs random field (GRF) models. A fast search method is used to improve the precision and computation speed. Coding and decoding results show that, with only 8/spl sim/30% additional bandwidth over a single view bit stream, one can transmit, store, and reconstruct stereoscopic video sequences with reasonably good performance.This paper proposes a stereo video coding system. To ensure compatibility with monoscopic transmission, one of the view sequences is coded and transmitted conforming to the MPEG standard, referred to as the reference stream, and the other view stream is referred to as target stream. Only a few frames of the latter are coded and transmitted, while the rest are skipped and reconstructed at the decoder using a novel stereoscopic frame compensation and interpolation technique, termed SFEI_BLCF. In disparity estimation, smooth and accurate disparity fields are obtained by using hierarchical Markov random field (MRF) and Gibbs random field (GRF) models. A fast search method is used to improve the precision and computation speed. Coding and decoding results show that, with only 8 30% additional bandwidth over a single view bit stream, one can transmit, store, and reconstruct stereoscopic video sequences with reasonably good performance.

Arbitrary view generation based on DIBR

An efficient depth image based rendering (DIBR) for generating arbitrary novel views is proposed. The proposed method fills the holes by preprocessing depth images and merging two desired images. In order to solve the visibility problem, the pixels in reference image are processed in an occlusion-compatible order. Furthermore, the reference images are corrected before 3D image warping so as to reduce the edge artifacts. Experimental results have show that the proposed method can provide a satisfactory image quality.

Objective performance evaluation of video segmentation algorithms with ground-truth

While the development of particular video segmentation algorithms has attracted considerable research interest, relatively little effort has been devoted to provide a methodology for evaluating their performance. In this paper, we propose a methodology to objectively evaluate video segmentation algorithm with ground-truth, which is based on computing the deviation of segmentation results from the reference segmentation. Four different metrics based on classification pixels, edges, relative foreground area and relative position respectively are combined to address the spatial accuracy. Temporal coherency is evaluated by utilizing the difference of spatial accuracy between successive frames. The experimental results show the feasibility of our approach. Moreover, it is computationally more efficient than previous methods. It can be applied to provide an offline ranking among different segmentation algorithms and to optimally set the parameters for a given algorithm.

Efficient stereo disparity estimation for intermediate view synthesis

An efficient algorithm addressing robust disparity estimation for intermediate view synthesis is proposed. In the proposed method, a new adaptive-size window approach based on region information is introduced to stereo matching in order to overcome problems with fixed-size window. Dynamic programming (DP) technique is used to find optimized disparity values. The reliability of disparity estimation is then measured with a criterion based on uniqueness and smoothness constrains. In occluded areas and image points with unreliable disparity assignments, region-based interpolation strategy is applied to compensate the disparity values. After projecting the left to right and right to left disparities onto the intermediate image, an arbitrary intermediate view is synthesized. Experimental results with natural stereo pairs show that the proposed algorithms provide good disparity map and obtain the intermediate views with high quality.

Video object segmentation for head-shoulder sequences in the cellular neural networks architecture

MPEG-4 introduces the concept of video object to support content-based functionalities. Video object segmentation is a key step in defining the contents of any video sequences. Head-shoulder sequence (HSS) is typical in video conferencing and surveillance systems, in which real-time performance is required. Since background information can be obtained in advance and pre-stored, video segmentation for HSS can use background information a priori. To avoid the critical selection of threshold for gradient-based method, and to overcome the insufficiency of monochrome intensity-based change detection, an efficient color edge-based change detection scheme (CECD) is utilized in this paper. In order to meet the real-time performance for HSS, it is implemented in the cellular neural networks (CNN) architecture. The algorithm is mainly based on 3 by 3, linear templates. Because of CNNs high parallelism and computational abilities, real-time performance is achieved. Experimental results on several test sequences show the robustness of this approach. It can achieve better spatial accuracy and temporal coherency than COST211 AM.

A Novel Real-Time Eye Detection in Human-Computer Interaction

Human eyes detection is one of the critical technologies in free stereoscopic display system. And the eye detection in free stereoscopic display system should be pretty accurate and real-time. To satisfy this demand, bright pupil effect under active infrared illumination is applied to provide the position of human eyes approximately; then the real AdaBoost algorithm is applied to locate human faces preliminarily and detect human eyes precisely; at the same time, the Kalman algorithm is applied to track the eye positions already detected to make sure the precision and speed of the human eyes detection further.On circumstance of Windows XP, PentiumIV, 512Memory, 2.4GHZ, for a video sequence of 640*480-pixel images, eye detection rate is 92.5%; the average processing time for each image is less than 10ms, meeting the need of real-time; this new method is also robust when there is variation of facial expression or a little degree leaning of human face.

A rectification algorithm for un-calibrated multi-view images based on SIFT features

In this paper, we present an efficient rectification algorithm for un-calibrated multi-view images based on SIFT (Scale-invariant feature transform) feature matching. Un-calibrated rectification is necessary for some specific occasions and we extend generic stereo pair rectification to multi-view camera array with projection shift method. We bring in SIFT algorithm to extract and match features (key points) automatically. Block-division features extraction method is proposed and RANSAC is used to improve precision of rectifying transformation. From the experiments, we find that our method is effective to rectify parallel cameras array. Rectified images have uniform horizontal disparities and the vertical mismatches between adjacent views are eliminated.

Hardware solution of real-time depth estimation based on stereo vision

3D video representation with depth maps is one efficient framework of 3DTV/free viewpoint television (FTV). 3D effect can be reconstructed based on human visional principle of binocular disparity. Texture image plus depth map provides an efficient solution of 3DTV application and real-time depth estimation is one barrier due to huge processing time consumption on complexity algorithm in the processing. An efficient hardware implementation of depth estimation is proposed for real time execution in this paper. The estimation system includes dual-channel video capture, image preprocessing, density pixel matching, pixel correlation and post processing. Each step of the system takes advantage of the hardware resource and adopts pipeline architecture to reduce the latency of combinational logic module at every stage and increase the throughput of the disparity extraction module. Experimental results demonstrate that the proposed method reduces the processing time and improves the ability of high resolution processing. The system can reach 131fps and Full HD (1920×1080) resolution, is suitable for adaptive real-time depth estimation and can be used as a subsystem embedded in the capture system of stereo vision.

Fast mode decision for inter macroblocks in H.264/AVC

In this paper, a fast mode decision algorithm for inter macroblocks in H.264/AVC is proposed. The algorithm is able to classify all modes by both gradient operator and comparison of space-time correlation. Because only part of modes is used to compare with each other the computational complexity can be reduced greatly. The simulation results show that it takes about 75% of encoding time for other algorithm with similar visual quality.

A Curve Fitting Based Virtual Camera Centers Generation Approach of Arc Cameras Array

A flexible virtual camera centers calculation method is proposed. The convergent video sequences which are captured by the arc video camera arrays are more approximate to HVS (Human Visual System), so as to diminish the viewing discomfort. But because of the manual setting of cameras, the orientation errors of cameras are inevitable. Thus it is necessary to conduct the cameras’ orientation rectification. And the generation of virtual camera centers is an important part of rectification. In our proposal, the virtual camera centers generation method can give out the virtual baselines in various baseline distances adaptively without initial setting of convergent angles. The virtual camera centers are generated in an LMS (least mean square) manner. Thus it is more flexible for different kinds of cameras’ orientation setting. And our proposal also refers to the generation of homography between old camera plane and virtual camera plane. Experiments show that the delta vector between two sets of camera centers can approach the minimum norm until now.