Chao-Chung Cheng

A novel 2Dd-to-3D conversion system using edge information

Although three-dimensional (3D) displays enhance visual quality more than two-dimensional (2D) displays do, the depth information required for 3D displays is unavailable in the conventional 2D content. Therefore, converting 2D videos into 3D ones has become an important issue in emerging 3D applications. This work presents a novel algorithm that automatically converts 2D videos into 3D ones. The proposed algorithm utilizes the edge information to segment the image into object groups. A depth map is then assigned based on a hypothesized depth gradient model. Next, the depth map is block-based assigned by cooperating with a cross bilateral filter to generate visually comfortable depth maps efficiently and also diminish the block artifacts. A multiview video can be readily generated by using a depth image-based rendering method.

Hardware-efficient belief propagation

Belief propagation (BP) is an effective algorithm for solving energy minimization problems in computer vision. However, it requires enormous memory, bandwidth, and computation because messages are iteratively passed between nodes in the Markov random field (MRF). In this paper, we propose two methods to address this problem. The first method is a message passing scheme called tile-based belief propagation. The key idea of this method is that a message can be well approximated from other faraway ones. We split the MRF into many tiles and perform BP within each one. To preserve the global optimality, we store the outgoing boundary messages of a tile and use them when performing BP in the neighboring tiles. The tile-based BP only requires 1-5% memory and 0.2-1% bandwidth of the ordinary BP. The second method is an O(L) message construction algorithm for the robust functions commonly used for describing the smoothness terms in the energy function. We find that many variables in constructing a message are repetitive; thus these variables can be calculated once and reused many times. The proposed algorithms are suitable for parallel implementations. We design a low-power VLSI circuit for disparity estimation that can construct 440 M messages per second and generate high quality disparity maps in near real-time. We also implement the proposed algorithms on a GPU, which can calculate messages 4 times faster than the sequential O(L) method.

Hardware-Efficient Belief Propagation

Loopy belief propagation (BP) is an effective solution for assigning labels to the nodes of a graphical model such as the Markov random field (MRF), but it requires high memory, bandwidth, and computational costs. Furthermore, the iterative, pixel-wise, and sequential operations of BP make it difficult to parallelize the computation. In this paper, we propose two techniques to address these issues. The first technique is a new message passing scheme named tile-based BP that reduces the memory and bandwidth to a fraction of the ordinary BP algorithms without performance degradation by splitting the MRF into many tiles and only storing the messages across the neighboring tiles. The tile-wise processing also enables data reuse and pipeline, resulting in efficient hardware implementation. The second technique is an O(L) fast message construction algorithm that exploits the properties of robust functions for parallelization. We apply these two techniques to a very large-scale integration circuit for stereo matching that generates high-resolution disparity maps in near real-time. We also implement the proposed schemes on graphics processing unit (GPU) which is four-time faster than standard BP on GPU.

A block-based 2D-to-3D conversion system with bilateral filter

The three-dimensional (3D) displays provide a dramatic improvement of visual quality over the 2D displays. The conversion of existing 2D videos to 3D videos is necessary for multimedia application. This paper presents an automatic and robust system to convert 2D videos to 3D videos. The proposed 2D-to-3D conversion combines two major depth generation modules, the depth from motion and depth from geometrical perspective. A block-based algorithm is applied and cooperates with the bilateral filter to diminish block effect and generate comfortable depth map. After generating the depth map, the multi-view video is rendered to 3D display.

A real-time 1080p 2D-to-3D video conversion system

In this paper, we demonstrate a 2D-to-3D video conversion system capable of real-time 1920×1080p conversion. The proposed system generates 3D depth information by fusing cues from edge feature-based global scene depth gradient and texture-based local depth refinement. By combining the global depth gradient and local depth refinement, generated 3D images have comfortable and vivid quality, and algorithm has very low computational complexity. Software is based on a system with a multi-core CPU and a GPU. To optimize performance, we use several techniques including unified streaming dataflow, multi-thread schedule synchronization, and GPU acceleration for depth image-based rendering (DIBR). With proposed method, real-time 1920×1080p 2Dto- 3D video conversion running at 30fps is then achieved.

A 2D-to-3D conversion system using edge information

The three-dimensional (3D) displays provide a dramatic improvement of visual quality than the 2D displays do. However, for existed 2D contents, depth information is not recorded. Therefore, 2D-to-3D conversion is necessary. This paper presents an automatic system which convert 2D videos to 3D videos. The proposed system groups the blocks into regions using edge information. A prior hypothesis of depth gradient is used to assign depth of regions. Then, the bilateral filter is also applied to diminish block effect and to generate comfortable depth map for 3D visualization.

Relative Depth Layer Extraction for Monoscopic Video by Use of Multidimensional Filter

This paper presents a relative depth layer extraction system for monoscopic video, using multi-line filters and a layer selection algorithm. Main ideas are to extract multiple linear trajectory signals from videos and to determine their relative depths using the concept of motion parallax. The proposed superficial line model used for detecting slow moving objects provides sufficient taps within few frames to reduce frame buffer, while the closest-hit line model used for detecting fast motion objects provides few enough taps to prevent blurring. To increase the correctness of layer map, three-level layer map co-decision is used to compensate low texture region defect

Architecture design of stereo matching using belief propagation

We propose a new architecture for stereo matching using belief propagation. The architecture combines our fast, fully-parallel processing element (PE) and memory-efficient tile-based BP (TBP) algorithm. On the architectural level, we develop several novel techniques, including a three stage pipeline, a message forwarding scheme, and a boundary message reuse scheme, which greatly reduce the required bandwidth and power consumption without sacrificing performance. The simulation shows that the architecture can generate HDTV720p results at 30 fps when operating at 227MHz. The high-quality depth maps enable real-time depth image based rendering and many other important applications in the 3D TV industry.

Hybrid depth cueing for 2D-to-3D conversion system

The three-dimensional (3D) displays provide a dramatic improvement of visual quality than the 2D displays do. The conversion of existing 2D videos to 3D videos is necessary for multimedia application. This paper presents a robust system to convert 2D videos to 3D videos. The main concepts are to extract the depth information from motion parallax of moving picture and to depth information from geometrical perspective in non-moving scene. In the first part, depthinduced motion information is reconstructed by motion vector to disparity mapping. By warping the consecutive video frames to parallel view angle with the current frame, the frame with suitable baseline is selected to generate depth using motion parallax information. However, video may not have the depth-induced motion information in every case. For scene without motion parallax, depth from geometrical perspective is applied to generate scene depth map. Scene depth map is assigned depending on the scene mode and analyzed line structure in the video. Combining these two depth cues, the stereo effect is enhanced and provide spectacular depth map. The depth map is then used to render the multi-view video for 3D display.

Analysis of belief propagation for hardware realization

Belief propagation has become a popular technique for solving computer vision problems, such as stereo estimation and image denoising. However, it requires large memory and bandwidth, and hence naive hardware implementation is prohibitive. In this paper, we first analyze the memory and bandwidth requirements of the technique from the hardware perspective. Then, we propose a tile-based belief propagation algorithm that works with existing data reuse schemes and achieves bandwidth reduction by a factor of 10 to 400. We apply the proposed algorithm to stereo estimation and show that its performance is comparable to the original algorithm.