Wei-Jia Huang

A 2D to 3D video and image conversion technique based on a bilateral filter

A method to convert a 2D video or 2D still image into a depth map is presented. The depth map can consequently be converted into a pair of right and left images for viewing on a stereoscopic display. The depth map can be used for other applications such as multi-view generation. Our method is based on the use of an initial depth map and a bilateral filter approximated by a bilateral grid. For still images, the initial depth map is arbitrary set by default or can be chosen by the user, while for videos it is evaluated from motion estimation. Our method produces finely segmented depth maps; when converted to the appropriate format and seen on a stereoscopic display, the resulting image or video is both realistic and comfortable to watch.

31.2: Invited Paper: Enhanced Minimally Invasive Surgery by 2D to 3D Conversion

Minimally Invasive Surgery (MIS) is a growing surgical technique comparing to traditional open surgery since it dramatically reduces patient trauma and costs but MIS technique is difficult because of looking at a 2D screen to perform the surgery. Depth perception is thus lost due to 2D endoscopic capturing and display. Therefore, the paper proposes real-time 2D to 3D conversion for enhancing MIS to provide depth perception on 3D display for increasing surgical quality, improving in safety and speed of procedures, and also to lower learning curve. The proposed method to convert an endoscopic 2D video frame into a depth map and consequently into a pair of right and left image for viewing on a stereoscopic display is based on the use of a bilateral filter approximated by a bilateral grid. The architecture is to make edges sharp to enhance moving instruments and to preserve chamber geometry segmentation. Results of verification by simulated surgery and questionnaire for surgeons proof the feasibility of 2D to 3D conversion on MIS.

A real-time video 2D-to-3D with the bilateral grid

The new data structure, the bilateral grid, was presented by Jiawen et al. to make bilateral filter algorithm become simple implementation. Based on the data structure, the GPU CUDA-based optimization is proposed to have more efficiency in using GPU shared memory and massive multithreading. Meanwhile, a commercial application, the video 2d to 3d conversion which was presented by Ludovic et al. is also re-designed by applying the proposed CUDA-based bilateral grid three times to obtain better 3D quality in real-time. Depth map are created and modified by adjusting bilateral grid parameters.

Seam based dynamic programming for stereo matching

In recent years, stereo matching gets more and more attention in the fields of gaming and 3D display, as the need of intuitive human-computer interaction and image based multi-view rendering getting higher. To fit these kinds of applications, cost and memory efficiency of the matching algorithm are as important as the quality of the matching result. Therefore, dynamic programming (DP) which uses semi-global optimization and provides compromise between complexity and quality can perform stereo matching effectively and efficiently.

A video depth refinement with circuit model constraint

The raise of 3D image market is rapid. In previous work [Wang 2008], distortion and grid line bending constraints is adapted to image resizing while important areas remain the same scale. In another previous work [Lang 2010], disparity of input stereo image is remapped, and warped stereo image pairs are manipulated by similar constraints. In our system, we adapt image constraints similar to Langs work, but for a depth refinement purpose. Original random and sharp depth error causes deformation and large defect of hole-filling. These errors are refined, but with a consequence of blurred depth edge. So we adapt another circuit model constraint to sharpen the depth edge again, but maintain the smoothed results at the same time. This technique is modified from previous work [Grady 2006], pixels are electrical circuit nodes with resistance derived from RGB difference. Generally, temporal smooth in video processing needs information from several frames. But in our system, no multiple frames involve but only current frame. In the same scene, different frames share similar color and shape. Since depth map is refined by 2D image, and temporal 2D images are similar, refined depths tend to be smoothed and sharpened in similar way. Finally, depth map of previous frame is used as an initial depth map of current optimization. Experiment results in Figure 1 & 2 show significant improvement.