Woo-Seok Jang

Efficient disparity map estimation using occlusion handling for various 3D multimedia applications

Disparity acquisition is beneficial to various 3D multimedia applications, i.e., free viewpoint TV (FTV), 3DTV, and 3D scene model reconstruction. In this paper, we propose an efficient disparity map estimation method with occlusion handling taken into account. In order to detect occlusion, we obtain left and right initial disparity maps via optimization based on modified constant-space belief propagation, a low complexity algorithm. The initial disparity maps provide clues for occlusion detection. From the clues, an energy function is defined and optimized by energy minimization framework. We define two occlusion types and apply suitable occlusion handling processes, accordingly. The proposed occlusion handling method extends disparity values of visible pixels to occluded pixels. Experimental results show that our method provides near-ground truth occlusion and a superior disparity map compared to other stateof- the-art methods with occlusion handling. Our contribution provides solid advantages regarding disparity map estimation, which is useful for various 3D multimedia applications.

Discontinuity preserving disparity estimation with occlusion handling

We solve inherent problems from conventional stereo matching methods.Distance transform-based stereo matching increases the disparity quality at edges.Occlusion detection with three constraints helps reducing disparity errors.The proposed hole filling of occluded regions results in a better performance. In this paper, we propose a stereo matching algorithm based on distance transform to generate high-quality disparity maps with occlusion handling. In general, pixel intensities around object edges are smeared due to mixed values located between the object and its background. This leads to problems when identifying discontinuous disparities. In order to handle these problems, we present an edge control function according to distance transform values. Meanwhile, occluded regions occur, i.e., some portions are visible only in one image. An energy function is designed to detect such regions considering warping, cross check, and luminance difference constraints. Consequently, we replace the disparity in the occluded region with the one chosen from its neighboring disparities in the non-occluded region based on color and spatial correlations. In particular, the occlusion hole is filled according to region types. Experimental results show that the proposed method outperforms conventional stereo matching algorithms with occlusion handling.

Efficient depth map generation with occlusion handling for various camera arrays

In this paper, we propose a direct depth map acquisition method for the arc camera array as well as the parallel camera array. In conventional stereo matching algorithms, image rectification is necessary where disparity values are obtained by identifying correspondences in the identical horizontal line of stereo images. The acquired disparity values are then transformed to depth values. However, image rectification may fabricate unwanted outcomes related to the arc camera array. Thus, the proposed method excludes image rectification and directly extracts depth values using an epipolar constraint. In particular, occlusion detection and handling processes are inserted to increase depth map accuracy. Further, belief propagation-based energy optimization is employed to confirm occlusion regions. Then, reasonable depth values are assigned to obtained occlusion regions using distances and color differences of neighbor pixels. Experimental results show that compared to the conventional methods, the proposed method generate more stable depth maps with fewer limitations.

Gaze correction using 3D video processing for videoconferencing

In this paper, we propose a gaze correction method using 3D video processing techniques including depth estimation and virtual view synthesis. We design a set of color and depth cameras to reduce occlusion regions and improve the depth precision in the less-detailed region. The proposed algorithm deals with fully unsolved problems by fusing the depth data from both depth sensors. Furthermore, view synthesis is performed to create the gaze corrected image from the obtained depth information. Experimental results show our contribution is useful for videoconferencing.

Eye Gaze Correction for Video Conferencing Using Kinect v2

In video conferencing, eye gaze correction is beneficial for effective communication. In this era, video conferencing at homes using laptops is straightforward. In this paper, we propose an eye gaze correction method with a low-cost simple setup using Kinect v2. Our method detects an ellipse that connects edge points of the face after identifying several feature points within the face using Kinect v2 SDK. Then, we apply a 3D affine transform that allows eye gaze correction using camera space points that are acquired from depth information. Thus, in the preprocessing step, an ellipse model should be extracted when the user gazes the camera and display, respectively. Also, we fill holes that are caused by the affine transform using color inpainting. As a result, we produced a natural eye gaze-corrected image in real-time.

Efficient Depth Map Generation for Various Stereo Camera Arrangements

In this paper, we propose a direct depth map acquisition method for the convergence camera array as well as the parallel camera array. The conventional methods perform image rectification to reduce complexity and improve accuarcy. However, image rectification may lead to unwanted consequences for the convergence camera array. Thus, the proposed method excludes image rectification and directly extracts depth values using the epipolar constraint. In order to acquire a more accurate depth map, occlusion detection and handling processes are added. Reasonable depth values are assigned to the obtained occlusion region by the distance and color differences from neighboring pixels. Experimental results show that the proposed method has fewer limitations than the conventional methods and generates more accurate depth maps stably.

Occlusion Detection Using Warping and Cross-Checking Constraints for Stereo Matching

In this paper, we propose an occlusion detection algorithm which estimates occluded pixels automatically. In order to detect the occlusion, we obtain an initial disparity map with an optimization algorithm based on the modified constant-space belief propagation (CSBP) which has low complexity. The initial disparity map gives us clues for occlusion detection. These clues are the warping constraint and the cross check constraint. From both constraints, we define a potential energy function for occlusion detection and optimize it using an energy minimization framework. Experimental results show that the result of the occlusion detection from the proposed algorithm is very close to the ground truth.

Disparity map acquisition with occlusion handling using warping constraint

In this paper, we propose a stereo matching algorithm with occlusion handling. In order to detect occlusion, we obtain an initial disparity map via optimization based on modified constant-space belief propagation (CSBP). Such a method is advantageous due to its low complexity. The initial disparity maps provide clue for occlusion detection. From such clue, an energy function for occlusion detection is defined and optimized by energy minimization framework. We classify occlusion into two types from the obtained occlusion map and apply suitable occlusion handling process, respectively. The proposed occlusion handling method based on the potential energy function extends disparity values of visible pixels to occluded pixels. Experimental results show that generated disparity map of the proposed method has satisfactory quality.

Efficient Selection of Candidates for Fast Inter Mode Decision in H.264

In the H.264/AVC video codec, a rate-distortion optimization method for mode decision provides high coding efficiency. However, it increases encoding time significantly due to its high complexity. In this paper, we propose a fast algorithm for inter mode decision using motion cost during motion estimation to reduce the number of candidate modes. The proposed approach makes early 16 times 16 mode decision and early inactive P8 times 8 mode decision. Its encoding performance is similar to that of the fast inter mode decision in the H.264/AVC JM reference software; however, its encoding time is reduced by 30.95% on average. Especially, when quantization parameter (QP) is larger, we obtained better results.

Disparity fusion using depth and stereo cameras for accurate stereo correspondence

Three-dimensional content (3D) creation has received a lot of attention due to numerous successes of 3D entertainment. Accurate stereo correspondence is necessary for efficient 3D content creation. In this paper, we propose a disparity map estimation method based on stereo correspondence. The proposed system utilizes depth and stereo camera sets. While the stereo set carries out disparity estimation, depth camera information is projected to left and right camera positions using 3D transformation and upsampling is processed in accordance with the image size. The upsampled depth is used for obtaining disparity data of left and right positions. Finally, disparity data from each depth sensor are combined. In order to evaluate the proposed method, we applied view synthesis from the acquired disparity map. The experimental results demonstrate that our method produces more accurate disparity maps compared to the conventional approaches which use the single depth sensors.