Ka-Ho Ng

Novel Directional Gradient Descent Searches for Fast Block Motion Estimation

Search point pattern-based fast block motion estimation algorithms provide significant speedup for motion estimation but usually suffer from being easily trapped in local minima. This may lead to low robustness in prediction accuracy particularly for video sequences with complex motions. This problem is especially serious in one-at-a-time search (OTS) and block-based gradient descent search (BBGDS), which provide very high speedup ratio. A multipath search using more than one search path has been proposed to improve the robustness of BBGDS but the computational requirement is much increased. To tackle this drawback, a novel directional gradient descent search (DGDS) algorithm using multiple OTSs and gradient descent searches on the error surface in eight directions is proposed in this letter. The search point patterns in each stage depend on the minima found in these eight directions, and thus the global minimum can be traced more efficiently. In addition, a fast version of the DGDS (FDGDS) algorithm is also described to further improve the speed of DGDS. Experimental results show that DGDS reduces computation load significantly compared with the well-known fast block motion estimation algorithms. Moreover, FDGDS can achieve faster speedup compared with the UMHexagonS algorithm in H.264/AVC implementation while maintaining very similar rate-distortion performance.

Novel Point-Oriented Inner Searches for Fast Block Motion Estimation

Recently, an enhanced hexagon-based (EHS) search algorithm was proposed to speedup the original hexagon-based search (HS) using a 6-side-based fast inner search. However, this 6-side-based method is quite irregular by inspecting the distance between the inner search points and the coarse search points that would lower prediction accuracy. In this paper, a new point-oriented grouping strategy is proposed to develop fast inner search techniques for speeding up the HS and diamond search (DS) algorithms. Experimental results show that the new HS and DS using point-oriented inner searches are faster than their original algorithms up to 30% with negligible peak signal-to-noise ratio degradation

A Search Patterns Switching Algorithm for Block Motion Estimation

Center-biased fast motion estimation algorithms, e.g., block-based gradient descent search and diamond search, can perform much better than coarse-to-fine search algorithms, such as 2-D logarithmic search and three-step search. The latter type of algorithms, however, is more suitable for handling large motion content. To combine the advantages of both types of algorithms, an adaptive algorithm performing search patterns switching (SPS) is proposed in this paper. The proposed SPS algorithm classifies the motion content of a block using a simple yet efficient motion content classifier called error descent rate. Unlike other classifiers with heavy overhead, this classifier requires only the searching of a few points in the search window and then a division operation. Experimental results show that the proposed SPS algorithm is very robust.

Depth map misalignment correction and dilation for DIBR view synthesis

The quality of the synthesized views by Depth Image Based Rendering (DIBR) highly depends on the accuracy of the depth map, especially the alignment of object boundaries of texture image. In practice, the misalignment of sharp depth map edges is the major cause of the annoying artifacts at the disoccluded regions of the synthesized views. Conventional smooth filter approach blurs the depth map to reduce the disoccluded regions. The drawbacks are the degradation of 3D perception of the reconstructed 3D videos and the destruction of the texture in background regions. Conventional edge preserving filter utilizes the color image information in order to align the depth edges with color edges. Unfortunately, the characteristics of color edges and depth edges are very different which causes annoying boundaries artifacts in the synthesized virtual views. Recent solution of reliability-based approach uses reliable warping information from other views to fill the holes. However, it is not suitable for the view synthesis in video-plus-depth based DIBR applications. In this paper, a new depth map preprocessing approach is proposed. It utilizes Watershed color segmentation method to correct the depth map misalignment and then the depth map object boundaries are extended to cover the transitional edge regions of color image. This approach can handle the sharp depth map edges lying inside or outside the object boundaries in 2D sense. The quality of the disoccluded regions of the synthesized views can be significantly improved and unknown depth values can also be estimated. Experimental results show that the proposed method achieves superior performance for view synthesis by DIBR especially for generating large baseline virtual views.

A foreground biased depth map refinement method for DIBR view synthesis

The performance of view synthesis using depth image based rendering (DIBR) highly depends on the accuracy of depth map. Inaccurate boundary alignment between texture image and depth map especially for large depth discontinuities always cause annoying artifacts in disocclusion regions of the synthesized view. Pre-filtering approach and reliability-based approach have been proposed to tackle this problem. However, pre-filtering approach blurs the depth map with drawback of degradation of the depth map and may also cause distortion in non-hole region. Reliability-based approach uses reliable warping information from other views to fill up holes and is not suitable for the view synthesis with single texture video such as video-plus-depth based DIBR applications. This paper presents a simple and efficient depth map preprocessing method with use of texture edge information to refine depth pixels around the large depth discontinuities. The refined depth map can make the whole texture edge pixels assigned with foreground depth values. It can significantly improve the quality of the synthesized view by avoiding incorrect use of foreground texture information in hole filling. The experimental results show the proposed method achieves superior performance for view synthesis by DIBR especially for large baseline.

Depth-aided exemplar-based hole filling for DIBR view synthesis

Quality of synthesized view by Depth-Image-Based Rendering (DIBR) highly depends on hole filling, especially for synthesized view with large disocclusion. Many hole filling methods are proposed to improve the synthesized view quality and inpainting is the most popular approach to recover the disocclusions. However, the conventional inpainting either makes the hole regions blurred via diffusion or propagates the foreground information to the disoclusion regions. Annoying artifacts are created in the synthesized virtual views. This paper proposes a depth-aided exemplar-based inpainting method for recovering large disoclusion. It consists of two processes, warped depth map filling and warped color image filling. Since depth map can be considered as a grey-scale image without texture, it is much easier to be filled. Disoccluded regions of color image are predicted based on its associated filled depth map information. Regions with texture lying around the background have higher priority to be filled than other regions and disoccluded regions are filled by propagating the background texture through the exemplar-based inpainting. Thus artifacts created by diffusion or using foreground information for prediction can be eliminated. Experimental results show texture can be recovered in large disocclusions and the proposed method has better visual quality compared to existing methods.

Subsampled Block-Matching for Zoom Motion Compensated Prediction

Motion compensated prediction plays a vital role in achieving enormous video compression efficiency in advanced video coding standards. Most practical motion compensated prediction techniques implicitly assume pure translational motions in the video contents for effective operation. Some attempts aiming at more general motion models are usually too complex requiring parameter estimation in practical implementation. In this paper, zoom motion compensation is investigated to extend the assumed model to support both zoom and translation motions. To accomplish practical complexity, a novel and efficient subsampled block-matching zoom motion estimation technique is proposed which makes use of the interpolated reference frames for subpixel motion estimation in a conventional hybrid video coding structure. Specially designed subsampling patterns in block matching are used to realize the translation and zoom motion estimation and compensation. No zoom parameters estimation and additional frame buffers are required in the encoder implementation. The complexity of the decoder is similar to the conventional hybrid video codec that supports subpixel motion compensation. The overall increase in memory requirement and computational complexity is moderate. Experimental results show that the new technique can achieve up to 9.89% bitrate reduction using KTA2.2r1 reference software implementation.

An adaptive background biased depth map hole-filling method for Kinect

The launch of Kinect provides a convenient way to access the depth information in real time. However the depth map quality still needs to be enhanced for 3D visual applications. In this paper, an adaptive background biased depth map hole-filling method is proposed. First, depth holes caused by abnormal reflection are filled by color similarity in-painting, and a soft decision for color similarity checking is performed by the use of probabilities in random walks color segmentation. Afterwards it is assumed that the lost information in the rest of depth holes belongs to the background. The background depth information is extracted by automatic thresholding in the neighborhood of each hole. Depth holes are in-painted with the background information in their local neighborhood. Combination of color similarity in-painting and background biased in-painting is able to perform depth map hole-filling adaptively for different kinds of depth holes for Kinect. The hole-filling results and virtual view synthesis results show that the Kinect depth map quality can be improved significantly by the proposed method.

Multiple block-size search algorithm for fast block motion estimation

Although variable block-size motion estimation provides significant video quality and coding efficiency improvement, it requires much higher computational complexity compared with fixed block size motion estimation. The reason is that the current motion estimation algorithms are mainly designed for fixed block size. Current variable block-size motion estimation implementation simply applies these existing motion estimation algorithms independently for different block sizes to find the best block size and the corresponding motion vector. Substantial computation is wasted because distortion data reuse among motion searches of different block sizes is not considered. In this paper, a motion estimation algorithm intrinsically designed for variable block-size video coding is presented. The proposed multiple block-size search (MBSS) algorithm unifies the motion searches for different block sizes into a single searching process instead of independently performing the search for each block size. In this unified search, the suboptimal motion vectors for different block sizes are used to determine the next search steps. Its prediction quality is comparable with that obtained by performing motion search for different block sizes independently while the computational load is substantially reduced. Experimental results show that the prediction quality of MBSS is similar to full search.

Watershed and Random Walks based depth estimation for semi-automatic 2D to 3D image conversion

Depth map estimation from a single image is the key problem for the 2D to 3D image conversion. Many 2D to 3D converting processes, either automatic or semi-automatic, are proposed before. Quality of the depth map from automatic methods is low and there exists wrong depth values due to errors estimation in depth cue extraction. The semi-automatic approaches can generate a better quality of depth map based on the user-defined labels, which indicate a rough estimation of depth values in the scene, to generate the rest of depth value and reconstruct the stereoscopic image. However, they require complexity system and are very computational intensive. A simplified approach is to combine the depth maps from Graph Cuts and Random Walks to persevering the sharp boundary and fine detail inside the objects. The drawback is the time consuming of the energy minimization in the Graph Cuts. In this paper, a fast Watershed segmentation based on the priority queue, which indicates the neighbor distance relationship, is used to replace the Graph Cuts to generate the hard constraints depth map. It is appended to the neighbor cost in the Random Walks to generate the final depth map with hard constraints in the objects boundaries regions and fine detail inside objects. The Watershed and Random Walks are low computational intensive and can achieve approximate real time estimation which results in a fast stereoscopic conversion process. Experimental results demonstrate that it can produce good quality stereoscopic image in very short time.