Shiori Sugimoto

Block-adaptive palette-based prediction for depth map coding

This paper proposes a novel prediction scheme for depth map coding. We utilize the fact that depth values are largely dependent on objects and one small block consists of only a small number of objects at most. The proposed method approximates each block with one palette and one object shape map. The palette consists of two representative depth values for foreground object and background object in the target block. The object shape map expresses which object exists at each pixel. These data are encoded only at the blocks where the proposed method is used. In order to reduce the bitrates required to transfer palettes and object shape maps, the proposed method enables their prediction by utilizing spatial and temporal correlations. Experiments are conducted on three kinds of depth maps; estimated from multiview images, captured by special sensors, and generated by computer. The results show that the proposed method reduces the bitrate by up to 40% and about 20% on average for 13 sequences relative to the MPEG-4 AVC/H.264.

Decoder-side macroblock information derivation for efficient multiview video plus depth map coding

This paper proposes a novel method that utilizes inter-view correlation in order to reduce the bitrate required to represent the prediction mode including the intra prediction direction, the motion vectors, the reference picture indexes, and the sub-block partitioning. The proposed method derives these kinds of macroblock (MB) information at the decoder side by comparing each image signal predictor with the view synthesis pictures. To accurately evaluate the predictors, we also propose a measure that minimizes the impact of errors in the depth information and view-dependent image signals. Experiments show that the proposed method reduces the bitrate by up to 25% relative to the international multiview video coding standard, and about 20% relative to the conventional view synthesis prediction method.

Backward view synthesis prediction using virtual depth map for multiview video plus depth map coding

View synthesis prediction has been studied as an efficient inter-view prediction scheme. Existing view synthesis prediction schemes fall into two types according to the pixel warping direction. While backward warping based view synthesis prediction enables block-based processing, forward warping based view synthesis prediction can handle occlusions properly. This paper proposes a two-step warping based view synthesis prediction; a virtual depth map is first generated by forward warping, and then prediction signals are generated by block-based backward warping using the virtual depth map. The technique of backward-warping-aware depth inpainting is also proposed. Experiments show that the proposed VSP scheme can achieve the decoder runtime reductions of about 37% on average with slight bitrate reductions relative to the conventional forward warping based VSP. Compared to the conventional backward warping based VSP, the proposed method reduces the bitrate for the synthesized views by up to 2.9% and about 2.2% on average.

Multi-layered image retargeting

Pixel domain methods for content-aware image retargeting have a performance limitation in that they distort important objects when an image is shrunken too much. To address this problem, we have designed and developed a novel 3D space domain image retargeting which rearranges objects with occlusions and deforms objects. With this method image formation is modeled as a quasi-3D space. The method also reduces image width by shrinking a part of the image space. Instead of directly optimizing the position and shape of objects, it closes objects by shrinking the gap space between them and deforms objects by shrinking spaces that include parts of them. It is capable of being extended to higher dimension problems such as multi-view image retargeting because it needs to optimize only one quasi-3D space whether the input is ordinary images or more complex ones such as video and stereo images. We have also designed a simple algorithm that makes use of dynamic programming. An experiment we conducted demonstrated the algorithms effectiveness.

Improved disparity vector derivation for inter-view residual prediction in 3D-HEVC

Inter-view residual prediction and advanced residual prediction (ARP) are efficient tools for coding dependent views of 3D video. It can predict the residue of motion compensated prediction (MCP) using the additional derived disparity vector. However, its coding performance depends on the accuracy of disparity vector derivation. In this paper, we propose the improved disparity vector derivation scheme for ARP. In the proposed scheme, the disparity vector is derived from the corresponding block in the reference picture. And that corresponding block is pointed by the same as MCP. Moreover, the disparity vector can be derived not only from the current reference block, but also from the blocks on the all other reference pictures included in the current reference picture lists. In addition, the disparity vector can be derived from both of the blocks predicted by disparity compensated prediction (DCP) and the blocks predicted by MCP and ARP because the derived disparity vector for ARP is stored in additional disparity vector field. Experimental results show that 0.2% bitrate reduction of synthesized views, and up to 0.5% for each dependent views in the reference software of 3D-HEVC.

Disparity vector derivation based on consistency of depth and disparity in 3D-HEVC

In this paper, we propose a novel disparity vector derivation scheme. In the proposed scheme, the preliminary disparity vector is derived while keeping consistency between the preliminary disparity and depth values of the block in the base view indicated by the preliminary disparity. And then, the final DV is derived from depth value of the block indicated by preliminary DV. The proposed derivation can derive a appropriate disparity even in the case that the disparity vector from neighboring block is disabling. Moreover, the derived DV can be utilized for advanced motion vector prediction in addition to the previous usage. The experimental results show that 0.2% BD-rate reduction with no complexity increase in the reference software of 3D-HEVC.

Depth-based weighted bi-prediction for video plus depth map coding

Video with depth is attracting attention as a promising approach to future 3D video. Unlike conventional video coding technologies, video with depth should take the correlations between video and depth into account. This paper proposes a novel bi-prediction scheme for video coding that exploits the structure similarity between video and depth. In the proposed scheme, a prediction image is generated by the pixel-wise weighted average of motion compensated reference images, and weights are decided by considering pixel similarities in the depth domain. The proposed pixel-wise weighted bi-prediction offers implicit depth-based block partitioning with consideration of the pixel correlations between predicted and reference blocks. Compared to using explicit, arbitrary shaped block partitioning, the proposed method offers better prediction for non-rigid objects, and requires no increase in motion information. Experiments show that the proposed method can achieve up to 17% bitrate reduction for bi-predictive frames and about 8% on average for 6 sequences relative to the existing biprediction scheme.