Wooshik Kim

Depth map distortion analysis for view rendering and depth coding

Video representations that support view synthesis based on depth maps, such as multiview plus depth (MVD), have been recently proposed, raising interest in efficient tools for depth map coding. In this paper, we derive a new distortion metric that takes into consideration camera parameters and global video characteristics in order to quantify the effect of lossy coding of depth maps on synthesized view quality. In addition, a new skip mode selection method is proposed based on local video characteristics. Experimental results with the proposed mode selection scheme show coding gains of up to 2 dB for the synthesized views, as well as better subjective quality.

Depth map coding with distortion estimation of rendered view

New data formats that include both video and the corresponding depth maps, such as multiview plus depth (MVD), enable new video applications in which intermediate video views (virtual views) can be generated using the transmitted/stored video views (reference views) and the corresponding depth maps as inputs. We propose a depth map coding method based on a new distortion measurement by deriving relationships between distortions in coded depth map and rendered view. In our experiments we use a codec based on H.264/AVC tools, where the rate-distortion (RD) optimization for depth encoding makes use of the new distortion metric. Our experimental results show the efficiency of the proposed method, with coding gains of up to 1.6 dB in interpolated frame quality as compared to encoding the depth maps using the same coding tools but applying RD optimization based on conventional distortion metrics.

Edge-adaptive transforms for efficient depth map coding

In this work a new set of edge-adaptive transforms (EATs) is presented as an alternative to the standard DCTs used in image and video coding applications. These transforms avoid filtering across edges in each image block, thus, they avoid creating large high frequency coefficients. These transforms are then combined with the DCT in H.264/AVC and a transform mode selection algorithm is used to choose between DCT and EAT in an RD-optimized manner. These transforms are applied to coding depth maps used for view synthesis in a multi-view video coding system, and provides up to 29% bit rate reduction for a fixed quality in the synthesized views.

Bit-depth scalable coding for high dynamic range video

This paper presents a technique for coding high dynamic range videos. The proposed coding scheme is scalable, such that both standard dynamic range and high dynamic range representations of a video can be extracted from one bit stream. A localized inverse tone mapping method is proposed for efficient inter-layer prediction, which applies a scaling factor and an offset to each macroblock, per color channel. The scaling factors and offsets are predicted from neighboring macroblocks, and then the differences are entropy coded. The proposed inter-layer prediction technique is independent of the forward tone mapping method and is able to cover a wide range of bit-depths and various color spaces. Simulations are performed based on H.264/AVC SVC common software and core experiment conditions. Results show the effectiveness of the proposed method.

Edge-aware intra prediction for depth-map coding

This work proposes a new intra prediction coding scheme for depth map images used in view interpolation. The main goal is to design a prediction scheme which can reduce the prediction error energy in blocks with arbitrary edge shapes. This will reduce the rate needed to encode such blocks while also eliminating some of the annoying artifacts caused by quantization. Since depth maps typically consist of smooth regions separated by edges, we find it sufficient to design prediction schemes which can make effective use of edge information. Working from the intra prediction framework in H.264, we provide a graph representation of pixels in a block and pixels from previously coded blocks and construct an edge-aware prediction scheme based on this. We also employ existing rate-distortion (RD) optimization methods to further improve the coding performance. Our proposed methods reduce the bit rate for depth maps by up to 29% for a fixed interpolated PSNR for some sequences.

Graph based transforms for depth video coding

In this paper a graph-based transform is proposed as an alternative to the discrete cosine transform. An image or video signal is represented as a graph signal, where the graph is generated so as not to cross an image edge in a local region, i.e., square block. Then, spectral representation of graph signal is used to form transform kernels by finding eigenvectors of Laplacian matrix of the graph. This method requires to include additional information, i.e., edge map or adjacency matrix, into a bitstream so that a decoder can regenerate the exactly same graph used at an encoder. The novelty of this paper includes finding the optimal adjacency matrix and compressing it using context-based adaptive binary arithmetic coding. Coding efficiency improvement can be achieved when an image block contains arbitrarily shaped edges by applying the transform not across the edges. The proposed transform is applied to coding depth maps used for view synthesis in a multi-view video coding system, and provides 14% bit rate savings on average.

Interplane prediction for RGB video coding

This paper presents an efficient video coding method in RGB space. Generally RGB space is regarded as a bad space from a compression point of view. Each RGB component contains the color information along with luminance information. This redundancy degrades the coding efficiency. To exploit the intercolor redundancy, the RGB space is usually converted into another color space. But it inevitably causes color distortion due to the rounding operation involved. For professional applications it is essential to maintain the original color fidelity. To cope with these problems we propose an inter-plane prediction (IPP) that increases the coding efficiency in RGB space while avoiding color distortion. The proposed IPP exploits the redundancy of the residual images, which exists even after intra/inter prediction in MC-DCT based codec. The simulation results show that by applying the IPP to the MPEG-4 AVC/H.264 4:4:4 codec we can achieve up to 40% higher coding efficiency than that of the coding without IPP option. We also show there exists an achievable PSNR limit if color space conversion is involved. In other words, coding in a given space is important to keep the fidelity of the original data for professional applications.

Depth map coding using graph based transform and transform domain sparsification

Depth map compression is important for compact “texture-plus-depth” representation of a 3D scene, where texture and depth maps captured from multiple camera viewpoints are coded into the same format. Having received such format, the decoder can synthesize any novel intermediate view using texture and depth maps of two neighboring captured views via depth-image-based rendering (DIBR). In this paper, we combine two previously proposed depth map compression techniques that promote sparsity in the transform domain for coding gain-graph-based transform (GBT) and transform domain sparsification (TDS) — together under one unified optimization framework. The key to combining GBT and TDS is to adaptively select the simplest transform per block that leads to a sparse representation. For blocks without detected prominent edges, the synthesized views distortion sensitivity to depth map errors is low, and TDS can effectively identify a sparse depth signal in fixed DCT domain within a large search space of good signals with small synthesized view distortion. For blocks with detected prominent edges, the synthesized views distortion sensitivity to depth map errors is high, and the search space of good depth signals for TDS to find sparse representations in DCT domain is small. In this case, GBT is first performed on a graph defining all detected edges, so that filtering across edges is avoided, resulting in a sparsity count ρ in GBT. We then incrementally add the most important edge to an initial no-edge graph, each time performing TDS in the resulting GBT domain, until the same sparsity count ρ is achieved. Experimentation on two sets of multiview images showed gain of up to 0.7dB in PSNR in synthesized view quality compared to previous techniques that employ either GBT or TDS alone.

Depth Map Coding Optimization Using Rendered View Distortion for 3D Video Coding

In order to improve 3D video coding efficiency, we propose methods to estimate rendered view distortion in synthesized views as a function of the depth map quantization error. Our approach starts by calculating the geometric error caused by the depth map error based on the camera parameters. Then, we estimate the rendered view distortion based on the local video characteristics. The estimated rendered view distortion is used in the rate-distortion optimized mode selection for depth map coding. A Lagrange multiplier is derived using the proposed distortion metric, which is estimated based on an autoregressive model. Experimental results show the efficiency of the proposed methods, with average savings of 43% in depth map bitrate as compared with encoding the depth maps using the same coding tools but with the rate-distortion optimization based on the conventional distortion metric.

A new color transform for RGB coding

This paper presents a new color transform (YSbSr) that increases the coding efficiency for RGB space by reducing the conversion error and coding error propagation that happen during the color space conversion. Traditionally the use of color transforms has been limited to maintain the compatibility with black-and-white transmission. But for high quality video for professional applications, the focus should change accordingly. Recently during the course of development for Professional Extensions of the MPEG-4 AVC H.264 standard, new color transforms have been proposed. These transforms are focusing on the decorrelation and reversibility using a lifting scheme for integer implementation. But in a real coding scenario, lossless coding is rare and coding error is inevitable. So we have to deal with the propagation of coding error due to the backward conversion to RGB space. Even though integer mapping is a charming factor, it is more desirable not to employ integer mapping if we lose the original goal, high decorrelation gain with small conversion error. Since the conversion equation takes up a very small portion of coder complexity, the difference between integer mappings and floating point operation is diluted. Based on this observation, we propose a new color transform that gives: high decorrelation gain with small rounding and conversion error. The simulation results show that we can achieve almost up to 2 dB gain compared with the traditional YCbCr color space using the proposed transform. It makes the proposed color transform a good candidate for applications that need to preserve the original color fidelity.