Fabian Jäger

Contour-based segmentation and coding for depth map compression

Emerging video technologies like 3DTV and Free Viewpoint Video require to transmit more information than just 2D color data. To allow rendering of arbitrary viewpoints of a video scene a new data format including 2D color and an accompanying depth map has been proposed. Consequently this new technique requires an efficient coding method not only for color information, but also for depth data. Depth maps are characterized by segments describable by piecewise linear functions bounded by sharp edges. Preserving these depth discontinuities is a crucial requirement for high quality view synthesis. To adapt to these characteristics we propose a novel algorithm for depth map compression explicitly signaling the location of discontinuities. Experimental results show that the proposed method yields up to 9dB PSNR gain compared to a JPEG-2000 encoder in high-bitrate scenarios. Subjective quality assessment of synthesized views using compressed depth maps can prove the superior visual quality originating from less geometric distortions.

Simplified depth map intra coding with an optional depth lookup table

3D Video is a new technology, which requires the transmission of depth data alongside conventional 2D video. The additional depth information allows to synthesize arbitrary viewpoints at the receiver and enables adaptation of the perceived depth impression and driving of multi-view auto-stereoscopic displays. In contrast to natural video signals, depth maps are characterized by piecewise smooth regions bounded by sharp edges along depth discontinuities. Conventional video coding methods tend to introduce ringing artifacts along these depth discontinuities, which lead to visually disturbing geometric distortions in the view synthesis process. Therefore, preserving the described signal characteristics of depth maps is a crucial requirement for new depth coding algorithms. In this paper a novel simplified intra-coding mode is presented, which works as an alternative coding path to the conventional transform-based intra coding scheme. The proposed intra coding mode yields up to 1.3% BD-rate savings in terms of total bitrate, including texture bitrate. An optional Depth Lookup Table can further increase the bitrate reduction to 2.1%.

Depth-based block partitioning for 3D video coding

3D video is an emerging technology that bundles depth information with texture videos to allow for view synthesis applications at the receiver. Depth discontinuities define object boundaries in both, depth maps and the collocated texture video. Therefore, segmentation of the depth map can be used for more accurate partitioning of the corresponding texture component. In this paper, already coded depth maps are used to increase coding efficiency for texture videos by deriving an arbitrarily shaped partitioning of a texture block based on the collocated depth block. By applying motion compensation to each partition and combining them afterwards by a depth-based segmentation mask, highly accurate prediction signals can be formed that reduce the remaining residual signal significantly. Simulation results show bitrate savings of up to 5.7% for the dependent texture views and up to 2.3% with respect to the total bitrate.

Model-based intra coding for depth maps in 3D video using a depth lookup table

3D Video is a new technology, which requires the transmission of depth data alongside conventional 2D video. The additional depth information allows to synthesize arbitrary viewpoints at the receiver and enables adaptation of the perceived depth impression and driving of multi-view auto-stereoscopic displays. In contrast to natural video signals, depth maps are characterized by piecewise smooth regions bounded by sharp edges along depth discontinuities. Conventional video coding methods tend to introduce ringing artifacts along these depth discontinuities, which lead to visually disturbing geometric distortions in the view synthesis process. Preserving the described signal characteristics of depth maps is therefore a crucial requirement for new depth coding algorithms. In this paper a novel model-based intra-coding mode is presented, which works as an addition to conventional transform-based intra coding tools. The proposed intra coding mode yields up to 42% BD-rate savings in terms of depth rate and up to 2:5% in terms of total rate. The average bitrate savings are approximately 24% for depth rate and 1.5% for the total rate including texture and depth.

Decoder complexity reduction for the scalable extension of HEVC

In the current standardization process of the scalable extension to High Efficiency Video Coding (SHVC) a high level syntax multi-loop approach is close to completion. On the one hand this multi-loop approach offers a reasonable rate-distortion performance while only minimal modifications to the encoder and decoder in both layers are required. On the other hand this approach requires full reconstruction of all pictures of all layers at the decoder side which, in the case of quality scalability with two layers, doubles the decoder complexity. In this paper high layer modifications to the prediction structure similar to the scalable extension of H.264 - AVC are implemented in SHVC and studied. These modifications allow for an enhancement layer decoder implementation to skip a significant amount of motion compensation and deblocking operations in the base layer. It is shown that the decoder complexity can hereby be reduced up to 55% for the random access configuration and up to 64% for the low delay configuration compared to SHVC. An overall coding performance increase of 1.2% when decoding the enhancement layer is reported while when only decoding the base layer a drift can be observed between -0.16 dB for random access and -0.39 dB for low delay.

Single-loop SNR scalability using binary residual refinement coding

In scalable video coding, a stream contains multiple layers of different temporal resolution, spatial resolution, or quality. Each enhancement layer can utilize the information from the lower layers in order to improve the overall coding efficiency. In the draft scalable extension of HEVC the enhancement layer uses the lower layer reconstructed pixel values for prediction. While this dual-loop approach is efficient in terms of rate-distortion performance, it doubles the decoder complexity. In this paper a method is presented that allows for single-loop decoding using a new coefficient refinement scheme for the enhancement layer. The scheme conceptually allows for binary mapping between arbitrary quantizer step sizes in base and enhancement layer. Simulation results reveal a promising rate-distortion performance, especially for small quantizer deltas.

Median trilateral loop filter for depth map video coding

Emerging extensions to conventional stereo video technologies like 3D Video require to add depth information to 2D video data. This supplementary data needs to be coded efficiently and transmitted to the receiver where arbitrary viewpoints are generated by using this additional information. The depth maps are characterized by piecewise smooth regions, which are bounded by sharp edges describing depth discontinuities along object boundaries. Preserving these characteristics and especially depth discontinuities is a crucial requirement for depth map coding. When coding depth maps by means of a conventional hybrid video coder, ringing artifacts are introduced along the sharp edges and result in quality degradation when using the reconstructed depth maps for view synthesis. To reduce these ringing artifacts and also to better align object boundaries in video and depth data, a new in-loop filter is proposed, which reconstructs the described characteristics of depth maps.

Segment-wise prediction in 3D video coding

3D video is an emerging technology that bundles depth information with texture videos to allow for view synthesis applications at the receiver. Depth discontinuities define object boundaries in both, depth maps and the collocated texture video. Therefore, depth segmentation can be utilized for a fine-grained motion field segmentation of the corresponding texture component. In this paper, depth information is used to increase coding efficiency for texture videos by deriving an arbitrarily shaped segmentation mask. By applying independent motion compensation to each of these segments and subsequently merging the two prediction signals, a highly accurate prediction signal can be generated that reduces the energy in the remaining texture residual signal. Simulation results show bitrate savings of up to 2.8% for the dependent texture views and up to about 1.0% with respect to the total bitrate.

Low complexity transform coding for depth maps in 3D video

3D video is a new technology, which requires transmission of depth data alongside conventional 2D video. The additional depth information allows to synthesize arbitrary viewpoints at the receiver for adaptation of perceived depth impression and for driving of multi-view auto-stereoscopic displays. Depth maps typically show different signal characteristics compared to textured video data. Piecewise smooth regions are bounded by sharp edges resembling depth discontinuities. These edges lead to strong ringing artifacts when depth maps are coded with DCT-based transform codecs, such as AVC or its successor HEVC. In this paper alternative transforms are proposed to be used for coding depth maps for 3D video. By replacing the DCT with these transforms, ringing artifacts in the reconstructed depth maps are reduced and at the same time the complexity of the transform stage is lowered significantly. For high quality depth map coding the proposed alternative transforms can even increase coding efficiency.

Content-adaptive encoder optimization of the H.264/AVC deblocking filter for visual quality improvement

This paper presents a new self-adapting and content-sensitive optimization technique for the H.264/AVC in-loop deblocking filter [1], focussing on the visual enhancement of the perceived reconstruction quality. Performed frame-wisely at the encoder side, the proposed algorithm first identifies visually important image regions in the currently decoded and blocky frame, including natural and artificial edge areas, and then optimizes the filter inherent threshold decisions, finding the best trade-off between preserving natural and suppressing artificial edges. As fast optimization criterion, the low complex Edge-PSNR [2] is employed, which has proved a very good congruence to the human visual quality sensation, much better than standard PSNR. In this work, the optimization behavior of all thresholds is analyzed, showing their mostly good-natured curve shape for convex optimization, but high content dependency of the optimal values. The in-loop coding results for AVC evidence the approachs high capability for visual improvement, whose general design is easily portable to other AVC deblocking based architectures like HEVC.