Olgierd Stankiewicz

High Efficiency 3D Video Coding Using New Tools Based on View Synthesis

We propose a new coding technology for 3D video represented by multiple views and the respective depth maps. The proposed technology is demonstrated as an extension of the recently developed high efficiency video coding (HEVC). One base views are compressed into a standard bitstream (like in HEVC). The remaining views and the depth maps are compressed using new coding tools that mostly rely on view synthesis. In the decoder, those views and the depth maps are derived via synthesis in the 3D space from the decoded baseview and from data corresponding to small disoccluded regions. The shapes and locations of those disoccluded regions can be derived by the decoder without any side information transmitted. To achieve high compression efficiency, we propose several new tools such as depth-based motion prediction, joint high frequency layer coding, consistent depth representation, and nonlinear depth representation. The experiments show high compression efficiency of the proposed technology. The bitrate needed for transmission of two side views with depth maps is mostly less than 50% than that of the bitrate for a single-view video.

Coding of multiple video+depth using HEVC technology and reduced representations of side views and depth maps

During the last two decades, a new technology generation of video compression was introduced about each 9 years. Each new compression-technology generation provides halving of necessary bitrates as compared to the last previous generation. This increasing single-view compression performance is related to increasing compression performance of multiview video coding. For multiview video with associated depth maps, additional significant bitrate reduction may be achieved. The paper reports the original compression technology that was designed and developed at Poznań University of Technology in response to MPEG Call for Proposals on 3D Video Coding Technology. The main idea of this technique is to predict very efficiently the side views and the depth maps from the base view.

Extensions of the HEVC technology for efficient multiview video coding

The paper deals with multiview video coding using the new technology of High Efficiency Video Coding (HEVC). Implementation of multiview video coding in the framework of HEVC is described together with new specific tools proposed by the authors. Extensive experimental results are reported for compression performance comparison of MVC (ISO 14496-10), HEVC simulcast and two versions of proposed “multiview HEVC”. For “multiview HEVC” the results indicate significant bitrate reduction of about 50%, as compared to the state-of-the-art MVC technology standardized as a part of AVC (MPEG-4, H.264).

A practical approach to acquisition and processing of free viewpoint video

We deal with the processing of multiview video acquired by the use of practical thus relatively simple acquisition systems that have a limited number of cameras located around a scene on independent tripods. The real-camera locations are nearly arbitrary as it would be required in the real-world Free-Viewpoint Television systems. The appropriate test video sequences are also reported. We describe a family of original extensions and adaptations of the multiview video processing algorithms adapted to arbitrary camera positions around a scene. The techniques constitute the video processing chain for Free-Viewpoint Television as they are aimed at estimating the parameters of such a multi-camera system, video correction, depth estimation and virtual view synthesis. Moreover, we demonstrate the need for new compression technology capable of efficient compression of sparse convergent views. The experimental results for processing the proposed test sequences are reported.

Similarity measures for depth estimation

This paper deals with similarity measures for stereoscopic depth estimation. These measures are used for matching of image pairs, which is the first step of the estimation process. We analyze influence of these similarity measures on performance of depth estimation with use of commonly known measures and compare the results with some novel proposals. The performance is judged by increase of quality of view synthesis, which is the main aim of this paper. Experimental results over a variety of moving material demonstrate that considerable gain can be attained without any modifications to estimation core and with tuning of matching stage only. Finally, some guidelines on design of well performing similarity measures are given. For the sake of paper, the whole work is described in context of belief-propagation algorithm, but the results and conclusions apply in general for many other state-of-the art optimization techniques.

Error concealment for MVC and 3D video coding

In this paper we propose a novel approach to error concealment that can be applied to MVC and other 3D video coding technologies. The image content, that is lost due to errors, is recovered with use of multiple error-concealment techniques. In our work we have used three techniques: well-known temporal- and intra-based techniques and a novel inter-view technique. Proposed inter-view recovery employs Depth Image Based Rendering (DIBR), which requires neighboring views and corresponding depth maps. Those depth maps can be delivered in the bit-stream or estimated in the receiver. In order to obtain the final reconstruction, the best technique is selected locally. For that, an original recovery quality measurement method, based on cross-checking, has been proposed. The idea has been implemented and assessed experimentally, with use of 3D video test sequences. The objective and subjective results show that the proposed approach provide good quality of reconstructed video.

Depth map inter-view consistency refinement for multiview video

This paper describes a technique for inter-view depth map consistency improvement for automatically and semi-automatically estimated depth maps. The goal is to improve 3D scene representation consistency by exchanging spatial information between all depth maps in a multiview sequence. Presented technique is based on iterative inter-view information exchange followed by depth quality assessment stage which prevents depth quality loss. The depth-map consistency improvement yields in better multi-view compression ratio and virtual view quality.

3D video compression by coding of disoccluded regions

In this paper, we present the most efficient coding tools that are used in a new video compression technology for multiple views with the depth maps. This very well performing technology was designed and developed in response to MPEG Call for Proposals on 3D Video Coding Technology. The proposed technology exploits a tool that reduces the side views and the side depth maps to small disoccluded regions. This way only one central view and one depth map are coded in the HEVC syntax while the remaining views and the depth maps are synthesized in the decoder from the small disoccluded regions and from the central-view data. Therefore, the bitrate needed for a side view is mostly below 20% of the bitrate for single-view video.

Experiments on acquisition and processing of video for free-viewpoint television

The paper describes an experimental multiview video production, processing and delivery chain developed at Poznan University of Technology for research on free-viewpoint television. The multiview-video acquisition system consists of HD camera units with wireless synchronization, wireless control, video storage and power supply units. Therefore no cabling is needed in the system, which is important for shooting real-world events. The system is mostly used for nearly circular setup of cameras but the locations of cameras are arbitrary, and the procedures for system calibration and multiview video correction are consid-ered. The paper deals also with adoption for circular camera arrangement of the techniques implemented in Depth Estimation Reference Software and View Synthesis Reference Software.

New results in free-viewpoint television systems for horizontal virtual navigation

The paper presents the concept of a practical free-viewpoint television system with purely optical depth estimation. The system consists of camera modules that contain pairs or triples of cameras together with the respective microphones. The camera modules can be sparsely located in arbitrary positions around a scene. Each camera module is equivalent to a video camera with a depth sensor and microphones. The hardware requirements, the video and audio processing algorithms and the preliminary experimental results are reported. In particular, for such systems, a compression technique is discussed that is more efficient than the new 3D-HEVC technology. A set of new test sequences obtained with the use of camera pairs are presented.