Jacek Konieczny

High Efficiency 3D Video Coding Using New Tools Based on View Synthesis

We propose a new coding technology for 3D video represented by multiple views and the respective depth maps. The proposed technology is demonstrated as an extension of the recently developed high efficiency video coding (HEVC). One base views are compressed into a standard bitstream (like in HEVC). The remaining views and the depth maps are compressed using new coding tools that mostly rely on view synthesis. In the decoder, those views and the depth maps are derived via synthesis in the 3D space from the decoded baseview and from data corresponding to small disoccluded regions. The shapes and locations of those disoccluded regions can be derived by the decoder without any side information transmitted. To achieve high compression efficiency, we propose several new tools such as depth-based motion prediction, joint high frequency layer coding, consistent depth representation, and nonlinear depth representation. The experiments show high compression efficiency of the proposed technology. The bitrate needed for transmission of two side views with depth maps is mostly less than 50% than that of the bitrate for a single-view video.

Depth-based inter-view prediction of motion vectors for improved multiview video coding

The paper deals with efficient exploitation of mutual correlation that exists in motion fields of individual views in multiview video. The paper describes a new technique for efficient representation of motion data in multiview video bitstreams that carry also depth maps. These depth maps may be used in order to derive motion information from neighboring views. Such inter-view prediction of motion vectors is the core idea of InterView Direct compression mode that is proposed in this paper. Application of the new mode yields bitrate reductions between 2% and 13% depending on an individual test sequence, compression scenario and variant of the state-of-the-art multiview compression reference technique. This improvement has been demonstrated by extensive experimental tests that used standard multiview test video sequences.

Coding of multiple video+depth using HEVC technology and reduced representations of side views and depth maps

During the last two decades, a new technology generation of video compression was introduced about each 9 years. Each new compression-technology generation provides halving of necessary bitrates as compared to the last previous generation. This increasing single-view compression performance is related to increasing compression performance of multiview video coding. For multiview video with associated depth maps, additional significant bitrate reduction may be achieved. The paper reports the original compression technology that was designed and developed at Poznań University of Technology in response to MPEG Call for Proposals on 3D Video Coding Technology. The main idea of this technique is to predict very efficiently the side views and the depth maps from the base view.

Extensions of the HEVC technology for efficient multiview video coding

The paper deals with multiview video coding using the new technology of High Efficiency Video Coding (HEVC). Implementation of multiview video coding in the framework of HEVC is described together with new specific tools proposed by the authors. Extensive experimental results are reported for compression performance comparison of MVC (ISO 14496-10), HEVC simulcast and two versions of proposed “multiview HEVC”. For “multiview HEVC” the results indicate significant bitrate reduction of about 50%, as compared to the state-of-the-art MVC technology standardized as a part of AVC (MPEG-4, H.264).

Extended Inter-View Direct mode for Multiview Video Coding

This paper presents a novel compression tool that improves compression performance of Multiview Video Coding when depth information is available for the reference view. This tool is proposed as a new Extended Inter-View Direct mode of macroblock coding and exploits improved inter-view prediction of motion information. For this prediction, the idea is to use depth information in order to predict motion vectors more accurately, as it is possible in the classic Multiview Video Coding. The Extended Inter-View Direct mode is designed especially for the purpose of joint video and depth coding, where depth information for coded view is not available. Extensive experiments show the potential of the new mode to reduce bitstream by 2% to 11% depending on test sequence, compression scenario and variant of the state-of-the-art multiview compression reference technique used.

3D video compression by coding of disoccluded regions

In this paper, we present the most efficient coding tools that are used in a new video compression technology for multiple views with the depth maps. This very well performing technology was designed and developed in response to MPEG Call for Proposals on 3D Video Coding Technology. The proposed technology exploits a tool that reduces the side views and the side depth maps to small disoccluded regions. This way only one central view and one depth map are coded in the HEVC syntax while the remaining views and the depth maps are synthesized in the decoder from the small disoccluded regions and from the central-view data. Therefore, the bitrate needed for a side view is mostly below 20% of the bitrate for single-view video.

Depth-based inter-view motion data prediction for HEVC-based multiview video coding

The paper deals with efficient coding of motion data for compression of multiview video with depth maps. This research was done in the context of the new compression technology called High Efficiency Video Coding. In the proposed approach, motion vectors and reference frame indices are predicted from reference view for each pixel using the Depth-Based Motion Prediction (DBMP). The new codec with DBMP compression tool was compared to the multiview HEVC-based codec i.e. a set of HEVC codecs augmented with inter-view texture prediction tools adopted from classic multiview codec (MVC) based on the AVC technology. Extensive experiments show the potential of the new DBMP predictor to reduce bitrate up to 12% against the multiview HEVC-based codec.

Fast depth estimation on mobile platforms and FPGA devices

In this paper, we propose a fast technique for real-time depth estimation that is implementable on mobile devices like smartphones and tablets. Moreover, the FPGA-based implementations of this technique are also reported. The idea of this technique is to use small-block matching and exploit the recently estimated disparity values in order to enhance spatial consistency of the output disparity map. Using the Middlebury stereoscopic test images, the proposed real-time technique is compared to other techniques known from the references. The obtained results demonstrate high efficiency of the proposed technique implemented both on mobile platforms and FPGA devices.

Football player detection in video broadcast

The paper describes a novel segmentation system based on the combination of Histogram of Oriented Gradients (HOG) descriptors and linear Support Vector Machine (SVM) classification for football video. Recently, HOG methods were widely used for pedestrian detection. However, presented experimental results show that combination of HOG and SVM is very promising for locating and segmenting players. In proposed system a dominant color based segmentation for football playfield detection and a 3D playfield modeling based on Hough transform is introduced. Experimental evaluation of the system is done for SD (720×576) and HD (1280×720) test sequences. Additionally, we test proposed system performance for different lighting conditions (non-uniform pith lightning, multiple player shadows) as well as for various positions of the cameras used for acquisition.

Depth-enhanced compression for 3D video

The original idea of the paper is to exploit depth maps in order to increase compression efficiency for multiview video. Depth information is used to establish a 3D mapping between each pixel in an encoded frame and its counterpart in the reference view. With this mapping motion vectors and reference frame indices can be obtained independently for each pixel in a coded picture by a simple derivation of the motion information assigned to the corresponding pixel in the reference view. The goal of the paper is to explore a practical way to use this idea in multiview video coding. This goal has been achieved by experimental testing of various sets of direct and skip modes with and without inter-frame motion prediction. It was shown that either “all modes” have to be used or “all but not classic direct mode” should be used. The respective experimental results have been provided in the paper.