Adam Grzelka

New results in free-viewpoint television systems for horizontal virtual navigation

The paper presents the concept of a practical free-viewpoint television system with purely optical depth estimation. The system consists of camera modules that contain pairs or triples of cameras together with the respective microphones. The camera modules can be sparsely located in arbitrary positions around a scene. Each camera module is equivalent to a video camera with a depth sensor and microphones. The hardware requirements, the video and audio processing algorithms and the preliminary experimental results are reported. In particular, for such systems, a compression technique is discussed that is more efficient than the new 3D-HEVC technology. A set of new test sequences obtained with the use of camera pairs are presented.

Immersive visual media — MPEG-I: 360 video, virtual navigation and beyond

In the paper we present a method for increasing the quality of views synthesized with typical Depth-Image-Based Rendering (DIBR) view synthesis algorithms. In the proposed idea the resolution of input real views and corresponding depth maps is doubled before the view synthesis. After the synthesis, the resolution of a synthesized view is downsampled back to the original resolution. This approach is transparent for the view synthesis algorithms, thus can be used with any DIBR method. In the paper, tests for two synthesis algorithms (the state-of-the-art MPEG reference software and our view synthesis method) are presented. For both algorithms, the proposed upsampling improves objective and subjective quality of synthesized views.

Graph-based multiview depth estimation using segmentation

This paper presents a new depth estimation method for multiview systems with arbitrary camera locations. The method exploits the graph cuts method, where vertices of the graph represent segments used for controlling the trade-off between the quality of depth maps and the time of estimation, while preserving the original resolution of a depth map. Moreover, the inter-view consistency of the depth maps, crucial for free-viewpoint television systems, is ensured by introduction of suitable connections in the optimized graph. It makes the proposed method the first that allows generation of spatially-consistent multiview depth maps using segmentation-based estimation. A new method of the adaptive calculation of the smoothing coefficient was also presented. The performance of the proposed algorithm was tested and compared with the state-of-the-art DERS method, showing an significant improvement, both in terms of the depth maps fidelity and the time of estimation.

Methods of high efficiency compression for transmission of spatial representation of motion scenes

The paper reports the hitherto obtained results of the research project that deals with scene representations that are suitable for free navigation around the scene. These scene representations are acquired from a limited number of cameras located around the scene. The project is aimed at the development of an efficient compression technology and the experimental environment for testing such technology. The paper reports some new multiview test sequences with the circular camera arrangement, including the hybrid ones that have been acquired using also the depth sensors. Finally, the paper also briefly reports the recent results of the new compression technology being under development. This technology outperforms the state-of-the-art 3D-HEVC.

Multiview synthesis — Improved view synthesis for virtual navigation

In the paper, we propose a new method for the virtual view synthesis called as Multiview Synthesis. In our approach, virtual views are synthesized using two neighboring real views, but the disoccluded areas are not inpainted, but filled by the information from the further real views — additional views, traditionally not included in the view synthesis. The whole synthesis is performed with triangles rather than with individual pixels. In the proposed Multiview Synthesis, additional steps of adaptive color correction, blurred edge removal and spatial edge blurring are also included. The experimental results show that both the objective and subjective quality of the synthesized views is significantly higher than the quality of views obtained using the state-of-the-art MPEG reference view synthesis software.

Optimization of camera positions for free-navigation applications

In the article we deal with the problem of camera positioning in sparse multiview systems with applications to free navigation. The limited number of the cameras, though makes the system relatively practical, implies problems with proper depth estimation and virtual view synthesis, due to increased amount of the occluded areas. We present experimental results for the optimal positioning of the cameras, depending on two factors - characteristics of an acquired scene and the multi-camera system (linear or circular camera setup). The results show the correlation between the number of occlusions in the scene and a gain from using camera pairs instead of uniformly distributed cameras.

Depth map upsampling and refinement for FTV systems

In free-viewpoint television, high-quality depth maps are substantial for a virtual view synthesis for free navigation purposes. In this paper, we propose a new method of depth map quality improvement and resolution increase. Our method is intended for low resolution depth maps acquired through estimation using multiview video. The proposed depth map quality improvement is based on segmentation of an acquired high-resolution image. In the estimated depth map, searching for outliers is performed in the neighborhood created from similar segments in the acquired view. Experimental results show high effectiveness of the presented method of depth maps refinement, especially for object edges.

Temporal enhancement of graph-based depth estimation method

This paper presents the temporal enhancement of the graph-based depth estimation method, designed for multiview systems with arbitrarily located cameras. The primary goal of the proposed enhancement is to increase the quality of estimated depth maps and simultaneously decrease the time of estimation. The method consists of two stages: the temporal enhancement of segmentation required in used depth estimation method, and the exploitation of depth information from the previous frame in the energy function minimization. Performed experiments show that for all tested sequences the quality of estimated depth maps was increased. Even if only one cycle of optimization is used in proposed method, the quality is higher than for unmodified method, apart from number of cycles. Therefore, use of proposed enhancement allows estimating depth of better quality even with 40% reduction of estimation time.

The influence of a lossy compression on the quality of estimated depth maps

The paper considers the influence of a lossy compression of multiview video on the quality of estimated depth maps. We have encoded a set of views with commonly known compression techniques used in compact IP cameras: HEVC (MPEG H part 2 / H.265), A VC (MPEG 4 part 10 / H.264) and MPEG 2 part 2 / H.262. Basing on the decoded data we have estimated depth maps and rendered a set of the intermediate virtual views. A drop of the quality was studied by comparison of virtual views rendered out of the depth maps estimated from an uncompressed video. The obtained results show that with a reasonable compression rate, lower quality of virtual views is acceptable for real implementable FTV systems.