Adrian Dziembowski

A practical approach to acquisition and processing of free viewpoint video

We deal with the processing of multiview video acquired by the use of practical thus relatively simple acquisition systems that have a limited number of cameras located around a scene on independent tripods. The real-camera locations are nearly arbitrary as it would be required in the real-world Free-Viewpoint Television systems. The appropriate test video sequences are also reported. We describe a family of original extensions and adaptations of the multiview video processing algorithms adapted to arbitrary camera positions around a scene. The techniques constitute the video processing chain for Free-Viewpoint Television as they are aimed at estimating the parameters of such a multi-camera system, video correction, depth estimation and virtual view synthesis. Moreover, we demonstrate the need for new compression technology capable of efficient compression of sparse convergent views. The experimental results for processing the proposed test sequences are reported.

Experiments on acquisition and processing of video for free-viewpoint television

The paper describes an experimental multiview video production, processing and delivery chain developed at Poznan University of Technology for research on free-viewpoint television. The multiview-video acquisition system consists of HD camera units with wireless synchronization, wireless control, video storage and power supply units. Therefore no cabling is needed in the system, which is important for shooting real-world events. The system is mostly used for nearly circular setup of cameras but the locations of cameras are arbitrary, and the procedures for system calibration and multiview video correction are consid-ered. The paper deals also with adoption for circular camera arrangement of the techniques implemented in Depth Estimation Reference Software and View Synthesis Reference Software.

New results in free-viewpoint television systems for horizontal virtual navigation

The paper presents the concept of a practical free-viewpoint television system with purely optical depth estimation. The system consists of camera modules that contain pairs or triples of cameras together with the respective microphones. The camera modules can be sparsely located in arbitrary positions around a scene. Each camera module is equivalent to a video camera with a depth sensor and microphones. The hardware requirements, the video and audio processing algorithms and the preliminary experimental results are reported. In particular, for such systems, a compression technique is discussed that is more efficient than the new 3D-HEVC technology. A set of new test sequences obtained with the use of camera pairs are presented.

Immersive visual media — MPEG-I: 360 video, virtual navigation and beyond

In the paper we present a method for increasing the quality of views synthesized with typical Depth-Image-Based Rendering (DIBR) view synthesis algorithms. In the proposed idea the resolution of input real views and corresponding depth maps is doubled before the view synthesis. After the synthesis, the resolution of a synthesized view is downsampled back to the original resolution. This approach is transparent for the view synthesis algorithms, thus can be used with any DIBR method. In the paper, tests for two synthesis algorithms (the state-of-the-art MPEG reference software and our view synthesis method) are presented. For both algorithms, the proposed upsampling improves objective and subjective quality of synthesized views.

Graph-based multiview depth estimation using segmentation

This paper presents a new depth estimation method for multiview systems with arbitrary camera locations. The method exploits the graph cuts method, where vertices of the graph represent segments used for controlling the trade-off between the quality of depth maps and the time of estimation, while preserving the original resolution of a depth map. Moreover, the inter-view consistency of the depth maps, crucial for free-viewpoint television systems, is ensured by introduction of suitable connections in the optimized graph. It makes the proposed method the first that allows generation of spatially-consistent multiview depth maps using segmentation-based estimation. A new method of the adaptive calculation of the smoothing coefficient was also presented. The performance of the proposed algorithm was tested and compared with the state-of-the-art DERS method, showing an significant improvement, both in terms of the depth maps fidelity and the time of estimation.

Methods of high efficiency compression for transmission of spatial representation of motion scenes

The paper reports the hitherto obtained results of the research project that deals with scene representations that are suitable for free navigation around the scene. These scene representations are acquired from a limited number of cameras located around the scene. The project is aimed at the development of an efficient compression technology and the experimental environment for testing such technology. The paper reports some new multiview test sequences with the circular camera arrangement, including the hybrid ones that have been acquired using also the depth sensors. Finally, the paper also briefly reports the recent results of the new compression technology being under development. This technology outperforms the state-of-the-art 3D-HEVC.

Multiview synthesis — Improved view synthesis for virtual navigation

In the paper, we propose a new method for the virtual view synthesis called as Multiview Synthesis. In our approach, virtual views are synthesized using two neighboring real views, but the disoccluded areas are not inpainted, but filled by the information from the further real views — additional views, traditionally not included in the view synthesis. The whole synthesis is performed with triangles rather than with individual pixels. In the proposed Multiview Synthesis, additional steps of adaptive color correction, blurred edge removal and spatial edge blurring are also included. The experimental results show that both the objective and subjective quality of the synthesized views is significantly higher than the quality of views obtained using the state-of-the-art MPEG reference view synthesis software.

Optimization of camera positions for free-navigation applications

In the article we deal with the problem of camera positioning in sparse multiview systems with applications to free navigation. The limited number of the cameras, though makes the system relatively practical, implies problems with proper depth estimation and virtual view synthesis, due to increased amount of the occluded areas. We present experimental results for the optimal positioning of the cameras, depending on two factors - characteristics of an acquired scene and the multi-camera system (linear or circular camera setup). The results show the correlation between the number of occlusions in the scene and a gain from using camera pairs instead of uniformly distributed cameras.

Depth map upsampling and refinement for FTV systems

In free-viewpoint television, high-quality depth maps are substantial for a virtual view synthesis for free navigation purposes. In this paper, we propose a new method of depth map quality improvement and resolution increase. Our method is intended for low resolution depth maps acquired through estimation using multiview video. The proposed depth map quality improvement is based on segmentation of an acquired high-resolution image. In the estimated depth map, searching for outliers is performed in the neighborhood created from similar segments in the acquired view. Experimental results show high effectiveness of the presented method of depth maps refinement, especially for object edges.

View and depth preprocessing for view synthesis enhancement

In the paper, two preprocessing methods for virtual view synthesis are presented. In the first approach, both horizontal and vertical resolutions of the real views and the corresponding depth maps are doubled in order to perform view synthesis on images with densely arranged points. In the second method, real views are filtered in order to eliminate blurred or improperly shifted edges of the objects. Both methods are performed prior to synthesis, thus they may be applied to different Depth-Image-Based Rendering algorithms. In the paper, for both proposed methods, the achieved quality gains are presented.