Dirk Farin

The effects of multiview depth video compression on multiview rendering

This article investigates the interaction between different techniques for depth compression and view synthesis rendering with multiview video plus scene depth data. Two different approaches for depth coding are compared, namely H.264/MVC, using temporal and inter-view reference images for efficient prediction, and the novel platelet-based coding algorithm, characterized by being adapted to the special characteristics of depth-images. Since depth-images are a 2D representation of the 3D scene geometry, depth-image errors lead to geometry distortions. Therefore, the influence of geometry distortions resulting from coding artifacts is evaluated for both coding approaches in two different ways. First, the variation of 3D surface meshes is analyzed using the Hausdorff distance and second, the distortion is evaluated for 2D view synthesis rendering, where color and depth information are used together to render virtual intermediate camera views of the scene. The results show that-although its rate-distortion (R-D) performance is worse-platelet-based depth coding outperforms H.264, due to improved sharp edge preservation. Therefore, depth coding needs to be evaluated with respect to geometry distortions.

Depth-Image Compression Based on an R-D Optimized Quadtree Decomposition for the Transmission of Multiview Images

This paper presents a novel depth-image coding algorithm that concentrates on the special characteristics of depth images: smooth regions delineated by sharp edges. The algorithm models these smooth regions using piecewise-linear functions and sharp edges by a straight line. To define the area of support for each modeling function, we employ a quadtree decomposition that divides the image into blocks of variable size, each block being approximated by one modeling function containing one or two surfaces. The subdivision of the quadtree and the selection of the type of modeling function is optimized such that a global rate-distortion trade-off is realized. Additionally, we present a predictive coding scheme that improves the coding performance of the quadtree decomposition by exploiting the correlation between each block of the quadtree. Experimental results show that the described technique improves the resulting quality of compressed depth images by 1.5-4 dB when compared to a JPEG-2000 encoder.

Robust camera calibration for sport videos using court models

We propose an automatic camera calibration algorithm for court sports. The obtained camera calibration parameters are required for applications that need to convert positions in the video frame to real-world coordinates or vice versa. Our algorithm uses a model of the arrangement of court lines for calibration. Since the court model can be specified by the user, the algorithm can be applied to a variety of different sports. The algorithm starts with a model initialization step which locates the court in the image without any user assistance or a-priori knowledge about the most probable position. Image pixels are classified as court line pixels if they pass several tests including color and local texture constraints. A Hough transform is applied to extract line elements, forming a set of court line candidates. The subsequent combinatorial search establishes correspondences between lines in the input image and lines from the court model. For the succeeding input frames, an abbreviated calibration algorithm is used, which predicts the camera parameters for the new image and optimizes the parameters using a gradient-descent algorithm. We have conducted experiments on a variety of sport videos (tennis, volleyball, and goal area sequences of soccer games). Video scenes with considerable difficulties were selected to test the robustness of the algorithm. Results show that the algorithm is very robust to occlusions, partial court views, bad lighting conditions, or shadows.

The Effect of Depth Compression on Multiview Rendering Quality

This paper presents a comparative study on different techniques for depth-image compression and its implications on the quality of multiview video plus depth virtual view rendering. A novel coding algorithm for depth images that concentrates on their special characteristics, namely smooth regions delineated by sharp edges, is compared to H.264 intra-coding with depth- images. These two coding techniques are evaluated in the context of multiview video plus depth representations, where depth information is used to render virtual intermediate camera views of the scene. Therefore it is important to evaluate the influence of depth-image coding artifacts on the quality of rendered virtual views. The results of this evaluation show, that the coding algorithm specialized on the characteristics of depth images outperforms H.264 intra-coding, although its RD-performance is worse.

Robust background estimation for complex video sequences

Knowing the background image of a video scene simplifies the general video-object segmentation problem and therefore it is required by several automatic segmentation algorithms. This paper presents a new background estimation algorithm which is applicable to complex video sequences where many objects are simultaneously visible and the background is visible for a short time period only. The algorithm applies a rough segmentation of the input images into foreground and background regions to exclude the foreground objects from background synthesis. This prevents a bias of the synthesized background image towards the color of foreground objects. Experiments show that the obtained background images differ significantly less from the real background than those obtained with previous algorithms.

System architecture for free-viewpoint video and 3D-TV

This paper presents a system architecture of an acquisition, compression and rendering system for 3D- TV and free-viewpoint video applications. We show that the proposed system yields two distinct advantages. First, it achieves an efficient compression of 3D/multi-view video by extending a standard H.264 encoder such that near backward compatibility is retained. Second, the proposed system can efficiently compress both 3D-TV and free- viewpoint multi-view video datasets using the single proposed system architecture.

Fast camera calibration for the analysis of sport sequences

Semantic analysis of sport sequences requires camera calibration to obtain player and ball positions in real-world coordinates. For court sports like tennis, the marker lines on the field can be used to determine the calibration parameters. We propose a real-time calibration algorithm that can be applied to all court sports simply by exchanging the court model. The algorithm is based on (1) a specialized court-line detector, (2) a RANSAC-based line parameter estimation, (3) a combinatorial optimization step to localize the court within the set of detected line segments, and (4) an iterative court-model tracking step. Our results show real-time calibration of, e.g., tennis and soccer sequences with a computation time of only about 6 ms per frame.

Current and Emerging Topics in Sports Video Processing

Sports video processing is an interesting topic for research, since the clearly defined game rules in sports provide the rich domain knowledge for analysis. Moreover, it is interesting because many specialized applications for sports video processing are emerging. This paper gives an overview of sports video research, where we describe both basic algorithmic techniques and applications.

Video-object segmentation using multi-sprite background subtraction

The background subtraction algorithm is a frequently-used object segmentation technique because of its algorithmic simplicity. However, we show that, for general rotational camera-motion, it is impractical, or even impossible, to use a single background image. As a solution, we propose to use multi-sprite backgrounds which enables processing of arbitrary rotational camera-motion. The paper describes a complete video-object segmentation system employing multi-sprites. The system generates object masks and background sprites that are compatible with MPEG-4 object-oriented video-coding tools. Good segmentation results are also obtained for sequences which cannot be processed with ordinary background images.

Multiview depth-image compression using an extended H.264 encoder

This paper presents a predictive-coding algorithm for the compression of multiple depth-sequences obtained from a multicamera acquisition setup. The proposed depth-prediction algorithm works by synthesizing a virtual depth-image that matches the depth-image (of the predicted camera). To generate this virtual depth-image, we use an image-rendering algorithm known as 3D image-warping. This newly proposed prediction technique is employed in a 3D coding system in order to compress multiview depth-sequences. For this purpose, we introduce an extended H.264 encoder that employs two prediction techniques: a block-based motion prediction and the previously mentioned 3D image-warping prediction. This extended H.264 encoder adaptively selects the most efficient prediction scheme for each image-block using a rate-distortion criterion. We present experimental results for several multiview depth-sequences, which show a quality improvement of about 2.5 dB as compared to H.264 inter-coded depth-images.