André Redert

Advanced three-dimensional television system technologies

We describe the goals of the ATTEST project, which started in March 2002 as part of the Information Society Technologies (IST) programme, sponsored by the European Commission. In the 2-year project, several industrial and academic partners cooperate towards a flexible, 2D-compatible and commercially feasible 3D-TV system-for broadcast environments. An entire 3D-video chain will be developed. We discuss the goals for content creation, coding, transmission, display and the central role that human 3D perception research will play in optimizing the entire chain. The goals include the development of a new 3D camera, algorithms to convert existing 2D-video material into 3D, a 2D-compatible coding and transmission scheme for 3D video using MPEG-2/4/7, and two new autostereoscopic displays. With the combination of industrial and academic partners and the technological progress obtained from earlier 3D projects, we expect to achieve the ATTEST goal of developing the first commercially feasible European 3D-TV broadcast system.

Stages as Models of Scene Geometry

Reconstruction of 3D scene geometry is an important element for scene understanding, autonomous vehicle and robot navigation, image retrieval, and 3D television. We propose accounting for the inherent structure of the visual world when trying to solve the scene reconstruction problem. Consequently, we identify geometric scene categorization as the first step toward robust and efficient depth estimation from single images. We introduce 15 typical 3D scene geometries called stages, each with a unique depth profile, which roughly correspond to a large majority of broadcast video frames. Stage information serves as a first approximation of global depth, narrowing down the search space in depth estimation and object localization. We propose different sets of low-level features for depth estimation, and perform stage classification on two diverse data sets of television broadcasts. Classification results demonstrate that stages can often be efficiently learned from low-dimensional image representations.

Depth Information by Stage Classification

Recently, methods for estimating 3D scene geometry or absolute scene depth information from 2D image content have been proposed. However, general applicability of these methods in depth estimation may not be realizable, as inconsistencies may be introduced due to a large variety of possible pictorial content. We identify scene categorization as the first step towards efficient and robust depth estimation from single images. To that end, we describe a limited number of typical 3D scene geometries, called stages, each having a unique depth pattern and thus providing a specific context for stage objects. This type of scene information narrows down the possibilities with respect to individual objects locations, scales and identities. We show how these stage types can be efficiently learned and how they can lead to robust extraction of depth information. Our results indicate that stages without much variation and object clutter can be detected robustly, with up to 60% success rate.

Philips 3D Solutions: From Content Creation to Visualization

Philips is realizing an end-to-end 3D display solution from 3D content creation to visualization. This development fits in our long-standing tradition of combining expertise in video processing with our strength in display development to create the most exciting and best viewing experience. Philips developed several high-quality 3D displays, ranging in resolution, viewing angle, depth experience, and sizes from 4 to 40 and up. Backwards compatibility with 2D content is enabled via signal processing or opto-electronic 3D & 2D dual mode displays. Content creation and conversion methods are provided, which are a key factor for the success of 3D displays. Fully automatic conversion from monoscopic 2D content into 3D enables the re-use of all existing 2D video material. Further methods enable 3D animation/design, 2D to 3D conversion in post- production and live capture of new 3D content. Our efforts in MPEG standardization towards the 2D-plus-depth format for 3D video enables a flexible interface between the variety in 3D content creation methods and the range in 3D displays. Furthermore, the 3D format is compatible with existing 2D content, standards and infrastructure. Currently, Philips offers several commercial 3D products for professional use such as digital signage, and progress is being made towards consumer products such as 3DTV.

Temporal stabilization of video object segmentation for 3D-TV applications

We present a method for improving the temporal stability of video object segmentation algorithms for 3D-TV applications. First, two quantitative measures to evaluate temporal stability without ground-truth are presented. Then, a pseudo-3D curve evolution method, which spatio-temporally stabilizes the estimated object segments is introduced. Temporal stability is achieved by re-distributing existing object segmentation errors such that they are less disturbing when the scene is rendered and viewed in 3D. Our starting point is the hypothesis that if making segmentation errors are inevitable, they should be made in a temporally consistent way for 3D TV applications. This hypothesis is supported by the experiments, which show that there is significant improvement in segmentation quality both in terms of the objective quantitative measures and in terms of the viewing comfort in subjective perceptual tests. This shows that it is possible to increase the object segmentation quality without increasing the actual segmentation accuracy.

Challenges in 3DTV image processing

Philips provides autostereoscopic three-dimensional display systems that will bring the next leap in visual experience, adding true depth to video systems. We identified three challenges specifically for 3D image processing: 1) bandwidth and complexity of 3D images, 2) conversion of 2D to 3D content, and 3) object-based image/depth processing. We discuss these challenges and our solutions via several examples. In conclusion, the solutions have enabled the market introduction of several professional 3D products, and progress is made rapidly towards consumer 3DTV.

Accurate and robust marker localization algorithm for camera calibration

We design a new scheme for the accurate localization of markers in images of calibration plates. This forms the basis of so-called fixed or strong camera calibration methods. We focus on the most widely used plates with circular markers arranged in a regular grid. Our algorithm is fully automatic and very robust. We assess its results by using synthetic images containing noise and other practical image detoriations. Compared with the ground truth of the marker positions, we obtain a localization precision of 0.01 pixel in high-quality images, and 0.03 pixel in very bad quality images (high noise, low contrast, lens distortion, CCD misplacements). With these results, the algorithm outperforms all existing algorithms from the literature.

3-D scene reconstruction with viewpoint adaptation on stereo displays

We propose a generic algorithm for the geometrically correct reconstruction of 3-D scenes on stereo displays with viewpoint adaptation. This forms the basis of multiviewpoint systems, which are currently the most promising candidates for real-time implementations of 3-D visual communication systems. The reconstruction algorithm needs 3-D tracking of the viewers eyes with respect to the display. We analyze the effect of eye-tracking errors. A simple bound is derived, below which reconstruction errors cannot be observed. We design a multiviewpoint system using a recently introduced image-based scene representation. The design formed the basis of the real-time multiviewpoint system that was recently built in the European PANORAMA project. Experiments with both natural and synthetic scenes show that the proposed reconstruction algorithm performs well. The experiments are performed by computer simulations and the real-time PANORAMA system.

Visualization of arbitrary-shaped 3D scenes on depth-limited 3D displays

We propose a depth scaling method that enables visualization of arbitrary-shaped 3D scenes on 3D displays. Most current 3D displays have a depth limitation, while the scene to be displayed has not. The trivial solutions as clipping or linear scaling of the scenes 3D bounding box suffer from nonoptimal utilization of the displays capabilities. Our approach uses spatially adaptive depth scaling that maximizes the perceptual 3D effect. From the original scene geometry, the topology and local depth ordering among objects are preserved, while depth linearity is disregarded. The scaling method applies to nearly all 3D displays, such as glasses-based, head-tracked, multiview, holographic and volumetric 3D displays. Subjective tests with the dynamic dimension display system show that our method significantly increases the perceptual 3D effect.

Depth Estimation Via Stage Classification

We identify scene categorization as the first step towards efficient and robust depth estimation from single images. Categorizing the scene into one of the geometric classes greatly reduces the possibilities in subsequent phases. To that end, we introduce 15 typical 3D scene geometries, called stages, each having a unique depth profile and roughly corresponding to a large majority of all images. In this work, we do not attempt to derive a precise depth map, but only to decide on the appropriate stage. The subsequent phase of parameter estimation would result in a more detailed background depth profile.