Timo Schairer

Fusion of range and color images for denoising and resolution enhancement with a non-local filter

We present an integrated method for post-processing of range data which removes outliers, smoothes the depth values and enhances the lateral resolution in order to achieve visually pleasing 3D models from low-cost depth sensors with additional (registered) color images. The algorithm is based on the non-local principle and adapts the original NL-Means formulation to the characteristics of typical depth data. Explicitly handling outliers in the sensor data, our denoising approach achieves unbiased reconstructions from error-prone input data. Taking intra-patch similarity into account, we reconstruct strong discontinuities without disturbing artifacts and preserve fine detail structures, obtaining piece-wise smooth depth maps. Furthermore, we exploit the dependencies of the depth data with additionally available color information and increase the lateral resolution of the depth maps. We finally discuss how to parallelize the algorithm in order to achieve fast processing times that are adequate for post-processing of data from fast depth sensors such as time-of-flight cameras.

Robust non-local denoising of colored depth data

We give a brief discussion of denoising algorithms for depth data and introduce a novel technique based on the NL-means filter. A unified approach is presented that removes outliers from depth data and accordingly achieves an unbiased smoothing result. This robust denoising algorithm takes intra-patch similarity and optional color information into account in order to handle strong discontinuities and to preserve fine detail structure in the data. We achieve fast computation times with a GPU-based implementation. Results using data from a time-of-flight camera system show a significant gain in visual quality.

Realistic Depth Blur for Images with Range Data

We present a system that allows for changing the major camera parameters after the acquisition of an image. Using the high dynamic range composition technique and additional range information captured with a small and low-cost time-of-flight camera, our setup enables us to set the main parameters of a virtual camera system and to compute the resulting image. Hence, the aperture size and shape, exposure time, as well as the focus can be changed in a postprocessing step. Since the depth-of-field computation is sensitive to proper range data, it is essential to process the color and depth data in an integrated manner. We use a non-local filtering approach to denoise and upsample the range data. The same technique is used to infer missing information regarding depth and color which occur due to the parallax between both cameras as well as due to the lens camera model that we use to simulate the depth of field in a physically correct way.

Visual mapping with uncertainty for correspondence-free localization using Gaussian process regression

We present a framework that allows for localization based on very low resolution omnidirectional image data using regression techniques. Previous related methods are constrained to image data labeled with exact position information acquired in the training phase. We relax this constraint and propose to learn local heteroscedastic Gaussian processes by accumulating odometry data which can easily be acquired. The processes are used as a probabilistic map to predict recording positions of newly acquired images by a fusion of the uncertain training data. In contrast to many feature-based approaches, our framework does not rely on any explicit correspondences over images as well as over positions and only imposes very weak assumptions on the type and quality of the image representations.

Normalized Cross-Correlation using SOFT

We present a novel method to compute the Normalized Cross-Correlation (NCC) of spherical signals such as omnidirectional images for the purpose of image alignment, i.e. orientation estimation, or template tracking. Expanding the reference image and the search template in the spherical harmonics basis, we use the Fourier representation of SO(3) as introduced by Kostelec and Rockmore. We show that the normalized version of the cross-correlation function with regard to orientation can be expressed in terms of standard correlation terms and the NCC can therefore be computed efficiently in O (N3 log2(N)). The approach is validated with experiments on real image sequences in a template matching application.

Learning to localize with Gaussian process regression on omnidirectional image data

We present a probabilistic localization and orientation estimation method for mobile agents equipped with omnidirectional vision. In our appearance-based framework, a scene is learned in an offline step by modeling the variation of the image energy in the frequency domain via Gaussian process regression. The metric localization of novel views is then solved by maximizing the joint predictive probability of the Gaussian processes using a particle filter which allows to incorporate a motion model in the prediction step. Based on the position estimate, a synthetic view is generated and used as a reference for the orientation estimation which is also performed in the Fourier space. Using real as well as virtual data, we show that this framework allows for robust localization in 2D and 3D scenes based on very low resolution images and with competitive computational load.

Application of particle filters to vision-based orientation estimation using harmonic analysis

We investigate the application of particle filters to estimate the orientation of a mobile agent based on an omnidirectional video stream. By applying spherical signal analysis to sequences of low resolution input images we perform a real-time estimation of the relative camera rotation. We use normalized cross-correlation computed with a fast frequency domain approach that yields unbiased estimates which can be further processed by a particle filter. We discuss the quaternion representation of the rotational state space and evaluate methods for meaningful averaging. A prototype system is presented and experiments with real image sequences show that robust estimation of the rotational motion is achievable even when the input images are corrupted, e.g. due to occlusions.

Photorealistic Real-Time Visualization of Cultural Heritage: A Case Study of Friedrichsburg Castle in Germany

Figure 1: Real-time rendering of the virtual reconstruction of the Friedrichsburg Castle This paper presents a novel highly immersive and interactive VR (virtual reality) installation targeted on photorealistic real-time visualization. Although applicable to many other scenarios, this work is focused primarily on virtual reconstructions in the context of cultural heritage projects. We address two shortcomings in most of the current virtual reconstructions, namely interactivity and realism. On the one hand many of them are presented either as a movie or using semi-interactive techniques. In both cases the imagery is pre-rendered and therefore the visualization is lacking interactivity. On the other hand interactive real-time presentations often are neither intuitive to navigate nor visually pleasant. We extended a real-time rendering software based on global illumination to adapt to the special needs of the visualization of virtual scenes that stem from the field of cultural heritage. A HDR (high dynamic range) daylight simulation was developed in conjunction with techniques and algorithms to significantly speed up the calculation time and increase the visual quality of the scene. To account for the different lighting situations encountered in the visualization of indoor and outdoor scenes, we developed a high dynamic range rendering pipeline that uses a dynamic tone mapping algorithm similar to human vision. To provide interactive access to the high quality 3D model even for unskilled users, we developed a very intuitive user interface based on a simple touchscreen for navigating the virtual scene. The combination of the real-time presentation of the photorealistic reconstruction and the intuitive navigation interface leads to a highly immersive and interactive VR installation. Since we are currently working on a virtual reconstruction of a Renaissance castle located in southern Germany, we will therefore use this reconstruction as a case study to present the developed features and to prove their relevance and usefulness. The virtual reconstruction is displayed using our VR installation and will be accessible to the public in the State Museum of Hohenzollern by August 2007.

Increased accuracy orientation estimation from omnidirectional images using the spherical Fourier transform

Orientation estimation based on image data is a key technique in many applications and robust estimates are possible in case of omnidirectional images. A very efficient technique is to solve the problem in Fourier space. In this paper we present a fast and simple method to overcome one of the main draw-backs of this approach, namely the large quantization steps. Due to high memory demands, the Fourier-based solution can be computed on low-resolution input only and the resulting rotation estimate is given on an equiangular grid. We estimate the mode of the likelihood density based on the grid values in order to obtain a rotation estimate of increased accuracy. We show results on data captured with a spherical video camera and validate the approach comparing the orientation estimates of the real data to the ground-truth values.

Screen Space Spherical Harmonic Occlusion

Email: sebastian.herholz@googlemail.com schairer@gris.uni-tuebingen.de References Simulating global illumination demands for solving the lighting integral of a surface point. Ambient Occlusion (AO): pro: appealing approximation of local occlusion con: no directional information (static gray shadows) Screen Space Directional Occlusion (SSDO) [Ritschel09]: pro: combination of occlusion sampling and lighting con: no easy integration into existing systems Spherical Harmonic Lighting (SHL) [Green03]: pro: compact representation of directional occlusion information using Spherical Harmonics (SH) con: expensive calculation of SH occlusion function using ray tracing Idea: Use screen space sampling for the calculation of the SH occlusion function to achieve directional lighting e ects per pixel. Performance Comparison