Frank Verbiest

Visual Modeling with a Hand-Held Camera

In this paper a complete system to build visual models from camera images is presented. The system can deal with uncalibrated image sequences acquired with a hand-held camera. Based on tracked or matched features the relations between multiple views are computed. From this both the structure of the scene and the motion of the camera are retrieved. The ambiguity on the reconstruction is restricted from projective to metric through self-calibration. A flexible multi-view stereo matching scheme is used to obtain a dense estimation of the surface geometry. From the computed data different types of visual models are constructed. Besides the traditional geometry- and image-based approaches, a combined approach with view-dependent geometry and texture is presented. As an application fusion of real and virtual scenes is also shown.

Surviving Dominant Planes in Uncalibrated Structure and Motion Recovery

In this paper we address the problem of uncalibrated structure and motion recovery from image sequences that contain dominant planes in some of the views. Traditional approaches fail when the features common to three consecutive views are all located on a plane. This happens because in the uncalibrated case there is a fundamental ambiguity in relating the structure before and after the plane. This is, however, a situation that is often hard to avoid in man-made environments. We propose a complete approach that detects the problem and defers the computation of parameters that are ambiguous in projective space (i.e. the registration between partial reconstructions only sharing a common plane and poses of cameras only seeing planar features) till after self-calibration. Also a new linear self-calibration algorithm is proposed that couples the intrinsics between multiple subsequences. The final result is a complete metric 3D reconstruction of both structure and motion for the whole sequence. Experimental results on real image sequences show that the approach yields very good results.

3D recording for archaeological fieldwork

Until recently, archaeologists have had limited 3D recording options because of the complexity and expense of the necessary recording equipment. We outline a system that helps archaeologists acquire 3D models without using equipment more complex or delicate than a standard digital camera.

Drift detection and removal for sequential structure from motion algorithms

In sequential Structure from Motion algorithms for extended image or video sequences, error build up caused by drift poses a problem as feature tracks that normally represent a single scene point will have distinct 3D reconstructions. For the final bundle adjustment to remove this drift, it must be told about these 3D-3D correspondences through a change in the cost function. However, as a bundle adjustment is a nonlinear optimization technique, the drift needs to be removed from the supplied initial solution to allow for convergence of the bundle adjustment to the real global optimum. Before drift can be removed, it has to be detected. This is accomplished through understanding of the long term behavior of drift which leaves 3D reconstructions from short sequences intact. Drift detection boils down to identifying reconstructions of the same scene part that only differ up to a projective transformation. After detection, the drift can be removed from future processed images and an Adapted Bundle Adjustment using correspondences supplied by the drift detection can remove the drift from previous images. Several experiments on real video sequences demonstrate the merit of drift detection and removal.

Image-based 3D acquisition of archaeological heritage and applications

In this paper an approach is presented that obtains virtual models from sequences of images. The system can deal with uncalibrated image sequences acquired with a hand-held camera. Based on tracked or matched features the relations between multiple views are computed. From this both the structure of the scene and the motion of the camera are retrieved. The ambiguity on the reconstruction is restricted from projective to metric through auto-calibration. A flexible multi-view stereo matching scheme is used to obtain a dense estimation of the surface geometry. From the computed data virtual models can be constructed or, inversely, virtual models can be included in the original images.

Photometric stereo with coherent outlier handling and confidence estimation

In photometric stereo a robust method is required to deal with outliers, such as shadows and non-Lambertian reflections. In this paper we rely on a probabilistic imaging model that distinguishes between inliers and outliers, and formulate the problem as a Maximum-Likelihood estimation problem. To signal which imaging model to use a hidden binary inlier map is introduced, which, to account for the fact that inlier/outlier pixels typically group together, is modelled as a Markov Random Field. To make inference of model parameters and hidden variables tractable a mean field Expectation-Maximization (EM) algorithm is used. If for each pixel we add the scaled normal, i.e. albedo and normal combined, to the model parameters, it would not be possible to obtain a confidence estimate in the result. Instead, each scaled normal is added as a hidden variable, the distribution of which, approximated by a Gaussian, is also estimated in the EM algorithm. The covariance matrix of the recovered approximate Gaussian distribution serves as a confidence estimate of the scaled normal. We demonstrate experimentally the effectiveness or our approach.

As time flies by: mixed image and model-based rendering of an historical landscape from helicopter images

For the preservation of cultural heritage to be successful the general public must be able to experience sites and reconstructions in an intuitive, yet convincingly real way. In this paper, a pipeline is discussed that can be employed to generate an interactive presentation of landscape reconstruction through a Quicktime VR object movie. Images of the current landscape are registered with virtual reconstructions of the same landscape through time.

Image-based rendering for photo-realistic visualization

Digital models of objects have long stood synonymous for 3D models with fixed textures. More recently, image-based rendering has made its way to offer an effective alternative. Rather than producing a 3D model, views are created by interpolating between images taken from many different viewpoints. The advantages are that the resulting visualizations look very realistic and that a wider range of objects can be dealt with. One can go a step further and follow a similar strategy for the illumination direction: for every reference viewpoint, now take multiple images, each with a different illumination. Again, images for novel illumination directions follow from interpolation. But there is an obvious catch in that many, many images are needed, unless at least a crude 3D shape model and preferably also a model for the surface reflectance are used to support the interpolation from more sparsely taken images. Even then, the number of images needed remains appreciable. Hence, we present a dome to efficiently capture such images. Fortunately, it is possible to generate this crude 3D model, as well as the surface reflectance characteristics’ directly from the images. We describe methods to achieve this. Also, in order to make real-time visualization possible, all critical steps of the visualization pipeline are programmed on off-the-shelf graphics hardware. The dome provides an easy to use and structured acquisition procedure. Yet, the applicability of the algorithms is not limited to structured input. The images could, for example be taken with a hand-held camera.

Real-time image based rendering from uncalibrated images

We present a novel real-time image-based rendering system for generating realistic novel views of complex scenes from a set of uncalibrated images. A combination of structure-and-motion and stereo techniques is used to obtain calibrated cameras and dense depth maps for all recorded images. These depth maps are converted into restrictive quadtrees, which allow for adaptive, view-dependent tessellations while storing per-vertex quality. When rendering a novel view, a subset of suitable cameras is selected based upon a ranking criterion. In the spirit of the unstructured lumigraph rendering approach a blending field is evaluated, although the implementation is adapted on several points. We alleviate the need for the creation of a geometric proxy for each novel view while the camera blending field is sampled in a more optimal, non-uniform way and combined with the per-vertex quality to reduce texture artifacts. In order to make real-time visualization possible, all critical steps of the visualization pipeline are programmed in a highly optimized way on commodity graphics hardware using the OpenGL Shading Language. The proposed system can handle complex scenes such as large outdoor scenes as well as small objects with a large number of acquired images.

Virtual Models from Video and Vice-Versa

In this paper an approach is presented that obtains virtual models from sequences of images. The system can deal with uncalibrated image sequences acquired with a hand-held camera. Based on tracked or matched features the relations between multiple views are computed. From this both the structure of the scene and the motion of the camera are retrieved. The ambiguity on the reconstruction is restricted from projective to metric through auto-calibration. A flexible multi-view stereo matching scheme is used to obtain a dense estimation of the surface geometry. From the computed data virtual models can be constructed or, inversely, virtual models can be included in the original images.