Simon Fuhrmann

Fusion of depth maps with multiple scales

Multi-view stereo systems can produce depth maps with large variations in viewing parameters, yielding vastly different sampling rates of the observed surface. We present a new method for surface reconstruction by integrating a set of registered depth maps with dramatically varying sampling rate. The method is based on the construction of a hierarchical signed distance field represented in an incomplete primal octree by incrementally adding triangulated depth maps. Due to the adaptive data structure, our algorithm is able to handle depth maps with varying scale and to consistently represent coarse, low-resolution regions as well as small details contained in high-resolution depth maps. A final surface mesh is extracted from the distance field by construction of a tetrahedral complex from the scattered signed distance values and applying the Marching Tetra-hedra algorithm on the partition. The output is an adaptive triangle mesh that seamlessly connects coarse and highly detailed regions while avoiding filling areas without suitable input data.

Photometric stereo for outdoor webcams

We present a photometric stereo technique that operates on time-lapse sequences captured by static outdoor webcams over the course of several months. Outdoor webcams produce a large set of uncontrolled images subject to varying lighting and weather conditions. We first automatically select a suitable subset of the captured frames for further processing, reducing the dataset size by several orders of magnitude. A camera calibration step is applied to recover the camera response function, the absolute camera orientation, and to compute the light directions for each image. Finally, we describe a new photometric stereo technique for non-Lambertian scenes and unknown light source intensities to recover normal maps and spatially varying materials of the scene.

Ambient point clouds for view interpolation

View interpolation and image-based rendering algorithms often produce visual artifacts in regions where the 3D scene geometry is erroneous, uncertain, or incomplete. We introduce ambient point clouds constructed from colored pixels with uncertain depth, which help reduce these artifacts while providing non-photorealistic background coloring and emphasizing reconstructed 3D geometry. Ambient point clouds are created by randomly sampling colored points along the viewing rays associated with uncertain pixels. Our real-time rendering system combines these with more traditional rigid 3D point clouds and colored surface meshes obtained using multiview stereo. Our resulting system can handle larger-range view transitions with fewer visible artifacts than previous approaches.

MVE: a multi-view reconstruction environment

We present MVE, the Multi-View Environment. MVE is an end-to-end multi-view geometry reconstruction software which takes photos of a scene as input and produces a surface triangle mesh as result. The system covers a structure-from-motion algorithm, multi-view stereo reconstruction, generation of extremely dense point clouds, and reconstruction of surfaces from point clouds. In contrast to most image-based geometry reconstruction approaches, our system is focused on reconstruction of multi-scale scenes, an important aspect in many areas such as cultural heritage. It allows to reconstruct large datasets containing some detailed regions with much higher resolution than the rest of the scene. Our system provides a graphical user interface for structure-from-motion reconstruction, visual inspection of images, depth maps, and rendering of scenes and meshes.

Direct Resampling for Isotropic Surface Remeshing

We present a feature-sensitive remeshing algorithm for relaxation-based methods. The first stage of the algorithm creates a new mesh from scratch by resampling the reference mesh with an exact vertex budget with either uniform or non-uniform vertex distribution according to a density function. The newly introduced samples on the mesh surface are triangulated directly in 3D by constructing a mutual tessellation. The second stage of the algorithm optimizes the positions of the mesh vertices by building a weighted centroidal Voronoi tessellation to obtain a precise isotropic placement of the samples. We achieve isotropy by employing Lloyd’s relaxation method, but other relaxation schemes are applicable. The proposed algorithm handles diverse meshes of arbitrary genus and guarantees that the remeshed model has the same topology as the input mesh. The density function can be defined by the user or derived automatically from the estimated curvature at the mesh vertices. A subset of the mesh edges may be tagged as sharp features to preserve the characteristic appearance of technical models. The new method can be applied to large meshes and produces results faster than previously achievable.

MVE-An image-based reconstruction environment

We present an image-based reconstruction system, the Multi-View Environment. MVE is an end-to-end multi-view geometry reconstruction software which takes photos of a scene as input and produces a textured surface mesh as result. The system covers a structure-from-motion algorithm, multi-view stereo reconstruction, generation of extremely dense point clouds, reconstruction of surfaces from point clouds, and surface texturing. In contrast to most image-based geometry reconstruction approaches, our system is focused on reconstruction of multi-scale scenes, an important aspect in many areas such as cultural heritage. It allows to reconstruct large datasets containing some detailed regions with much higher resolution than the rest of the scene. Our system provides a graphical user interface for visual inspection of the individual steps of the pipeline, i.e., the structure-from-motion result, multi-view stereo depth maps, and rendering of scenes and meshes. Display Omitted HighlightsEnd-to-end multi-view geometry reconstruction and texturing pipeline.Multi-scale reconstruction approach.

Scene Reconstruction from Community Photo Collections

The literally billions of images available from online photo-sharing sites offer an I unprecedented wealth of information but also add additional layers of complexity for reconstruction applications.

Geometric Point Light Source Calibration

We present a light position calibration technique based on a general arrangement of at least two reflective spheres in a single image. Contrary to other techniques we do not directly intersect rays for triangulation but instead solve for the optimal light position by evaluating the image-space error of the light highlights reflected from the spheres. This approach has been very successful in the field of Structure-from-Motion estimation. It has not been applied to light source calibration because determining the reflection point on the sphere to project the highlight back in the image is a challenging problem. We show a solution and define a novel, non-linear error function to recover the position of a point light source. We also introduce a light position estimation that is based on observing the light source directly in multiple images which does not use any reflections. Finally, we evaluate both proposed techniques and the classical ray intersection method in several scenarios with real data.

Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction

The ultimate goal of many image-based modeling systems is to render photo-realistic novel views of a scene without visible artifacts. Existing evaluation metrics and benchmarks focus mainly on the geometric accuracy of the reconstructed model, which is, however, a poor predictor of visual accuracy. Furthermore, using only geometric accuracy by itself does not allow evaluating systems that either lack a geometric scene representation or utilize coarse proxy geometry. Examples include a light field and most image-based rendering systems. We propose a unified evaluation approach based on novel view prediction error that is able to analyze the visual quality of any method that can render novel views from input images. One key advantage of this approach is that it does not require ground truth geometry. This dramatically simplifies the creation of test datasets and benchmarks. It also allows us to evaluate the quality of an unknown scene during the acquisition and reconstruction process, which is useful for acquisition planning. We evaluate our approach on a range of methods, including standard geometry-plus-texture pipelines as well as image-based rendering techniques, compare it to existing geometry-based benchmarks, demonstrate its utility for a range of use cases, and present a new virtual rephotography-based benchmark for image-based modeling and rendering systems.

Automatic construction of statistical shape models for vertebrae

For segmenting complex structures like vertebrae, a priori knowledge by means of statistical shape models (SSMs) is often incorporated. One of the main challenges using SSMs is the solution of the correspondence problem. In this work we present a generic automated approach for solving the correspondence problem for vertebrae. We determine two closed loops on a reference shape and propagate them consistently to the remaining shapes of the training set. Then every shape is cut along these loops and parameterized to a rectangle. There, we optimize a novel combined energy to establish the correspondences and to reduce the unavoidable area and angle distortion. Finally, we present an adaptive resampling method to achieve a good shape representation. A qualitative and quantitative evaluation shows that using our method we can generate SSMs of higher quality than the ICP approach.