Roland Ruiters

Integrated high-quality acquisition of geometry and appearance for cultural heritage

Current research trends demonstrate that, for a wide range of applications in cultural heritage, 3D shape acquisition alone is not sufficient. To generate a digital replica of a real world object the digitized geometric models have to be complemented with information pertaining to optical properties of the object surface. We therefore propose an integrated system for acquiring both the 3D shape and reflectance properties necessary for obtaining a photo-realistic digital replica. The proposed method is suitable for the digitization of objects showing the complex reflectance behavior, for example specularities and meso-scale interreflections, often encountered in the field of cultural heritage. We demonstrate the performance of our system with four challenging examples. By using Bidirectional Texture Functions, our structured light based approach is able to achieve good geometric precision while preserving tiny details such as scratches and engravings.

BTF compression via sparse tensor decomposition

In this paper, we present a novel compression technique for Bidirectional Texture Functions based on a sparse tensor decomposition. We apply the K‐SVD algorithm along two different modes of a tensor to decompose it into a small dictionary and two sparse tensors. This representation is very compact, allowing for considerably better compression ratios at the same RMS error than possible with current compression techniques like PCA, N‐mode SVD and Per Cluster Factorization. In contrast to other tensor decomposition based techniques, the use of a sparse representation achieves a rendering performance that is at high compression ratios similar to PCA based methods.

A Multi-camera, Multi-projector Super-Resolution Framework for Structured Light

In this work, we present a framework for multi-camera, multi-projector object acquisition based on structured light. This approach allows the reconstruction of an object without moving either the object or the acquisition setup, avoiding any registration of independent measurements. To overcome the resolution limitations of the individual projectors, we introduce a novel super-resolution scheme. By exploiting high dynamic range imaging, we are able to handle even complicated objects, exhibiting strong specularities. We show that, combined with an iterated bundle adjustment, these improvements increase the accuracy of the obtained point cloud.

Multi-view Normal Field Integration for 3D Reconstruction of Mirroring Objects

In this paper, we present a novel, robust multi-view normal field integration technique for reconstructing the full 3D shape of mirroring objects. We employ a turntable-based setup with several cameras and displays. These are used to display illumination patterns which are reflected by the object surface. The pattern information observed in the cameras enables the calculation of individual volumetric normal fields for each combination of camera, display and turntable angle. As the pattern information might be blurred depending on the surface curvature or due to non-perfect mirroring surface characteristics, we locally adapt the decoding to the finest still resolvable pattern resolution. In complex real-world scenarios, the normal fields contain regions without observations due to occlusions and outliers due to interreflections and noise. Therefore, a robust reconstruction using only normal information is challenging. Via a non-parametric clustering of normal hypotheses derived for each point in the scene, we obtain both the most likely local surface normal and a local surface consistency estimate. This information is utilized in an iterative min-cut based variational approach to reconstruct the surface geometry.

Patch-based texture interpolation

In this paper, we present a novel exemplar‐based technique for the interpolation between two textures that combines patch‐based and statistical approaches. Motivated by the notion of texture as a largely local phenomenon, we warp and blend small image neighborhoods prior to patch‐based texture synthesis. In addition, interpolating and enforcing characteristic image statistics faithfully handles high frequency detail. We are able to create both intermediate textures as well as continuous transitions. In contrast to previous techniques computing a global morphing transformation on the entire input exemplar images, our localized and patch‐based approach allows us to successfully interpolate between textures with considerable differences in feature topology for which no smooth global warping field exists.

Fusing Structured Light Consistency and Helmholtz Normals for 3D Reconstruction.

In this paper, we propose a 3D reconstruction approach which combines a structured light based consistency measure with dense normal information obtained by exploiting the Helmholtz reciprocity principle. This combination compensates for the individual limitations of techniques providing normal information, which are mainly affected by low-frequency drift, and those providing positional information, which are often not well-suited to recover fine details. To obtain Helmholtz reciprocal samples, we employ a turntable-based setup. Due to the reciprocity, the structured light directly provides the occlusion information needed during the normal estimation for both the cameras and light sources. We perform the reconstruction by solving one global variational problem which integrates all available measurements simultaneously, over all cameras, light source positions and turntable rotations. For this, we employ an octree-based continuous min-cut framework in order to alleviate metrification errors while maintaining memory efficiency. We evaluate the performance of our algorithm both on synthetic and real-world data.

WebGL-based streaming and presentation framework for bidirectional texture functions

Museums and Cultural Heritage institutions have a growing interest in presenting their collections to a broader community via the Internet. The photo-realistic presentation of interactively inspectable digital 3D replicas of artifacts is one of the most challenging problems in this field. For this purpose, we seek not only a 3D geometry but also a powerful material representation capable of reproducing the full visual appeal of an object. In this paper, we propose a WebGL-based presentation framework in which reflectance information is represented via Bidirectional Texture Functions. Our approach works out-of-the-box in modern web browsers and allows for the progressive transmission and interactive rendering of digitized artifacts consisting of 3D geometry and reflectance information. We handle the huge amount of data needed for this representation by employing a novel progressive streaming approach for BTFs which allows for the smooth interactive inspection of a steadily improving version during the download.

Heightfield and spatially varying BRDF Reconstruction for Materials with Interreflections

Photo‐realistic reproduction of material appearance from images has widespread use in applications ranging from movies over advertising to virtual prototyping. A common approach to this task is to reconstruct the small scale geometry of the sample and to capture the reflectance properties using spatially varying BRDFs. For this, multi‐view and photometric stereo reconstruction can be used, both of which are limited regarding the amount of either view or light directions and suffer from either low‐ or high‐frequency artifacts, respectively. In this paper, we propose a new algorithm combining both techniques to recover heightfields and spatially varying BRDFs while at the same time overcoming the above mentioned drawbacks. Our main contribution is a novel objective function which allows for the reconstruction of a heightfield and high quality SVBRDF including view dependent effects. Thereby, our method also avoids both low and high frequency artifacts. Additionally, our algorithm takes inter‐reflections into account allowing for the reconstruction of undisturbed representations of the underlying material. In our experiments, including synthetic and real‐world data, we show that our approach is superior to state‐of‐the‐art methods regarding reconstruction error as well as visual impression. Both the reconstructed geometry and the recovered SVBRDF are highly accurate, resulting in a faithful reproduction of the materials characteristic appearance, which is of paramount importance in the context of material rendering.

Example-based Interpolation and Synthesis of Bidirectional Texture Functions

Bidirectional Texture Functions (BTF) have proven to be a well‐suited representation for the reproduction of measured real‐world surface appearance and provide a high degree of realism. We present an approach for designing novel materials by interpolating between several measured BTFs. For this purpose, we transfer concepts from existing texture interpolation methods to the much more complex case of material interpolation. We employ a separation of the BTF into a heightmap and a parallax compensated BTF to cope with problems induced by parallax, masking and shadowing within the material. By working only on the factorized representation of the parallax compensated BTF and the heightmap, it is possible to efficiently perform the material interpolation. By this novel method to mix existing BTFs, we are able to design plausible and realistic intermediate materials for a large range of different opaque material classes. Furthermore, it allows for the synthesis of tileable and seamless BTFs and finally even the generation of gradually changing materials following user specified material distribution maps.

Data Driven Surface Reflectance from Sparse and Irregular Samples

In recent years, measuring surface reflectance has become an established method for high quality renderings. In this context, especially non‐parametric representations got a lot of attention as they allow for a very accurate representation of complex reflectance behavior. However, the acquisition of this data is a challenging task especially if complex object geometry is involved. Capturing images of the object under varying illumination and view conditions results in irregular angular samplings of the reflectance function with a limited angular resolution. Classical data‐driven techniques, like tensor factorization, are not well suited for such data sets as they require a resampling of the high dimensional measurement data to a regular grid. This grid has to be on a much higher angular resolution to avoid resampling artifacts which in turn would lead to data sets of enormous size. To overcome these problems we introduce a novel, compact data‐driven representation of reflectance functions based on a sum of separable functions which are fitted directly to the irregular set of data without any further resampling. The representation allows for efficient rendering and is also well suited for GPU applications. By exploiting spatial coherence of the reflectance function over the object a very precise reconstruction even of specular materials becomes possible already with a sparse input sampling. This would be impossible using standard data interpolation techniques. Since our algorithm exclusively operates on the compressed representation, it is both efficient in terms of memory use and computational complexity, depending only sub‐linearly on the size of the fully tabulated data. The quality of the reflectance function is evaluated on synthetic data sets as ground truth as well as on real world measurements.