Christopher Schwartz

Integrated high-quality acquisition of geometry and appearance for cultural heritage

Current research trends demonstrate that, for a wide range of applications in cultural heritage, 3D shape acquisition alone is not sufficient. To generate a digital replica of a real world object the digitized geometric models have to be complemented with information pertaining to optical properties of the object surface. We therefore propose an integrated system for acquiring both the 3D shape and reflectance properties necessary for obtaining a photo-realistic digital replica. The proposed method is suitable for the digitization of objects showing the complex reflectance behavior, for example specularities and meso-scale interreflections, often encountered in the field of cultural heritage. We demonstrate the performance of our system with four challenging examples. By using Bidirectional Texture Functions, our structured light based approach is able to achieve good geometric precision while preserving tiny details such as scratches and engravings.

Design and Implementation of Practical Bidirectional Texture Function Measurement Devices Focusing on the Developments at the University of Bonn

Understanding as well as realistic reproduction of the appearance of materials play an important role in computer graphics, computer vision and industry. They enable applications such as digital material design, virtual prototyping and faithful virtual surrogates for entertainment, marketing, education or cultural heritage documentation. A particularly fruitful way to obtain the digital appearance is the acquisition of reflectance from real-world material samples. Therefore, a great variety of devices to perform this task has been proposed. In this work, we investigate their practical usefulness. We first idey a set of necessary attributes and establish a general categorization of different designs that have been realized. Subsequently, we provide an in-depth discussion of three particular implementations by our work group, demonstrating advantages and disadvantages of different system designs with respect to the previously established attributes. Finally, we survey the existing literature to compare our implementation with related approaches.

A Multi-camera, Multi-projector Super-Resolution Framework for Structured Light

In this work, we present a framework for multi-camera, multi-projector object acquisition based on structured light. This approach allows the reconstruction of an object without moving either the object or the acquisition setup, avoiding any registration of independent measurements. To overcome the resolution limitations of the individual projectors, we introduce a novel super-resolution scheme. By exploiting high dynamic range imaging, we are able to handle even complicated objects, exhibiting strong specularities. We show that, combined with an iterated bundle adjustment, these improvements increase the accuracy of the obtained point cloud.

Fusing Structured Light Consistency and Helmholtz Normals for 3D Reconstruction.

In this paper, we propose a 3D reconstruction approach which combines a structured light based consistency measure with dense normal information obtained by exploiting the Helmholtz reciprocity principle. This combination compensates for the individual limitations of techniques providing normal information, which are mainly affected by low-frequency drift, and those providing positional information, which are often not well-suited to recover fine details. To obtain Helmholtz reciprocal samples, we employ a turntable-based setup. Due to the reciprocity, the structured light directly provides the occlusion information needed during the normal estimation for both the cameras and light sources. We perform the reconstruction by solving one global variational problem which integrates all available measurements simultaneously, over all cameras, light source positions and turntable rotations. For this, we employ an octree-based continuous min-cut framework in order to alleviate metrification errors while maintaining memory efficiency. We evaluate the performance of our algorithm both on synthetic and real-world data.

DOME II: a parallelized BTF acquisition system

Bidirectional Texture Functions (BTFs) provide a realistic depiction of the appearance of many real-world materials as they contain the spatially varying light scattering behavior of the material surface. Since editing of existing BTF data is still in its early stages, materials have to be measured from real-world samples. In contrast to the related Spatially Varying BRDFs (SVBRDFs), the reflectance information encoded in a BTF also includes non-local scattering effects and therefore does not obey energy conservation or reciprocity. While this higher degree of freedom also contributes to an increased realism, it inadvertently calls for an extensive measurement of reflectance samples, as many regularization approaches from BRDF measurement do not apply. In this paper, we present an automated, parallelized, robust, fast and transportable setup for the acquisition of BTFs from flat samples as well as 3D objects using camera and light arrays: the DOME II. In contrast to previous camera array approaches, the present setup, which is comprised of high-quality industry grade components, overcomes several issues regarding stability, reliability and precision. It achieves a well balanced state-of-the-art acquisition performance in terms of speed and quality at reasonable costs.

WebGL-based streaming and presentation framework for bidirectional texture functions

Museums and Cultural Heritage institutions have a growing interest in presenting their collections to a broader community via the Internet. The photo-realistic presentation of interactively inspectable digital 3D replicas of artifacts is one of the most challenging problems in this field. For this purpose, we seek not only a 3D geometry but also a powerful material representation capable of reproducing the full visual appeal of an object. In this paper, we propose a WebGL-based presentation framework in which reflectance information is represented via Bidirectional Texture Functions. Our approach works out-of-the-box in modern web browsers and allows for the progressive transmission and interactive rendering of digitized artifacts consisting of 3D geometry and reflectance information. We handle the huge amount of data needed for this representation by employing a novel progressive streaming approach for BTFs which allows for the smooth interactive inspection of a steadily improving version during the download.

Effective Visualization of Short Routes

In this work we develop a new alternative to conventional maps for visualization of relatively short paths as they are frequently encountered in hotels, resorts or museums. Our approach is based on a warped rendering of a 3D model of the environment such that the visualized path appears to be straight even though it may contain several junctions. This has the advantage that the beholder of the image gains a realistic impression of the surroundings along the way which makes it easy to retrace the route in practice. We give an intuitive method for generation of such images and present results from user studies undertaken to evaluate the benefit of the warped images for orientation in unknown environments.

Example-based Interpolation and Synthesis of Bidirectional Texture Functions

Bidirectional Texture Functions (BTF) have proven to be a well‐suited representation for the reproduction of measured real‐world surface appearance and provide a high degree of realism. We present an approach for designing novel materials by interpolating between several measured BTFs. For this purpose, we transfer concepts from existing texture interpolation methods to the much more complex case of material interpolation. We employ a separation of the BTF into a heightmap and a parallax compensated BTF to cope with problems induced by parallax, masking and shadowing within the material. By working only on the factorized representation of the parallax compensated BTF and the heightmap, it is possible to efficiently perform the material interpolation. By this novel method to mix existing BTFs, we are able to design plausible and realistic intermediate materials for a large range of different opaque material classes. Furthermore, it allows for the synthesis of tileable and seamless BTFs and finally even the generation of gradually changing materials following user specified material distribution maps.

Data Driven Surface Reflectance from Sparse and Irregular Samples

In recent years, measuring surface reflectance has become an established method for high quality renderings. In this context, especially non‐parametric representations got a lot of attention as they allow for a very accurate representation of complex reflectance behavior. However, the acquisition of this data is a challenging task especially if complex object geometry is involved. Capturing images of the object under varying illumination and view conditions results in irregular angular samplings of the reflectance function with a limited angular resolution. Classical data‐driven techniques, like tensor factorization, are not well suited for such data sets as they require a resampling of the high dimensional measurement data to a regular grid. This grid has to be on a much higher angular resolution to avoid resampling artifacts which in turn would lead to data sets of enormous size. To overcome these problems we introduce a novel, compact data‐driven representation of reflectance functions based on a sum of separable functions which are fitted directly to the irregular set of data without any further resampling. The representation allows for efficient rendering and is also well suited for GPU applications. By exploiting spatial coherence of the reflectance function over the object a very precise reconstruction even of specular materials becomes possible already with a sparse input sampling. This would be impossible using standard data interpolation techniques. Since our algorithm exclusively operates on the compressed representation, it is both efficient in terms of memory use and computational complexity, depending only sub‐linearly on the size of the fully tabulated data. The quality of the reflectance function is evaluated on synthetic data sets as ground truth as well as on real world measurements.

WebGL-based streaming and presentation of objects with bidirectional texture functions

Museums and Cultural Heritage institutions have a growing interest in presenting their collections to a broader community via the Internet. The photo-realistic presentation of interactively inspectable virtual surrogates is one of the most challenging problems in this field. For this purpose, we seek to employ not only a 3D geometry but also a powerful material representation capable of reproducing the full visual appeal of an object. In this article, we propose a WebGL-based presentation framework in which reflectance information is represented via Bidirectional Texture Functions (BTF). Our approach works out-of-the-box in modern Web browsers and allows for the progressive transmission and interactive rendering of digitized artifacts consisting of 3D geometry and reflectance information. We handle the huge amount of data needed for this representation by employing a novel progressive streaming approach for BTFs, which allows for the smooth interactive inspection of a steadily improving version during the download. We demonstrate an interesting use-case of this technique at a cross section of Cultural Heritage, medical education, and research and provide an evaluation of the capabilities of our framework in the scope of BTF compression and transmission.