Matteo Colaianni

Real-time pixel luminance optimization for dynamic multi-projection mapping

Using projection mapping enables us to bring virtual worlds into shared physical spaces. In this paper, we present a novel, adaptable and real-time projection mapping system, which supports multiple projectors and high quality rendering of dynamic content on surfaces of complex geometrical shape. Our system allows for smooth blending across multiple projectors using a new optimization framework that simulates the diffuse direct light transport of the physical world to continuously adapt the color output of each projector pixel. We present a real-time solution to this optimization problem using off-the-shelf graphics hardware, depth cameras and projectors. Our approach enables us to move projectors, depth camera or objects while maintaining the correct illumination, in realtime, without the need for markers on the object. It also allows for projectors to be removed or dynamically added, and provides compelling results with only commodity hardware.

Interactive model-based reconstruction of the human head using an RGB-D sensor

We present a novel method for the interactive markerless reconstruction of human heads using a single commodity RGB‐D sensor. Our entire reconstruction pipeline is implemented on the graphics processing unit and allows to obtain high‐quality reconstructions of the human head using an interactive and intuitive reconstruction paradigm. The core of our method is a fast graphics processing unit‐based nonlinear quasi‐Newton solver that allows us to leverage all information of the RGB‐D stream and fit a statistical head model to the observations at interactive frame rates. By jointly solving for shape, albedo and illumination parameters, we are able to reconstruct high‐quality models including illumination corrected textures. All obtained reconstructions have a common topology and can be directly used as assets for games, films and various virtual reality applications. We show motion retargeting, retexturing and relighting examples. The accuracy of the presented algorithm is evaluated by a comparison against ground truth data. Copyright

Interactive Painting and Lighting in Dynamic Multi-Projection Mapping

Digital Art and Design is a major part of our culture. Creating assets in a completely digital environment is well established and understood. Currently we see a rise in mixed-reality applications that aim to combine the traditional real-world based way of working with their digital counterparts. In this paper we present a non-destructive, immersive, fully dynamic mixed-reality painting system for real-world objects that combines the workflow of a traditional airbrush artist with the power of digital media. With this system we can completely alter the appearance of any Lambertian object.

Volume Extraction from Body Scans for Bra Sizing

We present a novel method for bra sizing based on surface scan data. While the current standard of finding the proper size for brassieres is based on only two 1-dimensional measurements, our approach takes the entire shape information into account. We propose to use the breast’s volume as a good approximation for that shape. To compute the breast volume we introduce a robust and automatic algorithm based on measuring the volume difference of a torso with and without the breasts. Finally, we compare our novel sizing strategy to the traditional sizing of bras. We will show a better distribution across the population of test-subjects. Using the surface scan data in combination with our volume based sizing approach we can furthermore generate mean body shapes for every bra size which is of great benefit for bra development.

A Pose Invariant Statistical Shape Model for Human Bodies

We present a complete pipeline for constructing a statistical shape model that is invariant to deviations in the scan pose while encoding the space of human pose and body shape in an efficient manner. A dense cross-parameterization between a large set of high-quality 3D scans is computed using a fast and robust volume aware non-rigid registration method. Our approach uses a novel encoding that automatically decorrelates shape and pose leading to a statistical model that is oblivious under transformations induced by pose. This allows us to efficiently compensate pose variations in captured input data leading to a compact representation for pose as well as body shape. We present a local as well as a global skeletal encoding and compare both approaches. Finally, we analyze the generalization properties and accuracy of our approach against two state-of-the-art methods. We apply our model to the data clustering problem and use it as a prior for non-rigid shape matching.

Anisotropic deformation for local shape control

We present a novel approach to mesh deformation that enables simple context sensitive manipulation of 3D geometry. The method is based on locally anisotropic transformations and is extended to global control directions. This allows intuitive directional modeling within an easy to implement framework. The proposed method complements current sculpting paradigms by providing further possibilities for intuitive surface-based editing without the need for additional host geometries. We show the anisotropic deformation to be seamlessly transferable to free boundary parameterization methods, which allows us to solve the hard problem of flattening compression garments in the domain of apparel design.

Adaptive stray-light compensation in dynamic multi-projection mapping

Projection-based mixed reality is an effective tool to create immersive visualizations on real-world objects. Its wide range of applications includes art installations, education, stage shows, and advertising. In this work, we enhance a multi-projector system for dynamic projection mapping by handling various physical stray-light effects: interreflection, projector black-level, and environmental light in real time for dynamic scenes. We show how all these effects can be efficiently simulated and accounted for at runtime, resulting in significantly improved projection mapping results. By adding a global optimization step, we can further increase the dynamic range of the projection.

Anisotropic surface based deformation

We present a novel approach to mesh deformation that enables simple context sensitive manipulation of 3D geometry. The method is based on locally anisotropic scaling. This allows an intuitive directional modeling within an easy to implement framework. The proposed method ideally complements current intuitive sculpting paradigms by further possibilities of surface based editing without the need of additional host geometries. We also show the anisotropy to be seamlessly transferable to free boundary parameterization methods, which allows to solve the hard problem of flattening compressive garments in the domain of apparel development.

Stray-light compensation in dynamic projection mapping

Projection based mixed-reality is an effective tool to create immersive visualizations on real-world objects. This is used in a wide range of applications like art-installations, education, stage shows and advertising. In this work, we enhance a multi-projector system for dynamic projection mapping, by handling various physical stray-light effects: interreflection, projector black-level and environment light in real-time for dynamic scenes. We show how all these effects can be efficiently simulated and accounted for at run time, resulting in significantly improved projection mapping results.

Reality forge: interactive dynamic multi-projection mapping

Creating digitals assets, especially surface textures is a tedious, and often unintuitive task. We built a fun to use, interactive projection-mapping system that aims to simplify this process.