Jan Herling

Advanced self-contained object removal for realizing real-time Diminished Reality in unconstrained environments

While Augmented Reality has always been restricted to adding artificial content to the real environment, Diminished Reality allows for removing real world content. Existing approaches however, either require complex setups or are not applicable in real-time. In this paper we present our approach for removing real-world objects from a live video stream of the users real environment. Our approach is based on a simple setup and neither requires any pre-processing nor any information on the structure and location of the objects to be removed or on their background. Our approach is based on the identification of the objects to be removed combined with an image completion and synthesis algorithm. The performance of our approach is one to two magnitudes better than that of previous work in the area of image completion, providing real-time object cancellation on standard laptop or tablet computers.

PixMix: A real-time approach to high-quality Diminished Reality

Diminished Reality (DR) allows to remove objects from a video stream while preseving a frame to frame coherence. Some approaches apply a pseudo-DR, allowing for the removal of objects only, while their background can be observed by a second camera. Most real DR approaches are highly computational expensive, not even allowing for interactive rates and/or apply significant restrictions regarding the uniformity of the background, or allow linear camera movements or even a static camera only. In this paper we will present a real-time capable Diminished Reality approach for high-quality image manipulation. Our approach achieves a significantly better performance and image quality for almost planar but non-trivial image backgrounds. Our Diminished Reality pipeline provides coherent video streams even for nonlinear camera movements due to the integration of homography based object tracking.

High-Quality Real-Time Video Inpaintingwith PixMix

While image inpainting has recently become widely available in image manipulation tools, existing approaches to video inpainting typically do not even achieve interactive frame rates yet as they are highly computationally expensive. Further, they either apply severe restrictions on the movement of the camera or do not provide a high-quality coherent video stream. In this paper we will present our approach to high-quality real-time capable image and video inpainting. Our PixMix approach even allows for the manipulation of live video streams, providing the basis for real Diminished Reality (DR) applications. We will show how our approach generates coherent video streams dealing with quite heterogeneous background environments and non-trivial camera movements, even applying constraints in real-time.

Interactive Bits: Prototyping of Mixed Reality Applications and Interaction Techniques through Visual Programming

The development of mixed reality (MR) applications typically requires a substantial amount of effort and programming skills. This is due to the fact that no standard interfaces and devices exist. Therefore, user interface elements and interaction techniques typically have to be implemented individually. In this paper we present our approach for supporting the development of mixed reality applications and interaction techniques using the concept of interactive bits. We developed this via a component-based approach allowing for modeling interaction techniques, object behaviors, virtual devices through to entire MR applications. We show how this process can be significantly enhanced by using a visual programming environment. Finally we present a set of typical examples demonstrating the potential of the approach.

An adaptive training-free feature tracker for mobile phones

While tracking technologies based on fiducial markers have dominated the development of Augmented Reality (AR) applications for almost a decade, various real-time capable approaches to markerless tracking have recently been presented. However, most existing approaches do not yet achieve sufficient frame rates for AR on mobile phones or at least require an extensive training phase in advance. In this paper we will present our approach on feature based tracking applying robust SURF features. The implementation is more than one magnitude faster than previous ones, allowing running even on mobile phones at highly interactive rates. In contrast to other feature based approaches on mobile phones, our implementation may immediately track features captured from a photo without any training. Further, the approach is not restricted to planar surfaces, but may use features of 3D objects.

Markerless Tracking for Augmented Reality

Augmented Reality (AR) tries to seamlessly integrate virtual content into the real world of the user. Ideally, the virtual content would behave exactly like real objects. This requires a correct and precise estimation of the user’s viewpoint (respectively that of a camera) with respect to the coordinate system of the virtual content. This can be achieved by an appropriate 6-DoF tracking system.

Random model variation for universal feature tracking

Feature based tracking approaches become more and more common for Augmented Reality (AR). However, most upcoming AR solutions are designed for mobile devices, in particular for smartphones and tablet computers, lacking sufficient performance for the execution of state-of-the art feature based approaches at interactive frame rates. In this paper we will present our approach significantly increasing the speed of feature based tracking, thus allowing for real-time applications even on mobile devices. Our approach applies a randomized pose initialization, is applicable to any feature detector and does not require any feature appearance attributes, such as descriptors or ferns.

Exploring augmented live video streams for remote participation

Augmented video streams display information within the context of the physical environment. In contrast to Augmented Reality, they do not require special equipment, they can support many users and are location-independent. In this paper we are exploring the potentials of augmented video streams for remote participation. We present our design considerations for remote participation user interfaces, briefly describe their development and explain the design of three different application scenarios: watching a pervasive game, observing the quality of a production process and exploring interactive science exhibits. The paper also discusses how to develop high quality augmented video streams along with which information and control options are required in order to obtain a viable remote participation interface.

Live Will Never Be the Same

While anticipated in Hollywood movies and sci-fi literature, a perfect artifical reality – i.e. the ultimate VR environment – will probably not become real within the next couple of years. However, using Augmented Reality (AR) technologies in order to seamlessly embed artificial content into the real environment, might be a much more feasible way to remove the clear border between reality and virtuality. In this paper we will look into lately developed AR technologies and how they relate to recent trends in movie and TV productions. We will further anticipate what this will mean to live broadcasts and which implications this might have for a widespread individual usage of sophisticated AR technology.

Supporting Reusability of VR and AR Interface Elements and Interaction Techniques

In contrast to 2D environments which apply well established user interface elements and generally accepted interaction techniques, VR and AR applications typically provide rather individual and specific realizations. This often leads to inconsistent user interfaces and a long and cumbersome development process. In this paper we show how we extended our approach on modeling VR and AR interface elements and interaction techniques represented by interaction and behavior objects by some simple yet powerful mechanisms: modules, templates, and inheritance. We will also show how specific examples could benefit from that approach.