Dieter Schmalstieg

Pose tracking from natural features on mobile phones

In this paper we present two techniques for natural feature tracking in real-time on mobile phones. We achieve interactive frame rates of up to 20 Hz for natural feature tracking from textured planar targets on current-generation phones. We use an approach based on heavily modified state-of-the-art feature descriptors, namely SIFT and Ferns. While SIFT is known to be a strong, but computationally expensive feature descriptor, Ferns classification is fast, but requires large amounts of memory. This renders both original designs unsuitable for mobile phones. We give detailed descriptions on how we modified both approaches to make them suitable for mobile phones. We present evaluations on robustness and performance on various devices and finally discuss their appropriateness for augmented reality applications.

The studierstube augmented reality project

Our starting point for developing the Studierstube system was the belief that augmented reality, the less obtrusive cousin of virtual reality, has a better chance of becoming a viable user interface for applications requiring manipulation of complex three-dimensional information as a daily routine. In essence, we are searching for a 3-D user interface metaphor as powerful as the desktop metaphor for 2-D. At the heart of the Studierstube system, collaborative augmented reality is used to embed computer-generated images into the real work environment. In the first part of this paper, we review the user interface of the initial Studierstube system, in particular the implementation of collaborative augmented reality, and the Personal Interaction Panel, a two-handed interface for interaction with the system. In the second part, an extended Studierstube system based on a heterogeneous distributed architecture is presented. This system allows the user to combine multiple approaches augmented reality, projection displays, and ubiquitous computingto the interface as needed. The environment is controlled by the Personal Interaction Panel, a twohanded, pen-and-pad interface that has versatile uses for interacting with the virtual environment. Studierstube also borrows elements from the desktop, such as multitasking and multi-windowing. The resulting software architecture is a user interface management system for complex augmented reality applications. The presentation is complemented by selected application examples.

First steps towards handheld augmented reality

In this paper we describe the first stand-alone AugmentedReality (AR) system with self-tracking running on anunmodified personal digital assistant (PDA) with acommercial camera. The project exploits the readyavailability of consumer devices with a minimal need forinfrastructure. The application provides the user with athree-dimensional augmented view of the environment.Our system achieves good overlay registration accuracyby using a popular marker-based tracking toolkit(ARToolKit), which runs directly on the PDA. Weintroduce an optional client/server architecture that isbased on wireless networking and is able to dynamicallyand transparently offload the tracking task in order toprovide better performance in select areas. Thehardware/software framework is modular and can beeasily combined with many elements of an existing ARframework. As a demonstration of the effectiveness, wepresent a 3D navigation application that guides a userthrough an unknown building to a chosen location.

Real-Time Detection and Tracking for Augmented Reality on Mobile Phones

In this paper, we present three techniques for 6DOF natural feature tracking in real time on mobile phones. We achieve interactive frame rates of up to 30 Hz for natural feature tracking from textured planar targets on current generation phones. We use an approach based on heavily modified state-of-the-art feature descriptors, namely SIFT and Ferns plus a template-matching-based tracker. While SIFT is known to be a strong, but computationally expensive feature descriptor, Ferns classification is fast, but requires large amounts of memory. This renders both original designs unsuitable for mobile phones. We give detailed descriptions on how we modified both approaches to make them suitable for mobile phones. The template-based tracker further increases the performance and robustness of the SIFT- and Ferns-based approaches. We present evaluations on robustness and performance and discuss their appropriateness for Augmented Reality applications.

Mathematics and geometry education with collaborative augmented reality

Abstract Construct3D is a 3D geometric construction tool specifically designed for mathematics and geometry education. It is based on the mobile collaborative augmented reality system “Studierstube”. We describe our efforts in developing a system for the improvement of spatial abilities and maximization of transfer of learning. In order to support various teacher–student interaction scenarios we implemented flexible methods for context and user dependent rendering of parts of the construction. Together with hybrid hardware setups they allow the use of Construct3D in todays classrooms and provide a testbed for future evaluations. Means of application and integration in mathematics and geometry education at high school as well as university level are being discussed. Anecdotal evidence supports our claim that Construct3D is easy to learn, encourages experimentation with geometric constructions and improves spatial skills.

Towards massively multi-user augmented reality on handheld devices

Augmented Reality (AR) can naturally complement mobile computing on wearable devices by providing an intuitive interface to a three-dimensional information space embedded within physical reality. Unfortunately, current wearable AR systems are relatively complex, expensive, fragile and heavy, rendering them unfit for large-scale deployment involving untrained users outside constrained laboratory environments. Consequently, the scale of collaborative multi-user experiments have not yet exceeded a handful of participants. In this paper, we present a system architecture for interactive, infrastructure-independent multi-user AR applications running on off-the-shelf handheld devices. We implemented a four-user interactive game installation as an evaluation setup to encourage playful engagement of participants in a cooperative task. Over the course of five weeks, more than five thousand visitors from a wide range of professional and socio-demographic backgrounds interacted with our system at four different locations.

Using transparent props for interaction with the virtual table

The Virtual Table presents stereoscopic graphics to a user in a workbench-like setting. This paper reports on a user interface and new interaction techniques for the Virtual Table based on transparent props—a tracked hand-held pen and a pad. These props, but in particular the pad, are augmented with 3D graphics from the Virtual Table’s display. This configuration creates a very powerful and flexible interface for two-handed interaction that can be applied to other back-projected stereographic displays as well: the pad can serve as a palette for tools and controls as well as a window-like see-through interface, a plane-shaped and throughthe-plane tool, supporting a variety of new interaction techniques.

Experiences with Handheld Augmented Reality

In this paper, we present Studierstube ES, a framework for the development of handheld Augmented Reality. The applications run self-contained on handheld computers and smartphones with Windows CE. A detailed description of the performance critical tracking and rendering components are given. We also report on the implementation of a client-server architecture for multi-user applications, and a game engine for location based museum games that has been built on top of this infrastructure. Details on two games that were created, permanently deployed and evaluated in two Austrian museums illustrate the practical value of the framework and lessons learned from using it.

Handheld Augmented Reality for underground infrastructure visualization

In this paper, we present an Augmented Reality (AR) system for aiding field workers of utility companies in outdoor tasks such as maintenance, planning or surveying of underground infrastructure. Our work addresses these issues using spatial interaction and visualization techniques for mobile AR applications and as well as for a new mobile device design. We also present results from evaluations of the prototype application for underground infrastructure spanning various user groups. Our application has been driven by feedback from industrial collaborators in the utility sector, and includes a translation tool for automatically importing data from utility company databases of underground assets.

Construct3D: A Virtual Reality Application for Mathematics and Geometry Education

Construct3D is a three dimensional geometric construction tool based on the collaborative augmented reality system ‘Studierstube’. Our setup uses a stereoscopic head mounted display (HMD) and the Personal Interaction Panel (PIP) - a two-handed 3D interaction tool that simplifies 3D model interaction. Means of application in mathematics and geometry education at high school as well as university level are being discussed. A pilot study summarizes the strengths and possible extensions of our system. Anecdotal evidence supports our claim that the use of Construct3D is easy to learn and encourages experimentation with geometric constructions.