Nate Hagbi

Shape recognition and pose estimation for mobile augmented reality

In this paper we present Nestor, a system for real-time recognition and camera pose estimation from planar shapes. The system allows shapes that carry contextual meanings for humans to be used as Augmented Reality (AR) tracking fiducials. The user can teach the system new shapes at runtime by showing them to the camera. The learned shapes are then maintained by the system in a shape library. Nestor performs shape recognition by analyzing contour structures and generating projective invariant signatures from their concavities. The concavities are further used to extract features for pose estimation and tracking. Pose refinement is carried out by minimizing the reprojection error between sample points on each image contour and its library counterpart. Sample points are matched by evolving an active contour in real time. Our experiments show that the system provides stable and accurate registration, and runs at interactive frame rates on a Nokia N95 mobile phone.

Line Segmentation for Degraded Handwritten Historical Documents

We propose a novel approach for text line segmentation based on adaptive local projection profiles. Our algorithm is suitable for degraded documents with text lines written in large skew. The main novelty of our approach is applying the local algorithm in an incremental manner that adapts to the skew of each text line as it progresses. The proposed approach achieves very accurate results on a set of degraded documents with lines written in different skew angles and curvatures.

In-place 3D sketching for authoring and augmenting mechanical systems

We present a framework for authoring three-dimensional virtual scenes for Augmented Reality (AR) which is based on hand sketching. Sketches consisting of multiple components are used to construct a 3D virtual scene augmented on top of the real drawing. Model structure and properties can be modified by editing the sketch itself and printed content can be combined with hand sketches to form a single scene. Authoring by sketching opens up new forms of interaction that have not been previously explored in Augmented Reality. To demonstrate the technology, we implemented an application that constructs 3D AR scenes of mechanical systems from freehand sketches, and animates the scenes using a physics engine. We provide examples of scenes composed from trihedral solid models, forces, and springs. Finally, we describe how sketch interaction can be used to author complicated physics experiments in a natural way.

Shape Recognition and Pose Estimation for Mobile Augmented Reality

Nestor is a real-time recognition and camera pose estimation system for planar shapes. The system allows shapes that carry contextual meanings for humans to be used as Augmented Reality (AR) tracking targets. The user can teach the system new shapes in real time. New shapes can be shown to the system frontally, or they can be automatically rectified according to previously learned shapes. Shapes can be automatically assigned virtual content by classification according to a shape class library. Nestor performs shape recognition by analyzing contour structures and generating projective-invariant signatures from their concavities. The concavities are further used to extract features for pose estimation and tracking. Pose refinement is carried out by minimizing the reprojection error between sample points on each image contour and its library counterpart. Sample points are matched by evolving an active contour in real time. Our experiments show that the system provides stable and accurate registration, and runs at interactive frame rates on a Nokia N95 mobile phone.

In-Place Sketching for content authoring in Augmented Reality games

Sketching leverages human skills for various purposes. In-Place Augmented Reality Sketching experiences build on the intuitiveness and flexibility of hand sketching for tasks like content creation. In this paper we explore the design space of In-Place Augmented Reality Sketching, with particular attention to content authoring in games. We propose a contextual model that offers a framework for the exploration of this design space by the research community. We describe a sketch-based AR racing game we developed to demonstrate the proposed model. The game is developed on top of our shape recognition and 3D registration library for mobile AR.

In-place Augmented Reality

In this paper we present a new vision-based approach for transmitting virtual models for augmented reality (AR). A two dimensional representation of the virtual models is embedded in a printed image. We apply image-processing techniques to interpret the printed image and extract the virtual models, which are then overlaid back on the printed image. The main advantages of our approach are: (1) the image of the embedded virtual models and their behaviors are understandable to a human without using an AR system, and (2) no database or network communication is required to retrieve the models. The latter is useful in scenarios with large numbers of users. We implemented an AR system that demonstrates the feasibility of our approach. Applications in education, advertisement, gaming, and other domains can benefit from our approach, since content providers need only to publish the printed content and all virtual information arrives with it.

Carving for topology simplification of polygonal meshes

The topological complexity of polygonal meshes has a large impact on the performance of various geometric processing algorithms, such as rendering and collision detection algorithms. Several approaches for simplifying topology have been discussed in the literature. These methods operate locally on models, which makes their effect on the topology hard to predict and analyze. Most existing methods tend to exhibit several disturbing artifacts, such as shrinking of the input and splitting of its components. We propose a novel top-down approach for topology simplification that avoids most problems that are common in existing methods. We start with a simple, genus-zero mesh that bounds the input and gradually introduce topologic features by a series of carving operations. This process yields a multiresolution stream of meshes with increasing topologic level of detail. We further present a carving algorithm that is based on constrained Delaunay tetrahedralization. The algorithm first constructs the tetrahedral mesh of the complement of the input with respect to its convex hull. It then proceeds to eliminate tetrahedra in a prioritized manner. We present quality results for two families of meshes that are difficult to simplify by all existing methods known to us: topologically complex meshes and highly clustered meshes.

Fast and Accurate Skew Estimation Based on Distance Transform

Document skew estimation is an important step in the process of document analysis. In this paper we discuss the properties of the distance transform of binarized documents and derive a fast and accurate method for detecting document skew. The method is based on the observation that the dominant orientation of the gradient of the distance transform accurately reflects the skew of the document. Our experiments suggest the method is robust to large variations in text properties, such as difference in script and page layout, and provides estimation accuracy of state-of-the-art skew detection methods.

In-Place Augmented Reality

In this paper, we present a vision-based approach for transmitting virtual models for Augmented Reality, which we name In-Place Augmented Reality (IPAR). A two-dimensional representation of the virtual models is embedded in a printed image. We apply computer vision techniques to interpret the printed image and extract the virtual models, which are then overlaid on the printed image. The main advantages of our approach are: (1) the image of the embedded virtual models and their behaviors are understandable to a human without using an AR system and (2) no database or network communication is required to retrieve the models. To demonstrate the technology and test its usability, we implemented several applications and performed a user evaluation. We discuss how the proposed technique can be used for the development of applications in different domains such as education, advertisement, and gaming.

In-Place Sketching for Augmented Reality Games

Sketching leverages human skills for various purposes, such as content authoring. This paper explores the different aspects of sketch-based interaction in augmented reality. We explore the design space of sketching in in-place augmented reality with particular attention to AR games. We propose a contextual model that offers a framework for the exploration of the design space by the research community. We describe different case studies of sketch-based AR games we developed based on our shape-based tracking library and our sketch-based 3-D game engine. We finally present user feedback from our games and introduce guidelines for AR game designers focusing on sketch-based interaction.