Byung-Kuk Seo

One-handed interaction with augmented virtual objects on mobile devices

We present a one-handed approach for augmented reality and interaction on mobile devices. The proposed application considers common situations with mobile devices such as when a users hand holds a mobile device while the other hand is free. It also supports natural augmented reality environment such as when a user interacts with augmented reality contents anytime and anywhere without special equipment such as visual markers or tags. In our approach, a virtual object is augmented on the palm of a users free hand, as if the virtual object is just sitting on the palm, using a palm pose estimation method. The augmented virtual object reacts (e.g. moving or animation) to motions of the hand such as opening or closing the hand based on fingertip tracking. Moreover, it provides tactile interactions with the virtual object by wearing a tactile glove with vibration sensors. This paper describes how to implement the augmented reality application, and preliminary results show its potential as a new approach to mobile augmented reality interaction.

Optimal Local Searching for Fast and Robust Textureless 3D Object Tracking in Highly Cluttered Backgrounds

Edge-based tracking is a fast and plausible approach for textureless 3D object tracking, but its robustness is still very challenging in highly cluttered backgrounds due to numerous local minima. To overcome this problem, we propose a novel method for fast and robust textureless 3D object tracking in highly cluttered backgrounds. The proposed method is based on optimal local searching of 3D-2D correspondences between a known 3D object model and 2D scene edges in an image with heavy background clutter. In our searching scheme, searching regions are partitioned into three levels (interior, contour, and exterior) with respect to the previous object region, and confident searching directions are determined by evaluating candidates of correspondences on their region levels; thus, the correspondences are searched among likely candidates in only the confident directions instead of searching through all candidates. To ensure the confident searching direction, we also adopt the region appearance, which is efficiently modeled on a newly defined local space (called a searching bundle). Experimental results and performance evaluations demonstrate that our method fully supports fast and robust textureless 3D object tracking even in highly cluttered backgrounds.

Robust hand detection for augmented reality interface

For interactive augmented reality, vision-based and hand-gesture-based interface are most desirable due to being natural and human-friendly. However, detecting hands and recognizing hand gestures in cluttered background are still challenging. Especially, if the background includes a large skin-colored region, the problem becomes more difficult. In this paper, we focus on detecting a hand reliably and propose an effective method. Our method is basically based on the assumption that a hand-forearm region (including a hand and part of a forearm) has different brightness from other skin-colored regions. Specifically, we first segment the hand-forearm region from other skin-colored regions based on the brightness difference which is represented by edges in this paper. Then, we extract the hand region from the hand-forearm region by detecting a feature point which indicates the wrist. Finally, we extract the hand by using the brightness-based segmentation which is slightly different from the hand-forearm region detection. We verify the effectiveness of our method by implementing a simple hand gesture interface based on our method and applying it to augmented reality applications.

Optimal illumination for discriminating objects with different spectra.

Color differences are determined by illumination, the spectral reflectance of objects, and the spectral sensitivity of the imaging sensor. We explore the optimal illumination conditions that best separate one object from another. Given two objects with distinct spectra, we derive the optimal illumination spectrum to maximize their color distance with a plain RGB camera. In practice, it is crucial to compose the most appropriate illuminations using available lighting sources, since creating an arbitrary illumination spectrum is unrealistic. Therefore, we derive the optimal linear combination of the provided illumination sources. Finally, we verify the effectiveness of the methods through experiments.

A tracking framework for augmented reality tours on cultural heritage sites

Visual tracking for augmented reality tours is still challenging for cultural heritage sites because of the great variation of tracking targets and environments on such sites. Even at todays state of the art, it is almost impossible to apply just one tracking method to all the various environments with any hope of success. This paper presents a tracking framework to overcome this problem. It consists of different tracking flows, each efficiently using robust visual cues of the target scene. Analysis of the tracking environment enables more practical tracking at the sites. The reliability of the tracking framework is verified through on-site demonstrations at Gyeong-bokgung, the most symbolic cultural heritage site in Korea.

Undistorted projection onto dynamic surface

For projector-based augmented reality systems, geometric correction is a crucial function. There have been many researches on the geometric correction in the literature. However, most of them focused only on static projection surfaces and could not give us a solution for dynamic surfaces (with varying geometry in time). In this paper, we aim at providing a simple and robust framework for projecting augmented reality images onto dynamic surfaces without image distortion. For this purpose, a new technique for embedding pattern images into the augmented reality images, which allows simultaneous display and correction, is proposed and its validity is shown in experimental results.

Augmented reality-based on-site tour guide: a study in Gyeongbokgung

This paper presents an on-site tour guide using augmented reality in which past life is virtually reproduced and visualized at cultural heritage sites. In the tour guide, animated 3-D virtual characters are superimposed on the cultural heritage sites by visually tracking simple geometric primitives of the sites such as rectangles and estimating camera poses (positions and orientations) that can be considered as a tourists viewpoints. Contextual information, such as a tourists locations and profiles, is used to support personalized tour guides. In particular, the tourists locations are obtained by visually recognizing wooden tablets of the cultural heritage sites. The prototype of the augmented reality tour guide was tested at Gangnyeongjeon and Gyotaejeon in Gyeongbokgung, which is a symbolic cultural heritage site in Korea and its user evaluation is discussed.

Simultaneous Geometric and Radiometric Adaptation to Dynamic Surfaces With a Mobile Projector-Camera System

Existing geometric and radiometric compensation methods for direct-projected augmented reality focus on static projection surfaces rather than dynamic surfaces (with varying geometry in time). We aim at providing an effective framework for projecting a sequence of augmented reality images onto dynamic surfaces without geometric and radiometric distortion. We present our design of a special pattern image for simultaneous geometric and radiometric compensation and evaluate two different techniques for embedding the pattern image into augmented reality images. The validity of the proposed method is examined through a variety of experiments with a mobile projector-camera system.

Subjective Evaluation on Visual Perceptibility of Embedding Complementary Patterns for Nonintrusive Projection-Based Augmented Reality

In projection-based augmented reality (AR) alleviating visual distraction of patterns has been a great challenge. As a representative one, a method of embedding patterns and their complements (hereafter, we call the pairs complementary patterns) into AR images alternately has been proposed recently. This paper presents subjective evaluation results and their statistical analysis on the visual perceptibility of embedding complementary patterns in different ways in a standard hardware environment. Then, we explore the constraints for embedded complementary patterns to be less perceptible. As expected, high projector refresh rate and low pattern strength were the general conditions for a decrease in the perception of embedded complementary patterns. However, reducing pattern size and projecting complementary patterns with an interval were also among the factors affecting the results. Detailed constraints are given in the experimental results. Also, we present which constraint is more dominant for pattern perceptibility.

Direct-Projected AR Based Interactive User Interface for Medical Surgery

In the field of computer aided surgery, augmented reality (AR) technology has been successfully used for enhancing accuracy of surgery and making surgeons convenient by visually assisting them in performing a number of complicated and time-consuming medical operations. However, there are still medical operations that do not receive the benefit of AR technology. As a representative one, surgeons still use an ink pen when they mark surgical targets for scheduling an operation. The ink pen is inconvenient because the mark drawn by the foreign matter is not easily modified or deleted. And the ink pen is also unlikely to be sanitary. In this paper, we propose an interactive user interface based on direct-projected augmented reality (DirectAR) technology for handling all these problems with the ink pen and its validity is shown in experimental results.