Youngkyoon Jang

3D Finger CAPE: Clicking Action and Position Estimation under Self-Occlusions in Egocentric Viewpoint

In this paper we present a novel framework for simultaneous detection of click action and estimation of occluded fingertip positions from egocentric viewed single-depth image sequences. For the detection and estimation, a novel probabilistic inference based on knowledge priors of clicking motion and clicked position is presented. Based on the detection and estimation results, we were able to achieve a fine resolution level of a bare hand-based interaction with virtual objects in egocentric viewpoint. Our contributions include: (i) a rotation and translation invariant finger clicking action and position estimation using the combination of 2D image-based fingertip detection with 3D hand posture estimation in egocentric viewpoint. (ii) a novel spatio-temporal random forest, which performs the detection and estimation efficiently in a single framework. We also present (iii) a selection process utilizing the proposed clicking action detection and position estimation in an arm reachable AR/VR space, which does not require any additional device. Experimental results show that the proposed method delivers promising performance under frequent self-occlusions in the process of selecting objects in AR/VR space whilst wearing an egocentric-depth camera-attached HMD.In this paper we present a novel framework for simultaneous detection of click action and estimation of occluded fingertip positions from egocentric viewed single-depth image sequences. For the detection and estimation, a novel probabilistic inference based on knowledge priors of clicking motion and clicked position is presented. Based on the detection and estimation results, we were able to achieve a fine resolution level of a bare hand-based interaction with virtual objects in egocentric viewpoint. Our contributions include: (i) a rotation and translation invariant finger clicking action and position estimation using the combination of 2D image-based fingertip detection with 3D hand posture estimation in egocentric viewpoint. (ii) a novel spatio-temporal random forest, which performs the detection and estimation efficiently in a single framework. We also present (iii) a selection process utilizing the proposed clicking action detection and position estimation in an arm reachable AR/VR space, which does not require any additional device. Experimental results show that the proposed method delivers promising performance under frequent self-occlusions in the process of selecting objects in AR/VR space whilst wearing an egocentric-depth camera-attached HMD.

Unified Context-Aware Augmented Reality Application Framework for User-Driven Tour Guides

We propose a unified context-aware augmented reality application framework that supports intelligent guidance and enables users to participate in content generation in museum guidance. It helps a user find personal interesting artifacts in art galleries by exploiting context-based behavior generation. The framework also enables them to combine augmented contents with different information to change the shape of content according to their preferences. Furthermore, it allows the users to label real objects for attaching new contents over the objects. Through demonstration in an art gallery, we found that the resulting system effectively guided users to visit and enabled them to participate in tour guidance.

A Study on Touchless Finger Vein Recognition Robust to the Alignment and Rotation of Finger

With increases in recent security requirements, biometric technology such as fingerprints, faces and iris recognitions have been widely used in many applications including door access control, personal authentication for computers, internet banking, automatic teller machines and border-crossing controls. Finger vein recognition uses the unique patterns of finger veins in order to identify individuals at a high level of accuracy. This paper proposes new device and methods for touchless finger vein recognition. This research presents the following five advantages compared to previous works. First, by using a minimal guiding structure for the finger tip, side and the back of finger, we were able to obtain touchless finger vein images without causing much inconvenience to user. Second, by using a hot mirror, which was slanted at the angle of 45 degrees in front of the camera, we were able to reduce the depth of the capturing device. Consequently, it would be possible to use the device in many applications having size limitations such as mobile phones. Third, we used the holistic texture information of the finger veins based on a LBP (Local Binary Pattern) without needing to extract accurate finger vein regions. By using this method, we were able to reduce the effect of non-uniform illumination including shaded and highly saturated areas. Fourth, we enhanced recognition performance by excluding non-finger vein regions. Fifth, when matching the extracted finger vein code with the enrolled one, by using the bit-shift in both the horizontal and vertical directions, we could reduce the authentic variations caused by the translation and rotation of finger. Experimental results showed that the EER (Equal Error Rate) was 0.07423% and the total processing time was 91.4ms.

Video-Based Object Recognition Using Novel Set-of-Sets Representations

We address the problem of object recognition in egocentric videos, where a user arbitrarily moves a mobile camera around an unknown object. Using a video that captures variation in an objects appearance owing to camera motion (more viewpoints, scales, clutter and lighting conditions), can accumulate evidence and improve object recognition accuracy. Most previous work has taken a single image as input, or tackled a video simply by a collection i.e. sum of frame-based recognition scores. In this paper, beyond frame-based recognition, we propose two novel set-of-sets representations of a video sequence for object recognition. We combine the techniques of bag of words for a set of data spatially distributed thus heterogeneous, and manifold for a set of data temporally smooth and homogeneous, to construct the two proposed set-of-sets representations. We also propose methods to perform matching for the two representations respectively. The representations and matching techniques are evaluated on our video-based object recognition datasets, which contain 830 videos of ten objects and four environmental variations. The experiments on the challenging new datasets show that our proposed solution significantly outperforms the traditional frame-based methods.

Stroke-based semi-automatic region of interest detection algorithm for in-situ painting recognition

In the case of illumination and view direction changes, the ability to accurately detect the Regions of Interest (ROI) is important for robust recognition. In this paper, we propose a stroke-based semi-automatic ROI detection algorithm using adaptive thresholding and a Hough-transform method for in-situ painting recognition. The proposed algorithm handles both simple and complicated texture painting cases by adaptively finding the threshold. It provides dominant edges by using the determined threshold, thereby enabling the Hough-transform method to succeed. Next, the proposed algorithm is easy to learn, as it only requires minimal participation from the user to draw a diagonal line from one end of the ROI to the other. Even though it requires a stroke to specify two vertex searching regions, it detects unspecified vertices by estimating probable vertex positions calculated by selecting appropriate lines comprising the predetected vertices. In this way, it accurately (1.16 error pixels) detects the painting region, even though a user sees the painting from the flank and gives inaccurate (4.53 error pixels) input points. Finally, the proposed algorithm provides for a fast processing time on mobile devices by adopting the Local Binary Pattern (LBP) method and normalizing the size of the detected ROI; the ROI image becomes smaller in terms of general code format for recognition, while preserving a high recognition accuracy (99.51%). As such, it is expected that this work can be used for a mobile gallery viewing system.

Metaphoric Hand Gestures for Orientation-Aware VR Object Manipulation With an Egocentric Viewpoint

We present a novel natural user interface framework, called Meta-Gesture, for selecting and manipulating rotatable virtual reality (VR) objects in egocentric viewpoint. Meta-Gesture uses the gestures of holding and manipulating the tools of daily use. Specifically, the holding gesture is used to summon a virtual object into the palm, and the manipulating gesture to trigger the function of the summoned virtual tool. Our contributions are broadly threefold: 1) Meta-Gesture is the first to perform bare hand-gesture-based orientation-aware selection and manipulation of very small (nail-sized) VR objects, which has become possible by combining a stable 3-D palm pose estimator (publicly available) with the proposed static-dynamic (SD) gesture estimator; 2) the proposed novel SD random forest, as an SD gesture estimator can classify a 3-D static gesture and its action status hierarchically, in a single classifier; and 3) our novel voxel coding scheme, called layered shape pattern, which is configured by calculating the fill rate of point clouds (raw source of data) in each voxel on the top of the palm pose estimation, allows for dispensing with the need for preceding hand skeletal tracking or joint classification while defining a gesture. Experimental results show that the proposed method can deliver promising performance, even under frequent occlusions, during orientation-aware selection and manipulation of objects in VR space by wearing head-mounted display with an attached egocentric-depth camera (see the supplementary video available at: http://ieeexplore.ieee.org).

Smart Wristband: Touch-and-Motion–Tracking Wearable 3D Input Device for Smart Glasses

The smart wristband is a novel type of wearable input device for smart glasses, and it can control multi-dimensional contents by using touch and motion. The smart wristband uses a touch-and-motion–tracking system with a touch screen panel (TSP) and inertial measurement unit (IMU) to help users control the smart glasses’ interface accurately and quickly without environmental noise, distortion, and multi-leveled pattern recognition tasks.

Unified Visual Perception Model for context-aware wearable AR

We propose Unified Visual Perception Model (UVPM), which imitates the human visual perception process, for the stable object recognition necessarily required for augmented reality (AR) in the field. The proposed model is designed based on the theoretical bases in the field of cognitive informatics, brain research and psychological science. The proposed model consists of Working Memory (WM) in charge of low-level processing (in a bottomup manner), Long-Term Memory (LTM) and Short-Term Memory (STM), which are in charge of high-level processing (in a top-down manner). WM and LTM/STM are mutually complementary to increase recognition accuracies. By implementing the initial prototype of each boxes of the model, we could know that the proposed model works for stable object recognition. The proposed model is available to support context-aware AR with the optical see-through HMD.

Local Feature Descriptors for 3D Object Recognition in Ubiquitous Virtual Reality

This paper represents 3D object recognition, which is an extension of the common feature point-based object recognition, based on novel descriptors utilizing local angles (for shape), gradient orientations (for texture of corners), and color information. First, the proposed algorithm extracts complementary feature points by randomly sampling the positions of the object edges. Then, it generates the proposed descriptors combining local angle patterns, gradient orientations, and color information. After making the descriptors, the method learns a codebook to enable the proposed algorithm to integrate the extracted feature points into a histogram through this codebook. Finally, the method classifies the query histogram based on a classifier. We expect that the proposed algorithm is robust to less textured and similar-shaped objects. The proposed method could be used as a core technology of the initial step of the information retrieval.

Adaptive Lip Feature Point Detection Algorithm for Real-Time Computer Vision-Based Smile Training System

This paper presents an adaptive lip feature point detection algorithm for the proposed real-time smile training system using visual instructions. The proposed algorithm can detect a lip feature point irrespective of lip color with minimal user participation, such as drawing a line on a lip on the screen. Therefore, the proposed algorithm supports adaptive feature detection by real-time analysis for a color histogram. Moreover, we develop a supportive guide model as visual instructions for the target expression. By using the guide model, users can train their smile expression intuitively because they can easily identify the differences between their smile and target expression. We also allow users to experience the smile training system using the proposed methods and we evaluated the effectiveness of these methods through usability tests. As experimental results, the proposed algorithm for feature detection had 3.4 error pixels and we found that the proposed methods could be an effective approach for training smile expressions in real-time processing.