Trish Keaton

A Robust Finger Tracking Method for Multimodal Wearable Computer Interfacing

Mobile wearable computers are intended to provide users with real-time access to information in a natural and unobtrusive manner. Computing and sensing in these devices must be reliable, easy to interact with, transparent, and configured to support different needs and complexities. This paper presents a vision-based robust finger tracking algorithm combined with audio-based control commands that is integrated into a multimodal unobtrusive user interface, wherein the interface may be used to segment out objects of interest in the environment by encircling them with the users pointing fingertip. In order to quickly extract the objects encircled by the user from a complex scene, this unobtrusive interface uses a single head-mounted camera to capture color images, which are then processed using algorithms to perform: color segmentation, fingertip shape analysis, perturbation model learning, and robust fingertip tracking. This interface is designed to be robust to changes in the environment and users movements by incorporating a state-space estimation with uncertain models algorithm, which attempts to control the influence of uncertain environment conditions on the systems fingertip tracking performance by adapting the tracking model to compensate for the uncertainties inherent in the data collected with a wearable computer

A level set method for the extraction of roads from multispectral imagery

With the advances in remote sensing technologies, the extraction of roads and other linear features from satellite and aerial imagery has gained substantial interest in recent years. The introduction of satellite imagery characterized by high spectral and spatial resolutions has made possible the development of new viable approaches for the accurate, and cost-effective extraction of linear features with minimal human intervention. This paper presents a semi-automated method for the extraction of roads from high resolution (1 meter) pan-sharpened multispectral IKONOS imagery. An operator provides an initial seed point on the road of interest, then the region is grown using a level set method. Further analysis through iterative smoothing refines the extracted region to accurately estimate the road centerline despite the presence of cars on the road, changes in the pavement or surface properties of the road, or obstruction resulting from foliage or shadows cast on the road by neighboring trees. Initial results have demonstrated the utility of the algorithm in efficiently extracting roads from high resolution satellite imagery with minimal human interaction. Over 97 % delineation accuracy was achieved on manually ground truthed IKONOS image samples overlooking both urban and rural locations.

Robust finger tracking for wearable computer interfacing

Key to the design of human-machine gesture interface applications is the ability of the machine to quickly and efficiently identify and track the hand movements of its user. In a wearable computer system equipped with head-mounted cameras, this task is extremely difficult due to the uncertain camera motion caused by the users head movement, the user standing still then randomly walking, and the users hand or pointing finger abruptly changing directions at variable speeds. This paper presents a tracking methodology based on a robust state-space estimation algorithm, which attempts to control the influence of uncertain environment conditions on the systems performance by adapting the tracking model to compensate for the uncertainties inherent in the data. Our system tracks a users pointing gesture from a single head mounted camera, to allow the user to encircle an object of interest, thereby coarsely segmenting the object. The snapshot of the object is then passed to a recognition engine for identification, and retrieval of any pre-stored information regarding the object. A comparison of our robust tracker against a plain Kalman tracker showed a 15% improvement in the estimated position error, and exhibited a faster response time.

Wide baseline image registration with application to 3-D face modeling

Establishing correspondence between features in two images of the same scene taken from different viewing angles is a challenging problem in image processing and computer vision. However, its solution is an important step in many applications like wide baseline stereo, three-dimensional (3-D) model alignment, creation of panoramic views, etc. In this paper, we propose a technique for registration of two images of a face obtained from different viewing angles. We show that prior information about the general characteristics of a face obtained from video sequences of different faces can be used to design a robust correspondence algorithm. The method works by matching two-dimensional (2-D) shapes of the different features of the face (e.g., eyes, nose etc.). A doubly stochastic matrix, representing the probability of match between the features, is derived using the Sinkhorn normalization procedure. The final correspondence is obtained by minimizing the probability of error of a match between the entire constellation of features in the two sets, thus taking into account the global spatial configuration of the features. The method is applied for creating holistic 3-D models of a face from partial representations. Although this paper focuses primarily on faces, the algorithm can also be used for other objects with small modifications.

Browsing the environment with the SNAP&TELL wearable computer system

This paper provides an overview of a multi-modal wearable computer system, SNAP&TELL. The system performs real-time gesture tracking, combined with audio-based control commands, in order to recognize objects in an environment, including outdoor landmarks. The system uses a single camera to capture images, which are then processed to perform color segmentation, fingertip shape analysis, robust tracking, and invariant object recognition, in order to quickly identify the objects encircled and SNAPped by the user’s pointing gesture. In addition, the system returns an audio narration, TELLing the user information concerning the object’s classification, historical facts, usage, etc. This system provides enabling technology for the design of intelligent assistants to support “Web-On-The-World” applications, with potential uses such as travel assistance, business advertisement, the design of smart living and working spaces, and pervasive wireless services and internet vehicles.

Evolving roads in IKONOS multispectral imagery

The introduction of satellite imagery characterized by high spectral and spatial resolutions has made possible the development of new viable approaches for the accurate, and cost-effective extraction of linear features with minimal human intervention. This paper presents a semi-automated method for the extraction of roads from (1-meter) pan-sharpened multispectral IKONOS imagery. An operator provides an initial seed point on the road of interest, then the region is evolved using a level set method. Further analysis through iterative smoothing refines the extracted region to accurately estimate the road centerline despite the presence of cars on the road, changes in the pavement or surface properties of the road, or obstruction resulting from foliage or shadows cast on the road by neighboring trees. Initial results have demonstrated the utility of the algorithm in efficiently extracting roads from high resolution satellite imagery with minimal human interaction. Over 97% delineation accuracy was achieved on manually ground truthed IKONOS image samples overlooking both urban and rural locations.

Comparison of robust estimation and Kalman filtering applied to fingertip tracking in human-machine interfaces

This paper studies the application of robust state-space estimation with uncertain models to tracking problems in human-machine interfaces. The need for robust methods arises from the desire to control the influence of uncertain environmental conditions on system performance, such as the effect of abrupt variations in object speed and motion characteristics. This paper produces models for motion uncertainties associated with a human hand, and applies them to a robust state-space estimation algorithm used to track a users pointing fingertip. Then a comparison is performed between the results from the robust tracker against a Kalman filter.

Automatic extraction of urban regions from multispectral SPOT satellite imagery

This paper presents a system for the automatic extraction of urban regions from multispectral satellite images using a pseudo-supervised method to classify unlabeled images. This is accomplished through a preclassification process which automatically extracts representative training data from images without the need for ground truth data or user intervention. In the preclassification stage, a technique is developed to find a set of prototypes from regions exhibiting urban-like characteristics, then a subspace is constructed by applying principle component analysis over the set of urban prototypes. Image points are then preclassified based upon their distance to the urban subspace. A set of representative training data is extracted from the preclassified urban and non-urban regions, which is used to train a Gaussian mixture model based Bayesian classifier. The final extraction of urban-like regions is determined by the trained Bayesian classifier using a maximum a posteriori (MAP) criterion. Our experimental results show that the proposed technique is effective in automatically extracting urban regions from SPOT multispectral satellite images.

Wide baseline image registration using prior information

Establishing correspondence between features in two images of the same scene taken from different viewing angles in a challenging problem in image processing and computer vision. However, its solution is an important step in many applications like wide baseline stereo, 3D model alignment, creation of panoramic views etc. In this paper, we propose a technique for registration of two images of a face obtained from different viewing angles. We show that prior information about the general characteristics of a face obtained from video sequences of different faces can be used to design a robust correspondence algorithm. The method works by matching 2D shapes of the different features of the face. A doubly stochastic matrix, representing the probability of match between the features, is derived using the Sinkhorn normalization procedure. The final correspondence is obtained by minimizing the probability of error of a match between the entire constellations of features in the two sets, thus taking into account the global spatial configuration of the features. The method is applied for creating holistic 3D models of a face from partial representations. Although this paper focuses primarily on faces, the algorithm can also be used for other objects with small modifications.