Jakub Segen

Shadow gestures: 3D hand pose estimation using a single camera

This paper describes a system that uses a camera and a point light source to track a users hand in three dimensions. Using depth cues obtained from projections of the hand and its shadow, the system computes the 3D position and orientation of two fingers (thumb and pointing finger). The system recognizes one dynamic and two static gestures. Recognition and pose estimation are user independent and robust. The system operates at the rate of 60 Hz and can be used as an intuitive input interface to applications that require multi-dimensional control. Examples include 3D fly-thrus, object manipulation and computer games.

A camera-based system for tracking people in real time

This paper describes a system for real-time tracking of people in video sequences. The input to the system is live or recorded video data acquired by a stationary camera in an environment where the primary moving objects are people. The output consists of trajectories which give the spatio-temporal coordinates of individual persons as they move in the environment. The system uses a new model-based approach to object tracking. It identifies feature points in each video frame, matches feature points across frames to produce feature paths, then groups short-lived and partially overlapping feature paths into longer living trajectories representing motion of individual persons. The path grouping is based on a novel model-based algorithm for motion clustering. The system runs on an SGI Indy workstation at an average rate of 14 frames a second. The system has numerous applications since various statistics and indicators of human activity can be derived from the motion trajectories. Examples of these indicators described in the paper include people counts, presence and time spent in a region, traffic density maps and directional traffic statistics.

Human-computer interaction using gesture recognition and 3D hand tracking

This paper describes a real time system for human-computer interaction through gesture recognition and three dimensional hand-tracking. Using two cameras that are focused at the users hand the system recognizes three gestures and tracks the hand in three dimensions. The system can simultaneously track two fingers (thumb and pointing finger) and output their poses. The pose for each finger consists of three positional coordinates and two angles (azimuth and elevation). By moving the thumb and the pointing finger in 3D, the user can control 10 degrees of freedom in a smooth and natural fashion. We have used this system as a multi-dimensional input-interface to computer games, terrain navigation software and graphical editors. In addition to providing 10 degrees of control, our system is much more natural and intuitive to use compared to traditional input devices. The system is user independent, and operates at the rate of 60 Hz.

Fast and accurate 3D gesture recognition interface

A video-based gesture recognition system can serve as a natural and accurate 3D user input device. We describe a two-camera system, that recognizes three gesture classes: two static and one dynamic. For one of these gestures (pointing), the system estimates five parameters of 3D pose: position and pointing direction. The recognition is robust, independent of the user and fast (60 Hz), and the estimated pose is very stable. We describe some of the interface applications that demonstrate the benefits of the system: control of a video game, piloting a virtual reality fly-through, and interaction with a 3D scene editor.

Model learning and recognition of nonrigid objects

A method of learning structural models of 2D shape from real data is described and demonstrated. These models can be used to classify nonrigid shapes, even if they are partially occluded, and to label their parts. The representation of a single shape is a layered graph whose vertices correspond to n-ary relations. A class of shapes is represented as a probability model whose outcome is a graph. The method is based on two types of learning: unsupervised learning used to discover relations, and supervised learning used to build class models. The class models are constructed incrementally, by matching and merging graphs representing shape instances. This process uses a fast graph-matching heuristic which seeks a simplest representation of a graph. An important feature is the self-generation of symbolic primitives by an unsupervised learning process. This feature makes it possible to apply the system to any set of shape data without adjustments, while other methods might require the user to provide a different set of primitives for each case.<<ETX>>

Learning graph models of shape

This paper describes an implemented system that learns structural models of shape from noisy image data. These models can be used to recognize deformable shapes with an arbitrary orientation, even if they are partially occluded. The representation used for instances and concepts of shape is a multilevel graph, whose vertices correspond to n-ary relations. The system exhibits two types of learning a constructive induction (involving learning from observations) used to discover relations, and learning from examples used to build the concept model. A concept model is constructed incrementally, by matching and merging graph instances. This process relies on a novel, very efficient graph matching method, that seeks a simplest representation of a graph. The performance of the system is demonstrated with real examples.

Look ma, no mouse!

magine taking a virtual flight over theYosemite Valley. You are in control of theflight—you soar up through the clouds,dive deep into the ravines, flying inwhichever direction you please—all bysimply moving your hand through the air.No gloves to wear and no complex key-board commands to remember. You haveabsolutely no devices to control—you point a fin-ger, pretend your hand is a fighter jet, and just fly.Today’s home computers are becoming increas-ingly powerful while also becoming very afford-able. Complex real-time animations that requiredhigh-end graphics workstations a few years ago cannow be done on home computers equipped withinexpensive graphics cards. A whole suite of inter-active 3D applications is now accessible to theaverage user. Examples range from molecular sim-ulators to 3D presentation tools that allow manip-ulation of virtual objects. However, what has notchanged is the complex interface to these applica-tions. 3D fly-thru’s, for example, require intricatemaneuvers and learning the input controls isextremely painstaking unless, of course, you are ateenager. The problem is with the way we com-municate with computers.Traditional computer interfaces (keyboard,mouse, joystick) are not always appropriate forinteracting with 3D applications. First of all, theycontrol too few parameters. For instance, whileplacing an observer in a 3D space requires sixdegrees of freedom, a mouse allows us to controlonly two. The issue here is not only the number ofparameters but also the ease with which theseparameters can be manipulated. Certain newdevices like the SpaceBall

Automatic discovery of robotic grasp configurations

A method that enables a robot system to learn how to grasp an object is presented. The method combines an automatic grasp discovery process with a visual recognition technique. When an object is seen the first time, the system experiments with it, seeking a way of grasping and lifting the object by trial and error, using visual information and input from the robot gripper. A discovered grasp configuration is saved along with the objects shape. When the same object is presented again in a different position and orientation, and the system recognizes its shape, the grasp information is retrieved and transformed to match the position and orientation of the object, so it can be picked in the first trial. The approach makes fewer assumptions, and requires less prior information about objects, than nonlearning grasp-determination methods. The presented method was implemented in a system with a robot and a servo-controlled two-finger gripper. Several examples of its operation are reported.<<ETX>>

Gesture based 3D man-machine interaction using a single camera

This paper describes a new gesture based input interface system that allows users to control both 2D and 3D applications using simple hand gestures. Using a single camera attached to the computer, the system tracks the users hand in three dimensions and computes up to four parameters in real-time (60 Hz). The system recognizes three gestures that can be interpreted as discrete commands to applications. This system is an off-shoot of an earlier system called Gesture VR that requires multiple cameras. Since the new system uses a single video source it can run readily on a standard home computer equipped with an inexpensive camera and is, therefore, accessible to most users. The system can be used with applications that require 2D and 2D interactions. Examples discussed in this paper include 3D virtual fly-throughs, graphical scene composers and video games.

Visual interface for conducting virtual orchestra

A real-time visual recognition system, that enables a human conductor to control an electronic orchestra using gestures of a traditional conductors baton, is described. The positions of the baton and conductors hand are identified in a sequence of images from a pair of cameras, and tracked in 3D space. Gestures defining the musical beat are detected in the batons trajectory, and conveyed to a sound synthesis system, as events that control the tempo and the phase of the music. Parameters that can enable the control of the volume of sound are computed from the range variations of the baton and the hand. The systems response is nearly instantaneous. The beat detection is reliable, and precise enough to be used by a professional conductor. It has been used to conduct an integrated electronic performance that combined a synthesized orchestra and animated ballet.