Robert Rae

Guiding attention for grasping tasks by gestural instruction: the GRAVIS-robot architecture

A major goal for the realization of a new generation of intelligent robots is the capability of instructing work tasks by interactive demonstration. To make such a process efficient and convenient for the human user requires that both the robot and the user can establish and maintain a common focus of attention. We describe a hybrid architecture that combines neural networks and finite stale machines into a flexible framework for controlling the behaviour of a vision based robot called GRAVIS-robot (Gestural Recognition Active Vision System robot). It consists of a binocular camera head, a 6 DOF robot arm and a 9 DOF multifingered hand. We focus primarily on nonverbal communication based on gestural commands of a human instructor which will at a later stage be complemented by spoken instructions.

Image Based Recognition of Graze Direction Using Adaptive Methods

Human-machine interfaces based on gaze recognition can greatly simplify the handling of computer applications. However, most of the existing systems have problems with changing environments and different users. As a solution we use (i) adaptive components which can be trained online and (ii) detect common facial features, i.e. eyes, nose and mouth, for gaze recognition. In a first step an adaptive color histogram segmentation method roughly determines the region of interest including the user’s face. Within this region we then use a hierarchical recognition approach to detect the facial features. In the last stage of our system these feature positions are used to estimate gaze direction by detailed analysis of the eye region. We achieve an average precision of 1.5 ‡ for the gaze pan and 2.5 ‡ for the tilt angle while the user looks on a computer screen. The system runs at a rate of one frame per second on a common workstation.

Integrating context-free and context-dependent attentional mechanisms for gestural object reference

Abstract.We present a vision system for human-machine interaction based on a small wearable camera mounted on glasses. The camera views the area in front of the user, especially the hands. To evaluate hand movements for pointing gestures and to recognise object references, an approach to integrating bottom-up generated feature maps and top-down propagated recognition results is introduced. Modules for context-free focus of attention work in parallel with the hand gesture recognition. In contrast to other approaches, the fusion of the two branches is on the sub-symbolic level. This method facilitates both the integration of different modalities and the generation of auditory feedback.

Facial Feature Detection Using Neural Networks

Many human-machine interfaces based on face gestures are strongly user-dependent. We want to overcome this limitation by using common facial features like eyes, nose and mouth for gaze recognition. In a first step an adaptive color histogram segmentation method roughly determines the region of interest including the users face. Within this region we then use a hierarchical recognition approach to detect the facial features. Our system is based on a what-where neural network architecture and allows a fast and robust recognition rate. In the future we intend to use the conspicuous features for estimation of gaze directions.

Kommunizierende Agenten: Gestische und natürlichsprachliche Interaktion

SummaryAn important scientific method within cognitive science consists in the synthesis of cognitive abilities, of forms of behavior by developing specific artificial agents. Many current approaches make use of the notion of an agent in order to develop concepts of cognitive behavior on different levels of abstraction. Basic properties of agents are: reactivity, autonomy, goal directed activity, and communication. This contribution examines the communicative aspect, i.e. the interaction by gesture or language and their integration, e.g. in identifying referents. Since we conceive communicating agents as systems able to synthesize such interactions as well as their integration, this will be illustrated with respect to two artificial systems. The GRAVIS system detects objects as well as pointing gestures of an instructor, and the camera agent is able to focus on specific objects. The CoRA system processes situated natural language instructions, and the simulated robot agent is able to integrate the use of language, perception and action. Finally we propose an integration of both approaches.ZusammenfassungEin wichtiger Forschungsansatz innerhalb der Kognitionswissenschaft besteht in der Synthese kognitiver Fähigkeiten und Verhaltensweisen durch den Bau geeigneter, künstlicher Systeme. Viele der in jüngerer Zeit verfolgten Ansätze nutzen dabei den Begriff des Agenten, um kognitives Verhalten auf unterschiedlichen Abstraktionsebenen zu konzeptualisieren. Grundeigenschaften von Agenten sind u.a. Reaktivität, Autonomie, Zielgerichtetheit und Kommunikationsfähigkeit. In dem vorliegenden Beitrag interessieren wir uns für die Ebene kommunikativen Verhaltens, bei der Gestik und Sprache zur Vermittlung eines Sachverhaltes zusammenwirken, und verstehen unter einem kommunizierenden Agenten ein System, das für solches Verhalten einen nennenswerten Kompetenzausschnitt realisiert. Wir stellen hierzu beispielhaft zwei Systeme vor. Das System GRAVIS demonstriert die Referenzierung von Objekten im Blickfeld eines Kamerakopfes durch Handzeigegestik eines Instrukteurs. Das System CoRA demonstriert die Verarbeitung sprachlicher Handlungsanweisungen im Kontext der Kooperation mit einem simulierten Roboter. Abschließend wird die Integration der beiden Ansätze diskutiert.

Communicating agents: Gesture and natural language interaction

SummaryAn important scientific method within cognitive science consists in the synthesis of cognitive abilities, of forms of behavior by developing specific artificial agents. Many current approaches make use of the notion of an agent in order to develop concepts of cognitive behavior on different levels of abstraction. Basic properties of agents are: reactivity, autonomy, goal directed activity, and communication. This contribution examines the communicative aspect, i.e. the interaction by gesture or language and their integration, e.g. in identifying referents. Since we conceive communicating agents as systems able to synthesize such interactions as well as their integration, this will be illustrated with respect to two artificial systems. The GRAVIS system detects objects as well as pointing gestures of an instructor, and the camera agent is able to focus on specific objects. The CoRA system processes situated natural language instructions, and the simulated robot agent is able to integrate the use of language, perception and action. Finally we propose an integration of both approaches.ZusammenfassungEin wichtiger Forschungsansatz innerhalb der Kognitionswissenschaft besteht in der Synthese kognitiver Fähigkeiten und Verhaltensweisen durch den Bau geeigneter, künstlicher Systeme. Viele der in jüngerer Zeit verfolgten Ansätze nutzen dabei den Begriff des Agenten, um kognitives Verhalten auf unterschiedlichen Abstraktionsebenen zu konzeptualisieren. Grundeigenschaften von Agenten sind u.a. Reaktivität, Autonomie, Zielgerichtetheit und Kommunikationsfähigkeit. In dem vorliegenden Beitrag interessieren wir uns für die Ebene kommunikativen Verhaltens, bei der Gestik und Sprache zur Vermittlung eines Sachverhaltes zusammenwirken, und verstehen unter einem kommunizierenden Agenten ein System, das für solches Verhalten einen nennenswerten Kompetenzausschnitt realisiert. Wir stellen hierzu beispielhaft zwei Systeme vor. Das System GRAVIS demonstriert die Referenzierung von Objekten im Blickfeld eines Kamerakopfes durch Handzeigegestik eines Instrukteurs. Das System CoRA demonstriert die Verarbeitung sprachlicher Handlungsanweisungen im Kontext der Kooperation mit einem simulierten Roboter. Abschließend wird die Integration der beiden Ansätze diskutiert.