Timo Sowa

Imitation Games with an Artificial Agent: From Mimicking to Understanding Shape-Related Iconic Gestures

We describe an anthropomorphic agent that is engaged in an imitation game with the human user. In imitating natural gestures demonstrated by the user, the agent brings together gesture recognition and synthesis on two levels of representation. On the mimicking level, the essential form features of the meaning-bearing gesture phase (stroke) are extracted and reproduced by the agent. Meaning-based imitation requires extracting the semantic content of such gestures and re-expressing it with possibly alternative gestural forms. Based on a compositional semantics for shape-related iconic gestures, we present first steps towards this higher-level gesture imitation in a restricted domain.

Temporal Symbolic Integration Applied to a Multimodal System Using Gestures and Speech

This paper presents a technical approach for temporal symbol integration aimed to be generally applicable in unimodal and multimodal user interfaces. It draws its strength from symbolic data representation and an underlying rule-based system, and is embedded in a multiagent system. The core method for temporal integration is motivated by findings from cognitive science research. We discuss its application for a gesture recognition task and speech-gesture integration in a Virtual Construction scenario. Finally an outlook of an empirical evaluation is given.

Interpretation of Shape-Related Iconic Gestures in Virtual Environments

So far, approaches towards gesture recognition focused mainly on deictic and emblematic gestures. Iconics, viewed as iconic signs in the sense of Peirce, are different from deictics and emblems, for their relation to the referent is based on similarity. In the work reported here, the breakdown of the complex notion of similarity provides the key idea towards a computational model of gesture semantics for iconic gestures. Based on an empirical study, we describe first steps towards a recognition model for shape-related iconic gestures and its implementation in a prototype gesture recognition system. Observations are focused on spatial concepts and their relation to features of iconic gestural expressions. The recognition model is based on a graph-matching method which compares the decomposed geometrical structures of gesture and object.

A Computational Model for the Representation and Processing of Shape in Coverbal Iconic Gestures 1

When humans describe the shape of objects, they often use iconic gestures to depict what they want to convey to a listener. Gesturing gives them the ability to express spatial concepts directly in the spatial medium and thus provides an important communicative resource for spatial language. In order to harvest this resource in language comprehension systems, the composite signal conveyed in two different media has to be re-integrated to a common, unified meaning. In a corpus study, we examined the morphological variety of shape-related iconic gestures and the kind of shape information they express. We distinguish four sub-types of iconic gestures and show that the most frequent type, called dimensional gestures, and the lexical affiliates they co-occur with, contain information about an object’s spatial extent, the course of its boundary, and the spatial relations between object parts. An analysis of the verbal utterances shows that adjectives and nouns are predominant among the lexical affiliates in our scenario. Based on the empirical results, a computational model for the representation and processing of multimodal shape descriptions is proposed.

Multimodale Interaktion in der Virtuellen Realität

Virtuelle Realitat oder Virtual Reality (VR) bezeichnet ein neuartiges Kommunikationsmedium, das die unmittelbare Wechselwirkung des Menschen mit raumlich organisierten rechnergenerierten Darstellungen erlaubt. Verbunden mit korperlich verankerter Interaktion finden insbesondere gestische Eingaben starkes Interesse. Dieser Beitrag gibt einen Uberblick uber Forschungsarbeiten im Labor fur Kunstliche Intelligenz und Virtuelle Realitat an der Universitat Bielefeld, mit denen Grundlagen fur den Einsatz gestischer und sprachlicher Interaktionstechniken entwickelt werden; als Erprobungsdomane dient ein Szenario des virtuellen Konstruierens. Fur die schnelle Erfassung komplexer Hand-Armgesten werden derzeit Datenhandschuhe und Korper-Tracker eingesetzt. Die Auswertung erfolgt mit wissensbasierten Ansatzen, die atomare Formelemente der Gestik symbolisch beschreiben und zu groseren Einheiten zusammensetzen. Ein zweites Thema ist die multimodale Interaktion durch sprachlich-gestische Eingaben, z.B. wenn auf einen Gegenstand gezeigt („dieses Rohr“) oder eine Drehrichtung („so herum“) signalisiert wird. Schlieslich wird dargestellt, wie die Ansatze zur Formbeschreibung von Gesten fur die Synthese naturlich wirkender gestischer Ausgaben mit einer artikulierten, anthropomorphen Figur ubertragen werden konnen, die in laufenden Arbeiten mit Sprachausgaben koordiniert werden.