Hannes Rieser

Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications

Communicating face-to-face, interlocutors frequently produce multimodal meaning packages consisting of speech and accompanying gestures. We discuss a systematically annotated speech and gesture corpus consisting of 25 route-and-landmark-description dialogues, the Bielefeld Speech and Gesture Alignment corpus (SaGA), collected in experimental face-to-face settings. We first describe the primary and secondary data of the corpus and its reliability assessment. Then we go into some of the projects carried out using SaGA demonstrating the wide range of its usability: on the empirical side, there is work on gesture typology, individual and contextual parameters influencing gesture production and gestures’ functions for dialogue structure. Speech-gesture interfaces have been established extending unification-based grammars. In addition, the development of a computational model of speech-gesture alignment and its implementation constitutes a research line we focus on.

Deixis: how to determine demonstrated objects using a pointing cone

We present a collaborative approach towards a detailed understanding of the usage of pointing gestures accompanying referring expressions. This effort is undertaken in the context of human-machine interaction integrating empirical studies, theory of grammar and logics, and simulation techniques. In particular, we attempt to measure the precision of the focussed area of a pointing gesture, the so-called pointing cone. The pointing cone serves as a central concept in a formal account of multi-modal integration at the linguistic speech-gesture interface as well as in a computational model of processing multi-modal deictic expressions.

On factoring out a gesture typology from the Bielefeld speech-and-gesture-alignment corpus (SAGA)

People communicate multimodally. Most prominently, they co-produce speech and gesture. How do they do that? Studying the interplay of both modalities has to be informed by empirically observed communication behavior. We present a corpus built of speech and gesture data gained in a controlled study. We describe 1) the setting underlying the data; 2) annotation of the data; 3) reliability evalution methods and results; and 4) applications of the corpus in the research domain of speech and gesture alignment.

Interactive Gesture in Dialogue: a PTT Model

Gestures are usually looked at in isolation or from an intra-propositional perspective essentially tied to one speaker. The Bielefeld multi-modal Speech-And-Gesture-Alignment (SAGA) corpus has many interactive gestures relevant for the structure of dialogue (Rieser 2008, 2009). To describe them, a dialogue theory is needed which can serve as a speech-gesture interface. PTT (Poesio and Traum 1997, Poesio and Rieser submitted a) can do this job in principle, how this can be achieved is the main topic of this paper. As a precondition, the empirical research procedure from systematic corpus annotation via gesture typology to a partial ontology for gestures is described. It is then explained how PTT is extended to provide an incremental modelling of speech plus gesture in an assertion-acknowledgement adjacency pair where grounding between dialogue participants is obtained through gesture.

On Grounding Natural Kind Terms in Human-Robot Communication

Our contribution situates Human-Robot Communication, especially the grounding of Natural Kind Terms, in the interface of Artificial Intelligence, Cognitive Psychology, Philosophy, Robotics and Semantics. We investigate whether a robot can be grounded in the sense favoured in Artificial Intelligence and Philosophy.We thus extend the notion of grounding to social symbol grounding using an interactive perspective addressing the question how grounding can be achieved in detail in interaction. For the acquisition of Natural Kind Terms we establish the notions of foundational common ground and foundational grounding in contrast to the established common ground and grounding. We introduce the robot setting used and provide a deep evaluation of a tutorial dialogue between a user and the robot. We investigate these Human-Robot Communication data from an ethno-methodological and an “omniscient” perspective (the latter amounting to consideration of automatic speech recognition results) and test whether these perspectives matter for analysing grounding. We show that the robot has acquired a partial concept of a Natural Kind Term—represented by statistics over visual object features—and that this is shared knowledge, hence the first step of a grounding sequence. Finally, we argue that grounding of robots can be achieved and extended to situated structures of considerable complexity.

Augenbewegungen als kognitionswissenschaftlicher Forschungsgegenstand

Wir gehen in dieser Arbeit von der Hypothese aus, das die Augenbewegungsforschung eine Moglichkeit darstellt, Aufschlusse uber die Organisation kognitiver Strukturen und die Dynamik kognitiver Prozesse zu erhalten. Die Fruchtbarkeit dieser Hypothese zeigen wir anhand der Beschreibung einer Reihe von Experimenten und Studien, die in Projekten des SFB 360 der Universitat Bielefeld, „Situierte Kunstliche Kommunikatoren“, durchgefuhrt wurden. Einleitend werden die verwendete Eyetracker-Technologie und die praktizierten Methoden der Aufbereitung von Augenbewegungsdaten, Clusteranalyse und attentional landscapes, beschrieben.

How do iconic gestures convey visuo-spatial information? bringing together empirical, theoretical, and simulation studies

We investigate the question of how co-speech iconic gestures are used to convey visuo-spatial information in an interdisciplinary way, starting with a corpus-based empirical and theoretical perspective on how a typology of gesture form and a partial ontology of gesture meaning are related. Results provide the basis for a computational modeling approach that allows us to simulate the production of speaker-specific gesture forms to be realized with virtual agents. An evaluation of our simulation results and our methodology shows that the model is able to successfully approximate human gestural behavior use of iconic gestures, and moreover, that gestural behavior can improve how humans rate a virtual agent in terms of eloquence, competence, human-likeness, or likeability.

Kohärenzkonstitution im gesprochenen Deutsch

Das Projekt “Theoretische Grundlagen und Simulation von Prozessen der Koharenzkonstitution im gesprochenen Deutsch” hat sich schwerpunktmasig mit dem Phanomen der “Reparatur” auseinandergesetzt. Reparaturen entstehen in der gesprochenen Sprache unter anderem dann, wenn einem Sprecher ein tatsachlicher oder ein vermeintlicher Fehler unterlauft, den er, sofern er ihn bemerkt, zumeist zu reparieren sucht. Ein typisches Beispiel einer Reparatur findet sich in (3.1) .

Depiktionsmetonymien und ihre Auflösung im Diskurs

Gegenstand dieses Beitrags sind Formen der nicht-wortlichen Interpretation. Es geht dabei um Depiktionsmetonymien, also um Ausdrucke wie Flugzeug,Rumpf oder Fahrgestell, die verwandt werden, um bestimmte Spielzeug-Aggregate zu benennen. Dazu diskutieren wir zunachst die Depiktionsleistung derartiger Aggregate im Rahmen einer dazu entwickelten semiotischen Theorie depiktionierender Objekte. Im Anschluss skizzieren wir den pragmatischen Hintergrund fur die Auflosung von Depiktionsmetonymien. Im Vordergrund stehen dabei die Funktion von Kontext und Dialoggeschichte sowie die Rolle verschiedener Wissensquellen. Anschliesend erortern wir die Rolle der Depiktionsleistung bei der Auflosung von Depiktionsmetonymien, wobei wir zwei unterschiedliche Beschreibungsansatze von Metonymien einander gegenuberstellen. Die Schlussdiskussion enthalt neben weiterfuhrenden Perspektiven Uberlegungen dazu, wie sich die hier behandelten Losungen zu denen in psychologisch motivierten Theorien verhalten.

Zur Semiotik von Repräsentationsrelationen. Eine Fallstudie

Magritte reflektierte in den Jahren 1927 bis 1935 Unterschiede zwischen Depiktionen (Bildern), Wortern und Gegenstanden. In diesem Zusammenhang entstand sein wohl bekanntestes Bild „Dies ist keine Pfeife“, nachfolgend in einer Version von 1948 wiedergegeben.