Gaëlle Ferré

Creating and exploiting multimodal annotated corpora: the ToMA project

The paper presents a project aiming at collecting, annotating and exploiting a dialogue corpus from a multimodal perspective. The goal of the project is the description of the different parameters involved in a natural interaction process. Describing such complex mechanism requires corpora annotated in different domains. This paper first presents the corpus and the scheme used in order to annotate the different domains that have to be taken into consideration, namely phonetics, morphology, syntax, prosody, discourse and gestures. Several examples illustrating the interest of such a resource are then proposed.

A multimodal approach to markedness in spoken French

This study aims at examining the links between marked structures in the syntactic and prosodic domains (fronting and focal accent), and the way the two types of contrast can be reinforced by gestures. It was conducted on a corpus of 1h30 of spoken French, involving three pairs of speakers in dialogues. Results show that although the tendency is for marked constructions both in syntax and prosody not to be reinforced by gestures, there is still a higher proportion of gesture reinforcing with prosodic marking than with syntactic fronting. The paper describes which eyebrow and head movements as well as hand gestures are more liable to accompany the two operations. Beyond these findings, the study gives an insight into the current models proposed in the literature for gesture-speech production.

Functions of three open-palm hand gestures

Abstract This study proposes an analysis of some pragmatic gestures with three types of open-palm gestures: beats and two instances of the hand flip, elsewhere called the ‘palm-up open-hand’ gesture (Müller, 2004; Cienki & Müller, 2008). Drawing upon three different corpora (political speeches made at the European Parliament, a television show in which the role of this parliament is presented and a corpus of conversational speech recorded in a lab), it proposes an analysis into prosodic, discursive and modal gestures. The paper - through the discussion of particular examples - will address the issues of the type of prosodic and discourse units which are marked by these gestures. Using the same methodological framework, the type of grammatical modality conveyed by open-palm gestures will also be considered.

The Corpus of Interactional Data: A Large Multimodal Annotated Resource

The availability of annotated datasets had been steadily growing for written language and benefited to linguistic studies and natural language processing. The situation for face-to-face spontaneous conversation is more contrasted for several reasons: technicalities in handling raw data (split across several sources and medias), need for a often difficult and time-consuming transcription, large variety of annotation that can be performed. We propose in this chapter a complete annotation workflow, starting from raw data (speech and video) to a richly annotated dataset with many linguistic information (morpho-syntax, prosody, gesture studies, discourse analysis). Our approach consisted in gathering experts from the different domains and work together on the establishment of an abstract schema encoded with types feature structures. We detail how the annotation workflow had been used for developing of a richly annotated version of the Corpus of Interactional Data. The corpus as well as the annotation described here are available through the Speech and Language Data Repository.