Federica Cavicchio

Modifications of phonetic labial targets in emotive speech: effects of the co-production of speech and emotions

This paper describes how the visual and acoustic characteristics of some Italian phones (/’a/, /b/, /v/) are modifled in emotive speech by the expression of joy, surprise, sadness, disgust, anger, and fear. In this research we speciflcally analyze the interaction between labial conflgurations, peculiar to each emotion, and the articulatory lip movements of the Italian vowel /’a/ and consonants /b/ and /v/, deflned by phonetic-phonological rules. This interaction was quantifled examining the variations of the following parameters: lip opening, upper and lower lip vertical displacements, lip rounding, anterior/posterior movements (protrusion) of upper lip and lower lip, left and right lip corner horizontal displacements, left and right corner vertical displacements, and asymmetry parameters calculated as the difierence between right and left corner position along the horizontal and the vertical axes. Moreover, we present the correlations between articulatory data and the spectral features of the co-produced acoustic signal.

Irony in a judicial debate: analyzing the subtleties of irony while testing the subtleties of an annotation scheme

Irony has been studied by famous scholars across centuries, as well as more recently in cognitive and pragmatic research. The prosodic and visual signals of irony were also studied. Irony is a communicative act in which the Sender’s literal goal is to communicate a meaning x, but through this meaning the Sender has the goal to communicate another meaning, y, which is contrasting, sometimes even opposite, to meaning x. In this case we have an antiphrastic irony. So an ironic act is an indirect speech act, in that its true meaning, the one really intended by the Sender, is not the one communicated by the literal meaning of the communicative act: it must be understood through inferences by the Addressee. The ironic statement may concern an event, object or person, and in this case, the Addressee, or a third person, or even the Sender itself (Self-irony). In this paper we define irony in terms of a goal and belief view of communication, and show how the annotation scheme, the Anvil-Score, and illustrate aspects of its expressive power by applying it to a particular case: ironic communication in a judicial debate.

Modifications of Speech Articulatory Characteristics in the Emotive Speech

The aim of the research is the phonetic articulatory description of emotive speech achievable studying the labial movements, which are the product of the compliance with both the phonetic-phonological constraints and the lip configuration required for the visual encoding of emotions. In this research we analyse the interaction between labial configurations, peculiar to six emotions (anger, disgust, joy, fear, surprise and sadness), and the articulatory lip movements defined by phonetic-phonological rules, specific to the vowel /’a/ and consonants /b/ and /v/.

Multimodal corpora annotation: validation methods to assess coding scheme reliability

Many multimodal corpora have been collected and annotated in the last years. Unfortunately, in many cases most of the multimodal coding schemes have been shown not to be reliable. This poor reliability may be caused either by the nature of multimodal data or by the nature of statistic methods to assess reliability. In this paper we will review the statistical measures currently used to assess agreement on multimodal corpora annotation. We will also propose alternative statistical methods to the well known kappa statistics.

Coordinating the Generation of Signs in Multiple Modalities in an Affective Agent

In order to be believable, embodied conversational agents (ECAs) must show expression of emotions in a consistent and natural looking way across modalities. The ECA has to be able to display coordinated signs of emotion during realistic emotional behaviour. Such a capability requires one to study and represent emotions and coordination of modalities during non-basic realistic human behaviour, to define languages for representing such behaviours to be displayed by the ECA, to have access to mono-modal representations such as gesture repositories. This chapter is concerned about coordinating the generation of signs in multiple modalities in such an affective agent. Designers of an affective agent need to know how it should coordinate its facial expression, speech, gestures and other modalities in view of showing emotion. This synchronisation of modalities is a main feature of emotions.

The Rovereto Emotion and Cooperation Corpus: a new resource to investigate cooperation and emotions

The Rovereto Emotion and Cooperation Corpus (RECC) is a new resource collected to investigate the relationship between cooperation and emotions in an interactive setting. Previous attempts at collecting corpora to study emotions have shown that this data are often quite difficult to classify and analyse, and coding schemes to analyse emotions are often found not to be reliable. We collected a corpus of task-oriented (MapTask-style) dialogues in Italian, in which the segments of emotional interest are identified using psycho-physiological indexes (Heart Rate and Galvanic Skin Conductance) which are highly reliable. We then annotated these segments in accordance with novel multimodal annotation schemes for cooperation (in terms of effort) and facial expressions (an indicator of emotional state). High agreement was obtained among coders on all the features. The RECC corpus is to our knowledge the first resource with psycho-physiological data aligned with verbal and nonverbal behaviour data.

(Non)cooperative Dialogues: The Role of Emotions

Objective: The effect of emotion on (non)co-operation in unscripted, ecological communication is investigated. Background: The participants in an interaction are generally cooperative in that, for instance, they tend to reduce the chance of misunderstandings in communication. However, it is also clear that cooperation is not complete. Positive and negative emotional states also appear to be connected to the participants’ commitment to cooperate or not, respectively. So far, however, it has proven remarkably difficult to test this because of the lack of entirely objective measurements of both cooperation levels and emotional responses. Method: In this article, the authors present behavioral methods and coding schemes for analyzing cooperation and (surface) indicators of emotions in face-to-face interactions and show that they can be used to study the correlation between emotions and cooperation effectively. Results: The authors observed large negative correlations between heart rate and cooperation, and a group of facial expressions was found to be predictive of the level of cooperation of the speakers. Conclusion: It is possible to develop reliable methods to code for cooperation, and with such coding schemes it is possible to confirm the commonsense prediction that noncooperative behavior by a conversational participant affects the other participant in ways that can be measured quantitatively. Application: These results shed light on an aspect of interaction that is crucial to building adaptive systems able to measure cooperation and to respond to the user’s affective states. The authors expect their methods to be applicable to building and testing such interaction systems.