Francesca D'Errico

Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing

Social Signal Processing is the research domain aimed at bridging the social intelligence gap between humans and machines. This paper is the first survey of the domain that jointly considers its three major aspects, namely, modeling, analysis, and synthesis of social behavior. Modeling investigates laws and principles underlying social interaction, analysis explores approaches for automatic understanding of social exchanges recorded with different sensors, and synthesis studies techniques for the generation of social behavior via various forms of embodiment. For each of the above aspects, the paper includes an extensive survey of the literature, points to the most important publicly available resources, and outlines the most fundamental challenges ahead.

Social Signal Processing: The Research Agenda

The exploration of how we react to the world and interact with it and each other remains one of the greatest scientific challenges. Latest research trends in cognitive sciences argue that our common view of intelligence is too narrow, ignoring a crucial range of abilities that matter immensely for how people do in life. This range of abilities is called social intelligence and includes the ability to express and recognise social signals produced during social interactions like agreement, politeness, empathy, friendliness, conflict, etc., coupled with the ability to manage them in order to get along well with others while winning their cooperation. Social Signal Processing (SSP) is the new research domain that aims at understanding and modelling social interactions (human-science goals), and at providing computers with similar abilities in human-computer interaction scenarios (technological goals). SSP is in its infancy, and the journey towards artificial social intelligence and socially-aware computing is still long. This research agenda is a twofold, a discussion about how the field is understood by people who are currently active in it and a discussion about issues that the researchers in this formative field face.

Social signals: a framework in terms of goals and beliefs

The paper defines the notion of social signal, in terms of a cognitive model of mind and social interaction, as a communicative or informative signal or a cue that directly or indirectly provides information about “social facts”: social interactions, social emotions, social attitudes, evaluations and stances, social relations, and social identities. This notion is compared with other connected notions in Semiotics, Ethology, and Psychology, several types of informative and communicative signals and cues are exemplified, and their process of production and interpretation is overviewed, while considering the role of context and previous knowledge in it. The implications of this model are outlined for the construction of systems for the analysis of social signals and their simulation in Virtual Agents.

Dominance signals in debates

The paper analyzes the signals of dominance in different modalities displayed during TV talk shows and debates. Dominance is defined, according to a model in terms of goals and beliefs, as a persons having more power than others. A scheme is presented for the annotation of signals of dominance in political debates: based on the analysis of videotaped data, a typology is proposed of strategies to convey dominance, and the corresponding signals are overviewed. Strategies range from the aggressive ones of imperiousness, judgement, invasion, norm violation and defiance, to the more subtle touchiness and victimhood, ending up with haughtiness, irony and ridicule, easiness, carelessness and assertiveness.

The Perception of Charisma from Voice: A Cross-Cultural Study

This paper provides an overview of previous works on the acoustic parameters of charismatic voice and illustrates MASCharP, a scale for measuring the perception of charisma in voice. A study is then presented on the perception of charisma through the temporal and pitch structure of the voices of an Italian and a French politician. Results show some cultural differences in charisma perception and how acoustic features such as pitch (normal, higher, or lower) and types of pauses (short or long) can affect the Proactive-Attractive and Calm-Benevolent dimensions of charisma. The same dimension of charisma can be conveyed by different acoustic correlates of voice by connecting them to the dimension of leader extraversion-introversion.

How Charisma Is Perceived from Speech: A Multidimensional Approach

A leaders charisma is conveyed by various multiple aspects of his perceivable behavior among which the acoustic-prosodic characteristics of speech. We present here a study on the perception of charisma in political speech that aims to investigate the notion of charisma and to validate a theoretical framework on a multidimensional scale of charisma perception. The study points out that a multidimensional approach of charisma allows to better analyze which factors are related to specific aspects of speech. We finally clustered the charismatic voice of an Italian political leader in three factors: Proactive-Attracting, Benevolent-Competent and Authoritarian.

Social signals: A psychological perspective

This chapter defines social signals as communicative or informative signals that directly or indirectly provide meanings concerning social interactions, social attitudes, social relationships and social emotions, and presents classical literature and recent research on social signals in the domains of cognitive and social psychology. In terms of the modalities exploited, it overviews some cases of gestures, head movements, and gaze items as social signals. In terms of the meanings conveyed, special focus is devoted to persuasion and agreement, the former seen as a way to influence others’ attitudes, the latter as a possible outcome of a persuasive process: the persuasive import of gesture and gaze, and the function of words and body signals in agreement are highlighted. In the realm of social relationships, the relation of dominance and the blatant and subtle signals that convey it in political debates are illustrated; finally the nature and function of the social emotions of shame, pride and enthusiasm are described, and some studies are presented on their multimodal communication.

Towards the Parody Machine. Qualitative Analysis and Cognitive Processes in the Parody of a Politician

The paper presents a model of parody, viewed as a distorted imitation of a text, discourse, behavior or trait of a person performed in order to elicit laughter. Focusing on the parody of politicians as a way to discredit them for persuasive purposes, a qualitative analysis is presented of the parody of a Mayor of Rome by an Italian comedian. The role of allusion in parody, and its consequent dependency on culturally shared knowledge are highlighted, and the function of ridicule as a form of moralistic aggression is stressed. Finally a first flash of a procedure for the construction of a parody machine is provided, based on the cognitive process of parody in humans.

Conceptual analysis of social signals: the importance of clarifying terminology

As a burgeoning field, Social Signal Processing (SSP) needs a solid grounding in the disciplines that have developed important concepts in the study of communication. However, the number and diversity of terms developed in linguistics, psychology, and the behavioural sciences may seem confusing for scholars who are not versed in the subtleties of conceptual analysis and theoretical developments. Indeed, different disciplines sometimes use the same term to mean different things or, conversely, use different terms to mean the same thing. The goals of this article are to present an overview of the different concepts developed in the various disciplines that studied animal and human communication, and to understand the differences and commonalities between concepts emerging from these disciplines. We conclude that such an understanding will greatly improve the efficiency of pluridisciplinary research projects, for the advancement of SSP requires that we look at the complexity of communication from different angles.

Discrediting Body. A Multimodal Strategy to Spoil the Other's Image

In political persuasion the persuader, beside bearing logical arguments and triggering emotions, must present ones own image ones ethos of a credible and reliable person, by enhancing three dimensions of it: competence, benevolence and dominance. In a parallel way, s/he may cast discredit on the opponent by criticizing, accusing or insulting on the same three dimensions. The work provides a description and a typology of multimodal discrediting moves focusing on the discrediters multimodal behavior. Based on an Italian corpus of political debates the analysis points out which facial expressions, gaze behavior, gestures, postures and prosodic features are used to convey discredit concerning the three target features of competence, benevolence and dominance. The qualitative study of discredit signals highlights the close relation with emotions and in this sense particular attention is given to them as triggers of the Audiences evaluationThis book constitutes the thoroughly refereed post-conference proceedings of the International Workshop on Multimodal Communication in Political Speech: Shaping Minds and Social Actions, held in Rome, Italy, during November 10-12, 2010. The 16 regular papers were carefully reviewed and selected from 33 submissions and presented with three key-notes. The purpose of the Political Speech workshops is to provide a forum for discussing research areas of persuasive agents and social signal processing. This book covers topics on multimodal aspects of political communication, including persuasion, fallacies, racist discourse, as well as music, autobiographic memories, metonymies, dominant postures, rhetorical strategies, interruptions, intonation, and voice appeal.