Roderick Cowie

Emotion recognition in human-computer interaction

Two channels have been distinguished in human interaction: one transmits explicit messages, which may be about anything or nothing; the other transmits implicit messages about the speakers themselves. Both linguistics and technology have invested enormous efforts in understanding the first, explicit channel, but the second is not as well understood. Understanding the other partys emotions is one of the key tasks associated with the second, implicit channel. To tackle that task, signal processing and analysis techniques have to be developed, while, at the same time, consolidating psychological and linguistic analyses of emotion. This article examines basic issues in those areas. It is motivated by the PKYSTA project, in which we aim to develop a hybrid system capable of using information from faces and voices to recognize peoples emotions.

The SEMAINE corpus of emotionally coloured character interactions

We have recorded a new corpus of emotionally coloured conversations. Users were recorded while holding conversations with an operator who adopts in sequence four roles designed to evoke emotional reactions. The operator and the user are seated in separate rooms; they see each other through teleprompter screens, and hear each other through speakers. To allow high quality recording, they are recorded by five high-resolution, high framerate cameras, and by four microphones. All sensor information is recorded synchronously, with an accuracy of 25 μs. In total, we have recorded 20 participants, for a total of 100 character conversational and 50 non-conversational recordings of approximately 5 minutes each. All recorded conversations have been fully transcribed and annotated for five affective dimensions and partially annotated for 27 other dimensions. The corpus has been made available to the scientific community through a web-accessible database.

Prosodic Characteristics of Skilled Reading: Fluency and Expressiveness in 8- 10-year-old Readers

Statistical methods of describing prosody were used to study fluency, expressiveness and their relationship among 8—10-year-old readers. 67 children were rated on fluency and expressiveness. The two were partially independent in the full sample: expressiveness rarely occurred without fluency, but fluency occurred without expressiveness. A balanced subsample of 24 was selected for closer instrumental and statistical analysis. There were robust relationships between expressiveness and variables associated with pitch mobility; and between fluency and measures associated with temporal organization. Interactions indicated that the relationships were not simple. Differences between groups depended on sentence content and position —expressive readers distinguished sentences more sharply according to content, and the groups diverged on some measures as the passage progressed. Also, measures associated primarily with either fluency or expression often showed secondary sensitivity to the other: temporal organization was associated with fluency, but worsened over time among inexpressive readers; and readers who were both fluent and expressive were distinctive in several respects. Some measures offer a basis for rules aimed at assigning individuals to skill categories, particularly the magnitude of pitch movements and reading time per syllable. The rules distinguish well among readers who were either at one of the extremes of skill, or fluent but inexpressive; it is harder to discriminate among the other readers (who have mixed skill patterns). The effects suggest psychological hypotheses about the underlying mechanisms.

Speech production in profound postlingual deafness.

Publisher Summary Deafness is a problem because of the barrier to communication that it creates. The effect of this barrier is to prevent the deaf from understanding what others say, but it might impede them in speaking intelligibly. This problem is known for those who are deaf from birth or early infancy, and who have to learn to speak with little or no knowledge of what language sounds like—the pre-lingually deaf. Deafness might afflict people who have learned to speak—the post-lingually deaf. This chapter focuses on the speech problems that the post-lingually deaf face. Their speech is considered on two main levels. On a descriptive level, the speech problems that are associated with the post-lingual deafness have been documented. The chapter discusses the mechanisms that might underlie these problems. To develop this kind of understanding, evidence on speech when feedback is reduced in other ways, has been presented.

Social Signal Processing: The Research Agenda

The exploration of how we react to the world and interact with it and each other remains one of the greatest scientific challenges. Latest research trends in cognitive sciences argue that our common view of intelligence is too narrow, ignoring a crucial range of abilities that matter immensely for how people do in life. This range of abilities is called social intelligence and includes the ability to express and recognise social signals produced during social interactions like agreement, politeness, empathy, friendliness, conflict, etc., coupled with the ability to manage them in order to get along well with others while winning their cooperation. Social Signal Processing (SSP) is the new research domain that aims at understanding and modelling social interactions (human-science goals), and at providing computers with similar abilities in human-computer interaction scenarios (technological goals). SSP is in its infancy, and the journey towards artificial social intelligence and socially-aware computing is still long. This research agenda is a twofold, a discussion about how the field is understood by people who are currently active in it and a discussion about issues that the researchers in this formative field face.

Building the databases needed to understand rich, spontaneous human behaviour

One of the motives for studying faces and gestures is the role that they play in spontaneous, socially rich interaction between humans. If computers are to interact with humans in that mode (or to analyse what they are doing in it), methods of interpreting the non-verbal signals that they use are critical. It is becoming clear that developing those methods requires databases whose complexity is of a very different order from those that are standard elsewhere. Samples cannot be generated to order, because acting does not reproduce the way features are distributed in spontaneous action. Data collections need to be very large, because there are extensive situational, individual, and cultural differences. It is a very large task to provide annotations that adequately capture the meaning of facial expressions or gestures. These problems are not insoluble, but sustained and well-directed efforts are needed to solve them.

Towards a conceptual framework of research on social signal processing

Social Signal Processing (SSP) as an emerging research area can draw on material from many disciplines, but it needs effective ways to organise the material. We propose a framework that integrates concepts, drawn primarily from psychology, but with input from other disciplines, in a way that indicates how they relate to SSP. We identify seven core constructs: state; indicators that convey info about it; process by which indicators are generated; types of inference involved in identifying states from indicators; perceptual issues, including recovery of indicators from flux of activity and accuracy of impressions about the state; the role of macro-context; and the different levels at which analysis may be couched (the individual, the dyad, the group, the organisation). These may or may not be reflected in the structure of an SSP system, but the natural subtlety and context-sensitivity of human communication makes it important that people designing systems should consider how they relate to its task, and decide how best to take account of them. The analysis works from simple models which consider only states and indicators to models which embrace all seven constructs. At each stage it points to the literatures which discuss the relevant issues. It is fully acknowledged that attempting such a synthesis raises many problems (not least of terminology), and that alternative frameworks deal at least as well with subsets of the issues. It remains to be seen whether they could be extended to cover a comparable range.

Politeness and social signals

In the literature, politeness has been researched within many disciplines. Although Brown and Levinson’s theory of politeness (1978, 1987) is often cited, it is primarily a linguistic theory and has been criticized for its lack of generalizability to all cultures. Consequently, there is a need for a more comprehensive approach to understand and explain politeness. We suggest applying a social signal framework that considers politeness as a communicative state. By doing so, we aim to unify and explain politeness and its corresponding research and identify further research needed in this area.