Dirk Heylen

Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing

Social Signal Processing is the research domain aimed at bridging the social intelligence gap between humans and machines. This paper is the first survey of the domain that jointly considers its three major aspects, namely, modeling, analysis, and synthesis of social behavior. Modeling investigates laws and principles underlying social interaction, analysis explores approaches for automatic understanding of social exchanges recorded with different sensors, and synthesis studies techniques for the generation of social behavior via various forms of embodiment. For each of the above aspects, the paper includes an extensive survey of the literature, points to the most important publicly available resources, and outlines the most fundamental challenges ahead.

Brain-Computer Interfacing and Games

Recently research into Brain-Computer Interfacing (BCI) applications for healthy users, such as games, has been initiated. But why would a healthy person use a still-unproven technology such as BCI for game interaction? BCI provides a combination of information and features that no other input modality can offer. But for general acceptance of this technology, usability and user experience will need to be taken into account when designing such systems. Therefore, this chapter gives an overview of the state of the art of BCI in games and discusses the consequences of applying knowledge from Human-Computer Interaction (HCI) to the design of BCI for games. The integration of HCI with BCI is illustrated by research examples and showcases, intended to take this promising technology out of the lab. Future research needs to move beyond feasibility tests, to prove that BCI is also applicable in realistic, real-world settings.

Generation of Facial Expressions from Emotion Using a Fuzzy Rule Based System

We propose a fuzzy rule-based system to map representations of the emotional state of an animated agent onto muscle contraction values for the appropriate facial expressions. Our implementation pays special attention to the way in which continuous changes in the intensity of emotions can be displayed smoothly on the graphical face. The rule system we have defined implements the patterns described by psychologists and researchers dealing with facial expressions of humans, including rules for displaying blends of expressions.

ParleE: An Adaptive Plan Based Event Appraisal Model of Emotions

We propose ParleE, a quantitative, flexible and adaptive model of emotions for a conversational agent in a multi-agent environment capable of multimodal communication. ParleE appraises events based on learning and a probabilistic planning algorithm. ParleE also models personality and motivational states and their role in determining the way the agent experiences emotion.

DIALOGUE-ACT TAGGING USING SMART FEATURE SELECTION; RESULTS ON MULTIPLE CORPORA

This paper presents an overview of our on-going work on dialogue-act classification. Results are presented on the ICSI, switchboard, and on a selection of the AMI corpus, setting a baseline for forthcoming research. For these corpora the best accuracy scores obtained are 89.27%, 65.68% and 59.76%, respectively. We introduce a smart compression technique for feature selection and compare the performance from a subset of the AMI transcriptions with AMI-ASR output for the same subset.

Social Signal Processing: The Research Agenda

The exploration of how we react to the world and interact with it and each other remains one of the greatest scientific challenges. Latest research trends in cognitive sciences argue that our common view of intelligence is too narrow, ignoring a crucial range of abilities that matter immensely for how people do in life. This range of abilities is called social intelligence and includes the ability to express and recognise social signals produced during social interactions like agreement, politeness, empathy, friendliness, conflict, etc., coupled with the ability to manage them in order to get along well with others while winning their cooperation. Social Signal Processing (SSP) is the new research domain that aims at understanding and modelling social interactions (human-science goals), and at providing computers with similar abilities in human-computer interaction scenarios (technological goals). SSP is in its infancy, and the journey towards artificial social intelligence and socially-aware computing is still long. This research agenda is a twofold, a discussion about how the field is understood by people who are currently active in it and a discussion about issues that the researchers in this formative field face.

Collaborative Practices that Support Creativity in Design

Design is a ubiquitous, collaborative and highly material activity. Because of the embodied nature of the design profession, designers apply certain collaborative practices to enhance creativity in their everyday work. Within the domain of industrial design, we studied two educational design departments over a period of eight months. Using examples from our fieldwork, we develop our results around three broad themes related to collaborative practices that support the creativity of design professionals: 1) externalization, 2) use of physical space, and 3) use of bodies. We believe that these themes of collaborative practices could provide new insights into designing technologies for supporting a varied set of design activities. We describe two conceptual collaborative systems derived from the results of our study.

The multiLis corpus - dealing with individual differences in nonverbal listening behavior

Computational models that attempt to predict when a virtual human should backchannel are often based on the analysis of recordings of face-to-face conversations between humans. Building a model based on a corpus brings with it the problem that people differ in the way they behave. The data provides examples of responses of a single person in a particular context but in the same context another person might not have provided a response. Vice versa, the corpus will contain contexts in which the particular listener recorded did not produce a backchannel response, where another person would have responded. Listeners can differ in the amount, the timing and the type of backchannels they provide to the speaker, because of individual differences - related to personality, gender, or culture, for instance. To gain more insight in this variation we have collected data in which we record the behaviors of three listeners interacting with one speaker. All listeners think they are having a one-on-one conversation with the speaker, while the speaker actually only sees one of the listeners. The context, in this case the speakers actions, is for all three listeners the same and they respond to it individually. This way we have created data on cases in which different persons show similar behaviors and cases in which they behave differently. With the recordings of this data collection study we can start building our model of backchannel behavior for virtual humans that takes into account similarities and differences between persons.

Virtual meeting rooms: from observation to simulation

Much working time is spent in meetings and, as a consequence, meetings have become the subject of multidisciplinary research. Virtual Meeting Rooms (VMRs) are 3D virtual replicas of meeting rooms, where various modalities such as speech, gaze, distance, gestures and facial expressions can be controlled. This allows VMRs to be used to improve remote meeting participation, to visualize multimedia data and as an instrument for research into social interaction in meetings. This paper describes how these three uses can be realized in a VMR. We describe the process from observation through annotation to simulation and a model that describes the relations between the annotated features of verbal and non-verbal conversational behavior. As an example of social perception research in the VMR, we describe an experiment to assess human observers’ accuracy for head orientation.

AFFECT IN TUTORING DIALOGUES

ABSTRACT This paper is about INES, an intelligent, multimodal tutoring environment, and how we build a tutor agent in the environment that tries to be sensitive to the mental state of the student that interacts with it. The environment was primarily designed to help students practice nursing tasks. For example, one of the implemented tasks is to give a virtual patient a subcutaneous injection. The students can interact multimodally using speech and a haptic device under the guidance of the virtual embodied tutor. INES takes into account elements of the students character and an appraisal of the students actions to estimate the mental state of the student. This information is used to plan and execute the actions and responses of the tutor agent.