Francesca Bonin

Open challenges in modelling, analysis and synthesis of human behaviour in human–human and human–machine interactions

Modelling, analysis and synthesis of behaviour are the subject of major efforts in computing science, especially when it comes to technologies that make sense of human–human and human–machine interactions. This article outlines some of the most important issues that still need to be addressed to ensure substantial progress in the field, namely (1) development and adoption of virtuous data collection and sharing practices, (2) shift in the focus of interest from individuals to dyads and groups, (3) endowment of artificial agents with internal representations of users and context, (4) modelling of cognitive and semantic processes underlying social behaviour and (5) identification of application domains and strategies for moving from laboratory to the real-world products.

How Do We React to Context? Annotation of Individual and Group Engagement in a Video Corpus

One of the main challenges of recent years is to create socially intelligent machines: machines able not only to communicate but also to understand social signals and make sense of the various social contexts. In this paper we describe a new annotation method for the analysis of involvement, aiming at exploring the relation between the individual and the social context in terms of perceived involvement, starting from the idea that the group is more than the sum of its parts. We present a description of the annotation method and preliminary results of an analysis of a multi-party casual conversation extracted from a multimedia multimodal corpus. The work aims to explore the mechanisms by which we react to social context so that we can develop automatic dialogue systems that are able to adapt to their environment in a similar way.

Time for laughter

Social signals are integral to conversational interaction and constitute a large part of the social dynamics of multiparty communication. Moreover, social signals may also have a function in discourse structure. We focus on laughter, exploring the extent to which laughter can be shown to signal the structural unfolding of conversation and whether laughter may be used in the signaling of topic changes. Recent research supports this hypothesis. We investigate the relation between laughter and topic changes from two different points of view (temporal distribution and content distribution) as visible in the TableTalk corpus and also in the AMI corpus. Consistent results emerge from studies of these two corpora. Laughter is less likely very soon after a topic change than it is before a topic change. In both studies, we find solo laughter significantly more frequent in times of topic transition than in times of topic continuity. This contradicts previous research about the social dynamics of shared versus solo laughter considering solo laughs as signals of topic continuation. We conclude that laughter has quantifiable discourse functionality concomitant with social signaling capacity.

From multimodal analysis to real-time interactions with virtual agents

One of the aims in building multimodal user interfaces is to make the interaction between user and systems as natural as possible. Possibly the most natural form of interaction we know is the way we communicate with other humans. By building virtual agents, we aim to recreate this natural form of interaction in human–machine communication. This is even more important for virtual agents that communicate with humans in a real-time face-to-face setting. While the promises of such natural interfaces are longstanding [1,9], their development is not straightforward. Understanding of human–human interaction is needed to an

Topics for the Future: Genre Differentiation, Annotation, and Linguistic Content Integration in Interaction Analysis

In this paper we discuss three topics central to discussion of the future of multimodal research -- genre differentiation, stardardization of annotation, and integration of social and verbal context.

International workshop on multimodal analyses enabling artificial agents in human- machine interaction (workshop summary)

In this paper a brief overview of the third workshop on Multimodal Analyses enabling Artificial Agents in Human-Machine Interaction. The paper is focussing on the main aspects intended to be discussed in the workshop reflecting the main scope of the papers presented during the meeting. The MA3HMI 2016 workshop is held in conjunction with the 18th ACM International Conference on Mulitmodal Interaction (ICMI 2016) taking place in Tokyo, Japan, in November 2016. This year, we have solicited papers concerning the different phases of the development of multimodal systems. Tools and systems that address real-time conversations with artificial agents and technical systems are also within the scope.

Multimodal Analyses enabling Artificial Agents in Human-Machine Interaction

This book constitutes the thoroughly refereed post-workshop proceedings of the Second Workshop on Multimodal Analyses Enabling Artificial Agents in Human Interaction, MA3HMI 2014, held in Conjunction with INTERSPEECH 2014, in Singapore, Singapore, on September 14th, 2014.The 9 revised papers presented together with a keynote talk were carefully reviewed and selected from numerous submissions. They are organized in two sections: human-machine interaction and dialogs and speech recognition.