Ionut Damian

The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time

Automatic detection and interpretation of social signals carried by voice, gestures, mimics, etc. will play a key-role for next-generation interfaces as it paves the way towards a more intuitive and natural human-computer interaction. The paper at hand introduces Social Signal Interpretation (SSI), a framework for real-time recognition of social signals. SSI supports a large range of sensor devices, filter and feature algorithms, as well as, machine learning and pattern recognition tools. It encourages developers to add new components using SSIs C++ API, but also addresses front end users by offering an XML interface to build pipelines with a text editor. SSI is freely available under GPL at http://openssi.net.

The TARDIS Framework: Intelligent Virtual Agents for Social Coaching in Job Interviews

The TARDIS project aims to build a scenario-based serious-game simulation platform for NEETs and job-inclusion associations that supports social training and coaching in the context of job interviews. This paper presents the general architecture of the TARDIS job interview simulator, and the serious game paradigm that we are developing.

Natural interaction with culturally adaptive virtual characters

Recently, the verbal and non-verbal behavior of virtual characters has become more and more sophisticated due to advances in behavior planning and rendering. Nevertheless, the appearance and behavior of these characters is in most cases based on the cultural background of their designers. Especially in combination with new natural interaction interfaces, there is the risk that characters developed for a particular culture might not find acceptance when being presented to another culture. A few attempts have been made to create characters that reflect a particular cultural background. However, interaction with these characters still remains an awkward experience in particular when it comes to non-verbal interaction. In many cases, human users either have to choose actions from a menu their avatar has to execute or they have to struggle with obtrusive interaction devices. In contrast, our paper combines an approach to the generation of culture-specific behaviors with full body avatar control based on the Kinect sensor. A first study revealed that users are able to easily control an avatar through their body movements and immediately adapt its behavior to the cultural background of the agents they interact with.

A Job Interview Simulation: Social Cue-Based Interaction with a Virtual Character

This paper presents an approach that makes use of a virtual character and social signal processing techniques to create an immersive job interview simulation environment. In this environment, the virtual character plays the role of a recruiter which reacts and adapts to the users behavior thanks to a component for the automatic recognition of social cues (conscious or unconscious behavioral patterns). The social cues pertinent to job interviews have been identified using a knowledge elicitation study with real job seekers. Finally, we present two user studies to investigate the feasibility of the proposed approach as well as the impact of such a system on users.

NovA: Automated Analysis of Nonverbal Signals in Social Interactions

Previous studies have shown that the success of interpersonal interaction depends not only on the contents we communicate explicitly, but also on the social signals that are conveyed implicitly. In this paper, we present NovA (NOnVerbal behavior Analyzer), a system that analyzes and facilitates the interpretation of social signals conveyed by gestures, facial expressions and others automatically as a basis for computer-enhanced social coaching. NovA records data of human interactions, automatically detects relevant behavioral cues as a measurement for the quality of an interaction and creates descriptive statistics for the recorded data. This enables us to give a user online generated feedback on strengths and weaknesses concerning his social behavior, as well as elaborate tools for offline analysis and annotation.

Individualized agent interactions

Individualized virtual agents can enhance the users perception of a virtual scenario. However, most systems only provide customization for visual features of the characters. In this paper, we describe an approach to individualizing the non-verbal behavior of virtual agents. To this end, we present a software framework which is able to visualize individualized non-verbal behavior. For demonstration purposes, we designed four behavioral profiles that simulate prototypical behaviors for differences in personality and gender. These were then tested in an evaluation study.

Context-Aware Automated Analysis and Annotation of Social Human--Agent Interactions

The outcome of interpersonal interactions depends not only on the contents that we communicate verbally, but also on nonverbal social signals. Because a lack of social skills is a common problem for a significant number of people, serious games and other training environments have recently become the focus of research. In this work, we present NovA (Nonverbal behavior Analyzer), a system that analyzes and facilitates the interpretation of social signals automatically in a bidirectional interaction with a conversational agent. It records data of interactions, detects relevant social cues, and creates descriptive statistics for the recorded data with respect to the agents behavior and the context of the situation. This enhances the possibilities for researchers to automatically label corpora of human--agent interactions and to give users feedback on strengths and weaknesses of their social behavior.

Games are Better than Books: In-Situ Comparison of an Interactive Job Interview Game with Conventional Training

Technology-enhanced learning environments are designed to help users practise social skills. In this paper, we present and evaluate a virtual job interview training game which has been adapted to the special requirements of young people with low chances on the job market. The evaluation spanned three days, during which we compared the technology-enhanced training with a traditional learning method usually practised in schools, i.e. reading a job interview guide. The results are promising as professional career counsellors rated the pupils who trained with the system significantly better than those who learned with the traditional method.

Cultural behaviors of virtual agents in an augmented reality environment

This paper presents a pilot evaluation study that investigates the physiological response of users when interacting with virtual agents that resemble cultural behaviors in an Augmented Reality environment. In particular, we analyze users from the Arab and German cultural backgrounds. The initial results of our analysis are promising and show that users tend to have a higher physiological arousal towards virtual agents that do not exhibit behaviors of their cultural background.

Who’s Afraid of Job Interviews? Definitely a Question for User Modelling

We define job interviews as a domain of interaction that can be modelled automatically in a serious game for job interview skills training. We present four types of studies: (1) field-based human-to-human job interviews, (2) field-based computer-mediated human-to-human interviews, (3) lab-based wizard of oz studies, (4) field-based human-to-agent studies. Together, these highlight pertinent questions for the user modelling field as it expands its scope to applications for social inclusion. The results of the studies show that the interviewees suppress their emotional behaviours and although our system recognises automatically a subset of those behaviours, the modelling of complex mental states in real-world contexts poses a challenge for the state-of-the-art user modelling technologies. This calls for the need to re-examine both the approach to the implementation of the models and/or of their usage for the target contexts.