Nadia Bianchi-Berthouze

Affective Body Expression Perception and Recognition: A Survey

Thanks to the decreasing cost of whole-body sensing technology and its increasing reliability, there is an increasing interest in, and understanding of, the role played by body expressions as a powerful affective communication channel. The aim of this survey is to review the literature on affective body expression perception and recognition. One issue is whether there are universal aspects to affect expression perception and recognition models or if they are affected by human factors such as culture. Next, we discuss the difference between form and movement information as studies have shown that they are governed by separate pathways in the brain. We also review psychological studies that have investigated bodily configurations to evaluate if specific features can be identified that contribute to the recognition of specific affective states. The survey then turns to automatic affect recognition systems using body expressions as at least one input modality. The survey ends by raising open questions on data collecting, labeling, modeling, and setting benchmarks for comparing automatic recognition systems.

Does Body Movement Engage You More in Digital Game Play? and Why?

In past years, computer game designers have tried to increase player engagement by improving the believability of characters and environment. Today, the focus is shifting toward improving the game controller. This study seeks to understand engagement on the basis of the body movements of the player. Initial results from two case-studies suggest that an increase in body movement imposed, or allowed, by the game controller results in an increase in the players engagement level. Furthermore, they lead us to hypothesize that an increased involvement of the body can afford the player a stronger affective experience. We propose that the contribution of full-body experience is three-fold: (a) it facilitates the feeling of presence in the digital environment (fantasy); (b) it enables the affective aspects of human-human interaction (communication); and (c) it unleashes the regulatory properties of emotion (affect).

Understanding the Role of Body Movement in Player Engagement

The introduction of full-body controllers has made computer games more accessible and promises to provide a more natural and engaging experience to players. However, the relationship between body movement and game engagement is not yet well understood. In this article, I consider how body movement affects the players experience during game play. I start by presenting a taxonomy of body movements observed during game play. These are framed in the context of a body of previously published research that is then embedded into a novel model of engagement. This model describes the relationship between the taxonomy of movement and the type of engagement that each class of movement facilitates. I discuss the factors that may inhibit or enhance such relationship. Finally, I conclude by considering how the proposed model could lead to a more systematic and effective use of body movement for enhancing game experience.

Movement-based sports video games: Investigating motivation and gaming experience

Video game consoles that enable gamers to use active body movements are becoming increasingly popular. Yet, little is known about the influence of movement on how gamers experience such games. This study takes an exploratory approach, using different data collection methods. A theory about the relationship between body movement and gaming experience emerges through the systematic collection and analysis of data obtained from interviews, questionnaires, video observations and a motion capture system. A Grounded Theory analysis of the interviews reveals two distinct motivations (to achieve and to relax) with which gamers approach such games, together with two corresponding movement control strategies. Four movement-specific items are found to influence immersion in movement-based interaction: natural control, mimicry of movements, proprioceptive feedback, and physical challenge. These results are verified by exploiting the movement patterns of gamers playing the Nintendo Wii Boxing game. This theory others insights to game designers as to how to design future generations of movement-based games. Whilst a controller that leaves more space for appropriation can be appealing to a larger population, its design may fail to promote and motivate physical activity and emotional well-being.

Modeling Multimodal Expression of User's Affective Subjective Experience

With the growing importance of information technology in our everyday life, new types of applications are appearing that require the understanding of information in a broad sense. Information that includes affective and subjective content plays a major role not only in an individual’s cognitive processes but also in an individual’s interaction with others. We identify three key points to be considered when developing systems that capture affective information: embodiment (experiencing physical reality), dynamics (mapping experience and emotional state with its label) and adaptive interaction (conveying emotive response, responding to a recognized emotional state). We present two computational systems that implement those principles: MOUE (Model Of User Emotions) is an emotion recognition system that recognizes the user’s emotion from his/her facial expressions, and from it, adaptively builds semantic definitions of emotion concepts using the user’s feedback; MIKE (Multimedia Interactive Environment for Kansei communication) is an interactive adaptive system that, along with the user, co-evolves a language for communicating over subjective impressions.

What Does Touch Tell Us about Emotions in Touchscreen-Based Gameplay?

The increasing number of people playing games on touch-screen mobile phones raises the question of whether touch behaviors reflect players’ emotional states. This prospect would not only be a valuable evaluation indicator for game designers, but also for real-time personalization of the game experience. Psychology studies on acted touch behavior show the existence of discriminative affective profiles. In this article, finger-stroke features during gameplay on an iPod were extracted and their discriminative power analyzed. Machine learning algorithms were used to build systems for automatically discriminating between four emotional states (Excited, Relaxed, Frustrated, Bored), two levels of arousal and two levels of valence. Accuracy reached between 69% and 77% for the four emotional states, and higher results (~89%) were obtained for discriminating between two levels of arousal and two levels of valence. We conclude by discussing the factors relevant to the generalization of the results to applications other than games.

Modeling human affective postures: an information theoretic characterization of posture features

One of the challenging issues in affective computing is to give a machine the ability to recognize the mood of a person. Efforts in that direction have mainly focused on facial and oral cues. Gestures have been recently considered as well, but with less success. Our aim is to fill this gap by identifying and measuring the saliency of posture features that play a role in affective expression. As a case study, we collected affective gestures from human subjects using a motion capture system. We first described these gestures with spatial features, as suggested in studies on dance. Through standard statistical techniques, we verified that there was a statistically significant correlation between the emotion intended by the acting subjects, and the emotion perceived by the observers. We used Discriminant Analysis to build affective posture predictive models and to measure the saliency of the proposed set of posture features in discriminating between 4 basic emotional states: angry, fear, happy, and sad. An information theoretic characterization of the models shows that the set of features discriminates well between emotions, and also that the models built over‐perform the human observers. Copyright

A categorical approach to affective gesture recognition

Studies on emotion are currently receiving a lot of attention. The importance of emotion in the development and support of intelligent and social behaviour has been highlighted by studies in psychology and neurology. Hence, the recognition of affective states has also become a critical feature in robot social development, with robots assumed to take on a role as social companion. In this paper, we address the issue of endowing robots with the ability to learn incrementally to recognize the affective state of their human partner by interpreting their gestural cues. We propose a model that can self-organize postural features into affective categories, and use contextual feedback from the partner to drive the learning process.

Motivating people with chronic pain to do physical activity: opportunities for technology design

Physical activity is important for improving quality of life in people with chronic pain. However, actual or anticipated pain exacerbation, and lack of confidence when doing physical activity, make it difficult to maintain and build towards long-term activity goals. Research guiding the design of interactive technology to motivate and support physical activity in people with chronic pain is lacking. We conducted studies with: (1) people with chronic pain, to understand how they maintained and increased physical activity in daily life and what factors deterred them; and (2) pain-specialist physiotherapists, to understand how they supported people with chronic pain. Building on this understanding, we investigated the use of auditory feedback to address some of the psychological barriers and needs identified and to increase self-efficacy, motivation and confidence in physical activity. We conclude by discussing further design opportunities based on the overall findings.

Naturalistic affective expression classification by a multi-stage approach based on hidden Markov models

In naturalistic behaviour, the affective states of a person change at a rate much slower than the typical rate at which video or audio is recorded (e.g. 25fps for video). Hence, there is a high probability that consecutive recorded instants of expressions represent a same affective content. In this paper, a multi-stage automatic affective expression recognition system is proposed which uses Hidden Markov Models (HMMs) to take into account this temporal relationship and finalize the classification process. The hidden states of the HMMs are associated with the levels of affective dimensions to convert the classification problem into a best path finding problem in HMM. The system was tested on the audio data of the Audio/Visual Emotion Challenge (AVEC) datasets showing performance significantly above that of a one-stage classification system that does not take into account the temporal relationship, as well as above the baseline set provided by this Challenge. Due to the generality of the approach, this system could be applied to other types of affective modalities.