Andrea Kleinsmith

Affective Body Expression Perception and Recognition: A Survey

Thanks to the decreasing cost of whole-body sensing technology and its increasing reliability, there is an increasing interest in, and understanding of, the role played by body expressions as a powerful affective communication channel. The aim of this survey is to review the literature on affective body expression perception and recognition. One issue is whether there are universal aspects to affect expression perception and recognition models or if they are affected by human factors such as culture. Next, we discuss the difference between form and movement information as studies have shown that they are governed by separate pathways in the brain. We also review psychological studies that have investigated bodily configurations to evaluate if specific features can be identified that contribute to the recognition of specific affective states. The survey then turns to automatic affect recognition systems using body expressions as at least one input modality. The survey ends by raising open questions on data collecting, labeling, modeling, and setting benchmarks for comparing automatic recognition systems.

Cross-cultural differences in recognizing affect from body posture

Conveyance and recognition of human emotion and affective expression is influenced by many factors, including culture. Within the user modeling field, it has become increasingly necessary to understand the role affect can play in personalizing interactive interfaces using embodied animated agents. However, little research within the computer science field aims at understanding cultural differences within this vein. Therefore, we conducted a study to evaluate if differences exist in the way various cultures perceive emotion from body posture. We used static posture images of affectively expressive avatars to conduct recognition experiments with subjects from three cultures. After analyzing the subjects judgments using multivariate analysis, we grounded the identified differences into a set of low-level posture features. We then used Mixture Discriminant Analysis (MDA) and an unsupervised expectation maximization (EM) model to build separate cultural models for affective posture recognition. Our results could prove useful to aide designers in creating more effective affective avatars.

Automatic Recognition of Non-Acted Affective Postures

The conveyance and recognition of affect and emotion partially determine how people interact with others and how they carry out and perform in their day-to-day activities. Hence, it is becoming necessary to endow technology with the ability to recognize users affective states to increase the technologies effectiveness. This paper makes three contributions to this research area. First, we demonstrate recognition models that automatically recognize affective states and affective dimensions from non-acted body postures instead of acted postures. The scenario selected for the training and testing of the automatic recognition models is a body-movement-based video game. Second, when attributing affective labels and dimension levels to the postures represented as faceless avatars, the level of agreement for observers was above chance level. Finally, with the use of the labels and affective dimension levels assigned by the observers as ground truth and the observers level of agreement as base rate, automatic recognition models grounded on low-level posture descriptions were built and tested for their ability to generalize to new observers and postures using random repeated subsampling validation. The automatic recognition models achieve recognition percentages comparable to the human base rates as hypothesized.

Recognizing Affective Dimensions from Body Posture

The recognition of affective human communication may be used to provide developers with a rich source of information for creating systems that are capable of interacting well with humans. Posture has been acknowledged as an important modality of affective communication in many fields. Behavioral studies have shown that posture can communicate discrete emotion categories as well as affective dimensions. In the affective computing field, while models for the automatic recognition of discrete emotion categories from posture have been proposed, to our knowledge, there are no models for the automatic recognition of affective dimensions from static posture. As a continuation of our previous study, the two main goals of this study are: i) to build automatic recognition models to discriminate between levels of affective dimensions based on low-level postural features; and ii) to investigate both the discriminative power and the limitations of the postural features proposed. The models were built on the basis of human observers ratings of posture according to affective dimensions directly (instead of emotion category) in conjunction with our posture features.

A categorical approach to affective gesture recognition

Studies on emotion are currently receiving a lot of attention. The importance of emotion in the development and support of intelligent and social behaviour has been highlighted by studies in psychology and neurology. Hence, the recognition of affective states has also become a critical feature in robot social development, with robots assumed to take on a role as social companion. In this paper, we address the issue of endowing robots with the ability to learn incrementally to recognize the affective state of their human partner by interpreting their gestural cues. We propose a model that can self-organize postural features into affective categories, and use contextual feedback from the partner to drive the learning process.

Towards unsupervised detection of affective body posture nuances

Recently, researchers have been modeling three to nine discrete emotions for creating affective recognition systems. However, in every day life, humans use a rich and powerful language for defining a large variety of affective states. Thus, one of the challenging issues in affective computing is to give computers the ability to recognize a variety of affective states using unsupervised methods. In order to explore this possibility, we describe affective postures representing 4 emotion categories using low level descriptors. We applied multivariate analysis to recognize and categorize these postures into nuances of these categories. The results obtained show that low-level posture features may be used for this purpose, leaving the naming issue to interactive processes.

Recognizing emotion from postures: cross-cultural differences in user modeling

The conveyance and recognition of human emotion and affective expression is influenced by many factors, including culture. Within the area of user modeling, it has become increasingly necessary to understand the role affect can play in personalizing interactive interfaces using embodied animated agents. Currently, little research focuses on the importance of emotion expression through body posture. Furthermore, little research aims at understanding cultural differences within this vein. Therefore, our goal is to evaluate whether or not differences exist in the way various cultures perceive emotion from body posture. We used images of 3D affectively expressive avatars to conduct recognition experiments with subjects from 3 cultures. The subjects judgments were analyzed using multivariate analysis. We grounded the identified differences into a set of low-level posture features. Our results could prove useful for constructing affective posture recognition systems in cross-cultural environments.

Grounding affective dimensions into posture features

Many areas of today’s society are seeing an increased importance in the creation of systems capable of interacting with users on an affective level through a variety of modalities. Our focus has been on affective posture recognition. However, a deeper understanding of the relationship between emotions in terms of postural expressions is required. The goal of this study was to identify affective dimensions that human observers use when discriminating between postures, and to investigate the possibility of grounding this affective space into a set of posture features. Using multidimensional scaling, arousal, valence, and action tendency were identified as the main factors in the evaluation process. Our results showed that, indeed, low-level posture features could effectively discriminate between the affective dimensions.

Form as a cue in the automatic recognition of non-acted affective body expressions

The advent of whole-body interactive technology has increased the importance of creating systems that take into account body expressions to determine the affective state of the user. In doing so, the role played by the form and motion information needs to be understood. Neuroscience studies have shown that biological motion is recognized by separate pathways in the brain. This paper investigates the contribution of body configuration (form) in the automatic recognition of non-acted affective dynamic expressions in a video game context. Sequences of static postures are automatically extracted from motion capture data and presented to the system which is a combination of an affective posture recognition module and a sequence classification rule to finalize the affective state of each sequence. Our results show that using form information only, the system recognition reaches performances very close to the agreement between observers who viewed the affective expressions as animations containing both form and temporal information.

Multi-score learning for affect recognition: the case of body postures

An important challenge in building automatic affective state recognition systems is establishing the ground truth. When the groundtruth is not available, observers are often used to label training and testing sets. Unfortunately, inter-rater reliability between observers tends to vary from fair to moderate when dealing with naturalistic expressions. Nevertheless, the most common approach used is to label each expression with the most frequent label assigned by the observers to that expression. In this paper, we propose a general pattern recognition framework that takes into account the variability between observers for automatic affect recognition. This leads to what we term a multi-score learning problem in which a single expression is associated with multiple values representing the scores of each available emotion label. We also propose several performance measurements and pattern recognition methods for this framework, and report the experimental results obtained when testing and comparing these methods on two affective posture datasets.