Joris H. Janssen

Emotional sweating across the body: comparing 16 different skin conductance measurement locations.

Skin conductance (SC) is one of the most commonly used measures in psychophysiological studies involving emotional arousal and is traditionally measured at the fingers or the palms (i.e., the palmar locations) of the hand. Palmar skin conductance recording positions are, however, not always preferred for ambulatory recordings in real-life situations. This study quantifies the responsiveness and similarity with the finger of 16 different recording positions of skin conductance while watching emotional film fragments. Findings indicated foot, fingers and shoulders being most responsive, whereas arm, back, armpit, and thighbone were least responsive. The measurements at the foot were most similar with those of the finger. In contrast, arm, back, and armpit traces differed most from the finger trace. Taken together, foot and shoulders are the best alternatives to the finger for ambulatory measurement of skin conductance to reflect emotional arousal. These findings can help new applications using skin conductance, like automated emotion measurements, to come to fruition.

Affective Man-Machine Interface: Unveiling Human Emotions through Biosignals

As is known for centuries, humans exhibit an electrical profile. This profile is altered through various psychological and physiological processes, which can be measured through biosignals; e.g., electromyography (EMG) and electrodermal activity (EDA). These biosignals can reveal our emotions and, as such, can serve as an advanced man-machine interface (MMI) for empathic consumer products. However, such a MMI requires the correct classification of biosignals to emotion classes. This chapter starts with an introduction on biosignals for emotion detection. Next, a state-of-the-art review is presented on automatic emotion classification. Moreover, guidelines are presented for affective MMI. Subsequently, a research is presented that explores the use of EDA and three facial EMG signals to determine neutral, positive, negative, and mixed emotions, using recordings of 21 people. A range of techniques is tested, which resulted in a generic framework for automated emotion classification with up to 61.31% correct classification of the four emotion classes, without the need of personal profiles. Among various other directives for future research, the results emphasize the need for parallel processing of multiple biosignals.

Intimate Heartbeats: Opportunities for Affective Communication Technology

Despite a variety of new communication technologies, loneliness is prevalent in Western countries. Boosting emotional communication through intimate connections has the potential to reduce loneliness. New technologies might exploit biosignals as intimate emotional cues because of their strong relationship to emotions. Through two studies, we investigate the possibilities of heartbeat communication as an intimate cue. In the first study (N = 32), we demonstrate, using self-report and behavioral tracking in an immersive virtual environment, that heartbeat perception influences social behavior in a similar manner as traditional intimate signals such as gaze and interpersonal distance. In the second study (N = 34), we demonstrate that a sound of the heartbeat is not sufficient to cause the effect; the stimulus must be attributed to the conversational partner in order to have influence. Together, these results show that heartbeat communication is a promising way to increase intimacy. Implications and possibilities for applications are discussed.

Avatars Versus Agents: A Meta-Analysis Quantifying the Effect of Agency on Social Influence

Existing research has investigated whether virtual representations perceived to be controlled by humans (i.e., avatars) or those perceived to be controlled by computer algorithms (i.e., agents) are more influential. A meta-analysis (N = 32) examined the model of social influence in virtual environments (Blascovich, 2002) and investigated whether agents and avatars in virtual environments elicit different levels of social influence. Results indicated that perceived avatars produced stronger responses than perceived agents. Level of immersion (desktop vs. fully immersive), dependent variable type (subjective vs. objective), task type (competitive vs. cooperative vs. neutral), and actual control of the representation (human vs. computer) were examined as moderators. An interaction effect revealed that studies conducted on a desktop that used objective measures showed a stronger effect for agency than those that were conducted on a desktop but used subjective measures. Competitive and cooperative tasks showed greater agency effects than neutral tasks. Studies in which both conditions were actually human controlled showed greater agency effects than studies in which both conditions were actually computer controlled. We discuss theoretical and design implications for human–computer interaction and computer-mediated communication.

Tune in to your emotions: a robust personalized affective music player

The emotional power of music is exploited in a personalized affective music player (AMP) that selects music for mood enhancement. A biosignal approach is used to measure listeners’ personal emotional reactions to their own music as input for affective user models. Regression and kernel density estimation are applied to model the physiological changes the music elicits. Using these models, personalized music selections based on an affective goal state can be made. The AMP was validated in real-world trials over the course of several weeks. Results show that our models can cope with noisy situations and handle large inter-individual differences in the music domain. The AMP augments music listening where its techniques enable automated affect guidance. Our approach provides valuable insights for affective computing and user modeling, for which the AMP is a suitable carrier application.

How affective technologies can influence intimate interactions and improve social connectedness

Affective computing holds the promise of creating effortless, integrated, and automatic ways of communicating emotions within our intimate social network. This could augment awareness systems and connectedness devices, reducing loneliness and improving health and well-being. Through two experiments, we investigate the effects of quantity and automaticity of emotion communication on perceived intimacy in mediated settings. In the first experiment (N=48), we manipulated the number of communicated emoticons. Results show that increases in communicated emoticon quantity lead to strong increases in perceived intimacy. In the second experiment (N=34), we compare automatic and user-initiated communication of emoticons. Results show that user-initiated communication of emoticons is experienced as more intimate than automatic communication. These results are discussed in light of the interpersonal process model of intimacy and can help the design of applications aimed at improving social interactions through affective communication technology.

Machines Outperform Laypersons in Recognizing Emotions Elicited by Autobiographical Recollection

Over the last decade, an increasing number of studies have focused on automated recognition of human emotions by machines. However, performances of machine emotion recognition studies are difficult to interpret because benchmarks have not been established. To provide such a benchmark, we compared machine with human emotion recognition. We gathered facial expressions, speech, and physiological signals from 17 individuals expressing 5 different emotional states. Support vector machines achieved an 82% recognition accuracy based on physiological and facial features. In experiments with 75 humans on the same data, a maximum recognition accuracy of 62.8% was obtained. As machines outperformed humans, automated emotion recognition might be ready to be tested in more practical applications.

Directing Physiology and Mood through Music: Validation of an Affective Music Player

Music is important in everyday life, as it provides entertainment and influences our moods. As music is widely available, it is becoming increasingly difficult to select songs to suit our mood. An affective music player can remove this obstacle by taking a desired mood as input and then selecting songs that direct toward that desired mood. In the present study, we validate the concept of an affective music player directing the energy dimension of mood. User models were trained for 10 participants based on skin conductance changes to songs from their own music database. Based on the resulting user models, the songs that most increased or decreased the skin conductance level of the participants were selected to induce either a relatively energized or a calm mood. Experiments were conducted in a real-world office setting. The results showed that a reliable prediction can be made of the impact of a song on skin conductance, that skin conductance and mood can be directed toward an energized or calm state and that skin conductance remains in these states for at least 30 minutes. All in all, this study shows that the concept and models of the affective music player worked in an ecologically valid setting, suggesting the feasibility of using physiological responses in real-life affective computing applications.

Personalized affective music player

We introduce and test an affective music player (AMP) that selects music for mood enhancement. Through a concise overview of content, construct, and ecological validity, we elaborate five considerations that form the foundation of the AMP. Based on these considerations, computational models are developed, using regression and kernel density estimation. We show how these models can be used for music selection and how they can be extended to fit in other systems. Subsequently, the success of the models is illustrated with a user test. The AMP augments music listening, where its techniques, in general, enable automated affect guidance. Finally, we argue that our AMP is readily applicable to real-world situations as it can 1) cope with noisy situations, 2) handle the large inter-individual differences apparent in the musical domain, and 3) integrate context or other information, all in real-time.

A three-component framework for empathic technologies to augment human interaction

Empathy can be considered one of our most important social processes. In that light, empathic technologies are the class of technologies that can augment empathy between two or more individuals. To provide a basis for such technologies, a three component framework is presented based on psychology and neuroscience, consisting of cognitive empathy, emotional convergence, and empathic responding. These three components can be situated in affective computing and social signal processing and pose different opportunities for empathic technologies. To leverage these opportunities, automated measurement possibilities for each component are identified using (combinations of) facial expressions, speech, and physiological signals. Thereafter, methodological challenges are discussed, including ground truth measurements and empathy induction. Finally, a research agenda is presented for social signal processing. This framework can help to further research on empathic technologies and ultimately bring it to fruition in meaningful innovations. In turn, this could enhance empathic behavior, thereby increasing altruism, trust, cooperation, and bonding.