Joyce H. D. M. Westerink

Affective Man-Machine Interface: Unveiling Human Emotions through Biosignals

As is known for centuries, humans exhibit an electrical profile. This profile is altered through various psychological and physiological processes, which can be measured through biosignals; e.g., electromyography (EMG) and electrodermal activity (EDA). These biosignals can reveal our emotions and, as such, can serve as an advanced man-machine interface (MMI) for empathic consumer products. However, such a MMI requires the correct classification of biosignals to emotion classes. This chapter starts with an introduction on biosignals for emotion detection. Next, a state-of-the-art review is presented on automatic emotion classification. Moreover, guidelines are presented for affective MMI. Subsequently, a research is presented that explores the use of EDA and three facial EMG signals to determine neutral, positive, negative, and mixed emotions, using recordings of 21 people. A range of techniques is tested, which resulted in a generic framework for automated emotion classification with up to 61.31% correct classification of the four emotion classes, without the need of personal profiles. Among various other directives for future research, the results emphasize the need for parallel processing of multiple biosignals.

Emotional and psychophysiological responses to tempo, mode, and percussiveness

People often listen to music to influence their emotional state. However, the specific musical characteristics which cause this process are not yet fully understood. We have investigated the influence of the musical characteristics of tempo, mode, and percussiveness on our emotions. In a quest towards ecologically valid results, 32 participants listened to 16 pop and 16 rock songs while conducting an office task. They rated experienced arousal, valence, and tension, while skin conductance and cardiovascular responses were recorded. An increase in tempo was found to lead to an increase in reported arousal and tension and a decrease in heart rate variability. More arousal was reported during minor than major mode songs. Level and frequency of skin conductance responses increased with an increase in percussiveness. Physiological responses revealed patterns that might not have been revealed by self-report. Interaction effects further suggest that musical characteristics interplay in modulating emotions. So, tempo, mode, and percussiveness indeed modulate our emotions and, consequently, can be used to direct emotions. Music presentation revealed subtly different results in a laboratory setting, where music was altered with breaks, from those in a more ecologically valid setting where continuous music was presented. All in all, this enhances our understanding of the influence of music on emotions and creates opportunities seamlessly to tap into listeners’ emotional state through their physiological responses.

Smart technologies for long-term stress monitoring at work

Due to the growing pace of life, stress became one of the major factors causing health problems. We have developed a framework for measuring stress in real-life conditions continuously and unobtrusively. In order to provide meaningful, useful and actionable information, we present stress information, derived from sensor measurements, in the context of persons activities. In this paper, we describe our framework, discuss how we address arising challenges and evaluate our approach on basis of the field studies we have conducted. The main results of the evaluation are that the results of long-term measurements of stress reveal people information about their behavioral patterns that they perceive as meaningful and useful, and trigger their ideas about behavioral changes necessary to achieve a better stress balance.

Intimate Heartbeats: Opportunities for Affective Communication Technology

Despite a variety of new communication technologies, loneliness is prevalent in Western countries. Boosting emotional communication through intimate connections has the potential to reduce loneliness. New technologies might exploit biosignals as intimate emotional cues because of their strong relationship to emotions. Through two studies, we investigate the possibilities of heartbeat communication as an intimate cue. In the first study (N = 32), we demonstrate, using self-report and behavioral tracking in an immersive virtual environment, that heartbeat perception influences social behavior in a similar manner as traditional intimate signals such as gaze and interpersonal distance. In the second study (N = 34), we demonstrate that a sound of the heartbeat is not sufficient to cause the effect; the stimulus must be attributed to the conversational partner in order to have influence. Together, these results show that heartbeat communication is a promising way to increase intimacy. Implications and possibilities for applications are discussed.

Computing emotion awareness through galvanic skin response and facial electromyography

To improve human-computer interaction (HCI), computers need to recognize and respond properly to their user’s emotional state. This is a fundamental application of affective computing, which relates to, arises from, or deliberately influences emotion. As a first step to a system that recognizes emotions of individual users, this research focuses on how emotional experiences are expressed in six parameters (i.e., mean, absolute deviation, standard deviation, variance, skewness, and kurtosis) of not baseline-corrected physiological measurements of the galvanic skin response (GSR) and of three electromyography signals: frontalis (EMG1), corrugator supercilii (EMG2), and zygomaticus major (EMG3). The 24 participants were asked to watch film scenes of 120 seconds, which they rated afterward. These ratings enabled us to distinguish four categories of emotions: negative, positive, mixed, and neutral. The skewness and kurtosis of the GSR, the skewness of the EMG2, and four parameters of EMG3, discriminate between the four emotion categories. This, despite the coarse time windows that were used. Moreover, rapid processing of the signals proved to be possible. This enables tailored HCI facilitated by an emotional awareness of systems.

Considerations for emotion-aware consumer products

Emotion-aware consumer products require reliable, short-term emotion assessment (i.e., unobtrusive, robust, and lacking calibration). To explore the feasibility of this, an experiment was conducted where the galvanic skin response (GSR) and three electromyography (EMG) signals (frontalis, corrugator supercilii, and zygomaticus major) were recorded on 24 participants who watched eight 2-min emotion inducing film fragments. The unfiltered psychophysiological signals were processed and six statistical parameters (i.e., mean, absolute deviation, standard deviation, variance, skewness, and kurtosis) were derived for each 10-s interval of the film fragment. For each physiological signal, skewness and kurtosis discriminated among affective states, accompanied by other parameters, depending on the signal. The skewness parameter also showed to indicate mixed emotions. Moreover, a mapping of events in the fragments on the signals showed the importance of short-term emotion assessment. Hence, this research identified generic features, denoted important considerations, and illustrated the feasibility of emotion-aware consumer products.

The influence of music on mood and performance while driving

Mood can influence our everyday behaviour and people often seek to reinforce, or to alter their mood, for example by turning on music. Music listening while driving is a popular activity. However, little is known about the impact of music listening while driving on physiological state and driving performance. In the present experiment, it was investigated whether individually selected music can induce mood and maintain moods during a simulated drive. In addition, effects of positive, negative, and no music on driving behaviour and physiological measures were assessed for normal and high cognitive demanding rides. Subjective mood ratings indicated that music successfully maintained mood while driving. Narrow lane width drives increased task demand as shown in effort ratings and increased swerving. Furthermore, respiration rate was lower during music listening compared to rides without music, while no effects of music were found on heart rate. Overall, the current study demonstrates that music listening in car influences the experienced mood while driving, which in turn can impact driving behaviour. Practitioners Summary: Even though it is a popular activity, little is known about the impact of music while driving on physiological state and performance. We examined whether music can induce moods during high and low simulated drives. The current study demonstrates that in car music listening influences mood which in turn can impact driving behaviour. The current study shows that listening to music can positively impact mood while driving, which can be used to affect state and safe behaviour. Additionally, driving performance in high demand situations is not negatively affected by music.

Emotion measurement platform for daily life situations

The growing interest in affective computing is expected to have its beneficial impact on consumer lifestyle products. Especially emphatic applications — applications that make you feel they really understand you — will serve the current consumer interest in enhanced and meaningful experiences. To do so, the applications will have to measure the users emotional experience. Well-established psychophysiological techniques bear promise, but so far have mainly been validated in laboratory situations. To also apply them in real-life situations, we built an emotion measurement platform. This platform shows that emotional experiences can be measured in a relatively unobtrusive way, while at the same time it enables us to gather knowledge on emotional experiences in everyday-life and it offers the opportunity to prototype emphatic application concepts and test them in relevant situations.

Tune in to your emotions: a robust personalized affective music player

The emotional power of music is exploited in a personalized affective music player (AMP) that selects music for mood enhancement. A biosignal approach is used to measure listeners’ personal emotional reactions to their own music as input for affective user models. Regression and kernel density estimation are applied to model the physiological changes the music elicits. Using these models, personalized music selections based on an affective goal state can be made. The AMP was validated in real-world trials over the course of several weeks. Results show that our models can cope with noisy situations and handle large inter-individual differences in the music domain. The AMP augments music listening where its techniques enable automated affect guidance. Our approach provides valuable insights for affective computing and user modeling, for which the AMP is a suitable carrier application.

Machines Outperform Laypersons in Recognizing Emotions Elicited by Autobiographical Recollection

Over the last decade, an increasing number of studies have focused on automated recognition of human emotions by machines. However, performances of machine emotion recognition studies are difficult to interpret because benchmarks have not been established. To provide such a benchmark, we compared machine with human emotion recognition. We gathered facial expressions, speech, and physiological signals from 17 individuals expressing 5 different emotional states. Support vector machines achieved an 82% recognition accuracy based on physiological and facial features. In experiments with 75 humans on the same data, a maximum recognition accuracy of 62.8% was obtained. As machines outperformed humans, automated emotion recognition might be ready to be tested in more practical applications.