Johannes Bernardus Fransiscus van Erp

Using neurophysiological signals that reflect cognitive or affective state: six recommendations to avoid common pitfalls

Estimating cognitive or affective state from neurophysiological signals and designing applications that make use of this information requires expertise in many disciplines such as neurophysiology, machine learning, experimental psychology, and human factors. This makes it difficult to perform research that is strong in all its aspects as well as to judge a study or application on its merits. On the occasion of the special topic “Using neurophysiological signals that reflect cognitive or affective state” we here summarize often occurring pitfalls and recommendations on how to avoid them, both for authors (researchers) and readers. They relate to defining the state of interest, the neurophysiological processes that are expected to be involved in the state of interest, confounding factors, inadvertently “cheating” with classification analyses, insight on what underlies successful state estimation, and finally, the added value of neurophysiological measures in the context of an application. We hope that this paper will support the community in producing high quality studies and well-validated, useful applications.

Social touch in human–computer interaction

Touch is our primary non-verbal communication channel for conveying intimate emotions and as such essential for our physical and emotional wellbeing. In our digital age, human social interaction is often mediated. However, even though there is increasing evidence that mediated touch affords affective communication, current communication systems (such as videoconferencing) still do not support communication through the sense of touch. As a result, mediated communication does not provide the intense affective experience of co-located communication. The need for ICT mediated or generated touch as an intuitive way of social communication is even further emphasized by the growing interest in the use of touch-enabled agents and robots for healthcare, teaching, and telepresence applications. Here, we review the important role of social touch in our daily life and the available evidence that affective touch can be mediated reliably between humans and between humans and digital agents. We base our observations on evidence from psychology, computer science, sociology, and neuroscience with focus on the first two. Our review shows that mediated affective touch can modulate physiological responses, increase trust and affection, help to establish bonds between humans and avatars or robots, and initiate pro-social behavior. We argue that ICT mediated or generated social touch can (a) intensify the perceived social presence of remote communication partners and (b) enable computer systems to more effectively convey affective information. However, this research field on the crossroads of ICT and psychology is still embryonic and we identify several topics that can help to mature the field in the following areas: establishing an overarching theoretical framework, employing better research methodologies, developing basic social touch building blocks, and solving specific ICT challenges.

Effortless passive BCIs for healthy users

While a BCI usually aims to provide an alternative communication channel for disabled users who have difficulties to move or to speak, we focused on BCIs as a way to retrieve and use information about an individuals cognitive or affective state without requiring any effort or intention of the user to convey this information. Providing only an extra channel of information rather than a replacement of certain functions, such BCIs could be useful for healthy users as well. We describe the results of our studies on neurophysiological correlates of attention, workload and emotion, as well as our efforts to include physiological variables. We found different features in EEG to be indicative of attention and workload, while emotional state may be better measured by physiological variables like heart rate and skin conductance. Potential applications are described. We argue that major challenges lie in hardware and generalization issues.

Public Understanding of Visual Representations of Uncertainty in Temperature Forecasts

Multiday weather forecasts often include graphical representations of uncertainty. However, visual representations of probabilistic events are often misinterpreted by the general public. Although various uncertainty visualizations are now in use, the parameters that determine their successful deployment are still unknown. At the same time, a correct understanding of possible weather forecast outcomes will enable the public to make better decisions and will increase their trust in these predictions. We investigated the effects of the visual form and width of temperature forecast visualizations with uncertainty on estimates of the probability that the temperature could exceed a given value. The results suggest that people apply an internal model of the uncertainty distribution that closely resembles a normal distribution, which confirms previous findings. Also, the visualization type appears to affect the applied internal model, in particular the probability estimates of values outside the depicted uncertainty range. Furthermore, we find that perceived uncertainty does not necessarily map linearly to visual features, as identical relative positions to the range are being judged differently depending on the width of the uncertainty range. Finally, the internal model of the uncertainty distribution is related to participants’ numeracy. We include some implications for makers or designers of uncertainty visualizations.

Editorial: Using neurophysiological signals that reflect cognitive or affective state

The central question of this Frontiers Research Topic is: What can we learn from brain and other physiological signals about an individuals cognitive and affective state and how can we use this information? This question reflects three important issues which are addressed by the 22 articles in this volume: (1) the combination of central and peripheral neurophysiological measures; (2) the diversity of cognitive and affective processes reflected by these measures; and (3) how to apply these measures in real world applications.

Tactile Cuing to Augment Multisensory Human–Machine Interaction.

Tactile displays promise to improve the information-processing capacity of operators, especially when used in conjunction with visual and auditory displays. In this article, we describe current applications and future directions in tactile cuing.

Physiological signals distinguish between reading emotional and non-emotional sections in a novel

We are interested in monitoring an individual’s emotions during the reading of a novel. While physiological responses to experimentally induced emotions are often small and inconsistent, being engaged in a novel may elicit relatively strong responses. We analyzed EEG, ECG, skin conductance and respiration of 57 readers reading a complete, yet to be published novel written by a popular contemporary writer (Arnon Grunberg) and compared physiology during the reading of pre-defined sections that were either emotionally intense or not. Heart rate was lower while reading emotional sections. We also found effects of emotional intensity on breathing variability and alpha asymmetry. Most of the examined physiological variables were strongly affected by time on task. We could estimate whether the physiological data of an individual reader were collected during the reading of an emotional or non-emotional section with an accuracy of up to 72% using individualized models and after correcting data for individually modeled time-related effects. Our results imply that using our methodology, it is possible to examine fluctuating reader’s emotion during the reading of a novel after reading, but not (yet) in real time.

Observing Touch from Video: The Influence of Social Cues on Pleasantness Perceptions

In order to advance the understanding of affective touch perceptions, and in particular to inform the design of physical human-robot interactions, an online video study was conducted in which observed stroking touches were assessed on perceived pleasantness. Touches were applied at different velocities and either with a human hand, a robot hand, a mannequin hand, or a plastic tube. In line with earlier research, it was found that stroking touches with a velocity of ca. 3 cm/s were rated as most pleasant. Moreover, the subjective pleasantness scores suggest that the stimulus type interacts with the stroking velocity. The possible roles that social agency, expectations, and anthropomorphism may play in perceptions of pleasantness are discussed.

Simulating Affective Touch: Using a Vibrotactile Array to Generate Pleasant Stroking Sensations

Gentle stroking touches are rated most pleasant when applied at a velocity of between 1–10 cm/s. Such touches are considered highly relevant in social interactions. Here, we investigate whether stroking sensations generated by a vibrotactile array can produce similar pleasantness responses, with the ultimate goal of using this type of haptic display in technology mediated social touch. A study was conducted in which participants received vibrotactile stroking stimuli of different velocities and intensities, applied to their lower arm. Results showed that the stimuli were perceived as continuous stroking sensations in a straight line. Furthermore, pleasantness ratings for low intensity vibrotactile stroking followed an inverted U-curve, similar to that found in research into actual stroking touches. The implications of these findings are discussed.

Emotional Responses to Multisensory Environmental Stimuli: A Conceptual Framework and Literature Review

How we perceive our environment affects the way we feel and behave. The impressions of our ambient environment are influenced by its entire spectrum of physical characteristics (e.g., luminosity, sound, scents, temperature) in a dynamic and interactive way. The ability to manipulate the sensory aspects of an environment such that people feel comfortable or exhibit a desired behavior is gaining interest and social relevance. Although much is known about the sensory effects of individual environmental characteristics, their combined effects are not a priori evident due to a wide range of non-linear interactions in the processing of sensory cues. As a result, it is currently not known how different environmental characteristics should be combined to effectively induce desired emotional and behavioral effects. To gain more insight into this matter, we performed a literature review on the emotional effects of multisensory stimulation. Although we found some interesting mechanisms, the outcome also reveals that empirical evidence is still scarce and haphazard. To stimulate further discussion and research, we propose a conceptual framework that describes how environmental interventions are likely to affect human emotional responses. This framework leads to some critical research questions that suggest opportunities for further investigation.