Sven-Thomas Graupner

Virtual friend or threat? The effects of facial expression and gaze interaction on psychophysiological responses and emotional experience

The present study aimed to investigate the impact of facial expression, gaze interaction, and gender on attention allocation, physiological arousal, facial muscle responses, and emotional experience in simulated social interactions. Participants viewed animated virtual characters varying in terms of gender, gaze interaction, and facial expression. We recorded facial EMG, fixation duration, pupil size, and subjective experience. Subjects rapid facial reactions (RFRs) differentiated more clearly between the characters happy and angry expression in the condition of mutual eye-to-eye contact. This finding provides evidence for the idea that RFRs are not simply motor responses, but part of an emotional reaction. Eye movement data showed that fixations were longer in response to both angry and neutral faces than to happy faces, thereby suggesting that attention is preferentially allocated to cues indicating potential threat during social interaction.

Attentional dynamics during free picture viewing: Evidence from oculomotor behavior and electrocortical activity

Most empirical evidence on attentional control is based on brief presentations of rather abstract stimuli. Results revealed indications for a dynamic interplay between bottom-up and top-down attentional mechanisms. Here we used a more naturalistic task to examine temporal signatures of attentional mechanisms on fine and coarse time scales. Subjects had to inspect digitized copies of 60 paintings, each shown for 40 s. We simultaneously measured oculomotor behavior and electrophysiological correlates of brain activity to compare early and late intervals (1) of inspection time of each picture (picture viewing) and (2) of the full experiment (time on task). For picture viewing, we found an increase in fixation duration and a decrease of saccadic amplitude while these parameters did not change with time on task. Furthermore, early in picture viewing we observed higher spatial and temporal similarity of gaze behavior. Analyzing electrical brain activity revealed changes in three components (C1, N1 and P2) of the eye fixation-related potential (EFRP); during picture viewing; no variation was obtained for the power in the frontal beta- and in the theta activity. Time on task analyses demonstrated no effects on the EFRP amplitudes but an increase of power in the frontal theta and beta band activity. Thus, behavioral and electrophysiological measures similarly show characteristic changes during picture viewing, indicating a shifting balance of its underlying (bottom-up and top-down) attentional mechanisms. Time on task also modulated top-down attention but probably represents a different attentional mechanism.

Saccadic context indicates information processing within visual fixations: Evidence from event-related potentials and eye-movements analysis of the distractor effect

Attention, visual information processing, and oculomotor control are integrated functions of closely related brain mechanisms. Recently, it was shown that the processing of visual distractors appearing during a fixation is modulated by the amplitude of its preceding saccade (Pannasch & Velichkovsky, 2009). So far, this was demonstrated only at the behavioral level in terms of saccadic inhibition. The present study investigated distractor-related brain activity with cortical eye fixation-related potentials (EFRPs). Moreover, the following saccade was included as an additional classification criterion. Eye movements and EFRPs were recorded during free visual exploration of paintings. During some of the fixations, a visual distractor was shown as an annulus around the fixation position, 100 ms after the fixation onset. The saccadic context of a fixation was classified by its preceding and following saccade amplitudes with the cut-off criterion set to 4° of visual angle. The prolongation of fixation duration induced by distractors was largest for fixations preceded and followed by short saccades. EFRP data revealed a difference in distractor-related P2 amplitude between the saccadic context conditions, following the same trend as in eye movements. Furthermore, influences of the following saccade amplitude on the latency of the saccadic inhibition and on the N1 amplitude were found. The EFRP results cannot be explained by the influence of saccades per se since this bias was removed by subtracting the baseline from the distractor EFRP. Rather, the data suggest that saccadic context indicates differences in how information is processed within single visual fixations.

Social communication with virtual agents: The effects of body and gaze direction on attention and emotional responding in human observers

In social communication, the gaze direction of other persons provides important information to perceive and interpret their emotional response. Previous research investigated the influence of gaze by manipulating mutual eye contact. Therefore, gaze and body direction have been changed as a whole, resulting in only congruent gaze and body directions (averted or directed) of another person. Here, we aimed to disentangle these effects by using short animated sequences of virtual agents posing with either direct or averted body or gaze. Attention allocation by means of eye movements, facial muscle response, and emotional experience to agents of different gender and facial expressions were investigated. Eye movement data revealed longer fixation durations, i.e., a stronger allocation of attention, when gaze and body direction were not congruent with each other or when both were directed towards the observer. This suggests that direct interaction as well as incongruous signals increase the demands of attentional resources in the observer. For the facial muscle response, only the reaction of muscle zygomaticus major revealed an effect of body direction, expressed by stronger activity in response to happy expressions for direct compared to averted gaze when the virtual characters body was directed towards the observer. Finally, body direction also influenced the emotional experience ratings towards happy expressions. While earlier findings suggested that mutual eye contact is the main source for increased emotional responding and attentional allocation, the present results indicate that direction of the virtual agents body and head also plays a minor but significant role.

8.2: Bi‐Directional OLED Microdisplay for Interactive HMD

First prototypes of bi-directional OLED microdisplay devices have been designed, that combine both display and camera functionality on a single CMOS chip (OLED-on-CMOS). Major aim of this integration is to provide capabilities for eye-tracking in see-through HMD to achieve gaze-based human-display-interaction, e.g. in augmented-reality applications.

Bi‐directional OLED microdisplay for interactive see‐through HMDs: Study toward integration of eye‐tracking and informational facilities

— First prototypes of bi-directional OLED microdisplay devices that combine both display and camera functionality on a single CMOS chip (OLED-on-CMOS) have been designed. The major goal of this integration is to provide capabilities for eye-tracking in see-through HMDs to achieve gaze-based human-display interaction, e.g., in augmented-reality applications. The development of the prototype was accompanied by user studies with a simulated bi-directional microdisplay consisting of a commercially available eye-tracker and a see-through HMD. These tests were aimed at providing basic minimum requirements in terms of temporal and spatial resolution of an eye-tracker to be implemented within the prototype, as well as to evaluate ergonomics of an appropriate user-interface design. A description of the current state of the hardware architecture and design aspects for bi-directional OLED microdisplays are also presented.

Continuous Gaze Cursor Feedback in Various Tasks: Influence on Eye Movement Behavior, Task Performance and Subjective Distraction

Using gaze as input modality has often been promoted as a method for advanced computer interaction. One important detail in gaze controlled interfaces is the design of optimal feedback. Highlighting the current point of gaze by a gaze contingent cursor represents a simple form of feedback. In an experimental study, we investigated the influence of gaze cursor feedback on eye movement behavior, task performance and subjective distraction. Participants of the study completed three different tasks (gaze typing, reading and image exploration) with five different feedback conditions. No-feedback was implemented as baseline condition and compared with gaze cursor feedback of various spatial precision and temporal delays. A blue, semitransparent small dot served as gaze cursor. The observed findings are discussed in the context of user friendly feedback for gaze based computer interaction.

Evaluating requirements for gaze-based interaction in a see-through head mounted display

This study suggests an approach to the evaluation of gaze-based interaction with information displayed on a see-through HMD. For these purposes, a mock-up system consisting of a head-mounted eye tracker and a see-through HMD was developed. In a series of three experiments subjects performed a simple point selection task using gaze control. Various design issues were addressed throughout including the size of interactive elements, their position on the screen, temporal and spatial resolution parameters of the eye tracker as well as algorithms used to enhance data quality. Reaction time and hit rate were measured as objective indicators of performance.