Ilkka Kosunen

The influence of implicit and explicit biofeedback in first-person shooter games

To understand how implicit and explicit biofeedback work in games, we developed a first-person shooter (FPS) game to experiment with different biofeedback techniques. While this area has seen plenty of discussion, there is little rigorous experimentation addressing how biofeedback can enhance human-computer interaction. In our two-part study, (N=36) subjects first played eight different game stages with two implicit biofeedback conditions, with two simulation-based comparison and repetition rounds, then repeated the two biofeedback stages when given explicit information on the biofeedback. The biofeedback conditions were respiration and skin-conductance (EDA) adaptations. Adaptation targets were four balanced player avatar attributes. We collected data with psycho¬physiological measures (electromyography, respiration, and EDA), a game experience questionnaire, and game-play measures. According to our experiment, implicit biofeedback does not produce significant effects in player experience in an FPS game. In the explicit biofeedback conditions, players were more immersed and positively affected, and they were able to manipulate the game play with the biosignal interface. We recommend exploring the possibilities of using explicit biofeedback interaction in commercial games.

Predicting term-relevance from brain signals

Term-Relevance Prediction from Brain Signals (TRPB) is proposed to automatically detect relevance of text information directly from brain signals. An experiment with forty participants was conducted to record neural activity of participants while providing relevance judgments to text stimuli for a given topic. High-precision scientific equipment was used to quantify neural activity across 32 electroencephalography (EEG) channels. A classifier based on a multi-view EEG feature representation showed improvement up to 17% in relevance prediction based on brain signals alone. Relevance was also associated with brain activity with significant changes in certain brain areas. Consequently, TRPB is based on changes identified in specific brain areas and does not require user-specific training or calibration. Hence, relevance predictions can be conducted for unseen content and unseen participants. As an application of TRPB we demonstrate a high-precision variant of the classifier that constructs sets of relevant terms for a given unknown topic of interest. Our research shows that detecting relevance from brain signals is possible and allows the acquisition of relevance judgments without a need to observe any other user interaction. This suggests that TRPB could be used in combination or as an alternative for conventional implicit feedback signals, such as dwell time or click-through activity.

Keep your opponents close: social context affects EEG and fEMG linkage in a turn-based computer game

In daily life, we often copy the gestures and expressions of those we communicate with, but recent evidence shows that such mimicry has a physiological counterpart: interaction elicits linkage, which is a concordance between the biological signals of those involved. To find out how the type of social interaction affects linkage, pairs of participants played a turn-based computer game in which the level of competition was systematically varied between cooperation and competition. Linkage in the beta and gamma frequency bands was observed in the EEG, especially when the participants played directly against each other. Emotional expression, measured using facial EMG, reflected this pattern, with the most competitive condition showing enhanced linkage over the facial muscle-regions involved in smiling. These effects were found to be related to self-reported social presence: linkage in positive emotional expression was associated with self-reported shared negative feelings. The observed effects confirmed the hypothesis that the social context affected the degree to which participants had similar reactions to their environment and consequently showed similar patterns of brain activity. We discuss the functional resemblance between linkage, as an indicator of a shared physiology and affect, and the well-known mirror neuron system, and how they relate to social functions like empathy.

Emotionally Adapted Games --- An Example of a First Person Shooter

This paper discusses a specific customization technology --- Psychological Customization - which enables the customization of information presented on a computer-based system in real-time and its application to manipulating emotions when playing computer games. The possibilities of customizing different elements of games to manipulate emotions are presented and a definition of emotionally adaptive games is given. A psychophysiologically adaptive game is discussed as an example of emotionally adapted games.

Incorporating subliminal perception in synthetic environments

Advanced interactive visualization such as in virtual environments and ubiquitous interaction paradigms pose new challenges and opportunities in considering real-time responses to subliminal cues. In this paper, we propose a synthetic reality platform that, combined with psychophysiological recordings, enables us to study in realtime the effects of various subliminal cues. We endeavor to integrate various aspects known to be relevant to implicit perception. The context is of consumer experience and choice of an artifact where the generation of subliminal perception through an intelligent 3D interface controls the spatio-temporal aspects of the information displayed and of the emergent narrative. One novel contribution of this work is the programmable nature of the interface that exploits known perceptive phenomena (e.g. masking, crowding and change blindness) to generate subliminal perception.

Experience Assessment and Design in the Analysis of Gameplay

We report research on player modeling using psychophysiology and machine learning, conducted through interdisciplinary collaboration between researchers of computer science, psychology, and game design at Aalto University, Helsinki. First, we propose the Play Patterns And eXperience (PPAX) framework to connect three levels of game experience that previously had remained largely unconnected: game design patterns, the interplay of game context with player personality or tendencies, and state-of-the-art measures of experience (both subjective and non-subjective). Second, we describe our methodology for using machine learning to categorize game events to reveal corresponding patterns, culminating in an example experiment. We discuss the relation between automatically detected event clusters and game design patterns, and provide indications on how to incorporate personality profiles of players in the analysis. This novel interdisciplinary collaboration combines basic psychophysiology research with game design patterns and machine learning, and generates new knowledge about the interplay between game experience and design.

Design and implementation of an affect-responsive interactive photo frame

This paper describes an affect-responsive interactive photo-frame application that offers its user a different experience with every use. It relies on visual analysis of activity levels and facial expressions of its users to select responses from a database of short video segments. This ever-growing database is automatically prepared by an offline analysis of user-uploaded videos. The resulting system matches its user’s affect along dimensions of valence and arousal, and gradually adapts its response to each specific user. In an extended mode, two such systems are coupled and feed each other with visual content. The strengths and weaknesses of the system are assessed through a usability study, where a Wizard-of-Oz response logic is contrasted with the fully automatic system that uses affective and activity-based features, either alone, or in tandem.

Extracting relevance and affect information from physiological text annotation

We present physiological text annotation, which refers to the practice of associating physiological responses to text content in order to infer characteristics of the user information needs and affective responses. Text annotation is a laborious task, and implicit feedback has been studied as a way to collect annotations without requiring any explicit action from the user. Previous work has explored behavioral signals, such as clicks or dwell time to automatically infer annotations, and physiological signals have mostly been explored for image or video content. We report on two experiments in which physiological text annotation is studied first to (1) indicate perceived relevance and then to (2) indicate affective responses of the users. The first experiment tackles the user’s perception of relevance of an information item, which is fundamental towards revealing the user’s information needs. The second experiment is then aimed at revealing the user’s affective responses towards a -relevant- text document. Results show that physiological user signals are associated with relevance and affect. In particular, electrodermal activity was found to be different when users read relevant content than when they read irrelevant content and was found to be lower when reading texts with negative emotional content than when reading texts with neutral content. Together, the experiments show that physiological text annotation can provide valuable implicit inputs for personalized systems. We discuss how our findings help design personalized systems that can annotate digital content using human physiology without the need for any explicit user interaction.

Neuroadaptive Meditation in the Real World

Meditation and mindfulness techniques are useful for both treatment of various disorders as well as improving the quality of life in general. Meditation offers intriguing possibilities for BCI as it is targeted at able-bodied general population and goes beyond the traditional explicit control BCI paradigm. In previous work, we have shown how neurofeedback can be successfully applied in a laboratory setting to improve the meditation experience. This position paper aims to expand this work in two ways. First, we explore the problems and issues that might arise when moving from the laboratory setting to the normal, everyday world. Second, we will consider the possibilities of extending the neurofeedback with other forms of physiological computing. Our position is that meditation and relaxation applications provide a perfect application area for bringing BCI into the real world.

BCI for Physiological Text Annotation

Automatic annotation of media content has become a critically important task for many digital services as the quantity of available online media content has grown exponentially. One approach is to annotate the content using the physiological responses of the media consumer. In the present paper, we reflect on three case studies that use brain signals for implicit text annotation to discuss the challenges faced when bringing passive brain-computer interfaces for physiological text annotation to the real world.