Boris Reuderink

Turning shortcomings into challenges: Brain–computer interfaces for games

In recent years we have seen a rising interest in brain-computer interfacing for human-computer interaction and potential game applications. Until now, however, we have almost only seen attempts where BCI is used to measure the affective state of the user or in neurofeeedback games. There have hardly been any attempts to design BCI games where BCI is considered to be one of the possible input modalities that can be used to control the game. One reason may be that research still follows the paradigms of the traditional, medically oriented, BCI approaches. In this paper we discuss current BCI research from the viewpoint of games and game design. It is hoped that this survey will make clear that we need to design different games than we used to, but that such games can nevertheless be interesting and exciting.

Brain-Computer Interfacing and Games

Recently research into Brain-Computer Interfacing (BCI) applications for healthy users, such as games, has been initiated. But why would a healthy person use a still-unproven technology such as BCI for game interaction? BCI provides a combination of information and features that no other input modality can offer. But for general acceptance of this technology, usability and user experience will need to be taken into account when designing such systems. Therefore, this chapter gives an overview of the state of the art of BCI in games and discusses the consequences of applying knowledge from Human-Computer Interaction (HCI) to the design of BCI for games. The integration of HCI with BCI is illustrated by research examples and showcases, intended to take this promising technology out of the lab. Future research needs to move beyond feasibility tests, to prove that BCI is also applicable in realistic, real-world settings.

Affective Pacman: A Frustrating Game for Brain-Computer Interface Experiments

We present the design and development of Affective Pacman, na game that induces frustration to study the effect of user state changes on the EEG signal. Affective Pacman is designed to induce frustration for short periods, and allows the synchronous recording of a wide range of sensors, such as physiological sensors and EEG in addition to the game state. A self-assessment is integrated in the game to track changes in user state. Preliminary results indicate a signicant effect of the frustration induction on the EEG.

Valence, arousal and dominance in the EEG during game play

In this paper, we describe our investigation of traces of naturally occurring emotions in electrical brain signals, that can be used to build interfaces that respond to our emotional state. This study confirms a number of known affective correlates in a realistic, uncontrolled environment for the emotions of valence (or pleasure), arousal and dominance: (1) a significant decrease in frontal power in the theta range is found for increasingly positive valence, (2) a significant frontal increase in power in the alpha range is associated with increasing emotional arousal, (3) a significant right posterior power increase in the delta range correlates with increasing arousal and (4) asymmetry in power in the lower alpha bands correlates with self-reported valence. Furthermore, asymmetry in the higher alpha bands correlates with self-reported dominance. These last two effects provide a simple measure for subjective feelings of pleasure and feelings of control.

Decision-Level Fusion for Audio-Visual Laughter Detection

Laughter is a highly variable signal, which can be caused by a spectrum of emotions. This makes the automatic detection of laughter a challenging, but interesting task. We perform automatic laughter detection using audio-visual data from the AMI Meeting Corpus. Audio-visual laughter detection is performed by fusing the results of separate audio and video classifiers on the decision level. This results in laughter detection with a significantly higher AUC-ROC than single-modality classification.

How Much Control Is Enough? Influence of Unreliable Input on User Experience

Brain-computer interfaces (BCI) provide a valuable new input modality within human-computer interaction systems. However, like other body-based inputs such as gesture or gaze based systems, the system recognition of input commands is still far from perfect. This raises important questions, such as what level of control should such an interface be able to provide. What is the relationship between actual and perceived control? And in the case of applications for entertainment in which fun is an important part of user experience, should we even aim for the highest level of control, or is the optimum elsewhere? In this paper, we evaluate whether we can modulate the amount of control and if a game can be fun with less than perfect control. In the experiment users (n=158) played a simple game in which a hamster has to be guided to the exit of a maze. The amount of control the user has over the hamster is varied. The variation of control through confusion matrices makes it possible to simulate the experience of using a BCI, while using the traditional keyboard for input. After each session the user completed a short questionnaire on user experience and perceived control. Analysis of the data showed that the perceived control of the user could largely be explained by the amount of control in the respective session. As expected, user frustration decreases with increasing control. Moreover, the results indicate that the relation between fun and control is not linear. Although at lower levels of control fun does increase with improved control, the level of fun drops just before perfect control is reached (with an optimum around 96%). This poses new insights for developers of games who want to incorporate some form of BCI or other modality with unreliable input in their game: for creating a fun game, unreliable input can be used to create a challenge for the user.

The Impact of Loss of Control on Movement BCIs

Brain-computer interfaces (BCIs) are known to suffer from spontaneous changes in the brain activity. If changes in the mental state of the user are reflected in the brain signals used for control, the behavior of a BCI is directly influenced by these states. We investigate the influence of a state of loss of control in a variant of Pacman on the performance of BCIs based on motor control. To study the effect a temporal loss of control has on the BCI performance, BCI classifiers were trained on electroencephalography (EEG) recorded during the normal control condition, and the classification performance on segments of EEG from the normal and loss of control condition was compared. Classifiers based on event-related desynchronization unexpectedly performed significantly better during the loss of control condition; for the event-related potential classifiers there was no significant difference in performance.

Evaluating User Experience of Actual and Imagined Movement in BCI Gaming

Most research on Brain-Computer Interfaces BCI focuses on developing ways of expression for disabled people who are not able to communicate through other means. Recently it has been shown that BCI can also be used in games to give users a richer experience and new ways to interact with a computer or game console. This paper describes research conducted to find out what the differences are between using actual and imagined movement as modalities in a BCI game. Results show that there are significant differences in user experience and that actual movement is a more robust way of communicating through a BCI.

Looking around with your brain in a virtual world

Offline analysis pipelines have been developed and evaluated for the detection of covert attention from electroen-cephalography recordings, and the detection of overt attention in terms of eye movement based on electrooculographic measurements. Some additional analysis were done in order to prepare the pipelines for use in a real-time system. This real-time system and a game application in which these pipelines are to be used were implemented. The game is set in a virtual environment where player is a wildlife photographer on an uninhabited island. Overt attention is used to adjust the angle of the first person camera, when the player is tracking animals. When making a photograph, the animal will flee when it notices it is looked at directly, so covert attention is required to get a good shot. Future work will entail user tests with this system to evaluate usability, user experience, and characteristics of the signals related to overt and covert attention when used in such an immersive environment.

Guessing what's on your mind: using the N400 in brain computer interfaces

In this paper, a method is proposed for using a simple neurophysiological brain response, the N400 potential, to determine a deeper underlying brain state. The goal is to construct a BCI that can determine what the user is thinking about, where thinking about is defined as being primed on. The results indicate that a subject can prime himself on a physical object by actively thinking about it during the experiment, as opposed to being shown explicit priming stimuli. Probe words are presented that elicit an N400 response which amplitude is modulated by the associative relatedness of the probe word to the object the user has primed himself on.