Laurens R. Krol

Towards BCI-Based Implicit Control in Human–Computer Interaction

In this chapter a specific aspect of Physiological Computing, that of implicit Human–Computer Interaction, is defined and discussed. Implicit Interaction aims at controlling a computer system by behavioural or psychophysiological aspects of user state, independently of any intentionally communicated command. This introduces a new type of Human–Computer Interaction, which in contrast to most forms of interaction implemented nowadays, does not require the user to explicitly communicate with the machine. Users can focus on understanding the current state of the system and developing strategies for optimally reaching the goal of the given interaction. For example, the system can assess the user state by means of passive Brain-Computer Interfaces, which the user needs not even be aware of. Based on this information and the given context the system can adapt automatically to the current strategies of the user. In a first study, a proof of principle is given, by implementing an Implicit Interaction to guide simple cursor movements in a 2D grid to a target. The results of this study clearly indicate the high potential of Implicit Interaction and introduce a new bandwidth of applications for passive Brain-Computer Interfaces.

Neuroadaptive technology enables implicit cursor control based on medial prefrontal cortex activity

Significance The human brain continuously and automatically processes information concerning its internal and external context. We demonstrate the elicitation and subsequent detection and decoding of such “automatic interpretations” by means of context-sensitive probes in an ongoing human–computer interaction. Through a sequence of such probe–interpretation cycles, the computer accumulates responses over time to model the operator’s cognition, even without that person being aware of it. This brings human cognition directly into the human–computer interaction loop, expanding traditional notions of “interaction.” The concept introduces neuroadaptive technology—technology which automatically adapts to an estimate of its operator’s mindset. This technology bears relevance to autoadaptive experimental designs, and opens up paradigm-shifting possibilities for human–machine systems in general. The effectiveness of today’s human–machine interaction is limited by a communication bottleneck as operators are required to translate high-level concepts into a machine-mandated sequence of instructions. In contrast, we demonstrate effective, goal-oriented control of a computer system without any form of explicit communication from the human operator. Instead, the system generated the necessary input itself, based on real-time analysis of brain activity. Specific brain responses were evoked by violating the operators’ expectations to varying degrees. The evoked brain activity demonstrated detectable differences reflecting congruency with or deviations from the operators’ expectations. Real-time analysis of this activity was used to build a user model of those expectations, thus representing the optimal (expected) state as perceived by the operator. Based on this model, which was continuously updated, the computer automatically adapted itself to the expectations of its operator. Further analyses showed this evoked activity to originate from the medial prefrontal cortex and to exhibit a linear correspondence to the degree of expectation violation. These findings extend our understanding of human predictive coding and provide evidence that the information used to generate the user model is task-specific and reflects goal congruency. This paper demonstrates a form of interaction without any explicit input by the operator, enabling computer systems to become neuroadaptive, that is, to automatically adapt to specific aspects of their operator’s mindset. Neuroadaptive technology significantly widens the communication bottleneck and has the potential to fundamentally change the way we interact with technology.

Evaluation of a Dry EEG System for Application of Passive Brain-Computer Interfaces in Autonomous Driving

We tested the applicability and signal quality of a 16 channel dry electroencephalography (EEG) system in a laboratory environment and in a car under controlled, realistic conditions. The aim of our investigation was an estimation how well a passive Brain-Computer Interface (pBCI) can work in an autonomous driving scenario. The evaluation considered speed and accuracy of self-applicability by an untrained person, quality of recorded EEG data, shifts of electrode positions on the head after driving-related movements, usability, and complexity of the system as such and wearing comfort over time. An experiment was conducted inside and outside of a stationary vehicle with running engine, air-conditioning, and muted radio. Signal quality was sufficient for standard EEG analysis in the time and frequency domain as well as for the use in pBCIs. While the influence of vehicle-induced interferences to data quality was insignificant, driving-related movements led to strong shifts in electrode positions. In general, the EEG system used allowed for a fast self-applicability of cap and electrodes. The assessed usability of the system was still acceptable while the wearing comfort decreased strongly over time due to friction and pressure to the head. From these results we conclude that the evaluated system should provide the essential requirements for an application in an autonomous driving context. Nevertheless, further refinement is suggested to reduce shifts of the system due to body movements and increase the headsets usability and wearing comfort.

Automated Task Load Detection with Electroencephalography: Towards Passive Brain–Computer Interfacing in Robotic Surgery

Automatic detection of the current task load of a surgeon in the theatre in real time could provide helpful information, to be used in supportive systems. For example, such information may enable the system to automatically support the surgeon when critical or stressful periods are detected, or to communicate to others when a surgeon is engaged in a complex maneuver and should not be disturbed. Passive brain–computer interfaces (BCI) infer changes in cognitive and affective state by monitoring and interpreting ongoing brain activity recorded via an electroencephalogram. The resulting information can then be used to automatically adapt a technological system to the human user. So far, passive BCI have mostly been investigated in laboratory settings, even though they are intended to be applied in real-world settings. In this study, a passive BCI was used to assess changes in task load of skilled surgeons performing both simple and complex surgical training tasks. Results indicate that the introduced methodo...

Meyendtris: a hands-free, multimodal tetris clone using eye tracking and passive BCI for intuitive neuroadaptive gaming

This paper introduces a completely hands-free version of Tetris that uses eye tracking and passive brain-computer interfacing (a real-time measurement and interpretation of brain activity) to replace existing game elements, as well as introduce novel ones. In Meyendtris, dwell time-based eye tracking replaces the games direct control elements, i.e. the movement of the tetromino. In addition to that, two mental states of the player influence the game in real time by means of passive brain-computer interfacing. First, a measure of the players relaxation is used to modulate the speed of the game (and the corresponding music). Second, when upon landing of a tetromino a state of error perception is detected in the players brain, this last landed tetromino is destroyed. Together, this results in a multimodal, hands-free version of the classic Tetris game that is no longer hindered by manual input bottlenecks, while engaging novel mental abilities of the player.

SEREEGA: Simulating event-related EEG activity

BACKGROUND Electroencephalography (EEG) is a popular method to monitor brain activity, but it is difficult to evaluate EEG-based analysis methods because no ground-truth brain activity is available for comparison. Therefore, in order to test and evaluate such methods, researchers often use simulated EEG data instead of actual EEG recordings. Simulated data can be used, among other things, to assess or compare signal processing and machine learning algorithms, to model EEG variabilities, and to design source reconstruction methods. NEW METHOD We present SEREEGA, Simulating Event-Related EEG Activity. SEREEGA is a free and open-source MATLAB-based toolbox dedicated to the generation of simulated epochs of EEG data. It is modular and extensible, at initial release supporting five different publicly available head models and capable of simulating multiple different types of signals mimicking brain activity. This paper presents the architecture and general workflow of this toolbox, as well as a simulated data set demonstrating some of its functions. The toolbox is available at https://github.com/lrkrol/SEREEGA. RESULTS The simulated data allows established analysis pipelines and classification methods to be applied and is capable of producing realistic results. COMPARISON WITH EXISTING METHODS Most simulated EEG is coded from scratch. The few open-source methods in existence focus on specific applications or signal types, such as connectivity. SEREEGA unifies the majority of past simulation methods reported in the literature into one toolbox. CONCLUSION SEREEGA is a general-purpose toolbox to simulate ground-truth EEG data.

Physiological Effects of Adaptive Cruise Control Behaviour in Real Driving

We examined physiological responses to behavior of an Adaptive Cruise Control (ACC) system during real driving. ACC is an example of automating a task that used to be performed by the user. In order to preserve the link between the user and an automated system such that they work together optimally, physiological signals reflecting mental state may be useful. We asked 15 participants to use an ACC at designated times while driving a track. When the ACC was activated, the car decelerated either strongly or softly, which was either according to expectation or not. Heart rate, eye blinks, and brain signals (EEG) were recorded. Heart rate and blink duration were the same following the announcement of an upcoming expected or unexpected deceleration profile. Heart rate and blink duration increased when a strong compared to a soft deceleration profile was announced, consistent with a state of arousal or startle. This was only found for the first half of the trials, when the driver was expected to be more alert and engaged (as also evidenced by decreasing heart rate, and increasing EEG alpha and blink duration over the trials). We conclude that for ACC behavior that is relevant for the driver, heart rate and blink duration may be used as a source of information about mental state elicited by the ACC, which could be used to evaluate driving experience.

Towards a Conceptual Framework for Cognitive Probing

Cognitive probing combines the ability of computers to interpret ongoing measures of arbitrary brain activity, with the ability of those same computers to actively elicit cognitive responses from their users. Purposefully elicited responses can be interpreted in order to learn about the user, enable symbiotic and implicit interaction, and support neuroadaptive technology. We propose a working definition of cognitive probing that allows it to be generalised across different applications and disciplines.

A task-independent workload classifier for neuroadaptive technology: Preliminary data

Passive brain-computer interfacing allows computer systems direct access to aspects of their users cognition. In essence, a computer system can gain information about its user without this user needing to explicitly communicate it. Based on this information, human-computer interaction can be made more symmetrical, solving an age-old but still fundamental problem of present-day interaction techniques. For practical real-world application of this technology, it is important that cognitive states can be identified accurately and efficiently. Here we present preliminary data demonstrating it is possible to calibrate a task-independent classifier to identify when a user is under heavy workload across different activities. We used different types of mental arithmetic and even a semantic task. Task-independent classification is an important step towards real-world practical application of this technology.