Marcel A. J. van Gerven

The brain–computer interface cycle

Brain-computer interfaces (BCIs) have attracted much attention recently, triggered by new scientific progress in understanding brain function and by impressive applications. The aim of this review is to give an overview of the various steps in the BCI cycle, i.e., the loop from the measurement of brain activity, classification of data, feedback to the subject and the effect of feedback on brain activity. In this article we will review the critical steps of the BCI cycle, the present issues and state-of-the-art results. Moreover, we will develop a vision on how recently obtained results may contribute to new insights in neurocognition and, in particular, in the neural representation of perceived stimuli, intended actions and emotions. Now is the right time to explore what can be gained by embracing real-time, online BCI and by adding it to the set of experimental tools already available to the cognitive neuroscientist. We close by pointing out some unresolved issues and present our view on how BCI could become an important new tool for probing human cognition.

Attention modulations of posterior alpha as a control signal for two-dimensional brain-computer interfaces

Research on brain-computer interfaces (BCIs) is gaining strong interest. This is motivated by BCIs being applicable for helping disabled, for gaming, and as a tool in cognitive neuroscience. Often, motor imagery is used to produce (binary) control signals. However, finding other types of control signals that allow the discrimination of multiple classes would help to increase the applicability of BCIs. We have investigated if modulation of posterior alpha activity by means of covert spatial attention in two dimensions can be reliably classified in single trials. Magnetoencephalography (MEG) data were collected for 15 subjects who were engaged in a task where they covertly had to visually attend left, right, up or down during a period of 2500 ms. We then classified the trials using support vector machines. The four orientations of covert attention could be reliably classified up to a maximum of 69% correctly classified trials (25% chance level) without the need for lengthy and burdensome subject training. Low classification performance in some subjects was explained by a low alpha signal. These findings support the case that modulation of alpha activity by means of covert spatial attention is promising as a control signal for a two-dimensional BCI.

Dynamic Bayesian networks as prognostic models for clinical patient management

Prognostic models in medicine are usually been built using simple decision rules, proportional hazards models, or Markov models. Dynamic Bayesian networks (DBNs) offer an approach that allows for the incorporation of the causal and temporal nature of medical domain knowledge as elicited from domain experts, thereby allowing for detailed prognostic predictions. The aim of this paper is to describe the considerations that must be taken into account when constructing a DBN for complex medical domains and to demonstrate their usefulness in practice. To this end, we focus on the construction of a DBN for prognosis of carcinoid patients, compare performance with that of a proportional hazards model, and describe predictions for three individual patients. We show that the DBN can make detailed predictions, about not only patient survival, but also other variables of interest, such as disease progression, the effect of treatment, and the development of complications. Strengths and limitations of our approach are discussed and compared with those offered by traditional methods.

Identifying object categories from event-related EEG: toward decoding of conceptual representations

Multivariate pattern analysis is a technique that allows the decoding of conceptual information such as the semantic category of a perceived object from neuroimaging data. Impressive single-trial classification results have been reported in studies that used fMRI. Here, we investigate the possibility to identify conceptual representations from event-related EEG based on the presentation of an object in different modalities: its spoken name, its visual representation and its written name. We used Bayesian logistic regression with a multivariate Laplace prior for classification. Marked differences in classification performance were observed for the tested modalities. Highest accuracies (89% correctly classified trials) were attained when classifying object drawings. In auditory and orthographical modalities, results were lower though still significant for some subjects. The employed classification method allowed for a precise temporal localization of the features that contributed to the performance of the classifier for three modalities. These findings could help to further understand the mechanisms underlying conceptual representations. The study also provides a first step towards the use of concept decoding in the context of real-time brain-computer interface applications.

Efficient Bayesian multivariate fMRI analysis using a sparsifying spatio-temporal prior

Bayesian logistic regression with a multivariate Laplace prior is introduced as a multivariate approach to the analysis of neuroimaging data. It is shown that, by rewriting the multivariate Laplace distribution as a scale mixture, we can incorporate spatio-temporal constraints which lead to smooth importance maps that facilitate subsequent interpretation. The posterior of interest is computed using an approximate inference method called expectation propagation and becomes feasible due to fast inversion of a sparse precision matrix. We illustrate the performance of the method on an fMRI dataset acquired while subjects were shown handwritten digits. The obtained models perform competitively in terms of predictive performance and give rise to interpretable importance maps. Estimation of the posterior of interest is shown to be feasible even for very large models with thousands of variables.

Iconic and multi-stroke gesture recognition

Many handwritten gestures, characters, and symbols comprise multiple pendown strokes separated by penup strokes. In this paper, a large number of features known from the literature are explored for the recognition of such multi-stroke gestures. Features are computed from a global gesture shape. From its constituent strokes, the mean and standard deviation of each feature are computed. We show that using these new stroke-based features, significant improvements in classification accuracy can be obtained between 10% and 50% compared to global feature representations. These results are consistent over four different databases, containing iconic pen gestures, handwritten symbols, and upper-case characters. Compared to two other multi-stroke recognition techniques, improvements between 25% and 39% are achieved, averaged over all four databases.

Covert attention allows for continuous control of brain-computer interfaces

While brain‐computer interfaces (BCIs) can be used for controlling external devices, they also hold the promise of providing a new tool for studying the working brain. In this study we investigated whether modulations of brain activity by changes in covert attention can be used as a continuous control signal for BCI. Covert attention is the act of mentally focusing on a peripheral sensory stimulus without changing gaze direction. The ongoing brain activity was recorded using magnetoencephalography in subjects as they covertly attended to a moving cue while maintaining fixation. Based on posterior alpha power alone, the direction to which subjects were attending could be recovered using circular regression. Results show that the angle of attention could be predicted with a mean absolute deviation of 51° in our best subject. Averaged over subjects, the mean deviation was ∼70°. In terms of information transfer rate, the optimal data length used for recovering the direction of attention was found to be 1700 ms; this resulted in a mean absolute deviation of 60° for the best subject. The results were obtained without any subject‐specific feature selection and did not require prior subject training. Our findings demonstrate that modulations of posterior alpha activity due to the direction of covert attention has potential as a control signal for continuous control in a BCI setting. Our approach will have several applications, including a brain‐controlled computer mouse and improved methods for neuro‐feedback that allow direct training of subjects’ ability to modulate posterior alpha activity.

Brain-computer interfacing using modulations of alpha activity induced by covert shifts of attention.

BackgroundVisual brain-computer interfaces (BCIs) often yield high performance only when targets are fixated with the eyes. Furthermore, many paradigms use intense visual stimulation, which can be irritating especially in long BCI sessions. However, BCIs can more directly directly tap the neural processes underlying visual attention. Covert shifts of visual attention induce changes in oscillatory alpha activity in posterior cortex, even in the absence of visual stimulation. The aim was to investigate whether different pairs of directions of attention shifts can be reliably differentiated based on the electroencephalogram. To this end, healthy participants (N = 8) had to strictly fixate a central dot and covertly shift visual attention to one out of six cued directions.ResultsCovert attention shifts induced a prolonged alpha synchronization over posterior electrode sites (PO and O electrodes). Spectral changes had specific topographies so that different pairs of directions could be differentiated. There was substantial variation across participants with respect to the direction pairs that could be reliably classified. Mean accuracy for the best-classifiable pair amounted to 74.6%. Furthermore, an alpha power index obtained during a relaxation measurement showed to be predictive of peak BCI performance (r = .66).ConclusionsResults confirm posterior alpha power modulations as a viable input modality for gaze-independent EEG-based BCIs. The pair of directions yielding optimal performance varies across participants. Consequently, participants with low control for standard directions such as left-right might resort to other pairs of directions including top and bottom. Additionally, a simple alpha index was shown to predict prospective BCI performance.

Interpreting single trial data using groupwise regularisation

Univariate statistical approaches are often used for the analysis of neuroimaging data but are unable to detect subtle interactions between different components of brain activity. In contrast, multivariate approaches that use classification as a basis are well-suited to detect such interactions, allowing the analysis of neuroimaging data on the single trial level. However, multivariate approaches typically assign a non-zero contribution to every component, making interpretation of the results troublesome. This paper introduces groupwise regularisation as a novel method for finding sparse, and therefore easy to interpret, models that are able to predict the experimental condition to which single trials belong. Furthermore, the obtained models can be constrained in various ways by placing features extracted from the data that are thought to belong together into groups. In order to learn models from data, we introduce a new algorithm that makes use of stability conditions that have been derived in this paper. The algorithm is used to classify multisensor EEG signals recorded for a motor imagery task using (groupwise) regularised logistic regression as the underlying classifier. We show that regularisation dramatically reduces the number of features without reducing the classification rate. This improves model interpretability as it finds features in the data such as mu and beta desynchronisation in the motor cortex contralateral to the imagined movement. By choosing particular groupings we can constrain the regularised solutions such that a lower number of sensors is used or a model is obtained that generalises well over subjects. The identification of a small number of groups of features that best explain the data make groupwise regularisation a useful new tool for single trial analysis.

Modality-Independent Decoding of Semantic Information from the Human Brain

An ability to decode semantic information from fMRI spatial patterns has been demonstrated in previous studies mostly for 1 specific input modality. In this study, we aimed to decode semantic category independent of the modality in which an object was presented. Using a searchlight method, we were able to predict the stimulus category from the data while participants performed a semantic categorization task with 4 stimulus modalities (spoken and written names, photographs, and natural sounds). Significant classification performance was achieved in all 4 modalities. Modality-independent decoding was implemented by training and testing the searchlight method across modalities. This allowed the localization of those brain regions, which correctly discriminated between the categories, independent of stimulus modality. The analysis revealed large clusters of voxels in the left inferior temporal cortex and in frontal regions. These voxels also allowed category discrimination in a free recall session where subjects recalled the objects in the absence of external stimuli. The results show that semantic information can be decoded from the fMRI signal independently of the input modality and have clear implications for understanding the functional mechanisms of semantic memory.