Lukas Dominique Josef Fiederer

Deep learning with convolutional neural networks for EEG decoding and visualization

Deep learning with convolutional neural networks (deep ConvNets) has revolutionized computer vision through end‐to‐end learning, that is, learning from the raw data. There is increasing interest in using deep ConvNets for end‐to‐end EEG analysis, but a better understanding of how to design and train ConvNets for end‐to‐end EEG decoding and how to visualize the informative EEG features the ConvNets learn is still needed. Here, we studied deep ConvNets with a range of different architectures, designed for decoding imagined or executed tasks from raw EEG. Our results show that recent advances from the machine learning field, including batch normalization and exponential linear units, together with a cropped training strategy, boosted the deep ConvNets decoding performance, reaching at least as good performance as the widely used filter bank common spatial patterns (FBCSP) algorithm (mean decoding accuracies 82.1% FBCSP, 84.0% deep ConvNets). While FBCSP is designed to use spectral power modulations, the features used by ConvNets are not fixed a priori. Our novel methods for visualizing the learned features demonstrated that ConvNets indeed learned to use spectral power modulations in the alpha, beta, and high gamma frequencies, and proved useful for spatially mapping the learned features by revealing the topography of the causal contributions of features in different frequency bands to the decoding decision. Our study thus shows how to design and train ConvNets to decode task‐related information from the raw EEG without handcrafted features and highlights the potential of deep ConvNets combined with advanced visualization techniques for EEG‐based brain mapping. Hum Brain Mapp 38:5391–5420, 2017.

Visualization of the Amygdalo-Hippocampal Border and its Structural Variability by 7T and 3T Magnetic Resonance Imaging

The amygdala and the hippocampus are two adjacent structures in the medial temporal lobe that have been broadly investigated in functional and structural neuroimaging due to their central importance in sensory perception, emotion, and memory. Exact demarcation of the amygdalo‐hippocampal border (AHB) is, however, difficult in conventional structural imaging. Recent evidence suggests that, due to this difficulty, functional activation sites with high probability of being located in the hippocampus may erroneously be assigned to the amygdala, and vice versa. In the present study, we investigated the potential of ultra‐high‐field magnetic resonance imaging (MRI) in single sessions for detecting the AHB in humans. We show for the first time the detailed structure of the AHB as it can be visualized in T1‐weighted 7T in vivo images at 0.5‐mm3 isotropic resolution. Compared to data acquired at 3T, 7T images revealed considerably more structural detail in the AHB region. Thus, we observed a striking inter‐hemispheric and interindividual variability of the exact anatomical configuration of the AHB that points to the necessity of individual imaging of the AHB as a prerequisite for accurate anatomical assignment in this region. The findings of the present study demonstrate the usefulness of ultra‐high‐field structural MRI to resolve anatomical ambiguities of the human AHB. Highly accurate morphometric and functional investigations in this region at 7T may allow addressing such hitherto unexplored issues as whether the structural configuration of the AHB is related to functional differences in amygdalo‐hippocampal interaction. Hum Brain Mapp 35:4316–4329, 2014.

The role of blood vessels in high-resolution volume conductor head modeling of EEG

Reconstruction of the electrical sources of human EEG activity at high spatiotemporal accuracy is an important aim in neuroscience and neurological diagnostics. Over the last decades, numerous studies have demonstrated that realistic modeling of head anatomy improves the accuracy of source reconstruction of EEG signals. For example, including a cerebrospinal fluid compartment and the anisotropy of white matter electrical conductivity were both shown to significantly reduce modeling errors. Here, we for the first time quantify the role of detailed reconstructions of the cerebral blood vessels in volume conductor head modeling for EEG. To study the role of the highly arborized cerebral blood vessels, we created a submillimeter head model based on ultra-high-field-strength (7 T) structural MRI datasets. Blood vessels (arteries and emissary/intraosseous veins) were segmented using Frangi multi-scale vesselness filtering. The final head model consisted of a geometry-adapted cubic mesh with over 17 × 106 nodes. We solved the forward model using a finite-element-method (FEM) transfer matrix approach, which allowed reducing computation times substantially and quantified the importance of the blood vessel compartment by computing forward and inverse errors resulting from ignoring the blood vessels. Our results show that ignoring emissary veins piercing the skull leads to focal localization errors of approx. 5 to 15 mm. Large errors (>2 cm) were observed due to the carotid arteries and the dense arterial vasculature in areas such as in the insula or in the medial temporal lobe. Thus, in such predisposed areas, errors caused by neglecting blood vessels can reach similar magnitudes as those previously reported for neglecting white matter anisotropy, the CSF or the dura — structures which are generally considered important components of realistic EEG head models. Our findings thus imply that including a realistic blood vessel compartment in EEG head models will be helpful to improve the accuracy of EEG source analyses particularly when high accuracies in brain areas with dense vasculature are required.

An autonomous robotic assistant for drinking

Stroke and neurodegenerative diseases, among a range of other neurologic disorders, can cause chronic paralysis. Patients suffering from paralysis may remain unable to achieve even basic everyday tasks such as liquid intake. Currently, there is a great interest in developing robotic assistants controlled via brain-machine interfaces (BMIs) to restore the ability to perform such tasks. This paper describes an autonomous robotic assistant for liquid intake. The components of the system include autonomous online detection both of the cup to be grasped and of the mouth of the user. It plans motions of the robot arm under the constraints that the cup stays upright while moving towards the mouth and that the cup stays in direct contact with the users mouth while the robot tilts it during the drinking phase. To achieve this, our system also includes a technique for online estimation of the location of the users mouth even under partial occlusions by the cup or robot arm. We tested our system in a real environment and in a shared-control setting using frequency-specific modulations recorded by electroencephalography (EEG) from the brain of the user. Our experiments demonstrate that our BMI-controlled robotic system enables a reliable liquid intake. We believe that our approach can easily be extended to other useful tasks including food intake and object manipulation.

A brain-computer interface for high-level remote control of an autonomous, reinforcement-learning-based robotic system for reaching and grasping

We present an Internet-based brain-computer interface (BCI) for controlling an intelligent robotic device with autonomous reinforcement-learning. BCI control was achieved through dry-electrode electroencephalography (EEG) obtained during imaginary movements. Rather than using low-level direct motor control, we employed a high-level control scheme of the robot, acquired via reinforcement learning, to keep the users cognitive load low while allowing control a reaching-grasping task with multiple degrees of freedom. High-level commands were obtained by classification of EEG responses using an artificial neural network approach utilizing time-frequency features and conveyed through an intuitive user interface. The novel ombination of a rapidly operational dry electrode setup, autonomous control and Internet connectivity made it possible to conveniently interface subjects in an EEG laboratory with remote robotic devices in a closed-loop setup with online visual feedback of the robots actions to the subject. The same approach is also suitable to provide home-bound patients with the possibility to control state-of-the-art robotic devices currently confined to a research environment. Thereby, our BCI approach could help severely paralyzed patients by facilitating patient-centered research of new means of communication, mobility and independence.

Mapping the fine structure of cortical activity with different micro-ECoG electrode array geometries

OBJECTIVE Innovations in micro-electrocorticography (µECoG) electrode array manufacturing now allow for intricate designs with smaller contact diameters and/or pitch (i.e. inter-contact distance) down to the sub-mm range. The aims of the present study were: (i) to investigate whether frequency ranges up to 400 Hz can be reproducibly observed in µECoG recordings and (ii) to examine how differences in topographical substructure between these frequency bands and electrode array geometries can be quantified. We also investigated, for the first time, the influence of blood vessels on signal properties and assessed the influence of cortical vasculature on topographic mapping. APPROACH The present study employed two µECoG electrode arrays with different contact diameters and inter-contact distances, which were used to characterize neural activity from the somatosensory cortex of minipigs in a broad frequency range up to 400 Hz. The analysed neural data were recorded in acute experiments under anaesthesia during peripheral electrical stimulation. MAIN RESULTS We observed that µECoG recordings reliably revealed multi-focal cortical somatosensory response patterns, in which response peaks were often less than 1 cm apart and would thus not have been resolvable with conventional ECoG. The response patterns differed by stimulation site and intensity, they were distinct for different frequency bands, and the results of functional mapping proved independent of cortical vascular. Our analysis of different frequency bands exhibited differences in the number of activation peaks in topographical substructures. Notably, signal strength and signal-to-noise ratios differed between the two electrode arrays, possibly due to their different sensitivity for variations in spatial patterns and signal strengths. SIGNIFICANCE Our findings that the geometry of µECoG electrode arrays can strongly influence their recording performance can help to make informed decisions that maybe important in number of clinical contexts, including high-resolution brain mapping, advanced epilepsy diagnostics or brain-machine interfacing.

Electrical Stimulation of the Human Cerebral Cortex by Extracranial Muscle Activity: Effect Quantification With Intracranial EEG and FEM Simulations

Objective: Electric fields (EF) of approx. 0.2 V/m have been shown to be sufficiently strong to both modulate neuronal activity in the cerebral cortex and have measurable effects on cognitive performance. We hypothesized that the EF caused by the electrical activity of extracranial muscles during natural chewing may reach similar strength in the cerebral cortex and hence might act as an endogenous modality of brain stimulation. Here, we present first steps toward validating this hypothesis. Methods: Using a realistic volume conductor head model of an epilepsy patient having undergone intracranial electrode placement and utilizing simultaneous intracranial and extracranial electrical recordings during chewing, we derive predictions about the chewing-related cortical EF strength to be expected in healthy individuals. Results: We find that in the region of the temporal poles, the expected EF strength may reach amplitudes in the order of 0.1-1 V/m. Conclusion: The cortical EF caused by natural chewing could be large enough to modulate ongoing neural activity in the cerebral cortex and influence cognitive performance. Significance: The present study lends first support for the assumption that extracranial muscle activity might represent an endogenous source of electrical brain stimulation. This offers a new potential explanation for the puzzling effects of gum chewing on cognition, which have been repeatedly reported in the literature.

Acting thoughts: Towards a mobile robotic service assistant for users with limited communication skills

As autonomous service robots become more affordable and thus available also for the general public, there is a growing need for user friendly interfaces to control the robotic system. Currently available control modalities typically expect users to be able to express their desire through either touch, speech or gesture commands. While this requirement is fulfilled for the majority of users, paralyzed users may not be able to use such systems. In this paper, we present a novel framework, that allows these users to interact with a robotic service assistant in a closed-loop fashion, using only thoughts. The brain-computer interface (BCI) system is composed of several interacting components, i.e., non-invasive neuronal signal recording and decoding, high-level task planning, motion and manipulation planning as well as environment perception. In various experiments, we demonstrate its applicability and robustness in real world scenarios, considering fetch-and-carry tasks and tasks involving human-robot interaction. As our results demonstrate, our system is capable of adapting to frequent changes in the environment and reliably completing given tasks within a reasonable amount of time. Combined with high-level planning and autonomous robotic systems, interesting new perspectives open up for non-invasive BCI-based human-robot interactions.

Deep Learning Based BCI Control of a Robotic Service Assistant Using Intelligent Goal Formulation

As autonomous service robots become more affordable and thus available for the general public, there is a growing need for user-friendly interfaces to control these systems. Control interfaces typically get more complicated with increasing complexity of the robotic tasks and the environment. Traditional control modalities as touch, speech or gesture commands are not necessarily suited for all users. While non-expert users can make the effort to familiarize themselves with a robotic system, paralyzed users may not be capable of controlling such systems even though they need robotic assistance most. In this paper, we present a novel framework, that allows these users to interact with a robotic service assistant in a closed-loop fashion, using only thoughts. The system is composed of several interacting components: non-invasive neuronal signal recording and co-adaptive deep learning which form the brain-computer interface (BCI), high-level task planning based on referring expressions, navigation and manipulation planning as well as environmental perception. We extensively evaluate the BCI in various tasks, determine the performance of the goal formulation user interface and investigate its intuitiveness in a user study. Furthermore, we demonstrate the applicability and robustness of the system in real world scenarios, considering fetch-and-carry tasks and tasks involving human-robot interaction. As our results show, the system is capable of adapting to frequent changes in the environment and reliably accomplishes given tasks within a reasonable amount of time. Combined with high-level planning using referring expressions and autonomous robotic systems, interesting new perspectives open up for non-invasive BCI-based human-robot interactions.

The dynamics of error processing in the human brain as reflected by high-gamma activity in noninvasive and intracranial EEG

&NA; Error detection in motor behavior is a fundamental cognitive function heavily relying on local cortical information processing. Neural activity in the high‐gamma frequency band (HGB) closely reflects such local cortical processing, but little is known about its role in error processing, particularly in the healthy human brain. Here we characterize the error‐related response of the human brain based on data obtained with noninvasive EEG optimized for HGB mapping in 31 healthy subjects (15 females, 16 males), and additional intracranial EEG data from 9 epilepsy patients (4 females, 5 males). Our findings reveal a multiscale picture of the global and local dynamics of error‐related HGB activity in the human brain. On the global level as reflected in the noninvasive EEG, the error‐related response started with an early component dominated by anterior brain regions, followed by a shift to parietal regions, and a subsequent phase characterized by sustained parietal HGB activity. This phase lasted for more than 1 s after the error onset. On the local level reflected in the intracranial EEG, a cascade of both transient and sustained error‐related responses involved an even more extended network, spanning beyond frontal and parietal regions to the insula and the hippocampus. HGB mapping appeared especially well suited to investigate late, sustained components of the error response, possibly linked to downstream functional stages such as error‐related learning and behavioral adaptation. Our findings establish the basic spatio‐temporal properties of HGB activity as a neural correlate of error processing, complementing traditional error‐related potential studies. Graphical abstract Figure. No caption available.