Markus Plank

EEG theta and Mu oscillations during perception of human and robot actions

The perception of others’ actions supports important skills such as communication, intention understanding, and empathy. Are mechanisms of action processing in the human brain specifically tuned to process biological agents? Humanoid robots can perform recognizable actions, but can look and move differently from humans, and as such, can be used in experiments to address such questions. Here, we recorded EEG as participants viewed actions performed by three agents. In the Human condition, the agent had biological appearance and motion. The other two conditions featured a state-of-the-art robot in two different appearances: Android, which had biological appearance but mechanical motion, and Robot, which had mechanical appearance and motion. We explored whether sensorimotor mu (8–13 Hz) and frontal theta (4–8 Hz) activity exhibited selectivity for biological entities, in particular for whether the visual appearance and/or the motion of the observed agent was biological. Sensorimotor mu suppression has been linked to the motor simulation aspect of action processing (and the human mirror neuron system, MNS), and frontal theta to semantic and memory-related aspects. For all three agents, action observation induced significant attenuation in the power of mu oscillations, with no difference between agents. Thus, mu suppression, considered an index of MNS activity, does not appear to be selective for biological agents. Observation of the Robot resulted in greater frontal theta activity compared to the Android and the Human, whereas the latter two did not differ from each other. Frontal theta thus appears to be sensitive to visual appearance, suggesting agents that are not sufficiently biological in appearance may result in greater memory processing demands for the observer. Studies combining robotics and neuroscience such as this one can allow us to explore neural basis of action processing on the one hand, and inform the design of social robots on the other.

Simultaneous Neural and Movement Recording in Large-Scale Immersive Virtual Environments

Virtual reality (VR) allows precise control and manipulation of rich, dynamic stimuli that, when coupled with on-line motion capture and neural monitoring, can provide a powerful means both of understanding brain behavioral relations in the high dimensional world and of assessing and treating a variety of neural disorders. Here we present a system that combines state-of-the-art, fully immersive, 3D, multi-modal VR with temporally aligned electroencephalographic (EEG) recordings. The VR system is dynamic and interactive across visual, auditory, and haptic interactions, providing sight, sound, touch, and force. Crucially, it does so with simultaneous EEG recordings while subjects actively move about a 20×20 ft2 space. The overall end-to-end latency between real movement and its simulated movement in the VR is approximately 40 ms. Spatial precision of the various devices is on the order of millimeters. The temporal alignment with the neural recordings is accurate to within approximately 1 ms. This powerful combination of systems opens up a new window into brain-behavioral relations and a new means of assessment and rehabilitation of individuals with motor and other disorders.

Resting-state fMRI activity predicts unsupervised learning and memory in an immersive virtual reality environment

In the real world, learning often proceeds in an unsupervised manner without explicit instructions or feedback. In this study, we employed an experimental paradigm in which subjects explored an immersive virtual reality environment on each of two days. On day 1, subjects implicitly learned the location of 39 objects in an unsupervised fashion. On day 2, the locations of some of the objects were changed, and object location recall performance was assessed and found to vary across subjects. As prior work had shown that functional magnetic resonance imaging (fMRI) measures of resting-state brain activity can predict various measures of brain performance across individuals, we examined whether resting-state fMRI measures could be used to predict object location recall performance. We found a significant correlation between performance and the variability of the resting-state fMRI signal in the basal ganglia, hippocampus, amygdala, thalamus, insula, and regions in the frontal and temporal lobes, regions important for spatial exploration, learning, memory, and decision making. In addition, performance was significantly correlated with resting-state fMRI connectivity between the left caudate and the right fusiform gyrus, lateral occipital complex, and superior temporal gyrus. Given the basal ganglias role in exploration, these findings suggest that tighter integration of the brain systems responsible for exploration and visuospatial processing may be critical for learning in a complex environment.

EEG Gamma Band Oscillations Differentiate the Planning of Spatially Directed Movements of the Arm Versus Eye: Multivariate Empirical Mode Decomposition Analysis

The neural dynamics underlying the coordination of spatially-directed limb and eye movements in humans is not well understood. Part of the difficulty has been a lack of signal processing tools suitable for the analysis of nonstationary electroencephalographic (EEG) signals. Here, we use multivariate empirical mode decomposition (MEMD), a data-driven approach that does not employ predefined basis functions. High-density EEG, and arm and eye movements were synchronously recorded in 10 subjects performing time-constrained reaching and/or eye movements. Subjects were allowed to move both the hand and the eyes, only the hand, or only the eyes following a 500-700 ms delay interval where the hand and gaze remained on a central fixation cross. An additional condition involved a nonspatially-directed “lift” movement of the hand. The neural activity during a 500 ms delay interval was decomposed into intrinsic mode functions (IMFs) using MEMD. Classification analysis revealed that gamma band (30 Hz <;) IMFs produced more classifiable features differentiating the EEG according to the different upcoming movements. A benchmark test using conventional algorithms demonstrated that MEMD was the best algorithm for extracting oscillatory bands from EEG, yielding the best classification of the different movement conditions. The gamma rhythm decomposed using MEMD showed a higher correlation with the eventual movement accuracy than any other band rhythm and than any other algorithm.

Human Cortical θ during Free Exploration Encodes Space and Predicts Subsequent Memory

Spatial representations and walking speed in rodents are consistently related to the phase, frequency, and/or amplitude of θ rhythms in hippocampal local field potentials. However, neuropsychological studies in humans have emphasized the importance of parietal cortex for spatial navigation, and efforts to identify the electrophysiological signs of spatial navigation in humans have been stymied by the difficulty of recording during free exploration of complex environments. We resolved the recording problem and experimentally probed brain activity of human participants who were fully ambulant. On each of 2 d, electroencephalography was synchronized with head and body movement in 13 subjects freely navigating an extended virtual environment containing numerous unique objects. θ phase and amplitude recorded over parietal cortex were consistent when subjects walked through a particular spatial separation at widely separated times. This spatial displacement θ autocorrelation (STAcc) was quantified and found to be significant from 2 to 8 Hz within the environment. Similar autocorrelation analyses performed on an electrooculographic channel, used to measure eye movements, showed no significant spatial autocorrelations, ruling out eye movements as the source of STAcc. Strikingly, the strength of an individuals STAcc maps from day 1 significantly predicted object location recall success on day 2. θ was also significantly correlated with walking speed; however, this correlation appeared unrelated to STAcc and did not predict memory performance. This is the first demonstration of memory-related, spatial maps in humans generated during active spatial exploration.

Dopamine function and the efficiency of human movement

To sustain successful behavior in dynamic environments, active organisms must be able to learn from the consequences of their actions and predict action outcomes. One of the most important discoveries in systems neuroscience over the last 15 years has been about the key role of the neurotransmitter dopamine in mediating such active behavior. Dopamine cell firing was found to encode differences between the expected and obtained outcomes of actions. Although activity of dopamine cells does not specify movements themselves, a recent study in humans has suggested that tonic levels of dopamine in the dorsal striatum may in part enable normal movement by encoding sensitivity to the energy cost of a movement, providing an implicit “motor motivational” signal for movement. We investigated the motivational hypothesis of dopamine by studying motor performance of patients with Parkinson disease who have marked dopamine depletion in the dorsal striatum and compared their performance with that of elderly healthy adults. All participants performed rapid sequential movements to visual targets associated with different risk and different energy costs, countered or assisted by gravity. In conditions of low energy cost, patients performed surprisingly well, similar to prescriptions of an ideal planner and healthy participants. As energy costs increased, however, performance of patients with Parkinson disease dropped markedly below the prescriptions for action by an ideal planner and below performance of healthy elderly participants. The results indicate that the ability for efficient planning depends on the energy cost of action and that the effect of energy cost on action is mediated by dopamine.

EEG Activity During Movement Planning Encodes Upcoming Peak Speed and Acceleration and Improves the Accuracy in Predicting Hand Kinematics

The relationship between movement kinematics and human brain activity is an important and fundamental question for the development of neural prosthesis. The peak velocity and the peak acceleration could best reflect the feedforward-type movement; thus, it is worthwhile to investigate them further. Most related studies focused on the correlation between kinematics and brain activity during the movement execution or imagery. However, human movement is the result of the motor planning phase as well as the execution phase and researchers have demonstrated that statistical correlations exist between EEG activity during the motor planning and the peak velocity and the peak acceleration using grand-average analysis. In this paper, we examined whether the correlations were concealed in trial-to-trial decoding from the low signal-to-noise ratio of EEG activity. The alpha and beta powers from the movement planning phase were combined with the alpha and beta powers from the movement execution phase to predict the peak tangential speed and acceleration. The results showed that EEG activity from the motor planning phase could also predict the peak speed and the peak acceleration with a reasonable accuracy. Furthermore, the decoding accuracy of the peak speed and the peak acceleration could both be improved by combining band powers from the motor planning phase with the band powers from the movement execution.

Causal analysis of cortical networks involved in reaching to spatial targets.

The planning of goal-directed movement towards targets in different parts of space is an important function of the brain. Such visuo-motor planning and execution is known to involve multiple brain regions, including visual, parietal, and frontal cortices. To understand how these brain regions work together to both plan and execute goal-directed movement, it is essential to describe the dynamic causal interactions among them. Here we model causal interactions of distributed cortical source activity derived from non-invasively recorded EEG, using a combination of ICA, minimum-norm distributed source localization (cLORETA), and dynamical modeling within the Source Information Flow Toolbox (SIFT). We differentiate network causal connectivity of reach planning and execution, by comparing the causal network in a speeded reaching task with that for a control task not requiring goal-directed movement. Analysis of a pilot dataset (n=5) shows the utility of this technique and reveals increased connectivity between visual, motor and frontal brain regions during reach planning, together with decreased cross-hemisphere visual coupling during planning and execution, possibly related to task demands.

Alpha and beta band power changes predict reaction time and endpoint error during planning reaching movements

Reaching is one of the fundamental tasks in daily life thus the neural processing of such a task will benefit brain computer interface (BCI) development. Even the simplest reaching task results from a movement planning phase as well as a movement execution phase. To facilitate the neural decoding of human reaching, brain activity in both movement planning and execution phases should be investigated. Many related studies focused on the correlation between movement kinematics and brain activity during movement execution or imagery. In this paper, we studied whether particular reaching parameters were encoded in the brain activity recorded by high-density EEG during the movement planning phase. Alpha and beta power over ten brain regions during the movement planning phase were correlated with reaction time/endpoint error during execution. The planning phase was divided into 2 movement planning intervals: 1) [0 250]ms and 2) [250 500]ms, with respect to the target onset to study the temporal characteristics of the power changes. Our results show that the greater the increase of the alpha band power in the left frontal region and the smaller the decrease of the beta band power in the right frontal region in the movement planning interval 2, the longer is the reaction time. A decrease of the beta band power in the motor and frontal regions from the two planning intervals correspond to a higher endpoint error. Moreover, higher alpha band power in the movement planning interval 2 over left parietal regions was related to smaller endpoint errors.

Neurocognitive stages of spatial cognitive mapping measured during free exploration of a large-scale virtual environment.

Using a novel, fully mobile virtual reality paradigm, we investigated the EEG correlates of spatial representations formed during unsupervised exploration. On day 1, subjects implicitly learned the location of 39 objects by exploring a room and popping bubbles that hid the objects. On day 2, they again popped bubbles in the same environment. In most cases, the objects hidden underneath the bubbles were in the same place as on day 1. However, a varying third of them were misplaced in each block. Subjects indicated their certainty that the object was in the same location as the day before. Compared with bubble pops revealing correctly placed objects, bubble pops revealing misplaced objects evoked a decreased negativity starting at 145 ms, with scalp topography consistent with generation in medial parietal cortex. There was also an increased negativity starting at 515 ms to misplaced objects, with scalp topography consistent with generation in inferior temporal cortex. Additionally, misplaced objects elicited an increase in frontal midline theta power. These findings suggest that the successive neurocognitive stages of processing allocentric space may include an initial template matching, integration of the object within its spatial cognitive map, and memory recall, analogous to the processing negativity N400 and theta that support verbal cognitive maps in humans.