Joseph E. O'Doherty

Learning to Control a Brain–Machine Interface for Reaching and Grasping by Primates

Reaching and grasping in primates depend on the coordination of neural activity in large frontoparietal ensembles. Here we demonstrate that primates can learn to reach and grasp virtual objects by controlling a robot arm through a closed-loop brain–machine interface (BMIc) that uses multiple mathematical models to extract several motor parameters (i.e., hand position, velocity, gripping force, and the EMGs of multiple arm muscles) from the electrical activity of frontoparietal neuronal ensembles. As single neurons typically contribute to the encoding of several motor parameters, we observed that high BMIc accuracy required recording from large neuronal ensembles. Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance. Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move. Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.

Cortical Ensemble Adaptation to Represent Velocity of an Artificial Actuator Controlled by a Brain-Machine Interface

Monkeys can learn to directly control the movements of an artificial actuator by using a brain-machine interface (BMI) driven by the activity of a sample of cortical neurons. Eventually, they can do so without moving their limbs. Neuronal adaptations underlying the transition from control of the limb to control of the actuator are poorly understood. Here, we show that rapid modifications in neuronal representation of velocity of the hand and actuator occur in multiple cortical areas during the operation of a BMI. Initially, monkeys controlled the actuator by moving a hand-held pole. During this period, the BMI was trained to predict the actuator velocity. As the monkeys started using their cortical activity to control the actuator, the activity of individual neurons and neuronal populations became less representative of the animals hand movements while representing the movements of the actuator. As a result of this adaptation, the animals could eventually stop moving their hands yet continue to control the actuator. These results show that, during BMI control, cortical ensembles represent behaviorally significant motor parameters, even if these are not associated with movements of the animals own limb.

A brain-machine interface instructed by direct intracortical microstimulation

Brain–machine interfaces (BMIs) establish direct communication between the brain and artificial actuators. As such, they hold considerable promise for restoring mobility and communication in patients suffering from severe body paralysis. To achieve this end, future BMIs must also provide a means for delivering sensory signals from the actuators back to the brain. Prosthetic sensation is needed so that neuroprostheses can be better perceived and controlled. Here we show that a direct intracortical input can be added to a BMI to instruct rhesus monkeys in choosing the direction of reaching movements generated by the BMI. Somatosensory instructions were provided to two monkeys operating the BMI using either: (a) vibrotactile stimulation of the monkeys hands or (b) multi-channel intracortical microstimulation (ICMS) delivered to the primary somatosensory cortex (S1) in one monkey and posterior parietal cortex (PP) in the other. Stimulus delivery was contingent on the position of the computer cursor: the monkey placed it in the center of the screen to receive machine–brain recursive input. After 2 weeks of training, the same level of proficiency in utilizing somatosensory information was achieved with ICMS of S1 as with the stimulus delivered to the hand skin. ICMS of PP was not effective. These results indicate that direct, bi-directional communication between the brain and neuroprosthetic devices can be achieved through the combination of chronic multi-electrode recording and microstimulation of S1. We propose that in the future, bidirectional BMIs incorporating ICMS may become an effective paradigm for sensorizing neuroprosthetic devices.

Unscented Kalman Filter for Brain-Machine Interfaces

Brain machine interfaces (BMIs) are devices that convert neural signals into commands to directly control artificial actuators, such as limb prostheses. Previous real-time methods applied to decoding behavioral commands from the activity of populations of neurons have generally relied upon linear models of neural tuning and were limited in the way they used the abundant statistical information contained in the movement profiles of motor tasks. Here, we propose an n-th order unscented Kalman filter which implements two key features: (1) use of a non-linear (quadratic) model of neural tuning which describes neural activity significantly better than commonly-used linear tuning models, and (2) augmentation of the movement state variables with a history of n-1 recent states, which improves prediction of the desired command even before incorporating neural activity information and allows the tuning model to capture relationships between neural activity and movement at multiple time offsets simultaneously. This new filter was tested in BMI experiments in which rhesus monkeys used their cortical activity, recorded through chronically implanted multielectrode arrays, to directly control computer cursors. The 10th order unscented Kalman filter outperformed the standard Kalman filter and the Wiener filter in both off-line reconstruction of movement trajectories and real-time, closed-loop BMI operation.

Future developments in brain-machine interface research

Neuroprosthetic devices based on brain-machine interface technology hold promise for the restoration of body mobility in patients suffering from devastating motor deficits caused by brain injury, neurologic diseases and limb loss. During the last decade, considerable progress has been achieved in this multidisciplinary research, mainly in the brain-machine interface that enacts upper-limb functionality. However, a considerable number of problems need to be resolved before fully functional limb neuroprostheses can be built. To move towards developing neuroprosthetic devices for humans, brain-machine interface research has to address a number of issues related to improving the quality of neuronal recordings, achieving stable, long-term performance, and extending the brain-machine interface approach to a broad range of motor and sensory functions. Here, we review the future steps that are part of the strategic plan of the Duke University Center for Neuroengineering, and its partners, the Brazilian National Institute of Brain-Machine Interfaces and the École Polytechnique Fédérale de Lausanne (EPFL) Center for Neuroprosthetics, to bring this new technology to clinical fruition.

Adaptive decoding for brain-machine interfaces through bayesian parameter updates

Brain-machine interfaces (BMIs) transform the activity of neurons recorded in motor areas of the brain into movements of external actuators. Representation of movements by neuronal populations varies over time, during both voluntary limb movements and movements controlled through BMIs, due to motor learning, neuronal plasticity, and instability in recordings. To ensure accurate BMI performance over long time spans, BMI decoders must adapt to these changes. We propose the Bayesian regression self-training method for updating the parameters of an unscented Kalman filter decoder. This novel paradigm uses the decoders output to periodically update its neuronal tuning model in a Bayesian linear regression. We use two previously known statistical formulations of Bayesian linear regression: a joint formulation, which allows fast and exact inference, and a factorized formulation, which allows the addition and temporary omission of neurons from updates but requires approximate variational inference. To evaluate these methods, we performed offline reconstructions and closed-loop experiments with rhesus monkeys implanted cortically with microwire electrodes. Offline reconstructions used data recorded in areas M1, S1, PMd, SMA, and PP of three monkeys while they controlled a cursor using a handheld joystick. The Bayesian regression self-training updates significantly improved the accuracy of offline reconstructions compared to the same decoder without updates. We performed 11 sessions of real-time, closed-loop experiments with a monkey implanted in areas M1 and S1. These sessions spanned 29 days. The monkey controlled the cursor using the decoder with and without updates. The updates maintained control accuracy and did not require information about monkey hand movements, assumptions about desired movements, or knowledge of the intended movement goals as training signals. These results indicate that Bayesian regression self-training can maintain BMI control accuracy over long periods, making clinical neuroprosthetics more viable.

A learning-based approach to artificial sensory feedback leads to optimal integration

Proprioception—the sense of the bodys position in space—is important to natural movement planning and execution and will likewise be necessary for successful motor prostheses and brain–machine interfaces (BMIs). Here we demonstrate that monkeys were able to learn to use an initially unfamiliar multichannel intracortical microstimulation signal, which provided continuous information about hand position relative to an unseen target, to complete accurate reaches. Furthermore, monkeys combined this artificial signal with vision to form an optimal, minimum-variance estimate of relative hand position. These results demonstrate that a learning-based approach can be used to provide a rich artificial sensory feedback signal, suggesting a new strategy for restoring proprioception to patients using BMIs, as well as a powerful new tool for studying the adaptive mechanisms of sensory integration.

Cortical modulations increase in early sessions with brain-machine interface.

Background During planning and execution of reaching movements, the activity of cortical motor neurons is modulated by a diversity of motor, sensory, and cognitive signals. Brain-machine interfaces (BMIs) extract part of these modulations to directly control artificial actuators. However, cortical modulations that emerge in the novel context of operating the BMI are poorly understood. Methodology/Principal Findings Here we analyzed the changes in neuronal modulations that occurred in different cortical motor areas as monkeys learned to use a BMI to control reaching movements. Using spike-train analysis methods we demonstrate that the modulations of the firing-rates of cortical neurons increased abruptly after the monkeys started operating the BMI. Regression analysis revealed that these enhanced modulations were not correlated with the kinematics of the movement. The initial enhancement in firing rate modulations declined gradually with subsequent training in parallel with the improvement in behavioral performance. Conclusions/Significance We conclude that the enhanced modulations are related to computational tasks that are significant especially in novel motor contexts. Although the function and neuronal mechanism of the enhanced cortical modulations are open for further inquiries, we discuss their potential role in processing execution errors and representing corrective or explorative activity. These representations are expected to contribute to the formation of internal models of the external actuator and their decoding may facilitate BMI improvement.

Virtual Active Touch Using Randomly Patterned Intracortical Microstimulation

Intracortical microstimulation (ICMS) has promise as a means for delivering somatosensory feedback in neuroprosthetic systems. Various tactile sensations could be encoded by temporal, spatial, or spatiotemporal patterns of ICMS. However, the applicability of temporal patterns of ICMS to artificial tactile sensation during active exploration is unknown, as is the minimum discriminable difference between temporally modulated ICMS patterns. We trained rhesus monkeys in an active exploration task in which they discriminated periodic pulse-trains of ICMS (200 Hz bursts at a 10 Hz secondary frequency) from pulse trains with the same average pulse rate, but distorted periodicity (200 Hz bursts at a variable instantaneous secondary frequency). The statistics of the aperiodic pulse trains were drawn from a gamma distribution with mean inter-burst intervals equal to those of the periodic pulse trains. The monkeys distinguished periodic pulse trains from aperiodic pulse trains with coefficients of variation 0.25 or greater. Reconstruction of movement kinematics, extracted from the activity of neuronal populations recorded in the sensorimotor cortex concurrent with the delivery of ICMS feedback, improved when the recording intervals affected by ICMS artifacts were removed from analysis. These results add to the growing evidence that temporally patterned ICMS can be used to simulate a tactile sense for neuroprosthetic devices.

Stochastic Facilitation of Artificial Tactile Sensation in Primates

Artificial sensation via electrical or optical stimulation of brain sensory areas offers a promising treatment for sensory deficits. For a brain–machine–brain interface, such artificial sensation conveys feedback signals from a sensorized prosthetic limb. The ways neural tissue can be stimulated to evoke artificial sensation and the parameter space of such stimulation, however, remain largely unexplored. Here we investigated whether stochastic facilitation (SF) could enhance an artificial tactile sensation produced by intracortical microstimulation (ICMS). Two rhesus monkeys learned to use a virtual hand, which they moved with a joystick, to explore virtual objects on a computer screen. They sought an object associated with a particular artificial texture (AT) signaled by a periodic ICMS pattern delivered to the primary somatosensory cortex (S1) through a pair of implanted electrodes. During each behavioral trial, aperiodic ICMS (i.e., noise) of randomly chosen amplitude was delivered to S1 through another electrode pair implanted 1 mm away from the site of AT delivery. Whereas high-amplitude noise worsened AT detection, moderate noise clearly improved the detection of weak signals, significantly raising the proportion of correct trials. These findings suggest that SF could be used to enhance prosthetic sensation.