R. Jacob Vogelstein

Restoring the sense of touch with a prosthetic hand through a brain interface.

Significance Our ability to manipulate objects relies fundamentally on sensory signals originating from the hand. To restore motor function with upper-limb neuroprostheses requires that somatosensory feedback be provided to the tetraplegic patient or amputee. Accordingly, we have developed approaches to convey sensory information critical for object manipulation—information about contact location, pressure, and timing—through intracortical microstimulation of somatosensory cortex. In experiments with nonhuman primates, we show that we can elicit percepts that are projected to a localized patch of skin, that track the pressure exerted on the skin, and that signal the timing of contact events. We anticipate that the proposed biomimetic feedback will constitute an important step in restoring touch to individuals who have lost it. Our ability to manipulate objects dexterously relies fundamentally on sensory signals originating from the hand. To restore motor function with upper-limb neuroprostheses requires that somatosensory feedback be provided to the tetraplegic patient or amputee. Given the complexity of state-of-the-art prosthetic limbs and, thus, the huge state space they can traverse, it is desirable to minimize the need for the patient to learn associations between events impinging on the limb and arbitrary sensations. Accordingly, we have developed approaches to intuitively convey sensory information that is critical for object manipulation—information about contact location, pressure, and timing—through intracortical microstimulation of primary somatosensory cortex. In experiments with nonhuman primates, we show that we can elicit percepts that are projected to a localized patch of skin and that track the pressure exerted on the skin. In a real-time application, we demonstrate that animals can perform a tactile discrimination task equally well whether mechanical stimuli are delivered to their native fingers or to a prosthetic one. Finally, we propose that the timing of contact events can be signaled through phasic intracortical microstimulation at the onset and offset of object contact that mimics the ubiquitous on and off responses observed in primary somatosensory cortex to complement slowly varying pressure-related feedback. We anticipate that the proposed biomimetic feedback will considerably increase the dexterity and embodiment of upper-limb neuroprostheses and will constitute an important step in restoring touch to individuals who have lost it.

A Multichip Neuromorphic System for Spike-Based Visual Information Processing

We present a multichip, mixed-signal VLSI system for spike-based vision processing. The system consists of an 80 60 pixel neuromorphic retina and a 4800 neuron silicon cortex with 4,194,304 synapses. Its functionality is illustrated with experimental data on multiple components of an attention-based hierarchical model of cortical object recognition, including feature coding, salience detection, and foveation. This model exploits arbitrary and reconfigurable connectivity between cells in the multichip architecture, achieved by asynchronously routing neural spike events within and between chips according to a memory-based look-up table. Synaptic parameters, including conductance and reversal potential, are also stored in memory and are used to dynamically configure synapse circuits within the silicon neurons.

Demonstration of a Semi-Autonomous Hybrid Brain–Machine Interface Using Human Intracranial EEG, Eye Tracking, and Computer Vision to Control a Robotic Upper Limb Prosthetic

To increase the ability of brain-machine interfaces (BMIs) to control advanced prostheses such as the modular prosthetic limb (MPL), we are developing a novel system: the Hybrid Augmented Reality Multimodal Operation Neural Integration Environment (HARMONIE). This system utilizes hybrid input, supervisory control, and intelligent robotics to allow users to identify an object (via eye tracking and computer vision) and initiate (via brain-control) a semi-autonomous reach-grasp-and-drop of the object by the MPL. Sequential iterations of HARMONIE were tested in two pilot subjects implanted with electrocortico-graphic (ECoG) and depth electrodes within motor areas. The subjects performed the complex task in 71.4% (20/28) and 67.7% (21/31) of trials after minimal training. Balanced accuracy for detecting movements was 91.1% and 92.9%, significantly greater than chance accuracies (p <; 0.05). After BMI-based initiation, the MPL completed the entire task 100% (one object) and 70% (three objects) of the time. The MPL took approximately 12.2 s for task completion after system improvements implemented for the second subject. Our hybrid-BMI design prevented all but one baseline false positive from initiating the system. The novel approach demonstrated in this proof-of-principle study, using hybrid input, supervisory control, and intelligent robotics, addresses limitations of current BMIs.

Simultaneous Neural Control of Simple Reaching and Grasping With the Modular Prosthetic Limb Using Intracranial EEG

Intracranial electroencephalographic (iEEG) signals from two human subjects were used to achieve simultaneous neural control of reaching and grasping movements with the Johns Hopkins University Applied Physics Lab (JHU/APL) Modular Prosthetic Limb (MPL), a dexterous robotic prosthetic arm. We performed functional mapping of high gamma activity while the subject made reaching and grasping movements to identify task-selective electrodes. Independent, online control of reaching and grasping was then achieved using high gamma activity from a small subset of electrodes with a model trained on short blocks of reaching and grasping with no further adaptation. Classification accuracy did not decline (p <; 0.05, one-way ANOVA) over three blocks of testing in either subject. Mean classification accuracy during independently executed overt reach and grasp movements for (Subject 1, Subject 2) were (0.85, 0.81) and (0.80, 0.96), respectively, and during simultaneous execution they were (0.83, 0.88) and (0.58, 0.88), respectively. Our models leveraged knowledge of the subjects individual functional neuroanatomy for reaching and grasping movements, allowing rapid acquisition of control in a time-sensitive clinical setting. We demonstrate the potential feasibility of verifying functionally meaningful iEEG-based control of the MPL prior to chronic implantation, during which additional capabilities of the MPL might be exploited with further training.

Dynamic control of the central pattern generator for locomotion

We show that an ongoing locomotor pattern can be dynamically controlled by applying discrete pulses of electrical stimulation to the central pattern generator (CPG) for locomotion. Data are presented from a pair of experiments on biological (wetware) and electrical (hardware) models of the CPG demonstrating that stimulation causes brief deviations from the CPG’s limit cycle activity. The exact characteristics of the deviation depend strongly on the phase of stimulation. Applications of this work are illustrated by examples showing how locomotion can be controlled by using a feedback loop to monitor CPG activity and applying stimuli at the appropriate times to modulate motor output. Eventually, this approach could lead to development of a neuroprosthetic device for restoring locomotion after paralysis.

The open connectome project data cluster: scalable analysis and vision for high-throughput neuroscience

We describe a scalable database cluster for the spatial analysis and annotation of high-throughput brain imaging data, initially for 3-d electron microscopy image stacks, but for time-series and multi-channel data as well. The system was designed primarily for workloads that build connectomes---neural connectivity maps of the brain---using the parallel execution of computer vision algorithms on high-performance compute clusters. These services and open-science data sets are publicly available at openconnecto.me. The system design inherits much from NoSQL scale-out and data-intensive computing architectures. We distribute data to cluster nodes by partitioning a spatial index. We direct I/O to different systems---reads to parallel disk arrays and writes to solid-state storage---to avoid I/O interference and maximize throughput. All programming interfaces are RESTful Web services, which are simple and stateless, improving scalability and usability. We include a performance evaluation of the production system, highlighting the effectiveness of spatial data organization.

A real-time virtual integration environment for the design and development of neural prosthetic systems

We have developed a virtual integration environment (VIE) for the development of neural prosthetic systems. The VIE is a software environment that modularizes the core functions of a neural prosthetic system — receiving signals, decoding signals and controlling a real or simulated device. Complete prosthetic systems can be quickly assembled by linking pre-existing modules together through standard interfaces. Systems can be simulated in real-time, and simulated components can be swapped out for real hardware. This paper is the first of two companion papers that describe the VIE and its use. In this paper, we first describe the architecture of the VIE and review implemented modules. We then describe the use of the VIE for the real-time validation of neural decode algorithms from pre-recorded data, the use of the VIE in closed loop primate experiments and the use of the VIE in the clinic.

Fast approximate quadratic programming for graph matching.

Quadratic assignment problems arise in a wide variety of domains, spanning operations research, graph theory, computer vision, and neuroscience, to name a few. The graph matching problem is a special case of the quadratic assignment problem, and graph matching is increasingly important as graph-valued data is becoming more prominent. With the aim of efficiently and accurately matching the large graphs common in big data, we present our graph matching algorithm, the Fast Approximate Quadratic assignment algorithm. We empirically demonstrate that our algorithm is faster and achieves a lower objective value on over 80% of the QAPLIB benchmark library, compared with the previous state-of-the-art. Applying our algorithm to our motivating example, matching C. elegans connectomes (brain-graphs), we find that it efficiently achieves performance.

Designing Tyrosine-Derived Polycarbonate Polymers for Biodegradable Regenerative Type Neural Interface Capable of Neural Recording

Next-generation neuroprosthetic limbs will require a reliable long-term neural interface to residual nerves in the peripheral nervous system (PNS). To this end, we have developed novel biocompatible materials and a fabrication technique to create high site-count microelectrodes for stimulating and recording from regenerated peripheral nerves. Our electrodes are based on a biodegradable tyrosine-derived polycarbonate polymer system with suitable degradation and erosion properties and a fabrication technique for deployment of the polymer in a porous, degradable, regenerative, multiluminal, multielectrode conduit. The in vitro properties of the polymer and the electrode were tuned to retain mechanical strength for over 24 days and to completely degrade and erode within 220 days. The fabrication technique resulted in a multiluminal conduit with at least 10 functioning electrodes maintaining recording site impedance in the single-digit kOhm range. Additionally, in vivo results showed that neural signals could be recorded from these devices starting at four weeks postimplantation and that signal strength increased over time. We conclude that our biodegradable regenerative-type neural interface is a good candidate for chronic high fidelity recording electrodes for integration with regenerated peripheral nerves.

WiiEMG: A real-time environment for control of the Wii with surface electromyography

We have developed a hardware and software platform, the WiiEMG, for controlling the Wii™ video game console with surface electromyography (EMG). WiiEMG is intended to assist with training and performance assessment of myoelectric control of upper arm prostheses. For this application, a player wears skin surface electrodes over myoelectric control sites. A real-time signal analysis system acquires amplified EMG signals and classifies the activity patterns associated with different motions. In addition, the amplitude of this pattern is used as a velocity signal, which is differentiated to give acceleration. Finally, a scaled version of this acceleration value is input as an analog voltage into a modified Wiimote™ in place of the normal accelerometer, and the Wiimote communicates the data to the console. To evaluate the systems performance, six able-bodied subjects were used to test the WiiEMG by playing the game Wii Tennis™ using myoelectric control. Results are reported that show how users develop improved EMG control after only a few brief trials. Improved muscle and EMG control has the potential to benefit myoelectric limb use as well as motor skills rehabilitation.