Alexander J. Doud

Continuous Three-Dimensional Control of a Virtual Helicopter Using a Motor Imagery Based Brain-Computer Interface

Brain-computer interfaces (BCIs) allow a user to interact with a computer system using thought. However, only recently have devices capable of providing sophisticated multi-dimensional control been achieved non-invasively. A major goal for non-invasive BCI systems has been to provide continuous, intuitive, and accurate control, while retaining a high level of user autonomy. By employing electroencephalography (EEG) to record and decode sensorimotor rhythms (SMRs) induced from motor imaginations, a consistent, user-specific control signal may be characterized. Utilizing a novel method of interactive and continuous control, we trained three normal subjects to modulate their SMRs to achieve three-dimensional movement of a virtual helicopter that is fast, accurate, and continuous. In this system, the virtual helicopters forward-backward translation and elevation controls were actuated through the modulation of sensorimotor rhythms that were converted to forces applied to the virtual helicopter at every simulation time step, and the helicopters angle of left or right rotation was linearly mapped, with higher resolution, from sensorimotor rhythms associated with other motor imaginations. These different resolutions of control allow for interplay between general intent actuation and fine control as is seen in the gross and fine movements of the arm and hand. Subjects controlled the helicopter with the goal of flying through rings (targets) randomly positioned and oriented in a three-dimensional space. The subjects flew through rings continuously, acquiring as many as 11 consecutive rings within a five-minute period. In total, the study group successfully acquired over 85% of presented targets. These results affirm the effective, three-dimensional control of our motor imagery based BCI system, and suggest its potential applications in biological navigation, neuroprosthetics, and other applications.

EEG Control of a Virtual Helicopter in 3-Dimensional Space Using Intelligent Control Strategies

Films like Firefox, Surrogates, and Avatar have explored the possibilities of using brain-computer interfaces (BCIs) to control machines and replacement bodies with only thought. Real world BCIs have made great progress toward that end. Invasive BCIs have enabled monkeys to fully explore 3-D space using neuroprosthetics. However, noninvasive BCIs have not been able to demonstrate such mastery of 3-D space. Here, we report our work, which demonstrates that human subjects can use a noninvasive BCI to fly a virtual helicopter to any point in a 3-D world. Through use of intelligent control strategies, we have facilitated the realization of controlled flight in 3-D space. We accomplished this through a reductionist approach that assigns subject-specific control signals to the crucial components of 3-D flight. Subject control of the helicopter was comparable when using either the BCI or a keyboard. By using intelligent control strategies, the strengths of both the user and the BCI system were leveraged and accentuated. Intelligent control strategies in BCI systems such as those presented here may prove to be the foundation for complex BCIs capable of doing more than we ever imagined.

Cortical Imaging of Event-Related (de)Synchronization During Online Control of Brain-Computer Interface Using Minimum-Norm Estimates in Frequency Domain

It is of wide interest to study the brain activity that correlates to the control of brain-computer interface (BCI). In the present study, we have developed an approach to image the cortical rhythmic modulation associated with motor imagery using minimum-norm estimates in the frequency domain (MNEFD). The distribution of cortical sources of mu activity during online control of BCI was obtained with the MNEFD. Contralateral decrease (event-related desynchronization) and ipsilateral increase (event-related synchronization) are localized in the sensorimotor cortex during online control of BCI in a group of human subjects. Statistical source analysis revealed that maximum correlation with movement imagination is localized in sensorimotor cortex.

Imaging the social brain: multi-subjects EEG recordings during the “Chicken's game”

In this study we measured simultaneously by EEG hyperscannings the neuroelectric activity in 6 couples of subjects during the performance of the “Chickens game”, derived from game theory. The simultaneous recording of the EEG in couples of interacting subjects allows to observe and model directly the neural signature of human interactions in order to understand the cerebral processes generating and generated by social cooperation or competition. Results suggested that the one of the most consistently activated structure in this particular social interaction paradigm is the left orbitofrontal cortex. Connectivity results also showed a significant involvement of the orbitofrontal regions of both hemispheres across the observed population. Taken together, results confirms that the study of the brain activities in humans during social interactions can take benefit from the simultaneous acquisition of brain activity during such interaction.

Estimation of the cortical activity from simultaneous multi-subject recordings during the prisoner's dilemma

One of the most challenging questions open in Neuroscience today is the characterization of the brain responses during social interaction. A major limitation of the approaches used in most of the studies performed so far is that only one of the participating brains is measured each time. The “interaction” between cooperating, competing or communicating brains is thus not measured directly, but inferred by independent observations aggregated by cognitive models and assumptions that link behavior and neural activation. In this paper, we present the results of the simultaneous neuroelectric recording of 5 couples of subjects engaged in cooperative games (EEG hyperscanning). The simultaneous recordings of couples of interacting subjects allows to observe and model directly the neural signature of human interactions in order to understand the cerebral processes generating and generated by social cooperation or competition. We used a paradigm called Prisoners dilemma derived from the game theory. Results collected in a population of 10 subjects suggested that the most consistently activated structure in social interaction paradigms is the orbitofrontal region (roughly described by the Brodmann area 10) during the condition of competition.

Simultaneous estimation of cortical activity during social interactions by using EEG hyperscannings

In this paper we show how the possibility of recording simultaneously the cerebral neuroelectric activity in different subjects (EEG hyperscanning) during the execution of different tasks could return useful information about the “internal” cerebral state of the subjects. We present the results obtained by EEG hyperscannings during ecological task (such as the execution of a card game) as well as that obtained in a series of couples of subjects during the performance of the Prisoners Dilemma Game. The simultaneous recordings of couples of interacting subjects allows to observe and to model directly the neural signature of human interactions in order to understand the cerebral processes generating and generated by social cooperation or competition. Results obtained in a study of different groups recorded during the card game revealed a larger activity in prefrontal and anterior cingulated cortex in different frequency bands for the player that leads the game when compared to other players. Results collected in a population of 10 subjects during the performance of the Prisoners Dilemma suggested that the most consistently activated structure is the orbitofrontal region (roughly described by the Brodmann area 10) during the condition of competition in both the tasks. It could be speculated whether the pattern of cortical connectivity between different cortical areas in different subjects could be employed as a tool for assessing the outcome of the task in advance.

Combined rTMS and virtual reality brain-computer interface training for motor recovery after stroke

OBJECTIVE Combining repetitive transcranial magnetic stimulation (rTMS) with brain-computer interface (BCI) training can address motor impairment after stroke by down-regulating exaggerated inhibition from the contralesional hemisphere and encouraging ipsilesional activation. The objective was to evaluate the efficacy of combined rTMS  +  BCI, compared to sham rTMS  +  BCI, on motor recovery after stroke in subjects with lasting motor paresis. APPROACH Three stroke subjects approximately one year post-stroke participated in three weeks of combined rTMS (real or sham) and BCI, followed by three weeks of BCI alone. Behavioral and electrophysiological differences were evaluated at baseline, after three weeks, and after six weeks of treatment. MAIN RESULTS Motor improvements were observed in both real rTMS  +  BCI and sham groups, but only the former showed significant alterations in inter-hemispheric inhibition in the desired direction and increased relative ipsilesional cortical activation from fMRI. In addition, significant improvements in BCI performance over time and adequate control of the virtual reality BCI paradigm were observed only in the former group. SIGNIFICANCE When combined, the results highlight the feasibility and efficacy of combined rTMS  +  BCI for motor recovery, demonstrated by increased ipsilesional motor activity and improvements in behavioral function for the real rTMS  +  BCI condition in particular. Our findings also demonstrate the utility of BCI training alone, as shown by behavioral improvements for the sham rTMS  +  BCI condition. This study is the first to evaluate combined rTMS and BCI training for motor rehabilitation and provides a foundation for continued work to evaluate the potential of both rTMS and virtual reality BCI training for motor recovery after stroke.

Cortical Imaging of Sensorimotor Rhythm during On-line Control of Brain-computer Interface

It is of wide interest to study the brain activity that correlates to the control of Brain-Computer Interface (BCI). In the present study, we propose an approach to image the cortical rhythmic modulation by motor imagery using minimum-norm estimates (MNE) in the frequency domain. Cortical distribution of mu activity during online control of BCI was obtained with the MNE. Statistical source analysis revealed maximum correlation with one-dimensional movement localized in sensorimotor cortex.

Continuous 3D control of a virtual helicopter using a motor imagery based BCI

Brain-computer interfaces (BCIs) are devices that allow for thought-based control of computer systems. However, sophisticated control of multi-dimensional BCIs has only recently been achieved in non-invasive systems. The design of these systems has focused on giving users fast, autonomous control that is both intuitive and accurate. Through the use of electroencephalographic recording techniques, sensorimotor rhythms induced from motor imaginations may be captured and a control signal may be characterized. Here we have trained two subjects with an interactive and continuous protocol to modulate their sensorimotor rhythms to control three-dimensions of motion of a virtual helicopter to reach randomly positioned and oriented rings. The subject group acquired 88% of presented targets and achieved as many as 11 consecutive rings in a five-minute period. Subjects learned to fly quickly, continuously and accurately through golden rings positioned and oriented randomly throughout a 3D virtual space.

Automated Electro-Mechanical Assessment of Psychomotor Skill for High-Stakes Certification in Surgical Robotics

In recent years, surgical robotic systems (SRS) have been employed to great effect in a wide range of minimally invasive procedures [1]. Yet despite the increasing use of these devices in clinical practice, there exists to date no definitive, objective system of measurement of the skill level of a surgeon using an SRS. Assessments of patient outcomes, surgical speed and instrument docking time have shown that improvement in skill using an SRS requires in-depth training, with exposure to as many as 20 cases of a given procedure to attain proficiency [2]. The model employed for the certification of a surgeon in the practice of minimally invasive laparoscopic surgery has addressed a parallel problem since its inception by establishing a set of exercises comprising what is now known as the fundamentals of laparoscopic surgery (FLS) assessment [3]. This high stakes examination tests the surgeon’s psychomotor skills in the crucial subtasks involved in the successful execution of laparoscopic surgery. The foundations of robotic surgery (FRS) project is an ongoing collaborative effort to create a similar, high-stakes assessment for surgical robotics undertaken by an international consortium of authoritative surgeons from multiple specialties. This consortium has conceptually identified and built consensus for the general system concept, subtasks, and objective performance metrics for robotic surgery, but does not intend to design or prototype working models [4]. We herein present an instrumented, electro-mechanical design and prototype of the overall system and two of the seven subtasks developed for eventual inclusion in the complete FRS high-stakes assessment [4]. The first subtask, the ring tower transfer, requires that the surgeon move a small ring positioned at the base of one tower to the base of another tower while avoiding contact between the ring and the tower along the way. The second subtask, knot tying, requires the test taker to use a provided silk suture to draw together two metal rings and secure them with a surgical knot. In both cases, our system provides automated, objective evaluation of the required performance metrics.