José del R. Millán

Noninvasive brain-actuated control of a mobile robot by human EEG

Brain activity recorded noninvasively is sufficient to control a mobile robot if advanced robotics is used in combination with asynchronous electroencephalogram (EEG) analysis and machine learning techniques. Until now brain-actuated control has mainly relied on implanted electrodes, since EEG-based systems have been considered too slow for controlling rapid and complex sequences of movements. We show that two human subjects successfully moved a robot between several rooms by mental control only, using an EEG-based brain-machine interface that recognized three mental states. Mental control was comparable to manual control on the same task with a performance ratio of 0.74.

Combining Brain–Computer Interfaces and Assistive Technologies: State-of-the-Art and Challenges

In recent years, new research has brought the field of electroencephalogram (EEG)-based brain–computer interfacing (BCI) out of its infancy and into a phase of relative maturity through many demonstrated prototypes such as brain-controlled wheelchairs, keyboards, and computer games. With this proof-of-concept phase in the past, the time is now ripe to focus on the development of practical BCI technologies that can be brought out of the lab and into real-world applications. In particular, we focus on the prospect of improving the lives of countless disabled individuals through a combination of BCI technology with existing assistive technologies (AT). In pursuit of more practical BCIs for use outside of the lab, in this paper, we identify four application areas where disabled individuals could greatly benefit from advancements in BCI technology, namely, “Communication and Control”, “Motor Substitution”, “Entertainment”, and “Motor Recovery”. We review the current state of the art and possible future developments, while discussing the main research issues in these four areas. In particular, we expect the most progress in the development of technologies such as hybrid BCI architectures, user–machine adaptation algorithms, the exploitation of users’ mental states for BCI reliability and confidence measures, the incorporation of principles in human–computer interaction (HCI) to improve BCI usability, and the development of novel BCI technology including better EEG devices.

The BCI competition III: validating alternative approaches to actual BCI problems

A brain-computer interface (BCI) is a system that allows its users to control external devices with brain activity. Although the proof-of-concept was given decades ago, the reliable translation of user intent into device control commands is still a major challenge. Success requires the effective interaction of two adaptive controllers: the users brain, which produces brain activity that encodes intent, and the BCI system, which translates that activity into device control commands. In order to facilitate this interaction, many laboratories are exploring a variety of signal analysis techniques to improve the adaptation of the BCI system to the user. In the literature, many machine learning and pattern classification algorithms have been reported to give impressive results when applied to BCI data in offline analyses. However, it is more difficult to evaluate their relative value for actual online use. BCI data competitions have been organized to provide objective formal evaluations of alternative methods. Prompted by the great interest in the first two BCI Competitions, we organized the third BCI Competition to address several of the most difficult and important analysis problems in BCI research. The paper describes the data sets that were provided to the competitors and gives an overview of the results.

Person Authentication Using Brainwaves (EEG) and Maximum A Posteriori Model Adaptation

In this paper, we investigate the use of brain activity for person authentication. It has been shown in previous studies that the brainwave pattern of every individual is unique and that the electroencephalogram (EEG) can be used for biometric identification. EEG-based biometry is an emerging research topic and we believe that it may open new research directions and applications in the future. However, very little work has been done in this area and was focusing mainly on person identification but not on person authentication. Person authentication aims to accept or to reject a person claiming an identity, i.e., comparing a biometric data to one template, while the goal of person identification is to match the biometric data against all the records in a database. We propose the use of a statistical framework based on Gaussian mixture models and maximum a posteriori model adaptation, successfully applied to speaker and face authentication, which can deal with only one training session. We perform intensive experimental simulations using several strict train/test protocols to show the potential of our method. We also show that there are some mental tasks that are more appropriate for person authentication than others

Control strategies for active lower extremity prosthetics and orthotics: a review

Technological advancements have led to the development of numerous wearable robotic devices for the physical assistance and restoration of human locomotion. While many challenges remain with respect to the mechanical design of such devices, it is at least equally challenging and important to develop strategies to control them in concert with the intentions of the user.This work reviews the state-of-the-art techniques for controlling portable active lower limb prosthetic and orthotic (P/O) devices in the context of locomotive activities of daily living (ADL), and considers how these can be interfaced with the user’s sensory-motor control system. This review underscores the practical challenges and opportunities associated with P/O control, which can be used to accelerate future developments in this field. Furthermore, this work provides a classification scheme for the comparison of the various control strategies.As a novel contribution, a general framework for the control of portable gait-assistance devices is proposed. This framework accounts for the physical and informatic interactions between the controller, the user, the environment, and the mechanical device itself. Such a treatment of P/Os – not as independent devices, but as actors within an ecosystem – is suggested to be necessary to structure the next generation of intelligent and multifunctional controllers.Each element of the proposed framework is discussed with respect to the role that it plays in the assistance of locomotion, along with how its states can be sensed as inputs to the controller. The reviewed controllers are shown to fit within different levels of a hierarchical scheme, which loosely resembles the structure and functionality of the nominal human central nervous system (CNS). Active and passive safety mechanisms are considered to be central aspects underlying all of P/O design and control, and are shown to be critical for regulatory approval of such devices for real-world use.The works discussed herein provide evidence that, while we are getting ever closer, significant challenges still exist for the development of controllers for portable powered P/O devices that can seamlessly integrate with the user’s neuromusculoskeletal system and are practical for use in locomotive ADL.

Collecting complex activity datasets in highly rich networked sensor environments

We deployed 72 sensors of 10 modalities in 15 wireless and wired networked sensor systems in the environment, in objects, and on the body to create a sensor-rich environment for the machine recognition of human activities. We acquired data from 12 subjects performing morning activities, yielding over 25 hours of sensor data. We report the number of activity occurrences observed during post-processing, and estimate that over 13000 and 14000 object and environment interactions occurred. We describe the networked sensor setup and the methodology for data acquisition, synchronization and curation. We report on the challenges and outline lessons learned and best practice for similar large scale deployments of heterogeneous networked sensor systems. We evaluate data acquisition quality for on-body and object integrated wireless sensors; there is less than 2.5% packet loss after tuning. We outline our use of the dataset to develop new sensor network self-organization principles and machine learning techniques for activity recognition in opportunistic sensor configurations. Eventually this dataset will be made public.

On the need for on-line learning in brain-computer interfaces

We motivate the need for on-line learning in brain-computer interfaces (BCI) and illustrate its benefits with the simplest method, namely fixed learning rates. However, the use of this method is supported by the risk of hampering the user to acquire suitable control of the BCI if the embedded classifier changes too rapidly. We report the results with 3 beginner subjects in a series of consecutive recordings, where the classifiers are iteratively trained with the data of a given session and tested on the next session. Interestingly, performance improved over sessions significantly for 2 of the subjects. These results show that on-line learning improves systematically the performance of the subjects. Moreover, performance with online learning is statistically similar to that obtained training the classifier off-line with the same amount of data.

The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition

There is a growing interest on using ambient and wearable sensors for human activity recognition, fostered by several application domains and wider availability of sensing technologies. This has triggered increasing attention on the development of robust machine learning techniques that exploits multimodal sensor setups. However, unlike other applications, there are no established benchmarking problems for this field. As a matter of fact, methods are usually tested on custom datasets acquired in very specific experimental setups. Furthermore, data is seldom shared between different groups. Our goal is to address this issue by introducing a versatile human activity dataset recorded in a sensor-rich environment. This database was the basis of an open challenge on activity recognition. We report here the outcome of this challenge, as well as baseline performance using different classification techniques. We expect this benchmarking database will motivate other researchers to replicate and outperform the presented results, thus contributing to further advances in the state-of-the-art of activity recognition methods.

Linear classification of low-resolution EEG patterns produced by imagined hand movements

Electroencephalograph (EEG)-based brain-computer interfaces (BCIs) require on-line detection of mental states from spontaneous EEG signals. In this framework, surface Laplacian (SL) transformation of EEG signals has proved to improve the recognition scores of imagined motor activity. The results we obtained in the first year of an European project named adaptive brain interfaces (ABI) suggest that: 1) the detection of mental imagined activity can be obtained by using the signal space projection (SSP) method as a classifier and 2) a particular type of electrodes can be used in such a BCI device, reconciling the benefits of SL waveforms and the need for the use of few electrodes. Recognition of mental activity was attempted on both raw and SL-transformed EEG data from five healthy people performing two mental tasks, namely imagined right and left hand movements.

Brain-actuated interaction

Over the last years evidence has accumulated that shows the possibility to analyze human brain activity on-line and translate brain states into actions such as selecting a letter from a virtual keyboard or moving a robotics device. These initial results have been obtained with either invasive approaches (requiring surgical implantation of electrodes) or synchronous protocols (where brain signals are time-locked to external cues). In this paper we describe a portable noninvasive brain-computer interface that allows the continuous control of a mobile robot in a house-like environment and also the operation of a virtual keyboard. The interface works asynchronously (the person makes self-paced decisions on when to switch from one mental task to the next) and uses 8 surface electrodes to measure electroencephalogram signals from which a statistical classifier recognizes 3 different mental states. Here we report results with five volunteers during their brain-actuated interaction experiments with the mobile robot and the virtual keyboard. Two of the participants successfully moved the robot between several rooms, while the other three participants managed to write messages with the virtual keyboard. One of the latter volunteers is a physically impaired person suffering from spinal muscular atrophy.