Jiahui Pan

A Hybrid Brain Computer Interface to Control the Direction and Speed of a Simulated or Real Wheelchair

Brain-computer interfaces (BCIs) are used to translate brain activity signals into control signals for external devices. Currently, it is difficult for BCI systems to provide the multiple independent control signals necessary for the multi-degree continuous control of a wheelchair. In this paper, we address this challenge by introducing a hybrid BCI that uses the motor imagery-based mu rhythm and the P300 potential to control a brain-actuated simulated or real wheelchair. The objective of the hybrid BCI is to provide a greater number of commands with increased accuracy to the BCI user. Our paradigm allows the user to control the direction (left or right turn) of the simulated or real wheelchair using left- or right-hand imagery. Furthermore, a hybrid manner can be used to control speed. To decelerate, the user imagines foot movement while ignoring the flashing buttons on the graphical user interface (GUI). If the user wishes to accelerate, then he/she pays attention to a specific flashing button without performing any motor imagery. Two experiments were conducted to assess the BCI control; both a simulated wheelchair in a virtual environment and a real wheelchair were tested. Subjects steered both the simulated and real wheelchairs effectively by controlling the direction and speed with our hybrid BCI system. Data analysis validated the use of our hybrid BCI system to control the direction and speed of a wheelchair.

A Hybrid BCI System Combining P300 and SSVEP and Its Application to Wheelchair Control

In this paper, a hybrid brain-computer interface (BCI) system combining P300 and steady-state visual evoked potential (SSVEP) is proposed to improve the performance of asynchronous control. The four groups of flickering buttons were set in the graphical user interface. Each group contained one large button in the center and eight small buttons around it, all of which flashed at a fixed frequency (e.g., 7.5 Hz) to evoke SSVEP. At the same time, the four large buttons of the four groups were intensified through shape and color changes in a random order to produce P300 potential. During the control state, the user focused on a desired group of buttons (target buttons) to evoke P300 potential and SSVEP, simultaneously. Discrimination between the control and idle states was based on the detection of both P300 and SSVEP on the same group of buttons. As an application, this method was used to produce a “go/stop” command in real-time wheelchair control. Several experiments were conducted, and data analysis results showed that combining P300 potential and SSVEP significantly improved the performance of the BCI system in terms of detection accuracy and response time.

Detecting awareness in patients with disorders of consciousness using a hybrid brain–computer interface

OBJECTIVE The bedside detection of potential awareness in patients with disorders of consciousness (DOC) currently relies only on behavioral observations and tests; however, the misdiagnosis rates in this patient group are historically relatively high. In this study, we proposed a visual hybrid brain-computer interface (BCI) combining P300 and steady-state evoked potential (SSVEP) responses to detect awareness in severely brain injured patients. APPROACH Four healthy subjects, seven DOC patients who were in a vegetative state (VS, n = 4) or minimally conscious state (MCS, n = 3), and one locked-in syndrome (LIS) patient attempted a command-following experiment. In each experimental trial, two photos were presented to each patient; one was the patients own photo, and the other photo was unfamiliar. The patients were instructed to focus on their own or the unfamiliar photos. The BCI system determined which photo the patient focused on with both P300 and SSVEP detections. MAIN RESULTS Four healthy subjects, one of the 4 VS, one of the 3 MCS, and the LIS patient were able to selectively attend to their own or the unfamiliar photos (classification accuracy, 66-100%). Two additional patients (one VS and one MCS) failed to attend the unfamiliar photo (50-52%) but achieved significant accuracies for their own photo (64-68%). All other patients failed to show any significant response to commands (46-55%). SIGNIFICANCE Through the hybrid BCI system, command following was detected in four healthy subjects, two of 7 DOC patients, and one LIS patient. We suggest that the hybrid BCI system could be used as a supportive bedside tool to detect awareness in patients with DOC.

Discrimination Between Control and Idle States in Asynchronous SSVEP-Based Brain Switches: A Pseudo-Key-Based Approach

A steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) can operate as an asynchronous brain switch. When SSVEP is detected with the “on/off” button flickering at a fixed frequency, the subject is identified as in the control state. Otherwise, he is in the idle state. Generally, the detection of the idle/control state is based on a predefined threshold, which is related to power. However, due to the variability of the electroencephalogram (EEG) signal, it is difficult to find an optimal threshold to achieve a high true-positive rate (TPR) in the control state while maintaining a low false-positive rate (FPR) in the idle state. In this paper, a novel pseudo-key-based approach is presented for better discriminating the control and idle states. A dedicated “on/off” button (target key) and several additional buttons (pseudo-keys) are displayed on the graphical user interface (GUI), and all of these buttons flash at different frequencies. The control state is identified from the EEG signal under two conditions. The first is a common thresholding condition, where the power ratio of the target key frequency component to a certain neighboring frequency band is above a predefined threshold. The second is a comparison condition, where the power of the target key frequency component is higher than any of the pseudo-keys. The effectiveness of the proposed approach is validated by several experiments. Further analysis shows that introducing the pseudo-keys can significantly reduce the probability that the SSVEP will be detected in response to the flickering target key in the idle state without substantially affecting the detection in the control state, providing strong evidence in support of our approach.

Multimodal BCIs: Target Detection, Multidimensional Control, and Awareness Evaluation in Patients With Disorder of Consciousness

Despite rapid advances in the study of brain-computer interfaces (BCIs) in recent decades, two fundamental challenges, namely, improvement of target detection performance and multidimensional control, continue to be major barriers for further development and applications. In this paper, we review the recent progress in multimodal BCIs (also called hybrid BCIs), which may provide potential solutions for addressing these challenges. In particular, improved target detection can be achieved by developing multimodal BCIs that utilize multiple brain patterns, multimodal signals, or multisensory stimuli. Furthermore, multidimensional object control can be accomplished by generating multiple control signals from different brain patterns or signal modalities. Here, we highlight several representative multimodal BCI systems by analyzing their paradigm designs, detection/control methods, and experimental results. To demonstrate their practicality, we report several initial clinical applications of these multimodal BCI systems, including awareness evaluation/detection in patients with disorder of consciousness (DOC). As an evolving research area, the study of multimodal BCIs is increasingly requiring more synergetic efforts from multiple disciplines for the exploration of the underlying brain mechanisms, the design of new effective paradigms and means of neurofeedback, and the expansion of the clinical applications of these systems.

Detecting number processing and mental calculation in patients with disorders of consciousness using a hybrid brain-computer interface system

BackgroundFor patients with disorders of consciousness such as coma, a vegetative state or a minimally conscious state, one challenge is to detect and assess the residual cognitive functions in their brains. Number processing and mental calculation are important brain functions but are difficult to detect in patients with disorders of consciousness using motor response-based clinical assessment scales such as the Coma Recovery Scale-Revised due to the patients’ motor impairments and inability to provide sufficient motor responses for number- and calculation-based communication.MethodsIn this study, we presented a hybrid brain-computer interface that combines P300 and steady state visual evoked potentials to detect number processing and mental calculation in Han Chinese patients with disorders of consciousness. Eleven patients with disorders of consciousness who were in a vegetative state (n = 6) or in a minimally conscious state (n = 3) or who emerged from a minimally conscious state (n = 2) participated in the brain-computer interface-based experiment. During the experiment, the patients with disorders of consciousness were instructed to perform three tasks, i.e., number recognition, number comparison, and mental calculation, including addition and subtraction. In each experimental trial, an arithmetic problem was first presented. Next, two number buttons, only one of which was the correct answer to the problem, flickered at different frequencies to evoke steady state visual evoked potentials, while the frames of the two buttons flashed in a random order to evoke P300 potentials. The patients needed to focus on the target number button (the correct answer). Finally, the brain-computer interface system detected P300 and steady state visual evoked potentials to determine the button to which the patients attended, further presenting the results as feedback.ResultsTwo of the six patients who were in a vegetative state, one of the three patients who were in a minimally conscious state, and the two patients that emerged from a minimally conscious state achieved accuracies significantly greater than the chance level. Furthermore, P300 potentials and steady state visual evoked potentials were observed in the electroencephalography signals from the five patients.ConclusionsNumber processing and arithmetic abilities as well as command following were demonstrated in the five patients. Furthermore, our results suggested that through brain-computer interface systems, many cognitive experiments may be conducted in patients with disorders of consciousness, although they cannot provide sufficient behavioral responses.

A Novel Audiovisual Brain-Computer Interface and Its Application in Awareness Detection

Currently, detecting awareness in patients with disorders of consciousness (DOC) is a challenging task, which is commonly addressed through behavioral observation scales such as the JFK Coma Recovery Scale-Revised. Brain-computer interfaces (BCIs) provide an alternative approach to detect awareness in patients with DOC. However, these patients have a much lower capability of using BCIs compared to healthy individuals. This study proposed a novel BCI using temporally, spatially, and semantically congruent audiovisual stimuli involving numbers (i.e., visual and spoken numbers). Subjects were instructed to selectively attend to the target stimuli cued by instruction. Ten healthy subjects first participated in the experiment to evaluate the system. The results indicated that the audiovisual BCI system outperformed auditory-only and visual-only systems. Through event-related potential analysis, we observed audiovisual integration effects for target stimuli, which enhanced the discriminability between brain responses for target and nontarget stimuli and thus improved the performance of the audiovisual BCI. This system was then applied to detect the awareness of seven DOC patients, five of whom exhibited command following as well as number recognition. Thus, this audiovisual BCI system may be used as a supportive bedside tool for awareness detection in patients with DOC.

Visual Fixation Assessment in Patients with Disorders of Consciousness Based on Brain-Computer Interface

Visual fixation is an item in the visual function subscale of the Coma Recovery Scale-Revised (CRS-R). Sometimes clinicians using the behavioral scales find it difficult to detect because of the motor impairment in patients with disorders of consciousness (DOCs). Brain-computer interface (BCI) can be used to improve clinical assessment because it directly detects the brain response to an external stimulus in the absence of behavioral expression. In this study, we designed a BCI system to assist the visual fixation assessment of DOC patients. The results from 15 patients indicated that three showed visual fixation in both CRS-R and BCI assessments and one did not show such behavior in the CRS-R assessment but achieved significant online accuracy in the BCI assessment. The results revealed that electroencephalography-based BCI can detect the brain response for visual fixation. Therefore, the proposed BCI may provide a promising method for assisting behavioral assessment using the CRS-R.

Emotion-Related Consciousness Detection in Patients With Disorders of Consciousness Through an EEG-Based BCI System

For patients with disorders of consciousness (DOC), such as vegetative state (VS) and minimally conscious state (MCS), detecting and assessing the residual cognitive functions of the brain remain challenging. Emotion-related cognitive functions are difficult to detect in patients with DOC using motor response-based clinical assessment scales such as the Coma Recovery Scale-Revised (CRS-R) because DOC patients have motor impairments and are unable to provide sufficient motor responses for emotion-related communication. In this study, we proposed an EEG-based brain-computer interface (BCI) system for emotion recognition in patients with DOC. Eight patients with DOC (5 VS and 3 MCS) and eight healthy controls participated in the BCI-based experiment. During the experiment, two movie clips flashed (appearing and disappearing) eight times with a random interstimulus interval between flashes to evoke P300 potentials. The subjects were instructed to focus on the crying or laughing movie clip and to count the flashes of the corresponding movie clip cued by instruction. The BCI system performed online P300 detection to determine which movie clip the patients responsed to and presented the result as feedback. Three of the eight patients and all eight healthy controls achieved online accuracies based on P300 detection that were significantly greater than chance level. P300 potentials were observed in the EEG signals from the three patients. These results indicated the three patients had abilities of emotion recognition and command following. Through spectral analysis, common spatial pattern (CSP) and differential entropy (DE) features in the delta, theta, alpha, beta, and gamma frequency bands were employed to classify the EEG signals during the crying and laughing movie clips. Two patients and all eight healthy controls achieved offline accuracies significantly greater than chance levels in the spectral analysis. Furthermore, stable topographic distribution patterns of CSP and DE features were observed in both the healthy subjects and these two patients. Our results suggest that cognitive experiments may be conducted using BCI systems in patients with DOC despite the inability of such patients to provide sufficient behavioral responses.

A gaze-independent audiovisual brain-computer Interface for detecting awareness of patients with disorders of consciousness

BackgroundCurrently, it is challenging to detect the awareness of patients who suffer disorders of consciousness (DOC). Brain-computer interfaces (BCIs), which do not depend on the behavioral response of patients, may serve for detecting the awareness in patients with DOC. However, we must develop effective BCIs for these patients because their ability to use BCIs does not as good as healthy users.MethodsBecause patients with DOC generally do not exhibit eye movements, a gaze-independent audiovisual BCI is put forward in the study where semantically congruent and incongruent audiovisual number stimuli were sequentially presented to evoke event-related potentials (ERPs). Subjects were required to pay attention to congruent audiovisual stimuli (target) and ignore the incongruent audiovisual stimuli (non-target). The BCI system was evaluated by analyzing online and offline data from 10 healthy subjects followed by being applied to online awareness detection in 8 patients with DOC.ResultsAccording to the results on healthy subjects, the audiovisual BCI system outperformed the corresponding auditory-only and visual-only systems. Multiple ERP components, including the P300, N400 and late positive complex (LPC), were observed using the audiovisual system, strengthening different brain responses to target stimuli and non-target stimuli. The results revealed the abilities of three of eight patients to follow commands and recognize numbers.ConclusionsThis gaze-independent audiovisual BCI system represents a useful auxiliary bedside tool to detect the awareness of patients with DOC.