Mohammad Moghadamfalahi

Noninvasive Brain–Computer Interfaces for Augmentative and Alternative Communication

Brain-computer interfaces (BCIs) promise to provide a novel access channel for assistive technologies, including augmentative and alternative communication (AAC) systems, to people with severe speech and physical impairments (SSPI). Research on the subject has been accelerating significantly in the last decade and the research community took great strides toward making BCI-AAC a practical reality to individuals with SSPI. Nevertheless, the end goal has still not been reached and there is much work to be done to produce real-world-worthy systems that can be comfortably, conveniently, and reliably used by individuals with SSPI with help from their families and care givers who will need to maintain, setup, and debug the systems at home. This paper reviews reports in the BCI field that aim at AAC as the application domain with a consideration on both technical and clinical aspects.

Language-Model Assisted Brain Computer Interface for Typing: A Comparison of Matrix and Rapid Serial Visual Presentation

Noninvasive electroencephalography (EEG)-based brain-computer interfaces (BCIs) popularly utilize event-related potential (ERP) for intent detection. Specifically, for EEG-based BCI typing systems, different symbol presentation paradigms have been utilized to induce ERPs. In this manuscript, through an experimental study, we assess the speed, recorded signal quality, and system accuracy of a language-model-assisted BCI typing system using three different presentation paradigms: a 4 × 7 matrix paradigm of a 28-character alphabet with row-column presentation (RCP) and single-character presentation (SCP), and rapid serial visual presentation (RSVP) of the same. Our analyses show that signal quality and classification accuracy are comparable between the two visual stimulus presentation paradigms. In addition, we observe that while the matrix-based paradigm can be generally employed with lower inter-trial-interval (ITI) values, the best presentation paradigm and ITI value configuration is user dependent. This potentially warrants offering both presentation paradigms and variable ITI options to users of BCI typing systems.

FlashType

Brain computer interfaces (BCIs) offer individuals with disabilities an alternative channel of communication and control, hence they have been receiving increasing interest. BCIs can also be useful for healthy individuals in situations limiting their movement or where other computer interaction modalities need to be supplemented. Event-related and steady state visually evoked potentials (SSVEPs) are the top two brain signal types used in developing BCIs that allow the user to make a choice from a discrete set of options, including the selection of commands from a menu for a robot or computer to perform, as well as typing letters, symbols, or icons for communication. Popular BCI speller paradigms, such as the P300 Matrix Speller, RSVP Keyboard TM or SSVEP spellers in which the letters on the keyboard display flicker, are sensitive to the font, size and presentation speed. In addition, sensitivity to eye gaze control plays a significant role in usability of most of these keyboards. We present a code-VEP based BCI, utilized in a language model assisted keyboard application. Utilizing a cursor based selection method, stimuli and targets are separated. FlashTypeTM separates visual stimulation from alphabet presentation to achieve performance invariance under presentation variations. Therefore, FlashTypeTM can be used for all languages, including the ones containing symbols and icons. FlashTypeTM, contains a Static Keyboard, a row of Suggested Characters and a row of Predicted Words. FlashTypeTM, by default, uses only one EEG electrode and four stimuli. The system can operate using only one stimulus at a lower selection rate, useful for individuals with limited or no gaze control. This feature is to be explored in future. Replacing letters with text or icons representing commands would allow controlling a computer or robot. In this study, FlashTypeTM has been evaluated by three individuals performing 10 Mastery tasks. In depth experimentation, such as assessing the system with potential end users writing long passages of text, will be done in future.

^{\text{TM}}

Auditory-evoked noninvasive electroencephalography (EEG) based brain-computer interfaces (BCIs) could be useful for improved hearing aids in the future. This manuscript investigates the role of frequency and spatial features of audio signal in EEG activities in an auditory BCI system with the purpose of detecting the attended auditory source in a cocktail party setting. A cross correlation based feature between EEG and speech envelope is shown to be useful to discriminate attention in the case of two different speakers. Results indicate that, on average, for speaker and direction (of arrival) of audio signals classification, the presented approach yields 91% and 86% accuracy, respectively.

: A Context-Aware c-VEP-Based BCI Typing Interface Using EEG Signals

One of the principal application areas for brain-computer interface (BCI) technology is augmentative and alternative communication (AAC), typically used by people with severe speech and physical disabilities (SSPI). Existing word- and phrase-based AAC solutions that employ BCIs that utilize electroencephalography (EEG) are sometimes supplemented by icons. Icon-based BCI systems that use binary signaling methods, such as P300 detection, combine hierarchical layouts with some form of scanning. The rapid serial visual presentation (RSVP) IconMessenger combines P300 signal detection with the icon-based semantic message construction system of iconCHAT. Language models are incorporated in the inference engine and some modifications that facilitate the use of RSVP were performed such as icon semantic role order selection and the tight fusion of language evidence and EEG evidence. The results of a study conducted with 10 healthy participants suggest that the system has potential as an AAC system in real-time typi...

Toward a brain interface for tracking attended auditory sources

Noninvasive EEG (electroencephalography) based auditory attention detection could be useful for improved hearing aids in the future. This work is a novel attempt to investigate the feasibility of online modulation of sound sources by probabilistic detection of auditory attention, using a noninvasive EEG-based brain computer interface. Proposed online system modulates the upcoming sound sources through gain adaptation which employs probabilistic decisions (soft decisions) from a classifier trained on offline calibration data. In this work, calibration EEG data were collected in sessions where the participants listened to two sound sources (one attended and one unattended). Cross-correlation coefficients between the EEG measurements and the attended and unattended sound source envelope (estimates) are used to show differences in sharpness and delays of neural responses for attended versus unattended sound source. Salient features to distinguish attended sources from the unattended ones in the correlation patterns have been identified, and later they have been used to train an auditory attention classifier. Using this classifier, we have shown high offline detection performance with single channel EEG measurements compared to the existing approaches in the literature which employ large number of channels. In addition, using the classifier trained offline in the calibration session, we have shown the performance of the online sound source modulation system. We observe that online sound source modulation system is able to keep the level of attended sound source higher than the unattended source.

RSVP IconMessenger: icon-based brain-interfaced alternative and augmentative communication

Multichannel electroencephalography (EEG) is widely used in non-invasive brain computer interfaces (BCIs) for user intent inference. EEG can be assumed to be a Gaussian process with unknown mean and autocovariance, and the estimation of parameters is required for BCI inference. However, the relatively high dimensionality of the EEG feature vectors with respect to the number of labeled observations lead to rank deficient covariance matrix estimates. In this manuscript, to overcome ill-conditioned covariance estimation, we propose a structure for the covariance matrices of the multichannel EEG signals. Specifically, we assume that these covariances can be modeled as a Kronecker product of temporal and spatial covariances. Our results over the experimental data collected from the users of a letter-by-letter typing BCI show that with less number of parameter estimations, the system can achieve higher classification accuracies compared to a method that uses full unstructured covariance estimation. Moreover, in order to illustrate that the proposed Kronecker product structure could enable shortening the BCI calibration data collection sessions, using Cramer-Rao bound analysis on simulated data, we demonstrate that a model with structured covariance matrices will achieve the same estimation error as a model with no covariance structure using fewer labeled EEG observations.

EEG-assisted modulation of sound sources in the auditory scene

Brain–Computer Interfaces (BCIs) seek to infer some task symbol, a task relevant instruction, from brain symbols, classifiable physiological states. For example, in a motor imagery robot control task a user would indicate their choice from a dictionary of task symbols (rotate arm left, grasp, etc.) by selecting from a smaller dictionary of brain symbols (imagined left or right hand movements). We examine how a BCI infers a task symbol using selections of brain symbols. We offer a recursive Bayesian decision framework which incorporates context prior distributions (e.g., language model priors in spelling applications), accounts for varying brain symbol accuracy and is robust to single brain symbol query errors. This framework is paired with Maximum Mutual Information (MMI) coding which maximizes a generalization of ITR. Both are applicable to any discrete task and brain phenomena (e.g., P300, SSVEP, MI). To demonstrate the efficacy of our approach we perform SSVEP “Shuffle” Speller experiments and compare our recursive coding scheme with traditional decision tree methods including Huffman coding. MMI coding leverages the asymmetry of the classifier’s mistakes across a particular user’s SSVEP responses; in doing so it offers a 33% increase in letter accuracy though it is 13% slower in our experiment.

Spatio-temporal EEG models for brain interfaces

A simulation framework could decrease the burden of attending long and tiring experimental sessions on the potential users of brain computer interface (BCI) systems. Specifically during the initial design of a BCI, a simulation framework that could replicate the operational performance of the system would be a useful tool for designers to make design choices. In this manuscript, we develop a Monte Carlo based probabilistic simulation framework for electroencephalography (EEG) based BCI design. We employ one event related potential (ERP) based typing and one steady state evoked potential (SSVEP) based control interface as testbeds. We compare the results of simulations with real time experiments. Even though over and under estimation of the performance is possible, the statistical results over the Monte Carlo simulations show that the developed framework generally provides a good approximation of the real time system performance.

Recursive Bayesian Coding for BCIs

RSVP Keyboard™ is a non-invasive electroencephalography (EEG) based brain computer interface (BCI) for letter by letter typing. In this system a sequence of symbols is presented on a computer screen in rapid serial visual presentation scheme to query a users intent. EEG evidence and language model are used in conjunction for joint inference of the intended symbol. Usually repetition of sequences is necessary to achieve high confidence in the intended symbol selection. This repetition usually results in degradation in the speed of typing while compensating for accuracy. In this manuscript, we develop a mathematical framework for active sequence selection that would optimize the amount of evidence obtained from user and would improve both typing speed and accuracy simultaneously. Our analysis based on Monte-Carlo simulation shows that one can effectively improve both typing speed and accuracy by optimizing the sequence of queries to be asked from the BCI user.