Keyword-spotting and speech onset detection in EEG-based Brain Computer Interfaces

Abstract

The growing intervention of automated speech recognition applications in everyday life drives improvement in Brain-Computer Interfaces (BCI) for speech processing. By incorporating ASR (Automatic Speech Recognition)-based “keyword spotting” or “wake up commands”, we provide techniques for assessing when a BCI should start decoding, improving accuracy and efficiency for end users. Here, we use high density scalp EEG collected while participants listened to continuous speech in an audio-only, clear context, or while they watched highly noisy, naturalistic audiovisual movie clips. We designed three speech processing deep learning models: A sentence spotter (SS) model, Phoneme vs. Silence (PS) classifier and finally, Audio vs. Audio-visual (AV) stimuli induced EEG response classifier. The overall goal of this study is to design and examine the performance of these techniques for various speech processing applications. We use Long-Short Term Memory (LSTM) and Gated Recurrent Units (GRUs) neural network architectures and evaluated them on 16 participants EEG data. We show 98.15% accuracy for our AV classifier, 98.56 F1 score on SS model and 70.33 F1 score on the PS model. 11Please refer to https://github.com/Madhusakth/BCI-Sentence-Silence-Spotter for the code.