Key Band Image Sequences and A Hybrid Deep Neural Network for Recognition of Motor Imagery EEG

Abstract

Deep neural network is a promising method to recognize motor imagery electroencephalography (MI-EEG), which is often used as the source signal of a rehabilitation system; and the core issues are the data representation and the matched neural networks. MI-EEG images is one of the main expressions, however, all the measured data of a trial are usually integrated into one image, causing information loss, especially in the time dimension; and the neural network architecture might not fully extract the features over the <inline-formula> <tex-math notation= LaTeX >$\\alpha $ </tex-math></inline-formula> and <inline-formula> <tex-math notation= LaTeX >$\\beta $ </tex-math></inline-formula> frequency bands, which are closely related to MI. In this paper, we propose a key band imaging method (KBIM). A short time Fourier transform is applied to each electrode of the MI-EEG signal to generate a time-frequency image, and the parts corresponding to the <inline-formula> <tex-math notation= LaTeX >$\\alpha $ </tex-math></inline-formula> and <inline-formula> <tex-math notation= LaTeX >$\\beta $ </tex-math></inline-formula> bands are intercepted, fused, and further arranged into the EEG electrode map by the nearest neighbor interpolation method, forming two key band image sequences. In addition, a hybrid deep neural network named the parallel multimodule convolutional neural network and long short-term memory network (PMMCL) is designed for the extraction and fusion of the spatial-spectral and temporal features of two key band image sequences to realize automatic classification of MI-EEG signals. Extensive experiments are conducted on two public datasets, and the accuracies after 10-fold cross-validation are 97.42% and 77.33%, respectively. Statistical analysis shows the superb discrimination ability for multiclass MI-EEG too. The results demonstrate that KBIM can preserve the integrity of the feature information, and they well match with PMMCL.