Motor Imagery Signal Classification Based on Multi-scale Self-attentional Mechanism

doi:10.3778/j.issn.1673-9418.2108004

Abstract

Abstract: Brain-computer interface (BCI) technology based on motor imagery electroencephalograph (EEG) signals has developed rapidly, and compared with traditional methods, deep learning has achieved competitive results. However, it is still a challenge to design and train an end-to-end network to fully extract the potential features of motor-imaging EEG signals. Based on the temporal and spatial characteristics of EEG, a multi-scale spatiotemporal self-attention network model based on attention mechanism is proposed to classify motor imagery EEG signals into four categories, such as left hand, right hand, foot and tongue/rest. Since the amplitude and response time of motor imagery EEG signal vary with the subjects, it is impossible to determine the most relevant brain region with motor imagery. Therefore, the self-attention mechanism is used to select the best channel by automatically weighting the higher weight to the motion-related channel and the lower weight to the non-motion-related channel in space. And parallel multi-scale TCN layer is used to extract the temporal feature at different scales and to eliminate the noise in time domain. The fusion module fuses the extracted spatial and temporal features, and finally inputs them to the classification module for classification. The model achieves accuracy of 79.26%, 85.90% and 96.96% on the BCI competition datasets IV-2a, IV-2b and HGD dataset. Compared with the baseline method, this method has a higher accuracy in subject classification. The results show that the method has good performance, robustness and transfer learning ability.

Key words: motor imagery, attention, brain-computer interface, deep learning, multi-scale

摘要： 基于运动想象脑电信号的脑机接口（BCI）技术近年来发展迅速，并且与传统方法相比，深度学习取得了具有竞争力的效果。然而如何设计和训练一个端到端网络来充分提取运动想象脑电信号的潜在特征仍然是一个挑战。从脑电的时间和空间特征出发，提出了一个基于注意力机制的多尺度时空自注意力网络模型用于运动想象脑电信号四分类（左手、右手、脚、舌头/休息）。由于运动想象脑电信号的幅值与响应时间因人而异，无法确切确定与运动想象最相关的大脑区域，在空间上使用自注意力机制自动将较高的权重加权到与运动相关的通道，将较低的权重加权到与运动无关的通道来选择最佳通道；在时间上，使用并行多尺度TCN层提取不同尺度下的时间域特征信息，消除时间域上的噪声。多尺度融合模块融合提取的空间和时间域特征，最后输入到特征分类模块进行分类。提出的模型在BCI竞赛数据集IV-2a、IV-2b数据集和HGD数据集上分别达到79.26%、85.90%和96.96%的精度。与现有方法相比，该方法在单被试分类中具有更高的准确率。结果表明，该方法具有较好的性能、鲁棒性和迁移学习能力。

关键词: 运动想象, 注意力, 脑机接口, 深度学习, 多尺度

LIU Jing, ZHAO Wei, DONG Zehao, WANG Shaohua, WANG Yu. Motor Imagery Signal Classification Based on Multi-scale Self-attentional Mechanism[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(6): 1427-1440.

刘京, 赵薇, 董泽浩, 王少华, 王余. 融合多尺度自注意力机制的运动想象信号解析[J]. 计算机科学与探索, 2023, 17(6): 1427-1440.

References

[1] LOTTE F, BOUGRAIN L, CICHOCKI A, et al. A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update[J]. Journal of Neural Enginee-ring, 2018, 15: 031005.
[2] GALáN F, NUTTIN M, LEW E, et al. A brain-actuated wheel-chair: asynchronous and non-invasive brain-computer inter-faces for continuous control of robots[J]. Clinical Neuro-physiology, 2008, 119(9): 2159-2169.
[3] GUAN C. High performance P300 speller for brain-computer interface[C]//Proceedings of the 2004 IEEE International Work-shop on Biomedical Circuits and Systems, Singapore, Dec 1-3, 2004. Piscataway: IEEE, 2004.
[4] SCHWARTZ A B, CUI X T, WEBER D J, et al. Brain-con-trolled interfaces: movement restoration with neural prost-hetics[J]. Neuron, 2006, 52(1): 205-220.
[5] 吕林洋. 基于运动想象的脑电信号分类及实时控制研究[D]. 保定: 河北大学, 2021.
LV L Y. Research on EEG classification and real-time control based on motor imagery[D]. Baoding: Hebei University, 2021.
[6] TAYEB Z, FEDJAEV J, GHABOOSI N, et al. Validating deep neural networks for online decoding of motor imagery move-ments from EEG signals[J]. Sensors, 2019, 19(1): 210.
[7] GONG A, LIU J, CHEN S, et al. Time-frequency cross mu-tual information analysis of the brain functional networks underlying multiclass motor imagery[J]. Journal of Motor Behavior, 2018, 50: 254-267.
[8] 沈永龙. 基于运动想象的脑机接口关键技术研究[D]. 保定：河北大学, 2021.
SHEN Y L. Research on key technologies of brain com-puter interface based on motor imagery[D]. Baoding: Hebei University, 2021.
[9] RAMOSER H, MULLER-GERKING J, PFURTSCHELLER G. Optimal spatial filtering of single trial EEG during ima-gined hand movement[J]. IEEE Transactions on Rehabili-tation Engineering, 2000, 8: 441-446.
[10] ANG K K, CHIN Z Y, ZHANG H, et al. Filter bank com-mon spatial pattern (FBCSP) in brain-computer interface[C]//Proceedings of the 2008 IEEE International Joint Con-ference on Neural Networks, Hong Kong, China, Jun 1-8, 2008. Piscataway: IEEE, 2008: 2390-2397.
[11] JIN J, MIAO Y, DALY I, et al. Correlation-based channel selection and regularized feature optimization for MI-based BCI[J]. Neural Networks, 2019, 118: 262-270.
[12] KUMAR S, SHARMA A, MAMUN K, et al. A deep lear-ning approach for motor imagery EEG signal classification[C]//Proceedings of the 2016 3rd Asia-Pacific World Con-gress on Computer Science and Engineering, Nadi, Dec 5-6, 2016. Piscataway: IEEE, 2016: 34-39.
[13] DE J, ZHANG X, LIN F, et al. Transduction on directed graphs via absorbing random walks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(7): 1770-1784.
[14] MA J, LIN F, WESARG S, et al. A novel Bayesian model incorporating deep neural network and statistical shape model for pancreas segmentation[C]//LNCS 11073: Proceedings of the 21st International Conference on Medical Image Com-puting and Computer Assisted Intervention, Granada, Sep 16-20, 2018. Cham: Springer, 2018: 480-487.
[15] CRAIK A, HE Y, CONTRERAS-VIDAL J L. Deep learning for electroencephalogram (EEG) classification tasks: a review[J]. Journal of Neural Engineering, 2019, 16(3): 031001.
[16] ANTONIADES A, SPYROU L, TOOK C C, et al. Deep lear-ning for epileptic intracranial EEG data[C]//Proceedings of the 2016 IEEE 26th International Workshop on Machine Lear-ning for Signal Processing, Vietrisul Mare, Sep 13-16, 2016. Piscataway: IEEE, 2016: 1-6.
[17] THODOROFF P, PINEAU J, LIM A. Learning robust fea-tures using deep learning for automatic seizure detection[C]//Proceedings of the 1st Machine Learning in Health Care, Los Angeles, Aug 19-20, 2016: 178-190.
[18] BASHIVAN P, RISH I, YEASIN M, et al. Learning repre-sentations from EEG with deep recurrent-convolutional neu-ral networks[J]. arXiv:1511.06448, 2015.
[19] VERNON L, AMELIA S, NICHOLAS R W M, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces[J]. Journal of Neural Engi-neering, 2016, 15(5).
[20] STOBER S, CAMERON D J, GRAHN J A. Using convo-lutional neural networks to recognize rhythm stimuli from electroencephalography recordings[C]//Advances in Neural Information Processing Systems 27, Montreal, Dec 8-13, 2014: 1449-1457.
[21] TABAR Y R, HALICI U. A novel deep learning approach for classification of EEG motor imagery signals[J]. Journal of Neural Engineering, 2017, 14(1): 016003.
[22] SAKHAVI S, GUAN C, YAN S. Learning temporal infor-mation for brain computer interface using convolutional neu-ral networks[J]. IEEE Transactions on Neural Networks & Learning Systems, 2018, 29: 5619-5629.
[23] ZHU X, LI P, LI C, et al. Separated channel convolutional neural network to realize the training free motor imagery BCI systems[J]. Biomedical Signal Processing and Control, 2019, 49: 396-403.
[24] SCHIRRMEISTER R T, SPRINGENBERG J T, FIEDERER L D J, et al. Deep learning with convolutional neural net-works for EEG decoding and visualization[J]. Human Brain Mapping, 2017, 38: 5391-5420.
[25] AZAB A M, MIHAYLOVA L, ANG K K, et al. Weighted transfer learning for improving motor imagery-based brain-computer interface[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 27: 1352-1359.
[26] SONG Y, WANG D, YUE K, et al. EEG-based motor ima-gery classification with deep multi-task learning[C]//Procee-dings of the 2019 International Joint Conference on Neural Networks, Budapest, Jul 14-19, 2019. Piscataway: IEEE, 2019: 1-8.
[27] LI Y, ZHANG X R, ZHANG B, et al. A channel-projection mixed-scale convolutional neural network for motor ima-gery EEG decoding[J]. IEEE Transactions on Neural Sys-tems and Rehabilitation Engineering, 2019, 27(6): 1170-1180.
[28] AMIN S U, ALSULAIMAN M, MUHAMMAD G, et al. Deep learning for EEG motor imagery classification based on multi-layer CNN feature fusion[J]. Future Generation Com-puter Systems, 2019, 101: 542-554.
[29] WU H, NIU Y, LI F, et al. A parallel multiscale filter bank convolutional neural networks for motor imagery EEG classi-fication[J]. Frontiers in Neuroscience, 2019, 13: 1275.
[30] INGOLFSSON T M, HERSCHE M, WANG X, et al. EEG-TCNet: an accurate temporal convolutional network for em-bedded motor-imagery brain-machine interfaces[C]//Procee-dings of the 2020 IEEE International Conference on Sys-tems, Man, and Cybernetics, Toronto, Oct 11-14, 2020. Pis-cataway: IEEE, 2020: 2958-2965.
[31] ZHANG H, GOODFELLOW I, METAXAS D, et al. Self-attention generative adversarial networks[C]//Proceedings of the 2019 International Conference on Machine Learning, Long Beach, Jun 9-15, 2019: 7354-7363.
[32] BAI S, KOLTER J Z, KOLTUN V. An empirical evaluation of generic convolutional and recurrent networks for seque-nce modeling[J]. arXiv:1803.01271, 2018.
[33] SHELHAMER E, LONG J, DARRELL T. Fully convolu-tional networks for semantic segmentation[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pat-tern Recognition, Boston, Jun 8-10, 2015. Piscataway: IEEE, 2015: 3431-3440.
[34] OORD A V D, DIELEMAN S, ZEN H, et al. WaveNet: a generative model for raw audio[J]. arXiv:1609.03499, 2016.
[35] ZHU D, YAO H, JIANG B, et al. Negative log likelihood ratio loss for deep neural network classification[J]. arXiv:1804.10690, 2018.
[36] KINGMA D P, BA J. Adam: a method for stochastic opti-mization[J]. arXiv:1412.6980, 2014.
[37] ZHAO X, ZHANG H, ZHU G, et al. A multi-branch 3D convolutional neural network for EEG-based motor imagery classification[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 27(10): 2164-2177.
[38] LIU X, SHEN Y, LIU J, et al. Parallel spatial-temporal self-attention CNN-based motor imagery classification for BCI[J]. Frontiers in Neuroscience, 2020, 14.
[39] SHAHTALEBI S, MOHAMMADI A. Bayesian optimized spectral filters coupled with ternary ECOC for single-trial EEG classification[J]. IEEE Transactions on Neural Systems, 2018, 26(12): 2249-2259.
[40] WANG J, FENG Z, REN X, et al. Feature subset and time segment selection for the classification of EEG data based motor imagery[J]. Biomedical Signal Processing and Control, 2020, 61: 102026.
[41] TANG X, LI W, LI X, et al. Motor imagery EEG recogni-tion based on conditional optimization empirical mode decom-position and multi-scale convolutional neural network[J]. Expert Systems with Applications, 2020, 149: 113285.
[42] LU N, LI T, REN X, et al. A deep learning scheme for motor imagery classification based on restricted Boltzmann machi-nes[J]. IEEE Transactions on Neural Systems and Rehabili-tation Engineering, 2017, 25(6): 566-576.
[43] LUO T J, ZHOU C L, CHAO F. Exploring spatial-frequency-sequential relationships for motor imagery classification with recurrent neural network[J]. BMC Bioinformatics, 2018, 19(1): 1-18.