Time Series Anomaly Detection Model with Dual Attention Mechanism

doi:10.3778/j.issn.1673-9418.2304005

Abstract

Abstract: As an important part of time series research, time series anomaly detection has attracted extensive attention and research in academia and industry. In view of the deep local features and complex dependency in time series data, an anomaly detection model with dual attention mechanism is proposed. The model adopts autoencoder structure. The encoder is composed of a squeeze excitation attention block (SEAB) and a probsparse self-attention block (PSAB). SEAB mines local features containing important information by assigning greater weights to sequence segments with strong discriminability using dynamic weighted window partitioning. PSAB adopts sparse self-attention mechanism to retain dot products with higher weights, eliminate redundant timing features, and reduce time complexity, so as to capture the long-term dependence of time series. Experimental results show that the proposed model achieves the highest F1 score of 0.97 among 9 comparison models and outperforms all other comparison models in 8 of 14 tested datasets in terms of F1 score, which can effectively identify abnormal situation in time series data and achieve advanced anomaly detection performance.

Key words: time series, anomaly detection, deep learning, attention, autoencoder

摘要： 时间序列异常检测作为时间序列研究的重要组成部分，已经引起学术界和工业界的广泛关注和研究。针对时间序列数据中蕴含的深层局部特征和复杂的前后依赖关系，提出一种融合双重注意力机制的异常检测模型。该模型采用自编码器结构，由挤压激励注意力模块（SEAB）和概率稀疏自注意力模块（PSAB）组成编码器。SEAB通过利用动态加权窗口划分，为具有强可辨识性的子序列片段赋予更大的权重，使模型能够更加有效地挖掘出具有重要信息的局部特征。PSAB则采用稀疏自注意力机制，保留具有较高权重的点积，去除冗余的时序特征，降低了时间复杂度，从而捕获时间序列的长期依赖关系。实验结果表明，该模型在9个对比模型中取得了最高的[F1]分数0.97，并在14个测试数据集中有8个[F1]分数超过其他所有对比模型，因此可有效地识别时间序列数据中的异常情况，并具备先进的异常检测性能。

关键词: 时间序列, 异常检测, 深度学习, 注意力, 自编码器

YANG Chaocheng, YAN Xuanhui, CHEN Rongjun, LI Hanzhang. Time Series Anomaly Detection Model with Dual Attention Mechanism[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(3): 740-754.

杨超城, 严宣辉, 陈容均, 李汉章. 融合双重注意力机制的时间序列异常检测模型[J]. 计算机科学与探索, 2024, 18(3): 740-754.

References

[1] CHOI Y, LIM H, CHOI H, et al. GAN-based anomaly detection and localization of multivariate time series data for power plant[C]//Proceedings of the 2020 IEEE International Conference on Big Data and Smart Computing, Busan, Feb 19-22, 2020. Piscataway: IEEE, 2020: 71-74.
[2] 段雪源, 付钰, 王坤. 基于VAE-WGAN的多维时间序列异常检测方法[J]. 通信学报, 2022, 43(3): 1-13.
DUAN X Y, FU Y, WANG K. Multi-dimensional time series anomaly detection method based on VAE-WGAN[J]. Journal on Communications, 2022, 43(3): 1-13.
[3] CHENG W, MA T, WANG X, et al. Anomaly detection for Internet of things time series data using generative adversarial networks with attention mechanism in smart agriculture[J]. Frontiers in Plant Science, 2022, 13: 1-18.
[4] XU H, CHEN W, ZHAO N, et al. Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in Web applications[C]//Proceedings of the 2018 World Wide Web Conference on World Wide Web, Lyon, Apr 23-27, 2018. New York: ACM, 2018: 187-196.
[5] YANG K, WANG Y, HAN X, et al. Unsupervised anomaly detection for time series data of spacecraft using multi-task learning[J]. Applied Sciences, 2022, 12(13): 6296-6313.
[6] PARK D, HOSHI Y, KEMP C C. A multimodal anomaly detector for robot-assisted feeding using an LSTM-based variational autoencoder[J]. IEEE Robotics and Automation Letters, 2018, 3(3): 1544-1551.
[7] XIA X, PAN X, LI N, et al. GAN-based anomaly detection: a review[J]. Neurocomputing, 2022, 493: 497-535.
[8] BRAUCKHOFF D, SALAMATIAN K, MAY M. Applying PCA for traffic anomaly detection: problems and solutions[C]//Proceedings of the 28th IEEE International Conference on Computer Communications, Joint Conference of the IEEE Computer and Communications Societies, Rio de Janeiro, Apr 19-25, 2009. Piscataway: IEEE, 2009: 2866-2870.
[9] JIANLIANG M, HAIKUN S, LING B. The application on intrusion detection based on K-means cluster algorithm Meng[C]//Proceedings of the 2009 International Forum on Information Technology and Applications, Chengdu, May 15-17, 2009. Piscataway: IEEE, 2009: 150-152.
[10] LIU F T, TING K M, ZHOU Z H. Isolation forest[C]//Proceedings of the 8th IEEE International Conference on Data Mining, Pisa, Dec 15-19, 2008. Washington: IEEE Computer Society, 2008: 413-422.
[11] HUANG T, ZHU Y, ZHANG Q, et al. An LOF-based adaptive anomaly detection scheme for cloud computing[C]//Proceedings of the IEEE 37th Annual Computer Software and Applications Conference, Kyoto, Jul 22-26, 2013. Washington: IEEE Computer Society, 2013: 206-211.
[12] FAROOQI A H, MUNIR A. Intrusion detection system for IP multimedia subsystem using K-nearest neighbor classifier[C]//Proceedings of the 2008 IEEE International Multitopic Conference. Piscataway: IEEE, 2008: 423-428.
[13] HABEEBA R A A, NASARUDDINA F, GANIB A, et al. Real-time big data processing for anomaly detection: a survey[J]. International Journal of Information Management, 2019, 45: 289-307.
[14] HUNDMAN K, CONSTANTINOU V, LAPORTE C, et al. Detecting spacecraft anomalies using LSTMs and nonparametric dynamic thresholding[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, London, Aug 19-23, 2018. New York: ACM, 2018: 387-395.
[15] BASHAR M A, NAYAK R. TAnoGAN: time series anomaly detection with generative adversarial networks[C]//Proceedings of the 2020 IEEE Symposium Series on Computational Intelligence, Canberra, Dec 1-4, 2020. Piscataway: IEEE, 2020: 1778-1785.
[16] GEIGER A, LIU D, ALNEGHEIMISH S, et al. TadGAN: time series anomaly detection using generative adversarial networks[C]//Proceedings of the 2020 IEEE International Conference on Big Data , Atlanta, Dec 10-13, 2020. Piscataway: IEEE, 2020: 33-43.
[17] LI D, CHEN D, JIN B, et al. MAD-GAN: multivariate anomaly detection for time series data with generative adversarial networks[C]//Proceedings of the 28th International Conference on Artificial Neural Networks, Munich, Sep 17-19, 2019: 703-716.
[18] WANG X, PI D, ZHANG X, et al. Variational transformer-based anomaly detection approach for multivariate time series[J]. Measurement, 2022, 191: 110791-110808.
[19] HOCHREITER S, SCHMIDHUBER J. Long short-term memory[J]. Neural Computation, 1997, 9(8): 1735-1780.
[20] CHUNG J, GULCEHRE C, CHO K, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[J]. arXiv:1412.3555, 2014.
[21] ZHANG Y, ZHU Y, LI X, et al. Anomaly detection based on mining six local data features and BP neural network[J]. Symmetry, 2019, 11(4): 571-591.
[22] ZHAO P, CHANG X, WANG M. A novel multivariate time-series anomaly detection approach using an unsupervised deep neural network[J]. IEEE Access, 2021, 9: 109025-109041.
[23] YU Q, JIBIN L, JIANG L. An improved ARIMA-based traffic anomaly detection algorithm for wireless sensor networks[J]. International Journal of Distributed Sensor Networks, 2016, 12: 9653230-9653239.
[24] CHOI K, YI J, PARK C, et al. Deep learning for anomaly detection in time-series data: review, analysis, and guidelines[J]. IEEE Access, 2021, 9: 120043-120065.
[25] DAI X, GAO Z. From model, signal to knowledge: a data-driven perspective of fault detection and diagnosis[J]. IEEE Transactions on Industrial Informatics, 2013, 9(4): 2226-2238.
[26] XIE Y, ZHANG Y. An intelligent anomaly analysis for intrusion detection based on SVM[C]//Proceedings of the 2012 International Conference on Computer Science and Information Processing, Xi’an, Aug 24-26, 2012. Piscataway: IEEE, 2012: 739-742.
[27] CHEN N, TU H, DUAN X, et al. Semisupervised anomaly detection of multivariate time series based on a variational autoencoder[J]. Applied Intelligence, 2023, 53(5): 6074-6098.
[28] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems 30, Long Beach, Dec 4-9, 2017: 5998-6008.
[29] CHEN Z, JIAZE E, ZHANG X, et al. Multi-task time series forecasting with shared attention[C]//Proceedings of the 20th International Conference on Data Mining Workshops, Sorrento, Nov 17-20, 2020. Piscataway: IEEE, 2020: 917-925.
[30] YAN J, LIU J, WANG L, et al. Land-cover classification with time-series remote sensing images by complete extraction of multiscale timing dependence[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022, 15: 1953-1967.
[31] FAN J, HUANG Y, ZHANG K, et al. DWNet: dual-window deep neural network for time series prediction[J]. Complexity, 2021: 1125630.
[32] YANG D, WANG J, YAN X, et al. Subway air quality modeling using improved deep learning framework[J]. Process Safety and Environmental Protection, 2022, 163: 487-497.
[33] HU J, SHEN L, ALBANIE S, et al. Squeeze-and-excitation networks[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 7132-7141.
[34] CHEN R, YAN X, WANG S, et al. DA-Net: dual-attention network for multivariate time series classification[J]. Information Sciences, 2022, 610: 472-487.
[35] LIU Z, LIN Y, CAO Y, et al. Swin Transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021: 10002-10022.
[36] ZHOU H, ZHANG S, PENG J, et al. Informer: beyond efficient transformer for long sequence time-series forecasting[C]//Proceedings of the 35th AAAI Conference on Artificial Intelligence, the 33rd Conference on Innovative Applications of Artificial Intelligence, the 11th Symposium on Educational Advances in Artificial Intelligence, Feb 2-9, 2021. Menlo Park: AAAI, 2021: 11106-11115.
[37] RAKTHANMANON T, CAMPANA B, MUEEN A, et al. Searching and mining trillions of time series subsequences under dynamic time warping[C]//Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM, 2012: 262-270.
[38] MORGACHEV G, GONCHAROV A, STRIJOV V. Distance function selection for multivariate time-series[C]//Proceedings of the 2019 International Conference on Artificial Intelligence: Applications and Innovations, Belgrade, Sep 30-Oct 4, 2019. Piscataway: IEEE, 2019: 66-70.
[39] ZHU G, ZHAO H, LIU H, et al. A novel LSTM-GAN algorithm for time series anomaly detection[C]//Proceedings of the 2019 Prognostics and System Health Management Conference, Qingdao, Oct 25-27, 2019. Piscataway: IEEE, 2019: 1-6.
[40] LAVIN A, AHMAD S. Evaluating real-time anomaly detection algorithms—the numenta anomaly benchmark[C]//Proceedings of the 14th IEEE International Conference on Machine Learning and Applications, Miami, Dec 9-11, 2015. Piscataway: IEEE, 2015: 38-44.
[41] GARG A, ZHANG W, SAMARAN J, et al. An evaluation of anomaly detection and diagnosis in multivariate time series[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(6): 2508-2517.
[42] HUET A, NAVARRO J M, ROSSI D. Local evaluation of time series anomaly detection algorithms[C]//Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, Aug 14-18, 2022.New York: ACM, 2022: 635-645.
[43] MA J, PERKINS S. Time-series novelty detection using one-class support vector machines[J]. The International Joint Conference on Neural Networks, 2003, 3: 1741-1745.
[44] ZONG B, SONG Q, MIN M R, et al. Deep autoencoding Gaussian mixture model for unsupervised anomaly detection[C]//Proceedings of the 6th International Conference on Learning Representations, Vancouver, Apr 30-May 3, 2018: 1-19.
[45] AUDIBERT J, MICHIARDI P, GUYARD F, et al. USAD:unsupervised anomaly detection on multivariate time series[C]//Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Aug 23-27, 2020.New York: ACM, 2020: 3395-3404.
[46] LIN S, CLARK R, BIRKE R, et al. Anomaly detection for time series using VAE-LSTM hybrid model[C]//Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, Barcelona, May 4-8, 2020. Piscataway: IEEE, 2020: 4322-4326.
[47] CARMONA C U, AUBET F X, FLUNKERT V, et al. Neural contextual anomaly detection for time series[C]//Proceedings of the 33rd International Joint Conference on Artificial Intelligence,Vienna, Jul 23-29, 2022: 2843-2851.
[48] WANG R, LIU C, MOU X, et al. Deep contrastive one-class time series anomaly detection[J]. arXiv:2207.01472, 2022.
[49] CHEN L, CHEN D, YANG F, et al. A deep multi-task representation learning method for time series classification and retrieval[J]. Information Sciences, 2021, 555: 17-32.