Research on Anomaly Detection System of Online Multi-node Log Flow

doi:10.3778/j.issn.1673-9418.2002015

Abstract

Abstract:

With the increasing amount of logs produced by nodes in CNGrid, traditional manual methods for abnormal log analysis can no longer meet the need of daily analysis. In order to analyze the log automatically and efficiently, a two-stage detection method is proposed in this paper. In the first stage, the log patterns are classified during preprocessing, then the principal component analysis is used for anomaly detection and the sequence of log types is defined as a log flow pattern. The abnormal flow patterns obtained from anomaly detection are extracted by the definition. Finally, the hierarchical clustering algorithm is used to simplify the results of the flow pattern and the results are saved. In the second stage, through the detection model and flow pattern obtained in the first stage, the log flow information can be monitored and analyzed in real time and the corresponding flow pattern can be matched. Finally, the experiment is carried out on real logs in CNGrid, and the results are visualized in real time. These greatly reduce the manual work of operations.

Key words: principal component analysis (PCA), log flow pattern, hierarchical clustering, visualization

摘要：

随着国家高性能计算环境各个节点产生日志数量不断增加，采用传统的人工方式进行异常日志分析已不能满足日常的分析需求。为了高效自动化地分析日志，提出了一个两阶段的检测方法。第一阶段首先在预处理时将日志模式进行分类，然后使用主成分分析方法进行异常检测，并将日志类型的有序排列定义为一种日志流量模式，通过该定义将异常检测中得到的异常流量模式提取出来，最后使用层次聚类算法简化流量模式的结果并保存。第二阶段通过第一阶段得到的检测模型和流量模式，可以实时监测分析日志流量信息并匹配对应流量模式。最终基于高性能计算环境中的真实日志进行实验分析，并实现了结果的可视化展示，从而大大减轻了运维人员的工作负担。

关键词: 主成分分析（PCA）, 日志流量模式, 层次聚类, 可视化

WANG Xiaodong, ZHAO Yining, XIAO Haili, WANG Xiaoning, CHI Xuebin. Research on Anomaly Detection System of Online Multi-node Log Flow[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(11): 1828-1837.

王晓东，赵一宁，肖海力，王小宁，迟学斌. 线上多节点日志流量异常检测系统的研究[J]. 计算机科学与探索, 2020, 14(11): 1828-1837.

References

[1] Zhao Y N, Xiao H L. The design of LARGE: log analyzing framework in grid environment[J]. E-science Technology & Application, 2016, 7(3): 3-7. 赵一宁, 肖海力. 网格环境日志分析框架LARGE的设计[J]. 科研信息化技术与应用, 2016, 7(3): 3-7.
[2] Dokas P, Ertoz L, Kumar V, et al. Data mining for network intrusion detection[C]//Proceedings of the 2002 National Science Foundation Workshop on Next Generation Data Mining, Marriott, Nov 1-3, 2002: 21-30.
[3] Vaarandi R. A data clustering algorithm for mining patterns from event logs[C]//Proceedings of the 3rd IEEE Workshop on IP Operations & Management, Kansas City, Oct 3, 2003. Piscataway: IEEE, 2003: 119-126.
[4] Vaarandi R. A breadth-first algorithm for mining frequent patterns from event logs[C]//LNCS 3283: Proceedings of the IFIP International Conference on Intelligence in Communication Systems, Bangkok, Nov 23-26, 2004. Berlin, Heidelberg: Springer, 2004: 293-308.
[5] Makanju A A O, Zincir-Heywood A N, Milios E E. Clustering event logs using iterative partitioning[C]//Proceedings of the 15th ACM SIGKDD International Conference on Know-ledge Discovery and Data Mining, Paris, Jun 28-Jul 1, 2009. New York: ACM, 2009: 1255-1264.
[6] Xu W, Huang L, Fox A, et al. Detecting large-scale system problems detection by mining console logs[C]//Proceedings of the ACM SIGOPS 22nd Symposium on Operating Sys-tems Principles, Oct 2009. New York: ACM, 2009: 117-132.
[7] Fronza I, Sillitti A, Succi G, et al. Failure prediction based on log files using random indexing and support vector machines[J]. Journal of Systems & Software, 2013, 86(1): 2-11.
[8] Weiss G M, Hirsh H. Learning to predict rare events in event sequences[C]//Proceedings of the 4th International Conference on Knowledge Discovery and Data Mining, New York, Aug 27-31, 1998. Menlo Park: AAAI, 1998: 359-363.
[9] Yamanishi K, Maruyama Y. Dynamic syslog mining for network failure monitoring[C]//Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, Aug 21-24, 2005. New York: ACM, 2005: 499-508.
[10] Yuan D, Mai H H, Xiong W W, et al. SherLog: error diagnosis by connecting clues from run-time logs[J]. ACM SIGPLAN Notices, 2010, 45(3): 143-154.
[11] Peng W, Li T, Ma S. Mining logs files for data-driven system management[J]. SIGKDD Explorations, 2005, 7(1): 44-51.
[12] Prewett J E. Analyzing cluster log files using logsurfer[C]//Proceedings of the 4th Annual Conference on Linux Clusters, San Jose, Jun 11, 2003. San Jose: Linux Clusters, 2003: 1-26.
[13] Hansen S E, Atkins E T. Automated system monitoring and notification with Swatch[C]//Proceedings of the 7th Conference on Systems Administration, Monterey, Nov 1-5, 1993. Ber-keley: USENIX Association, 1993: 145-152.
[14] Zhao Y N, Xiao H L. Extracting log patterns from system logs in LARGE[C]//Proceedings of the 2016 IEEE Inter-national Parallel and Distributed Processing Symposium Workshops, Chicago, May 23-27, 2016. Washington: IEEE Computer Society, 2016: 1645-1652.
[15] Zhao Y N, Wang X D, Xiao H L, et al. Improvement of the log pattern extracting algorithm using text similarity[C]//Proceedings of the 2018 IEEE International Parallel and Distributed Processing Symposium Workshops, Vancouver, May 21-25, 2018. Washington: IEEE Computer Society, 2018: 507-514.
[16] Rousseeuw P J. Silhouettes: a graphical aid to the interpre-tation and validation of cluster analysis[J]. Journal of Computational and Applied Mathematics, 1987, 20: 53-65.