Journal of Frontiers of Computer Science and Technology ›› 2020, Vol. 14 ›› Issue (11): 1828-1837.DOI: 10.3778/j.issn.1673-9418.2002015

Previous Articles     Next Articles

Research on Anomaly Detection System of Online Multi-node Log Flow

WANG Xiaodong, ZHAO Yining, XIAO Haili, WANG Xiaoning, CHI Xuebin   

  1. 1. Computer Network Information Center, Chinese Academy of Sciences, Beijing 100190, China
    2. University of Chinese Academy of Sciences, Beijing 100049, China
  • Online:2020-11-01 Published:2020-11-09

线上多节点日志流量异常检测系统的研究

王晓东赵一宁肖海力王小宁迟学斌   

  1. 1. 中国科学院 计算机网络信息中心,北京 100190
    2. 中国科学院大学,北京 100049

Abstract:

With the increasing amount of logs produced by nodes in CNGrid, traditional manual methods for abnormal log analysis can no longer meet the need of daily analysis. In order to analyze the log automatically and efficiently, a two-stage detection method is proposed in this paper. In the first stage, the log patterns are classified during preprocessing, then the principal component analysis is used for anomaly detection and the sequence of log types is defined as a log flow pattern. The abnormal flow patterns obtained from anomaly detection are extracted by the definition. Finally, the hierarchical clustering algorithm is used to simplify the results of the flow pattern and the results are saved. In the second stage, through the detection model and flow pattern obtained in the first stage, the log flow information can be monitored and analyzed in real time and the corresponding flow pattern can be matched. Finally, the experiment is carried out on real logs in CNGrid, and the results are visualized in real time. These greatly reduce the manual work of operations.

Key words: principal component analysis (PCA), log flow pattern, hierarchical clustering, visualization

摘要:

随着国家高性能计算环境各个节点产生日志数量不断增加,采用传统的人工方式进行异常日志分析已不能满足日常的分析需求。为了高效自动化地分析日志,提出了一个两阶段的检测方法。第一阶段首先在预处理时将日志模式进行分类,然后使用主成分分析方法进行异常检测,并将日志类型的有序排列定义为一种日志流量模式,通过该定义将异常检测中得到的异常流量模式提取出来,最后使用层次聚类算法简化流量模式的结果并保存。第二阶段通过第一阶段得到的检测模型和流量模式,可以实时监测分析日志流量信息并匹配对应流量模式。最终基于高性能计算环境中的真实日志进行实验分析,并实现了结果的可视化展示,从而大大减轻了运维人员的工作负担。

关键词: 主成分分析(PCA), 日志流量模式, 层次聚类, 可视化