高性能计算环境中日志模式提炼方法的研究

doi:10.3778/j.issn.1673-9418.2103066

计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (10): 2264-2272.DOI: 10.3778/j.issn.1673-9418.2103066

高性能计算环境中日志模式提炼方法的研究

王晓东¹^,², 赵一宁¹^,⁺(), 肖海力¹, 王小宁¹, 迟学斌¹^,²

1.中国科学院计算机网络信息中心,北京 100190
2.中国科学院大学,北京 100049

收稿日期:2021-03-19 修回日期:2021-05-20 出版日期:2022-10-01 发布日期:2021-05-31
通讯作者: + E-mail: zhaoyn@sccas.cn
作者简介:王晓东（1989—）,男,河南洛阳人,博士研究生,主要研究方向为日志处理分析、数据挖掘、机器学习。
赵一宁（1983—）,男,河北河间人,助理研究员,主要研究方向为日志处理分析、数据挖掘。
肖海力（1978—）,男,湖北天门人,硕士,研究员,主要研究方向为网格计算、分布式计算。
王小宁（1981—）,女,四川资阳人,博士,副研究员,主要研究方向为网格计算、分布式计算。
迟学斌（1963—）,男,吉林梅河口人,研究员,主要研究方向为并行计算与软件、网格计算技术、高性能计算机系统维护与管理。
基金资助:
中国科学院战略性先导科技专项项目（A类）(XDA19020101)

Research on Method of Log Pattern Extracting in High-Performance Computing Environment

WANG Xiaodong¹^,², ZHAO Yining¹^,⁺(), XIAO Haili¹, WANG Xiaoning¹, CHI Xuebin¹^,²

1. Computer Network Information Center, Chinese Academy of Sciences, Beijing 100190, China
2. University of Chinese Academy of Sciences, Beijing 100049, China

Received:2021-03-19 Revised:2021-05-20 Online:2022-10-01 Published:2021-05-31
About author:WANG Xiaodong, born in 1989, Ph.D. candidate. His research interests include log processing and analyzing, data mining and machine learning.
ZHAO Yining, born in 1983, research assistant. His research interests include log processing and analyzing and data mining.
XIAO Haili, born in 1978, M.S., professor. His research interests include grid computing and distributed computing.
WANG Xiaoning, born in 1981, Ph.D., associate professor. Her research interests include grid computing and distributed computing.
CHI Xuebin, born in 1963, professor. His research interests include parallel computing and software, grid computing technology, high performance computer system maintenance and management.
Supported by:
Strategic Priority Research Program of the Chinese Academy of Sciences(XDA19020101)

摘要/Abstract

摘要：

日志分析对于计算机系统的稳定运行起着至关重要的作用,然而日志通常是非结构化的,不利于自动化分析,如何自动化将日志的模式提炼出来并变成结构化的数据具有重要的实际意义。提出了LDmatch算法,该算法以单词匹配率为基础实现了一种日志模式提炼算法。传统的日志匹配算法在进行相似度计算时使用一对一单词匹配法,而LDmatch算法根据两条日志所包含的单词之间的最长公共子序列计算日志之间的相似度,并以此为基础进行日志分类。LDmatch算法还能实时得到日志模板并更新。除此之外,该算法的模式仓库使用了基于哈希表的数据结构进行存储,该存储结构细化了日志的分类,减少了日志匹配时的比较次数,从而提高了日志模式提炼算法的匹配效率。为了验证算法的优势,将LDmatch算法应用于开源数据集以及国家高性能计算环境实际产生的日志数据集,并且使用多种其他日志模式提炼算法进行对比并得出实验结果,最终证明了该算法在准确度、鲁棒性和效率上具有优势。

关键词: 日志模式提炼, 单词匹配率, 日志模板, 哈希表

Abstract:

Log analysis plays an important role in the stable operation of computer system. However, logs are usua-lly unstructured, which is not conducive to automatic analysis. How to categorize logs and turn them into structured data automatically is of great practical significance. In this paper, LDmatch algorithm is proposed, which imple-ments a log pattern extracting algorithm based on word matching rate. Traditional log matching algorithms use one-to-one word matching method in similarity calculation, while the proposed LDmatch algorithm calculates the simi-larity between logs according to the longest common subsequence (LCS) of words contained in two logs, and classi-fies logs based on the LCS. LDmatch algorithm can also get real-time log template and update. In addition, the pat-tern warehouse of the algorithm uses a data structure based on hash table for storage, which refines the classification of logs and reduces the times of comparison during log matching, thus improving the matching efficiency of the algorithm. In order to verify the advantages of the algorithm, it is applied to the open source data set and the actual log data set generated by the CNGrid. A variety of other log pattern extraction algorithms are used for comparison and experimental results are obtained. Finally, the advantages of the algorithm in accuracy, robustness and efficiency are proven.

Key words: log pattern extraction, word matching rate, log template, hash table

中图分类号:

TP316

王晓东, 赵一宁, 肖海力, 王小宁, 迟学斌. 高性能计算环境中日志模式提炼方法的研究[J]. 计算机科学与探索, 2022, 16(10): 2264-2272.

WANG Xiaodong, ZHAO Yining, XIAO Haili, WANG Xiaoning, CHI Xuebin. Research on Method of Log Pattern Extracting in High-Performance Computing Environment[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(10): 2264-2272.

图/表 7

图1 LARGE系统结构

Fig.1 System structure of LARGE

图2 日志模式提炼算法流程图

Fig.2 Flowchart of log pattern extracting algorithms

图3 日志模式仓库

Fig.3 Log patterns warehouse

图4 阈值-准确度

Fig.4 Threshold and accuracy

图5 不同模式提炼方法下的准确度

Fig.5 Accuracy in different pattern extracting methods

表1 不同种类日志上的实验结果

Table 1 Experimental results on different kinds of logs

日志种类	LDmatch 最优阈值	准确度
日志种类	LDmatch 最优阈值	LDmatch	Lmatch	Match
Andriod	0.90	0.912	0.912	0.689
Apache	0.55	1.000	1.000	1.000
BGL	0.50	0.845	0.836	0.645
Hadoop	0.40	0.881	0.877	0.871
HDFS	0.60	0.841	0.841	0.841
HealthApp	0.30	0.622	0.557	0.628
HPC	0.60	0.906	0.904	0.895
Linux	0.45	0.610	0.599	0.531
Mac	0.60	0.858	0.846	0.510
OpenSSH	0.75	0.678	0.678	0.373
OpenStack	0.70	0.838	0.807	0.733
Proxifier	0.70	0.517	0.517	0.517
Spark	0.25	0.920	0.718	0.599
Thunderbird	0.55	0.916	0.922	0.825
Windows	0.70	0.993	0.990	0.705
Zookeeper	0.60	0.964	0.964	0.894
平均值	—	0.831	0.810	0.703

图6 不同日志数量下模式提炼方法消耗的时间

Fig.6 Time consumed by pattern extracting methods under different number of logs

参考文献 16

[1]	赵一宁, 肖海力. 网格环境日志分析框架LARGE的设计[J]. 科研信息化技术与应用, 2016, 7(3): 3-7.
	ZHAO Y N, XIAO H L. The design of LARGE: log anal-yzing framework in grid environment[J]. E-science Techno-logy & Application, 2016, 7(3): 3-7.
[2]	XU W, HUANG L, FOX A, et al. Detecting large-scale system problems detection by mining console logs[C]// Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, Oct 2009. New York: ACM, 2009: 117-132.
[3]	LOU J G, FU Q, YANG S Q, et al. Mining invariants from console logs for system problem detection[C]// Proceedings of the 2010 USENIX Annual Technical Conference, Boston, Jun 23-25, 2010. Berkeley: USENIX Association, 2010: 231-244.
[4]	LIN Q W, ZHANG H Y, LOU J G, et al. Log clustering based problem identification for online service systems[C]// Proceedings of the 38th International Conference on Soft-ware Engineering Companion, Austin, May 14-22, 2016. Pisca-taway: IEEE, 2016: 102-111.
[5]	DU M, LI F F, ZHENG G N, et al. DeepLog: anomaly detection and diagnosis from system logs through deep lear-ning[C]// Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Dallas, Oct 30-Nov 3, 2017. New York: ACM, 2017: 1285-1298.
[6]	ZHAO Y N, XIAO H L. Extracting log patterns from sys-tem logs in LARGE[C]// Proceedings of the 2016 IEEE Inter-national Parallel and Distributed Processing Symposium Workshops, Chicago, May 23-27, 2016. Washington: IEEE Computer Society, 2016: 1645-1652.
[7]	ZHAO Y N, WANG X D, XIAO H L, et al. Improvement of log pattern extracting algorithm using text similarity[C]// Proceedings of the 2018 IEEE International Parallel and Distributed Processing Symposium Workshops, Vancouver, May 21-25, 2018. Washington: IEEE Computer Society, 2018: 507-514.
[8]	VAARANDI R. A data clustering algorithm for mining pat-terns from event logs[C]// Proceedings of the 3rd IEEE Work-shop on IP Operations & Management, Kansas City, Oct 3, 2003. Piscataway: IEEE, 2003: 119-126.
[9]	NAGAPPAN M, VOUK M A. Abstracting log lines to log event types for mining software system logs[C]// Proceedings of the 7th IEEE Working Conference on Mining Software Repositories, Cape Town, May 2-3, 2010. Piscataway: IEEE, 2010: 114-117.
[10]	FU Q, LOU J G, WANG Y, et al. Execution anomaly detec-tion in distributed systems through unstructured log analysis[C]// Proceedings of the 9th IEEE International Conference on Data Mining, Miami Beach, Dec 6-9, 2009. Piscataway: IEEE, 2009: 149-158.
[11]	TANG L, LI T, PERNG C S. LogSig: generating system events from raw textual logs[C]// Proceedings of the 20th ACM Conference on Information and Knowledge Management, Oct 2011. New York: ACM, 2011: 785-794.
[12]	MIZUTANI M. Incremental mining of system log format[C]// Proceedings of the 2013 IEEE International Conference on Services Computing, Santa Clara, Jun 28-Jul 3, 2013. Pis-cataway: IEEE, 2013: 595-602.
[13]	MESSAOUDI S, PANICHELLA A, BIANCULLI D, et al. A search-based approach for accurate identification of log message formats[C]// Proceedings of the 26th IEEE/ACM International Conference on Program Comprehension, May 2018. New York: ACM, 2018: 167-177.
[14]	DAI H T, LI H, CHEN C S, et al. Logram: efficient log parsing using n-gram dictionaries[J]. IEEE Transactions on Software Engineering, 2022, 48(3): 879-892.
[15]	NEDELKOSKI S, BOGATINOVSKI J, ACKER A, et al. Self-supervised log parsing[J]. arXiv:2003.07905, 2020.
[16]	ZHU J M, HE S L, LIU J Y, et al. Tools and benchmarks for automated log parsing[C]//Proceedings of the 41st Interna-tional Conference on Software Engineering:Software Engi-neering in Practice, Montreal, May 25-31, 2019. Piscataway: IEEE, 2019: 121-130.

高性能计算环境中日志模式提炼方法的研究

Research on Method of Log Pattern Extracting in High-Performance Computing Environment

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 16

相关文章 1

编辑推荐

Metrics