Intranet Anomaly Detection Method Using Flow Mining and Graph Mining

doi:10.3778/j.issn.1673-9418.1908003

Abstract

Abstract:

Evidence of malicious activity on the intranet is often hidden in large data streams, such as system logs that accumulate over months or years, whereas data flows are often unbounded, changing, and unlabeled. Therefore, in order to achieve highly accurate anomaly detection, this paper proposes an intranet anomaly detection method that integrates flow mining and graph mining, which not only gives full play to the unsupervised advantages of graph mining, but also integrates the good adaptive ability of flow mining. Through the ensemble classification and update, when the concept drift occurs, this paper uses the ensemble-based method to ensure that the ensemble adapts to the current concept, so that it can detect the malicious behavior of the intranet. Experiments show that this method is more effective than the traditional single-model based method, and can effectively detect the intranet anomalies, which changes their behavior over time to hide malicious activities. The method based on flow mining and graph mining proposed in this paper is very meaningful for the abnormal and unlabeled data of the intranet hidden in a large number of data streams.

Key words: anomaly detection, graph computing, intranet anomaly, ensemble learning

摘要：

内网恶意内部活动的证据通常隐藏在大型数据流中，例如数月或数年累积的系统日志，然而数据流往往是无界的、不断变化的和未标记的。因此，为实现高度准确的异常检测，提出集成流挖掘和图挖掘的内网异常检测方法，在发挥图挖掘的无监督优势的同时，融入了流挖掘的良好自适应能力。采用集成的方法，通过集成分类和更新，当出现概念漂移时，保证集成适应当前概念，使之可以检测到内网恶意行为。实验证明基于集成的方法比传统的单模型方法更有效，可以有效识别随时间改变其行为来隐藏恶意活动的内网异常，在面对隐藏在大量数据流中的内网异常且无标记的数据时，所提出的基于流挖掘和图挖掘的集成方法是十分有意义的。

关键词: 异常检测, 图计算, 内网异常, 集成学习

SUN Wei, ZHANG Yu. Intranet Anomaly Detection Method Using Flow Mining and Graph Mining[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(7): 1154-1163.

孙伟, 张羽. 利用流挖掘和图挖掘的内网异常检测方法[J]. 计算机科学与探索, 2020, 14(7): 1154-1163.

References

[1] Zeadally S, Yu B, Jeong D H, et al. Detecting insider threats: solutions and trends[J]. Information Security Journal: A Global Perspective, 2012, 21(4): 183-192.
[2] Thuraisingham B, Parveen P, Masud M M, et al. Big data analy-tics with applications in insider threat detection[M]. [S.l.]: Auer-bach Publications, 2017.
[3] Ransbotham S, Mitra S. Choice and chance: a conceptual model of paths to information security compromise[J]. Information Systems Research, 2009, 20(1): 121-139.
[4] Homoliak I, Toffalini F, Guarnizo J, et al. Insight into insi-ders and it: a survey of insider threat taxonomies, analysis, modeling, and countermeasures[J]. ACM Computing Surveys, 2019, 52(2): 30.
[5] Teixeira C H C, Fonseca A J, Serafini M, et al. Arabesque: a system for distributed graph mining-extended version[J]. arXiv: 1510.04233, 2015.
[6] Savage D, Zhang X Z, Yu X H, et al. Anomaly detection in online social networks[J]. Social Networks, 2014, 39: 62-70.
[7] Slimani T, Lazzez A. Efficient analysis of pattern and associa-tion rule mining approaches[J]. arXiv:1402.2892, 2014.
[8] Gheyas I A, Abdallah A E. Detection and prediction of insider threats to cyber security: a systematic literature review and meta-analysis[J]. Big Data Analytics, 2016, 1(1): 6.
[9] Kuang F J, Xu W D, Zhang S Y. A novel hybrid KPCA and SVM with GA model for intrusion detection[J]. Applied Soft Computing, 2014, 18: 178-184.
[10] Zhang J, Chen Y, Ju A K. Insider threat detection of adaptive optimization DBN for behavior logs[J]. Turkish Journal of Electrical Engineering & Computer Sciences, 2018, 26(2): 792-802.
[11] Liu L, De Vel O Y, Han Q L, et al. Detecting and preventing cyber insider threats: a survey[J]. IEEE Communications Surveys & Tutorials, 2018, 20(2): 1397-1417.
[12] Rudd E M, Rozsa A, Günther M, et al. A survey of stealth malware attacks, mitigation measures, and steps toward autonomous open world solutions[J]. IEEE Communications Surveys & Tutorials, 2017, 19(2): 1145-1172.
[13] Mishra P, Pilli E S, Varadharajan V, et al. Intrusion detection techniques in cloud environment: a survey[J]. Journal of Network and Computer Applications, 2017, 77: 18-47.
[14] Sun X, Dai J, Liu P, et al. Using Bayesian networks for pro-babilistic identification of zero-day attack paths[J]. IEEE Transactions on Information Forensics and Security, 2018, 13(10): 2506-2521.
[15] Sharif M I, Lee W, Cui W D, et al. Secure in-VM monitoring using hardware virtualization[C]//Proceedings of the 2009 ACM Conference on Computer and Communications Security, Chi-cago, Nov 9-13, 2009. New York: ACM, 2009: 477-487.
[16] Azaria A, Richardson A, Kraus S, et al. Behavioral analysis of insider threat: a survey and bootstrapped prediction in imba-lanced data[J]. IEEE Transactions on Computational Social Systems, 2014, 1(2): 135-155.
[17] Stephens G D, Maloof M A. Insider threat detection: U.S. Patent 8707431[P]. 2014-04-22.
[18] Das S, Liu Y, Zhang W, et al. Semantics-based online malware detection: towards efficient real-time protection against malware[J]. IEEE Transactions on Information Forensics and Security, 2017, 11(2): 289-302.
[19] Campos G O, Zimek A, Sander J, et al. On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study[J]. Data Mining and Knowledge Discovery, 2016, 30(4): 891-927.
[20] Akoglu L, Tong H H, Koutra D. Graph based anomaly detec-tion and description: a survey[J]. Data Mining and Knowledge Discovery, 2015, 29(3): 626-688.
[21] Buczak A L, Guven E. A survey of data mining and machine learning methods for cyber security intrusion detection[J]. IEEE Communications Surveys & Tutorials, 2016, 18(2):1153-1176.
[22] Bell A J C, Rogers M B, Pearce J M. The insider threat: behavioral indicators and factors influencing likelihood of intervention[J]. International Journal of Critical Infrastructure Protection, 2019, 24: 166-176.
[23] Ramirez A G, Lara C, Betev L, et al. Arhuaco: deep learning and isolation based security for distributed high-throughput computing[J]. arXiv:1801.04179, 2018.
[24] Gavai G, Sricharan K, Gunning D, et al. Supervised and unsupervised methods to detect insider threat from enterprise social and online activity data[J]. Journal of Wireless Mobile Networks, Ubiquitous Computing, and Dependable Applica-tions, 2015, 6(4): 47-63.
[25] Aberger C R, Tu S S, Olukotun K, et al. EmptyHeaded: a relational engine for graph processing[C]//Proceedings of the 2016 International Conference on Management of Data, San Francisco, Jun 26-Jul 1, 2016. New York: ACM, 2016:431-446.
[26] Zhang X L, Furtlehner C, Germain-Renaud C, et al. Data stream clustering with affinity propagation[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(7): 1644-1656.
[27] Sun Y, Tang K, Minku L L, et al. Online ensemble learning of data streams with gradually evolved classes[J]. IEEE Trans-actions on Knowledge and Data Engineering, 2016, 28(6): 1532-1545.
[28] Webb G I, Hyde R, Cao H, et al. Characterizing concept drift[J]. Data Mining and Knowledge Discovery, 2016, 30(4): 964-994.
[29] Brzezinski D, Stefanowski J. Combining block-based and online methods in learning ensembles from concept drifting data streams[J]. Information Sciences, 2014, 265: 50-67.
[30] Lv K, Chen Y, Hu C Z. Dynamic defense strategy against advanced persistent threat under heterogeneous networks[J]. Information Fusion, 2019, 49: 216-226.
[31] Chen L, Mei Q L. Mining frequent items in data stream using time fading model[J]. Information Sciences, 2014, 257: 54-69.
[32] Koutra D, Kang U, Vreeken J, et al. Summarizing and under-standing large graphs[J]. Statistical Analysis and Data Mining, 2015, 8(3): 183-202.
[33] Velampalli S, Jonnalagedda M V. Graph based knowledge dis-covery using MapReduce and SUBDUE algorithm[J]. Data & Knowledge Engineering, 2017, 111: 103-113.
[34] Padmanabhan S, Chakravarthy S. HDB-Subdue: a scalable approach to graph mining[C]//LNCS 5691: Proceedings of the 11th International Conference on Data Warehousing and Knowledge Discovery, Linz, Aug 31-Sep 2, 2009. Berlin,Heidelberg: Springer, 2009: 325-338.
[35] Legg P A, Buckley O, Goldsmith M, et al. Automated insider threat detection system using user and role-based profile assessment[J]. IEEE Systems Journal, 2015, 11(2): 503-512.
[36] Chowdhury S, Khanzadeh M, Akula R, et al. Botnet detection using graph-based feature clustering[J]. Journal of Big Data, 2017, 4: 14.
[37] Kim J, Park M, Kim H, et al. Insider threat detection based on user behavior modeling and anomaly detection algorithms[J]. Applied Sciences, 2019, 9(19): 4018.
[38] Haidar D, Gaber M M, Kovalchuk Y. AnyThreat: an oppor-tunistic knowledge discovery approach to insider threat detec-tion[J]. arXiv:1812.00257, 2018.
[39] Hubballi N, Santini J. Detecting TCP ACK storm attack: a state transition modelling approach[J]. IET Networks, 2018, 7(6): 429-434.
[40] Lazier C L, Argenti M. Techniques and systems for detecting anomalous operational data: U.S. Patent 9785495[P]. 2017-10-10.
[41] Jiang D D, Yao C, Xu Z Z, et al. Multi-scale anomaly detection for high-speed network traffic[J]. Transactions on Emerging Telecommunications Technologies, 2015, 26(3): 308-317.
[42] Hubballi N, Suryanarayanan V. False alarm minimization tech-niques in signature-based intrusion detection systems: a survey[J]. Computer Communications, 2014, 49: 1-17.
[43] Siddique K, Akhtar Z, Khan F A, et al. KDD Cup 99 data sets: a perspective on the role of data sets in network intrusion detection research[J]. IEEE Computer, 2019, 52(2): 41-51.
[44] Ring M, Wunderlich S, Scheuring D, et al. A survey of network-based intrusion detection data sets[J]. arXiv:1903. 02460v1, 2019.
[45] Cam-Winget N, Paul S, Anderson B, et al. Intrusion detection model for an internet-of-things operations environment: U.S. Patent Application 16/135756[P]. 2019-08-01.
[46] Liu J, Strohschein D, Samsi S, et al. Large scale organization and inference of an imagery dataset for public safety[J]. arXiv:1908.09006, 2019.
[47] Jiang M, Cui P, Beutel A, et al. Catching synchronized beha-viors in large networks: a graph mining approach[J]. ACM Transactions on Knowledge Discovery from Data, 2016, 10(4): 35.