User Behavior Analysis with RNN and Graph Neural Networks

doi:10.3778/j.issn.1673-9418.2005018

Abstract

Abstract:

With the increasing amount of logs produced by nodes in CNGrid (China National Grid), traditional manual methods for user behavior analysis can no longer meet the need of daily analysis. In recent years, deep learning has shown good results in key tasks related to computer sciences, such as intrusion detection, image recognition, natural language processing and malware detection. This paper demonstrates how to apply deep learning models to user behavior analysis. To this end, this paper classifies user behavior in CNGrid and extracts a large number of user operation sequences bounded to sessions. These sequences are put into deep learning models. This paper proposes a deep learning model that combines recurrent neural network (RNN) with graph neural network (GNN) to predict the user behavior. Graph neural network can catch the hidden state of the user’s local behavior, so it can be used as preprocessing. Recurrent neural network can catch the message of time sequence. The model is built by combining GNN and RNN to acquire both advantages. In order to verify the effectiveness of the model, this paper conducts experiments of the real user behavior datasets on CNGrid and compares them with a variety of other methods. Experimental results demonstrate the effectiveness of this novel deep learning model.

Key words: user behavior analysis, graph neural network (GNN), recurrent neural network (RNN)

摘要：

随着国家高性能计算环境（CNGrid）各个节点产生日志数量不断增加，采用传统的人工方式进行用户行为分析已不能满足日常的分析需求。近年来，深度学习在入侵检测、图像识别、自然语言处理和恶意软件检测等与计算机科学相关的关键任务中取得了良好的效果。演示了如何将深度学习模型应用于用户行为分析。为此，在CNGrid中对用户行为进行分类，提取大量绑定到会话的用户操作序列，然后将这些序列放入抽象的深度学习模型中。提出了一种基于图神经网络（GNN）和循环神经网络（RNN）的深度学习模型来预测用户行为。图神经网络能够捕捉用户局部行为的隐藏状态，可以作为预处理步骤。循环神经网络能够捕捉时间序列的信息。因此，通过将GNN和RNN相结合的方式来构建该模型，以获得两者的优点。为了验证模型的有效性，在CNGrid的真实用户行为数据集上进行了实验,并在实验中与多种不同的其他方法进行对比。实验结果证明了这种新的深度学习模型的有效性。

关键词: 用户行为分析, 图神经网络（GNN）, 循环神经网络（RNN）

WANG Xiaodong, ZHAO Yining, XIAO Haili, WANG Xiaoning, CHI Xuebin. User Behavior Analysis with RNN and Graph Neural Networks[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(5): 838-847.

王晓东, 赵一宁, 肖海力, 王小宁, 迟学斌. 使用GNN与RNN实现用户行为分析[J]. 计算机科学与探索, 2021, 15(5): 838-847.

References

[1] ZHAO Y N, XIAO H L. The design of LARGE: log anal-yzing framework in grid environment[J]. E-Science Tech-nology & Application, 2016, 7(3): 3-7.
赵一宁, 肖海力. 网格环境日志分析框架LARGE的设计[J]. 科研信息化技术与应用, 2016, 7(3): 3-7.
[2] SALAKHUTDINOV R, MNIH A. Probabilistic matrix factor-ization[C]//Proceedings of the 21st Annual Conference on Neural Information Processing Systems, Vancouver, Dec 3-6, 2007. Red Hook: Curran Associates, 2007: 1257-1264.
[3] KOREN Y, BELL R, VOLINSKY C. Matrix factorization tech-niques for recommender systems[J]. Computer, 2009, 42(8): 30-37.
[4] KOREN Y, BELL R M. Advances in collaborative filtering[M]//RICCI F, ROKACH L, SHAPIRA B, eds. Recommender Systems Handbook. Berlin, Heidelberg: Springer, 2011.
[5] SARWAR B M, KARYPIS G, KONSTAN J A, et al. Item-based collaborative filtering recommendation algorithms[C]//Proceedings of the 10th International World Wide Web Conference, Hong Kong, China, May 1-5, 2001. New York:ACM, 2001: 285-295.
[6] LINDEN G, SMITH B, YORK J. Amazon.com recommen-dations: item-to-item collaborative filtering[J]. IEEE Internet Computing, 2003, 7(1): 76-80.
[7] SHANI G, HECKERMAN D, BRAFMAN R I. An MDP-based recommender system[J]. Journal of Machine Learning Research, 2005, 6(1): 1265-1295.
[8] RENDLE S, FREUDENTHALER C, SCHMIDT-THIEME L. Factorizing personalized Markov chains for next-basket recommendation[C]//Proceedings of the 19th International Conference on World Wide Web, Raleigh, Apr 26-30, 2010. New York: ACM, 2010: 811-820.
[9] LECUN Y, BENGIO Y, HINTON G E. Deep learning[J]. Nature, 2015, 521(7553): 436-444.
[10] ZHANG G H, LIU B. Research on time series classification using CNN and bidirectional GRU[J]. Journal of Frontiers of Computer Science and Technology, 2019, 13(6): 916-927.
张国豪, 刘波. 采用CNN和Bidirectional GRU的时间序列分类研究[J]. 计算机科学与探索, 2019, 13(6): 916-927.
[11] ZHANG J J, HUANG H, HU Y, et al. Modified recurrent neural networks in spoken language understanding[J]. Computer Engineering and Applications, 2019, 55(18): 155-160.
张晶晶, 黄浩, 胡英, 等. 口语理解中改进循环神经网络的应用[J]. 计算机工程与应用, 2019, 55(18): 155-160.
[12] SHENG L S, LI C, LI X. Classification of mammography based on attention mechanism[J]. Computer Engineering and Applications, 2020, 56(8): 166-170.
盛龙帅, 李策, 李欣. 基于注意力机制的乳腺X线摄影分类方法[J]. 计算机工程与应用, 2020, 56(8): 166-170.
[13] SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, Dec 8-13, 2014. Red Hook: Curran Associates, 2014: 3104-3112.
[14] HIDASI B, KARATZOGLOU A, BALTRUNAS L, et al. Session-based recommendations with recurrent neural net-works[C]//Proceedings of the 4th International Conference on Learning Representations, San Juan, May 2-4, 2016: 1-10.
[15] TAN Y K, XU X X, LIU Y. Improved recurrent neural networks for session-based recommendations[C]//Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, Boston, Sep 15, 2016. New York: ACM, 2016: 17-22.
[16] TUAN T X, PHUONG T M. 3D convolutional networks for session-based recommendation with content features[C]// Proceedings of the 11th ACM Conference on Recommender Systems, Como, Aug 27-31, 2017. New York: ACM, 2017: 138-146.
[17] LI J, REN P, CHEN Z, et al. Neural attentive session-based recommendation[C]//Proceedings of the 2017 ACM Con-ference on Information and Knowledge Management, Sing-apore, Nov 6-10, 2017. New York: ACM, 2017: 1419-1428.
[18] LIU Q, ZENG Y F, MOKHOSI R, et al. STAMP: short-term attention/memory priority model for session-based recommendation[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, Aug 19-23, 2018. New York: ACM, 2018: 1831-1839.
[19] GORI M, MONFARDINI G, SCARSELLI F. A new model for learning in graph domains[C]//Proceedings of the 2005 International Joint Conference on Neural Networks, Montreal, Jul 31-Aug 4, 2005. Piscataway: IEEE, 2005: 1-6.
[20] SCARSELLI F, YONG S L, GORI M, et al. Graph neural networks for ranking Web pages[C]//Proceedings of the 2005 IEEE/WIC/ACM International Conference on Web Intelligence, Compiegne, Sep 19-22, 2015. Washington: IEEE Computer Society, 2005: 666-672.
[21] SCARSELLI F, GORI M, TSOI A C, et al. Computational capabilities of graph neural networks[J]. IEEE Transactions on Neural Networks, 2009, 20(1): 81-102.
[22] LI Y J, TARLOW D, BROCKSCHMIDT M, et al. Gated graph sequence neural networks[C]//Proceedings of the 4th International Conference on Learning Representations, San Juan, May 2-4, 2016: 1-20.
[23] LI Z Y, DING X, LIU T. Constructing narrative event evolutionary graph for script event prediction[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, Jul 13-19, 2018: 4201-4207.
[24] LI R Y, TAPASWI M, LIAO R J, et al. Situation recognition with graph neural networks[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 4183-4192.
[25] MARINO K, SALAKHUTDINOV R, GUPTA A. The more you know: using knowledge graphs for image classification [C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 20-28.
[26] SHUMAN D I, NARANG S K, FROSSARD P, et al. The emerging field of signal processing on graphs[J]. IEEE Signal Processing Magazine, 2013, 30(3): 83-98.
[27] BRUNA J, ZAREMBA W, SZLAM A, et al. Spectral net-works and locally connected networks on graphs[J]. arXiv:1312.6203, 2013.
[28] DEFFERRARD M, BRESSON X, VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Dec 5-10, 2016. Red Hook: Curran Associates, 2016: 3837-3845.
[29] WU S, TANG Y Y, ZHU Y Q, et al. Session-based recom-mendation with graph neural networks[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence, the 31st Innovative Applications of Artificial Intelligence Con-ference, the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, Honolulu, Jan 27-Feb 1, 2019. Menlo Park: AAAI, 2019: 346-353.
[30] GILMER J, SCHOENHOLZ S S, RILEY P F, et al. Neural message passing for quantum chemistry[J]. arXiv:1704.01212, 2017.
[31] FEY M, LENSSEN J E. Fast graph representation learning with PyTorch geometric[J]. arXiv:1903.02428, 2019.
[32] KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[C]//Proceedings of the 5th International Conference on Learning Representations, Toulon, Apr 24-26, 2017: 1-13.
[33] ADAM P, SAM G, FRANCISCO M, et al. PyTorch: an imperative style, high-performance deep learning library[C]//Proceedings of the Annual Conference on Neural Infor-mation Processing Systems, Vancouver, Dec 8-14, 2019. Red Hook: Curran Associates, 2019: 8024-8035.
[34] RENDLE S, FREUDENTHALER C, GANTNER Z, et al. BPR: Bayesian personalized ranking from implicit feedback[C]//Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, Montreal, Jun 18-21, 2009: 452-461.
[35] BIANCHI F M, GRATTAROLA D, LIVI L, et al. Graph neural networks with convolutional ARMA filters[J]. arXiv:1901.01343, 2019.
[36] WU F, ZHANG T, SOUZA JR A H, et al. Simplifying graph convolutional networks[C]//Proceedings of the 36th Intern-ational Conference on Machine Learning, Long Beach, Jun 9-15, 2019: 6861-6871.