Advances in Knowledge Graph Embedding Based on Graph Neural Networks

doi:10.3778/j.issn.1673-9418.2212063

Abstract

Abstract: As graph neural networks continue to develop, knowledge graph embedding methods based on graph neural networks are receiving increasing attention from researchers. Compared with traditional methods, they can better handle the diversity and complexity of entities, and capture the multiple features and complex relationships of entities, thereby improving the representation ability and application value of knowledge graphs. This paper firstly outlines the development history of knowledge graphs and the basic concepts of knowledge graphs and graph neural networks. Secondly, it focuses on discussing the design ideas and algorithm frameworks of knowledge graph embedding based on graph convolution, graph neural networks, graph attention, and graph autoencoders. Then, it describes the performance of graph neural network knowledge graph embedding in tasks such as link prediction, entity alignment, knowledge graph reasoning, and knowledge graph completion, while supplementing some research on commonsense knowledge graphs with graph neural networks. Finally, this paper makes a comprehensive summary, and future research directions are outlined with respect to some challenges and issues in knowledge graph embedding.

Key words: knowledge graph, knowledge graph embedding, graph neural network, representation learning

摘要： 随着图神经网络的发展，基于图神经网络的知识图谱嵌入方法日益受到研究人员的关注。相比传统的方法，它可以更好地处理实体的多样性和复杂性，并捕捉实体的多重特征和复杂关系，从而提高知识图谱的表示能力和应用价值。首先概述知识图谱的发展历程，梳理知识图谱和图神经网络的基本概念；其次着重讨论基于图卷积、图神经、图注意力以及图自编码器的知识图谱嵌入的设计思路和算法框架；然后描述图神经网络的知识图谱嵌入在链接预测、实体对齐、知识推理以及知识图谱补全等任务中的性能，同时补充图神经网络在常识性知识图谱中的一些研究；最后进行全面性的总结，并针对知识图谱嵌入存在的一些问题和挑战，勾画未来研究方向。

关键词: 知识图谱, 知识图谱嵌入, 图神经网络, 表示学习

YAN Zhaoyao, DING Cangfeng, MA Lerong, CAO Lu, YOU Hao. Advances in Knowledge Graph Embedding Based on Graph Neural Networks[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(8): 1793-1813.

延照耀, 丁苍峰, 马乐荣, 曹璐, 游浩. 面向图神经网络的知识图谱嵌入研究进展[J]. 计算机科学与探索, 2023, 17(8): 1793-1813.

References

[1] SINGHAL A. Introducing the knowledge graph: things, not strings[J]. Official Google Blog, 2012, 5: 16.
[2] 刘知远, 孙茂松, 林衍凯, 等. 知识表示学习研究进展[J]. 计算机研究与发展, 2016, 53(2): 247-261.
LIU Z Y, SUN M S, LIN Y K, et al. Knowledge representation learning: a review[J]. Journal of Computer Research and Development, 2016, 53(2): 247-261.
[3] MIKOLOV T, SUTSKEVER I, CHEN K, et al. Distributed representations of words and phrases and their compositionality[C]//Advances in Neural Information Processing Systems 26, Lake Tahoe, Dec 5-8, 2013: 3111-3119.
[4] 杨东华, 何涛, 王宏志, 等. 面向知识图谱的图嵌入学习研究进展[J]. 软件学报, 2022, 33(9): 3370-3390.
YANG D H, HE T, WANG H Z, et al. Survey on know-ledge graph embedding learning[J]. Journal of Software, 2022, 33(9): 3370-3390.
[5] SCARSELLI F, GORI M, TSOI A C, et al. The graph neural network model[J]. IEEE Transactions on Neural Networks, 2008, 20(1): 61-80.
[6] BORDES A, WESTON J, USUNIER N. Open question answering with weakly supervised embedding models[C]//LNCS 8724: Proceedings of the 2014 European Conference on Machine Learning and Knowledge Discovery in Database, Nancy, Sep 15-19, 2014. Cham: Springer, 2014: 165-180.
[7] DAIBER J, JAKOB M, HOKAMP C, et al. Improving efficiency and accuracy in multilingual entity extraction[C]//Proceedings of the 9th International Conference on Semantic Systems, Graz, Sep 4-6, 2013. New York: ACM, 2013: 121-124.
[8] HOFFMANN R, ZHANG C, LING X, et al. Knowledge-based weak supervision for information extraction of overlapping relations[C]//Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, Jun 19-24, 2011. Stroudsburg: ACL, 2011: 541-550.
[9] GUO Q, ZHUANG F, QIN C, et al. A survey on knowledge graph-based recommender systems[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34(8): 3549-3568.
[10] NICKEL M, TRESP V, KRIEGEL H P. A three-way model for collective learning on multi-relational data[C]//Proceedings of the 28th International Conference on Machine Learning, Bellevue, Jun 28-Jul 2, 2011. Madison: Omnipress, 2011: 809-816.
[11] YANG B, YIH S W, HE X, et al. Embedding entities and relations for learning and inference in knowledge bases[C]//Proceedings of the 3rd International Conference on Learning Representations, San Diego, May 7-9, 2015: 1-12.
[12] TROUILLON T, WELBL J, RIEDEL S, et al. Complex embeddings for simple link prediction[C]//Proceedings of the 33rd International Conference on Machine Learning, New York, Jun 19-24, 2016: 2071-2080.
[13] NICKEL M, ROSASCO L, POGGIO T. Holographic embeddings of knowledge graphs[C]//Proceedings of the 30th AAAI Conference on Artificial Intelligence, Phoenix Arizonam, Feb 12-17, 2016: 1955-1961.
[14] BORDES A, USUNIER N, GARCIA-DURAN A, et al. Translating embeddings for modeling multi-relational data[C]//Advances in Neural Information Processing Systems 26, Lake Tahoe, Dec 5-8, 2013: 2787-2795.
[15] SUN Z, DENG Z H, NIE J Y, et al. Rotate: knowledge graph embedding by relational rotation in complex space[J]. arXiv:1902.10197, 2019.
[16] DETTMERS T, MINERVINI P, STENETORP P, et al. Convolutional 2D knowledge graph embeddings[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence, the 30th Innovative Applications of Artificial Intelligence, and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, Feb 2-7, 2018. Menlo Park, AAAI, 2018: 1811-1818.
[17] 吴博, 梁循, 张树森, 等. 图神经网络前沿进展与应用[J]. 计算机学报, 2022, 45(1): 35-68.
WU B, LIANG X, ZHANG S S, et al. Advances and applications in graph neural network[J]. Chinese Journal of Computers, 2022, 45(1): 35-68.
[18] KHAMSI M A, KIRK W A. An introduction to metric spaces and fixed point theory[M]. New York: John Wiley & Sons, 2011.
[19] GILMER J, SCHOENHOLZ S S, RILEY P F, et al. Neural message passing for quantum chemistry[C]//Proceedings of the 34th International Conference on Machine Learning, Sydney, Aug 6-11, 2017: 1263-1272.
[20] BENGIO Y, COURVILLE A, VINCENT P. Representation learning: a review and new perspectives[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(8): 1798-1828.
[21] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553): 436-444.
[22] SCHLICHTKRULL M, KIPF T N, BLOEM P, et al. Modeling relational data with graph convolutional networks[C]//LNCS 10843: Proceedings of the 15th International Conference the Semantic Web, Heraklion, Jun 3-7, 2018. Cham: Springer, 2018: 593-607.
[23] YE R, LI X, FANG Y, et al. A vectorized relational graph convolutional network for multi-relational network alignment[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macao, China, Aug 10-16, 2019: 4135-4141.
[24] CAI L, YAN B, MAI G, et al. TransGCN: coupling transformation assumptions with graph convolutional networks for link prediction[C]//Proceedings of the 10th International Conference on Knowledge Capture, Marina Del Rey, Nov 19-21, 2019. New York: ACM, 2019: 131-138.
[25] VASHISHTH S, SANYAL S, NITIN V, et al. Composition-based multi-relational graph convolutional networks[J]. arXiv: 1911.03082, 2019.
[26] YU D, YANG Y, ZHANG R, et al. Knowledge embedding based graph convolutional network[C]//Proceedings of the Web Conference 2021, Ljubljana, Apr 19-23, 2021. New York: ACM, 2021: 1619-1628.
[27] SHANG C, TANG Y, HUANG J, et al. End-to-end structure-aware convolutional networks for knowledge base completion[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence, the 31st Innovative Applications of Artificial Intelligence Conference, the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, Honolulu, Jan 27-Feb 1, 2019. Menlo Park: AAAI, 2019: 3060-3067.
[28] YAN Y, LIU L, BAN Y, et al. Dynamic knowledge graph alignment[C]//Proceedings of the 35th AAAI Conference on Artificial Intelligence, the 33rd Conference on Innovative Applications of Artificial Intelligence, the 11th Symposium on Educational Advances in Artificial Intelligence, Feb 2-9, 2021. Menlo Park: AAAI, 2021: 4564-4572.
[29] WANG Z, REN Z, HE C, et al. Robust embedding with multi-level structures for link prediction[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macao, China, Aug 10-16, 2019: 5240-5246.
[30] WANG S, WEI X, DOS SANTOS C N, et al. Mixed-curvature multi-relational graph neural network for knowledge graph completion[C]//Proceedings of the Web Conference 2021, Ljubljana, Apr 19-23, 2021: 1761-1771.
[31] RICHARDSON M, DOMINGOS P. Markov logic networks[J]. Machine Learning, 2006, 62(1): 107-136.
[32] ZHANG Y, CHEN X, YANG Y, et al. Efficient probabilistic logic reasoning with graph neural networks[C]//Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Apr 26-30, 2020: 1-20.
[33] TERU K K, DENIS E G, HAMILTON W L. Inductive relation prediction by subgraph reasoning[C]//Proceedings of the 37th International Conference on Machine Learning, Jul 13-18, 2020: 9448-9457.
[34] LIU S, GRAU B, HORROCKS I, et al. INDIGO: GNN-based inductive knowledge graph completion using pair-wise encoding[C]//Advances in Neural Information Processing Systems 34, Dec 6-14, 2021: 2034-2045.
[35] CAO Y, LIU Z, LI C, et al. Multi-channel graph neural network for entity alignment[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Jul 28-Aug 2, 2019. Stroudsburg: ACL, 2019: 1452-1461.
[36] WANG Z, YANG J, YE X. Knowledge graph alignment with entity-pair embedding[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Nov 16-20, 2020. Stroudsburg: ACL, 2020: 1672-1680.
[37] MAO X, WANG W, XU H, et al. MRAEA: an efficient and robust entity alignment approach for cross-lingual knowledge graph[C]//Proceedings of the 13th International Conference on Web Search and Data Mining, Houston, Feb 3-7, 2020. New York: ACM, 2020: 420-428.
[38] NATHANI D, CHAUHAN J, SHARMA C, et al. Learning attention-based embeddings for relation prediction in know-ledge graphs[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Jul 28-Aug 2, 2019. Stroudsburg: ACL, 2019: 4710-4723.
[39] CHEN M, ZHANG Y, KOU X, et al. r-GAT: relational graph attention network for multi-relational graphs[J]. arXiv:2109. 05922, 2021.
[40] GUO L, WANG W, SUN Z, et al. Decentralized knowledge graph representation learning[J]. arXiv:2010.08114, 2020.
[41] LIU X, TAN H, CHEN Q, et al. RAGAT: relation aware graph attention network for knowledge graph completion[J]. IEEE Access, 2021, 9: 20840-20849.
[42] ZHANG Z, ZHUANG F, ZHU H, et al. Relational graph neural network with hierarchical attention for knowledge graph completion[C]//Proceedings of the 2020 AAAI Conference on Artificial Intelligence, New York, Feb 7-12, 2020: 9612-9619.
[43] LI Z, LIU H, ZHANG Z, et al. Learning knowledge graph embedding with heterogeneous relation attention networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(8): 3961-3973.
[44] ZHAO X, JIA Y, LI A, et al. Target relational attention-oriented knowledge graph reasoning[J]. Neurocomputing, 2021, 461: 577-586.
[45] JI K, HUI B, LUO G. Graph attention networks with local structure awareness for knowledge graph completion[J]. IEEE Access, 2020, 8: 224860-224870.
[46] SUN Z, WANG C, HU W, et al. Knowledge graph alignment network with gated multi-hop neighborhood aggregation[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence, the 32nd Innovative Applications of Artificial Intelligence Conference, the 10th AAAI Symposium on Educational Advances in Artificial Intelligence, New York, Feb 7-12, 2020. Menlo Park: AAAI, 2020: 222-229.
[47] ZHANG Z, HUANG J, TAN Q. Association rules enhanced knowledge graph attention network[J]. Knowledge-Based Systems, 2022, 239: 108038.
[48] DAI G, WANG X, ZOU X, et al. MRGAT: multi-relational graph attention network for knowledge graph completion[J]. Neural Networks, 2022, 154: 234-245.
[49] BALDI P. Autoencoders, unsupervised learning, and deep architectures[C]//Proceedings of the 2011 ICML Workshop on Unsupervised and Transfer Learning, Bellevue, Jul 2, 2011: 37-50.
[50] RUMELHART D E, HINTON G E, WILLIAMS R J. Learning internal representations by error propagation[M]. Cambridge: MIT Press, 1985.
[51] KINGMA D P, WELLING M. Auto-encoding variational Bayes[J]. arXiv:1312.6114, 2013.
[52] TSCHANNEN M, BACHEM O, LUCIC M. Recent advances in autoencoder-based representation learning[J]. arXiv:1812. 05069, 2018.
[53] SALHA G, LIMNIOS S, HENNEQUIN R, et al. Gravity-inspired graph autoencoders for directed link prediction[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, Nov 3-7, 2019. New York: ACM, 2019: 589-598.
[54] ZHANG S, ZHANG Z, ZHUANG F, et al. Compressing knowledge graph embedding with relational graph auto-encoder[C]//Proceedings of the 2020 IEEE 10th International Conference on Electronics Information and Emergency Communication, Beijing, Jul 17-19, 2020. Piscataway: IEEE, 2020: 366-370.
[55] JANG E, GU S, POOLE B. Categorical reparameterization with gumbel-softmax[J]. arXiv:1611.01144, 2016.
[56] YANG K. GCN-VAE for knowledge graph completion[EB/OL]. [2022-09-14]. http://snap.stanford.edu/class/cs224w-2019/project/26425038.pdf.
[57] HU K R, LIU H, ZHAN C J, et al. Learning knowledge graph embedding with a bi-directional relation encoding network and a convolutional autoencoder decoding network[J]. Neural Computing and Applications, 2021, 33(17): 11157-11173.
[58] TOUTANOVA K, CHEN D. Observed versus latent features for knowledge base and text inference[C]//Proceedings of the 3rd Workshop on Continuous Vector Space Models and Their Compositionality, Beijing, Jul 26-31, 2015. Stroudsburg: ACL, 2015: 57-66.
[59] SUN Z, HU W, LI C. Cross-lingual entity alignment via joint attribute-preserving embedding[C]//LNCS 10587: Proceedings of the 16th International Semantic Web Conference, Vienna, Oct 21-25, 2017. Cham: Springer, 2017: 628-644.
[60] XIONG W, HOANG T, WANG W Y. Deeppath: a reinforcement learning method for knowledge graph reasoning[J]. arXiv:1707.06690, 2017.
[61] SUCHANEK F M, KASNECI G, WEIKUM G. YAGO: a core of semantic knowledge[C]//Proceedings of the 16th International Conference on World Wide Web, Banff, May 8-12, 2007. New York: ACM, 2007: 697-706.
[62] NIE K, ZENG K, MENG Q. Knowledge reasoning method for military decision support knowledge graph mixing rule and graph neural networks learning together[C]//Proceedings of the 2020 Chinese Automation Congress, Shanghai, Nov 6-8, 2020. Piscataway: IEEE, 2020: 4013-4018.
[63] ZITNIK M, AGRAWAL M, LESKOVEC J. Modeling polypharmacy side effects with graph convolutional networks[J]. Bioinformatics, 2018, 34(13): i457-i466.
[64] CUI Z, HENRICKSON K, KE R, et al. Traffic graph convolutional recurrent neural network: a deep learning framework for network-scale traffic learning and forecasting[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 21(11): 4883-4894.
[65] CHEN Z, WANG Y, ZHAO B, et al. Knowledge graph completion: a review[J]. IEEE Access, 2020, 8: 192435-192456.
[66] LIN Y, LIU Z, SUN M, et al. Learning entity and relation embeddings for knowledge graph completion[C]//BONET B, KOENIG S. Proceedings of the 29th AAAI Conference on Artificial Intelligence, Austin, Jan 25-30, 2015. Menlo Park: AAAI, 2015: 2181-2187.
[67] SHEN Y, DING N, ZHENG H T, et al. Modeling relation paths for knowledge graph completion[J]. IEEE Transactions on Knowledge and Data Engineering, 2020, 33(11): 3607-3617.
[68] ZHANG Y, ZHENG W, WANG H, et al. Joint entity structural and attribute information for knowledge graph completion[C]//Proceedings of the 2021 International Conference on High Performance Computing and Communication, Xiamen, Dec 3-5, 2021: 264-269.
[69] JUNG J, JUNG J, KANG U. Learning to walk across time for interpretable temporal knowledge graph completion[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, New York, Aug 14-18, 2021. New York: ACM, 2021: 786-795.
[70] GARCíA-DURáN A, DUMAN?I? S, NIEPERT M. Learning sequence encoders for temporal knowledge graph completion[J]. arXiv:1809.03202, 2018.
[71] REITER R. On closed world data bases[M]. San Mateo: Morgan Kaufmann, 1981.
[72] ILIEVSKI F, SZEKELY P, ZHANG B. CSKG: the commonsense knowledge graph[C]//LNCS 12731: Proceedings of the 18th International Conference the Semantic Web, Jun 6-10, 2021. Cham: Springer, 2021: 680-696.
[73] SPEER R, CHIN J, HAVASI C. ConceptNet 5.5: an open multilingual graph of general knowledge[C]//Proceedings of the 31st AAAI Conference on Artificial Intelligence, San Francisco, Feb 4-9, 2017. Menlo Park: AAAI, 2017: 4444-4451.
[74] SAP M, LEBRAS R, ALLAWAY E, et al. ATOMIC: an atlas of machine commonsense for if-then reasoning[J]. arXiv:1811.00146, 2018.
[75] VRANDE?I? D, KR?TZSCH M. Wikidata: a free collaborative knowledgebase[J]. Communications of the ACM, 2014, 57(10): 78-85.
[76] MILLER G A. WordNet: a lexical database for English[J]. Communications of the ACM, 1995, 38(11): 39-41.
[77] SCHULER K K. VerbNet: a broad-coverage, comprehensive verb lexicon[D]. Philadelphia: University of Pennsylvania, 2005.
[78] BAKER C F, FILLMORE C J, LOWE J B. The Berkeley FrameNet project[C]//Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Montreal, 1998: 86-90.
[79] KRISHNA R, ZHU Y, GROTH O, et al. Visual genome: connecting language and vision using crowdsourced dense image annotations[J]. International Journal of Computer Vision, 2017, 123(1): 32-73.
[80] DENG J, DONG W, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, Jun 20-25, 2009. Washington: IEEE Computer Society, 2009: 248-255.
[81] YASUNAGA M, REN H, BOSSELUT A, et al. QA-GNN: reasoning with language models and knowledge graphs for question answering[J]. arXiv: 2104. 06378, 2021.
[82] SHWARTZ V, WEST P, BRAS R L, et al. Unsupervised commonsense question answering with self-talk[J]. arXiv: 2004.05483, 2020.
[83] BOSSELUT A, RASHKIN H, SAP M, et al. COMET: commonsense transformers for automatic knowledge graph construction[J]. arXiv:1906.05317, 2019.
[84] PALATUCCI M, POMERLEAU D, HINTON G E, et al. Zero-shot learning with semantic output codes[C]//Advances in Neural Information Processing Systems 22, Vancouver, Dec 7-10, 2009: 1410-1418.
[85] WANG X, YE Y, GUPTA A. Zero-shot recognition via semantic embeddings and knowledge graphs[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 6857-6866.
[86] KAMPFFMEYER M, CHEN Y, LIANG X, et al. Rethinking knowledge graph propagation for zero-shot learning[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 11479-11488.
[87] NAYAK N V, BACH S H. Zero-shot learning with common sense knowledge graphs[J]. arXiv:2006.10713, 2020.
[88] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems 30, Long Beach, Dec 4-9, 2017: 5998-6008.
[89] ROJAS-CARULLA M, SCH?LKOPF B, TURNER R, et al. Invariant models for causal transfer learning[J]. The Journal of Machine Learning Research, 2018, 19(1): 1309-1342.
[90] WU Q, ZHANG H, YAN J, et al. Handling distribution shifts on graphs: an invariance perspective[J]. arXiv:2202. 02466, 2022.
[91] ARJOVSKY M, BOTTOU L, GULRAJANI I, et al. Invariant risk minimization[J]. arXiv:1907.02893, 2019.
[92] LI H, WANG X, ZHANG Z, et al. OOD-GNN: out-of-distribution generalized graph neural network[J]. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(7): 7328-7340.