多级融合知识图谱补全模型

doi:10.3778/j.issn.1673-9418.2404032

摘要/Abstract

摘要： 知识图谱补全旨在通过预测缺失的三元组来扩展和完善知识图谱，多模态知识图谱补全融合了实体的本体信息，如实体描述、实体图像和实体属性，以获取更精确的实体表示。现有研究将不同模态投影到统一的空间中，以获取实体模态联合表示，再融合知识图谱结构信息作出预测。然而，现存方法融合多模态信息时难以捕捉实体背景知识的复杂交互，不可避免地存在信息丢失和特征提取能力不足的问题；同时过拟合及实体关系交互不足限制了二维卷积模型性能，导致难以融合知识图谱结构信息。因此，提出了多级融合知识图谱补全模型，从实体多模态信息融合与知识图谱结构信息融合两方面解决上述问题。为充分融合实体多模态信息，提出同时使用三种不同融合方法，以全面捕捉实体背景知识交互，并联合决策学习，旨在结合不同多模态融合方法提供的互补信息，以获取实体丰富多样的表示；为充分融合知识图谱结构信息，利用特征泛化来缓解二维卷积模型的过拟合问题，并结合特征重塑增强实体与关系间交互，以提升实体与关系间的上下文感知能力。实验结果表明，该模型在多个公开数据集上均取得较好性能。

关键词: 知识图谱补全, 多模态融合, 本体信息, 结构信息, 决策学习

Abstract: Knowledge graph completion aims to expand and enhance knowledge graphs by predicting missing triples. Multi-modal knowledge graph completion integrates entity ontology information such as entity descriptions, entity images, and entity attributes to obtain more accurate entity representations. Existing research projects different modalities into a unified space to obtain joint representations of entities, then combine knowledge graph structural information for predictions. However, existing methods have difficulty in capturing the complex interactions between entity background knowledge when fusing multi-modal information, which inevitably leads to information loss and insufficient feature extraction capabilities; overfitting and limited entity relation interactions restrict the performance of 2D convolution models, making it difficult to integrate knowledge graph structural information. Therefore, this paper uses a multi-level fusion knowledge graph completion model to address the above issues from two aspects: the fusion of entity multimodal information and the integration of knowledge graph structural information. To fully integrate entity multi-modal information, three different fusion methods are simultaneously used to comprehensively capture the interaction of entity background knowledge, along with decision learning, aiming to combine the complementary information provided by different multi-modal fusion methods to obtain rich and diverse entity representations. To fully integrate knowledge graph structural information, feature generalization is proposed to alleviate the overfitting issues of 2D convolution models, combined with feature reshaping to enhance interactions between entities and relations, thereby improving the contextual perception ability of entities and relations. Experiments on multiple public datasets demonstrate the superior performance of the proposed method.

Key words: knowledge graph completion, multi-modal fusion, ontology information, structural information, decision learning

叶志鸿, 吴运兵, 戴思翀, 曾智宏. 多级融合知识图谱补全模型[J]. 计算机科学与探索, 2025, 19(3): 724-737.

YE Zhihong, WU Yunbing, DAI Sichong, ZENG Zhihong. Multi-level Fusion Knowledge Graph Completion Model[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(3): 724-737.

参考文献

[1] RADHAKRISHNAN P, TALUKDAR P, VARMA V. ELDEN: improved entity linking using densified knowledge graphs[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: ACL, 2018: 1844-1853.
[2] ZHU Y K, ZHANG C, RÉ C, et al. Building a large-scale multimodal knowledge base system for answering visual queries[EB/OL]. [2024-02-23]. https://arxiv.org/abs/1507. 05670.
[3] ZHANG Z H, LIU H L, CHEN J Y, et al. An industry evaluation of embedding-based entity alignment[C]//Proceedings of the 28th International Conference on Computational Linguistics: Industry Track, 2020: 179-189.
[4] SUN R, CAO X Z, ZHAO Y, et al. Multi-modal knowledge graphs for recommender systems[C]//Proceedings of the 29th ACM International Conference on Information and Knowledge Management. New York: ACM, 2020: 1405-1414.
[5] BORDES A, USUNIER N, GARCIA-DURÁN A, et al. Translating embeddings for modeling multi-relational data[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems, 2013: 2787-2795.
[6] DETTMERS T, MINERVINI P, STENETORP P, et al. Convolutional 2D knowledge graph embeddings[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence and the 30th Innovative Applications of Artificial Intelligence Conference and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence. Palo Alto: AAAI, 2018: 1811-1818.
[7] GARCÍA-DURÁN A, NIEPERT M. KBlrn: end-to-end learning of knowledge base representations with latent, relational, and numerical features[C]//Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, 2018: 372-381.
[8] LIANG S, ZHU A J, ZHANG J S, et al. Hyper-node relational graph attention network for multi-modal knowledge graph completion[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2023, 19(2): 1-21.
[9] ZHANG Z Q, WANG J, YE J P, et al. Rethinking graph convolutional networks in knowledge graph completion[C]//Proceedings of the ACM Web Conference 2022. New York: ACM, 2022: 798-807.
[10] 杜雪盈, 刘名威, 沈立炜, 等. 面向链接预测的知识图谱表示学习方法综述[J]. 软件学报, 2024, 35(1): 87-117.
DU X Y, LIU M W, SHEN L W, et al. Survey on representation learning methods of knowledge graph for link prediction[J]. Journal of Software, 2024, 35(1): 87-117.
[11] SUN Z, DENG Z, NIE J Y, et al. RotatE: knowledge graph embedding by relational rotation in complex space[C]//Proceedings of the 7th International Conference on Learning Representations, New Orleans, May 6-9, 2019.
[12] EBISU T, ICHISE R. TorusE: knowledge graph embedding on a lie group[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1): 1819-1826.
[13] NICKEL M, TRESP V, KRIEGEL H P, et al. A three-way model for collective learning on multi-relational data[C]//Proceedings of the 28th International Conference on Machine Learning, 2011: 809-816.
[14] YANG B, YIH W T, HE X, et al. Embedding entities and relations for learning and inference in knowledge bases[C]// Proceedings of the 2015 International Conference on Learning Representations, San Diego, May 7-9, 2015: 1412-1420.
[15] ZHOU Z H, WANG C, FENG Y, et al. JointE: jointly utilizing 1D and 2D convolution for knowledge graph embedding[J]. Knowledge-Based Systems, 2022, 240: 108100.
[16] JIANG D, WANG R G, YANG J, et al. Kernel multi-attention neural network for knowledge graph embedding[J]. Knowledge-Based Systems, 2021, 227: 107188.
[17] ZHANG Z L, LI Z F, LIU H, et al. Multi-scale dynamic convolutional network for knowledge graph embedding[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34(5): 2335-2347.
[18] VASHISHTH S, SANYAL S, NITIN V, et al. InteractE: improving convolution-based knowledge graph embeddings by increasing feature interactions[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(3): 3009-3016.
[19] SCHLICHTKRULL M, KIPF T N, BLOEM P, et al. Modeling relational data with graph convolutional networks[C]//Proceedings of the 15th International Conference on the Semantic Web. Cham: Springer, 2018: 593-607.
[20] VASHISHTH S, SANYAL S, NITIN V, et al. Composition-based multi-relational graph convolutional networks[C]//Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Apr 26-30, 2020.
[21] YAO L, MAO C S, LUO Y. KG-BERT: BERT for knowledge graph completion[EB/OL]. [2024-02-23]. https://arxiv.org/abs/1909.03193.
[22] WANG L, ZHAO W, WEI Z Y, et al. SimKGC: simple contrastive knowledge graph completion with pre-trained language models[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2022: 4281-4294.
[23] XIE R B, LIU Z Y, LUAN H B, et al. Image-embodied knowledge representation learning[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2017: 3140-3146.
[24] WANG Z K, LI L J, LI Q D, et al. Multimodal data enhanced representation learning for knowledge graphs[C]//Proceedings of the 2019 International Joint Conference on Neural Networks. Piscataway: IEEE, 2019: 1-8.
[25] WANG M, WANG S, YANG H, et al. Is visual context really helpful for knowledge graph? A representation learning perspective[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York: ACM, 2021: 2735-2743.
[26] ZHANG Y Y, FANG Q, QIAN S S, et al. Multi-modal multi-relational feature aggregation network for medical knowledge representation learning[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York: ACM, 2020: 3956-3965.
[27] ZHAO Y, CAI X R, WU Y K, et al. MoSE: modality split and ensemble for multimodal knowledge graph completion[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2022: 10527-10536.
[28] LI X H, ZHAO X Y, XU J X, et al. IMF: interactive multimodal fusion model for link prediction[C]//Proceedings of the ACM Web Conference 2023. New York: ACM, 2023: 2572-2580.
[29] BOLLACKER K, EVANS C, PARITOSH P, et al. Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. New York: ACM, 2008: 1247-1250.
[30] SUCHANEK F M, KASNECI G, WEIKUM G. YAGO: a large ontology from Wikipedia and WordNet[J]. Journal of Web Semantics, 2008, 6(3): 203-217.
[31] AUER S, BIZER C, KOBILAROV G, et al. DBpedia: a nucleus for a web of open data[C]//Proceedings of the 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference. Berlin, Heidelberg: Springer, 2007: 722-735.
[32] SHANG C, TANG Y, HUANG J, et al. End-to-end structure- aware convolutional networks for knowledge base completion[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(1): 3060-3067.