多模态课程学习知识图谱实体预测方法研究

doi:10.3778/j.issn.1673-9418.2308085

摘要/Abstract

摘要： 现有知识图谱实体预测方法一方面只利用邻域和图结构信息增强节点信息，忽略了知识图谱之外的多模态信息对于知识图谱信息的增强；另一方面正负样本对比训练模型时负样本随机排序导致训练效果不佳，且没有额外的信息帮助负样本的训练过程。为此，提出了一种多模态课程学习知识图谱实体预测模型（MMCL）。首先把多模态信息引入知识图谱实现信息增强，利用生成对抗网络（GAN）优化多模态信息融合过程，生成器生成的样本增强知识图谱信息，同时也提升鉴别器判别三元组真伪的能力；其次利用课程学习算法根据负样本的难易程度对负样本从易到难排序，通过步调函数分层次地把排序的负样本加入到训练过程中，更有利于发挥负样本鉴别三元组真伪的效果，同时无标签学习避免了训练后期假阴性问题；多模态信息融合互相优化的鉴别器与课程学习训练模型共享参数，帮助提升负样本的训练效果。在FB15k-237和WN18RR两个数据集上进行实验，结果表明，MMCL与基线模型相比，在平均倒数排名（MRR）、Hits@1、Hits@3以及Hits@10四个性能评价指标均有明显提升，验证了所提模型的有效性和可行性。

关键词: 课程学习, 多模态, 生成对抗网络（GAN）, 负采样

Abstract: On the one hand, the existing knowledge graph entity prediction methods only use the neighborhood and graph structure information to enhance the node information, and ignore the multi-modal information outside the knowledge graph to enhance the knowledge graph information. On the other hand, when comparing positive and negative samples to train the model, the negative sample random ordering results in poor training effect, and there is no additional information to help the training process of negative samples. Therefore, a multi-modal curriculum learning knowledge graph entity prediction model (MMCL) is proposed. Firstly, multi-modal information is introduced into the knowledge graph to achieve information enhancement, and the multi-modal information fusion process is optimized using generative adversarial network (GAN). The samples generated by the generator enhance the knowledge graph information, and at the same time improve the discriminator??s ability to distinguish the truth and falsity of triples. Secondly, the course learning algorithm is used to sort the negative samples from easy to difficult according to the difficulty of the negative samples. By adding the sorted negative samples into the training process hierarchically through the pace function, it is more beneficial to playing the effect of negative samples in identifying the truth and falseness of triples, and at the same time, no label learning avoids the false-negative problem in the late training period. The discriminators share parameters with course learning training models to help improve the training effect of negative samples. Experiments are conducted on two datasets, FB15k-237 and WN18RR. The results show that compared with the baseline model, MMCL is significantly improved in mean reciprocal rank (MRR), Hits@1, Hits@3 and Hits@10. The validity and feasibility of the proposed model are verified.

Key words: curriculum learning, multi-modal, generative adversarial network (GAN), negative sample

许智宏, 郝雪梅, 王利琴, 董永峰, 王旭. 多模态课程学习知识图谱实体预测方法研究[J]. 计算机科学与探索, 2024, 18(6): 1590-1599.

XU Zhihong, HAO Xuemei, WANG Liqin, DONG Yongfeng, WANG Xu. Research on Knowledge Graph Entity Prediction Method of Multi-modal Curriculum Learning[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(6): 1590-1599.

参考文献

[1] WANG X, ZOU L, WANG C K, et al. Research on knowledge graph data management: a survey[J]. Journal of Software, 2019, 30(7): 2139-2174.
[2] LIU H X, WU Y X, YANG Y M. Analogical inference for multi-relational embeddings[C]//Proceedings of the 34th International Conference on Machine Learning, Sydney, Aug 6-11, 2017: 2168-2178.
[3] ZHANG Z Q, WANG J, YE J P, et al. Rethinking graph convolutional networks in knowledge graph completion[C]//Proceedings of the ACM Web Conference 2022, New York, Apr 25-29,?2022. New York: ACM, 2022: 798-807.
[4] WANG H W, REN H Y, LESKOVEC J. Relational message passing for knowledge graph completion[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, New York, Aug 23-27, 2021. New York: ACM, 2021: 1697-1707.
[5] VASHISHTH S, SANYAL S, NITIN N, et al. Composition-based multi-relational graph convolutional neworks[C]//Pro-ceedings of the 8th International Conference on Learning Representations, Addis Ababa, Apr 26-30, 2020.
[6] NATHANI D, CHAUHAN J, SHARMA C, et al. Learning attention based embeddings for relation prediction in knowledge graphs[C]//Proceedings of the 57th Conference of the Association for Computational Linguistics, Stroudsburg, Jul 28-Aug 2, 2019. Stroudsburg: ACL, 2019: 4710-4723.
[7] FU T S, JING Z, CHEN W. A comprehensive overview of knowledge graph completion[J]. Knowledge-Based Systems, 2022, 255: 109597.
[8] BORDES A, USUNIER N, GARCIA-DURAN A, et al. Translating embeddings for modeling multi-relational data[C]//Pro-ceedings of the 26th International Conference on Neural Information Processing Systems, Nevada, Dec 5-8, 2013: 2787-2795.
[9] WANG Z, ZHANG J, FENG J, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the 28th AAAI Conference on Artificial Intelligence, Quebec, Jul 27-31, 2014. Menlo Park: AAAI, 2014: 1112-1119.
[10] LIN Y, LIU Z, SUN M, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the 29th AAAI Conference on Artificial Intelligence, Texas, Jan 25-30, 2015. Menlo Park: AAAI, 2015: 2181-2187.
[11] JI G, HE S, XU L, et al. Knowledge graph embedding via dynamic mapping matrix[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, Jul 26-31, 2015. Stroudsburg: ACL, 2015: 687-696.
[12] YANY B, HE X. Embedding entities and relations for learning and inference in knowledge bases[C]//Proceedings of the 2015 International Conference on Learning Representations, San Diego, May 7-9, 2015: 1412-1420.
[13] PUJARA J, MIAO H, GETOOR L, et al. Ontology-aware partitioning for knowledge graph identification[C]//Proceedings of the 2013 Conference on Automated Knowledge Base Construction, San Francisco, Oct 27-28, 2013: 19-24.
[14] AN B, CHEN B, HAN X P, et al. Accurate text-enhanced knowledge graph representation learning[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Jun 1-6, 2018. Stroudsburg: ACL, 2018: 745-755.
[15] YAO L, MAO C S, LUO Y. KG-BERT: BERT for knowledge graph completion[J]. arXiv:1909.03193, 2019.
[16] XIE R B, LIU Z Y, LUAN H B, et al. Image-embodied know-ledge representation learning[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Aug 19-25, 2017: 3140-3146.
[17] LIU Y, LI H, GARCI?A-DURA?N A, et al. MMKG: multi-modal knowledge graphs[C]//Proceedings of the 16th International Conference on Semantic Web, Berlin, Sep 23-27, 2019: 459-474.
[18] LIANG S, ZHU A J, ZHANG J S, et al. Hyper-node relational graph attention network for multi-modal knowledge graph completion[J]. ACM Transactions on Multimedia Com-puting, Communications, and Applications, 2023, 19(2): 1-21.
[19] ZHANG Y C, ZHANG W. Knowledge graph completion with pre-trained multi-modal transformer and twins negative sampling[J]. arXiv:2209.07084, 2022.
[20] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lin-guistics: Human Language Technologies, Minneapolis, Jun 2-7, 2019. Stroudsburg: ACL, 2019: 4171-4186.
[21] REIMERS N, GUREVYCH I. Sentence-BERT: sentence embeddings using BERT-networks[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China, Nov 3-7, 2019: 3980-3990.
[22] CHEN L Y, LI Z, WANG Y J, et al. MMEA: entity alignment for multimodal knowledge graph[C]//Proceedings of the 13th International Conference on Knowledge Science, Engineering and Management, Hangzhou, Aug 28-30, 2017: 134-147.
[23] MICHAEL S, THOMAS N K, PETER B, et al. Modeling relational data with graph convolutional networks[C]//Proceedings of the 18th Extended Semantic Web Conference. Cham: Springer, 2018: 593-607.
[24] TROUILLON T, WELBL J, RIEDEL S, et al. Complex embeddings for simple link prediction[C]//Proceedings of the 33rd International Conference on Machine Learning, New York, Jun 19-24, 2016: 2071-2080.
[25] LI Z, LIU H, ZHANG Z, et al. Learning knowledge graph embedding with heterogeneous relation attention networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(8): 3961-3973.
[26] AHRABIAN K, FEIZI A, SALEHI Y, et al. Structure aware negative sampling in knowledge graphs[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2020: 6093-6101.
[27] TANG Z W, PEI S C, ZHANG Z, et al. Positive-unlabeled learning with adversarial data augmentation for knowledge graph completion[C]//Proceedings of the 31st International Joint Conference on Artificial Intelligence, Vienna, Jul 23-29, 2022: 2248-2254.