基于超图双重注意力机制的多模态谣言检测模型

doi:10.3778/j.issn.1673-9418.2411048

摘要/Abstract

摘要： 针对现有图神经网络模型在多模态新闻谣言检测任务中难以有效建模谣言传播过程中的复杂多节点关系、且对节点自身所含的多模态特征信息挖掘不足的问题，提出一种融合图文及情感等多模态特征的节点嵌入的超图网络谣言检测模型。该方法融合包含图文以及情感等多模态特征，得到信息更充分的初始节点嵌入；将融合后的节点特征导入构建的多重关系的超图结构中，刻画超过成对关系的群体交互模式；引入带门控机制双重注意力机制，自适应给关键节点和超边分配权重并突出高权重要素；得到的节点高层表示作为分类输入，最终提升新闻关系和传播模式的识别能力。通过实验验证，所提出的方法在三个公开数据集上取得了显著的性能提升，准确率分别为94.46%、97.36%和93.86%。此外，该方法在早期谣言检测任务中也表现出一定的有效性，展示了其在多模态信息融合及复杂关系建模方面的优势。

关键词: 图神经网络, 谣言检测, 多模态融合, 注意力机制

Abstract: To address the existing graph neural network models?? limitations in effectively modeling complex multi-node relationships during rumor propagation and insufficient exploration of multi-modal feature information within nodes for multi-modal fake news detection, this paper proposes a hypergraph-based rumor detection model incorporating multi-modal node embeddings. Firstly, the method fuses multi-modal features, including text, image, and sentiment to obtain more informative initial node embeddings. Then, it feeds the fused node representations into the constructed multi-relational hypergraph to capture group-level interaction patterns beyond pairwise relations. Finally, it introduces a gated dual-attention mechanism that adaptively assigns weights to key nodes and hyperedges and highlights high-importance factors. The resulting high-level node representations are used as classifier inputs, ultimately improving the identification of news relationships and propagation patterns. Experimental evaluations demonstrate that the proposed method achieves significant performance improvements on three publicly available datasets, obtaining accuracies of 94.46%, 97.36% and 93.86%, respectively. Furthermore, the method shows promising effectiveness in early rumor detection tasks, highlighting its advantages in multi-modal information fusion and complex relationship modeling.

Key words: graph neural networks, rumor detection, multi-modal fusion, attention mechanism

王安然, 袁得嵛, 潘语泉, 贾源. 基于超图双重注意力机制的多模态谣言检测模型[J]. 计算机科学与探索, 2025, 19(11): 3033-3045.

WANG Anran, YUAN Deyu, PAN Yuquan, JIA Yuan. Multi-modal Rumor Detection Model Based on Dual Attention Mechanism on Hypergraphs[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(11): 3033-3045.

参考文献

[1] VOSOUGHI S, ROY D, ARAL S. The spread of true and false news online[J]. Science, 2018, 359(6380): 1146-1151.
[2] NICKERSON R S. Confirmation bias: a ubiquitous phenomenon in many guises[J]. Review of General Psychology, 1998, 2(2): 175-220.
[3] CINELLI M, DE FRANCISCI MORALES G, GALEAZZI A, et al. The echo chamber effect on social media[J]. Proceedings of the National Academy of Sciences of the USA, 2021, 118(9): e2023301118.
[4] SHU K, SLIVA A, WANG S H, et al. Fake news detection on social media: a data mining perspective[J]. ACM SIGKDD Explorations Newsletter, 2017, 19(1): 22-36.
[5] 刘华玲, 陈尚辉, 曹世杰, 等. 基于多模态学习的虚假新闻检测研究[J]. 计算机科学与探索, 2023, 17(9): 2015-2029.
LIU H L, CHEN S H, CAO S J, et al. Survey of fake news detection with multi-model learning[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(9): 2015-2029.
[6] NEWMAN M L, PENNEBAKER J W, BERRY D S, et al. Lying words: predicting deception from linguistic styles[J]. Personality and Social Psychology Bulletin, 2003, 29(5): 665-675.
[7] SHRESTHA A, SPEZZANO F. Textual characteristics of news title and body to detect fake news: a reproducibility study[C]//Advances in Information Retrieval: the 43rd European Conference on IR Research. Cham: Springer, 2021: 120- 133.
[8] QIAN F, GONG C Y, SHARMA K, et al. Neural user response generator: fake news detection with collective user intelligence[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence. Palo Alto: AAAI, 2018: 3834-3840.
[9] DAI Y, SHOU L J, GONG M, et al. Graph fusion network for text classification[J]. Knowledge-Based Systems, 2022, 236: 107659.
[10] HAN Y, KARUNASEKERA S, LECKIE C. Graph neural networks with continual learning for fake news detection from social media[EB/OL]. [2024-09-14]. https://arxiv.org/abs/2007.03316.
[11] KHATTAR D, GOUD J S, GUPTA M, et al. MVAE: multimodal variational autoencoder for fake news detection[C]//Proceedings of the 2019 World Wide Web Conference. New York: ACM, 2019: 2915-2921.
[12] SINGHAL S, SHAH R R, CHAKRABORTY T, et al. SpotFake: a multi-modal framework for fake news detection[C]// Proceedings of the 5th IEEE International Conference on Multimedia Big Data. Piscataway: IEEE, 2019: 39-47.
[13] SINGHAL S, KABRA A, SHARMA M, et al. SpotFake+: a multimodal framework for fake news detection via transfer learning (student abstract)[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(10): 13915-13916.
[14] ZHOU X Y, WU J D, ZAFARANI R. Similarity-aware multi-modal fake news detection[C]//Advances in Knowledge Discovery and Data Mining - the 24th Pacific-Asia Conference. Cham: Springer, 2020: 354-367.
[15] PENG L W, JIAN S L, LI D S, et al. MRML: multimodal rumor detection by deep metric learning[C]//Proceedings of the 2023 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway: IEEE, 2023: 1-5.
[16] 王震宇, 朱学芳. 基于多模态Transformer的虚假新闻检测研究[J]. 情报学报, 2023, 42(12): 1477-1486.
WANG Z Y, ZHU X F. Research on fake news detection based on multimodal transformer[J]. Journal of the China Society for Scientific and Technical Information, 2023, 42(12): 1477-1486.
[17] DING K Z, WANG J L, LI J D, et al. Graph prototypical networks for few-shot learning on attributed networks[C]//Proceedings of the 29th ACM International Conference on Information & Knowledge Management. New York: ACM, 2020: 295-304.
[18] DING K Z, XU Z, TONG H H, et al. Data augmentation for deep graph learning: a survey[J]. ACM SIGKDD Explorations Newsletter, 2022, 24(2): 61-77.
[19] HAMILTON W L, YING R, LESKOVEC J. Inductive representation learning on large graphs[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 1025-1035.
[20] VELI?KOVI? P, CUCURULL G, CASANOVA A, et al. Graph attention networks[EB/OL]. [2024-09-14]. https://arxiv. org/abs/1710.10903.
[21] WANG X, JI H Y, SHI C, et al. Heterogeneous graph attention network[C]//Proceedings of the 2019 World Wide Web Conference. New York: ACM, 2019: 2022-2032.
[22] SHU K, CUI L M, WANG S H, et al. dEFEND: explainable fake news detection[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM, 2019: 395-405.
[23] YANG Z C, YANG D Y, DYER C, et al. Hierarchical attention networks for document classification[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: ACL, 2016: 1480-1489.
[24] CHENG L, GUO R C, SHU K, et al. Causal understanding of fake news dissemination on social media[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. New York: ACM, 2021: 148-157.
[25] MONTI F, FRASCA F, EYNARD D, et al. Fake news detection on social media using geometric deep learning[EB/OL]. [2024-09-15]. https://arxiv.org/abs/1902.06673.
[26] BIAN T, XIAO X, XU T Y, et al. Rumor detection on social media with bi-directional graph convolutional networks[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(1): 549-556.
[27] KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. [2024-09-15]. https://arxiv.org/abs/1609.02907.
[28] DOU Y T, SHU K, XIA C Y, et al. User preference-aware fake news detection[C]//Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2021: 2051- 2055.
[29] SHU K, MAHUDESWARAN D, WANG S H, et al. FakeNewsNet: a data repository with news content, social context, and spatiotemporal information for studying fake news on social media[J]. Big Data, 2020, 8(3): 171-188.
[30] REN Y X, ZHANG J W. Fake news detection on news-oriented heterogeneous information networks through hierarchical graph attention[C]//Proceedings of the 2021 International Joint Conference on Neural Networks. Piscataway: IEEE, 2021: 1-8.
[31] FENG Y F, YOU H X, ZHANG Z Z, et al. Hypergraph neural networks[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(1): 3558-3565.
[32] BAI S, ZHANG F H, TORR P H S. Hypergraph convolution and hypergraph attention[J]. Pattern Recognition, 2021, 110: 107637.
[33] DING K Z, WANG J L, LI J D, et al. Be more with less: hypergraph attention networks for inductive text classification[EB/OL]. [2024-09-15]. https://arxiv.org/abs/2011.00387.
[34] WU J Y, HOOI B. DECOR: degree-corrected social graph refinement for fake news detection[C]//Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. New York: ACM, 2023: 2582-2593.
[35] ZHOU D Y, HUANG J Y, SCH?LKOPF B. Learning with hypergraphs: clustering, classification, and embedding[C]//Advances in Neural Information Processing Systems 19, 2007: 1601-1608.
[36] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2024-09-15]. https://arxiv.org/abs/1810.04805.
[37] ZHANG H, WU C R, ZHANG Z Y, et al. ResNeSt: split-attention networks[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 2735-2745.
[38] ZHANG X Y, CAO J, LI X R, et al. Mining dual emotion for fake news detection[C]//Proceedings of the Web Conference 2021. New York: ACM, 2021: 3465-3476.
[39] GUO Q J, KANG Z, TIAN L, et al. TieFake: title-text similarity and emotion-aware fake news detection[C]//Proceedings of the 2023 International Joint Conference on Neural Networks. Piscataway: IEEE, 2023: 1-7.
[40] JEONG U, DING K Z, CHENG L, et al. Nothing stands alone: relational fake news detection with hypergraph neural networks[C]//Proceedings of the 2022 IEEE International Conference on Big Data. Piscataway: IEEE, 2023: 596-605.
[41] HONNIBAL M, MONTANI I. spaCy 2: natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing[EB/OL]. [2024-09-15]. https://github.com/explosion/spaCy.
[42] MA J, GAO W, MITRA P, et al. Detecting rumors from microblogs with recurrent neural networks[C]//Proceedings of the 25th International Joint Conference on Artificial Intelligence. Palo Alto: AAAI, 2016: 3818-3824.