Recommendation Method Based on Cross-Modal Graph Masking and Feature Enhancement

doi:10.3778/j.issn.1673-9418.2409043

Abstract

Abstract: In order to solve the problems of data noise and insufficient expression of multimodal information in traditional multimodal recommendation systems, a recommendation method based on cross-modal graph masking and feature enhancement is proposed in this paper. This method firstly uses a CLIP pre-trained model to extract semantically consistent text and visual features, and establishes an item-item graph based on similarity of item??s multimodal features to provide semantically rich contextual information for item-representation. Secondly, the cross-modal graph mask reconstruction method is designed which fully uses inter-modality features information to reduce data noise and enhance features, and then the graph convolutional network is used to learn user and item interaction information. In the final prediction, user preference scores are used to accurately capture the user??s preferences for the target item. By weighting the preference scores from different aspects, it can more effectively complete recommended tasks. Finally, a multi-task training strategy is employed to jointly optimize model learning, simultaneously considering multimodal information and interactive information, effectively improving the accuracy of multimodal recommendations. Experimental results on the four public datasets Women Clothing, Men Clothing, Beauty, Toys & Games show that the recommendation accuracy of the proposed model is superior to benchmark algorithms in terms of Recall and normalized discounted cumulative gain (NDCG) metrics, with an improvement of 8.04% on the Recall@20 metric and 7.31% on the NDCG@20 metric, respectively, compared with the most advanced baseline.

Key words: graph representation learning, multimodal recommendation system, graph neural networks, self-supervised learning

摘要： 提出了一种基于跨模态图掩码和特征增强的推荐方法，旨在减少传统多模态推荐系统中的数据噪声和解决多模态信息表达不足问题。该方法使用CLIP预训练模型提取语义一致性的文本和视觉特征，根据项目模态特征相似性建立项目-项目图，为项目表示提供语义丰富的上下文信息；设计跨模态图掩码重建方法，该方法充分利用模态间特征信息，减少数据噪声并增强特征，之后使用图卷积网络学习用户和项目交互信息；在最终预测时，使用用户偏好分数更准确地捕捉用户对目标项目的偏好，通过不同方面的偏好分数加权计算，更为有效地完成推荐任务。使用多任务联合学习框架进行模型训练，同时兼顾了多模态信息和交互信息，有效提升了多模态推荐的准确性。在Women Clothing、Men Clothing、Beauty、Toys & Games四个公共数据集上的实验结果表明，所提出的方法在召回率（Recall）和归一化折损累计增益（NDCG）指标上均优于现有的基准算法，在Recall@20指标上，比最先进的基线提升8.04%，在NDCG@20指标上，比最先进的基线提升7.31%。

关键词: 图表示学习, 多模态推荐系统, 图神经网络, 自监督学习

JING Li, ZHENG Gonghao, LI Xiaohan, YU Mengyuan. Recommendation Method Based on Cross-Modal Graph Masking and Feature Enhancement[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(9): 2470-2478.

景丽, 郑公浩, 李晓涵, 蔚梦媛. 基于跨模态图掩码和特征增强的推荐方法[J]. 计算机科学与探索, 2025, 19(9): 2470-2478.

References

[1] LYU Y Z, YIN H Z, LIU J, et al. Reliable recommendation with review-level explanations[C]//Proceedings of the 2021 IEEE 37th International Conference on Data Engineering. Piscataway: IEEE, 2021: 1548-1558.
[2] HE R N, MCAULEY J. VBPR: visual Bayesian personalized ranking from implicit feedback[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2016, 30(1): 144-150.
[3] RENDLE S, FREUDENTHALER C, GANTNER Z, et al. BPR: Bayesian personalized ranking from implicit feedback[C]//Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence. Arlington: AUAI Press, 2009: 452-461.
[4] ZHANG Y F, AI Q Y, CHEN X, et al. Joint representation learning for top-N recommendation with heterogeneous information sources[C]//Proceedings of the 2017 ACM Conference on Information and Knowledge Management. New York: ACM, 2017: 1449-1458.
[5] CHEN X, CHEN H X, XU H T, et al. Personalized fashion recommendation with visual explanations based on multimodal attention network: towards visually explainable recommendation[C]//Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2019: 765-774.
[6] KIM T, LEE Y C, SHIN K, et al. MARIO: modality-aware attention and modality-preserving decoders for multimedia recommendation[C]//Proceedings of the 31st ACM International Conference on Information & Knowledge Management. New York: ACM, 2022: 993-1002.
[7] LIU Z Y, ZHOU J. Introduction to graph neural networks[M]. Cham: Springer, 2020.
[8] WANG X, HE X N, WANG M, et al. Neural graph collaborative filtering[C]//Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2019: 165-174.
[9] HE X N, DENG K, WANG X, et al. LightGCN: simplifying and powering graph convolution network for recommendation[C]//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2020: 639-648.
[10] WEI Y W, WANG X, NIE L Q, et al. MMGCN: multi-modal graph convolution network for personalized recommendation of micro-video[C]//Proceedings of the 27th ACM International Conference on Multimedia. New York: ACM, 2019: 1437-1445.
[11] WEI Y W, WANG X, NIE L Q, et al. Graph-refined convolutional network for multimedia recommendation with implicit feedback[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York: ACM, 2020: 3541-3549.
[12] ZHANG J H, ZHU Y Q, LIU Q, et al. Mining latent structures for multimedia recommendation[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York: ACM, 2021: 3872-3880.
[13] ZHOU X, SHEN Z Q. A tale of two graphs: freezing and denoising graph structures for multimodal recommendation[C]//Proceedings of the 31st ACM International Conference on Multimedia. New York: ACM, 2023: 935-943.
[14] 孙秀娟, 孙福振, 李鹏程, 等. 融合掩码自编码器的自适应增强序列推荐[J]. 计算机科学与探索, 2024, 18(12): 3324-3334.
SUN X J, SUN F Z, LI P C, et al. Fusion of masked autoencoder for adaptive augmentation sequential recommendation[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(12): 3324-3334.
[15] 金海波, 冯雨静. 社交知识感知网络推荐算法[J]. 计算机科学与探索, 2025, 19(4): 1105-1114.
JIN H B, FENG Y J. Social knowledge-aware network recommendation algorithm[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(4): 1105-1114.
[16] YU P H, TAN Z Y, LU G M, et al. Multi-view graph convolutional network for multimedia recommendation[C]//Proceedings of the 31st ACM International Conference on Multimedia. New York: ACM, 2023: 6576-6585.
[17] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the 38th International Conference on Machine Learning, 2021: 8748-8763.
[18] MCPHERSON M, SMITH-LOVIN L, COOK J M. Birds of a feather: homophily in social networks[J]. Annual Review of Sociology, 2001, 27: 415-444.
[19] CHEN Y, WU L, ZAKI M. Iterative deep graph learning for graph neural networks: better and robust node embeddings[C]//Advances in Neural Information Processing Systems 33, 2020: 19314-19326.
[20] CHEN J, FANG H, SAAD Y. Fast approximate kNN graph construction for high dimensional data via recursive Lanczos bisection[J]. Journal of Machine Learning Research, 2009, 10: 1989-2012.
[21] VELICKOVIC P, CUCURULL G, CASANOVA A, et al. Graph attention networks[EB/OL]. [2024-07-16]. https://arxiv.org/abs/1710.10903.
[22] GILMER J, SCHOENHOLZ S S, RILEY P F, et al. Neural message passing for quantum chemistry[C]//Proceedings of the 34th International Conference on Machine Learning, 2017: 1263-1272.
[23] YANG Z. XLNet: generalized autoregressive pretraining for language understanding[EB/OL]. [2024-07-16]. https://arxiv.org/abs/1906.08237.
[24] HOU Z Y, LIU X, CEN Y K, et al. GraphMAE: self-supervised masked graph autoencoders[C]//Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. New York: ACM, 2022: 594-604.
[25] KIM Y, KIM T, SHIN W Y, et al. MONET: modality-embracing graph convolutional network and target-aware attention for multimedia recommendation[C]//Proceedings of the 17th ACM International Conference on Web Search and Data Mining. New York: ACM, 2024: 332-340.
[26] MCAULEY J, TARGETT C, SHI Q F, et al. Image-based recommendations on styles and substitutes[C]//Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2015: 43-52.
[27] ZHANG J H, ZHU Y Q, LIU Q, et al. Latent structure mining with contrastive modality fusion for multimedia recommendation[J]. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(9): 9154-9167.