Temporal Multimodal Sentiment Analysis with Composite Cross Modal Interaction Network

doi:10.3778/j.issn.1673-9418.2311004

Abstract

Abstract: To address the issues of insufficient modal fusion and weak interactivity caused by semantic feature differences between different modalities in multimodal emotion analysis, a temporal multimodal sentiment analysis model for composite cross modal interaction network (CCIN-SA) is constructed by studying and analyzing the potential correlations between different modalities. The model first uses a bidirectional gated loop unit and a multi-head attention mechanism to extract temporal features of text, visual, and speech modalities with contextual semantic information. Then, a cross modal attention interaction layer is designed to continuously strengthen the target mode using low order signals from auxiliary modes, enabling the target mode to learn information from auxiliary modes and capture potential adaptability between modes. Then it inputs the enhanced features into the composite feature fusion layer, further captures the similarity between different modalities through condition vectors, enhances the correlation degree of important features, and mines deeper level interactivity between modalities. Finally, using a multi-head attention mechanism, the composite cross modal enhanced features are concatenated and fused with low order signals to increase the weight of important features within the modality, preserve the unique feature information of the initial modality, and perform the final emotion classification task on the obtained multimodal fused features. The model evaluation is conducted on the CMU-MOSI and CMU-MOSEI datasets, and the results show that the model is improved in accuracy and F1 metrics compared with other existing models. It can be seen that the CCIN-SA model can effectively explore the correlation between different modalities and make more accurate emotional judgments.

Key words: cross modal interaction, attention mechanism, feature fusion, composite fusion layer, multimodal emotional analysis

摘要： 针对多模态情感分析中存在的不同模态间语义特征差异性导致模态融合不充分、交互性弱等问题，通过研究分析不同模态之间存在的潜在关联性，搭建一种基于复合跨模态交互网络的时序多模态情感分析（CCIN-SA）模型。该模型首先使用双向门控循环单元和多头注意力机制提取具有上下文语义信息的文本、视觉和语音模态时序特征；然后，设计跨模态注意力交互层，利用辅助模态的低阶信号不断强化目标模态，使得目标模态学习到辅助模态的信息，捕获模态间的潜在适应性；再将增强后的特征输入到复合特征融合层，通过条件向量进一步捕获不同模态间的相似性，增强重要特征的关联程度，挖掘模态间更深层次的交互性；最后，利用多头注意力机制将复合跨模态强化后的特征与低阶信号做拼接融合，提高模态内部重要特征的权重，保留初始模态独有的特征信息，将得到的多模态融合特征进行最终的情感分类任务。在CMU-MOSI和CMU-MOSEI数据集上进行模型评估，结果表明，CCIN-SA模型相比其他现有模型在准确率和F1指标上均有提高，能够有效挖掘不同模态间的关联性，做出更加准确的情感判断。

关键词: 跨模态交互, 注意力机制, 特征融合, 复合融合层, 多模态情感分析

YANG Li, ZHONG Junhong, ZHANG Yun, SONG Xinyu. Temporal Multimodal Sentiment Analysis with Composite Cross Modal Interaction Network[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(5): 1318-1327.

杨力, 钟俊弘, 张赟, 宋欣渝. 基于复合跨模态交互网络的时序多模态情感分析[J]. 计算机科学与探索, 2024, 18(5): 1318-1327.

References

[1] HONG M, JUNG J J. Multi-sided recommendation based on social tensor factorization[J]. Information Sciences, 2018, 447: 140-156.
[2] 张亚洲, 戎璐, 宋大为, 等. 多模态情感分析研究综述[J]. 模式识别与人工智能, 2020, 33(5): 426-438.
ZHANG Y Z, RONG L, SONG D W, et al. A survey on multimodal sentiment analysis[J]. Pattern Recognition and Artificial Intelligence, 2020, 33(5): 426-438.
[3] PORIA S, CAMBRIA E, HAZARIKA D, et al. Multi-level multiple attentions for contextual multimodal sentiment analysis[C]//Proceedings of the 2017 IEEE International Conference on Data Mining, New Orleans, Nov 18-21, 2017. Washington: IEEE Computer Society, 2017: 1033-1038.
[4] 刘继明, 张培翔, 刘颖, 等. 多模态的情感分析技术综述 [J]. 计算机科学与探索, 2021, 15(7): 1165-1182.
LIU J M, ZHANG P X, LIU Y, et al. Summary of multi-modal sentiment analysis technology[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(7): 1165-1182.
[5] FU Z W, LIU F, XU Q, et al. NHFNET: a non-homogeneous fusion network for multimodal sentiment analysis[C]//Proceedings of the 2022 IEEE International Conference on Multimedia and Expo. Piscataway: IEEE, 2022: 1-6.
[6] LIANG B, SU H, GUI L, et al. Aspect-based sentiment analy-sis via affective knowledge enhanced graph convolutional networks[J]. Knowledge-Based Systems, 2022, 235: 107643.
[7] ZHANG L, WANG S, LIU B. Deep learning for sentiment analysis: a survey[J]. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2018, 8(4): e1253.
[8] 谭荧, 张进, 夏立新. 社交媒体情境下的情感分析研究综述[J]. 数据分析与知识发现, 2020, 4(1): 1-11.
TAN Y, ZHANG J, XIA L X. A survey of sentiment analysis on social media[J]. Data Analysis and Knowledge Discovery, 2020, 4(1): 1-11.
[9] 刘路路, 杨燕, 王杰. ABAFN: 面向多模态的方面级情感分析模型[J]. 计算机工程与应用, 2022, 58(10): 193-199.
LIU L L, YANG Y, WANG J. ABAFN: aspect-based sentiment analysis model for multimodal[J]. Computer Engineering and Applications, 2022, 58(10): 193-199.
[10] HOU M, TANG J, ZHANG J, et al. Deep multimodal multi-linear fusion with high-order polynomial pooling[C]//Advances in Neural Information Processing Systems 32, Vancouver, Dec 8-14, 2019: 12156-12166.
[11] ZADEH A, CHEN M H, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Sep 9-11, 2017. Stroudsburg: ACL, 2017: 1103-1114.
[12] LIU Z, SHEN Y, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[C]//Proceedings of the 2018 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2018: 2247-2256.
[13] ZADEH A, LIANG P P, MAZUMDER N, et al. Memory fusion network for multi-view sequential learning[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence, the 30th Innovative Applications of Artificial Intelligence, and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, Feb 2-7, 2018. Menlo Park: AAAI, 2018: 5634-5641.
[14] TSAI Y H H, LIANG P P, ZADEH A, et al. Learning factorized multimodal representations[J]. arXiv:1806.06176v1, 2018.
[15] HAZARIKA D, ZIMMERMANN R, PORIA S, et al. MISA: modality-invariant and -specific representations for multimodal sentiment analysis[J]. arXiv:2005.03545, 2020.
[16] RAHMAN W, HASAN M K, LEE S, et al. Integrating multimodal information in large pretrained transformers[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2020: 2359-2369.
[17] YU W M, XU H, YUAN Z Q, et al. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis[C]//Proceedings of the 35th AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2021: 10790-10797.
[18] TSAI Y H, BAI S J, LINAG P P, et al. Multimodal transformer for unaligned multimodal language sequences[C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Jul 28-Aug 2, 2019. Stroudsburg: ACL, 2019: 6558-6569.
[19] TANG D, QIN B, LIU T. Document modeling with gated recurrent neural network for sentiment classification[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Sep 17-21, 2015. Stroudsburg: ACL, 2015: 1422-1432.
[20] ZADEH A, LIANG P P, PORIA S, et al. Multi-attention recurrent network for human communication comprehension[C]//Proceedings of the 2018 32nd AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2018: 5642-5649.
[21] DEGOTTEX G, KANE J, DRUGMAN T, et al. COVAREP—a collaborative voice analysis repository for speech technologies[C]//Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway: IEEE, 2014: 960-964.
[22] KIM Y. Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Oct 25- 29, 2014. Stroudsburg: ACL, 2014: 1746-1751.
[23] 马亚雄. 基于语音和面部表情的情感识别方法研究[D]. 武汉: 华中科技大学, 2021.
MA Y X. Research on emotion recognition based on speech and facial expressions[D]. Wuhan: Huazhong University of Science and Technology, 2021.
[24] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2016: 770-778.
[25] ZADEH A, ZELLERS R, PINCUS E, et al. Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages[J]. IEEE Intelligent Systems, 2016, 31(6): 82-88.
[26] ZADEH A, LIANG P, PORIA S, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Jul 15-20, 2018. Stroudsburg: ACL, 2018: 2236-2246.
[27] YANG Z, DAI Z, YANG Y, et al. XLNet: generalized auto-regressive pretraining for language understanding[C]//Advances in Neural Information Processing Systems 32, Vancouver, Dec?8-14, 2019: 5753-5763.