面向多模态情感分析的双模态交互注意力

doi:10.3778/j.issn.1673-9418.2105071

计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (4): 909-916.DOI: 10.3778/j.issn.1673-9418.2105071

面向多模态情感分析的双模态交互注意力

包广斌, 李港乐⁺(), 王国雄

兰州理工大学计算机与通信学院,兰州 730050

收稿日期:2021-05-19 修回日期:2021-08-03 出版日期:2022-04-01 发布日期:2021-08-05
通讯作者: + E-mail: 1450316716@qq.com
作者简介:包广斌（1975—）,男,甘肃兰州人,博士,副教授,主要研究方向为大数据分析、自然语言处理。
李港乐（1997—）,女,山东济宁人,硕士,主要研究方向为自然语言处理。
王国雄（1997—）,男,甘肃陇南人,硕士,主要研究方向为自然语言处理。
基金资助:
国家自然科学基金(51668043);甘肃省自然科学基金(18JR3RA156)

Bimodal Interactive Attention for Multimodal Sentiment Analysis

BAO Guangbin, LI Gangle⁺(), WANG Guoxiong

School of Computer and Communication, Lanzhou University of Technology, Lanzhou 730050, China

Received:2021-05-19 Revised:2021-08-03 Online:2022-04-01 Published:2021-08-05
About author:BAO Guangbin, born in 1975, Ph.D., associate professor. His research interests include big data analysis and natural language processing.
LI Gangle, born in 1997, M.S. Her research interest is natural language processing.
WANG Guoxiong, born in 1997, M.S. His research interest is natural language processing.
Supported by:
National Natural Science Foundation of China(51668043);Natural Science Foundation of Gansu Province(18JR3RA156)

摘要/Abstract

摘要：

针对现有多模态情感分析方法中存在情感分类准确率不高,难以有效融合多模态特征等问题,通过研究分析相邻话语之间的依赖关系和文本、语音和视频模态之间的交互作用,建立一种融合上下文和双模态交互注意力的多模态情感分析模型。该模型首先采用双向门控循环单元（BiGRU）捕获各模态中话语之间的相互依赖关系,得到各模态的上下文信息。为了学习不同模态之间的交互信息,提出了一种双模态交互注意力机制来融合两种模态的信息,并将其作为条件向量来区分各模态信息对于情感分类的重要程度;然后结合自注意力、全连接层组成多模态特征融合模块,挖掘模态内部和模态之间的关联性,获得跨模态联合特征。最后,将得到的上下文特征和跨模态联合特征进行拼接,经过一层全连接层后馈送至Softmax进行最终的情感分类。在公开的多模态情感分析数据集CMU-MOSI上对所提出的模型进行评估,实验结果表明,相比现有模型,该模型在多模态情感分类任务上的表现是有效的和先进的。

关键词: 多模态, 情感分析, 双向门控循环单元（BiGRU）, 上下文, 双模态交互注意力, 特征融合

Abstract:

Aiming at the problems of low accuracy of sentiment classification and difficulty in effectively fusing multimodal features in existing multimodal sentiment analysis methods, a multimodal sentiment analysis model combining context and bimodal interactive attention is established by analyzing the dependence between adjacent utterances and the interaction among text, audio and video modalities. Firstly, the model adopts a bidirectional gated recurrent unit (BiGRU) to capture the dependence between utterances in each modal, and the context information of each modal is obtained. In order to learn the interactive information between different modalities, a bimodal interactive attention mechanism is proposed to fuse the information of the two modalities and it is used as a condition vector to distinguish the importance of each modal for sentiment classification. Then, self-attention and fully connected layers are combined to form a multimodal feature fusion module, the correlation of information within and between modalities is mined, and cross-modal joint features are obtained. Finally, the obtained contextual features and cross-modal joint features are spliced, and then fed to Softmax for the final sentiment classification after a fully connected layer. The proposed model is evaluated on the public multimodal sentiment analysis dataset CMU-MOSI (CMU multimodal opinion-level sentiment intensity). The experimental results show that compared with the existing models, the performance of the proposed model on multimodal sentiment classification task is effective and advanced.

Key words: multimodal, sentiment analysis, bidirectional gated recurrent unit (BiGRU), context, bimodal interactive attention, feature fusion

中图分类号:

TP391

包广斌, 李港乐, 王国雄. 面向多模态情感分析的双模态交互注意力[J]. 计算机科学与探索, 2022, 16(4): 909-916.

BAO Guangbin, LI Gangle, WANG Guoxiong. Bimodal Interactive Attention for Multimodal Sentiment Analysis[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 909-916.

图/表 10

图1 文本、面部表情和语音语调对于情感分类的作用

Fig.1 Effect of text, facial expressions and voice intonation on sentiment classification

图2 融合上下文和双模态交互注意力的模型结构

Fig.2 Model structure combining context and bimodal interactive attention

图3 BiGRU结构模型图

Fig.3 BiGRU structure model diagram

图4 双模态交互注意力（Bim_Att）结构图

Fig.4 Structure diagram of bimodal interactive attention (Bim_Att)

表1 实验参数设置

Table 1 Experimental parameter setting

参数	值
词向量维度	50
BiGRU隐藏单元	300
全连接层神经元	100
Dropout	0.5
学习率	0.001
批处理	32
迭代次数	30
优化函数	Adam
损失函数	Categorical_crossentropy

表2 在MOSI数据集上的实验结果

Table 2 Experimental results on MOSI dataset %

Model	Accuracy	$F 1$ score
GME-LSTM^[9]	76.50	73.40
MARN^[17]	77.10	77.00
TFN^[11]	77.10	77.90
Dialogue-RNN^[14]	79.80	79.48
BC-LSTM^[13]	80.30	—
Multilogue-Net^[15]	81.19	80.10
Con-BIAM	81.91	85.40

表2 在MOSI数据集上的实验结果

Table 2 Experimental results on MOSI dataset %

Model	Accuracy	$F 1$ score
GME-LSTM^[9]	76.50	73.40
MARN^[17]	77.10	77.00
TFN^[11]	77.10	77.90
Dialogue-RNN^[14]	79.80	79.48
BC-LSTM^[13]	80.30	—
Multilogue-Net^[15]	81.19	80.10
Con-BIAM	81.91	85.40

图5 Con-BIAM模型在MOSI数据集上的混淆矩阵

Fig.5 Con-BIAM model confusion matrix on MOSI dataset

表3 不同模型在双模态、三模态特征融合的准确率

Table 3 Accuracy of different models in bimodal and trimodal feature fusion %

Metric	Dialogue-RNN^[14]	Multilogue-Net^[15]	Con-BIAM
T+A	79.80	80.18	80.45
V+T	78.90	80.06	80.98
A+V	73.90	75.16	63.96
A+V+T	79.80	81.19	81.91

表4 不同模型在双模态、三模态特征融合的 F 1分数

Table 4 F 1scores of different models in bimodal and trimodal feature fusion %

Metric	Dialogue-RNN^[14]	Multilogue-Net^[15]	Con-BIAM
T+A	78.32	79.88	84.14
V+T	78.12	79.84	84.43
A+V	73.92	74.04	75.20
A+V+T	79.48	80.10	85.40

图6 在MOSI数据集上的对比实验

Fig.6 Comparative experiment on MOSI dataset

参考文献 21

[1]	GHOSAL D, AKHTAR M S, CHAUHAN D S, et al. Contextual inter-modal attention for multi-modal sentiment analysis[C]// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Oct 31-Nov 4, 2018. Stroudsburg: ACL, 2018: 3454-3466.
[2]	林敏鸿, 蒙祖强. 基于注意力神经网络的多模态情感分析[J]. 计算机科学, 2020, 47(S2):508-514.
	LIN M H, MENG Z Q. Multimodal sentiment analysis based on attention neural network[J]. Computer Science, 2020, 47(S2):508-514.
[3]	刘继明, 张培翔, 刘颖, 等. 多模态的情感分析技术综述[J]. 计算机科学与探索, 2021, 15(7):1165-1182.
	LIU J M, ZHANG P X, LIU Y, et al. Summary of multi-modal sentiment analysis technology[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(7):1165-1182.
[4]	何俊, 张彩庆, 李小珍, 等. 面向深度学习的多模态融合技术研究综述[J]. 计算机工程, 2020, 46(5):1-11.
	HE J, ZHANG C Q, LI X Z, et al. Survey of research on multimodal fusion technology for deep learning[J]. Computer Engineering, 2020, 46(5):1-11.
[5]	PORIA S, CAMBRIA E, HAZARIKA D, et al. Multi-level multiple attentions for contextual multimodal sentiment analysis[C]// Proceedings of the 2017 IEEE International Conference on Data Mining, New Orleans, Nov 18-21, 2017. Washington: IEEE Computer Society, 2017: 1033-1038.
[6]	KUMAR A, VEPA J. Gated mechanism for attention based multi modal sentiment analysis[C]// Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, Barcelona, May 4-8, 2020. Piscataway:IEEE, 2020: 4477-4481.
[7]	LIN Z, FENG M, SANTOS C N, et al. A structured self-attentive sentence embedding[J]. arXiv: 1703. 03130, 2017.
[8]	ZADEH A, ZELLERS R, PINCUS E, et al. Multimodal senti-ment intensity analysis in videos: facial gestures and verbal messages[J]. IEEE Intelligent Systems, 2016, 31(6):82-88. DOI URL
[9]	CHEN M H, WANG S, LIANG P P, et al. Multimodal sentiment analysis with word-level fusion and reinforcement learning[C]// Proceedings of the 19th ACM International Conference on Multimodal Interaction, Glasgow, Nov 13-17, 2017. New York: ACM, 2017: 163-171.
[10]	张亚洲, 戎璐, 宋大为, 等. 多模态情感分析研究综述[J]. 模式识别与人工智能, 2020, 33(5):426-438.
	ZHANG Y Z, RONG L, SONG D W, et al. A survey on multimodal sentiment analysis[J]. Pattern Recognition and Artificial Intelligence, 2020, 33(5):426-438.
[11]	ZADEH A, CHEN M H, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Sep 9-11, 2017. Stroudsburg: ACL, 2017: 1103-1114.
[12]	ZADEH A, LIANG P P, PORIA S, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Jul 15-20, 2018. Stroudsburg: ACL, 2018: 2236-2246.
[13]	PORIA S, CAMBRIA E, HAZARIKA D, et al. Context-dependent sentiment analysis in user-generated videos[C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Jul 30-Aug 4, 2017. Stroudsburg: ACL, 2017: 873-883.
[14]	MAJUMDER N, PORIA S, HAZARIKA D, et al. Dialogue: an attentive RNN for emotion detection in conversations[C]// Proceedings of the 2019 AAAI Conference on Artificial Intelligence, Honolulu, Jan 27-Feb 1, 2019. Palo Alto: AAAI, 2019: 6818-6825.
[15]	SHENOY A, SARDANA A. Multilogue-Net: a context aware RNN for multi-modal emotion detection and sentiment analysis in conversation[C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Seattle, Jul 5-10, 2020. Stroudsburg: ACL, 2020: 19-28.
[16]	KIM T, LEE B. Multi-attention multimodal sentiment analysis[C]// Proceedings of the 2020 International Conference on Multimedia Retrieval, Dublin, Jun 8-11, 2020. New York: ACM, 2020: 436-441.
[17]	ZADEH A, LIANG P P, PORIA S, et al. Multi-attention recurrent network for human communication comprehension[C]// Proceedings of the 2018 AAAI Conference on Artificial Intelligence, New Orleans, Feb 2-7, 2018. Palo Alto: AAAI, 2018: 5642-5649.
[18]	XI C, LU G M, YAN J J. Multimodal sentiment analysis based on multi-head attention mechanism[C]// Proceedings of the 4th International Conference on Machine Learning and Soft Computing, Haiphong City, Jan 17-19, 2020. New York: ACM, 2020: 34-39.
[19]	VERMA S, WANG J W, GE Z F, et al. Deep-HOSeq: deep higher order sequence fusion for multimodal sentiment analysis[J]. arXiv: 2010. 08218, 2020.
[20]	TACHIBANA H, UENOYAMA K, AIHARA S. Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention[C]// Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, Apr 15-20, 2018. Piscataway: IEEE, 2018: 4784-4788.
[21]	EYBEN F, WÖLLMER M, SCHULLER B W. Opensmile: the munich versatile and fast open-source audio feature extractor[C]// Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Oct 25-29, 2010. New York: ACM, 2010: 1459-1462.

面向多模态情感分析的双模态交互注意力

Bimodal Interactive Attention for Multimodal Sentiment Analysis

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 21

相关文章 15

编辑推荐

Metrics

[1]	杨知桥, 张莹, 王新杰, 张东波, 王玉. 改进U型网络在视网膜病变检测中的应用研究[J]. 计算机科学与探索, 2022, 16(8): 1877-1884.
[2]	洪惠群, 沈贵萍, 黄风华. 表情识别技术综述[J]. 计算机科学与探索, 2022, 16(8): 1764-1778.
[3]	彭豪, 李晓明. 多尺度选择金字塔网络的小样本目标检测算法[J]. 计算机科学与探索, 2022, 16(7): 1649-1660.
[4]	赵小明, 杨轶娇, 张石清. 面向深度学习的多模态情感识别研究进展[J]. 计算机科学与探索, 2022, 16(7): 1479-1503.
[5]	刘颖, 王哲, 房杰, 朱婷鸽, 李琳娜, 刘继明. 基于图文融合的多模态舆情分析[J]. 计算机科学与探索, 2022, 16(6): 1260-1278.
[6]	赵运基, 范存良, 张新良. 融合多特征和通道感知的目标跟踪算法[J]. 计算机科学与探索, 2022, 16(6): 1417-1428.
[7]	程卫月, 张雪琴, 林克正, 李骜. 融合全局与局部特征的深度卷积神经网络算法[J]. 计算机科学与探索, 2022, 16(5): 1146-1154.
[8]	赵鹏飞, 谢林柏, 彭力. 融合注意力机制的深层次小目标检测算法[J]. 计算机科学与探索, 2022, 16(4): 927-937.
[9]	王燕妮, 余丽仙. 注意力与多尺度有效融合的SSD目标检测算法[J]. 计算机科学与探索, 2022, 16(2): 438-447.
[10]	肖泽管, 陈清亮. 融合多种类型语法信息的属性级情感分析模型[J]. 计算机科学与探索, 2022, 16(2): 395-402.
[11]	钱伍, 王国中, 李国平. 改进YOLOv5的交通灯实时检测鲁棒算法[J]. 计算机科学与探索, 2022, 16(1): 231-241.
[12]	李志欣, 陈圣嘉, 周韬, 马慧芳. 协同级联网络和对抗网络的目标检测[J]. 计算机科学与探索, 2022, 16(1): 217-230.
[13]	陈璠, 彭力. 多层级重叠条纹特征融合的行人重识别[J]. 计算机科学与探索, 2021, 15(9): 1753-1761.
[14]	武家伟, 孙艳春. 融合知识图谱和深度学习方法的问诊推荐系统[J]. 计算机科学与探索, 2021, 15(8): 1432-1440.
[15]	刘继明, 张培翔, 刘颖, 张伟东, 房杰. 多模态的情感分析技术综述[J]. 计算机科学与探索, 2021, 15(7): 1165-1182.