Journal of Frontiers of Computer Science and Technology ›› 2021, Vol. 15 ›› Issue (7): 1165-1182.DOI: 10.3778/j.issn.1673-9418.2012075

• Surveys and Frontiers • Previous Articles     Next Articles

Summary of Multi-modal Sentiment Analysis Technology

LIU Jiming, ZHANG Peixiang, LIU Ying, ZHANG Weidong, FANG Jie   

  1. 1. School of Communications and Information Engineering, Xi??an University of Posts and Telecommunications, Xi??an 710121, China
    2. Center for Image and Information Processing, Xi??an University of Posts and Telecommunications, Xi??an 710121, China
    3. International Joint Research Center for Wireless Communication and Information Processing Technology of Shaanxi Province, Xi??an 710121, China
    4. Key Laboratory of Electronic Information Application Technology for Crime Scene Investigation, Ministry of Public Security, Xi??an University of Posts and Telecommunications, Xi??an 710121, China
  • Online:2021-07-01 Published:2021-07-09

多模态的情感分析技术综述

刘继明张培翔刘颖张伟东房杰   

  1. 1.西安邮电大学 通信与信息工程学院,西安 710121
    2.西安邮电大学 图像与信息处理研究所,西安 710121
    3.陕西省无线通信与信息处理技术国际合作研究中心,西安 710121
    4.西安邮电大学 电子信息现场勘验应用技术公安部重点实验室,西安 710121

Abstract:

Sentiment analysis refers to the use of computers to automatically analyze and determine the emotions that people want to express. It can play a significant role in human-computer interaction and criminal investigation and solving cases. The advancement of deep learning and traditional feature extraction algorithms provides conditions for the use of multiple modalities for sentiment analysis. Combining multiple modalities for sentiment analysis can make up for the instability and limitations of single-modal sentiment analysis, and can effectively improve accuracy. In recent years, researchers have used three modalities of facial expression information, text information, and voice information to perform sentiment analysis. This paper mainly summarizes the multi-modal sentiment analysis technology from these three modalities. Firstly, it briefly introduces the basic concepts and research status of multi-modal sentiment analysis. Secondly, it summarizes the commonly used multi-modal sentiment analysis datasets. It gives a brief description of the existing single-modal emotion analysis technology based on facial expression information, text information and voice information. Next, the modal fusion technology is introduced in detail, and the existing results of the multi-modal sentiment analysis technology are mainly described according to different modal fusion methods. Finally, it discusses the problems of multi-modal sentiment analysis and future development direction.

Key words: multi-modal, sentiment analysis, modal fusion

摘要:

情感分析是指利用计算机自动分析确定人们所要表达的情感,其在人机交互和刑侦破案等领域都能发挥重大作用。深度学习和传统特征提取算法的进步为利用多种模态进行情感分析提供了条件。结合多种模态进行情感分析可以弥补单模态情感分析的不稳定性以及局限性等缺点,能够有效提高准确度。近年来,研究者多用面部表情信息、文本信息以及语音信息三种模态进行情感分析。主要从这三种模态对多模态情感分析技术进行综述:首先对多模态情感分析的基本概念以及研究现状进行简要介绍;其次总结了常用的多模态情感分析数据集;然后分别对现有的基于面部表情信息、文本信息和语音信息的单模态情感分析技术进行简要叙述;接下来详细介绍了模态融合技术,并依据不同的模态融合方式对多模态情感分析技术的现有成果进行重点描述;最后讨论了多模态情感分析存在的问题以及未来的发展方向。

关键词: 多模态, 情感分析, 模态融合