关键词: 多模态情绪识别, 得分网络补全, 融合重构

Abstract: Human multimodal emotion recognition utilizes various heterogeneous modal data, such as language, vision, and sound, to perceive and understand human emotions. Compared to single modalities, the complementary information in multimodal data helps to understand emotions more robustly. However, in practical multimodal scenarios, incomplete or missing modal information often exists, severely hindering the understanding of multimodal features and leading to a decline in emotion recognition accuracy. To address the issue of recognition accuracy decline caused by incomplete or missing modalities, which has not been effectively handled by previous multimodal emotion recognition methods, we propose an Incomplete Multimodal Emotion Recognition method based on Fusion Generative Diffusion Models. This method aims to enhance emotion recognition accuracy by reconstructing incomplete modal data information. First, a generative diffusion model based on cross-modal conditional stochastic differential equations is constructed. In the back-diffusion process, the available modal information is converted into dynamic constraints of the drift term through learnable projection to generate incomplete modal information features. Second, a two-way collaborative optimization framework of the incomplete modal generation network and the fusion reconstruction module is constructed. The joint objective function is used to realize the gradient backpropagation interaction of generation quality and feature fusion, and the emotional semantic consistency constraint of generated features and real features is established through a hierarchical attention mechanism. Results from several dataset tests indicate that the proposed multimodal emotion recognition method achieves superior performance in various incomplete modality scenarios.

Key words: multimodal emotion recognition, score network completion, fusion reconstruction