计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (4): 950-958.DOI: 10.3778/j.issn.1673-9418.2011039

• 图形图像 • 上一篇    

双流时间域信息交互的微表情识别卷积网络

朱伟杰, 陈莹+()   

  1. 江南大学 轻工过程先进控制教育部重点实验室,江苏 无锡 214122
  • 收稿日期:2020-11-12 修回日期:2021-01-14 出版日期:2022-04-01 发布日期:2021-02-04
  • 通讯作者: + E-mail: chenying@jiangnan.edu.cn
  • 作者简介:朱伟杰(1996—),男,安徽马鞍山人,硕士研究生,主要研究方向为深度学习、模式识别。
    陈莹(1976—),女,浙江丽水人,博士,教授,博士生导师,CCF会员,主要研究方向为模式识别、信息融合。
  • 基金资助:
    国家自然科学基金(61573168)

Micro-expression Recognition Convolutional Network for Dual-stream Temporal-Domain Information Interaction

ZHU Weijie, CHEN Ying+()   

  1. Key Laboratory of Advanced Process Control for Light Industry, Ministry of Education, Jiangnan University, Wuxi, Jiangsu 214122, China
  • Received:2020-11-12 Revised:2021-01-14 Online:2022-04-01 Published:2021-02-04
  • About author:ZHU Weijie, born in 1996, M.S. candidate. His research interests include deep learning and pattern recognition.
    CHEN Ying, born in 1976, Ph.D., professor, Ph.D. supervisor, member of CCF. Her research interests include pattern recognition and information fusion.
  • Supported by:
    National Natural Science Foundation of China(61573168)

摘要:

目前主流的深度学习方法用于微表情识别存在实验数据非常稀缺的问题,导致神经网络在学习的过程中知识获取有限进而难以提升精度。针对目前存在的问题,提出双流网络时间域信息交互的微表情识别方法,构建了双流时间域信息交互卷积神经网络(DSTICNN),网络对微表情序列进行处理,进而实现微表情自动识别。该算法通过改进深度互学习策略引导网络学习同一图像序列的不同时间域信息,来提高最终的识别率。算法基于不同时间尺度构建DSTICNN32和DSTICNN64,在训练阶段改良了深度互学习的损失函数。同时,在两流网络接近决策层的特征图加上了均方差损失,最终由交叉熵损失、JS散度损失和均方差损失来共同监督训练,使得两流网络互相学习加强,提高各自预测样本的能力。算法在CASME Ⅱ、SMIC数据库上进行了实验,结果表明该算法能有效提高微表情识别率,CASME Ⅱ数据库上提高6.83个百分点,SMIC数据库上提高1.65个百分点,整体算法优于现有算法。

关键词: 深度学习, 双流时间域信息, 交互, 微表情识别, 深度互学习

Abstract:

The current mainstream deep learning methods used for micro-expression recognition have the problem of very scarce experimental data, which leads to the limited knowledge acquisition of neural networks in the learning process and it is difficult to improve the accuracy. The dual-stream network temporal-domain information interaction micro-expression recognition method is proposed, and a dual-stream temporal-domain information inter-action neural convolution network (dual scale temporal interactive convolution neural network, DSTICNN), is constructed to process the micro-expression sequence, and then realize automatic recognition of micro-expressions. The algorithm improves the final recognition rate by improving the deep mutual learning strategy to guide the network to learn different temporal domain information of the same image sequence. The algorithm builds DSTICNN32 and DSTICNN64 based on different temporal scales, and improves the loss function of deep mutual learning in the training phase. At the same time, mean square error loss is added to the feature maps of the two-stream network close to the decision-making layer, and finally cross-entropy loss, JS divergence loss and mean square error loss are used to jointly supervise training, so that the two-stream network learns and strengthens each other and improves their respective prediction samples ability. The algorithm is tested on CASME Ⅱ and SMIC databases, and the results show that the algorithm in this paper can effectively improve the recognition rate of micro-expressions. The recognition rate is improved by 6.83 percentage points on the CASME Ⅱ database and 1.65 percentage points on the SMIC database. The overall algorithm is better than existing algorithms.

Key words: deep learning, dual-stream temporal-domain information, interaction, micro-expression recognition, deep mutual learning

中图分类号: