双流时间域信息交互的微表情识别卷积网络

doi:10.3778/j.issn.1673-9418.2011039

计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (4): 950-958.DOI: 10.3778/j.issn.1673-9418.2011039

• 图形图像 • 上一篇

双流时间域信息交互的微表情识别卷积网络

朱伟杰, 陈莹⁺()

江南大学轻工过程先进控制教育部重点实验室,江苏无锡 214122

收稿日期:2020-11-12 修回日期:2021-01-14 出版日期:2022-04-01 发布日期:2021-02-04
通讯作者: + E-mail: chenying@jiangnan.edu.cn
作者简介:朱伟杰（1996—）,男,安徽马鞍山人,硕士研究生,主要研究方向为深度学习、模式识别。
陈莹（1976—）,女,浙江丽水人,博士,教授,博士生导师,CCF会员,主要研究方向为模式识别、信息融合。
基金资助:
国家自然科学基金(61573168)

Micro-expression Recognition Convolutional Network for Dual-stream Temporal-Domain Information Interaction

ZHU Weijie, CHEN Ying⁺()

Key Laboratory of Advanced Process Control for Light Industry, Ministry of Education, Jiangnan University, Wuxi, Jiangsu 214122, China

Received:2020-11-12 Revised:2021-01-14 Online:2022-04-01 Published:2021-02-04
About author:ZHU Weijie, born in 1996, M.S. candidate. His research interests include deep learning and pattern recognition.
CHEN Ying, born in 1976, Ph.D., professor, Ph.D. supervisor, member of CCF. Her research interests include pattern recognition and information fusion.
Supported by:
National Natural Science Foundation of China(61573168)

摘要/Abstract

摘要：

目前主流的深度学习方法用于微表情识别存在实验数据非常稀缺的问题,导致神经网络在学习的过程中知识获取有限进而难以提升精度。针对目前存在的问题,提出双流网络时间域信息交互的微表情识别方法,构建了双流时间域信息交互卷积神经网络（DSTICNN）,网络对微表情序列进行处理,进而实现微表情自动识别。该算法通过改进深度互学习策略引导网络学习同一图像序列的不同时间域信息,来提高最终的识别率。算法基于不同时间尺度构建DSTICNN32和DSTICNN64,在训练阶段改良了深度互学习的损失函数。同时,在两流网络接近决策层的特征图加上了均方差损失,最终由交叉熵损失、JS散度损失和均方差损失来共同监督训练,使得两流网络互相学习加强,提高各自预测样本的能力。算法在CASME Ⅱ、SMIC数据库上进行了实验,结果表明该算法能有效提高微表情识别率,CASME Ⅱ数据库上提高6.83个百分点,SMIC数据库上提高1.65个百分点,整体算法优于现有算法。

关键词: 深度学习, 双流时间域信息, 交互, 微表情识别, 深度互学习

Abstract:

The current mainstream deep learning methods used for micro-expression recognition have the problem of very scarce experimental data, which leads to the limited knowledge acquisition of neural networks in the learning process and it is difficult to improve the accuracy. The dual-stream network temporal-domain information interaction micro-expression recognition method is proposed, and a dual-stream temporal-domain information inter-action neural convolution network (dual scale temporal interactive convolution neural network, DSTICNN), is constructed to process the micro-expression sequence, and then realize automatic recognition of micro-expressions. The algorithm improves the final recognition rate by improving the deep mutual learning strategy to guide the network to learn different temporal domain information of the same image sequence. The algorithm builds DSTICNN32 and DSTICNN64 based on different temporal scales, and improves the loss function of deep mutual learning in the training phase. At the same time, mean square error loss is added to the feature maps of the two-stream network close to the decision-making layer, and finally cross-entropy loss, JS divergence loss and mean square error loss are used to jointly supervise training, so that the two-stream network learns and strengthens each other and improves their respective prediction samples ability. The algorithm is tested on CASME Ⅱ and SMIC databases, and the results show that the algorithm in this paper can effectively improve the recognition rate of micro-expressions. The recognition rate is improved by 6.83 percentage points on the CASME Ⅱ database and 1.65 percentage points on the SMIC database. The overall algorithm is better than existing algorithms.

Key words: deep learning, dual-stream temporal-domain information, interaction, micro-expression recognition, deep mutual learning

中图分类号:

TP391

朱伟杰, 陈莹. 双流时间域信息交互的微表情识别卷积网络[J]. 计算机科学与探索, 2022, 16(4): 950-958.

ZHU Weijie, CHEN Ying. Micro-expression Recognition Convolutional Network for Dual-stream Temporal-Domain Information Interaction[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 950-958.

图/表 9

参考文献 28

[1]	HAGGARD E A. Micromomentary facial expressions as in-dicators of ego mechanisms in psychotherapy[J]. Methods of Research in Psychotherapy, 1966: 154-165.
[2]	张爱梅, 徐杨. 注意力分层双线性池化残差网络的表情识别[J]. 计算机工程与应用, 2020, 56(23):161-166.
	ZHANG A M, XU Y. Attention hierarchical bilinear pooling residual network for expression recognition[J]. Computer En-gineering and Applications, 2020, 56(23):161-166.
[3]	PFISTER T, LI X B, ZHAO G Y, et al. Recognising spon-taneous facial micro-expressions[C]// Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Nov 6-13, 2011. Washington: IEEE Computer Society, 2011: 1449-1456.
[4]	WANG Y D, SEE J, PHAN R C W, et al. LBP with six inter-section points: reducing redundant information in LBP-TOP for micro-expression recognition[C]// LNCS 9003: Proceedings of the 12th Asian Conference on Computer Vision, Singapore, Nov 1-5, 2014. Cham: Springer, 2014: 525-537.
[5]	张轩阁, 田彦涛, 郭艳君. 基于光流与LBP-TOP特征结合的微表情识别[J]. 吉林大学学报(信息科学版), 2015, 33(5):516-523.
	ZHANG X G, TIAN Y T, GUO Y J. Micro-expression reco-gnition based on feature combination of optical flow and LBP-TOP[J]. Journal of Jilin University (Information Sci-ence Edition), 2015, 33(5):516-523.
[6]	LIU Y J, ZHANG J K, YAN W J, et al. A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4):299-310. DOI URL
[7]	BEN X Y, ZHANG P, YAN R, et al. Gait recognition and micro-expression recognition based on maximum margin pro-jection with tensor representation[J]. Neural Computing and Applications, 2016, 27(8):2629-2646. DOI URL
[8]	LIONG S T, GAN Y S, YAU W C, et al. OFF-ApexNet on micro-expression recognition system[J]. Signal Processing: Image Communication, 2018, 74:129-139. DOI URL
[9]	刘汝涵, 徐丹. 视频放大和深度学习在微表情识别任务上的应用[J]. 计算机辅助设计与图形学学报, 2019, 31(9):1535-1541.
	LIU R H, XU D. Video amplification and deep learning in micro-expression recognition[J]. Journal of Computer-Aided Design & Computer Graphics, 2019, 31(9):1535-1541.
[10]	RATHI P, SHARMA R, SINGAL P, et al. Micro-expression recognition using 3D-CNN layering[M]// AI-Powered IoT for COVID-19. Boca Raton: CRC Press, 2020.
[11]	LI J, WANG Y, SEE J, et al. Micro-expression recognition based on 3D flow convolutional neural network[J]. Pattern Analysis and Applications, 2019, 22(4):1331-1339. DOI URL
[12]	REDDY S P T, KARRI S T, DUBEY S R, et al. Spon-taneous facial micro-expression recognition using 3D spatio-temporal convolutional neural networks[C]// Proceedings of the 2019 International Joint Conference on Neural Networks, Budapest, Jul 14-19, 2019. Piscataway: IEEE, 2019: 1-8.
[13]	JIA X T, BEN X Y, YUAN H, et al. Macro-to-micro trans-formation model for micro-expression recognition[J]. Jour-nal of Computational Science, 2018, 25:289-297.
[14]	XIA B, WANG W K, WANG S F, et al. Learning from macro-expression: a micro-expression recognition frame-work[C]// Proceedings of the 28th ACM International Con-ference on Multimedia, Seattle, Oct 12-16, 2020. New York: ACM, 2020: 2936-2944.
[15]	ZHANG Y, XIANG T, HOSPEDALES T M, et al. Deep mutual learning[C]// Proceedings of the 2018 IEEE Con-ference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 4320-4328.
[16]	TRAN D, BOURDEY L D, FERGUS R, et al. Learning spatiotemporal features with 3D convolutional networks[C]// Proceedings of the 2015 IEEE International Confe-rence on Computer Vision, Santiago, Dec 7-13, 2015. Was-hington: IEEE Computer Society, 2015: 4489-4497.
[17]	SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethin-king the inception architecture for computer vision[C]// Pro-ceedings of the 2016 IEEE Computer Vision and Pattern Recognition, Las Vegas, Jun 26-Jul 1, 2016. Washington: IEEE Computer Society, 2016: 2818-2826.
[18]	YAN W J, LI X B, WANG S J, et al. CASME II: an improved spontaneous micro-expression database and the baseline evaluation[J]. PLoS One, 2014, 9(1):e86041. DOI URL
[19]	LI X B, PFISTER T, HUANG X H, et al. A spontaneous micro-expression database: inducement, collection and base-line[C]// Proceedings of the 10th IEEE International Con-ference and Workshops on Automatic Face and Gesture Re-cognition, Shanghai, Apr 22-26, 2013. Washington: IEEE Computer Society, 2013: 1-6.
[20]	ZHAO G Y, PIETIKÄINEN M. Dynamic texture reco-gnition using local binary patterns with an application to facial expressions[J]. IEEE Transactions on Pattern Ana-lysis and Machine Intelligence, 2007, 29(6):915-928.
[21]	QUANG N V, CHUN J, TOKUYAMA T. CapsuleNet for micro-expression recognition[C]// Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Ges-ture Recognition, Lille, May 14-18, 2019. Piscataway: IEEE, 2019: 1-7.
[22]	TAKALKAR M A, XU M. Image based facial micro-expression recognition using deep learning on small data-sets[C]// Proceedings of the 2017 International Conference on Digital Image Computing: Techniques and Applications, Sydney, Nov 29-Dec 1, 2017. Piscataway: IEEE, 2017: 1-7.
[23]	HU C L, JIANG D B, ZOU H T, et al. Multi-task micro-expression recognition combining deep and handcrafted features[C]// Proceedings of the 24th International Confe-rence on Pattern Recognition, Beijing, Aug 20-24, 2018. Washington: IEEE Computer Society, 2018: 946-951.
[24]	LIU Y C, DU H M, ZHENG L, et al. A neural micro-expression recognizer[C]// Proceedings of the 14th IEEE In-ternational Conference on Automatic Face & Gesture Reco-gnition, Lille, May 14-18, 2019. Piscataway: IEEE, 2019: 1-4.
[25]	YU J H, ZHANG C Y, SONG Y, et al. ICE-GAN: identity-aware and capsule-enhanced GAN for micro-expression reco-gnition and synjournal[J]. arXiv: 2005. 04370v2, 2020.
[26]	ZHOU L, MAO Q R, XUE L Y. Dual-inception network for crossdatabase microexpression recognition[C]// Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, Lille, May 14-18, 2019. Piscataway: IEEE, 2019: 1-5.
[27]	XIA Z Q, HONG X P, GAO X Y, et al. Corrections to “spatiotemporal recurrent convolutional networks for re-cognizing spontaneous micro-expressions”[J]. IEEE Tran-sactions on Multimedia, 2020, 22(4):1111.
[28]	XIA Z Q, PENG W, KHOR H Q, et al. Revealing the invisible with model and data shrinking for composite-database micro-expression recognition[J]. IEEE Transactions on Image Processing, 2020, 29:8590-8605. DOI URL

Layer	DSTICNN32	Stride	DSTICNN64	Stride
Input	32×112×112		64×112×112
Conv1	(4×3×3,16)	Ss:2,Ts:2	(8×3×3, 16)	Ss:2,Ts:4
Pool1	1×2×2	Ss:2,Ts:1	1×2×2	Ss:2,Ts:1
Conv2	(3×3×3, 32)	Ss:1,Ts:1	(3×3×3, 32)	Ss:1,Ts:1
Pool2	2×2×2	Ss:2,Ts:2	2×2×2	Ss:2,Ts:2
Conv3	(3×3×3, 64)	Ss:1,Ts:1	(3×3×3, 64)	Ss:1,Ts:1
Pool3	2×2×2	Ss:2,Ts:2	2×2×2	Ss:2,Ts:2
Conv4	(4×3×3, 128)	Ss:1,Ts:1	(4×3×3, 128)	Ss:1,Ts:1
Pool4	1×2×2	Ss:1,Ts:2	1×2×2	Ss:1,Ts:2

Layer	DSTICNN32	Stride	DSTICNN64	Stride
Input	32×112×112		64×112×112
Conv1	(4×3×3,16)	Ss:2,Ts:2	(8×3×3, 16)	Ss:2,Ts:4
Pool1	1×2×2	Ss:2,Ts:1	1×2×2	Ss:2,Ts:1
Conv2	(3×3×3, 32)	Ss:1,Ts:1	(3×3×3, 32)	Ss:1,Ts:1
Pool2	2×2×2	Ss:2,Ts:2	2×2×2	Ss:2,Ts:2
Conv3	(3×3×3, 64)	Ss:1,Ts:1	(3×3×3, 64)	Ss:1,Ts:1
Pool3	2×2×2	Ss:2,Ts:2	2×2×2	Ss:2,Ts:2
Conv4	(4×3×3, 128)	Ss:1,Ts:1	(4×3×3, 128)	Ss:1,Ts:1
Pool4	1×2×2	Ss:1,Ts:2	1×2×2	Ss:1,Ts:2

方法	SMIC	CASME Ⅱ
LBP-TOP^[20]	52.80	63.41
Quang等^[21]	59.80	70.10
Takalkar等^[22]	—	75.57
Hu等^[23]	65.10	66.20
Liu等^[24]	75.30	82.00
ICE-GAN^[25]	79.10	86.80
Dual-Inception^[26]	61.49	81.32
STRCN-G^[27]	72.30	80.30
Xia等^[28]	66.00	81.31
DSTICNN32_JS_MSE	81.79	83.65
DSTICNN64_JS_MSE	85.93	80.60

方法	SMIC	CASME Ⅱ
LBP-TOP^[20]	52.80	63.41
Quang等^[21]	59.80	70.10
Takalkar等^[22]	—	75.57
Hu等^[23]	65.10	66.20
Liu等^[24]	75.30	82.00
ICE-GAN^[25]	79.10	86.80
Dual-Inception^[26]	61.49	81.32
STRCN-G^[27]	72.30	80.30
Xia等^[28]	66.00	81.31
DSTICNN32_JS_MSE	81.79	83.65
DSTICNN64_JS_MSE	85.93	80.60

方法	损失函数	SMIC	CASME Ⅱ
DSTICNN32	交叉熵	68.55	69.41
DSTICNN64	交叉熵	69.34	72.87
DSTICNN32	交叉熵+KL	72.64	76.50
DSTICNN64	交叉熵+KL	72.72	76.50
DSTICNN32	交叉熵+KL+均方差	76.66	82.21
DSTICNN64	交叉熵+KL+均方差	79.13	78.78
DSTICNN32	交叉熵+JS	76.45	81.61
DSTICNN64	交叉熵+JS	81.31	78.34
DSTICNN32	交叉熵+JS+均方差	81.70	83.65
DSTICNN64	交叉熵+JS+均方差	85.93	80.60

双流时间域信息交互的微表情识别卷积网络

Micro-expression Recognition Convolutional Network for Dual-stream Temporal-Domain Information Interaction

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 28

相关文章 15

编辑推荐

Metrics

[1]	安凤平, 李晓薇, 曹翔. 权重初始化-滑动窗口CNN的医学图像分类[J]. 计算机科学与探索, 2022, 16(8): 1885-1897.
[2]	曾凡智, 许露倩, 周燕, 周月霞, 廖俊玮. 面向智慧教育的知识追踪模型研究综述[J]. 计算机科学与探索, 2022, 16(8): 1742-1763.
[3]	刘艺, 李蒙蒙, 郑奇斌, 秦伟, 任小广. 视频目标跟踪算法综述[J]. 计算机科学与探索, 2022, 16(7): 1504-1515.
[4]	赵小明, 杨轶娇, 张石清. 面向深度学习的多模态情感识别研究进展[J]. 计算机科学与探索, 2022, 16(7): 1479-1503.
[5]	夏鸿斌, 肖奕飞, 刘渊. 融合自注意力机制的长文本生成对抗网络模型[J]. 计算机科学与探索, 2022, 16(7): 1603-1610.
[6]	孙方伟, 李承阳, 谢永强, 李忠博, 杨才东, 齐锦. 深度学习应用于遮挡目标检测算法综述[J]. 计算机科学与探索, 2022, 16(6): 1243-1259.
[7]	刘雅芬, 郑艺峰, 江铃燚, 李国和, 张文杰. 深度半监督学习中伪标签方法综述[J]. 计算机科学与探索, 2022, 16(6): 1279-1290.
[8]	程卫月, 张雪琴, 林克正, 李骜. 融合全局与局部特征的深度卷积神经网络算法[J]. 计算机科学与探索, 2022, 16(5): 1146-1154.
[9]	钟梦圆, 姜麟. 超分辨率图像重建算法综述[J]. 计算机科学与探索, 2022, 16(5): 972-990.
[10]	裴利沈, 赵雪专. 群体行为识别深度学习方法研究综述[J]. 计算机科学与探索, 2022, 16(4): 775-790.
[11]	许嘉, 韦婷婷, 于戈, 黄欣悦, 吕品. 题目难度评估方法研究综述[J]. 计算机科学与探索, 2022, 16(4): 734-759.
[12]	包广斌, 李港乐, 王国雄. 面向多模态情感分析的双模态交互注意力[J]. 计算机科学与探索, 2022, 16(4): 909-916.
[13]	邬开俊, 黄涛, 王迪聪, 白晨帅, 陶小苗. 视频异常检测技术研究进展[J]. 计算机科学与探索, 2022, 16(3): 529-540.
[14]	刘利平, 孙建, 高世妍. 单图像盲去模糊方法概述[J]. 计算机科学与探索, 2022, 16(3): 552-564.
[15]	刘颖, 郭莹莹, 房杰, 范九伦, 郝羽, 刘继明. 深度学习跨模态图文检索研究综述[J]. 计算机科学与探索, 2022, 16(3): 489-511.