Micro-expression Recognition Convolutional Network for Dual-stream Temporal-Domain Information Interaction

doi:10.3778/j.issn.1673-9418.2011039

Abstract

Abstract:

The current mainstream deep learning methods used for micro-expression recognition have the problem of very scarce experimental data, which leads to the limited knowledge acquisition of neural networks in the learning process and it is difficult to improve the accuracy. The dual-stream network temporal-domain information interaction micro-expression recognition method is proposed, and a dual-stream temporal-domain information inter-action neural convolution network (dual scale temporal interactive convolution neural network, DSTICNN), is constructed to process the micro-expression sequence, and then realize automatic recognition of micro-expressions. The algorithm improves the final recognition rate by improving the deep mutual learning strategy to guide the network to learn different temporal domain information of the same image sequence. The algorithm builds DSTICNN32 and DSTICNN64 based on different temporal scales, and improves the loss function of deep mutual learning in the training phase. At the same time, mean square error loss is added to the feature maps of the two-stream network close to the decision-making layer, and finally cross-entropy loss, JS divergence loss and mean square error loss are used to jointly supervise training, so that the two-stream network learns and strengthens each other and improves their respective prediction samples ability. The algorithm is tested on CASME Ⅱ and SMIC databases, and the results show that the algorithm in this paper can effectively improve the recognition rate of micro-expressions. The recognition rate is improved by 6.83 percentage points on the CASME Ⅱ database and 1.65 percentage points on the SMIC database. The overall algorithm is better than existing algorithms.

Key words: deep learning, dual-stream temporal-domain information, interaction, micro-expression recognition, deep mutual learning

摘要：

目前主流的深度学习方法用于微表情识别存在实验数据非常稀缺的问题,导致神经网络在学习的过程中知识获取有限进而难以提升精度。针对目前存在的问题,提出双流网络时间域信息交互的微表情识别方法,构建了双流时间域信息交互卷积神经网络（DSTICNN）,网络对微表情序列进行处理,进而实现微表情自动识别。该算法通过改进深度互学习策略引导网络学习同一图像序列的不同时间域信息,来提高最终的识别率。算法基于不同时间尺度构建DSTICNN32和DSTICNN64,在训练阶段改良了深度互学习的损失函数。同时,在两流网络接近决策层的特征图加上了均方差损失,最终由交叉熵损失、JS散度损失和均方差损失来共同监督训练,使得两流网络互相学习加强,提高各自预测样本的能力。算法在CASME Ⅱ、SMIC数据库上进行了实验,结果表明该算法能有效提高微表情识别率,CASME Ⅱ数据库上提高6.83个百分点,SMIC数据库上提高1.65个百分点,整体算法优于现有算法。

关键词: 深度学习, 双流时间域信息, 交互, 微表情识别, 深度互学习

CLC Number:

TP391

ZHU Weijie, CHEN Ying. Micro-expression Recognition Convolutional Network for Dual-stream Temporal-Domain Information Interaction[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 950-958.

朱伟杰, 陈莹. 双流时间域信息交互的微表情识别卷积网络[J]. 计算机科学与探索, 2022, 16(4): 950-958.

Figures/Tables 9

References 28

[1]	HAGGARD E A. Micromomentary facial expressions as in-dicators of ego mechanisms in psychotherapy[J]. Methods of Research in Psychotherapy, 1966: 154-165.
[2]	张爱梅, 徐杨. 注意力分层双线性池化残差网络的表情识别[J]. 计算机工程与应用, 2020, 56(23):161-166.
	ZHANG A M, XU Y. Attention hierarchical bilinear pooling residual network for expression recognition[J]. Computer En-gineering and Applications, 2020, 56(23):161-166.
[3]	PFISTER T, LI X B, ZHAO G Y, et al. Recognising spon-taneous facial micro-expressions[C]// Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Nov 6-13, 2011. Washington: IEEE Computer Society, 2011: 1449-1456.
[4]	WANG Y D, SEE J, PHAN R C W, et al. LBP with six inter-section points: reducing redundant information in LBP-TOP for micro-expression recognition[C]// LNCS 9003: Proceedings of the 12th Asian Conference on Computer Vision, Singapore, Nov 1-5, 2014. Cham: Springer, 2014: 525-537.
[5]	张轩阁, 田彦涛, 郭艳君. 基于光流与LBP-TOP特征结合的微表情识别[J]. 吉林大学学报(信息科学版), 2015, 33(5):516-523.
	ZHANG X G, TIAN Y T, GUO Y J. Micro-expression reco-gnition based on feature combination of optical flow and LBP-TOP[J]. Journal of Jilin University (Information Sci-ence Edition), 2015, 33(5):516-523.
[6]	LIU Y J, ZHANG J K, YAN W J, et al. A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4):299-310. DOI URL
[7]	BEN X Y, ZHANG P, YAN R, et al. Gait recognition and micro-expression recognition based on maximum margin pro-jection with tensor representation[J]. Neural Computing and Applications, 2016, 27(8):2629-2646. DOI URL
[8]	LIONG S T, GAN Y S, YAU W C, et al. OFF-ApexNet on micro-expression recognition system[J]. Signal Processing: Image Communication, 2018, 74:129-139. DOI URL
[9]	刘汝涵, 徐丹. 视频放大和深度学习在微表情识别任务上的应用[J]. 计算机辅助设计与图形学学报, 2019, 31(9):1535-1541.
	LIU R H, XU D. Video amplification and deep learning in micro-expression recognition[J]. Journal of Computer-Aided Design & Computer Graphics, 2019, 31(9):1535-1541.
[10]	RATHI P, SHARMA R, SINGAL P, et al. Micro-expression recognition using 3D-CNN layering[M]// AI-Powered IoT for COVID-19. Boca Raton: CRC Press, 2020.
[11]	LI J, WANG Y, SEE J, et al. Micro-expression recognition based on 3D flow convolutional neural network[J]. Pattern Analysis and Applications, 2019, 22(4):1331-1339. DOI URL
[12]	REDDY S P T, KARRI S T, DUBEY S R, et al. Spon-taneous facial micro-expression recognition using 3D spatio-temporal convolutional neural networks[C]// Proceedings of the 2019 International Joint Conference on Neural Networks, Budapest, Jul 14-19, 2019. Piscataway: IEEE, 2019: 1-8.
[13]	JIA X T, BEN X Y, YUAN H, et al. Macro-to-micro trans-formation model for micro-expression recognition[J]. Jour-nal of Computational Science, 2018, 25:289-297.
[14]	XIA B, WANG W K, WANG S F, et al. Learning from macro-expression: a micro-expression recognition frame-work[C]// Proceedings of the 28th ACM International Con-ference on Multimedia, Seattle, Oct 12-16, 2020. New York: ACM, 2020: 2936-2944.
[15]	ZHANG Y, XIANG T, HOSPEDALES T M, et al. Deep mutual learning[C]// Proceedings of the 2018 IEEE Con-ference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 4320-4328.
[16]	TRAN D, BOURDEY L D, FERGUS R, et al. Learning spatiotemporal features with 3D convolutional networks[C]// Proceedings of the 2015 IEEE International Confe-rence on Computer Vision, Santiago, Dec 7-13, 2015. Was-hington: IEEE Computer Society, 2015: 4489-4497.
[17]	SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethin-king the inception architecture for computer vision[C]// Pro-ceedings of the 2016 IEEE Computer Vision and Pattern Recognition, Las Vegas, Jun 26-Jul 1, 2016. Washington: IEEE Computer Society, 2016: 2818-2826.
[18]	YAN W J, LI X B, WANG S J, et al. CASME II: an improved spontaneous micro-expression database and the baseline evaluation[J]. PLoS One, 2014, 9(1):e86041. DOI URL
[19]	LI X B, PFISTER T, HUANG X H, et al. A spontaneous micro-expression database: inducement, collection and base-line[C]// Proceedings of the 10th IEEE International Con-ference and Workshops on Automatic Face and Gesture Re-cognition, Shanghai, Apr 22-26, 2013. Washington: IEEE Computer Society, 2013: 1-6.
[20]	ZHAO G Y, PIETIKÄINEN M. Dynamic texture reco-gnition using local binary patterns with an application to facial expressions[J]. IEEE Transactions on Pattern Ana-lysis and Machine Intelligence, 2007, 29(6):915-928.
[21]	QUANG N V, CHUN J, TOKUYAMA T. CapsuleNet for micro-expression recognition[C]// Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Ges-ture Recognition, Lille, May 14-18, 2019. Piscataway: IEEE, 2019: 1-7.
[22]	TAKALKAR M A, XU M. Image based facial micro-expression recognition using deep learning on small data-sets[C]// Proceedings of the 2017 International Conference on Digital Image Computing: Techniques and Applications, Sydney, Nov 29-Dec 1, 2017. Piscataway: IEEE, 2017: 1-7.
[23]	HU C L, JIANG D B, ZOU H T, et al. Multi-task micro-expression recognition combining deep and handcrafted features[C]// Proceedings of the 24th International Confe-rence on Pattern Recognition, Beijing, Aug 20-24, 2018. Washington: IEEE Computer Society, 2018: 946-951.
[24]	LIU Y C, DU H M, ZHENG L, et al. A neural micro-expression recognizer[C]// Proceedings of the 14th IEEE In-ternational Conference on Automatic Face & Gesture Reco-gnition, Lille, May 14-18, 2019. Piscataway: IEEE, 2019: 1-4.
[25]	YU J H, ZHANG C Y, SONG Y, et al. ICE-GAN: identity-aware and capsule-enhanced GAN for micro-expression reco-gnition and synjournal[J]. arXiv: 2005. 04370v2, 2020.
[26]	ZHOU L, MAO Q R, XUE L Y. Dual-inception network for crossdatabase microexpression recognition[C]// Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, Lille, May 14-18, 2019. Piscataway: IEEE, 2019: 1-5.
[27]	XIA Z Q, HONG X P, GAO X Y, et al. Corrections to “spatiotemporal recurrent convolutional networks for re-cognizing spontaneous micro-expressions”[J]. IEEE Tran-sactions on Multimedia, 2020, 22(4):1111.
[28]	XIA Z Q, PENG W, KHOR H Q, et al. Revealing the invisible with model and data shrinking for composite-database micro-expression recognition[J]. IEEE Transactions on Image Processing, 2020, 29:8590-8605. DOI URL

Layer	DSTICNN32	Stride	DSTICNN64	Stride
Input	32×112×112		64×112×112
Conv1	(4×3×3,16)	Ss:2,Ts:2	(8×3×3, 16)	Ss:2,Ts:4
Pool1	1×2×2	Ss:2,Ts:1	1×2×2	Ss:2,Ts:1
Conv2	(3×3×3, 32)	Ss:1,Ts:1	(3×3×3, 32)	Ss:1,Ts:1
Pool2	2×2×2	Ss:2,Ts:2	2×2×2	Ss:2,Ts:2
Conv3	(3×3×3, 64)	Ss:1,Ts:1	(3×3×3, 64)	Ss:1,Ts:1
Pool3	2×2×2	Ss:2,Ts:2	2×2×2	Ss:2,Ts:2
Conv4	(4×3×3, 128)	Ss:1,Ts:1	(4×3×3, 128)	Ss:1,Ts:1
Pool4	1×2×2	Ss:1,Ts:2	1×2×2	Ss:1,Ts:2

Layer	DSTICNN32	Stride	DSTICNN64	Stride
Input	32×112×112		64×112×112
Conv1	(4×3×3,16)	Ss:2,Ts:2	(8×3×3, 16)	Ss:2,Ts:4
Pool1	1×2×2	Ss:2,Ts:1	1×2×2	Ss:2,Ts:1
Conv2	(3×3×3, 32)	Ss:1,Ts:1	(3×3×3, 32)	Ss:1,Ts:1
Pool2	2×2×2	Ss:2,Ts:2	2×2×2	Ss:2,Ts:2
Conv3	(3×3×3, 64)	Ss:1,Ts:1	(3×3×3, 64)	Ss:1,Ts:1
Pool3	2×2×2	Ss:2,Ts:2	2×2×2	Ss:2,Ts:2
Conv4	(4×3×3, 128)	Ss:1,Ts:1	(4×3×3, 128)	Ss:1,Ts:1
Pool4	1×2×2	Ss:1,Ts:2	1×2×2	Ss:1,Ts:2

方法	SMIC	CASME Ⅱ
LBP-TOP^[20]	52.80	63.41
Quang等^[21]	59.80	70.10
Takalkar等^[22]	—	75.57
Hu等^[23]	65.10	66.20
Liu等^[24]	75.30	82.00
ICE-GAN^[25]	79.10	86.80
Dual-Inception^[26]	61.49	81.32
STRCN-G^[27]	72.30	80.30
Xia等^[28]	66.00	81.31
DSTICNN32_JS_MSE	81.79	83.65
DSTICNN64_JS_MSE	85.93	80.60

方法	SMIC	CASME Ⅱ
LBP-TOP^[20]	52.80	63.41
Quang等^[21]	59.80	70.10
Takalkar等^[22]	—	75.57
Hu等^[23]	65.10	66.20
Liu等^[24]	75.30	82.00
ICE-GAN^[25]	79.10	86.80
Dual-Inception^[26]	61.49	81.32
STRCN-G^[27]	72.30	80.30
Xia等^[28]	66.00	81.31
DSTICNN32_JS_MSE	81.79	83.65
DSTICNN64_JS_MSE	85.93	80.60

方法	损失函数	SMIC	CASME Ⅱ
DSTICNN32	交叉熵	68.55	69.41
DSTICNN64	交叉熵	69.34	72.87
DSTICNN32	交叉熵+KL	72.64	76.50
DSTICNN64	交叉熵+KL	72.72	76.50
DSTICNN32	交叉熵+KL+均方差	76.66	82.21
DSTICNN64	交叉熵+KL+均方差	79.13	78.78
DSTICNN32	交叉熵+JS	76.45	81.61
DSTICNN64	交叉熵+JS	81.31	78.34
DSTICNN32	交叉熵+JS+均方差	81.70	83.65
DSTICNN64	交叉熵+JS+均方差	85.93	80.60