Deep Residual Expression Recognition Network to Enhance Inter-class Discrimination

doi:10.3778/j.issn.1673-9418.2011042

Abstract

Abstract:

Deep facial expression recognition is a challenging task for neural networks to apply in pattern reco-gnition. Compared with face recognition tasks such as identity authentication and feature point recognition, there are a lot of redundant information in facial expression recognition task. To achieve good results, more accurate class-ification is needed. Most researches focus on the generalization and network structure of data and ignore the inter-class relationship of data. In this paper, a deep residual expression recognition network RMRnet (recall matrix distinguished residual net) based on class analysis is proposed. First, the data are fed to the backbone network(Resnet18) to obtain the confusion matrix, after that, the recall matrix is used to analyze the relationship between classes. Then, the branches of network structure are designed based on the inter-class relationship to distinguish the strong-related classes, while the rest classes are designed as the supplementary branch to balance the weak-related class. In the end, by adding those branches to the corresponding position of backbone network, RMRnet is constituted. Compared with baseline method and advanced methods in recent years, experimental results in the popular large-scale database show that the proposed method is more effective than the baseline method and has str-ong competitiveness among a group of advanced methods.

Key words: facial expression recognition, neural network, confusion matrix, inter-class association

摘要：

深度人脸表情识别是神经网络应用于模式识别上一项极具挑战性的任务。相对于身份认证和特征点识别等人脸识别任务，表情识别任务中存在着大量的冗余信息，要得到好的效果，需要更精确的分类。多数研究关注点在数据的泛化性和网络结构上，而忽视了数据的类间关系。提出了一种基于类间分析的深度残差表情识别网络RMRnet。首先，将数据通过骨干网络Resnet18得到混淆矩阵，进一步得到召回率矩阵分析类间关系;然后，凭借类间关系设计网络结构分支，进一步区分强联系类，设计补充支路平衡弱联系类;最后，将分支添加到骨干网络的相应位置，得到RMRnet网络模型。在流行的大型数据库上，与基准方法和近年来的先进方法的对比实验结果表明，提出的方法相较于基准方法效果良好，在一众先进方法中也有很强的竞争力。

关键词: 表情识别, 神经网络, 混淆矩阵, 类间关联

CLC Number:

TP391.41

HUANG Hao, GE Hongwei. Deep Residual Expression Recognition Network to Enhance Inter-class Discrimination[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(8): 1842-1849.

黄浩, 葛洪伟. 强化类间区分的深度残差表情识别网络[J]. 计算机科学与探索, 2022, 16(8): 1842-1849.

Figures/Tables 9

Fig.1 General pipeline of deep facial expression recognition system

Fig.2 A [[3×3, 64],[3×3, 64]]×2 residual block

Fig.3 Resnet18 experiment of balanced dataset (accuracy is 58.6%)

Fig.4 Resnet18 experiment with codistribution of original dataset (accuracy is 65.6%)

Fig.5 Pipeline of RMRnet network

Fig.6 Data flow diagram

Fig.7 Predicted value of RMRnet on RAF-DB

Fig.8 Predicted value of Resnet18 on RAF-DB

Table 1 Comparative experiment on AffectNet, RAF-DB, Ferplus

AffectNet		RAF-DB		Ferplus
Approach	Acc/%	Approach	Acc/%	Approach	Acc/%
DLP-CNN^[19]	54.47	DLP-CNN^[19]	74.20	Baseline^[21]	83.10
pACNN^[4]	55.33	EAU-Net^[1]	81.83	TFE-JL^[25]	84.30
EAU-Net^[1]	58.91	DeepExp3D^[24]	82.06	VGG13-PLD^[16]	84.99
Baseline^[21]	54.52	Baseline^[21]	82.88	SHCNN^[26]	86.54
IPA2LT^[23]	56.51	pACNN^[4]	83.05	ESR-9^[27]	87.15
RMRnet	58.43	RMRnet	86.14	RMRnet	87.26

References 27

[1]	DENG J, PANG G, ZhANG Z, et al. cGAN based facial expression recognition for human-robot interaction[J]. IEEE Access, 2019, 7: 9848-9859.
[2]	DING H, SRICHARAN K, CHELLAPPA R. ExprGAN: facial expression editing with controllable expression inten- sity[C]// Proceedings of the 32nd AAAI Conference on Artificial Intelligence, the 30th Innovative Applications of Artificial Intelligence, and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, Feb 2-7, 2018. Menlo Park: AAAI, 2018: 6781-6788.
[3]	YANG H, CIFTCI U, YIN L. Facial expression recognition by de-expression residue learning[C]// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 2168-2177.
[4]	ZHOU X, JIN K, SHAN Y, et al. Visually interpretable representation learning for depression recognition from facial images[J]. IEEE Transactions on Affective Computing, 2020, 11(3): 542-552.
[5]	GOH K M, NG C H, LIM L L, et al. Micro-expression rec-ognition: an updated review of current trends, challenges and solutions[J]. The Visual Computer, 2020, 36(3): 445-468.
[6]	WANG C, PENG M, BI T, et al. Micro-attention for micro-expression recognition[J]. Neurocomputing, 2020, 410: 354-362.
[7]	TAKALKAR M A, XU M, CHACZKO Z. Manifold feature integration for micro-expression recognition[J]. Multimedia Systems, 2020, 26(5): 535-551.
[8]	ZENG N, ZHANG H, SONG B, et al. Facial expression recognition via learning deep sparse autoencoders[J]. Neuro-computing, 2018, 273: 643-649.
[9]	LI S, DENG W H, DU J P. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild[C]// Proceedings of the 2017 IEEE Conference on Comp-uter Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017.Washington: IEEE Computer Society, 2017: 2584-2593.
[10]	LI Y, ZENG J, SHAN S G, et al. Occlusion aware facial expression recognition using CNN with attention mechanism[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2439-2450.
[11]	JIA X, ZHENG X, LI W, et al. Facial emotion distribution learning by exploiting low-rank label correlations locally[C]// Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Was-hington: IEEE Computer Society, 2019: 9841-9850.
[12]	LEE J, KIM S, KIM S, et al. Context-aware emotion reco-gnition networks[C]// Proceedings of the 2019 IEEE Intern-ational Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Washington:IEEE Computer Society, 2019: 10143-10152.
[13]	BADDAR W J, RO Y M. Mode variational LSTM robust to unseen modes of variation: application to facial expression recognition[C]// Proceedings of the 33rd AAAI Conference on Artificial Intelligence, the 31st Innovative Applications of Artificial Intelligence Conference, the 9th AAAI Sympo-sium on Educational Advances in Artificial Intelligence, Honolulu, Jan 27-Feb 1, 2019. Menlo Park:AAAI, 2019: 3215-3223.
[14]	XIE S, HU H, WU Y. Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition[J]. Pattern Recognition, 2019, 92: 177-191.
[15]	BAI M, XIE W, SHEN L. Disentangled feature based adversarial learning for facial expression recognition[C]// Proceedings of the 2019 IEEE International Conference on Image Processing, Taiwan, China, Sep 22, 2019. Piscataway:IEEE, 2019: 31-35.
[16]	BARSOUM E, ZHANG C, FERRER C C, et al. Training deep networks for facial expression recognition with crowd-sourced label distribution[C]// Proceedings of the 18th ACM International Conference on Multimodal Interaction,Tokyo, Nov 12-16, 2016. New York: ACM, 2016: 279-283.
[17]	BARROS P V A, PARISI G I, WERMTER S. A personalized affective memory model for improving emotion recognition[C]// Proceedings of the 36th International Confe-rence on Machine Learning, Long Beach, Jun 9-15, 2019: 485-494.
[18]	SHAN L, DENG W H. Deep facial expression recognition: a survey[J]. IEEE Transactions on Affective Computing, 2020. DOI: 10.1109/TAFFC.2020.2981446.
[19]	LI S, DENG W H. Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recog-nition[J]. IEEE Transactions on Image Processing, 2018, 28(1): 356-370.
[20]	ZHANG L, VERMA B, TJONDRONEGORO D, et al. Facial expression analysis under partial occlusion: a survey[J]. ACM Computing Surveys, 2018, 51(2): 1-49.
[21]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recog-nition, Las Vegas, Jun 26-Jul 1, 2016. Washington:IEEE Computer Society, 2016: 770-778.
[22]	MOLLAHOSSEINI A, HASANI B, MAHOOR M H. AffectNet: a database for facial expression, valence, and arousal computing in the wild[J]. IEEE Transactions on Affective Computing, 2017, 10(1): 18-31.
[23]	ZENG J B, SHAN S G, CHEN X L. Facial expression recognition with inconsistently annotated datasets[C]// LNCS 11217: Proceedings of the 15th European Conference on Com-puter Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 227-243.
[24]	KOUJAN M R, ALHARBAWEE L, GIANNAKAKIS G, et al. Real-time facial expression recognition“in the wild”by disentangling 3D expression from identity[J].arXiv:2005.05509, 2020.
[25]	LI M, XU H, HUANG X C, et al. Facial expression recog-nition with identity and emotion joint learning[J]. IEEE Trans-actions on Affective Computing, 2021, 12(2): 544-550.
[26]	MIAO S, XU H, HAN Z, et al. Recognizing facial expressions using a shallow convolutional neural network[J]. IEEE Access, 2019, 7: 78000-78011.
[27]	SIQUEIRA H, MAGG S, WERMTER S. Efficient facial feature learning with wide ensemble-based convolutional neural networks[C]// Proceedings of the 34th AAAI Conference on Artificial Intelligence, the 32nd Innovative Applications of Artificial Intelligence Conference, the 10th AAAI Symposium on Educational Advances in Artificial Intelligence, New York, Feb 7-12, 2020. Menlo Park: AAAI, 2020: 5800-5809.