强化类间区分的深度残差表情识别网络

doi:10.3778/j.issn.1673-9418.2011042

计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (8): 1842-1849.DOI: 10.3778/j.issn.1673-9418.2011042

强化类间区分的深度残差表情识别网络

黄浩¹^,², 葛洪伟¹

1. 江苏省模式识别与计算智能工程实验室（江南大学），江苏无锡 214122
2. 江南大学人工智能与计算机学院，江苏无锡 214122

收稿日期:2020-11-12 修回日期:2021-01-29 出版日期:2022-08-01 发布日期:2021-03-28
作者简介:黄浩（1996—），男，湖北黄冈人,硕士研究生，CCF学生会员，主要研究方向为模式识别、机器学习。
葛洪伟（1967—），男，江苏无锡人，博士，教授，博士生导师，主要研究方向为人工智能、模式识别、机器学习、图像处理与分析等。
基金资助:
江苏省研究生创新计划项目(KYLX16_0781);江苏高校优势学科建设工程资助项目。

Deep Residual Expression Recognition Network to Enhance Inter-class Discrimination

HUANG Hao¹^,², GE Hongwei¹

1. Jiangsu Provincial Engineering Laboratory of Pattern Recognition and Computational Intelligence (Jiangnan Uni-versity), Wuxi, Jiangsu 214122, China
2. School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, Jiangsu 214122, China

Received:2020-11-12 Revised:2021-01-29 Online:2022-08-01 Published:2021-03-28
About author:HUANG Hao, born in 1996, M.S. candidate, student member of CCF. His research interests include pattern recognition and machine learning.
GE Hongwei, born in 1967, Ph.D., professor, Ph.D. supervisor. His research interests include artificial intelligence, pattern recognition, machine learning, image processing and analysis, etc.
Supported by:
the Innovation Program for Graduate of Jiangsu Province(KYLX16_0781);and the Superior Discipline Construction Project of Jiangsu Higher Education Institutions.

摘要/Abstract

摘要：

深度人脸表情识别是神经网络应用于模式识别上一项极具挑战性的任务。相对于身份认证和特征点识别等人脸识别任务，表情识别任务中存在着大量的冗余信息，要得到好的效果，需要更精确的分类。多数研究关注点在数据的泛化性和网络结构上，而忽视了数据的类间关系。提出了一种基于类间分析的深度残差表情识别网络RMRnet。首先，将数据通过骨干网络Resnet18得到混淆矩阵，进一步得到召回率矩阵分析类间关系;然后，凭借类间关系设计网络结构分支，进一步区分强联系类，设计补充支路平衡弱联系类;最后，将分支添加到骨干网络的相应位置，得到RMRnet网络模型。在流行的大型数据库上，与基准方法和近年来的先进方法的对比实验结果表明，提出的方法相较于基准方法效果良好，在一众先进方法中也有很强的竞争力。

关键词: 表情识别, 神经网络, 混淆矩阵, 类间关联

Abstract:

Deep facial expression recognition is a challenging task for neural networks to apply in pattern reco-gnition. Compared with face recognition tasks such as identity authentication and feature point recognition, there are a lot of redundant information in facial expression recognition task. To achieve good results, more accurate class-ification is needed. Most researches focus on the generalization and network structure of data and ignore the inter-class relationship of data. In this paper, a deep residual expression recognition network RMRnet (recall matrix distinguished residual net) based on class analysis is proposed. First, the data are fed to the backbone network(Resnet18) to obtain the confusion matrix, after that, the recall matrix is used to analyze the relationship between classes. Then, the branches of network structure are designed based on the inter-class relationship to distinguish the strong-related classes, while the rest classes are designed as the supplementary branch to balance the weak-related class. In the end, by adding those branches to the corresponding position of backbone network, RMRnet is constituted. Compared with baseline method and advanced methods in recent years, experimental results in the popular large-scale database show that the proposed method is more effective than the baseline method and has str-ong competitiveness among a group of advanced methods.

Key words: facial expression recognition, neural network, confusion matrix, inter-class association

中图分类号:

TP391.41

黄浩, 葛洪伟. 强化类间区分的深度残差表情识别网络[J]. 计算机科学与探索, 2022, 16(8): 1842-1849.

HUANG Hao, GE Hongwei. Deep Residual Expression Recognition Network to Enhance Inter-class Discrimination[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(8): 1842-1849.

图/表 9

图1 深度表情识别通用流程

Fig.1 General pipeline of deep facial expression recognition system

图2 一个[[3×3，64],[3×3，64]]×2残差块

Fig.2 A [[3×3, 64],[3×3, 64]]×2 residual block

图3 平衡数据集的Resnet18实验(精度为58.6%)

Fig.3 Resnet18 experiment of balanced dataset (accuracy is 58.6%)

图4 原数据集同分布的Resnet18实验(精度为65.6%)

Fig.4 Resnet18 experiment with codistribution of original dataset (accuracy is 65.6%)

图5 RMRnet整体网络结构

Fig.5 Pipeline of RMRnet network

图6 数据流图

Fig.6 Data flow diagram

图7 RMRnet在RAF-DB上的预测值

Fig.7 Predicted value of RMRnet on RAF-DB

图8 Resnet18在RAF-DB上的预测值

Fig.8 Predicted value of Resnet18 on RAF-DB

表1 在AffectNet、RAF-DB、Ferplus上的对比实验

Table 1 Comparative experiment on AffectNet, RAF-DB, Ferplus

AffectNet		RAF-DB		Ferplus
Approach	Acc/%	Approach	Acc/%	Approach	Acc/%
DLP-CNN^[19]	54.47	DLP-CNN^[19]	74.20	Baseline^[21]	83.10
pACNN^[4]	55.33	EAU-Net^[1]	81.83	TFE-JL^[25]	84.30
EAU-Net^[1]	58.91	DeepExp3D^[24]	82.06	VGG13-PLD^[16]	84.99
Baseline^[21]	54.52	Baseline^[21]	82.88	SHCNN^[26]	86.54
IPA2LT^[23]	56.51	pACNN^[4]	83.05	ESR-9^[27]	87.15
RMRnet	58.43	RMRnet	86.14	RMRnet	87.26

参考文献 27

[1]	DENG J, PANG G, ZhANG Z, et al. cGAN based facial expression recognition for human-robot interaction[J]. IEEE Access, 2019, 7: 9848-9859.
[2]	DING H, SRICHARAN K, CHELLAPPA R. ExprGAN: facial expression editing with controllable expression inten- sity[C]// Proceedings of the 32nd AAAI Conference on Artificial Intelligence, the 30th Innovative Applications of Artificial Intelligence, and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, Feb 2-7, 2018. Menlo Park: AAAI, 2018: 6781-6788.
[3]	YANG H, CIFTCI U, YIN L. Facial expression recognition by de-expression residue learning[C]// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 2168-2177.
[4]	ZHOU X, JIN K, SHAN Y, et al. Visually interpretable representation learning for depression recognition from facial images[J]. IEEE Transactions on Affective Computing, 2020, 11(3): 542-552.
[5]	GOH K M, NG C H, LIM L L, et al. Micro-expression rec-ognition: an updated review of current trends, challenges and solutions[J]. The Visual Computer, 2020, 36(3): 445-468.
[6]	WANG C, PENG M, BI T, et al. Micro-attention for micro-expression recognition[J]. Neurocomputing, 2020, 410: 354-362.
[7]	TAKALKAR M A, XU M, CHACZKO Z. Manifold feature integration for micro-expression recognition[J]. Multimedia Systems, 2020, 26(5): 535-551.
[8]	ZENG N, ZHANG H, SONG B, et al. Facial expression recognition via learning deep sparse autoencoders[J]. Neuro-computing, 2018, 273: 643-649.
[9]	LI S, DENG W H, DU J P. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild[C]// Proceedings of the 2017 IEEE Conference on Comp-uter Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017.Washington: IEEE Computer Society, 2017: 2584-2593.
[10]	LI Y, ZENG J, SHAN S G, et al. Occlusion aware facial expression recognition using CNN with attention mechanism[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2439-2450.
[11]	JIA X, ZHENG X, LI W, et al. Facial emotion distribution learning by exploiting low-rank label correlations locally[C]// Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Was-hington: IEEE Computer Society, 2019: 9841-9850.
[12]	LEE J, KIM S, KIM S, et al. Context-aware emotion reco-gnition networks[C]// Proceedings of the 2019 IEEE Intern-ational Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Washington:IEEE Computer Society, 2019: 10143-10152.
[13]	BADDAR W J, RO Y M. Mode variational LSTM robust to unseen modes of variation: application to facial expression recognition[C]// Proceedings of the 33rd AAAI Conference on Artificial Intelligence, the 31st Innovative Applications of Artificial Intelligence Conference, the 9th AAAI Sympo-sium on Educational Advances in Artificial Intelligence, Honolulu, Jan 27-Feb 1, 2019. Menlo Park:AAAI, 2019: 3215-3223.
[14]	XIE S, HU H, WU Y. Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition[J]. Pattern Recognition, 2019, 92: 177-191.
[15]	BAI M, XIE W, SHEN L. Disentangled feature based adversarial learning for facial expression recognition[C]// Proceedings of the 2019 IEEE International Conference on Image Processing, Taiwan, China, Sep 22, 2019. Piscataway:IEEE, 2019: 31-35.
[16]	BARSOUM E, ZHANG C, FERRER C C, et al. Training deep networks for facial expression recognition with crowd-sourced label distribution[C]// Proceedings of the 18th ACM International Conference on Multimodal Interaction,Tokyo, Nov 12-16, 2016. New York: ACM, 2016: 279-283.
[17]	BARROS P V A, PARISI G I, WERMTER S. A personalized affective memory model for improving emotion recognition[C]// Proceedings of the 36th International Confe-rence on Machine Learning, Long Beach, Jun 9-15, 2019: 485-494.
[18]	SHAN L, DENG W H. Deep facial expression recognition: a survey[J]. IEEE Transactions on Affective Computing, 2020. DOI: 10.1109/TAFFC.2020.2981446.
[19]	LI S, DENG W H. Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recog-nition[J]. IEEE Transactions on Image Processing, 2018, 28(1): 356-370.
[20]	ZHANG L, VERMA B, TJONDRONEGORO D, et al. Facial expression analysis under partial occlusion: a survey[J]. ACM Computing Surveys, 2018, 51(2): 1-49.
[21]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recog-nition, Las Vegas, Jun 26-Jul 1, 2016. Washington:IEEE Computer Society, 2016: 770-778.
[22]	MOLLAHOSSEINI A, HASANI B, MAHOOR M H. AffectNet: a database for facial expression, valence, and arousal computing in the wild[J]. IEEE Transactions on Affective Computing, 2017, 10(1): 18-31.
[23]	ZENG J B, SHAN S G, CHEN X L. Facial expression recognition with inconsistently annotated datasets[C]// LNCS 11217: Proceedings of the 15th European Conference on Com-puter Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 227-243.
[24]	KOUJAN M R, ALHARBAWEE L, GIANNAKAKIS G, et al. Real-time facial expression recognition“in the wild”by disentangling 3D expression from identity[J].arXiv:2005.05509, 2020.
[25]	LI M, XU H, HUANG X C, et al. Facial expression recog-nition with identity and emotion joint learning[J]. IEEE Trans-actions on Affective Computing, 2021, 12(2): 544-550.
[26]	MIAO S, XU H, HAN Z, et al. Recognizing facial expressions using a shallow convolutional neural network[J]. IEEE Access, 2019, 7: 78000-78011.
[27]	SIQUEIRA H, MAGG S, WERMTER S. Efficient facial feature learning with wide ensemble-based convolutional neural networks[C]// Proceedings of the 34th AAAI Conference on Artificial Intelligence, the 32nd Innovative Applications of Artificial Intelligence Conference, the 10th AAAI Symposium on Educational Advances in Artificial Intelligence, New York, Feb 7-12, 2020. Menlo Park: AAAI, 2020: 5800-5809.

强化类间区分的深度残差表情识别网络

Deep Residual Expression Recognition Network to Enhance Inter-class Discrimination

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 27

相关文章 15

编辑推荐

Metrics

[1]	吕晓琦, 纪科, 陈贞翔, 孙润元, 马坤, 邬俊, 李浥东. 结合注意力与循环神经网络的专家推荐算法[J]. 计算机科学与探索, 2022, 16(9): 2068-2077.
[2]	安凤平, 李晓薇, 曹翔. 权重初始化-滑动窗口CNN的医学图像分类[J]. 计算机科学与探索, 2022, 16(8): 1885-1897.
[3]	于慧琳, 陈炜, 王琪, 高建伟, 万怀宇. 使用子图推理实现知识图谱关系预测[J]. 计算机科学与探索, 2022, 16(8): 1800-1808.
[4]	洪惠群, 沈贵萍, 黄风华. 表情识别技术综述[J]. 计算机科学与探索, 2022, 16(8): 1764-1778.
[5]	李玉轩, 洪学海, 汪洋, 唐正正, 班艳. 引入激活加权策略的分组排序学习方法[J]. 计算机科学与探索, 2022, 16(7): 1594-1602.
[6]	张雁操, 赵宇海, 史岚. 融合图注意力的多特征链接预测算法[J]. 计算机科学与探索, 2022, 16(5): 1096-1106.
[7]	欧阳柳, 贺禧, 瞿绍军. 全卷积注意力机制神经网络的图像语义分割[J]. 计算机科学与探索, 2022, 16(5): 1136-1145.
[8]	程卫月, 张雪琴, 林克正, 李骜. 融合全局与局部特征的深度卷积神经网络算法[J]. 计算机科学与探索, 2022, 16(5): 1146-1154.
[9]	童敢, 黄立波. Winograd快速卷积相关研究综述[J]. 计算机科学与探索, 2022, 16(5): 959-971.
[10]	裴利沈, 赵雪专. 群体行为识别深度学习方法研究综述[J]. 计算机科学与探索, 2022, 16(4): 775-790.
[11]	张少伟, 王鑫, 陈子睿, 王林, 徐大为, 贾勇哲. 有监督实体关系联合抽取方法研究综述[J]. 计算机科学与探索, 2022, 16(4): 713-733.
[12]	朱伟杰, 陈莹. 双流时间域信息交互的微表情识别卷积网络[J]. 计算机科学与探索, 2022, 16(4): 950-958.
[13]	卓天天, 桑庆兵. 注意力机制与复合卷积在手写识别中的应用[J]. 计算机科学与探索, 2022, 16(4): 888-897.
[14]	陆仲达, 张春达, 张佳奇, 王子菲, 许军华. 双分支网络的苹果叶部病害识别[J]. 计算机科学与探索, 2022, 16(4): 917-926.
[15]	马金林, 张裕, 马自萍, 毛凯绩. 轻量化神经网络卷积设计研究进展[J]. 计算机科学与探索, 2022, 16(3): 512-528.