Class-Balanced Modulation for Facial Expression Recognition

doi:10.3778/j.issn.1673-9418.2210079

Abstract

Abstract: Facial expression recognition (FER) aims at determining the types of facial expressions for given facial images, which has a broad application prospect in psychological diagnosis, human-computer interaction, etc. In practical tasks, various databases tend to have imbalanced data distributions among basic facial expressions. Such an issue has caused imbalanced feature distribution and inconsistent classifier optimization for various facial expressions, seriously affecting the performance of expression recognition models. To solve this issue, this paper proposes a class-balanced modulation mechanism for facial expression recognition (CBM-Net), which attempts to address the imbalanced data distribution problem by modulating the FER model in feature learning and classifier optimization stages. CBM-Net includes two modules of feature modulation and gradient modulation. The feature modulation module struggles to balance feature distributions for all facial expressions by increasing the separability between classes and the tightness within classes in the feature direction. The gradient modulation module uses the statistical information of batch training samples to reversely adjust the optimization gradient of each classifier to ensure that the convergence speed of each classifier is consistent, so that the performance of each classifier can be optimal at the same time. Qualitative and quantitative experiments on four popular datasets show that CBM-Net is effective in class-balanced modulation, and its effect is quite good compared with many advanced methods.

Key words: facial expression recognition (FER), class imbalance, class balance modulation, feature modulation, gradient modulation

摘要： 人脸表情识别（FER）旨在从人脸图片中判断表情的类别，在心理诊断、人机交互等领域有着广泛的应用前景。在实际任务中，不同表情数据的分布往往是不平衡的。数据的不平衡导致了各表情的特征分布不平衡和分类器优化不平衡，从而影响了表情识别模型的性能。为此，提出了一种类别平衡调制的人脸表情识别方法（CBM-Net），分别在特征学习阶段和分类器优化阶段对模型进行类别平衡调制。CBM-Net包括特征调制和梯度调制两个模块。特征调制模块通过在特征方向上增加类间的可分性与类内的紧密性，实现各类别的特征分布平衡。梯度调制模块利用批次训练样本的统计信息对各分类器的优化梯度进行反向调节，确保各类别的分类器收敛速度一致，使得各分类器性能同时达到最优。在四个流行的数据集上进行的定性和定量实验表明，CBM-Net在人脸表情识别的类别平衡调制上是有效的，与一众先进方法相比，效果也相当良好。

关键词: 人脸表情识别（FER）, 类别不平衡, 类别平衡调制, 特征调制, 梯度调制

LIU Chengguang, WANG Shanmin, LIU Qingshan. Class-Balanced Modulation for Facial Expression Recognition[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(12): 3029-3038.

刘成广, 王善敏, 刘青山. 类别平衡调制的人脸表情识别[J]. 计算机科学与探索, 2023, 17(12): 3029-3038.

References

[1] 洪惠群, 沈贵萍, 黄风华. 表情识别技术综述[J]. 计算机科学与探索, 2022, 16(8): 1764-1778.
HONG H Q, SHEN G P, HUANG F H. Summary of expression recognition technology[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(8): 1764-1778.
[2] 邵志文, 周勇, 马利庄, 等. 基于深度学习的表情动作单元识别综述[J]. 电子学报, 2022, 50(8): 2003-2017.
SHAO Z W, ZHOU Y, MA L Z, et al. Survey of expression action unit recognition based on deep learning[J]. Acta Electronica Sinica, 2022, 50(8): 2003-2017.
[3] DURIC Z, GRAY W D, HEISHMAN R, et al. Integrating perceptual and cognitive modeling for adaptive and intel-ligent human-computer interaction[J]. Proceedings of the IEEE, 2002, 90(7): 1272-1289.
[4] LI B, MEHTA S, ANEJA D, et al. A facial affect analysis system for autism spectrum disorder[C]//Proceedings of the 2019 IEEE International Conference on Image Processing, Taipei, China，Sep 22-25, 2019. Piscataway: IEEE, 2019: 4549-4553.
[5] JEONG M, KO B C. Driver’s facial expression recognition in real-time for safe driving[J]. Sensors, 2018, 18(12): 4270.
[6] LUCEY P, COHN J F, KANADE T, et al. The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression[C]//Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, Jun 13-18, 2010. Washington: IEEE Computer Society，2010: 94-101.
[7] VALSTAR M, PANTIC M. Induced disgust, happiness and surprise: an addition to the mmi facial expression database[C]//Proceedings of the 3rd Workshop on EMOTION, 2010: 65-70.
[8] LYONS M, AKAMATSU S, KAMACHI M, et al. Coding facial expressions with gabor wavelets[C]//Proceedings of the 3rd International Conference on Face & Gesture Recog-nition, Nara, Apr 14-16, 1998. Washington: IEEE Computer Society, 1998: 200-205.
[9] LI S, DENG W, DU J P. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 2584-2593.
[10] MOLLAHOSSEINI A, HASANI B, MAHOOR M H. Affect-Net: a database for facial expression, valence, and arousal computing in the wild[J]. IEEE Transactions on Affective Computing, 2017, 10(1): 18-31.
[11] GOODFELLOW I J, ERHAN D, CARRIER P L, et al. Chal-lenges in representation learning: a report on three machine learning contests[C]//LNCS 8228: Proceedings of the 20th International Conference on Neural Information Processing, Daegu, Nov 3-7, 2013: 117-124.
[12] DRUMMOND C, HOLTE R C. C4.15, class imbalance, and cost sensitivity: why under-sampling beats over-sampling[C]//Proceedings of the 2003 Workshop on Learning from Imbalanced Datasets II, Washington, Aug 21, 2003: 1-8.
[13] BUDA M, MAKI A, MAZUROWSKI M A. A systematic study of the class imbalance problem in convolutional neural networks[J]. Neural Networks, 2018, 106: 249-259.
[14] CHAWLA N V, BOWYER K W, HALL L O, et al. SMOTE: synthetic minority over-sampling technique[J]. Journal of Artificial Intelligence Research, 2002, 16: 321-357.
[15] HAN H, WANG W Y, MAO B H. Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning[C]//LNCS 3644: Proceedings of the 2005 International Conference on Intelligent Computing, Hefei, Aug 23-26, 2005: 878-887.
[16] HUANG C, LI Y, LOY C C, et al. Learning deep represen-tation for imbalanced classification[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 5375-5384.
[17] WANG Y X, RAMANAN D, HEBERT M. Learning to model the tail[C]//Proceedings of the Annual Conference on Neural Information Processing Systems 2017, Dec 4-9, 2017: 7029-7039.
[18] MALISIEWICZ T, GUPTA A, EFROS A A. Ensemble of exemplar-SVMs for object detection and beyond[C]//Pro-ceedings of the 2011 International Conference on Computer Vision, Barcelona, Nov 6-13, 2011. Washington: IEEE Com-puter Society, 2011: 89-96.
[19] DONG Q, GONG S, ZHU X. Class rectification hard mining for imbalanced deep learning[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 1869-1878.
[20] LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 2999-3007.
[21] CUI Y, JIA M, LIN T Y, et al. Class-balanced loss based on effective number of samples[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recog-nition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 9268-9277.
[22] KANG B, XIE S, ROHRBACH M, et al. Decoupling repre-sentation and classifier for long-tailed recognition[C]//Pro-ceedings of the 8th International Conference on Learning Representations, Addis Ababa, Apr 26-30, 2020: 1-19.
[23] REN M, ZENG W, YANG B, et al. Learning to reweight examples for robust deep learning[C]//Proceedings of the 35th International Conference on Machine Learning, Stock-holm, Jul 10-15, 2018: 4331-4340.
[24] DHALL A, RAMANA MURTHY O V, GOECKE R, et al. Video and image based emotion recognition challenges in the wild: EmotiW 2015[C]//Proceedings of the 2015 ACM International Conference on Multimodal Interaction, Seattle, May 15, 2015. New York: ACM, 2015: 423-426.
[25] LEE J, KIM S, KIM S, et al. Context-aware emotion recog-nition networks[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 10143-10152.
[26] KOH P W, LIANG P. Understanding black-box predictions via influence functions[C]//Proceedings of the 34th Interna-tional Conference on Machine Learning, Sydney, Aug 6-11, 2017: 1885-1894.
[27] SHE J, HU Y, SHI H, et al. Dive into ambiguity: latent dis-tribution mining and pairwise uncertainty estimation for facial expression recognition[C]//Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recog-nition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 6248-6257.
[28] 黄浩, 葛洪伟. 强化类间区分的深度残差表情识别网络[J]. 计算机科学与探索, 2022, 16(8): 1842-1849.
HUANG H, GE H W. Deep residual expression recognition network to enhance inter-class discrimination[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(8): 1842-1849.
[29] DENG J, GUO J, VERVERAS E, et al. Retinaface: single-shot multi-level face localisation in the wild[C]//Procee-dings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 5203-5212.
[30] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Con-ference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 770-778.
[31] GUO Y, ZHANG L, HU Y, et al. MS-CELEB-1M: a dataset and benchmark for large-scale face recognition[C]//LNCS 9907: Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 11-14, 2016. Cham: Sprin-ger, 2016: 87-102.
[32] VAN DER MAATEN L, HINTON G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9: 2579-2605.
[33] WANG K, PENG X, YANG J, et al. Suppressing uncertain-ties for large-scale facial expression recognition[C]//Procee-dings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Pis-cataway: IEEE, 2020: 6897-6906.
[34] ZHAO Z, LIU Q, ZHOU F. Robust lightweight facial expres-sion recognition network with label distribution training[C]//Proceedings of the 35th AAAI Conference on Artifi-cial Intelligence, the 33rd Conference on Innovative Appli-cations of Artificial Intelligence, the 11th Symposium on Educational Advances in Artificial Intelligence, Feb 2-9, 2021. Menlo Park: AAAI, 2021: 3510-3519.
[35] ZHANG Y, WANG C, DENG W. Relative uncertainty learning for facial expression recognition[C]//Advances in Neural Information Processing?Systems?34,?Dec?6-14, 2021: 17616-17627.
[36] RUAN D, YAN Y, CHEN S, et al. Deep disturbance-disen-tangled learning for facial expression recognition[C]//Pro-ceedings of the 28th ACM International Conference on Multimedia, Seattle, Oct 12-16, 2020. New York: ACM, 2020: 2833-2841.
[37] WEN Z, LIN W, WANG T, et al. Distract your attention: multi-head cross attention network for facial expression recognition[J]. arXiv:2109.07270, 2021.
[38] CAI J, MENG Z, KHAN A S, et al. Island loss for learning discriminative features in facial expression recognition[C]//Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, Xi??an, May 15-19, 2018. Washington: IEEE Computer Society, 2018: 302-309.
[39] ACHARYA D, HUANG Z, PANI PAUDEL D, et al. Cova-riance pooling for facial expression recognition[C]//Procee-dings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 367-374.
[40] GAO S H, CHENG M M, ZHAO K, et al. Res2Net: a new multi-scale backbone architecture[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 43(2): 652-662.
[41] LI H, WANG N, DING X, et al. Adaptively learning facial expression representation via CF labels and distillation[J]. IEEE Transactions on Image Processing, 2021, 30: 2016-2028.
[42] VO T H, LEE G S, YANG H J, et al. Pyramid with super resolution for in-the-wild facial expression recognition[J]. IEEE Access, 2020, 8: 131988-132001.
[43] GROSS R, MATTHEWS I, COHN J, et al. Multi-PIE[J]. Image and Vision Computing, 2010, 28(5): 807-813.