基于分割和多级掩膜学习的遮挡人脸识别方法

doi:10.3778/j.issn.1673-9418.2306082

摘要/Abstract

摘要： 现有的人脸识别方法无法有效消除遮挡造成的损坏特征的影响，随着网络层数加深，损坏特征与用于身份分类的有效特征变得难以分离，影响识别结果。针对上述问题，设计了一种基于分割和多级掩膜学习策略的遮挡人脸识别方法，模型由遮挡检测分割、特征提取、掩膜学习单元三大模块构成，无需依赖额外的遮挡检测器，且无论训练还是测试都只需要一次端到端的过程即可同时学习特征掩膜和深度抗遮挡特征。掩膜学习单元以不同尺寸的遮挡分割表示和不同阶段的人脸特征为输入，为特征提取的不同阶段生成对应的掩膜，通过掩膜运算在特征提取的各阶段有效消除遮挡造成的损坏特征的影响，最终构建特征金字塔融合各阶段特征进行身份分类。实验结果表明该方法可有效提高遮挡人脸识别的准确率，在经过遮挡处理的LFW数据集以及真实的口罩遮挡数据集MFR2、Mask_whn上的准确率分别达到了98.77%、96.70%、81.53%，与现有的主流方法相比分别提升了2.04、0.48、4.44个百分点。

关键词: 遮挡人脸识别, 多级掩膜学习, 遮挡检测分割, 特征金字塔

Abstract: Existing face recognition methods cannot effectively eliminate the influence of corrupted features caused by occlusion. As the features flow deeper, the corrupted features get entangled with the effective features used for identity classification, which affects the recognition results. To address the problem, this paper designs an occluded face recognition method based on segmentation and multi-stage mask learning strategy. The model consists of three components: occlusion detection and segmentation, feature extraction, and mask learning unit. The proposed method only needs one end-to-end process to learn feature masks and deep occlusion-robust features without relying on additional occlusion detectors. The mask learning units take different sizes of occlusion segmentation representations and facial features of different stages as input, generate corresponding feature masks for different stages of feature extraction, and effectively eliminate the influence of corrupted features caused by occlusion at each stage of feature extraction through mask operations. Finally, a feature pyramid is constructed to fuse features of different stages for identity classification. Experimental results show that the proposed method can effectively improve the accuracy of occluded face recognition. The accuracy on the occluded LFW dataset and the real masked datasets MFR2 and Mask_whn reach 98.77%, 96.70% and 81.53%, respectively, which has an accuracy improvement of 2.04, 0.48 and 4.44 percentage points compared with the existing mainstream methods.

Key words: occluded face recognition, multi-stage mask learning, occlusion detection and segmentation, feature pyramid

张铮, 芦天亮, 曹金璇. 基于分割和多级掩膜学习的遮挡人脸识别方法[J]. 计算机科学与探索, 2024, 18(7): 1814-1825.

ZHANG Zheng, LU Tianliang, CAO Jinxuan. Occluded Face Recognition Based on Segmentation and Multi-stage Mask Learning[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(7): 1814-1825.

参考文献

[1] TOMAR V, KUMAR N, SRIVASTAVA A R. Single sample face recognition using deep learning: a survey[J]. Artificial Intelligence Review, 2023, 56(S1): 1063-1111.
[2] LIU W, WANG X. Research advanced in the face recognition[J]. Highlights in Science, Engineering and Technology, 2023, 49: 448-456.
[3] KIM M, JAIN A K, LIU X. AdaFace: quality adaptive margin for face recognition[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 18-24, 2022. Piscataway: IEEE, 2022: 18729-18738.
[4] BOUTROS F, DAMER N, KIRCHBUCHNER F, et al. ElasticFace: elastic margin loss for deep face recognition[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 19-20, 2022. Piscataway: IEEE, 2022: 1577-1586.
[5] JEEVAN G, ZACHARIAS G C, NAIR M S, et al. An empirical study of the impact of masks on face recognition[J]. Pattern Recognition, 2022, 122: 108308.
[6] MIN R, HADID A, DUGELAY J L. Improving the recognition of faces occluded by facial accessories[C]//Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition, Santa Barbara, Mar 21-23, 2011. Piscataway: IEEE, 2011: 442-447.
[7] PARK S, LEE H, YOO J H, et al. Partially occluded facial image retrieval based on a similarity measurement[J]. Mathematical Problems in Engineering, 2015. DOI: 10.1155/2015/217568.
[8] LV J J, SHAO X H, HUANG J S, et al. Data augmentation for face recognition[J]. Neurocomputing, 2017, 230: 184-196.
[9] TRIGUEROS D S, MENG L, HARTNETT M. Enhancing convolutional neural networks for face recognition with occlusion maps and batch triplet loss[J]. Image and Vision Computing, 2018, 79: 99-108.
[10] HE M, ZHANG J, SHAN S, et al. Locality-aware channel-wise dropout for occluded face recognition[J]. IEEE Transactions on Image Processing, 2021, 31: 788-798.
[11] HUBER M, BOUTROS F, KIRCHBUCHNER F, et al. Mask-invariant face recognition through template-level knowledge distillation[C]//Proceedings of the 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition, Jodhpur, Dec 15-18, 2021. Piscataway: IEEE, 2021: 1-8.
[12] HAO S, CHEN C, CHEN Z, et al. A unified framework for masked and mask-free face recognition via feature rectification[C]//Proceedings of the 2022 IEEE International Conference on Image Processing, Bordeaux, Oct 16-19, 2022. Piscataway: IEEE, 2022: 726-730.
[13] CHEN Y A, CHEN W C, WEI C P, et al. Occlusion-aware face inpainting via generative adversarial networks[C]//Proceedings of the 2017 IEEE International Conference on Image Processing, Beijing, Sep 17-20, 2017. Piscataway: IEEE, 2017: 1202-1206.
[14] ZHAO F, FENG J, ZHAO J, et al. Robust LSTM-auto-encoders for face de-occlusion in the wild[J]. IEEE Transactions on Image Processing, 2017, 27(2): 778-790.
[15] HU B, ZHENG Z, LIU P, et al. Unsupervised eyeglasses removal in the wild[J]. IEEE Transactions on Cybernetics, 2020, 51(9): 4373-4385.
[16] WU Y, SINGH V, KAPOOR A. From image to video face inpainting: spatial-temporal nested GAN (STN-GAN) for usability recovery[C]//Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision, Snowmass Village, Mar 1-5, 2020. Washington: IEEE Computer Society, 2020: 2385-2394.
[17] JU Y J, LEE G H, HONG J H, et al. Complete face recovery GAN: unsupervised joint face rotation and de-occlusion from a single-view image[C]//Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, Jan 3-8, 2022. Piscataway: IEEE, 2022: 1173-1183.
[18] DING F, PENG P, HUANG Y, et al. Masked face recognition with latent part detection[C]//Proceedings of the 28th ACM International Conference on Multimedia, Seattle Oct 12-16, 2020. New York: ACM, 2020: 2281-2289.
[19] WAN W, CHEN J. Occlusion robust face recognition based on mask learning[C]//Proceedings of the 2017 IEEE International Conference on Image Processing, Beijing, Sep 17-20, 2017. Piscataway: IEEE, 2017: 3795-3799.
[20] SONG L, GONG D, LI Z, et al. Occlusion robust face recognition based on mask learning with pairwise differential siamese network[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 773-782.
[21] QIU H, GONG D, LI Z, et al. End2End occluded face recognition by masking corrupted features[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(10): 6939-6952.
[22] ZENG D, VELDHUIS R, SPREEUWERS L, et al. Occlusion-invariant face recognition using simultaneous segmentation[J]. IET Biometrics, 2021, 10(6): 679-691.
[23] QIN X, ZHANG Z, HUANG C, et al. U2-Net: going deeper with nested U-structure for salient object detection[J]. Pattern Recognition, 2020, 106: 107404.
[24] DENG J, GUO J, YANG J, et al. ArcFace: additive angular margin loss for deep face recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(10): 5962-5979.
[25] LIN T Y, DOLLáR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 936-944.
[26] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Italy, Oct 5-9, 2015. Cham: Springer, 2015: 234-241.
[27] PENG C, ZHANG X, YU G, et al. Large kernel matters—improve semantic segmentation by global convolutional network[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 1743-1751.
[28] SHI W, CABALLERO J, HUSZAR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 1874-1883.
[29] MASI I, MATHAI J, ABDALMAGEED W. Towards learning structure via consensus for face segmentation and parsing[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 5507-5517.
[30] ZHANG K, ZHANG Z, LI Z, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23(10): 1499-1503.
[31] YI D, LEI Z, LIAO S, et al. Learning face representation from scratch[EB/OL]. [2023-04-03]. https://arxiv.org/abs/1411. 7923.
[32] ANWAR A, RAYCHOWDHURY A. Masked face recognition for secure authentication[EB/OL]. [2023-04-03]. https://arxiv.org/abs/2008.11104.
[33] HUANG G B, MATTAR M, BERG T, et al. Labeled faces in the wild: a database for studying face recognition in unconstrained environments[C]//Proceedings of the 2008 Workshop on Faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition, Marseille, Oct 17, 2008: 617-624.
[34] WANG Z, HUANG B, WANG G, et al. Masked face recognition dataset and application[J]. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2023, 5(2): 298-304.
[35] GUO Y, ZHANG L, HU Y, et al. MS-Celeb-1M: a dataset and benchmark for large-scale face recognition[C]//Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 11-14, 2016. Cham: Springer, 2016: 87-102.
[36] WANG H, WANG Y, ZHOU Z, et al. CosFace: large margin cosine loss for deep face recognition[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-23, 2018. Piscataway: IEEE, 2018: 5265-5274.
[37] VAN DER MAATEN L, HINTON G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9: 2579-2605.