Face Inpainting Algorithm Combining Edge Information with Gated Convolution

doi:10.3778/j.issn.1673-9418.2001035

Abstract

Abstract:

For the face inpainting under arbitrary shape occlusion, the existing methods are easy to produce edge blur and distortion of the inpainting results. In this paper, an algorithm for face inpainting combining edge information with gated convolution is proposed. Firstly, the edge image of occluded area is generated by prior face knowledge to constrain the process of face inpainting. Secondly, the gated convolution holds the ability to extract accurate local feature in the absence of some pixels, and a gated convolution-based generative adversarial network (GAN) for image inpainting is designed. The model consists of two parts: edge connection GAN and image inpainting GAN. The edge connection network uses the binary occlusion image, the image to be repaired and its edge image for training, and realizes the automatic completion and connection of the missing edge image. The image inpainting GAN takes the completed edge image as the guidance information, and combines the occlusion image to repair the missing area. The experimental results show that the inpainting effect of this algorithm is better than that of other algorithms, and its evaluation indicators are better than those of the current image inpainting algorithms based on deep learning.

Key words: face inpainting, Canny edge, gated convolution, deep learning, generative adversarial network (GAN)

摘要：

针对任意形状遮挡下人脸修复，现有方法容易产生边缘模糊和恢复结果失真等问题。提出了一种结合边缘信息和门卷积的人脸修复算法。首先，通过先验人脸知识产生遮挡区域的边缘图，以约束人脸修复过程。其次，利用门卷积在部分像素缺失下的精确局部特征描述能力，设计面向图像修复的门卷积深度生成对抗网络（GAN）。该模型由边缘连接生成对抗网络和图像修复生成对抗网络两部分组成。边缘连接网络利用二值遮挡图和待修复图像及其边缘图的多源信息进行训练，实现对缺失边缘图像的自动补全和连接。图像修复网络以补全的边缘图为引导信息，联合遮挡图像进行缺失区域修复。实验结果表明：相比其他算法，该算法修复效果更好，其评价指标比当前基于深度学习的图像修复算法更优。

关键词: 人脸修复, Canny边缘, 门卷积, 深度学习, 生成对抗网络（GAN）

WANG Fuping, LI Wenlou, LIU Ying, LU Jin, GONG Yanchao. Face Inpainting Algorithm Combining Edge Information with Gated Convolution[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(1): 150-162.

王富平, 李文楼, 刘颖, 卢津, 公衍超. 结合边缘信息和门卷积的人脸修复算法[J]. 计算机科学与探索, 2021, 15(1): 150-162.

References

[1] WANG H Y, LIANG H Z. Recognition of local occluded facial expressions based on improved generative adversarial network[J]. Computer Engineering and Applications, 2020, 56(5): 141-146.
王海涌, 梁红珠. 基于改进的GAN的局部遮挡人脸表情识别[J]. 计算机工程与应用, 2020, 56(5): 141-146.
[2] SUN X, DING X L. Data augmentation method based on generative adversarial networks for facial expression reco-gnition sets[J]. Computer Engineering and Applications, 2020, 56(4): 115-121.
孙晓, 丁小龙. 基于生成对抗网络的人脸表情数据增强方法[J]. 计算机工程与应用, 2020, 56(4): 115-121.
[3] BERTALMIO M, SAPIRO G, CASELLES V, et al. Image inpainting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, Jul 23-28, 2000. New York: ACM, 2000: 417-424.
[4] BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3): 24.
[5] HE K M, SUN J. Computing nearest-neighbor fields via propagation-assisted KD-trees[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Reco-gnition, Rhode Island, Jun 16-21, 2012. Washington: IEEE Computer Society, 2012: 111-118.
[6] BERTALMíO M, VESE L A, SAPIRO G, et al. Simultaneous structure and texture image inpainting[J]. IEEE Transactions on Image Processing，2003, 12(8): 882-889.
[7] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[C]//Proceedings of the 2014 International Conference on Neural Information Pro-cessing Systems, Montreal, Dec 8-13, 2014. Red Hook: Curran Associates, 2014: 2672-2680.
[8] IIZUKA S, SIMO- SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 107.
[9] LI Y J, LIU S F, YANG J M, et al. Generative face com-pletion[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Piscataway: IEEE, 2017: 3911-3919.
[10] YU J, LIN Z, YANG J, et al. Generative image inpainting with contextual attention[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Com-puter Society, 2018: 5505-5514.
[11] LIU G L, REDA F A, SHIH K J, et al. Image inpainting for irregular holes using partial convolutions[C]//LNCS 11215: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Berlin, Heidelberg: Springer, 2018: 89-105.
[12] YU J, LIN Z, YANG J, et al. Free-form image inpainting with gated convolution[J]. arXiv:1806.03589, 2018.
[13] NAZERI K, NG E, JOSEPH T, et al. Edgeconnect: generative image inpainting with adversarial edge learning[J]. arXiv: 1901.00212, 2019.
[14] JO Y, PARK J. SC-FEGAN: face editing generative adver-sarial network with user??s sketch and color[C]//Proceedings of the 2019 IEEE International Conference on Computer Vision, Seoul, Oct 27-Nov 2，2019. Piscataway: IEEE, 2019: 1745-1753.
[15] XIONG W, YU J H, LIN Z, et al. Foreground-aware image inpainting[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 5840-5848.
[16] ZHENG C X, CHAM T J, CAI J F. Pluralistic image com-pletion[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 1438-1447.
[17] YANG Y, GUO X, MA J, et al. LaFIn: generative landmark guided face inpainting[J]. arXiv:1911.11394, 2019.
[18] WANG K F, GOU C, DUAN Y J, et al. Generative adver-sarial networks: the state of the art and beyond[J]. Acta Auto-matica Sinica, 2017, 43(3): 321-332.
王坤峰, 苟超, 段艳杰, 等. 生成式对抗网络的研究进展与展望[J]. 自动化学报, 2017, 43(3): 321-332．
[19] WU S Q, LI X M. Survey on research progress of generating adversarial networks[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(3): 377-388.
吴少乾, 李西明. 生成对抗网络的研究进展综述[J]. 计算机科学与探索, 2020, 14(3): 377-388.
[20] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Con-ference on Computer Vision and Pattern Recognition, Las Vegas, Jun 26-Jul 1, 2016. Piscataway: IEEE, 2016: 770-778.
[21] WANG T C, LIU M Y, ZHU J Y, et al. High-resolution image synthesis and semantic manipulation with conditional GANs[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 8798-8807.
[22] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409. 1556, 2014.
[23] JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]//LNCS 9906: Pro-ceedings of the 2016 European Conference on Computer Vision, Amsterdam, Oct 8-16, 2016. Berlin, Heidelberg: Springer, 2016: 694-711.
[24] GATYS L, ECKER A S, BETHGE M. Texture synthesis using convolutional neural networks[C]//Proceedings of the Advances in Neural Information Processing Systems, Mon-treal, Dec 7-12, 2015. Red Hook: Curran Associates, 2015: 262-270.
[25] GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 26-Jul 1, 2016. Washington: IEEE Computer Society, 2016: 2414-2423.
[26] LIU Z, LUO P, WANG X, et al. Deep learning face attributes in the wild[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 11-18, 2015. Washington: IEEE Computer Society, 2015: 3730-3738.
[27] KINGMA D P, BA J. Adam: a method for stochastic opti-mization[J]. arXiv:1412.6980, 2014.
[28] ABDOU I E, PRATT W K. Quantitative design and evalua-tion of enhancement/thresholding edge detectors[J]. Procee-dings of the IEEE, 1979, 67(5): 753-763.