结合语义先验和深度注意力残差的图像修复

doi:10.3778/j.issn.1673-9418.2208014

摘要/Abstract

摘要： 针对现有图像修复方法修复结果缺乏真实性、未灵活处理缺失区域和未缺失区域信息以及未有效处理不同阶段的图像特征信息等问题，提出结合语义先验和深度注意力残差组的图像修复方法。该图像修复方法主要由语义先验网络、深度注意力残差组与全尺度跳跃连接组成。语义先验网络学习缺失区域视觉元素的完整语义先验信息，利用学习到的语义信息对缺失区域进行补全。深度注意力残差组使生成器不仅能更加关注图像的缺失区域，而且能自适应地学习各个通道的特征。全尺度跳跃连接则可以将包含图像边界的低层次特征图与包含图像纹理与细节的高层次特征图结合起来对图像缺失区域进行修复。在CelebA-HQ数据集与Paris Street View数据集上进行了充分对比实验，实验结果表明，该方法优于当前代表性先进图像修复方法。

关键词: 图像修复, 编码解码, 语义先验, 注意力机制, 跳跃连接

Abstract: To overcome the shortcomings of existing image inpainting methods, such as the lack of authenticity in the inpainting results, the lack of effective processing of missing region and non-missing region information, and the lack of effective processing of image feature information in different stages, an image inpainting method combining semantic priors and deep attention residual group is proposed. The image inpainting network is mainly composed of semantic priors network, deep attention residual group and full-scale skip connection. The semantic priors network learns the complete semantic priors information of visual elements in the missing region, and uses the learned semantic information to complete the missing region. The deep attention residual group enables the generator not only to pay more attention to the missing area of the image, but also to learn the features of each channel adaptively. The full-scale skip connection can combine the low-level feature map containing the image boundary with the high-level feature map containing the image texture and detail to inpaint the missing area of the image. In this paper, a full comparison experiment is conducted on CelebA-HQ dataset and Paris Street View dataset, and the experimental results show that the proposed method is superior to the current representative advanced image inpainting methods.

Key words: image inpainting, encoder and decoder, semantic priors, attention mechanism, skip connection

陈晓雷, 杨佳, 梁其铎. 结合语义先验和深度注意力残差的图像修复[J]. 计算机科学与探索, 2023, 17(10): 2450-2461.

CHEN Xiaolei, YANG Jia, LIANG Qiduo. Image Inpainting Combining Semantic Priors and Deep Attention Residuals[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(10): 2450-2461.

参考文献

[1] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Advances in Neural Inform-ation Processing Systems 27, Montreal, Dec 8-13, 2014: 2672-2680.
[2] NAZERI K, NG E, JOSEPH T, et al. Edgeconnect: gener-ative image inpainting with adversarial edge learning[J].arXiv:1901.00212, 2019.
[3] XIONG W, YU J, LIN Z, et al. Foreground-aware image inpainting[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 5840-5848.
[4] 李海燕, 王伟华, 郭磊, 等. BDCN和U-net边缘生成两阶段修复算法[J]. 北京邮电大学学报, 2021, 44(5): 121-126.
LI H Y, WANG W H, GUO L, et al. Two-stage network inpainting algorithm based on BDCN and U-net edge gene-ration[J]. Journal of Beijing University of Posts and Tele-communications, 2021, 44(5): 121-126.
[5] RONNEBERGER O, FISCHER P, BROX T. U-Net: convo-lutional networks for biomedical image segmentation[C]//LNCS 9351: Proceedings of the 18th International Conf-erence on Medical Image Computing and Computer Assisted Intervention, Munich, Oct 5-9, 2015. Cham: Springer, 2015: 234-241.
[6] 胡凯, 赵健, 刘昱, 等. 结构引导的图像修复[J]. 北京航空航天大学学报, 2022, 48(7): 1269-1277.
HU K, ZHAO J, LIU Y, et al. Images inpainting via struc-ture guidance[J]. Journal of Beijing University of Aerona-utics and Astronautics, 2022, 48(7): 1269-1277.
[7] LI J, HE F, ZHANG L, et al. Progressive reconstruction of visual structure for image inpainting[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 5961-5970.
[8] SONG Y, YANG C, SHEN Y, et al. SPG-Net: segmentation prediction and guidance network for image inpainting[J]. arXiv:1805.03356, 2018.
[9] LIU H, JIANG B, XIAO Y, et al. Coherent semantic atten-tion for image inpainting[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 4170-4179.
[10] LIU H, WAN Z, HUANG W, et al. DeFLOCNet: deep image editing via flexible low-level controls[C]//Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 10765- 10774.
[11] ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//LNCS 11211: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer,2018: 294-310.
[12] ZHANG Y, LI K, LI K, et al. Residual non-local attention networks for image restoration[J]. arXiv:1903.10082, 2019.
[13] MAO X, SHEN C, YANG Y B. Image restoration using very deep convolutional encoder-decoder networks with sym-metric skip connections[C]//Advances in Neural Information Processing Systems 29, Barcelona, Dec 5-10, 2016: 2802-2810.
[14] 杨文霞, 王萌, 张亮. 基于密集连接块U-Net的语义人脸图像修复[J]. 计算机应用, 2020, 40(12): 3651-3657.
YANG W X, WANG M, ZHANG L. Semantic face image inpainting based on U-Net with dense blocks[J]. Journal of Computer Applications, 2020, 40(12): 3651-3657.
[15] MIYATO T, KATAOKA T, KOYAMA M, et al. Spectral normalization for generative adversarial networks[J]. arXiv:1802.05957, 2018.
[16] DEMIR U, UNAL G. Patch-based image inpainting with generative adversarial networks[J]. arXiv:1803.07422, 2018.
[17] PARK T, LIU M Y, WANG T C, et al. Semantic image synthesis with spatially-adaptive normalization[C]//Proce-edings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Pisc-ataway: IEEE, 2019: 2337-2346.
[18] ULYANOV D, VEDALDI A, LEMPITSKY V. Instance normalization: the missing ingredient for fast stylization[J].arXiv:1607.08022, 2016.
[19] HOU Q, ZHOU D, FENG J. Coordinate attention for efficient mobile network design[C]//Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 13713- 13722.
[20] KARRAS T, AILA T, LAINE S, et al. Progressive growing of GANs for improved quality, stability, and variation[J]. arXiv:1710.10196, 2017.
[21] DOERSCH C, SINGH S, GUPTA A, et al. What makes paris look like paris?[J]. ACM Transactions on Graphics, 2012, 31(4): 1-9.
[22] WANG N, ZHANG Y, ZHANG L. Dynamic selection net-work for image inpainting[J]. IEEE Transactions on Image Processing, 2021, 30: 1784-1798.
[23] LI J, WANG N, ZHANG L, et al. Recurrent feature reaso-ning for image inpainting[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 7757-7765.
[24] YU T, GUO Z, JIN X, et al. Region normalization for image inpainting[C]//Proceedings of the 34th AAAI Confe-rence on Artificial Intelligence, the 32nd Innovative Applic-ations of Artificial Intelligence Conference, the 10th AAAI Symposium on Educational Advances in Artificial Intellig-ence, New York, Feb 7-12, 2020. Menlo Park: AAAI, 2020: 12733-12740.