Review of Application of Generative Adversarial Networks in Image Restoration

doi:10.3778/j.issn.1673-9418.2307073

Abstract

Abstract: With the rapid development of generative adversarial networks, many image restoration problems that are difficult to solve based on traditional methods have gained new research approaches. With its powerful generation ability, generative adversarial networks can restore intact images from damaged images, so they are widely used in image restoration. In order to summarize the relevant theories and research on the problem of using generative adversarial networks to repair damaged images in recent years, based on the categories of damaged images and their adapted repair methods, the applications of image restoration are divided into three main aspects: image inpainting, image deblurring, and image denoising. For each aspect, the applications are further subdivided through technical principles, application objects and other dimensions. For the field of image inpainting, different image completion methods based on generative adversarial networks are discussed from the perspectives of using conditional guidance and latent coding. For the field of image deblurring, the essential differences between motion blurred images and static blurred images and their repair methods are explained. For the field of image denoising, personalized denoising methods for different categories of images are summarized. For each type of applications, the characteristics of the specific GAN models employed are summarized. Finally, the advantages and disadvantages of GAN applied to image restoration are summarized, and the future application scenarios are prospected.

Key words: image restoration, generative adversarial network, image inpainting, image deblurring, image denoising

摘要： 随着生成对抗网络的迅猛发展，许多基于传统方法难以较好解决的图像修复问题获得了新的研究途径。生成对抗网络凭借强大的生成能力，能从受损图像中恢复出完好的图像，故而在图像修复中得到较为广泛的应用。总结了近年来利用生成对抗网络修复受损图像问题的相关理论与研究，以受损图像的类别及其所适配的修复方法为主要划分依据，将图像修复的应用划分为图像补全、图像去模糊、图像去噪三个主要方面。针对每一方面，通过技术原理、应用对象等维度对图像修复的应用进一步细分。对于图像补全领域，从使用条件引导与潜在编码等角度探讨了基于生成对抗网络的不同图像补全方法；对于图像去模糊领域，阐释了运动模糊图像与静态模糊图像的本质不同及其修复方法；对于图像去噪领域，归纳了不同类别图像的个性化去噪方法。同时，对于每一类应用，分析了所采用的具体生成对抗网络模型的特点及其贡献。最后，总结了生成对抗网络应用于图像修复的优势与不足，并对未来应用场景进行了展望。

关键词: 图像修复, 生成对抗网络, 图像补全, 图像去模糊, 图像去噪

GONG Ying, XU Wentao, ZHAO Ce, WANG Binjun. Review of Application of Generative Adversarial Networks in Image Restoration[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(3): 553-573.

龚颖, 许文韬, 赵策, 王斌君. 生成对抗网络在图像修复中的应用综述[J]. 计算机科学与探索, 2024, 18(3): 553-573.

References

[1] GOODFELLOW I, BENGIO Y, COURVILLE A. Deep learning[M]. Cambridge: MIT Press, 2016: 331-360.
[2] HINTON G E, OSINDERO S, TEH Y W. A fast learning algorithm for deep belief nets[J]. Neural Computation, 2006, 18(7): 1527-1554.
[3] KINGMA D P, WELLING M. Auto-encoding variational Bayes[J]. arXiv:1312.6114, 2013.
[4] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, Dec 8-13, 2014. Cambridge: MIT Press, 2014: 2672-2680.
[5] RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[J]. arXiv:1511.06434, 2015.
[6] LECUN Y, BOTTOU L, BENGGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
[7] MIRZA M, OSINDERO S. Conditional generative adversarial nets[J]. arXiv:1411.1784, 2014.
[8] ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein generative adversarial networks[C]//Proceedings of the 34th International Conference on Machine Learning, Sydney, Aug 6, 2017: 214-223.
[9] MAO X, LI Q, XIE H, et al. Least squares generative adversarial networks[J]. arXiv:1611.04076, 2016.
[10] CHEN X, DUAN Y, HOUTHOOFT R, et al. InfoGAN: interpretable representation learning by information maximizing generative adversarial nets[C]//Advances in Neural Information Processing Systems 29, 2016: 2180-2188.
[11] 陈佛计, 朱枫, 吴清潇, 等. 生成对抗网络及其在图像生成中的应用研究综述[J]. 计算机学报, 2021, 44(2): 347-369.
CHEN F J, ZHU F, WU Q X, et al. A survey about image generation with generative adversarial nets[J]. Chinese Journal of Computers, 2021, 44(2): 347-369.
[12] BAO J, CHEN D, WEN F, et al. CVAE-GAN: fine-grained image generation through asymmetric training[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Piscataway: IEEE, 2017: 2745-2754.
[13] ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Venice, Oct 22-29, 2017. Piscataway: IEEE, 2017: 1125-1134.
[14] RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th Medical Image Computing and Computer-Assisted Intervention, Munich, Oct 5-9, 2015. Berlin: Springer International Publishing, 2015: 234-241.
[15] ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Piscataway: IEEE, 2017: 2223-2232.
[16] IIZUKA S, SIMOSERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1-14.
[17] YANG C, LU X, LIN Z, et al. High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Piscataway: IEEE, 2017: 6721-6729.
[18] YU J, LIN Z, YANG J, et al. Generative image inpainting with contextual attention[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 5505-5514.
[19] GULRAJANI I, AHMED F, ARJOVSKY M, et al. Improved training of Wasserstein GANs[C]//Advances in Neural Information Processing Systems 30, 2017: 5767-5777.
[20] YU J, LIN Z, YANG J, et al. Free-form image inpainting with gated convolution[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 4470-4479.
[21] ZHANG H, HU Z, LUO C, et al. Semantic image inpainting with progressive generative networks[C]//Proceedings of the 2018 ACM Conference on Multimedia, Seoul, Oct 22-26, 2018. New York: ACM, 2018: 1939-1947.
[22] GRAVES A. Supervised sequence labelling with recurrent neural networks[M]. Berlin, Heidelberg: Springer, 2012: 37-45.
[23] PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 26-Jul 1, 2016. Piscataway: IEEE, 2016: 2536-2544.
[24] YEH R, CHEN C, LIM T, et al. Semantic image inpainting with deep generative models[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, Jul 21-26, 2017. Piscataway: IEEE, 2017: 6882-6890.
[25] REN K, MENG L, FAN C, et al. Least squares DCGAN based semantic image inpainting[C]//Proceedings of the 2018 5th IEEE International Conference on Cloud Computing and Intelligence Systems, Nanjing, Nov 30-Dec 2, 2018. Piscataway: IEEE, 2018: 890-894.
[26] LAHIRI A, JAIN A K, AGRAWAL S, et al. Prior guided GAN based semantic inpainting[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Piscataway: IEEE, 2020: 13696-13705.
[27] GRAVES A, FERNANDEZ S, SCHMIDHUBER. Multi-dimensional recurrent neural networks[C]//Proceedings of the 17th International Conference on Artificial Neural Networks, Porto, Sep 9-13, 2007. Berlin: Springer, 2007: 549-558．
[28] KARRAS T, LAINE S, AILA T. A style-based generator architecture for generative adversarial networks[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 4401-4410.
[29] KARRAS T, LAINE S, AITTALA M, et al. Analyzing and improving the image quality of StyleGAN[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Piscataway: IEEE, 2020: 8107-8116.
[30] YU Y, ZHANG L. High-fidelity image inpainting with GAN inversion[J]. arXiv:2208.11850v1, 2022.
[31] GU J, SHEN Y, ZHOU B. Image processing using multi-code GAN prior[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Piscataway: IEEE, 2020: 3009-3018.
[32] POIRIER-GINTER Y, LALONDE J F. Robust unsupervised StyleGAN image restoration[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, Jun 18-22, 2023. Piscataway: IEEE, 2023: 22292-22301.
[33] ZHENG C, CHAM T J, CAI J. Pluralistic image completion[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 1438-1447.
[34] ZHAO L, MO Q, LIN S, et al. UCTGAN: diverse image inpainting based on unsupervised cross-space translation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Piscataway: IEEE, 2020: 5740-5749.
[35] LIU H, WAN Z, HUANG W, et al. PD-GAN: probabilistic diverse GAN for image inpainting[C]//Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 9371-9381.
[36] YILDIRIM B, PEHLIVAN H, BATUHAN B, et al. Diverse inpainting and editing with GAN inversion[J]. arXiv: 2307.15033, 2023.
[37] KUPYN O, BUDZAN V, MYKHAILYCH M, et al. DeblurGAN: blind motion deblurring using conditional adversarial networks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 8183-8192.
[38] NAH S, HYUN K, MU L. Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, Jul 21-26, 2017. Piscataway: IEEE, 2017: 3883-3891.
[39] JOHNSON J, ALAHI A, FEI-FEI L. Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 11-14, 2016. Cham: Springer, 2016: 694-711.
[40] KUPYN O, MARTYNIUK T, WU J, et al. DeblurGAN-v2: deblurring (orders-of-magnitude) faster and better[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 8877-8886.
[41] LIN T, DOLL′AR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, Jul 21-26, 2017. Piscataway: IEEE, 2017: 2117-2125.
[42] JOLICOEUR-MARTINEAU A. The relativistic discriminator: a key element missing from standard GAN[J]. arXiv:1807.00734, 2018.
[43] KUMAR J, MASTAN I D, RAMAN S. FMD-cGAN: fast motion deblurring using conditional generative adversarial networks[C]//Proceedings of the 6th International Conference on Computer Vision and Image Processing, Rupnagar, Dec 3-5, 2021. Switzerland: Springer, 2021: 362-377.
[44] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 26-Jul 1, 2016. Piscataway: IEEE, 2016: 770-778.
[45] LIM J H, YE J C. Geometric GAN[J]. arXiv:1705.02894, 2017.
[46] HOWARD A G, ZHU M, CHEN B, et al. MobileNets: efficient convolutional neural networks for mobile vision applications[J]. arXiv:1704.04861v1, 2017.
[47] GONG G, ZHANG K. Local blurred natural image restoration based on self-reference deblurring generative adversarial networks[C]//Proceedings of the 2019 IEEE International Conference on Signal and Image Processing Applications, Kuala Lumpur, Sep 17-19, 2019. Piscataway: IEEE, 2019: 231-235.
[48] ZHANG K, LUO W, ZHONG Y, et al. Deblurring by realistic blurring[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Piscataway: IEEE, 2020: 2737-2746.
[49] ZHAO S, ZHANG Z, HONG R, et al. FCL-GAN: a lightweight and real-time baseline for unsupervised blind image deblurring[C]//Proceedings of the 30th ACM International Conference on Multimedia, Lisbon, Oct 10-14, 2022. New York: ACM, 2022: 6220-6229.
[50] ZHU H Y, PENG X, CHANDRASEKHAR V, et al. DehazeGAN: when image dehazing meets differential programming[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, Jul 13-19, 2018: 1234-1240.
[51] LI Z, ZHENG C, SHU H, et al. Dual-scale single image dehazing via neural augmentation[J]. IEEE Transactions on Image Processing, 2022, 31: 6213-6223.
[52] KOSCHMIEDER H. Theorie der horizontalen Sichtweite[M]//Beitrage zur Physik der freien Atmosphare, Wiesbaden: VS Verlag für Sozialwissenschaften, 1924: 33-53.
[53] SWAMI K, DAS S K. CANDY: conditional adversarial networks based fully end-to-end system for single image haze removal[C]//Proceedings of the 2018 24th International Conference on Pattern Recognition, Beijing, Aug 20-24, 2018. Piscataway: IEEE, 2018: 3061-3067.
[54] LI R D, PAN J S, LI Z C, et al. Single image dehazing via conditional generative adversarial network[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 8202-8211.
[55] FU M, LIU H, YU Y, et al. DW-GAN: a discrete wavelet transform GAN for nonhomogeneous dehazing[C]//Procee-dings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 203-212.
[56] DONG Y, LIU Y, ZHANG H, et al. FD-GAN: generative adversarial networks with fusion-discriminator for single image dehazing[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, Feb 7-12, 2020. Menlo Park: AAAI, 2020: 10729-10736.
[57] KUMAR A, SANATHRA M, KHARE M, et al. Orthogonal transform based generative adversarial network for image dehazing[J]. arXiv:2206.01743v1, 2022.
[58] MEHTA A, SINHA H, MANDAL M, et al. Domain-aware unsupervised hyperspectral reconstruction for aerial image dehazing[C]//Proceedings of the 2021 IEEE/CVF Winter Conference on Applications of Computer Vision, Jan 3-8, 2021. Piscataway: IEEE, 2021: 413-422.
[59] ZHANG H, SINDAGI V, PATEL V M. Image de-raining using a conditional generative adversarial network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(11): 3943-3956.
[60] JIN X, CHEN Z, LIN J, et al. Unsupervised single image deraining with self-supervised constraints[C]//Proceedings of the 2019 IEEE International Conference on Image Processing, Taipei, China, Sep 22-25, 2019. Piscataway: IEEE, 2019:2761-2765.
[61] 王照乾, 孔韦韦, 滕金宝, 等. DenseNet生成对抗网络低照度图像增强方法[J]. 计算机工程与应用, 2022, 58(8): 214-220.
WANG Z Q, KONG W W, TENG J B, et al. Low illumination image enhancement method based on DenseNet GAN[J]. Computer Engineering and Applications, 2022, 58(8): 214-220.
[62] 杨艳春, 高晓宇, 党建武, 等. 基于WEMD和生成对抗网络重建的红外与可见光图像融合[J]. 光学精密工程, 2022, 30(3): 320-330.
YANG Y C, GAO X Y, DANG J W, et al. Infrared and visible image fusion based on WEMD and generative adversarial network reconstruction[J]. Optics and Precision Engineering, 2022, 30(3): 320-330.
[63] HAN J, LI W, FANG P, et al. Blind image decomposition[C]//Proceedings of the 17th European Conference on Computer Vision, Tel Aviv, Oct 23-27, 2022. Cham: Springer, 2022: 218-237.
[64] CHEN J, CHEN J, CHAO H, et al. Image blind denoising with generative adversarial network based noise modeling[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 3155-3164.
[65] TRAN L D, NGUYEN S M, ARAI M. GAN-based noise model for denoising real images[C]//Proceedings of the 15th Asian Conference on Computer Vision, Kyoto, Nov 30-Dec 4, 2020. Berlin: Springer, 2020: 560-572.
[66] YUAN Y, LIU S, ZHANG J, et al. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 701-710.
[67] WOLTERINK J M, LEINER T, VIERGEVER M A, et al. Generative adversarial networks for noise reduction in low-dose CT[J]. IEEE Transactions on Medical Imaging, 2017, 36(12): 2536-2545.
[68] YANG Q, YAN P, ZHANG Y, et al. Low-dose CT image denoising using a generative adversarial network with Wasserstein distance and perceptual loss[J]. IEEE Transactions on Medical Imaging, 2018, 37(6): 1348-1357.
[69] HUANG Z, ZHANG J, ZHANG Y, et al. DU-GAN: generative adversarial networks with dual-domain U-Net-based discriminators for low-dose CT denoising[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 71: 1-12.
[70] SCHONFELD E, SCHIELE B, KHOREVA A. A U-Net based discriminator for generative adversarial networks[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Piscataway: IEEE, 2020: 8207-8216.
[71] YUN S, HAN D, OH S J, et al. CutMix: regularization strategy to train strong classifiers with localizable features[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 6023-6032.
[72] YI X, BABYN P. Sharpness-aware low-dose CT denoising using conditional generative adversarial network[J]. Journal of Digital Imaging, 2018, 31: 655-669.
[73] KANG E, KOO H J, YANG D H, et al. Cycle-consistent adversarial denoising network for multiphase coronary CT angiography[J]. Medical physics, 2019, 46(2): 550-562.
[74] HUANG D, SWANSON E A, LIN C P, et al. Optical coherence tomography[J]. Science, 1991, 254(5035): 1178-1181.
[75] MA Y, CHEN X, ZHU W, et al. Speckle noise reduction in optical coherence tomography images based on edge-sensitive cGAN[J]. Biomedical Optics Express, 2018, 9(11): 5129-5146.
[76] HUANG Y, LU Z, SHAO Z, et al. Simultaneous denoising and super-resolution of optical coherence tomography images based on generative adversarial network[J]. Optics Express, 2019, 27(9): 12289-12307.
[77] CHEN Z, ZENG Z, SHEN H, et al. DN-GAN: denoising generative adversarial networks for speckle noise reduction in optical coherence tomography images[J]. Biomedical Signal Processing and Control, 2020, 55: 101632.
[78] GUO A, FANG L, QI M, et al. Unsupervised denoising of optical coherence tomography images with nonlocal-generative adversarial network[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 70: 1-12.
[79] SINGH P, PRADHAN G. A new ECG denoising framework using generative adversarial network[J]. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2020, 18(2): 759-764.
[80] CHATTERJEE S, THAKUR R S, YADAV R N, et al. Review of noise removal techniques in ECG signals[J]. IET Signal Processing, 2020, 14(9): 569-590.
[81] MONAKHOVA K, RICHTER S R, WALLER L, et al. Dancing under the stars: video denoising in starlight[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 19-24, 2022. Piscataway: IEEE, 2022: 16241-16251.
[82] 鲍先富, 强赞霞, 杨关. 功能解耦和谱特征融合的雪霾消除模型[J]. 计算机工程与应用, 2023, 59(13): 211-219.
BAO X F, QIANG Z X, YANG G. Snow and haze elimination model based on function decoupling and edge feature fusion[J]. Computer Engineering and Applications, 2023, 59(13): 211-219.
[83] OHAYON G, ADRAI T, VAKSMAN G, et al. High perceptual quality image denoising with a posterior sampling CGAN[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021: 1805-1813.
[84] LUGMAYR A, DANELLJAN M, ROMERO A, et al. Repaint: inpainting using denoising diffusion probabilistic models[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 19-24, 2022. Piscataway: IEEE, 2022: 11461-11471.
[85] WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2014: 13(4): 600-612.
[86] DU Y, LI X. Perceptually optimized generative adversarial network for single image dehazing[J]. arXiv:1805.01084v1, 2018.