Research on Edge-Guided Image Repair Algorithm

doi:10.3778/j.issn.1673-9418.2009091

Journal of Frontiers of Computer Science and Technology ›› 2022, Vol. 16 ›› Issue (3): 669-682.DOI: 10.3778/j.issn.1673-9418.2009091

• Graphics and Image • Previous Articles Next Articles

Research on Edge-Guided Image Repair Algorithm

JIANG Yi¹^,², XU Jiajie¹, LIU Xu¹^,⁺(), ZHU Junwu¹

1. School of Information Engineering, Yangzhou University, Yangzhou, Jiangsu 225127, China
2. State Key Laboratory of Ocean Engineering, Shanghai Jiao Tong University, Shanghai 200030, China

Received:2020-09-01 Revised:2020-10-20 Online:2022-03-01 Published:2020-10-23
About author:JIANG Yi, born in 1974, M.S., associate professor. Her research interests include artificial intelligence, mechanism design, image processing, etc.
XU Jiajie, born in 1995, M.S. His research interests include deep learning, image inpainting, etc.
LIU Xu, born in 1990, Ph.D. candidate. Her research interests include mechanism design, artificial intelligence, intelligent software, etc.
ZHU Junwu, born in 1972, Ph.D., professor, senior member of CCF. His research interests include artificial intelligence, intelligent software, educational informationization, etc.
Supported by:
National Natural Science Foundation of China(61872313);Open Project in the State Key Laboratory of Ocean Engineering(1907);Key Research Project in Education Information in Jiangsu Province(20180012);Project of Science and Technology of Yangzhou(YZ2018209);Project of Science and Technology of Yangzhou(YZ2019133)

边缘指导图像修复算法研究

姜艺¹^,², 胥加洁¹, 柳絮¹^,⁺(), 朱俊武¹

1.扬州大学信息工程学院,江苏扬州 225127
2.上海交通大学海洋工程国家重点实验室,上海 200030

通讯作者: + E-mail: sherryliu08@foxmail.com
作者简介:姜艺（1974—）,女,上海人,硕士,副教授,主要研究方向为人工智能、机制设计、图像处理等。
胥加洁（1995—）,男,江苏盐城人,硕士,主要研究方向为深度学习、图像修补等。
柳絮（1990—),女,江苏扬州人,博士研究生,主要研究方向为机制设计、人工智能、智能软件等。
朱俊武（1972—）,男,江苏江都人,博士,教授,CCF高级会员,主要研究方向为人工智能、智能软件、教育信息化等。
基金资助:
国家自然科学基金(61872313);海洋工程国家重点实验室项目(1907);江苏省教育信息化重点课题(20180012);扬州科技计划项目(YZ2018209);扬州科技计划项目(YZ2019133)

Abstract

Abstract:

The continuous development of deep learning technology has provided new ideas for image repair research over the years, and the image repair methods can understand the semantic information of image through the study of massive image data. Although the existing image repair methods have been able to generate desirable repair results, it is insufficient to deal with the details of missing part from the image when facing the image with more complex missing part, thus the restoration results are excessively smooth or blurry, and the complex structural information that misses from the image cannot be repaired well. In order to solve the issues above, an edge-guided image repair method based on generative adversarial networks technology and the corresponding algorithm are proposed in this paper, and the repair process is divided into two stages. First, the edge repair model is trained to generate more realistic edge information of the missing area. Then, the content generation model is trained to fill in the missing content information based on the edge information that has been repaired. Lastly, the experimental verification is conducted on the CelebA dataset and ParisStreet-View dataset to compare with the Shift-Net model,deep image prior (DIP) model and field factorization machine (FFM) model, and the visual qualitative analysis and quantitative index analysis are carried out on the experimental repair results. The experimental results prove that the repair method proposed in this paper for the missing complex structure information in the image is superior to the existing methods, and also reflect the edge information plays a crucial role in image repair.

Key words: deep learning, image repair, generative adversarial networks (GAN), edge-guided

摘要：

近年来,深度学习技术的不断发展为图像修复研究提供了新的思路,通过对海量图像数据的学习,使得图像修复方法能够理解图像的语义信息。虽然现有的图像修复方法已能够生成较好的图像修复结果,但遇到结构缺失较为复杂的图像时,对缺失部分细节处理能力较差,所生成的结果会过度平滑或模糊,不能很好地修复图像缺失的复杂结构信息。针对此问题,基于生成对抗网络技术提出了一种边缘指导图像修复的方法和对应算法,将图像修复工作分为两部分：首先训练边缘修复模型生成较为真实的缺失区域的边缘信息,再根据已修复好的边缘信息,训练内容生成模型填充缺失部分的内容信息。最后所提方法在CelebA数据集和ParisStreet-View数据集上与Shift-Net模型、深度图样先验（DIP）模型以及FFM模型进行了对比实验验证,并对实验修复结果进行了视觉上的定性分析和定量指标分析。实验结果证明提出的方法相对现有方法能更好地修复图像中缺失的复杂结构信息,反映出边缘信息在图像修复过程中具有重要的作用。

关键词: 深度学习, 图像修复, 生成对抗网络（GAN）, 边缘指导

CLC Number:

TP399

JIANG Yi, XU Jiajie, LIU Xu, ZHU Junwu. Research on Edge-Guided Image Repair Algorithm[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(3): 669-682.

姜艺, 胥加洁, 柳絮, 朱俊武. 边缘指导图像修复算法研究[J]. 计算机科学与探索, 2022, 16(3): 669-682.

Figures/Tables 15

Fig.1 Whole framework of model

Fig.2 Network structure parameter diagram of edge-guided repair model

Fig.3 Network structure parameter diagram of optimization network

Fig.4 Network structure parameter diagram of content filling model

Fig.5 Samples of irregular mask datasets

Fig.6 Irregular mask repair results on CelebA datasets

Fig.7 Regular mask repair results on CelebA datasets

Fig.8 Regular mask repair results on ParisStreet-View datasets

Fig.9 Irregular mask repair results on ParisStreet-View datasets

Fig.10 Results of edge-guided image repair method

Table 1 Regular mask repair results comparison on different datasets

模型	CelebA		ParisStreet-View
模型	PSNR/dB	SSIM	PSNR/dB	SSIM
Shift-Net	24.64	0.92	26.94	0.90
DIP	23.50	0.89	24.34	0.87
FFM	22.59	0.92	24.66	0.88
Proposed	26.76	0.93	27.56	0.92

Table 2 Irregular mask repair results comparison on different datasets

模型	CelebA		ParisStreet-View
模型	PSNR/dB	SSIM	PSNR/dB	SSIM
Shift-Net	23.20	0.86	20.40	0.71
DIP	21.90	0.83	18.48	0.63
FFM	20.43	0.84	18.71	0.70
Proposed	24.98	0.88	20.73	0.73

Fig.11 Comparison curves of repair results and original image under different mask sizes

Fig.12 Repair results comparison on CelebA datasets under different mask sizes

Fig.13 Repair results comparison on ParisStreet-View datasets under different mask sizes

References 28

[1]	BERTALMÍO M, SAPIRO G, CASELLES V, et al. Image inpainting[C]// Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, Jul 23-28, 2000. New York: ACM, 2000: 417-424.
[2]	SHEN J H, CHAN T F. Mathematical models for local non-texture inpaintings[J]. SIAM Journal on Applied Mathematics, 2002, 62(3): 1019-1043. DOI URL
[3]	CHAN T F, SHEN J H. Nontexture inpainting by curvature-driven diffusions[J]. Journal of Visual Communication and Image Representation, 2001, 12(4): 436-449. DOI URL
[4]	TSAI A, YEZZI A J, WILLSKY A S. Curve evolution imple-mentation of the Mumford-Shah functional for image segmenta-tion, denoising, interpolation, and magnification[J]. IEEE Transactions on Image Processing, 2001, 10(8): 1169-1186. DOI URL
[5]	SHEN J H, KANG S H, CHAN T F. Euler’s elastica and curvature-based inpainting[J]. SIAM Journal of Applied Mathematics, 2003, 63(2): 564-592. DOI URL
[6]	ESEDOGLU S, SHEN J. Digital inpainting based on the Mumford-Shah-Euler image model[J]. European Journal of Applied Mathematics, 2002, 13(4): 353-370.
[7]	BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3): 24.
[8]	DARABI S, SHECHTMAN E, BARNES C, et al. Image melding: combining inconsistent images using patch-based synjournal[J]. ACM Transactions on Graphics, 2012, 31(4): 82.
[9]	ULYANOV D, VEDALDI A, LEMPITSKY V S. Deep image prior[C]// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 9446-9454.
[10]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Advances in Neural Information Processing Systems, 2014, 3: 2672-2680.
[11]	LIU G L, REDA F A, SHIH K J, et al. Image inpainting for irregular holes using partial convolutions[C]// LNCS 11215:Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 89-105.
[12]	BALLESTER C, BERTALMÍO M, CASELLES V, et al. Filling-in by joint interpolation of vector fields and gray levels[J]. IEEE Transactions on Image Processing, 2001, 10(8): 1200-1211. DOI URL
[13]	张桂梅, 李艳兵. 结合纹理结构的分数阶TV模型的图像修复[J]. 中国图象图形学报, 2019, 24(5): 700-713.
	ZHANG G M, LI Y B. Image inpainting of fractional TV model combined with texture structure[J]. Journal of Image and Graphics, 2019, 24(5): 700-713.
[14]	许刚, 马爽. 动态尺度块匹配约束下的链式优化图像修复研究[J]. 电子学报, 2015, 43(3): 529-535. DOI
	XU G, MA S. Image completion using dynamic-scale patch matching and layer-wise chain optimization[J]. Acta Electronica Sinica, 2015, 43(3): 529-535.
[15]	高成英, 徐仙儿, 罗燕媚, 等. 基于稀疏表示的物体图像修复[J]. 计算机学报, 2019, 42(9): 1953-1965.
	GAO C Y, XU X E, LUO Y M, et al. Object image inpainting based on sparse representation[J]. Chinese Journal of Computers, 2019, 42(9): 1953-1965.
[16]	PATHAK D, KRÄHENBÜHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 2536-2544.
[17]	IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1-14.
[18]	SONG Y H, CHAO Y, SHEN Y J, et al. SPG-Net: segmenta-tion prediction and guidance network for image inpainting[C]// Proceedings of the 2018 British Machine Vision Conference, Newcastle, Sep 3-6, 2018. Durham: BMVA Press, 2018: 97.
[19]	YANG C, LU X, LIN Z, et al. High-resolution image inpainting using multi-scale neural patch synconfproc[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 4076-4084.
[20]	YEH R A, CHEN C, LIM T Y, et al. Semantic image inpainting with deep generative models[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 6882-6890.
[21]	LI Y J, LIU S F, YANG J M, et al. Generative face complet-ion[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 5892-5900.
[22]	DEMIR U, ÜNAL G B. Patch-based image inpainting with generative adversarial networks[J]. arXiv:1803.07422, 2018.
[23]	LIU Z W, LUO P, WANG X G, et al. Deep learning face attributes in the wild[C]// Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015. Washington: IEEE Computer Society, 2015: 3730-3738.
[24]	DOERSCH C, SINGH S, GUPTA A, et al. What makes Paris look like Paris[J]. Communications of the ACM, 2015, 58(12): 103-110.
[25]	PÉREZ P, GANGNET M, BLAKE A, et al. Poisson image editing[J]. ACM Transactions on Graphics, 2003, 22(3): 313-318. DOI URL
[26]	ZEILER M D. Adadelta: an adaptive learning rate method[J]. arXiv:1212.5701, 2012.
[27]	YAN Z Y, LI X M, LI M, et al. Shift-Net: image inpainting via deep feature rearrangement[C]// LNCS 11218: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 3-19.
[28]	TELEA A C. An image inpainting technique based on the fast marching method[J]. Journal of Graphics Tools, 2004, 9(1): 23-34. DOI URL

Research on Edge-Guided Image Repair Algorithm

边缘指导图像修复算法研究

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 15

References 28

Related Articles 15

Recommended Articles

Metrics

[1]	AN Fengping, LI Xiaowei, CAO Xiang. Medical Image Classification Algorithm Based on Weight Initialization-Sliding Window CNN [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(8): 1885-1897.
[2]	ZENG Fanzhi, XU Luqian, ZHOU Yan, ZHOU Yuexia, LIAO Junwei. Review of Knowledge Tracing Model for Intelligent Education [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(8): 1742-1763.
[3]	LIU Yi, LI Mengmeng, ZHENG Qibin, QIN Wei, REN Xiaoguang. Survey on Video Object Tracking Algorithms [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(7): 1504-1515.
[4]	ZHAO Xiaoming, YANG Yijiao, ZHANG Shiqing. Survey of Deep Learning Based Multimodal Emotion Recognition [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(7): 1479-1503.
[5]	XIA Hongbin, XIAO Yifei, LIU Yuan. Long Text Generation Adversarial Network Model with Self-Attention Mechanism [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(7): 1603-1610.
[6]	SUN Fangwei, LI Chengyang, XIE Yongqiang, LI Zhongbo, YANG Caidong, QI Jin. Review of Deep Learning Applied to Occluded Object Detection [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(6): 1243-1259.
[7]	LIU Yafen, ZHENG Yifeng, JIANG Lingyi, LI Guohe, ZHANG Wenjie. Survey on Pseudo-Labeling Methods in Deep Semi-supervised Learning [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(6): 1279-1290.
[8]	SHEN Ruicai, ZHAI Junhai, HOU Yingzhen. Multi-discriminator Generative Adversarial Networks Based on Selective Ensemble Learning [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(6): 1429-1438.
[9]	LIN Jiawei, WANG Shitong. Deep Adversarial-Reconstruction-Classification Networks for Unsupervised Domain Adaptation [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(5): 1107-1116.
[10]	CHENG Weiyue, ZHANG Xueqin, LIN Kezheng, LI Ao. Deep Convolutional Neural Network Algorithm Fusing Global and Local Features [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(5): 1146-1154.
[11]	ZHONG Mengyuan, JIANG Lin. Review of Super-Resolution Image Reconstruction Algorithms [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(5): 972-990.
[12]	PEI Lishen, ZHAO Xuezhuan. Survey of Collective Activity Recognition Based on Deep Learning [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 775-790.
[13]	XU Jia, WEI Tingting, YU Ge, HUANG Xinyue, LYU Pin. Review of Question Difficulty Evaluation Approaches [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 734-759.
[14]	ZHU Weijie, CHEN Ying. Micro-expression Recognition Convolutional Network for Dual-stream Temporal-Domain Information Interaction [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 950-958.
[15]	ZHANG Quangui, HU Jiayan, WANG Li. One Class Collaborative Filtering Recommendation Algorithm Coupled with User Common Characteristics [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(3): 637-648.