边缘指导图像修复算法研究

doi:10.3778/j.issn.1673-9418.2009091

计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (3): 669-682.DOI: 10.3778/j.issn.1673-9418.2009091

边缘指导图像修复算法研究

姜艺¹^,², 胥加洁¹, 柳絮¹^,⁺(), 朱俊武¹

1.扬州大学信息工程学院,江苏扬州 225127
2.上海交通大学海洋工程国家重点实验室,上海 200030

收稿日期:2020-09-01 修回日期:2020-10-20 出版日期:2022-03-01 发布日期:2020-10-23
通讯作者: + E-mail: sherryliu08@foxmail.com
作者简介:姜艺（1974—）,女,上海人,硕士,副教授,主要研究方向为人工智能、机制设计、图像处理等。
胥加洁（1995—）,男,江苏盐城人,硕士,主要研究方向为深度学习、图像修补等。
柳絮（1990—),女,江苏扬州人,博士研究生,主要研究方向为机制设计、人工智能、智能软件等。
朱俊武（1972—）,男,江苏江都人,博士,教授,CCF高级会员,主要研究方向为人工智能、智能软件、教育信息化等。
基金资助:
国家自然科学基金(61872313);海洋工程国家重点实验室项目(1907);江苏省教育信息化重点课题(20180012);扬州科技计划项目(YZ2018209);扬州科技计划项目(YZ2019133)

Research on Edge-Guided Image Repair Algorithm

JIANG Yi¹^,², XU Jiajie¹, LIU Xu¹^,⁺(), ZHU Junwu¹

1. School of Information Engineering, Yangzhou University, Yangzhou, Jiangsu 225127, China
2. State Key Laboratory of Ocean Engineering, Shanghai Jiao Tong University, Shanghai 200030, China

Received:2020-09-01 Revised:2020-10-20 Online:2022-03-01 Published:2020-10-23
About author:JIANG Yi, born in 1974, M.S., associate professor. Her research interests include artificial intelligence, mechanism design, image processing, etc.
XU Jiajie, born in 1995, M.S. His research interests include deep learning, image inpainting, etc.
LIU Xu, born in 1990, Ph.D. candidate. Her research interests include mechanism design, artificial intelligence, intelligent software, etc.
ZHU Junwu, born in 1972, Ph.D., professor, senior member of CCF. His research interests include artificial intelligence, intelligent software, educational informationization, etc.
Supported by:
National Natural Science Foundation of China(61872313);Open Project in the State Key Laboratory of Ocean Engineering(1907);Key Research Project in Education Information in Jiangsu Province(20180012);Project of Science and Technology of Yangzhou(YZ2018209);Project of Science and Technology of Yangzhou(YZ2019133)

摘要/Abstract

摘要：

近年来,深度学习技术的不断发展为图像修复研究提供了新的思路,通过对海量图像数据的学习,使得图像修复方法能够理解图像的语义信息。虽然现有的图像修复方法已能够生成较好的图像修复结果,但遇到结构缺失较为复杂的图像时,对缺失部分细节处理能力较差,所生成的结果会过度平滑或模糊,不能很好地修复图像缺失的复杂结构信息。针对此问题,基于生成对抗网络技术提出了一种边缘指导图像修复的方法和对应算法,将图像修复工作分为两部分：首先训练边缘修复模型生成较为真实的缺失区域的边缘信息,再根据已修复好的边缘信息,训练内容生成模型填充缺失部分的内容信息。最后所提方法在CelebA数据集和ParisStreet-View数据集上与Shift-Net模型、深度图样先验（DIP）模型以及FFM模型进行了对比实验验证,并对实验修复结果进行了视觉上的定性分析和定量指标分析。实验结果证明提出的方法相对现有方法能更好地修复图像中缺失的复杂结构信息,反映出边缘信息在图像修复过程中具有重要的作用。

关键词: 深度学习, 图像修复, 生成对抗网络（GAN）, 边缘指导

Abstract:

The continuous development of deep learning technology has provided new ideas for image repair research over the years, and the image repair methods can understand the semantic information of image through the study of massive image data. Although the existing image repair methods have been able to generate desirable repair results, it is insufficient to deal with the details of missing part from the image when facing the image with more complex missing part, thus the restoration results are excessively smooth or blurry, and the complex structural information that misses from the image cannot be repaired well. In order to solve the issues above, an edge-guided image repair method based on generative adversarial networks technology and the corresponding algorithm are proposed in this paper, and the repair process is divided into two stages. First, the edge repair model is trained to generate more realistic edge information of the missing area. Then, the content generation model is trained to fill in the missing content information based on the edge information that has been repaired. Lastly, the experimental verification is conducted on the CelebA dataset and ParisStreet-View dataset to compare with the Shift-Net model,deep image prior (DIP) model and field factorization machine (FFM) model, and the visual qualitative analysis and quantitative index analysis are carried out on the experimental repair results. The experimental results prove that the repair method proposed in this paper for the missing complex structure information in the image is superior to the existing methods, and also reflect the edge information plays a crucial role in image repair.

Key words: deep learning, image repair, generative adversarial networks (GAN), edge-guided

中图分类号:

TP399

姜艺, 胥加洁, 柳絮, 朱俊武. 边缘指导图像修复算法研究[J]. 计算机科学与探索, 2022, 16(3): 669-682.

JIANG Yi, XU Jiajie, LIU Xu, ZHU Junwu. Research on Edge-Guided Image Repair Algorithm[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(3): 669-682.

图/表 15

图1 整体模型框架

Fig.1 Whole framework of model

图2 边缘修复模型网络结构参数图

Fig.2 Network structure parameter diagram of edge-guided repair model

图3 优化网络的网络结构参数图

Fig.3 Network structure parameter diagram of optimization network

图4 内容填充模型网络结构参数图

Fig.4 Network structure parameter diagram of content filling model

图5 不规则掩膜数据集样本

Fig.5 Samples of irregular mask datasets

图6 CelebA数据集上不规则掩膜修复结果

Fig.6 Irregular mask repair results on CelebA datasets

图7 CelebA数据集上规则掩膜修复结果

Fig.7 Regular mask repair results on CelebA datasets

图8 ParisStreet-View数据集上规则掩膜修复结果

Fig.8 Regular mask repair results on ParisStreet-View datasets

图9 ParisStreet-View数据集上不规则掩膜修复结果

Fig.9 Irregular mask repair results on ParisStreet-View datasets

图10 边缘指导图像修复方法结果展示

Fig.10 Results of edge-guided image repair method

表1 不同数据集上规则掩膜修复结果对比

Table 1 Regular mask repair results comparison on different datasets

模型	CelebA		ParisStreet-View
模型	PSNR/dB	SSIM	PSNR/dB	SSIM
Shift-Net	24.64	0.92	26.94	0.90
DIP	23.50	0.89	24.34	0.87
FFM	22.59	0.92	24.66	0.88
Proposed	26.76	0.93	27.56	0.92

表2 不同数据集上不规则掩膜修复结果对比

Table 2 Irregular mask repair results comparison on different datasets

模型	CelebA		ParisStreet-View
模型	PSNR/dB	SSIM	PSNR/dB	SSIM
Shift-Net	23.20	0.86	20.40	0.71
DIP	21.90	0.83	18.48	0.63
FFM	20.43	0.84	18.71	0.70
Proposed	24.98	0.88	20.73	0.73

图11 不同掩膜下修复结果与原图的对比曲线

Fig.11 Comparison curves of repair results and original image under different mask sizes

图12 CelebA数据集不同掩膜大小下修复结果对比

Fig.12 Repair results comparison on CelebA datasets under different mask sizes

图13 ParisStreet-View数据集不同掩膜大小下修复结果对比图

Fig.13 Repair results comparison on ParisStreet-View datasets under different mask sizes

参考文献 28

[1]	BERTALMÍO M, SAPIRO G, CASELLES V, et al. Image inpainting[C]// Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, Jul 23-28, 2000. New York: ACM, 2000: 417-424.
[2]	SHEN J H, CHAN T F. Mathematical models for local non-texture inpaintings[J]. SIAM Journal on Applied Mathematics, 2002, 62(3): 1019-1043. DOI URL
[3]	CHAN T F, SHEN J H. Nontexture inpainting by curvature-driven diffusions[J]. Journal of Visual Communication and Image Representation, 2001, 12(4): 436-449. DOI URL
[4]	TSAI A, YEZZI A J, WILLSKY A S. Curve evolution imple-mentation of the Mumford-Shah functional for image segmenta-tion, denoising, interpolation, and magnification[J]. IEEE Transactions on Image Processing, 2001, 10(8): 1169-1186. DOI URL
[5]	SHEN J H, KANG S H, CHAN T F. Euler’s elastica and curvature-based inpainting[J]. SIAM Journal of Applied Mathematics, 2003, 63(2): 564-592. DOI URL
[6]	ESEDOGLU S, SHEN J. Digital inpainting based on the Mumford-Shah-Euler image model[J]. European Journal of Applied Mathematics, 2002, 13(4): 353-370.
[7]	BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3): 24.
[8]	DARABI S, SHECHTMAN E, BARNES C, et al. Image melding: combining inconsistent images using patch-based synjournal[J]. ACM Transactions on Graphics, 2012, 31(4): 82.
[9]	ULYANOV D, VEDALDI A, LEMPITSKY V S. Deep image prior[C]// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 9446-9454.
[10]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Advances in Neural Information Processing Systems, 2014, 3: 2672-2680.
[11]	LIU G L, REDA F A, SHIH K J, et al. Image inpainting for irregular holes using partial convolutions[C]// LNCS 11215:Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 89-105.
[12]	BALLESTER C, BERTALMÍO M, CASELLES V, et al. Filling-in by joint interpolation of vector fields and gray levels[J]. IEEE Transactions on Image Processing, 2001, 10(8): 1200-1211. DOI URL
[13]	张桂梅, 李艳兵. 结合纹理结构的分数阶TV模型的图像修复[J]. 中国图象图形学报, 2019, 24(5): 700-713.
	ZHANG G M, LI Y B. Image inpainting of fractional TV model combined with texture structure[J]. Journal of Image and Graphics, 2019, 24(5): 700-713.
[14]	许刚, 马爽. 动态尺度块匹配约束下的链式优化图像修复研究[J]. 电子学报, 2015, 43(3): 529-535. DOI
	XU G, MA S. Image completion using dynamic-scale patch matching and layer-wise chain optimization[J]. Acta Electronica Sinica, 2015, 43(3): 529-535.
[15]	高成英, 徐仙儿, 罗燕媚, 等. 基于稀疏表示的物体图像修复[J]. 计算机学报, 2019, 42(9): 1953-1965.
	GAO C Y, XU X E, LUO Y M, et al. Object image inpainting based on sparse representation[J]. Chinese Journal of Computers, 2019, 42(9): 1953-1965.
[16]	PATHAK D, KRÄHENBÜHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 2536-2544.
[17]	IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1-14.
[18]	SONG Y H, CHAO Y, SHEN Y J, et al. SPG-Net: segmenta-tion prediction and guidance network for image inpainting[C]// Proceedings of the 2018 British Machine Vision Conference, Newcastle, Sep 3-6, 2018. Durham: BMVA Press, 2018: 97.
[19]	YANG C, LU X, LIN Z, et al. High-resolution image inpainting using multi-scale neural patch synconfproc[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 4076-4084.
[20]	YEH R A, CHEN C, LIM T Y, et al. Semantic image inpainting with deep generative models[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 6882-6890.
[21]	LI Y J, LIU S F, YANG J M, et al. Generative face complet-ion[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 5892-5900.
[22]	DEMIR U, ÜNAL G B. Patch-based image inpainting with generative adversarial networks[J]. arXiv:1803.07422, 2018.
[23]	LIU Z W, LUO P, WANG X G, et al. Deep learning face attributes in the wild[C]// Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015. Washington: IEEE Computer Society, 2015: 3730-3738.
[24]	DOERSCH C, SINGH S, GUPTA A, et al. What makes Paris look like Paris[J]. Communications of the ACM, 2015, 58(12): 103-110.
[25]	PÉREZ P, GANGNET M, BLAKE A, et al. Poisson image editing[J]. ACM Transactions on Graphics, 2003, 22(3): 313-318. DOI URL
[26]	ZEILER M D. Adadelta: an adaptive learning rate method[J]. arXiv:1212.5701, 2012.
[27]	YAN Z Y, LI X M, LI M, et al. Shift-Net: image inpainting via deep feature rearrangement[C]// LNCS 11218: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 3-19.
[28]	TELEA A C. An image inpainting technique based on the fast marching method[J]. Journal of Graphics Tools, 2004, 9(1): 23-34. DOI URL

边缘指导图像修复算法研究

Research on Edge-Guided Image Repair Algorithm

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 15

参考文献 28

相关文章 15

编辑推荐

Metrics

[1]	安凤平, 李晓薇, 曹翔. 权重初始化-滑动窗口CNN的医学图像分类[J]. 计算机科学与探索, 2022, 16(8): 1885-1897.
[2]	曾凡智, 许露倩, 周燕, 周月霞, 廖俊玮. 面向智慧教育的知识追踪模型研究综述[J]. 计算机科学与探索, 2022, 16(8): 1742-1763.
[3]	刘艺, 李蒙蒙, 郑奇斌, 秦伟, 任小广. 视频目标跟踪算法综述[J]. 计算机科学与探索, 2022, 16(7): 1504-1515.
[4]	赵小明, 杨轶娇, 张石清. 面向深度学习的多模态情感识别研究进展[J]. 计算机科学与探索, 2022, 16(7): 1479-1503.
[5]	夏鸿斌, 肖奕飞, 刘渊. 融合自注意力机制的长文本生成对抗网络模型[J]. 计算机科学与探索, 2022, 16(7): 1603-1610.
[6]	孙方伟, 李承阳, 谢永强, 李忠博, 杨才东, 齐锦. 深度学习应用于遮挡目标检测算法综述[J]. 计算机科学与探索, 2022, 16(6): 1243-1259.
[7]	刘雅芬, 郑艺峰, 江铃燚, 李国和, 张文杰. 深度半监督学习中伪标签方法综述[J]. 计算机科学与探索, 2022, 16(6): 1279-1290.
[8]	申瑞彩, 翟俊海, 侯璎真. 选择性集成学习多判别器生成对抗网络[J]. 计算机科学与探索, 2022, 16(6): 1429-1438.
[9]	林佳伟, 王士同. 用于无监督域适应的深度对抗重构分类网络[J]. 计算机科学与探索, 2022, 16(5): 1107-1116.
[10]	程卫月, 张雪琴, 林克正, 李骜. 融合全局与局部特征的深度卷积神经网络算法[J]. 计算机科学与探索, 2022, 16(5): 1146-1154.
[11]	钟梦圆, 姜麟. 超分辨率图像重建算法综述[J]. 计算机科学与探索, 2022, 16(5): 972-990.
[12]	裴利沈, 赵雪专. 群体行为识别深度学习方法研究综述[J]. 计算机科学与探索, 2022, 16(4): 775-790.
[13]	许嘉, 韦婷婷, 于戈, 黄欣悦, 吕品. 题目难度评估方法研究综述[J]. 计算机科学与探索, 2022, 16(4): 734-759.
[14]	朱伟杰, 陈莹. 双流时间域信息交互的微表情识别卷积网络[J]. 计算机科学与探索, 2022, 16(4): 950-958.
[15]	张全贵, 胡嘉燕, 王丽. 耦合用户公共特征的单类协同过滤推荐算法[J]. 计算机科学与探索, 2022, 16(3): 637-648.