Journal of Frontiers of Computer Science and Technology ›› 2022, Vol. 16 ›› Issue (3): 669-682.DOI: 10.3778/j.issn.1673-9418.2009091

• Graphics and Image • Previous Articles     Next Articles

Research on Edge-Guided Image Repair Algorithm

JIANG Yi1,2, XU Jiajie1, LIU Xu1,+(), ZHU Junwu1   

  1. 1. School of Information Engineering, Yangzhou University, Yangzhou, Jiangsu 225127, China
    2. State Key Laboratory of Ocean Engineering, Shanghai Jiao Tong University, Shanghai 200030, China
  • Received:2020-09-01 Revised:2020-10-20 Online:2022-03-01 Published:2020-10-23
  • About author:JIANG Yi, born in 1974, M.S., associate professor. Her research interests include artificial intelligence, mechanism design, image processing, etc.
    XU Jiajie, born in 1995, M.S. His research interests include deep learning, image inpainting, etc.
    LIU Xu, born in 1990, Ph.D. candidate. Her research interests include mechanism design, artificial intelligence, intelligent software, etc.
    ZHU Junwu, born in 1972, Ph.D., professor, senior member of CCF. His research interests include artificial intelligence, intelligent software, educational informationization, etc.
  • Supported by:
    National Natural Science Foundation of China(61872313);Open Project in the State Key Laboratory of Ocean Engineering(1907);Key Research Project in Education Information in Jiangsu Province(20180012);Project of Science and Technology of Yangzhou(YZ2018209);Project of Science and Technology of Yangzhou(YZ2019133)

边缘指导图像修复算法研究

姜艺1,2, 胥加洁1, 柳絮1,+(), 朱俊武1   

  1. 1.扬州大学 信息工程学院,江苏 扬州 225127
    2.上海交通大学 海洋工程国家重点实验室,上海 200030
  • 通讯作者: + E-mail: sherryliu08@foxmail.com
  • 作者简介:姜艺(1974—),女,上海人,硕士,副教授,主要研究方向为人工智能、机制设计、图像处理等。
    胥加洁(1995—),男,江苏盐城人,硕士,主要研究方向为深度学习、图像修补等。
    柳絮(1990—),女,江苏扬州人,博士研究生,主要研究方向为机制设计、人工智能、智能软件等。
    朱俊武(1972—),男,江苏江都人,博士,教授,CCF高级会员,主要研究方向为人工智能、智能软件、教育信息化等。
  • 基金资助:
    国家自然科学基金(61872313);海洋工程国家重点实验室项目(1907);江苏省教育信息化重点课题(20180012);扬州科技计划项目(YZ2018209);扬州科技计划项目(YZ2019133)

Abstract:

The continuous development of deep learning technology has provided new ideas for image repair research over the years, and the image repair methods can understand the semantic information of image through the study of massive image data. Although the existing image repair methods have been able to generate desirable repair results, it is insufficient to deal with the details of missing part from the image when facing the image with more complex missing part, thus the restoration results are excessively smooth or blurry, and the complex structural information that misses from the image cannot be repaired well. In order to solve the issues above, an edge-guided image repair method based on generative adversarial networks technology and the corresponding algorithm are proposed in this paper, and the repair process is divided into two stages. First, the edge repair model is trained to generate more realistic edge information of the missing area. Then, the content generation model is trained to fill in the missing content information based on the edge information that has been repaired. Lastly, the experimental verification is conducted on the CelebA dataset and ParisStreet-View dataset to compare with the Shift-Net model,deep image prior (DIP) model and field factorization machine (FFM) model, and the visual qualitative analysis and quantitative index analysis are carried out on the experimental repair results. The experimental results prove that the repair method proposed in this paper for the missing complex structure information in the image is superior to the existing methods, and also reflect the edge information plays a crucial role in image repair.

Key words: deep learning, image repair, generative adversarial networks (GAN), edge-guided

摘要:

近年来,深度学习技术的不断发展为图像修复研究提供了新的思路,通过对海量图像数据的学习,使得图像修复方法能够理解图像的语义信息。虽然现有的图像修复方法已能够生成较好的图像修复结果,但遇到结构缺失较为复杂的图像时,对缺失部分细节处理能力较差,所生成的结果会过度平滑或模糊,不能很好地修复图像缺失的复杂结构信息。针对此问题,基于生成对抗网络技术提出了一种边缘指导图像修复的方法和对应算法,将图像修复工作分为两部分:首先训练边缘修复模型生成较为真实的缺失区域的边缘信息,再根据已修复好的边缘信息,训练内容生成模型填充缺失部分的内容信息。最后所提方法在CelebA数据集和ParisStreet-View数据集上与Shift-Net模型、深度图样先验(DIP)模型以及FFM模型进行了对比实验验证,并对实验修复结果进行了视觉上的定性分析和定量指标分析。实验结果证明提出的方法相对现有方法能更好地修复图像中缺失的复杂结构信息,反映出边缘信息在图像修复过程中具有重要的作用。

关键词: 深度学习, 图像修复, 生成对抗网络(GAN), 边缘指导

CLC Number: