图像平滑结构引导的图像修复

doi:10.3778/j.issn.1673-9418.2409004

摘要/Abstract

摘要： 针对现有图像修复方法存在参数量大、计算复杂度高以及大面积受损图像修复产生结构模糊或扭曲问题，提出一种图像平滑结构引导的轻量级图像修复方法。该方法由结构重建网络、结构增强网络和纹理修复网络组成。结构重建网络重建受损的边缘保留平滑图像，生成粗糙的平滑图像。结构增强网络进一步精细化该平滑图像，增强其连贯性，为RGB图像修复提供可靠的辅助结构信息。纹理修复网络利用增强后的结构信息调节修复过程，以重建受损的RGB图像。在编码阶段，为避免不同尺度的结构信息丢失，使用辅助特征聚合模块将多尺度结构信息融合到修复过程。在解码阶段，为了在图像缺失区域合成清晰的纹理和语义结构，使用特征合成模块对输入特征图建模局部和全局上下文，从而得到更合理的修复效果。此外，引入基于对比学习的对比损失以稳定和改进模型的训练效果。在CelebA-HQ、FFHQ、Paris StreetView以及Dunhuang数据集上进行实验，实验结果表明，该方法在参数量为9×106、计算复杂度为29 GFLOPs的情况下，修复效果均优于对比方法。

关键词: 图像修复, 辅助结构, 注意力机制, 对比学习

Abstract: To address the existing challenges in image inpainting methods, such as large parameter count, high computational complexity, and the issue of structural blurring or distortion in the restoration of severely damaged images, a lightweight image inpainting method guided by image smoothing structures is proposed. This method primarily comprises a structure reconstruction network, a structure enhancement network, and a texture restoration network. Firstly, the structure reconstruction network restores the damaged, edge-preserving smooth image to generate a rough smooth image. Then, the structure enhancement network further refines this smooth image, enhancing its coherence and providing reliable structural guidance for RGB image restoration. Finally, the texture restoration network utilizes the enhanced structural information to regulate the restoration process and reconstruct the damaged RGB image. In the encoding stage, to avoid the loss of structural information at different scales, an auxiliary feature aggregation module is used to fuse multi-scale structural information into the inpainting process. In the decoding stage, a feature synthesis module is employed to model both local and global contexts of the input feature maps, ensuring clearer textures and semantic structures in the missing regions of the image. This approach results in a more reasonable and coherent restoration. Additionally, contrast loss based on contrastive learning is introduced to stabilize and improve the model??s training performance. Experiments conducted on the CelebA-HQ, FFHQ, Paris StreetView, and Dunhuang datasets demonstrate that the proposed method outperforms existing methods in terms of restoration quality, with a parameter count of 9×106 and computational complexity of 29 GFLOPs.

Key words: image inpainting, auxiliary structures, attention mechanism, contrastive learning

石计亮, 张乾, 杨思红, 刘霜, 滕林, 柏武贰. 图像平滑结构引导的图像修复[J]. 计算机科学与探索, 2025, 19(8): 2149-2160.

SHI Jiliang, ZHANG Qian, YANG Sihong, LIU Shuang, TENG Lin, BAI Wuer. Image Inpainting Guided by Image Smoothness Structure[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(8): 2149-2160.

参考文献

[1] PATHAK D, KR?HENBüHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 2536-2544.
[2] YANG C, LU X, LIN Z, et al. High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 4076-4084.
[3] IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1-14.
[4] YU Y C, ZHAN F N, WU R L, et al. Diverse image inpainting with bidirectional and autoregressive transformers[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York: ACM, 2021: 69-78.
[5] LI W B, LIN Z, ZHOU K, et al. MAT: mask-aware transformer for large hole image inpainting[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 10748-10758.
[6] SHAMSOLMOALI P, ZAREAPOOR M, ZHOU H Y, et al. Distance-based weighted transformer network for image completion[J]. Pattern Recognition, 2024, 147: 110120.
[7] CHEN S, ATAPOUR-ABARGHOUEI A, SHUM H P H. HINT: high-quality inpainting transformer with mask-aware encoding and enhanced attention[J]. IEEE Transactions on Multimedia, 2024, 26: 7649-7660.
[8] REN Y R, YU X M, ZHANG R N, et al. StructureFlow: image inpainting via structure-aware appearance flow[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 181-190.
[9] LIAO L, XIAO J, WANG Z, et al. Image inpainting guided by coherence priors of semantics and textures[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 6535-6544.
[10] 陈晓雷, 杨佳, 梁其铎. 结合语义先验和深度注意力残差的图像修复[J]. 计算机科学与探索, 2023, 17(10): 2450-2461.
CHEN X L, YANG J, LIANG Q D. Image inpainting combining semantic priors and deep attention residuals[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(10): 2450-2461.
[11] XU S X, LIU D, XIONG Z W. E2I: generative inpainting from edge to image[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(4): 1308-1322.
[12] GUO X F, YANG H Y, HUANG D. Image inpainting via conditional texture and structure dual generation[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 14114-14123.
[13] 姜艺, 胥加洁, 柳絮, 等. 边缘指导图像修复算法研究[J]. 计算机科学与探索, 2022, 16(3): 669-682.
JIANG Y, XU J J, LIU X, et al. Research on edge-guided image repair algorithm[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(3): 669-682.
[14] YU Y S, DU D W, ZHANG L B, et al. Unbiased multi-modality guidance for image inpainting[C]//Proceedings of the 17th European Conference on Computer Vision. Cham: Springer, 2022: 668-684.
[15] LIU C, XU S X, PENG J L, et al. Towards interactive image inpainting via robust sketch refinement[EB/OL]. [2024-07-26]. https://arxiv.org/abs/2306.00407.
[16] HORITA D, YANG J, CHEN D, et al. A structure-guided diffusion model for large-hole image completion[C]//Proceedings of the 34th British Machine Vision Conference. Durham: BMVA Press, 2023: 258.
[17] CAO C J, DONG Q L, FU Y W. ZITS++: image inpainting by improving the incremental transformer on structural priors[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(10): 12667-12684.
[18] SUN S Q, REN W Q, GAO X W, et al. Restoring images in adverse weather conditions via histogram transformer[C]//Proceedings of the 18th European Conference on Computer Vision. Cham: Springer, 2024: 111-129.
[19] ZUO Z W, ZHAO L, LI A L, et al. Generative image inpainting with segmentation confusion adversarial training and contrastive learning[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(3): 3888-3896.
[20] MA X Z, KONG X, WANG S N, et al. Luna: linear unified nested attention[EB/OL]. [2024-07-26]. https://arxiv.org/abs/ 2106.01540.
[21] PHUTKE S S, MURALA S. Nested deformable multi-head attention for facial image inpainting[C]//Proceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision. Piscataway: IEEE, 2023: 6067-6076.
[22] 曹岩, 辛子昊, 邬开俊, 等. BMTA：多元场景下的大面积破损图像修复[J]. 计算机科学与探索, 2025, 19(6): 1553-1563.
CAO Y, XIN Z H, WU K J, et al. BMTA: inpainting of large area damaged images in multiple scenarios[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(6): 1553-1563.
[23] KARRAS T, AILA T M, LAINE S, et al. Progressive growing of GANs for improved quality, stability, and variation[EB/OL]. [2024-07-26]. https://arxiv.org/abs/1710.10196.
[24] KARRAS T, LAINE S, AILA T. A style-based generator architecture for generative adversarial networks[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vis-ion and Pattern Recognition. Piscataway: IEEE, 2019: 4396-4405.
[25] DOERSCH C, SINGH S, GUPTA A, et al. What makes Paris look like Paris?[J]. ACM Transactions on Graphics, 2012, 31(4): 1-9.
[26] LI L X, ZOU Q, ZHANG F, et al. Line drawing guided progressive inpainting of mural damage[EB/OL]. [2024-07-26]. https://arxiv.org/abs/2211.06649.
[27] LIU G L, REDA F A, SHIH K J, et al. Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the 15th European Conference on Computer Vision. Cham: Springer, 2018: 89-105.
[28] LI X G, GUO Q, LIN D, et al. MISF: multi-level interactive siamese filtering for high-fidelity image inpainting[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 1859-1868.
[29] PARK T, LIU M Y, WANG T C, et al. Semantic image synthesis with spatially-adaptive normalization[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 2332-2341.