基于几何约束和结构注意力机制的图像超分辨率

doi:10.3778/j.issn.1673-9418.2408084

摘要/Abstract

摘要： 针对将超分辨率技术应用于古建筑物图像修复时，如何在更大尺度的重建时更多地保留原始图像的几何结构问题，提出一种基于几何约束与结构注意力增强的超分辨率图像重建方法。该方法在GeoSR模型的基础上设计了多尺度特征融合ghost卷积模块，在不显著增加模型参数的同时，提升模型性能；同时引入结构注意力增强模块，在上采样时自适应地调整不同通道和空间位置的权重，使模型能够更加关注图像中的结构信息，从而更有效地捕捉和恢复图像细节；利用最小化混合均方误差和几何对齐误差构成损失函数，在训练中有效地恢复了细节和几何规律。在Cityscape、DIV2K等数据集上进行了详尽的实验，实验结果表明利用多尺度的ghost卷积模块能够在减小模型参数同时增强模型的性能，同时结构注意力增强模块的应用使模型对几何结构的重建更加出色。改进后的模型只有原始GeoSR模型参数的80.9%，但超分辨率重建效果对比目前的模型效果更好，特别是在4倍和8倍的重建尺度下，该模型效果提升更明显。

关键词: 单图像的超分辨率, 几何约束, 多尺度特征, ghost卷积, 结构注意力

Abstract: To address the issue of better preserving the geometric structure of the original image during large-scale reconstruction in super-resolution applications for ancient architectural image restoration, this paper proposes a super-resolution image reconstruction method based on geometric constraints and structural attention enhancement. This method builds upon the GeoSR model by designing a multi-scale feature fusion ghost convolution module, which improves model performance without significantly increasing the number of parameters. Additionally, a structural attention enhancement module is introduced, which adaptively adjusts the weights of different channels and spatial positions during upsampling, enabling the model to better focus on structural information in the image, thus more effectively capturing and restoring image details. Finally, a loss function that minimizes the hybrid mean squared error and geometric alignment error is utilized to effectively recover details and geometric patterns during training. Extensive experiments are conducted on the Cityscape, DIV2K, and other datasets, and the results show that the multi-scale ghost convolution module enhances model performance while reducing the number of parameters. Furthermore, the application of the structural attention enhancement module makes the model more outstanding in reconstructing geometric structures. The improved model only has 80.9% of the parameters of the original GeoSR model, while it achieves better super-resolution reconstruction performance compared with current models, with more noticeable improvements, particularly at 4x and 8x reconstruction scales.

Key words: single image super-resolution, geometric constraints, multi-scale features, ghost convolution, structural attention

辜翱, 方艳红. 基于几何约束和结构注意力机制的图像超分辨率[J]. 计算机科学与探索, 2025, 19(7): 1868-1877.

GU Ao, FANG Yanhong. Image Super-Resolution Based on Geometric Constraints and Structural Attention Mechanism[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(7): 1868-1877.

参考文献

[1] ZHOU D W, DUAN R, ZHAO L J, et al. Single image super-resolution reconstruction based on multi-scale feature mapping adversarial network[J]. Signal Processing, 2020, 166: 107251.
[2] SONG Z Y, ZHAO X Q, JIANG H M. Gradual deep residual network for super-resolution[J]. Multimedia Tools and Applications, 2021, 80(7): 9765-9778.
[3] LU Z Y, CHEN Y. Joint self-supervised depth and optical flow estimation towards dynamic objects[J]. Neural Processing Letters, 2023, 55(8): 10235-10249.
[4] SHANG W L, SOHN K, ALMEID D, et al. Understanding and improving convolutional neural networks via concatenated rectified linear units[C]//Proceedings of the 33rd Inter-national Conference on Machine Learning, 2016: 2217-2225.
[5] LU Z Y, CHEN Y. Pyramid frequency network with spatial attention residual refinement module for monocular depth estimation[J]. Journal of Electronic Imaging, 2022, 31(2): 023005.
[6] LU Z Y, WANG F. City scene super-resolution via geometric error minimization[J]. Journal of Electronic Imaging, 2024, 33(1): 013014.
[7] HAN K, WANG Y H, TIAN Q, et al. GhostNet: more features from cheap operations[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 1577-1586.
[8] WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the 15th European Conference on Computer Vision. Cham: Springer, 2018: 3-19.
[9] CANNY J. A computational approach to edge detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,1986, 8(6): 679-698.
[10] DUDA R O, HART P E. Use of the Hough transformation to detect lines and curves in pictures[J]. Communications of the ACM, 1972, 15(1): 11-15.
[11] BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of the 2012 British Machine Vision Conference, 2012.
[12] MARTIN D, FOWLKES C, TAL D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological stati-stics[C]//Proceedings of the 8th IEEE International Conference on Computer Vision. Piscataway: IEEE, 2001: 416-423.
[13] CORDTS M, OMRAN M, RAMOS S, et al. The cityscapes dataset for semantic urban scene understanding[C]//Proceed-ings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 3213-3223.
[14] ALI-BEY A, CHAIB-DRAA B, GIGUèRE P. GSV-Cities: toward appropriate supervised visual place recognition[J]. Neurocomputing, 2022, 513: 194-203.
[15] AGUSTSSON E, TIMOFTE R. NTIRE 2017 challenge on single image super-resolution: dataset and study[C]//Proceed-ings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 1122-1131.
[16] BOOR C D. Bicubic spline interpolation[J]. Journal of Math-ematics and Physics, 1962, 41: 212-218.
[17] KINGMA D P, BA J, HAMMAD M M. Adam: a method for stochastic optimization[EB/OL]. [2024-06-03]. https://arxiv.org/abs/1412.6980.
[18] HORé A, ZIOU D. Image quality metrics: PSNR vs. SSIM[C]//Proceedings of the 2010 20th International Conference on Pattern Recognition. Piscataway: IEEE, 2010: 2366-2369.
[19] SHI W Z, CABALLERO J, HUSZáR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 1874-1883.
[20] DONG C, LOY C C, HE K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.
[21] KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 1646-1654.
[22] LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 1132-1140.
[23] DONG C, LOY C C, TANG X O. Accelerating the super-resolution convolutional neural network[C]//Proceedings of the 14th European Conference on Computer Vision. Cham: Springer, 2016: 391-407.
[24] KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 1637-1645.
[25] LEDIG C, THEIS L, HUSZAR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 105-114.
[26] HARIS M, SHAKHNAROVICH G, UKITA N. Deep back-projection networks for super-resolution[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 1664-1673.
[27] LU Z Y, CHEN Y. Single image super-resolution based on a modified U-Net with mixed gradient loss[J]. Signal, Image and Video Processing, 2022, 16(5): 1143-1151.