Feature Refinement and Multi-scale Attention for Transformer Image Denoising Network

doi:10.3778/j.issn.1673-9418.2308091

Abstract

Abstract: In order to enhance the relevance of global context information, strengthen the attention to multi-scale features, improve the image denoising effect while preserving the details to the greatest extent, a Transformer based feature refinement and multi-scale attention image denoising network (TFRADNet) is proposed. The network not only uses Transformer in the codec part to solve the long-term dependence problem of large-scale images and improve the efficiency of model noise reduction, but also adds a position awareness layer after the up-sampling operation to enhance the network’s perception ability of pixel positions in the feature map. To cope with Transformer’s neglect of spatial relationships among pixels, which may result in local detail distortion, a feature refinement block (FRB) is designed at feature reconstruction stage. A serial structure is used to introduce nonlinear transformations layer by layer, to enhance the recognition of local image features with complex noise levels. Meanwhile, a multi-scale attention block (MAB) is designed, which adopts a parallel double-branch structure to jointly model spatial attention and channel attention, effectively capturing and weighting image features of different scales, and improving the model’s perception ability of multi-scale features. Experimental results on real noise datasets SIDD, DND and RNI15 show that TFRADNet can take into account global information and local details, and has stronger noise suppression ability and robustness than other advanced methods.

Key words: image denoising, feature refinement, multi-scale attention, Transformer, real noise

摘要： 为增强全局上下文信息的关联性，加强对多尺度特征的关注，在提升图像去噪效果的同时最大程度保留细节特征，提出一种基于Transformer的特征细化和多尺度注意力的图像去噪网络（TFRADNet）。该网络不仅在编解码器部分利用Transformer解决大规模图像的长程依赖问题，提高模型的去噪效率，还在上采样操作后加入位置感知层来增强网络对特征图中像素位置的感知能力。为了应对Transformer可能对像素间空间关系的忽略，导致局部细节失真，在特征重建阶段设计了特征细化模块（FRB），采用串行结构逐层引入非线性变换，加强对噪声水平复杂的图像局部特征的识别。同时，设计了多尺度注意力模块（MAB），采用并行双分支结构，对空间注意力和通道注意力联合建模，有效捕捉不同尺度的图像特征并进行加权，提高模型对多尺度特征的感知能力。在真实噪声数据集SIDD、DND和RNI15上的实验结果显示，TFRADNet能够兼顾全局信息和局部细节，相比其他先进方法展现出了更强的抑噪能力和稳健性。

关键词: 图像去噪, 特征细化, 多尺度注意力, Transformer, 真实噪声

YUAN Heng, GENG Yikun. Feature Refinement and Multi-scale Attention for Transformer Image Denoising Network[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(7): 1838-1851.

袁姮, 耿仪坤. 特征细化和多尺度注意力的Transformer图像去噪网络[J]. 计算机科学与探索, 2024, 18(7): 1838-1851.

References

[1] DABOV K, FOI A, KATKOVNIK V, et al. Image denoising by sparse 3-D transform-domain collaborative filtering[J]. IEEE Transactions on Image Processing, 2007, 16(8): 2080-2095.
[2] 钱冲, 常冬霞. 图拉普拉斯正则化稀疏变换学习图像去噪算法[J]. 计算机工程与应用, 2022, 58(5): 232-239.
QIAN C, CHANG D?X. Image denoising algorithm based on graph Laplacian regularized sparse transform learning[J]. Computer Engineering and Applications, 2022, 58(5): 232-239.
[3] ZHANG K, ZUO W M, CHEN Y J, et al. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising[J]. IEEE Transactions on Image Processing, 2017, 26(7): 3142-3155.
[4] ZHANG K, ZUO W M, ZHANG L. FFDNet: toward a fast and flexible solution for CNN-based image denoising[J]. IEEE Transactions on Image Processing, 2018, 27(9): 4608-4622.
[5] GUO S, YAN Z F, ZHANG K, et al. Toward convolutional blind denoising of real photographs[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 15-20, 2019. Piscataway: IEEE, 2019: 1712-1722.
[6] 丁岳皓, 吴昊, 孔凤玲, 等. 面向真实图像噪声的两阶段盲去噪[J]. 中国图象图形学报, 2023, 28(7): 2026-2036.
DING Y H, WU H, KONG F L, et al. A dual of real image noise-related blind denoising technique[J]. Journal of Image and Graphics, 2023, 28(7): 2026-2036.
[7] WU W C, LIU S J, ZHOU Y, et al. Dual residual attention network for image denoising[EB/OL]. [2023-06-29]. https://arxiv.org/abs/2305.04269.
[8] WU W C, LV G N, DUAN Y Y, et al. DCANet: dual convolutional neural network with attention for image blind denoising[EB/OL]. [2023-06-29]. https://arxiv.org/abs/2304.01498.
[9] 曲海成, 申磊. 面向目标检测的SAR图像去噪和语义增强[J]. 光子学报, 2022, 51(4): 329-343.
QU H C, SHEN L. SAR image denoising and semantic enhancement for object detection[J]. Acta Photonica Sinica, 2022, 51(4): 329-343.
[10] HUANG J, LIU X, PAN Y, et al. CasaPuNet: channel affine self-attention based progressively updated network for real image denoising[J]. IEEE Transactions on Industrial Informatics, 2023, 19(8): 9145-9156.
[11] MOU C, ZHANG J, FAN X, et al. COLA-Net: collaborative attention network for image restoration[J]. IEEE Transactions on Multimedia, 2022, 24: 1366-1377.
[12] JIANG B, LU Y, WANG J, et al. Deep image denoising with adaptive priors[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(8): 5124-5136.
[13] 耿玉标, 岳志远, 闫麒名, 等. 产品表面缺陷检测的多通路阈值收缩融合网络[J]. 计算机工程与应用, 2023, 59(10): 162-170.
GENG Y B, YUE Z Y, YAN Q M, et al. Multi-stream thres-hold shrinkage and fusion network for product surface defect detection[J]. Computer Engineering and Applications, 2023, 59(10): 162-170.
[14] JIANG J D, ZHENG L N, LUO F, et al. RedNet: residual encoder-decoder network for indoor RGB-D semantic segmentation[EB/OL]. [2023-06-29]. https://arxiv.org/abs/1806. 01054.
[15] ZAMIR S W, ARORA A, KHAN S, et al. Multi-stage progressive image restoration[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, Jun 20-25, 2021. Piscataway: IEEE, 2021: 14821-14831.
[16] JIANG Y Q, ZHANG C, LIU J. CS-PCN: context-space progressive collaborative network for image denoising[EB/OL]. [2023-06-29]. https://arxiv.org/abs/2305.10146.
[17] POTLAPALLI V, ZAMIR S W, KHAN S, et al. PromptIR: prompting for all-in-one blind image restoration[EB/OL]. [2023-06-29]. https://arxiv.org/abs/2306.13090.
[18] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems 30, Long Beach, Dec 4-9, 2017: 5998-6008.
[19] LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 11-17, 2021. Piscataway: IEEE, 2021: 10012-10022.
[20] LIANG J Y, CAO J Z, SUN G L, et al. SwinIR: image restoration using swin transformer[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 11-17, 2021. Piscataway: IEEE, 2021: 1833-1844.
[21] WANG Z D, CUN X D, BAO J M, et al. Uformer: a general U-shaped transformer for image restoration[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 18-24, 2022. Piscataway: IEEE, 2022: 17683-17693.
[22] YAO C, JIN S, LIU M Q, et al. Dense residual transformer for image denoising[J]. Electronics, 2022, 11(3): 418.
[23] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Oct 5- 9, 2015. Cham: Springer, 2015: 234-241.
[24] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[EB/OL]. [2023-06-29]. https://arxiv.org/abs/2010.11929.
[25] WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 3-19.
[26] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-23, 2018. Washington: IEEE Computer Society, 2018: 7132-7141.
[27] CHARBONNIER P, BLANC-FERAUD L, AUBERT G, et al. Two deterministic half-quadratic regularization algorithms for computed imaging[C]//Proceedings of the 1st International Conference on Image Processing, Austin, Nov 13-16, 1994. Piscataway: IEEE, 1994: 168-172.
[28] HUYNH-THU Q, GHANBARI M. Scope of validity of PSNR in image/video quality assessment[J]. Electronics Letters, 2008, 44(13): 800-801.
[29] ASSESSMENT I Q. From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 770-778.
[30] ABDELHAMED A, LIN S, BROWN M S. A high-quality denoising dataset for smartphone cameras[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-23, 2018. Washington: IEEE Computer Society, 2018: 1692-1700.
[31] PLOTZ T, ROTH S. Benchmarking denoising algorithms with real photographs[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 1586-1595.
[32] LEBRUN M, COLOM M, MOREL J M. The noise clinic: a blind image denoising algorithm[J]. Image Processing on Line, 2015, 5: 1-54.
[33] ZAMIR S W, ARORA A, KHAN S, et al. Learning enriched features for real image restoration and enhancement[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 492-511.
[34] ANWAR S, BARNES N. Real image denoising with feature attention[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 3155-3164.
[35] YUE Z S, YONG H W, ZHAO Q, et al. Variational denoising network: toward blind noise modeling and removal[C]//Advances in Neural Information Processing Systems 32, Vancouver, Dec 8-14, 2019: 1688-1699.
[36] ZAMIR S W, ARORA A, KHAN S, et al. CycleISP: real image restoration via improved data synthesis[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 2696-2705.
[37] CHENG S, WANG Y Z, HUANG H B, et al. NBNet: noise basis learning for image denoising with subspace projection[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, Jun 20-25, 2021. Piscataway: IEEE, 2021: 4896-4906.