基于Transformer的单幅图像去雾算法综述

doi:10.3778/j.issn.1673-9418.2307103

摘要/Abstract

摘要： 图像去雾是一种低级的计算机视觉任务，旨在对雾天降质图像进行预处理，通过恢复其颜色对比度、细节纹理等信息，提高图像的可视性和质量，还原清晰的无雾图像，为后续高级视觉任务（如目标检测、目标追踪、目标分割等）的执行奠定基础。近年来，基于神经网络的去雾算法取得了不错的去雾效果，诸多基于Transformer的图像去雾算法逐渐被提出。但目前缺少对基于Transformer的图像去雾算法进行全面分析和总结的综述。为了弥补这一空缺，对基于Transformer的日间图像、夜间图像和遥感图像去雾算法进行了全面的梳理和综述，其中不仅涵盖了各类去雾算法的基本原理，还探讨了这些算法在不同场景下的适用性和性能表现。此外，介绍了图像去雾任务中常用的数据集和评价指标。在此基础上，对现有的代表性图像去雾算法的性能从定量和定性两个角度进行了分析和评估，对比了典型去雾算法在去雾效果、运行速度、资源消耗等方面的表现。最后，总结了图像去雾技术的应用场景，对图像去雾领域仍然存在的挑战以及未来的发展方向进行了分析和展望。

关键词: Transformer, 图像去雾, 数字图像处理, 深度学习

Abstract: As a fundamental computer vision task, image dehazing aims to preprocess degraded images by restoring color contrast and texture information to improve visibility and image quality, thereby the clear images can be recovered for subsequent high-level visual tasks, such as object detection, tracking, and object segmentation. In recent years, neural network-based dehazing methods have achieved notable success, with a growing number of Transformer-based dehazing approaches being proposed. Up to now, there is a lack of comprehensive review that thoroughly analyzes Transformer-based image dehazing algorithms. To fill this gap, this paper comprehensively sorts out Transformer-based daytime, nighttime and remote sensing image dehazing algorithms, which not only covers the fundamental principles of various types of dehazing algorithms, but also explores the applicability and performance of these algorithms in different scenarios. In addition, the commonly used datasets and evaluation metrics in image dehazing tasks are introduced. On this basis, analysis of the performance of existing representative dehazing algorithms is carried out from both quantitative and qualitative perspectives, and the performance of typical dehazing algorithms in terms of dehazing effect, operation speed, resource consumption is compared. Finally, the application scenarios of image dehazing technology are summarized, and the challenges and future development directions in the field of image dehazing are analyzed and prospected.

Key words: Transformer, image dehazing, digital image processing, deep learning

张凯丽, 王安志, 熊娅维, 刘运. 基于Transformer的单幅图像去雾算法综述[J]. 计算机科学与探索, 2024, 18(5): 1182-1196.

ZHANG Kaili, WANG Anzhi, XIONG Yawei, LIU Yun. Survey of Transformer-Based Single Image Dehazing Methods[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(5): 1182-1196.

参考文献

[1] KATYAL S, KUMAR S, SAKHUJA R, et al. Object detection in foggy conditions by fusion of saliency map and YOLO[C]//Proceedings of the 12th International Conference on Sensing Technology, Limerick, Dec 4-8, 2018. Piscataway: IEEE, 2018: 154-159.
[2] SINDAGI V A, OZA P, YASARLA R, et al. Prior-based domain adaptive object detection for hazy and rainy conditions[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 763-780.
[3] ZHANG Q, ZHANG L Q, WANG D, et al. Global and local information aggregation network for edge-aware salient object detection[J]. Journal of Visual Communication and Image Representation, 2021, 81: 103350.
[4] SINGH D, KUMAR V A. Comprehensive review of computational dehazing techniques[J]. Archives of Computational Methods in Engineering, 2019, 26(5): 1395-1413.
[5] ZHAN Y, ZHAO W L. Instance search via instance level segmentation and feature representation[J]. Journal of Visual Communication and Image Representation, 2021, 79: 103253.
[6] ZHANG J, LI Z X, ZHANG C L. Stable self-attention adversarial learning for semi-supervised semantic image segmentation[J]. Journal of Visual Communication and Image Representation, 2021, 78: 103170.
[7] NISHITA T, MIYAWAKI Y, NAKAMAE E. A shading model for atmospheric scattering considering luminous intensity distribution of light sources[C]//Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 1987: 303-310.
[8] STARK J A. Adaptive image contrast enhancement using generalizations of histogram equalization[J]. IEEE Transactions on Image Processing, 2000, 9(5): 889-896.
[9] KIM J Y, KIM L S, HWANG S H. An advanced contrast enhancement using partially overlapped sub-block histogram equalization[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2001, 11(4): 475-484.
[10] GALDRAN A, VAZQUEZ-CORRAL J, PARDO D, et al. Fusionbased variational image dehazing[J]. IEEE Signal Processing Letters, 2019, 24(2): 151-155.
[11] HE K M, SUN J, TANG X O. Single image haze removal using dark channel prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 33(12): 2341-2353.
[12] FATTAL R. Single image dehazing[J]. Association for Com-puting Machinery Transactions on Graphics, 2008, 27(3): 72.
[13] ZHU Q S, MAI J M, SHAO L. Single image dehazing using color attenuation prior[C]//Proceedings of the 2014 British Machine Vision Conference, Nottingham, Sep 1-5, 2014.
[14] CAI B L, XU X M, JIA K, et al. DehazeNet: an end-to-end system for single image haze removal[J]. IEEE Transactions on Image Processing, 2016, 25(11): 5187-5198.
[15] LI B Y, PENG X L, WANG Z Y, et al. AOD-Net: all-in-one dehazing network[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Piscataway: IEEE, 2017: 4780-4788.
[16] ZHANG H, PATEL V M. Densely connected pyramid dehazing network[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition，Salt Lake City, Jun 18-23, 2018. Piscataway: IEEE, 2018: 3194-3203.
[17] LIU X H, MA Y R, SHI Z H, et al. GridDehazeNet: attention-based multi-scale network for image dehazing[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 7314-7323.
[18] QIN X, WANG Z L, BAI Y C, et al. FFA-Net: feature fusion attention network for single image dehazing[C]//Proceedings of the 2020 AAAI Conference on Artificial Intelligence, New York, Feb 7-12, 2020. Palo Alto: AAAI Press, 2020: 11908-11915.
[19] DONG H, PAN J S, XIANG L, et al. Multiscale boosted dehazing network with dense feature fusion[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 2154-2164.
[20] WU H, QU Y, LIN S, et al. Contrastive learning for compact single image dehazing[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Reco-gnition, Nashville, Jun 20-25, 2021. Piscataway: IEEE, 2021: 10546-10555.
[21] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144.
[22] 郑玉彤, 孙昊英, 宋伟. 隐空间转换的混合样本图像去雾[J]. 计算机工程与应用, 2023, 59(9): 225-236.
ZHENG Y T, SUN H Y, SONG W. Hybrid samples image dehazing via latent space translation[J]. Computer Engineering and Applications, 2023, 59(9): 225-236.
[23] 黄淑英, 汪斌, 李红霞, 等. 基于生成对抗网络的图像去雾算法[J]. 模式识别与人工智能, 2021, 34(11): 990-1003.
HUANG S Y, WANG B, LI H X, et al. Image dehazing based on generative adversarial network[J]. Pattern Recognition and Artificial Intelligence, 2021, 34(11): 990-1003.
[24] 禹晶, 徐东彬, 廖庆敏. 图像去雾技术研究进展[J].中国图象图形学报, 2011, 16(9): 1561-1576.
YU J, XU D B, LIAO Q M. Research on image dehazing technology[J]. Journal of Image and Graphics, 2011, 16(9): 1561-1576.
[25] GUI J, CONG X F, CAO Y, et al. A comprehensive survey on image dehazing based on deep learning[C]//Proceedings of the 30th International Joint Conference on Artificial Intelligence, Montreal, Aug 19-27, 2021: 4426-4433.
[26] 郑凤仙, 王夏黎, 何丹丹, 等. 单幅图像去雾算法研究综述[J]. 计算机工程与应用, 2022, 58(3): 1-14.
ZHENG F X, WANG X L, HE D D, et al. Survey on the research of single image dehazing methods[J]. Computer Engineering and Applications, 2022, 58(3): 1-14.
[27] 贾童瑶, 卓力, 李嘉锋, 等. 基于深度学习的单幅图像去雾研究进展[J]. 电子学报, 2023, 51(1): 231-245.
JIA T Y, ZHUO L, LI J F, et al. Research on the deep-learning-based single image dehazing methods[J]. Acta Electronica Sinica, 2023, 51(1): 231-245.
[28] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Proces-sing Systems 30, Long Beach, Dec 4-9, 2017: 5998-6008.
[29] ZHAO D, LI J, LI H Y, et al. Complementary feature enhanced network with vision transformer for image dehazing[EB/OL]. [2023-05-12]. https://arxiv.org/abs/2109.07100.
[30] YANG Y, ZHANG H W, WU X, et al. MSTFDN: multi-scale transformer fusion dehazing network[J]. Application Intelligence, 2022, 53(5): 5951-5962.
[31] SUAREZ P L, CARPIO D, SAPPA A D, et al. Transformer based image dehazing[C]//Proceedings of the 16th International Conference on Signal-Image Technology & Internet-Based Systems, Dijon, Oct 19-21, 2022. Piscataway: IEEE, 2022: 148-154.
[32] LIANG J Y, CAO J Z, SUN G L, et al. SwinIR: image restoration using swin transformer[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 11-17, 2021. Piscataway: IEEE, 2021: 1833-1844.
[33] SONG Y D, HE Z Q, QIAN H, et al. Vision transformer for single image dehazing[J]. IEEE Transactions on Image Processing, 2023, 32: 1927-1941.
[34] LIU Z, LINY T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021: 9992-10002.
[35] BA L J, KIROS J R, HINTON G E. Layer normalization [EB/OL]. [2023-05-12]. https://arxiv.org/abs/1607.06450.
[36] HENDRYCKS D, GIMPEL K. Bridging nonlinearities and stochastic regularizers with Gaussian error linear units[EB/OL]. [2023-05-12]. https://arxiv.org/abs/1606.08415.
[37] LIU J Z, YUAN H Q, YUAN Z Q. Visual transformer with stable prior and patch-level attention for single image dehazing[J]. Neurocomputing, 2023, 551.
[38] QIU Y W, ZHANG K H, WANG C X, et al. MB-TaylorFormer: multi-branch efficient transformer expanded by Taylor formula for image dehazing[EB/OL]. [2023-05-12]. https://arxiv.org/abs/2308.14036.
[39] TONG M, WANG Y Z, CUI P, et al. Semi-uformer: semi-supervised uncertainty-aware transformer for image dehazing[EB/OL]. [2023-05-12]. https://arxiv.org/abs/2210.16057.
[40] LIU Y Y, LIU H, LI L Y, et al. A data-centric solution to nonhomogeneous dehazing via vision transformer[EB/OL]. [2023-05-12]. https://arxiv.org/abs/2304.07874.
[41] WANG C, PAN J S, LIN W Y, et al. Self-Promer: self-prompt dehazing transformers with depth-consistency[EB/OL]. [2023-05-12]. https://arxiv.org/abs/2303.07033.
[42] GAO G L, CAO J, BAO C, et al. A novel transformer-based attention network for image dehazing[J]. Sensors, 2022, 22(9): 3428.
[43] CHEN S X, YE T, LIU Y, et al. Dual-former: hybrid self-attention transformer for efficient image restoration[EB/OL]. [2023-05-12]. https://arxiv.org/abs/2210.01069.
[44] XU J, CHEN Z X, LUO H, et al. An efficient dehazing algorithm based on the fusion of transformer and convolutional neural network[J]. Sensors, 2023, 23(1): 43.
[45] PARIHAR A S, JAVA A. Densely connected convolutional transformer for single image dehazing[J]. Journal of Visual Communication and Image Representation, 2022, 90: 103722.
[46] TAGHANAKI S A, BENTAIEB A, SHARMA A, et al. Select, attend, and transfer: light, learnable skip connections[C]//Proceedings of the 10th International Workshop on Machine Learning in Medical Imaging, Shenzhen, Oct 13, 2019. Cham: Springer, 2019: 417-425.
[47] CHEN S X, YE T, LIU Y, et al. DEHRFormer: real-time transformer for depth estimation and haze removal from varicolored haze scenes[EB/OL]. [2023-05-12]. https://arxiv.org/abs/2210.01069.
[48] GUO C L, YAN Q X, ANWAR S, et al. Image dehazing transformer with transmission-aware 3D position embedding[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 18-24, 2022. Piscataway: IEEE, 2022: 5802-5810.
[49] JIANG L M, ZHANG C X, HUANG M Y, et al. TSIT: a simple and versatile framework for image-to-image translation[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 206-222.
[50] WANG X T, YU K, DONG C, et al. Recovering realistic texture in image super-resolution by deep spatial feature transform[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-23, 2018. Washington: IEEE Computer Society, 2018: 606-615.
[51] LI X L, HUA Z, LI J J. Two-stage single image dehazing network using swin transformer[J]. The Institution of Engineering and Technology Image Processing, 2022, 16(9): 2518-2534.
[52] LIU Y, YAN Z S, CHEN S X, et al. NightHazeFormer: single nighttime haze removal using prior query transformer[EB/OL]. [2023-05-12]. https://arxiv.org/abs/2305.09533.
[53] DONG P W, WANG B. TransRA: transformer and residual attention fusion for single remote sensing image dehazing[J]. Multidimensional Systems and Signal Processing, 2022, 33(4): 1119-1138.
[54] KULKARNI A, MURALA S. Aerial image dehazing with attentive deformable transformers[C]//Proceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, Jan 2-7, 2023. Piscataway: IEEE, 2023: 6294-6303.
[55] CHI K C, YUAN Y, WANG Q. Trinity-Net: gradient-guided swin transformer-based remote sensing image dehazing and beyond[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-14.
[56] ANCUTI C, ANCUTI C O, DE VLEESCHOUWER C. D-HAZY: a dataset to evaluate quantitatively dehazing algorithms[C]//Proceedings of the 2016 IEEE International Conference on Image Processing, Phoenix, Sep 25-28, 2016. Piscataway: IEEE, 2016: 2226-2230.
[57] ANCUTI C, ANCUTI C O, TIMOFTE R, et al. I-HAZE: a dehazing benchmark with real hazy and haze-free indoor images[C]//Proceedings of the 19th International Conference on Advanced Concepts for Intelligent Vision Systems, Poitiers, Sep 24-27, 2018. Cham: Springer, 2018: 620-631.
[58] ANCUTI C, ANCUTI C O, TIMOFTE R, et al. O-HAZE: a dehazing benchmark with real hazy and haze-free outdoor images[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-23, 2018. Washington: IEEE Computer Society, 2018: 754-762.
[59] LI B Y, REN W Q, FU D P, et al. Benchmarking single-image dehazing and beyond[J]. IEEE Transactions on Image Processing, 2019, 28(1): 492-505.
[60] ANCUTI C, ANCUTI C O, SBERT M, et al. Dense-Haze: a benchmark for image dehazing with dense-Haze and haze-free images[C]//Proceedings of the 2019 IEEE International Conference on Image Processing, Taipei, China, Sep 22-25, 2019. Piscataway: IEEE, 2019: 1014-1018.
[61] ANCUTI C, ANCUTI C O, VASLUIANU F, et al. NTIRE 2020 challenge on nonhomogeneous dehazing[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Piscataway: IEEE, 2020: 2029-2044.
[62] ZHANG Y F, DING L, SHARMA G. HazeRD: an outdoor scene dataset and benchmark for single image dehazing[C]//Proceedings of the 2017 IEEE International Conference on Image Processing, Beijing, Sep 17-20, 2017. Piscataway: IEEE, 2017: 3205-3209.
[63] ZHANG J, CAO Y, ZHA Z J, et al. Nighttime dehazing with a synthetic benchmark[C]//Proceedings of the 28th ACM International Conference on Multimedia, Seattle, Oct 12-16, 2020. New York: ACM, 2020: 2355-2363.
[64] HUANG B H, LI Z, YANG C, et al. Single satellite optical imagery dehazing using SAR image prior based on conditional generative adversarial networks[C]//Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision, Snowmass Village, Mar 1-5, 2020. Piscataway: IEEE, 2020: 1795-1802.
[65] LIN D Y, XU G L, WANG X K, et al. A remote sensing image dataset for cloud removal[EB/OL]. [2023-05-12]. https://arxiv.org/abs/1901.00600.
[66] SCHARSTEIN D, HIRSCHMüLLER H, KITAJIMA Y, et al. High-resolution stereo datasets with subpixel-accurate ground truth[C]//Proceedings of the 36th German Conference on Pattern Recognition, Münster, Sep 2-5, 2014. Cham: Sprin-ger, 2014: 31-42.
[67] SILBERMAN N, HOIEM D, KOHLI P, et al. Indoor segmentation and support inference from RGBD images[C]//Proceedings of the 12th European Conference on Computer Vision, Florence, Oct 7-13, 2012. Berlin, Heidelberg: Springer, 2012: 746-760.
[68] WANG Z, BOVIK A C, SHEIKH R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[69] ZHANG J, CAO Y, FANG S, et al. Fast haze removal for nighttime image using maximum reflectance prior[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 7016-7024.
[70] LIU Y, YAN Z S, WUA M, et al. Nighttime image dehazing based on variational decomposition model[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 19-20, 2022. Pis-cataway: IEEE, 2022: 640-649.
[71] WU R Q, DUAN Z P, GUO C L, et al. RIDCP: revitalizing real image dehazing via high-quality codebook priors[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, Jun 17-24, 2023. Piscataway: IEEE, 2023: 22282-22291.
[72] YUY K, LIU H, FU M H, et al. A two-branch neural network for non-homogeneous dehazing via ensemble learning[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, Jun 19-25, 2021. Piscataway: IEEE, 2021: 193-202.