基于深度学习的红外可见光图像融合综述

doi:10.3778/j.issn.1673-9418.2306061

摘要/Abstract

摘要： 如何将多张图像中的互补信息保存到一张图像中用于全面表征场景是具有挑战性的课题。基于此课题，大量的图像融合方法被提出。红外可见光图像融合（IVIF）作为图像融合中一个重要分支，在语义分割、目标检测和军事侦察等实际领域都有着广泛的应用。近年来，深度学习技术引领了图像融合的发展方向，研究人员利用深度学习针对IVIF方向进行了探索。相关实验工作证明了应用深度学习方法来完成IVIF相较于传统方法有着显著优势。对基于深度学习的IVIF前沿算法进行了详细的分析论述。首先，从网络架构、方法创新以及局限性等方面报告了领域内的方法研究现状。其次，对IVIF方法中常用的数据集进行了简要介绍并给出了定量实验中常用评价指标的定义。对提到的一些具有代表性的方法进行了图像融合和语义分割的定性评估、定量评估实验以及融合效率分析实验来全方面地评估方法的性能。最后，给出了实验结论并对领域内未来可能的研究方向进行了展望。

关键词: 图像融合, 红外可见光图像, 深度学习

Abstract: How to preserve the complementary information in multiple images to represent the scene in one image is a challenging topic. Based on this topic, various image fusion methods have been proposed. As an important branch of image fusion, infrared and visible image fusion (IVIF) has a wide range of applications in segmentation, target detection and military reconnaissance fields. In recent years, deep learning has led the development direction of image fusion. Researchers have explored the field of IVIF using deep learning. Relevant experimental work has proven that applying deep learning to achieving IVIF has significant advantages compared with traditional methods. This paper provides a detailed analysis on the advanced algorithms for IVIF based on deep learning. Firstly, this paper reports on the current research status from the aspects of network architecture, method innovation, and limitations. Secondly, this paper introduces the commonly used datasets in IVIF methods and provides the definition of commonly used evaluation metrics in quantitative experiments. Qualitative and quantitative evaluation experiments of fusion and segmentation and fusion efficiency analysis experiments are conducted on some representative methods mentioned in the paper to comprehensively evaluate the performance of the methods. Finally, this paper provides conclusions and prospects for possible future research directions in the field.

Key words: image fusion, infrared and visible images, deep learning

王恩龙, 李嘉伟, 雷佳, 周士华. 基于深度学习的红外可见光图像融合综述[J]. 计算机科学与探索, 2024, 18(4): 899-915.

WANG Enlong, LI Jiawei, LEI Jia, ZHOU Shihua. Deep Learning-Based Infrared and Visible Image Fusion: A Survey[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(4): 899-915.

参考文献

[1] LEI J, LI J W, LIU J Y, et al. GALFusion: multi-exposure image fusion via a global-local aggregation learning network[J]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 1-15.
[2] LI J W, LIU J Y, ZHOU S H, et al. GeSeNet: a general semantic-guided network with couple mask ensemble for medical image fusion[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023. DOI: 10.1109/TNNLS.2023.3293274.
[3] LI J W, LIU J Y, ZHOU S H, et al. Infrared and visible image fusion based on residual dense network and gradient loss [J]. Infrared Physics & Technology, 2023, 128: 104486.
[4] WANG J, SONG K C, BAO Y Q, et al. CGFNet: cross-guided fusion network for RGB-T salient object detection [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(5): 2949-2961.
[5] 薛震, 张亮亮, 刘吉. 基于改进YOLOv7的融合图像多目标检测方法[J]. 兵器装备工程报, 2023, 44(6): 166-172.
XUE Z, ZHANG L L, LIU J. A fusion image multi object detection method based on improved YOLOv7[J]. Journal of Ordnance Equipment Engineering, 2023, 44(6): 166-172.
[6] LU R T, YANG X G, LI W P, et al. Robust infrared small target detection via multidirectional derivative-based weig-hted contrast measure[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 1-5.
[7] 邓楷文, 葛晨阳. 改进YOLOv5的轻量化红外交通目标检测[J]. 计算机工程与应用, 2023, 59(12): 184-192.
DENG K W, GE C Y. Research on lightweight of improved YOLOv5 infrared traffic detection network[J]. Computer Engineering and Applications, 2023, 59(12): 184-192.
[8] 别倩, 王晓, 徐新, 等. 红外-可见光跨模态的行人检测综述[J]. 中国图象图形学报, 2023, 28(5): 1287-1307.
BIE Q, WANG X, XU X, et al. Overview of infrared visible cross modal pedestrian detection[J]. Journal of Image and Graphics, 2023, 28(5): 1287-1307.
[9] LI B, WU W, WANG Q, et al. SiamRPN++: evolution of siamese visual tracking with very deep networks[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 4277-4286.
[10] LI C L, XIANG Z Q, TANG J, et al. RGBT tracking via noise-robust cross-modal ranking[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 33(9): 5019-5031.
[11] MULLER A C, NARAYANAN S. Cognitively-engineered multisensor image fusion for military applications[J]. Information Fusion, 2009, 10(2): 137-149.
[12] CHEN J, LI X J, LUO L B, et al. Infrared and visible image fusion based on target-enhanced multiscale transform decomposition[J]. Information Sciences, 2020, 508: 64-78.
[13] LIU X B, MEI W B, DU H Q. Structure tensor and nonsubsampled shearlet transform based algorithm for CT and MRI image fusion[J]. Neurocomputing, 2017, 235: 131-139.
[14] 邸敬, 郭文庆, 刘冀钊, 等. 基于NSCT域滚动引导滤波与自适应PCNN的医学图像融合[J]. 计算机应用研究, 2023, 40(7): 1-7.
DI J, GUO W Q, LIU J Z, et al. Medical image fusion based on NSCT domain rolling guided filtering and adaptive PCNN[J]. Application Research of Computers, 2023, 40(7): 1-7.
[15] 杨艳春, 裴佩佩, 党建武, 等. 基于交替梯度滤波器和改进PCNN的红外与可见光图像融合[J]. 光学精密工程, 2022,30(9): 1123-1138.
YANG Y C, PEI P P, DANG J W, et al. Infrared and visible image fusion based on alternating gradient filter and improved PCNN[J]. Optics and Precision Engineering, 2022, 30(9): 1123-1138.
[16] WANG J, PENG J Y, FENG X Y, et al. Fusion method for infrared and visible images by using non-negative sparse representation[J]. Infrared Physics & Technology, 2014, 67: 477-489.
[17] LIU Y, LIU S P, WANG Z F. A general framework for image fusion based on multi-scale transform and sparse representation[J]. Information Fusion, 2015, 24: 147-164.
[18] LI H, WU X J, KITTER J. MDLatLRR: a novel decomposition method for infrared and visible image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4733-4746.
[19] 王纪委, 曲怀敬, 魏亚南, 等. 结合线性稀疏表示和图像抠图的多聚焦图像融合方法[J]. 计算机应用研究, 2022, 39(6): 1879-1885.
WANG J W, QU H J, WEI Y N, et al. A multifocal image fusion method combining linear sparse representation and image matting[J]. Application Research of Computers, 2022, 39(6): 1879-1885.
[20] MOU J, GAO W, SONG Z X. Image fusion based on non-negative matrix factorization and infrared feature extraction [C]//Proceedings of the 2013 6th International Congress on Image and Signal Processing, Hangzhou, Dec 16-18, 2013. Washington: IEEE Computer Society, 2013: 1046-1050.
[21] FU Z Z, WANG X, XU J, et al. Infrared and visible images fusion based on RPCA and NSCT[J]. Infrared Physics & Technology, 2016, 77: 114-123.
[22] WANG A Z, WANG M H. RGB-D salient object detection via minimum barrier distance transform and saliency fusion[J]. IEEE Signal Processing Letters, 2017, 24: 663-667.
[23] YANG Y, ZHANG Y M, HUANG S Y, et al. Infrared and visible image fusion using visual saliency sparse representation and detail injection model[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-15.
[24] NIE R C, MA C Z, CAO J D, et al. A total variation with joint norms for infrared and visible image fusion[J]. IEEE Transactions on Multimedia, 2022, 24: 1460-1472.
[25] LI J W, LIU J Y, ZHOU S H, et al. Learning a coordinated network for detail-refinement multiexposure image fusion[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(2): 713-727.
[26] ZHANG X C, DEMIRIS Y. Visible and infrared image fusion using deep learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 10535-10554.
[27] 安晓东, 李亚丽, 王芳. 汽车驾驶辅助系统红外与可见光融合算法综述[J]. 计算机工程与应用, 2022, 58(19): 64-75.
AN X D, LI Y L, WANG F. Overview of infrared and visible image fusion algorithms for automotive driving assistance system[J]. Computer Engineering and Applications, 2022, 58(19): 64-75.
[28] 唐霖峰, 张浩, 徐涵, 等. 基于深度学习的图像融合方法综述[J]. 中国图象图形学报, 2023, 28(1): 3-36.
TANG L F, ZHANG H, XU H, et al. Deep learning-based image fusion: a survey[J]. Journal of Image and Graphics, 2023, 28(1): 3-36.
[29] LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2019, 28(5): 2614-2623.
[30] LI H, WU X J, KITTER J. RFN-Nest: an end-to-end residual fusion network for infrared and visible images[J]. Information Fusion, 2021, 73: 72-86.
[31] LI Q Q, HAN G L, LIU P X, et al. A multilevel hybrid transmission network for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-14.
[32] XU H, GONG M Q, TIAN X, et al. CUFD: an encoder-decoder network for visible and infrared image fusion based on common and unique feature decomposition[J]. Computer Vision and Image Understanding, 2022, 218: 103407.
[33] JIAN L H, YANG X M, LIU Z, et al. SEDRFuse: a symmetric encoder-decoder with residual block network for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-15.
[34] WANG X J, HUA Z, LI J J. PACCDU: pyramid attention cross-convolutional dual UNet for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-16.
[35] LI H, WU X J, DURRANI T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656.
[36] WANG Z S, WANG J Y, WU Y Y, et al. UNFusion: a unified multi-scale densely connected network for infrared and visible image fusion[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(6): 3360-3374.
[37] WANG Z S, WU Y Y, WANG J Y, et al. Res2Fusion: infrared and visible image fusion based on dense Res2Net and double nonlocal attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-12.
[38] LIANG P W, JIANG J J, LIU X M, et al. Fusion from decomposition: a self-supervised decomposition approach for image fusion[C]//Proceedings of the 17th European Conference on Computer Vision, Oct 23-27, 2022. Cham: Springer, 2022:719-735.
[39] LI H F, ZHAO J Z, LI J X, et al. Feature dynamic alignment and refinement for infrared-visible image fusion: translation robust fusion[J]. Information Fusion, 2023, 95: 26-41.
[40] TANG L F, XIANG X Y, ZHANG H, et al. DIVFusion: darkness-free infrared and visible image fusion[J]. Information Fusion, 2023, 91: 477-493.
[41] LI Z X, LIU J Y, LIU R S, et al. Multiple task-oriented encoders for unified image fusion[C]//Proceedings of the 2021 IEEE International Conference on Multimedia and Expo, Shenzhen, Jul 5-9, 2021. Washington: IEEE Computer Society, 2021:1-6.
[42] ZHU Z G, YANG X G, LU R T, et al. CLF-Net: contrastive learning for infrared and visible image fusion network[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-15.
[43] XIAO W X, ZHANG Y F, WANG H B, et al. Heterogeneous knowledge distillation for simultaneous infrared-visible image fusion and super-resolution[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-15.
[44] SU W J, HUANG Y D, LI Q F, et al. Infrared and visible image fusion based on adversarial feature extraction and stable image reconstruction[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-14.
[45] MA J Y, TANG L F, XU M L, et al. STDFusionNet: an infrared and visible image fusion network based on salient target detection[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-13.
[46] LONG Y Z, JIA H T, ZHONG Y D, et al. RXDNFuse: a aggregated residual dense network for infrared and visible image fusion[J]. Information Fusion, 2021, 69: 128-141.
[47] GUO C X, FAN D D, JIANG Z X, et al. MDFN: mask deep fusion network for visible and infrared image fusion without reference ground-truth[J]. Expert Systems with Applications, 2022, 211: 118631.
[48] LIU J Y, DIAN R W, LI S T, et al. SGFusion: a saliency guided deep-learning framework for pixel-level image fusion[J]. Information Fusion, 2023, 91: 205-214.
[49] WANG X, GUAN Z, YU S S, et al. Infrared and visible image fusion via decoupling network[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-13.
[50] XU H, MA J Y, JIANG J J, et al. U2Fusion: a unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 502-518.
[51] ZHANG H, XU H, XIAO Y, et al. Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity[C]//Proceedings of the 2020 AAAI Conference on Artificial Intelligence, New York, Feb 7-12, 2020. Menlo Park: AAAI, 2020: 12797-12804.
[52] ZHANG H, MA J Y. SDNet: a versatile squeeze-and-decomposition network for real-time image fusion[J]. International Journal of Computer Vision, 2021, 129(10): 2761-2785.
[53] CHENG C Y, XU T Y, WU X J. MUFusion: a general unsupervised image fusion network based on memory unit[J]. Information Fusion, 2023, 92: 80-92.
[54] TANG W, HE F Z, LIU Y, et al. DATFuse: infrared and visible image fusion via dual attention transformer[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(7): 3159-3172.
[55] LI J, ZHU J M, LI C, et al. CGTF: convolution-guided transformer for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-14.
[56] LIU J Y, WU Y H, WU G Y, et al. Learn to search a lightweight architecture for target-aware infrared and visible image fusion[J]. IEEE Signal Processing Letters, 2022, 29: 1614-1618.
[57] WANG D, LIU J Y, FAN X, et al. Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration[C]//Proceedings of the 31st International Joint Conference on Artificial Intelligence, Montreal, Aug 19-27, 2022: 3508-3515.
[58] XU H, MA J Y, YUAN J T, et al. RFNet: unsupervised network for mutually reinforcing multi-modal image registration and fusion[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 18-24, 2022. Washington: IEEE Computer Society, 2022: 19647-19656.
[59] XU H, YUAN J T, MA J Y. MURF: mutually reinforcing multi-modal image registration and fusion[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(10): 12148-12166.
[60] TANG L F, YUAN J T, MA J Y. Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network[J]. Information Fusion, 2022, 82: 28-42.
[61] WANG D, LIU J Y, LIU R S, et al. An interactively reinforced paradigm for joint infrared-visible image fusion and saliency object detection[J]. Information Fusion, 2023, 98: 101828.
[62] LI H F, CEN Y L, LIU Y, et al. Different input resolutions and arbitrary output resolution: a meta learning-based deep framework for infrared and visible image fusion[J]. IEEE Transactions on Image Processing, 2021, 30: 4070-4083.
[63] WANG B W, ZOU Y, ZHANG L F, et al. Multimodal super-resolution reconstruction of infrared and visible images via deep learning[J]. Optics and Lasers in Engineering, 2022, 156: 107078.
[64] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of ACM, 2020, 63(11): 139-144.
[65] MA J Y, YU W, LIANG P W, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26.
[66] MA J Y, ZHANG H, SHAO Z F, et al. GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14.
[67] ZHOU H B, HOU J L, ZHANG Y D, et al. Unified gradient-and intensity-discriminator generative adversarial network for image fusion[J]. Information Fusion, 2022, 88: 184-201.
[68] XU H, LIANG P W, YU W, et al. Learning a generative model for fusing infrared and visible images via conditional generative adversarial network with dual discriminators[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macao, China, Aug 10-16, 2019: 3954-3960.
[69] FU Y, WU X J, DURRANI T. Image fusion based on generative adversarial network consistent with perception[J]. Information Fusion, 2021, 72: 110-125.
[70] YANG Y, LIU J X, HUANG S Y, et al. Infrared and visible image fusion via texture conditional generative adversarial network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(12): 4771-4783.
[71] TANG Z M, XIAO G B, GUO J W, et al. Dual-attention-based feature aggregation network for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 1-13.
[72] WANG J X, XI X L, LI D M, et al. FusionGRAM: an infrared and visible image fusion framework based on gradient residual and attention mechanism[J]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 1-12.
[73] LI J, HUO H T, LI C, et al. AttentionFGAN: infrared and visible image fusion using attention-based generative adversarial networks[J]. IEEE Transactions on Multimedia, 2021, 23: 1383-1396.
[74] WANG Z S, SHAO W Y, CHEN Y L, et al. A cross-scale iterative attentional adversarial fusion network for infrared and visible images[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(8): 3677-3688.
[75] YIN H T, XIAO J H. Laplacian pyramid generative adversarial network for infrared and visible image fusion[J]. IEEE Signal Processing Letters, 2022, 29: 1988-1992.
[76] ZHOU H B, WU W, ZHANG Y D, et al. Semantic-supervised infrared and visible image fusion via a dual-discriminator generative adversarial network[J]. IEEE Transactions on Multimedia, 2023, 25: 635-648.
[77] LIU J Y, FAN X, HUANG Z B, et al. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 18-24, 2022. Piscataway: IEEE, 2022: 5792-5801.
[78] RAO Y J, WU D, HAN M N, et al. AT-GAN: a generative adversarial network with attention and transition for infrared and visible image fusion[J]. Information Fusion, 2023, 92: 336-349.
[79] HAN M N, YU K L, QIU J H, et al. Boosting target-level infrared and visible image fusion with regional information coordination[J]. Information Fusion, 2023, 92: 268-288.
[80] ZHANG J, JIAO L C, MA W P, et al. Transformer based conditional GAN for multimodal image fusion[J]. IEEE Transactions on Multimedia, 2023, 25: 8988-9001.
[81] JIA X Y, ZHU C, LI M Z, et al. LLVIP: a visible-infrared paired dataset for low-light vision[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 11-17, 2021. Piscataway: IEEE, 2021: 3489-3497.
[82] 孙彬, 高云翔, 诸葛吴为, 等. 可见光与红外图像融合质量评价指标分析[J]. 中国图象图形学报, 2023, 28(1): 144-155.
SUN B, GAO Y X, ZHUGE W W, et al. Analysis of quality objective assessment metrics for visible and infrared image fusion[J]. Journal of Image and Graphics, 2023, 28(1): 144-155.
[83] 杨艳春, 李娇, 王阳萍. 图像融合质量评价方法研究综述[J]. 计算机科学与探索, 2018, 12(7): 1021-1035.
YANG Y C, LI J, WANG Y P. Review of image fusion quality evaluation methods[J]. Journal of Frontiers of Computer Science and Technology, 2018, 12(7): 1021-1035.