基于深度学习的多聚焦图像融合方法前沿进展

doi:10.3778/j.issn.1673-9418.2306058

摘要/Abstract

摘要： 多聚焦图像融合是一种有效的图像融合技术，旨在将来自同一场景的不同焦平面的源图像组合起来以获得良好的融合结果，这意味着融合后的图像将在所有焦平面都聚焦，也就是包含着更丰富的场景信息。深度学习的发展促进了图像融合的巨大进步，而神经网络强大的特征提取和重构能力使融合结果大有可为。近年来，出现了越来越多的基于深度学习的多聚焦图像融合方法，如卷积神经网络（CNN）、生成对抗网络（GAN）和自动编码器等。为了给相关研究人员和技术人员提供有效的参考和理解，介绍了多聚焦图像融合的概念和一些评价指标，分析了十几种近几年基于深度学习的多聚焦图像融合前沿方法，讨论了各种方法的特点和创新性，并总结了它们的优缺点，回顾了多聚焦图像融合技术在各种场景中的应用，包括摄影可视化、医疗诊断和遥感检测等领域，介绍了目前多聚焦图像融合相关领域面临的一些挑战并展望了未来可能的研究趋势。

关键词: 深度学习, 图像融合, 多聚焦

Abstract: Multi-focus image fusion is an effective image fusion technology, which aims to combine source images from different focal planes of the same scene to obtain a good fusion result. This means that the fused image will focus on all focal planes, that is, it contains more abundant scene information. The development of deep learning promotes the great progress of image fusion, and the powerful feature extraction and reconstruction ability of neural network makes the fusion result promising. In recent years, more and more multi-focus image fusion methods based on deep learning have been proposed, such as convolutional neural network (CNN), generative adversarial network (GAN) and automatic encoder, etc. In order to provide effective reference for relevant researchers and technicians, firstly, this paper introduces the concept of multi-focus image fusion and some evaluation indicators. Then, it analyzes more than ten advanced methods of multi-focus image fusion based on deep learning in recent years, discusses the characteristics and innovation of various methods, and summarizes their advantages and disadvantages. In addition, it reviews the application of multi-focus image fusion technology in various scenes, including photographic visualization, medical diagnosis, remote sensing detection and other fields. Finally, it proposes some challenges faced by current multi-focus image fusion related fields and looks forward to future possible research trends.

Key words: deep learning, image fusion, multi-focus

李子奇, 苏宇轩, 孙俊, 张永宏, 夏庆锋, 尹贺峰. 基于深度学习的多聚焦图像融合方法前沿进展[J]. 计算机科学与探索, 2024, 18(9): 2276-2292.

LI Ziqi, SU Yuxuan, SUN Jun, ZHANG Yonghong, XIA Qingfeng, YIN Hefeng. Critical Review of Multi-focus Image Fusion Based on Deep Learning Method[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(9): 2276-2292.

参考文献

[1] LIU Y, LIU S, WANG Z. Multi-focus image fusion with dense SIFT[J]. Information Fusion, 2015, 23: 139-155.
[2] TANG H, XIAO B, LI W, et al. Pixel convolutional neural network for multi-focus image fusion[J]. Information Sciences, 2018, 433: 125-141.
[3] JIANG Z G, HAN D B, CHEN J, et al. A wavelet-based algorithm for multi-focus micro-image fusion[C]//Proceedings of the 3rd International Conference on Image and Graphics, Hong Kong, China, Dec 18-20, 2004.?Washington: IEEE Computer Society, 2004: 176-179.
[4] WANG Z, MA Y. Medical image fusion using m-PCNN[J]. Information Fusion, 2008, 9(2): 176-185.
[5] SUJATHA K, PUNITHAVATHANI D S. Optimized ensemble decision-based multi-focus image fusion using binary genetic Grey-Wolf optimizer in camera sensor networks[J]. Multimedia Tools and Applications, 2018, 77(2): 1735-1759.
[6] CHEN Z, WANG D, GONG S, et al. Application of multi-focus image fusion in visual power patrol inspection[C]//Proceedings of the 2017 IEEE 2nd Advanced Information Technology, Electronic and Automation Control Conference,?Chongqing, Mar 25-26, 2017. Piscataway: IEEE, 2017: 1688-1692.
[7] SONG Y, LI M T, LI Q L, et al. A new wavelet based multi-focus image fusion scheme and its application on optical microscopy[C]//Proceedings of the 2006 IEEE International Conference on Robotics and Biomimetics, Kunming, Dec 17-20, 2006. Piscataway: IEEE, 2006: 401-405.
[8] ZHANG H, XU H, TIAN X, et al. Image fusion meets deep learning: a survey and perspective[J]. Information Fusion, 2021, 76: 323-336.
[9] LI S, KANG X, FANG L, et al. Pixel-level image fusion: a survey of the state of the art[J]. Information Fusion, 2017, 33: 100-112.
[10] BHAT S, KOUNDAL D. Multi-focus image fusion techniques: a survey[J]. Artificial Intelligence Review, 2021, 54(8): 5735-5787.
[11] LIU Y, WANG L, CHENG J, et al. Multi-focus image fusion: a survey of the state of the art[J]. Information Fusion, 2020, 64: 71-91.
[12] ZHANG Q, SHI T, WANG F, et al. Robust sparse representation based multi-focus image fusion with dictionary construction and local spatial consistency[J]. Pattern Recognition, 2018, 83: 299-313.
[13] ZHANG Q, LIU Y, BLUM R S, et al. Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: a review[J]. Information Fusion, 2018, 40: 57-75.
[14] AMIN-NAJI M, AGHAGOLZADEH A. Multi-focus image fusion in DCT domain using variance and energy of Laplacian and correlation coefficient for visual sensor networks[J]. Journal of AI and Data Mining, 2018, 6(2): 233-250.
[15] KOU L, ZHANG L, ZHANG K, et al. A multi-focus image fusion method via region mosaicking on Laplacian pyramids[J]. PLoS One, 2018, 13(5): e0191085.
[16] BAVIRISETTI D P, XIAO G, ZHAO J, et al. Multi-scale guided image and video fusion: a fast and efficient approach[J]. Circuits, Systems, and Signal Processing, 2019, 38(12): 5576-5605.
[17] PAUL S, SEVCENCO I S, AGATHOKLIS P. Multi-exposure and multi-focus image fusion in gradient domain[J]. Journal of Circuits, Systems and Computers, 2016, 25(10): 1650123.
[18] LIU Y, LIU S, WANG Z. A general framework for image fusion based on multi-scale transform and sparse representation[J]. Information Fusion, 2015, 24: 147-164.
[19] LI J, GUO X, LU G, et al. DRPL: deep regression pair learning for multi-focus image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4816-4831.
[20] LIU Y, CHEN X, PENG H, et al. Multi-focus image fusion with a deep convolutional neural network[J]. Information Fusion, 2017, 36: 191-207.
[21] AMIN-NAJI M, AGHAGOLZADEH A, EZOJI M. Ensemble of CNN for multi-focus image fusion[J]. Information Fusion, 2019, 51: 201-214.
[22] YANG Y, NIE Z, HUANG S, et al. Multilevel features convolutional neural network for multifocus image fusion[J]. IEEE Transactions on Computational Imaging, 2019, 5(2): 262-273.
[23] TONG H S, WU X J, LI H. Improved dual channel pulse coupled neural network and its application to multi-focus image fusion[EB/OL]. [2023-03-25]. https://arxiv.org/abs/2002. 01102.
[24] GUO X, NIE R, CAO J, et al. Fully convolutional network-based multifocus image fusion[J]. Neural Computation, 2018, 30(7): 1775-1800.
[25] GUO X, NIE R, CAO J, et al. FuseGAN: learning to fuse multi-focus image via conditional generative adversarial network[J]. IEEE Transactions on Multimedia, 2019, 21(8): 1982-1996.
[26] GUO X, MENG L, MEI L, et al. Multi-focus image fusion with siamese self-attention network[J]. IET Image Processing, 2020, 14(7): 1339-1346.
[27] WANG Y, XU S, LIU J, et al. MFIF-GAN: a new generative adversarial network for multi-focus image fusion[J]. Signal Processing: Image Communication, 2021, 96: 116295.
[28] MA B, ZHU Y, YIN X, et al. SESF-Fuse: an unsupervised deep model for multi-focus image fusion[J]. Neural Computing and Applications, 2021, 33(11): 5793-5804.
[29] XU H, FAN F, ZHANG H, et al. A deep model for multi-focus image fusion based on gradients and connected regions[J]. IEEE Access, 2020, 8: 26316-26327.
[30] XU K, QIN Z, WANG G, et al. Multi-focus image fusion using fully convolutional two-stream network for visual sensors[J]. KSII Transactions on Internet and Information Systems, 2018, 12(5): 2253-2272.
[31] ZHAO W, WANG D, LU H. Multi-focus image fusion with a natural enhancement via a joint multi-level deeply supervised convolutional neural network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 29(4): 1102-1115.
[32] LI H, NIE R, CAO J, et al. Multi-focus image fusion using U-shaped networks with a hybrid objective[J]. IEEE Sensors Journal, 2019, 19(21): 9755-9765.
[33] ZHANG Y, LIU Y, SUN P, et al. IFCNN: a general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54: 99-118.
[34] HUANG J, LE Z, MA Y, et al. A generative adversarial network with adaptive constraints for multi-focus image fusion[J]. Neural Computing and Applications, 2020, 32: 15119-15129.
[35] DUAN Z, ZHANG T, LUO X, et al. DCKN: multi-focus image fusion via dynamic convolutional kernel network[J]. Signal Processing, 2021, 189: 108282.
[36] XIAO B, WU H, BI X. DTMNet: a discrete Tchebichef moments-based deep neural network for multi-focus image fusion[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021.
[37] XU H, MA J, LE Z, et al. FusionDN: a unified densely connected network for image fusion[C]//Proceedings of the 2020 AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2020: 12484-12491.
[38] ZHANG H, LE Z, SHAO Z, et al. MFF-GAN: an unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion[J]. Information Fusion, 2021, 66: 40-53.
[39] MA J, LE Z, TIAN X, et al. SMFuse: multi-focus image fusion via self-supervised mask-optimization[J]. IEEE Transactions on Computational Imaging, 2021, 7: 309-320.
[40] NEJATI M, SAMAVI S, SHIRANI S. Multi-focus image fusion using dictionary-based sparse representation[J]. Information Fusion, 2015, 25: 72-84.
[41] XU S, WEI X, ZHANG C, et al. MFFW: a new dataset for multi-focus image fusion[EB/OL]. [2023-03-25]. https://arxiv.org/abs/2002.04780.
[42] SAVI? S, BABI? Z. Multifocus image fusion based on the first level of empirical mode decomposition[C]//Proceedings of the 2012 19th International Conference on Systems, Signals and Image Processing, Vienna, Apr 11-13, 2012. Piscataway: IEEE, 2012: 604-607.
[43] MA H, LIAO Q, ZHANG J, et al. An α-matte boundary defocus model-based cascaded network for multi-focus image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 8668-8679.
[44] LIU Z, BLASCH E, XUE Z, et al. Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 34(1): 94-109.
[45] QU G, ZHANG D, YAN P. Information measure for performance of image fusion[J]. Electronics Letters, 2002, 38(7): 313-315.
[46] HOSSNY M, NAHAVANDI S, CREIGHTON D. Comments on ‘Information measure for performance of image fusion’[J]. Electronics Letters, 2008, 44(18): 1066-1067.
[47] BULANON D M, BURKS T F, ALCHANATIS V. Image fusion of visible and thermal images for fruit detection[J]. Biosystems Engineering, 2009, 103(1): 12-22.
[48] CUI G, FENG H, XU Z, et al. Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition[J]. Optics Communications, 2015, 341: 199-209.
[49] RAJALINGAM B, PRIYA R. Hybrid multimodality medical image fusion technique for feature enhancement in medical diagnosis[J]. International Journal of Engineering Science Invention, 2018, 2: 52-60.
[50] XYDEAS C S, PETROVIC V. Objective image fusion performance measure[J]. Electronics Letters, 2000, 36(4): 308-309.
[51] ESKICIOGLU A M, FISHER P S. Image quality measures and their performance[J]. IEEE Transactions on Communications, 1995, 43(12): 2959-2965.
[52] WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[53] LI S, HONG R, WU X. A novel similarity based quality metric for image fusion[C]//Proceedings of the 2008 International Conference on Audio, Language and Image Processing, Shanghai, Jul 7-9, 2008. Piscataway: IEEE, 2008: 167-172.
[54] CHEN Y, BLUM R S. A new automated quality assessment algorithm for image fusion[J]. Image and Vision Computing, 2009, 27(10): 1421-1432.
[55] HAN Y, CAI Y, CAO Y, et al. A new image fusion performance metric based on visual information fidelity[J]. Information Fusion, 2013, 14(2): 127-135.
[56] DO C M, JAVIDI B. Multifocus holographic 3-D image fusion using independent component analysis[J]. Journal of Display Technology, 2007, 3(3): 326-332.
[57] BENES R, DVORAK P, FAUNDEZ-ZANUY M, et al. Multi-focus thermal image fusion[J]. Pattern Recognition Letters, 2013, 34(5): 536-544.
[58] COSTA M G F, PINTO K M B, FUJIMOTO L B M, et al. Multi-focus image fusion for bacilli images in conventional sputum smear microscopy for tuberculosis[J]. Biomedical Signal Processing and Control, 2019, 49: 289-297.
[59] GABARDA S, CRISTóBAL G. On the use of a joint spatial-frequency representation for the fusion of multi-focus images[J]. Pattern Recognition Letters, 2005, 26(16): 2572-2578.
[60] CAO T, DINH A, WAHID K A, et al. Multi-focus fusion technique on low-cost camera images for canola phenotyping[J]. Sensors, 2018, 18(6): 1887.
[61] ZHANG X, LI X, LIU Z, et al. Multi-focus image fusion using image-partition-based focus detection[J]. Signal Processing, 2014, 102: 64-76.
[62] ZHANG X. Deep learning-based multi-focus image fusion: a survey and a comparative study[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(9): 4819-4838.