可变形卷积网络研究综述

doi:10.3778/j.issn.1673-9418.2209076

摘要/Abstract

摘要： 近年来，随着深度学习的快速发展，可变形卷积网络因其强大的特征提取能力受到广泛关注，克服了卷积神经网络中难以解决的一些问题，并且已在计算机视觉、自然语言处理等相关领域发挥重要作用。由于目前对可变形卷积网络进行系统性总结的研究还很少，为了给后续研究提供详细的参考依据，对可变形卷积网络引入以来的相关工作进行总结。首先，综述了近几年的高质量文献，从不变性特征的角度入手，对可变形卷积网络中的可变形卷积和可变形感兴趣区域池化等核心技术进行介绍。然后，将收集到的相关文献按照研究领域的不同进行分类，全面概括现阶段可变形卷积网络在图像识别和分类、目标检测、图像分割、目标追踪等研究领域的应用情况，同时还对比了重要网络模型的性能和优缺点。其次，通过梳理文献，分析可变形卷积网络存在的优势和不足，并根据现阶段存在的一些问题，探讨可变形卷积网络未来可能的研究趋势。最后，基于不变性特征的提取对可变形卷积网络进行了总结和展望。

关键词: 特征提取, 可变形卷积, 不变性特征, 感兴趣区域

Abstract: In recent years, with the rapid development of deep learning, deformable convolutional networks have received extensive attention because of their powerful feature extraction capabilities, overcoming some problems that are difficult to solve in convolutional neural networks, and have played an important role in computer vision, natural language processing and other related fields. Since there is a little research on systematic summary of the deformable convolutional network, in order to provide a detailed reference for subsequent research, this paper summarizes the related work since the introduction of the deformable convolutional network. Firstly, this paper reviews the high-quality literature in recent years, and introduces the core technologies such as deformable convolution and deformable region of interest pooling in deformable convolutional networks from the perspective of invariant features. Secondly, the collected relevant literature is classified according to different research fields, and the appli-cation of deformable convolutional networks in image recognition and classification, target detection, image seg-mentation, target tracking and other research fields is comprehensively summarized. At the same time, the perfor-mance, advantages and disadvantages of important network models are listed. Thirdly, by combing the literature, the advantages and disadvantages of the deformable convolutional network are analyzed, and the possible future research trends of the deformable convolutional network are discussed according to some problems existing at the present stage. Finally, the deformable convolutional networks are summarized and prospected based on invariant feature extraction.

Key words: feature extraction, deformable convolution, invariance feature, region of interest

刘卫光, 刘东, 王璐. 可变形卷积网络研究综述[J]. 计算机科学与探索, 2023, 17(7): 1549-1564.

LIU Weiguang, LIU Dong, WANG Lu. Survey of Deformable Convolutional Networks[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(7): 1549-1564.

参考文献

[1] DAI J, QI H, XIONG Y, et al. Deformable convolutional networks[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision,Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 764-773.
[2] AZULAY A, WEISS Y. Why do deep convolutional net-works generalize so poorly to small image transformations[J]. Machine Learning Research, 2019, 20: 1-25.
[3] COHEN T S, WELLING M. Group equivariant convolu-tional networks[J]. arXiv:1602.07576, 2016.
[4] FOLLMANN P, B?TTGER T. A rotationally-invariant con-volution module by feature map back-rotation[C]//Procee-dings of the 2018 IEEE Winter Conference on Applications of Computer Vision, Lake Tahoe, Mar 12-15, 2018. Washington: IEEE Computer Society, 2018: 784-792.
[5] JEON Y, KIM J. Active convolution: learning the shape of convolution for image classification[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Reco-gnition, Honolulu, Jul 21-26, 2017. Washington: IEEE Com-puter Society, 2017: 1846-1854.
[6] 魏驰宇, 刘蓉, 刘明, 等. 改进FCOS的复杂场景口罩佩戴检测算法[J]. 计算机工程与应用, 2023, 59(11): 188-194.
WEI C Y, LIU R, LIU M, et al. Improved algorithm of FCOS for complex scene mask wear detection[J]. Computer Engi-neering and Applications, 2023, 59(11): 188-194 .
[7] ZHU X Z, HU H, LIN S, et al. Deformable ConvNets v2: more deformable, better results[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recogni-tion, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 9308-9316.
[8] ZHU J, FANG L, GHAMISI P. Deformable convolutional neural networks for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(8): 1254-1258.
[9] 唐婷, 潘新. 一种基于可变形卷积的高光谱图像分类算法[J]. 光电子·激光, 2022, 33(5): 488-494.
TANG T, PAN X. A hyperspectral image classification algo-rithm based on deformable convolution[J]. Journal of Opto-electronics·Laser, 2022, 33(5): 488-494.
[10] DONG X, TAN L, ZHOU L, et al. An action recognition method based on deformable convolution network[J]. Journal of Physics. Conference Series, 2020, 1487(1): 12033.
[11] 王雪娇, 智敏. 基于可变形卷积神经网络的人体动作识别[J]. 计算机工程与科学, 2021, 43(1): 105-111.
WANG X J, ZHI M. Human motion recognition based on deformable convolutional neural network[J]. Computer Engi-neering & Science, 2021, 43(1): 105-111.
[12] WANG H, ZHANG Y, LIU C, et al. sEMG based hand ges-ture recognition with deformable convolutional network[J]. International Journal of Machine Learning and Cybernetics, 2022, 13(6): 1729-1738.
[13] MA P, MA J, WANG X, et al. Deformable convolutional networks for multi-view 3D shape classification[J]. Electro-nics Letters, 2018, 54(24): 1373-1375.
[14] YAO X, SUN K, BU X, et al. Classification of white blood cells using weighted optimized deformable convolutional neural networks[J]. Artificial Cells, Nanomedicine, and Bio-technology, 2021, 49(1): 147-155.
[15] 安鑫, 孙昊, 卓力, 等. 基于可变形卷积和自适应二维位置编码的鲁棒车牌识别方法[J]. 测控技术, 2023, 42(3): 11-18.
AN X, SUN H, ZHUO L, et al. Robust license plate recog-nition based on deformable convolution and adaptive 2D positional encoding[J]. Measurement & Control Technology, 2023, 42(3): 11-18.
[16] 王鉴, 张荣福. 基于可变形卷积的单帧图像眼球定位追踪[J]. 光学仪器, 2021, 43(6): 26-31.
WANG J, ZHANG R F. Single-frame image eyeball trac-king based on deformable convolution[J]. Optical Instruments, 2021, 43(6): 26-31.
[17] 黄凤琪, 陈明, 冯国富. 基于可变形卷积的改进YOLO目标检测算法[J]. 计算机工程, 2021, 47(10): 269-275.
HUANG F Q, CHEN M, FENG G F. Improved YOLO object detection algorithm based on deformable convolution[J]. Com-puter Engineering, 2021, 47(10): 269-275.
[18] 赵轩, 周凡, 余汉成. 基于改进特征提取及融合模块的YOLOv3模型[J]. 电子科技, 2022, 35(7): 40-45.
ZHAO X, ZHOU F, YU H C. Improved YOLOv3 model based on new feature extraction and fusion module[J]. Elec-tronic Science and Technology, 2022, 35(7): 40-45.
[19] LI D Y, WANG G F, ZHANG Y, et al. Coal gangue detec-tion and recognition algorithm based on deformable convo-lution YOLOv3[J]. IET Image Processing, 2022, 16(1): 134-144.
[20] 包俊, 刘宏哲. 融合可变形卷积网络的鱼眼图像目标检测[J]. 计算机工程, 2021, 47(4): 248-255.
BAO J, LIU H Z. Object detection in fisheye images com-bining deformable convolutional networks[J]. Computer Engi-neering, 2021, 47(4): 248-255.
[21] WEI X, WEI Y, LU X. RMDC: rotation-mask deformable convolution for object detection in top-view fisheye cameras[J]. Neurocomputing, 2022, 504: 99-108.
[22] GUAN S Y, ZHANG W Y, JIANG Y F. A surface defect detection method of the magnesium alloy sheet based on deformable convolution neural network[J]. Metalurgija, 2020, 59(3): 325-328.
[23] LIU Z, YANG B, DUAN G, et al. Visual defect inspection of metal part surface via deformable convolution and con-catenate feature pyramid neural networks[J]. IEEE Transac-tions on Instrumentation and Measurement, 2020, 69(12): 9681-9694.
[24] 李海丰, 景攀, 韩红阳. 基于可变形卷积与特征融合的机场道面裂缝检测算法[J]. 南京航空航天大学学报, 2021, 53(6): 981-988.
LI H F, JING P, HAN H Y. Airport pavement crack detec-tion algorithm based on deformable convolution and fea-ture fusion[J]. Journal of Nanjing University of Aeronautics & Astronautics, 2021, 53(6): 981-988.
[25] 孙志超, 王博, 张晓玲. 基于可变形残差卷积与伸缩式特征金字塔算法的PCB缺陷检测[J/OL]. 电讯技术(2022-04-27) [2023-04-11]. http://kns.cnki.net/kcms/detail/51.1267.tn.20220427.1608.012.html.
SUN Z C, WANG B, ZHANG X L. PCB defect detection based on deformable residual convolution and scalable fea-ture pyramid algorithm[J/OL]. Telecommunication Enginee-ring (2022-04-27) [2023-04-11]. http://kns.cnki.net/kcms/detail/51.1267.tn.20220427.1608.012.html.
[26] 朱红艳, 李泽平, 赵勇, 等. 基于多尺度融合和可变形卷积PCB缺陷检测算法[J]. 计算机工程与设计, 2022, 43(8): 2188-2196.
ZHU H Y, LI Z P, ZHAO Y, et al. PCB defect detection algorithm based on multi-scale fusion and deformable con-volution[J]. Computer Engineering and Design, 2022, 43(8): 2188-2196.
[27] 赵丽娟, 柳长安, 张正, 等. 输电线路中销钉缺陷的自适应检测技术研究[J]. 华中科技大学学报(自然科学版), 2023, 51(2): 109-115.
ZHAO L J, LIU C A, ZHANG Z, et al. Research on adaptive detection technology for pin defects in transmission lines[J]. Journal of Huazhong University of Science and Technology (Nature Science Edition), 2023, 51(2): 109-115.
[28] HAQUE M F, KANG D. Object localization and detection using SALNet with deformable convolutional network[J]. The Journal of Korean Institute of Information Technology, 2020, 18(1): 27-34.
[29] LIU Z, SHEN C, FAN X, et al. Scale-aware limited defor-mable convolutional neural networks for traffic sign detec-tion and classification[J]. IET Intelligent Transport Systems, 2020, 14(12): 1712-1722.
[30] LI C, ZHANG D, TIAN Z, et al. Few-shot learning with deformable convolution for multiscale lesion detection in mammography[J]. Medical Physics, 2020, 47(7): 2970-2985.
[31] DENG L, YANG M, LI H, et al. Restricted deformable con-volution-based road scene semantic segmentation using sur-round view cameras[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 21(10): 4350-4362.
[32] PLAYOUT C, AHMAD O, LECUE F, et al. Adaptable defor-mable convolutions for semantic segmentation of fisheye ima-ges in autonomous driving systems[J]. arXiv:2102.10191, 2021.
[33] JIN Q, MENG Z, PHAM T D, et al. DUNet: a deformable network for retinal vessel segmentation[J]. Knowledge-Based Systems, 2019, 178: 149-162.
[34] GURITA A, MOCANU I G. Image segmentation using encoder-decoder with deformable convolutions[J]. Sensors, 2021, 21(5): 1570.
[35] CHEN H, DING L, YAO F, et al. Panoptic segmentation of UAV images with deformable convolution network and mask scoring[C]//Proceedings of the 12th International Conference on Graphics and Image Processing, Xi??an, Nov 13-15, 2020. San Francisco: SPIE, 2021: 312-321.
[36] 张彬彬, 帕孜来·马合木提. 可变形有效感受野的人体图像语义分割算法[J]. 光电子·激光, 2021, 32(9): 953-961.
ZHANG B B, PAZILAT M. Body image semantic segmen-tation algorithm based on deformable effective receptive field[J]. Journal of Optoelectronics·Laser, 2021, 32(9): 953-961.
[37] 盛克峰, 李文星. 基于可变形卷积和语义嵌入式注意力机制的眼球超声图像分割方法[J]. 计算机系统应用, 2022, 31(2): 342-349.
SHENG K F, LI W X. Eyeball ultrasound image segmen-tation based on deformable convolution and semantic embed-ded attention mechanism[J]. Computer Systems & Applica-tions, 2022, 31(2): 342-349.
[38] ZHANG Y, TANG Y, FANG B, et al. Multi-object tracking using deformable convolution networks with tracklets upda-ting[J]. International Journal of Wavelets, Multiresolution and Information Processing, 2019, 17(6): 1950042.
[39] CAO W, CHEN X. Deformable convolutional networks trac-ker[J]. Destech Transactions on Computer Science and Engi-neering, 2019, 28(3): 231-235.
[40] LIU F, LIU D, TIAN J, et al. Cascaded one-shot deformable convolutional neural networks: developing a deep learning model for respiratory motion estimation in ultrasound sequ-ences[J]. Medical Image Analysis, 2020, 65: 101793.
[41] JIE L, LEI H, ZHIQIANG W, et al. Multi-task learning with deformable convolution[J]. Journal of Visual Communication and Image Representation, 2021, 77: 103-109.
[42] CHENG K, LUBAMBA E K, LIU Q. Action prediction based on partial video observation via context and temporal sequential network with deformable convolution[J]. IEEE Access, 2020, 8: 133527-133540.
[43] 翟强, 王陆洋,殷保群, 等. 基于尺度自适应卷积神经网络的人群计数算法[J]. 计算机工程, 2020, 46(2): 250-254.
ZHAI Q, WANG L Y, YIN B Q, et al. Crowd counting algorithm based on scale adaptive convolutional neural net-work[J]. Computer Engineering, 2020, 46(2): 250-254.
[44] BERROUK S, EL FAZZIKI A, SADGAL M. Hybrid defor-mable convolutional with recurrent neural network for optimal traffic congestion prediction: a fuzzy logic congestion index system[J]. International Journal of Advanced Computer Scie-nce and Applications, 2022, 13(5): 0130575.
[45] NIELSEN A H, IOSIFIDIS A, KARSTOFT H. Forecasting large-scale circulation regimes using deformable convolu-tional neural networks and global spatiotemporal climate data[J]. Scientific Reports, 2022, 12(1): 1-12.
[46] PENG J, BAO C, HU C, et al. Automated mammographic mass detection using deformable convolution and multiscale features[J]. Medical & Biological Engineering & Compu-ting, 2020, 58(7): 1405-1417.
[47] 王梦南, 赵涓涓, 肖宁, 等. 基于可变形卷积神经网络的肺结节假阳性识别[J]. 计算机工程与设计, 2022, 43(6): 1732-1739.
WANG M N, ZHAO J J, XIAO N, et al. False positive identification of pulmonary nodules based on deformable convolutional neural network[J]. Computer Engineering and Design, 2022, 43(6): 1732-1739.
[48] YING X, WANG L, WANG Y, et al. Deformable 3d con-volution for video super-resolution[J]. IEEE Signal Proces-sing Letters, 2020, 27: 1500-1504.
[49] WANG Y, YANG J, WANG L, et al. Light field image super-resolution using deformable convolution[J]. IEEE Transac-tions on Image Processing, 2020, 30: 1057-1071.
[50] KIM J, JI S, BAEK S, et al. Depth map super-resolution using guided deformable convolution[J]. IEEE Access, 2021, 9: 66626-66635.
[51] 蔡非凡, 万旺根. 基于可变形非局部三维卷积网络的视频超分辨率重建算法[J]. 工业控制计算机, 2022, 35(3): 54-56.
CAI F F, WAN W G. Video super resolution with deformable non-local 3D convolutional network[J]. Industrial Control Computer, 2022, 35(3): 54-56.
[52] 王文庆, 庞颖, 刘洋, 等. 基于可变形卷积的图像边缘智能提取方法[J]. 西安邮电大学学报, 2021, 26(1): 84-89.
WANG W Q, PANG Y, LIU Y, et al. Image edge intelligent extraction method based on deformable convolution[J]. Journal of Xi??an University of Posts and Telecommunica-tions, 2021, 26(1): 84-89.
[53] YANG Z, LIU S, HU H, et al. RepPoints: point set repre-sentation for object detection[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 9656-9665.
[54] CHEN F, WU F, XU J, et al. Adaptive deformable convo-lutional network[J]. Neurocomputing, 2021, 453: 853-864.
[55] DENG L, GONG Y, LU X, et al. Focus-enhanced scene text recognition with deformable convolutions[C]//Proceedings of the 5th International Conference on Computer and Com-munications, Chengdu, Dec 6-9, 2019. Piscataway: IEEE, 2019: 1685-1689.
[56] LAI S C, TAN H K, LAU P Y. 3D deformable convolution for action classification in videos[C]//Proceedings of the International Workshop on Advanced Imaging Technology 2021, Kagoshima, Jan 5-6, 2021. San Francisco: SPIE, 2021: 149-154.
[57] XI W, SUN L, SUN J. Upgrade your network inplace with deformable convolution[C]//Proceedings of the 19th Inter-national Symposium on Distributed Computing and Applic-ations for Business Engineering and Science, Xuzhou, Oct 16-19, 2020. Piscataway: IEEE, 2020: 239-242.
[58] CHETLUR S, WOOLLEY C, VANDERMERSCH P, et al. CUDNN: efficient primitives for deep learning[J]. arXiv:1410.0759, 2014.
[59] SUN Z, OZAY M, OKATANI T. Design of kernels in con-volutional neural networks for image classification[C]//LNCS 9911: Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 11-14, 2016. Cham: Sprin-ger, 2016: 51-66.
[60] GAO H, ZHU X, LIN S, et al. Deformable kernels: adap-ting effective receptive fields for object deformation[J]. arXiv:1910.02940, 2019.
[61] DING X, GUO Y, DING G, et al. ACNet: strengthening the kernel skeletons for powerful CNN via asymmetric con-volution blocks[C]//Proceedings of the 2019 IEEE/CVF Inter-national Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 1911-1920.
[62] HE K, LI C, YANG Y, et al. Integrating large circular kernels into CNNs through neural architecture search[J]. arXiv:2107.02451, 2021.