Research Progress of Deep Learning in Retinal Vessel Segmentation

doi:10.3778/j.issn.1673-9418.2103099

Abstract

Abstract:

The retinal features obtained by retinal blood vessel segmentation can be used to assist the diagnosis of diabetic retinopathy and other ocular diseases. In recent years, the automatic segmentation algorithm of blood vessels based on deep learning has attracted a lot of research. The reason is that the method can automatically extract image features and has the advantages of high accuracy and fast speed. This paper reviews the research on retinal blood vessel segmentation based on deep learning in recent years. It first discusses the establishment of fundus image databases, commonly used data enhancement, image preprocessing, and image slicing operations. Then, recent deep learning algorithms are classified as cascaded neural network, multi-path neural network, multi-scale neural network in the perspective of network architecture, and these networks are carried out introduction, comparison, performance analysis, complexity analysis and disadvantage analysis. Besides, the introduction of the research on the actual deployment of neural networks is also given. The results show that the data amount in the existing fundus image database is still limited, and the most commonly used methods of data enhancement and image preprocessing are respectively horizontal and vertical flipping and image gray-scaling. Observed from the performance achieved by existing research, cascaded and multi-path neural networks are more suitable for retinal vessel segmentation. Observed from the existing complexity, the inference time of many models can reach the millisecond level, and the computational consumption can reach below million. Observed from the shortcomings of existing algorithms, an algorithm can only solve part of the existing challenges. In the case of mobile device hardware resource constraints, light-weight neural network is a direction worthy of exploration.

Key words: deep learning, retinal vessel segmentation, neural networks

摘要：

视网膜血管分割得到的视网膜特征可以用于辅助糖尿病视网膜病变等眼病的诊断。近年来基于深度学习的血管自动分割算法以自动提取图像特征、精度高、速度快的这些优点吸引了大量研究。对近年基于深度学习的视网膜血管分割研究进行回顾，包括常见的眼底图像数据库、常用的数据增强、图像预处理、图像切片的操作。从网络架构的角度将近期的深度学习血管分割算法归类为级联结构神经网络、多路径神经网络、多尺度神经网络，并对网络进行介绍、对比、性能分析、复杂度分析、缺点分析。同时对于神经网络现实部署的研究也进行了介绍。结果表明，现有眼底图像数据库的数据量还较少，数据增强和图像预处理较多使用方法分别为水平竖直翻转和图像灰度化。从现有研究达到的性能上看，级联结构和多路径的神经网络较为适合视网膜血管的分割；从现有的复杂度来看，部分模型的推断时间可以达到毫秒级，计算消耗可以达到兆以下；从现有算法的缺点看，某个算法只能解决部分现有挑战。在移动设备硬件资源限制的情况下，轻量级的神经网络是一个值得探索的方向。

关键词: 深度学习, 视网膜血管分割, 神经网络

LI Lanlan, ZHANG Xiaohui, NIU Decao, HU Yihuang, ZHAO Tiesong, WANG Dabiao. Research Progress of Deep Learning in Retinal Vessel Segmentation[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(11): 2063-2076.

李兰兰，张孝辉，牛得草，胡益煌，赵铁松，王大彪. 深度学习在视网膜血管分割上的研究进展[J]. 计算机科学与探索, 2021, 15(11): 2063-2076.

References

[1] LI T, BO W, HU C Y, et al. Applications of deep learning in fundus images: a review[J]. Medical Image Analysis, 2021, 69: 101971.
[2] WU Y C, XIA Y, SONG Y, et al. NFN+: a novel network followed network for retinal vessel segmentation[J]. Neural Networks, 2020, 126: 153-162.
[3] LI X, JIANG Y C, LI M L, et al. Lightweight attention convo-lutional neural network for retinal vessel segmentation[J]. IEEE Transactions on Industrial Informatics, 2021, 17(3): 1958-1967.
[4] WU H S, WANG W, ZHONG J F, et al. SCS-Net: a scale and context sensitive network for retinal vessel segmentation[J]. Medical Image Analysis, 2021, 70: 102025.
[5] WANG S J, YU L Q, LI K, et al. DoFE: domain-oriented feature embedding for generalizable fundus image segmenta-tion on unseen datasets[J]. IEEE Transactions on Medical Imaging, 2020, 39(12): 4237-4248.
[6] YAN Z Q, YANG X, CHENG K T. A three-stage deep lear-ning model for accurate retinal vessel segmentation[J]. IEEE Journal of Biomedical and Health Informatics, 2019, 23(4): 1427-1436.
[7] GOLKAR E, RABBANI H, DEHGHANI A. Hybrid registra-tion of retinal fluorescein angiography and optical coherence tomography images of patients with diabetic retinopathy[J]. Biomedical Optics Express, 2021, 12(3): 1707-1724.
[8] WANG D Y, HAYTHAM A, POTTENBURGH J, et al. Hard attention net for automatic retinal vessel segmentation[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 24(12): 3384-3396.
[9] RODRIGUES é O, CONCI A, LIATSIS P. ELEMENT: multi-modal retinal vessel segmentation based on a coupled region growing and machine learning approach[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 24(12): 3507-3519.
[10] JIN Q G, MENG Z P, PHAM T D, et al. DUNet: a deformable network for retinal vessel segmentation[J]. Knowledge-Based Systems, 2019, 178: 149-162.
[11] OLIVEIRA A, PEREIRA S, SILVA C A. Retinal vessel seg-mentation based on fully convolutional neural networks[J]. Expert Systems with Applications, 2018, 112: 229-242.
[12] WU Y C, XIA Y, SONG Y, et al. Multiscale network followed network model for retinal vessel segmentation[C]//LNCS 11071: Proceedings of the 21st International Conference on Medical Image Computing and Computer Assisted Interven-tion, Granada, Sep 16-20, 2018. Cham: Springer, 2018: 119-126.
[13] LIAN S, LI L, LIAN G R, et al. A global and local enhanced residual U-Net for accurate retinal vessel segmentation[J]. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2021, 18(3): 852-862.
[14] WANG W, ZHONG J F, WU H S, et al. RVSeg-Net: an efficient feature pyramid cascade network for retinal vessel segmentation[C]//LNCS 12265: Proceedings of the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, Lima, Oct 4-8, 2020. Cham: Springer, 2020: 796-805.
[15] XU H, ZHU Y H, ZHEN T, et al. Survey of image semantic segmentation methods based on deep neural network[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(1): 47-59.
徐辉, 祝玉华, 甑彤, 等. 深度神经网络图像语义分割方法综述[J]. 计算机科学与探索, 2021, 15(1): 47-59.
[16] VLACHOS M, DERMATAS E. Multi-scale retinal vessel segmentation using line tracking[J]. Computerized Medical Imaging and Graphics, 2010, 34(3): 213-227.
[17] ZHAO Y T, LIU Y H, WU X Q, et al. Retinal vessel segmentation: an efficient graph cut approach with retinex and local phase[J]. PLoS One, 2015, 10(4): e0122332.
[18] FRAZ M M, REMAGNINO P, HOPPE A, et al. An ensemble classification-based approach applied to retinal blood vessel segmentation[J]. IEEE Transactions on Biomedical Enginee-ring, 2012, 59(9): 2538-2548.
[19] LI Q L, FENG B W, XIE L P, et al. A cross-modality learning approach for vessel segmentation in retinal images[J]. IEEE Transactions on Medical Imaging, 2016, 35(1): 109-118.
[20] GU Z W, CHENG J, FU H Z, et al. CE-Net: context encoder network for 2D medical image segmentation[J]. IEEE Tran-sactions on Medical Imaging, 2019, 38(10): 2281-2292.
[21] ZHOU Y K, CHEN Z L, SHEN H L, et al. A refined equili-brium generative adversarial network for retinal vessel seg-mentation[J]. Neurocomputing, 2021, 437: 118-130.
[22] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. arXiv:1406.2661, 2014.
[23] MINAEE S, BOYKOV Y, PORIKLI F, et al. Image segmen-tation using deep learning: a survey[J]. arXiv:2001.05566, 2020.
[24] SON J, PARK S J, JUNG K H. Retinal vessel segmentation in fundoscopic images with generative adversarial networks[J]. arXiv:1706.09318, 2017.
[25] YAN Z Q, YANG X, CHENG K T. Joint segment-level and pixel-wise losses for deep learning based retinal vessel seg-mentation[J]. IEEE Transactions on Biomedical Engineering, 2018, 65(9): 1912-1923.
[26] GUO S, WANG K, KANG H, et al. BTS-DSN: deeply supe-rvised neural network with short connections for retinal vessel segmentation[J]. International Journal of Medical Informatics, 2019, 126: 105-113.
[27] XIAO X, LIAN S, LUO Z M, et al. Weighted Res-UNet for high-quality retina vessel segmentation[C]//Proceedings of the 2018 9th International Conference on Information Tech-nology in Medicine and Education, Hangzhou, Oct 19-21, 2018. Piscataway: IEEE, 2018: 327-331.
[28] ZHANG B H, HUANG S L, HU S H. Multi-scale neural networks for retinal blood vessels segmentation[J]. arXiv:1804.04206, 2018.
[29] XU R, YE X C, JIANG G L, et al. Retinal vessel segmenta-tion via a semantics and multi-scale aggregation network[C]//Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, Barcelona, May 4-8, 2020. Piscataway: IEEE, 2020: 1085-1089.
[30] WANG B, QIU S, HE H G. Dual encoding U-Net for retinal vessel segmentation[C]//LNCS 11764: Proceedings of the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention, Shenzhen, Oct 13-17, 2019. Cham: Springer, 2019: 84-92.
[31] HAJABDOLLAHI M, ESFANDIARPOOR R, NAJARIAN K, et al. Low complexity convolutional neural network for vessel segmentation in portable retinal diagnostic devices[C]//Proceedings of the 2018 IEEE International Conference on Image Processing, Athens, Oct 7-10, 2018. Piscataway: IEEE, 2018: 2785-2789.
[32] WU Y C, XIA Y, SONG Y, et al. Vessel-Net: retinal vessel segmentation under multi-path supervision[C]//LNCS 11764: Proceedings of the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention, Shenzhen, Oct 13-17, 2019. Cham: Springer, 2019: 264-272.
[33] CHERUKURI V, KUMAR B G V, BALA R, et al. Multi-scale regularized deep network for retinal vessel segmentation[C]//Proceedings of the 2019 IEEE International Conference on Image Processing, Taipei, China, Sep 22-25, 2019. Pis-cataway: IEEE, 2019: 824-828.
[34] FENG S T, ZHUO Z S, PAN D R, et al. CcNet: a cross-connected convolutional network for segmenting retinal vessels using multi-scale features[J]. Neurocomputing, 2019, 392: 268-276.
[35] SONG J, LEE B. Development of automatic retinal vessel segmentation method in fundus images via convolutional neural networks[C]//Proceedings of the 39th Annual Inter-national Conference of the IEEE Engineering in Medicine and Biology Society, Jeju Island, Jul 11-15, 2017. Piscata-way: IEEE, 2017: 681-684.
[36] LONG J, SHELHAMER E, DARRELL T. Fully convolu-tional networks for semantic segmentation[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 7-12, 2015. Washington: IEEE Computer Society, 2015: 3431-3440.
[37] RONNEBERGER O, FISCHER P, BROX T. U-net: convo-lutional networks for biomedical image segmentation[C]//LNCS 9351: Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Inter-vention, Munich, Oct 5-9, 2015. Cham: Springer, 2015: 234-241.
[38] MILLETARI F, NAVAB N, AHMADI S A. V-Net: fully convolutional neural networks for volumetric medical image segmentation[C]//Proceedings of the 4th International Con-ference on 3D Vision, Stanford, Oct 25-28, 2016. Washington: IEEE Computer Society, 2016: 565-571.
[39] TIAN C, FANG T, FAN Y L, et al. Multi-path convolutional neural network in fundus segmentation of blood vessels[J]. Biocybernetics and Biomedical Engineering, 2020, 40(2): 583-595.
[40] KHAN T M, ABDULLAH F, NAQVI S S, et al. Shallow vessel segmentation network for automatic retinal vessel segmentation[C]//Proceedings of the 2020 International Joint Conference on Neural Networks, Glasgow, Jul 19-24, 2020. Piscataway: IEEE, 2020: 1-7.
[41] HU K, ZHANG Z Z, NIU X R, et al. Retinal vessel segmen-tation of color fundus images using multiscale convolutional neural network with an improved cross-entropy loss func-tion[J]. Neurocomputing, 2018, 309: 179-191.
[42] ALOM M Z, HASAN M, YAKOPCIC C, et al. Recurrent residual convolutional neural network based on U-Net (R2U-Net) for medical image segmentation[J]. arXiv:1802.06955, 2018.
[43] LIU B, GU L, LU F. Unsupervised ensemble strategy for retinal vessel segmentation[C]//LNCS 11764: Proceedings of the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention, Shenzhen, Oct 13-17, 2019. Cham: Springer, 2019: 111-119.
[44] KHAN T M, NAQVI S S, ARSALAN M, et al. Exploiting residual edge information in deep fully convolutional neural networks for retinal vessel segmentation[C]//Proceedings of the 2020 International Joint Conference on Neural Networks, Glasgow, Jul 19-24, 2020. Piscataway: IEEE, 2020: 1-8.
[45] YANG L, WANG H X, ZENG Q S, et al. A hybrid deep segmentation network for fundus vessels via deep-learning framework[J]. Neurocomputing, 2021, 448: 168-178.
[46] GRIDACH M. PyDiNet: pyramid dilated network for medical image segmentation[J]. Neural Networks, 2021, 140: 274-281.
[47] LAIBACHER T, WEYDE T, JALALI S. M2U-Net: effective and efficient retinal vessel segmentation for resource-con-strained environments[J]. arXiv:1811.07738, 2018.
[48] WEI J, FAN Z. Genetic U-Net: automatically designing light-weight U-shaped CNN architectures using the genetic algo-rithm for retinal vessel segmentation[J]. arXiv:2010.15560, 2020.
[49] ATLI ?, GEDIK O S. Sine-Net: a fully convolutional deep learning architecture for retinal blood vessel segmentation[J]. Engineering Science and Technology, an International Journal, 2021, 24(2): 271-283.
[50] SAMUEL P M, VEERAMALAI T. VSSC Net: vessel specific skip chain convolutional network for blood vessel segmenta-tion[J]. Computer Methods and Programs in Biomedicine, 2021, 198: 105769.
[51] LIANG J J, WEI J J, JIANG Z F. Generative adversarial networks GAN overview[J]. Journal of Frontiers of Com-puter Science and Technology, 2020, 14(1): 1-17.
梁俊杰, 韦舰晶, 蒋正锋. 生成对抗网络GAN综述[J]. 计算机科学与探索, 2020, 14(1): 1-17.