U型卷积网络在乳腺医学图像分割中的研究综述

doi:10.3778/j.issn.1673-9418.2307069

摘要/Abstract

摘要： U-Net及其变体模型在乳腺医学图像分割领域展现了卓越的性能，U-Net采用全卷积网络（FCN）结构进行语义分割，U-Net对称结构的高度灵活性和适应性可以通过调整网络深度、引入新的模块来适应不同的图像分割任务和挑战，这种创新结构对后续网络设计产生了深远影响。深入探讨了基于U型卷积网络在乳腺医学图像分割中的应用，并对近年来用于乳腺医学图像分割的U型卷积网络进行了分类与归纳。针对U-Net网络结构改进的乳腺医学图像分割技术进行了如下总结。阐述了目前广泛使用的乳腺医学图像数据集及评价指标，陈述了常用的数据增强方法；详细介绍了U-Net模型的网络结构以及用于乳腺医学图像的传统分割方法；对用于乳腺医学图像分割方法的U型网络结构按照残差结构、多尺度特征、膨胀机制、注意力机制、跳跃连接机制、结合Transformer等方面改进进行归纳总结。讨论了当下乳腺医学图像分割所遇到的问题与挑战，对未来的研究走向做出了展望。

关键词: 医学图像分割, U型卷积网络, 深度学习, 乳腺疾病, 图像处理

Abstract: U-Net and its variants have showcased exceptional performance in the domain of breast medical image segmentation. By employing a fully convolutional network (FCN) structure for semantic segmentation, the symmetrical structure of U-Net offers remarkable flexibility and adaptability. It can be tailored to diverse image segmentation tasks and challenges by adjusting network depth and incorporating new modules, leaving a significant impact on subsequent network designs. This paper aims to delve into the application of U-shaped convolutional networks in breast medical image segmentation, categorizing and summarizing U-shaped convolutional networks used for this purpose in recent years. It outlines the widely used breast medical image datasets and evaluation metrics, discusses common data augmentation techniques, and provides a detailed introduction to the network structure of the U-Net model along with traditional segmentation methods for breast medical images. Furthermore, it summarizes the improvements made to the U-Net network structure for breast medical image segmentation, including modifications like residual structures, multi-scale features, dilation mechanisms, attention mechanisms, skip connection mechanisms, and integration with Transformers. Finally, it addresses the current challenges and problems encountered in breast medical image segmentation and offers insights into future research directions.

Key words: medical image segmentation, U-shaped convolutional network, deep learning, breast disease, image processing

蒲秋梅, 殷帅, 李正茂, 赵丽娜. U型卷积网络在乳腺医学图像分割中的研究综述[J]. 计算机科学与探索, 2024, 18(6): 1383-1403.

PU Qiumei, YIN Shuai, LI Zhengmao, ZHAO Lina. Review of U-Net-Based Convolutional Neural Networks for Breast Medical Image Segmentation[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(6): 1383-1403.

参考文献

[1] DUFFY S W, TABáR L, YEN A M F, et al. Mammography screening reduces rates of advanced and fatal breast cancers: results in 549, 091 women[J]. Cancer, 2020, 126(13): 2971-2979.
[2] DEVITA V T, LAWRENCE T S, ROSENBERG S A. Cancer: principles & practice of oncology: primer of the molecular biology of cancer[M]. [S.l.]: Lippincott Williams & Wilkins, 2012.
[3] AKRAM S U, KANNALA J, EKLUND L, et al. Cell segmentation proposal network for microscopy image analysis[M]//Deep Learning and Data Labeling for Medical Applications. Cham: Springer, 2016: 21-29.
[4] SUN L, HE J, YIN X, et al. An image segmentation framework for extracting tumors from breast magnetic resonance images[J]. Journal of Innovative Optical Health Sciences, 2018, 11(4): 185001.
[5] ASGARI TAGHANAKI S, ABHISHEK K, COHEN J P, et al. Deep semantic segmentation of natural and medical images: a review[J]. Artificial Intelligence Review, 2021, 54: 137-178.
[6] QURESHI I, YAN J, ABBAS Q, et al. Medical image segmentation using deep semantic-based methods: a review of techniques, applications and emerging trends[J]. Information Fusion, 2023, 90: 316-352.
[7] BINDHU V. Biomedical image analysis using semantic segmentation[J]. Journal of Innovative Image Processing, 2019, 1(2): 91-101.
[8] ROTH H R, SHEN C, ODA H, et al. Deep learning and its application to medical mage segmentation[J]. Medical Imaging Technology, 2018, 36(2): 63-71.
[9] 周涛, 侯森宝, 陆惠玲, 等. 探析U-Net的改进机制及其在医学图像分割的应用[J]. 生物医学工程学杂志, 2022, 39(4): 806-825.
ZHOU T, HOU S B, LU H L, et al. Exploring and analyzing the improvement mechanism of U-Net and its application in medical image segmentation[J]. Journal of Biomedical Engineering, 2022, 39(4): 806-825.
[10] 周涛, 董雅丽, 霍兵强, 等. U-Net网络医学图像分割应用综述[J]. 中国图象图形学报, 2021, 26(9): 2058-2077.
ZHOU T, DONG Y L, HUO B Q, et al. U-Net and its applications in medical image segmentation:a review[J]. Journal of Image and Graphics, 2021, 26(9): 2058-2077.
[11] HUANG B, TIAN J, ZHANG H, et al. Deep semantic segmentation feature-based radiomics for the classification tasks in medical image analysis[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 25(7): 2655-2664.
[12] SINHA A, DOLZ J. Multi-scale self-guidedattention for medical image segmentation[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 25(1): 121-130.
[13] TAWFIK N, ELNEMR H A, FAKHR M, et al. Hybrid pixel-feature fusion system for multi-modal medical images[J]. Journal of Ambient Intelligence and Humanized Computing, 2021, 12(6): 6001-6018.
[14] AZAM M A, KHAN K B, SALAHUDDINS, et al. A review on multimodal medical image fusion: compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics[J]. Computers in Biology and Medicine, 2022, 144: 105253.
[15] HERMESSI H, MOURALI O, ZAGROUBA E. Multimodal medical image fusion review: theoretical background and recent advances[J]. Signal Processing, 2021, 183: 108036.
[16] KUMAR N, VERMA R, SHARMA S, et al. A dataset and a technique for generalized nuclear segmentation for computational pathology[J]. IEEE Transactions on Medical Imaging, 2017, 36(7): 1550-1560.
[17] SPANHOL F A, OLIVEIRA L S, PETITJEAN C, et al. A dataset for breast cancer histopathological image classification[J]. IEEE Transactions on Biomedical Engineering, 2015, 63(7): 1455-1462.
[18] BEJNORDI B E, VETA M, VAN DIEST PJ, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer[J]. Jama, 2017, 318(22): 2199-2210.
[19] BANDI P, GEESSINK O, MANSON Q, et al. From detection of individual metastases to classification of lymph node status at the patient level: the Camelyon17 challenge[J]. IEEE Transactions on Medical Imaging, 2018, 38(2): 550-560.
[20] BRANCATI N, ANNICIELLO A M, PATI P, et al. BRACS: a dataset for breast carcinoma subtyping in H&E histology images[J]. arXiv:2111.04740, 2021.
[21] MOREIRA I C, AMARAL I, DOMINGUESI, et al. INbreast: toward a full-field digital mammographic database[J]. Academic Radiology, 2012, 19(2): 236-248.
[22] 茅枫, 张燕娜, 刘乾宇, 等. 国外公开乳腺影像数据集的现状分析及其启示[J]. 中国医学科学院学报, 2021, 43(2): 271-277.
MAO F, ZHANG Y N, LIU Q Y, et al. Analysis and enlightenment of foreign open breast image datasets[J]. Acta Academiae Medicinae Sinicae, 2021, 43(2) :271-277.
[23] ROSE C, TURI D, WILLIAMS A, et al. Web services for the DDSM and digital mammography research[C]//Proceedings of the 2006 International Workshop on Digital Mammography, Manchester, Jun 18-21, 2006. Berlin, Heidelberg: Springer, 2006: 376-383.
[24] Digital mammography dataset[EB/OL]. [2023-05-16]. http://www.bcsc-research.org/data/ma-mmography_dataset.
[25] AL-DHABYANI W, GOMAA M, KHALED H, et al. Dataset of breast ultrasound images[J]. Data in Brief, 2020, 28: 104863.
[26] YAP M H, GOYAL M, OSMAN F, et al. Breast ultrasound region of interest detection and lesion localisation[J]. Artificial Intelligence in Medicine, 2020, 107: 101880.
[27] ZEISER F A, DA COSTA C A, ZONTA T, et al. Segmentation of masses on mammograms using data augmentation and deep learning[J]. Journal of Digital Imaging, 2020, 33: 858-868.
[28] KARTHIGA R, NARASIMHAN K, AMIRTHARAJAN R. Diagnosis of breast cancer for modern mammography using artificial intelligence[J]. Mathematics and Computers in Simulation, 2022, 202: 316-330.
[29] RAZALI N F, ISA I S, SULAIMAN S N, et al. High-level features in deeper deep learning layers for breast cancer classification[C]//Proceedings of the 2021 11th IEEE International Conference on Control System, Computing and Engineering, Penang, Aug 27-28, 2021. Piscataway: IEEE, 2021: 170-175.
[30] ALYAFI B, DIAZ O, MARTI R. DCGANs for realistic breast mass augmentation in X-ray mammography[C]//Proceedings of the Medical Imaging 2020: Computer-Aided Diagnosis, Houston, Feb 16-19, 2020. San Francisco: SPIE, 2020: 473-480.
[31] WU E, WU K, LOTTER W. Synthesizing lesions using contextual GANs improves breast cancer classification on mammograms[J]. arxiv:2006.00086, 2020.
[32] CHHIPA P C, UPADHYAY R, PIHLGREN G G, et al. Magnification prior: a self-supervised method for learning representations on breast cancer histopathological images[C]//Proceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa,?Jan 2-7, 2023. Piscataway: IEEE, 2023: 2717-2727.
[33] 沈嘉琳, 汪源源, 王涌, 等. 基于灰度阈值分割和动态规划的超声图像乳腺肿瘤边缘提取[J]. 航天医学与医学工程, 2005(4): 281-286.
SHEN J L, WANG Y Y, WANG Y, et al. Boundary extraction of ultrasonic breast tumor image based on gray-level threshold segmentation and dynamic programming[J]. Space Medicine & Medical Engineering, 2005(4): 281-286.
[34] HOUSSEIN E H, EMAM M M, ALI A A. An efficient multi-level thresholding segmentation method for thermography breast cancer imaging based on improved chimp optimization algorithm[J]. Expert Systems with Applications, 2021, 185: 115651.
[35] 冯宝, 陈业航, 刘壮盛, 等. 结合MRF能量和模糊速度的乳腺癌图像分割方法[J]. 自动化学报, 2020, 46(6): 1188-1199.
FENG B, CHEN Y H, LIU Z S, et al. Segmentation of breast cancer on DCE-MRI images with MRF energy and fuzzy speed function[J]. Acta Automatica Sinica, 2020, 46(6): 1188-1199.
[36] 周皓阳, 冯宝, 齐菲菲, 等. 结合MRF能量和DCE-MRI时域特征的乳腺癌灶分割算法[J]. 光子学报, 2021, 50(6): 205-216.
ZHOU H Y, FENG B, QI F F，et al. Combining MRF energy and DCE-MRI time-domain features for breast tumors segmentation algorithm[J]. Acta Photonica Sinica, 2021, 50(6):205-216.
[37] 陈磊, 张凯, 金振成. 数字乳房X射线图像的肿块分割算法研究[J]. 生物医学工程学杂志, 2008, 25(6): 1282-1284.
CHEN L, ZHANG K, JIN Z C. Study of mass segmentation algorithm for digital mammogram[J]. Journal of Biomedical Engineering, 2008, 25(6): 1282-1284.
[38] AMEER S F, NAYYEF Z T, FAHAD Z H, et al. Using morphological operation and watershed techniques for breast cancer detection[J]. International Journal of Online & Biomedical Engineering, 2020, 16(5): 140.
[39] 刘元振, 林伟, 朱玲英, 等. 区域生长法结合多竞争最小二乘拟合算法去除乳腺X线影像图像中胸大肌影[J]. 中国医学影像技术, 2022, 38(6): 923-927.
LIU Y Z, LIN W, ZHU L Y，et al. Regional growth method combining with multi-competitive least-squares for removal of pectoralis major in mammographic images[J]. Chinese Journal of Medical Imaging Technology, 2022, 38(6): 923-927.
[40] RAY S, DAS A, DHAL K G, et al. Whale optimizer-based clustering for breast histopathology image segmentation[J].International Journal of Swarm Intelligence Research, 2022, 13(1): 1-29.
[41] 范虹, 张程程, 侯存存, 等. 结合双树复小波变换和改进密度峰值快速搜索聚类的乳腺MR图像分割[J]. 电子学报, 2019, 47(10): 2149-2157.
FAN H, ZHANG C C, HOU C C, et al. Dual-tree complex wavelet transform and improved density peak fast search and clustering method for breast MR image segmentation[J]. Acta Electronica Sinical, 2019, 47(10): 2149-2157.
[42] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 7-12, 2015. Piscataway: IEEE, 2015: 3431-3440.
[43] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Oct 5-9, 2015. Cham: Springer, 2015: 234-241.
[44] VITTIKOP B S,DHOTRE S R. Automatic segmentation of MRI images for brain tumor using UNet[C]//Proceedings of the 2019 1st International Conference on Advances in Information Technology, Chikmagalur, Jul 25-27, 2019. Piscataway: IEEE, 2019: 507-511.
[45] 宋杰, 肖亮, 练智超, 等. 基于深度学习的数字病理图像分割综述与展望[J]. 软件学报, 2021, 32(5): 1427-1460.
SONG J, XIAO L, LIAN Z C, et al. Overview and prospect of deep learning for image segmentation in digital pathology [J]. Journal of Software, 2021, 32(5): 1427-1460.
[46] LOU A, GUAN S, LOEW M. DC-UNet: rethinking the U-Net architecture with dual channel efficient CNN for medical image segmentation[C]//Proceedings of the Medical Imaging 2021: Image Processing, Feb 15-19, 2021. San Francisco: SPIE, 2021: 758-768.
[47] ZHU H H, LIU D C, LIU J Y, et al. RMS-SE-UNet: a segmentation method for tumors in breast ultrasound images[C]//Proceedings of the 2021 IEEE 6th International Conference on Computer and Communication Systems, Chengdu, Apr 23-26, 2021. Piscataway: IEEE, 2021: 328-334.
[48] IBTEHAZ N, RAHMAN M S. MultiResUNet: rethinking the U-Net architecture for multimodal biomedical image segmentation[J]. Neural Networks, 2020, 121: 74-87.
[49] YU K, CHEN S, CHEN Y. Tumor segmentation in breast ultra-sound image by means of RES path combined with dense connection neural network[J]. Diagnostics, 2021, 11(9): 1565.
[50] 秦传波, 宋子玉, 曾军英, 等. 联合多尺度和注意力-残差的深度监督乳腺癌分割[J]. 光学精密工程, 2021, 29(4): 877-895.
QIN C B, SONG Z Y, ZENG J Y, et al. Deeply supervised breast cancer segmentation combined with multi-scale and attention-residuals[J]. Optics and Precision Engineering, 2021, 29(4): 877-895.
[51] YUAN S, QIU Z, LI P, et al. RMAU-Net: breast tumor segmentation network based on residual depthwise separable convolution and multiscale channel attention gates[J]. Applied Sciences, 2023, 13(20): 11362.
[52] CHEN G, DAI Y, ZHANG J. RRCNet: refinement residual convolutional network for breast ultrasound images segmentation[J]. Engineering Applications of Artificial Intelligence, 2023, 117: 105601.
[53] CHEN G, ZHOU L, ZHANG J, et al. ESKNet: an enhanced adaptive selection kernel convolution for ultrasound breast tumors segmentation[J]. Expert Systems with Applications, 2024: 123265.
[54] HAI J, QIAO K, CHEN J, et al. Fully convolutional densenet with multiscale context for automated breast tumor segmentation[J]. Journal of Healthcare Engineering, 2019: 8415485.
[55] PI J, QI Y, LOU M, et al. FS-UNet: mass segmentation in mammograms using an encoder-decoder architecture with feature strengthening[J]. Computers in Biology and Medicine, 2021, 137: 104800.
[56] LIU G, WANG J, LIU D, et al. A multiscale nonlocal feature extraction network for breast lesion segmentation in ultra-sound images[J]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 5011012.
[57] QI W, WU H C, CHAN S C. MDF-Net: a multi-scale dynamic fusion network for breast tumor segmentation of ultra-sound images[J]. IEEE Transactions on Image Processing, 2023, 32: 4842-4855.
[58] HAN C, HUANG X, ZHANG Y, et al. ECU-Net: multi-scale salient boundary detection and contrast feature enhancement U-Net for breast ultrasound image segmentation[J]. Signal, Image and Video Processing, 2023, 17(5): 2287-2295.
[59] LU Z, SHE C, WANG W, et al. LM-Net: a light-weight and multi-scale network for medical image segmentation[J]. Computers in Biology and Medicine, 2024, 168: 107717.
[60] PAN X, LI L, YANG D, et al. An accurate nuclei segmentation algorithm in pathological image based on deep semantic network[J]. IEEE Access, 2019, 7: 110674-110686.
[61] 唐浩漾, 王燕, 宋聪, 等. 基于空洞U-Net网络的乳腺细胞图像分割算法[J]. 光电子·激光, 2021, 32(5): 470-476.
TANG H Y, WANG Y, SONG C, et al. Breast cell image segmentation algorithm based on dilated U-Net network[J]. Journal of Optoelectronics·Laser, 2021, 32(5): 470-476.
[62] CHANG H T, LI J D, ALKHALEEFAH M, et al. 3D dense U-Net-ASPP for MRI imaging segmentation of breast tumors[C]//Proceedings of the 2023 International Conference on Consumer Electronics, Pingtung, China, Jul 17-19, 2023. Piscataway: IEEE, 2023: 847-848.
[63] SUN H, LI C, LIU B, et al. AUNet: attention-guided dense-upsampling networks for breast mass segmentation in whole mammograms[J]. Physics in Medicine & Biology, 2020, 65(5): 055005.
[64] LI S, DONG M, DU G, et al. Attention dense-U-Net for automatic breast mass segmentation in digital mammogram[J]. IEEE Access, 2019, 7: 59037-59047.
[65] 罗圣钦, 陈金怡, 李洪均. 基于注意力机制的多尺度残差UNet实现乳腺癌灶分割[J]. 计算机应用, 2022, 42(3): 818-824.
LUO S Q, CHEN J Y, LI H J. Multiscale residual UNet based on attention mechanism to realize breast cancer lesion segmentation[J]. Journal of Computer Applications, 2022, 42(3): 818-824.
[66] ZHU D, WANG Y, YANG M, et al. SU-Next: an image segmentation model of breast tumors based on U-Next and attention mechanisms[J]. Journal of Physics: Conference Series, 2023, 2575(1): 012001.
[67] YOU G, YANG X, LI X, et al. EfficientUNet: an efficient solution for breast tumour segmentation in ultrasound images [J]. IET Image Processing, 2024, 18: 523-534.
[68] MALEKMOHAMMADI A, SORYANI M, KOZEGAR E. Mass segmentation in automated breast ultrasound using an enhanced attentive UNet[J]. Expert Systems with Applications, 2024: 123095.
[69] 徐春燕, 谢嘉伟, 杨春霞, 等. 基于病理穿刺切片组织形态学分析的乳腺癌新辅助化疗疗效预测[J]. 四川大学学报(医学版), 2021, 52(2): 279-285.
XU C Y, XIE J W, YANG C X, et al. Prediction of response to neoadjuvant chemotherapy for breast cancer based on histomorphology analysis ofneedle biopsy images[J]. Journal of Sichuan University (Medical Sciences), 2021, 52(2): 279-285.
[70] BACCOUCHE A, GARCIA-ZAPIRAIN B, CASTILLO OLEA C, et al. Connected-UNets: a deep learning architecture for breast mass segmentation[J]. NPJ Breast Cancer, 2021, 7(1): 1-12.
[71] MISRA S, YOON C, KIM K J, et al. Deep learning-based multimodal fusion network for segmentation and classification of breast cancers using B-mode and elastography ultrasound images[J]. Bioengineering & Translational Medicine, 2022: e10480.
[72] QIU H T, SHI C. LM-UNet: lateral MLP augmented U-Net for medical image segmentation[J]. Research Square, 2023: 1-20.
[73] CHOWDARY G J, YOAGARAJAH P. EU-Net: enhanced U-shaped network for breast mass segmentation[J]. IEEE Journal of Biomedical and Health Informatics, 2023: 1-11.
[74] IQBAL A, SHARIF M. BTS-ST: swin Transformer network for segmentation and classification of multimodality breast cancer images[J]. Knowledge-Based Systems, 2023, 267: 110393.
[75] LI B, SU B, ZHANG X. U2-MNet: an improved neural network for breast tumor segmentation[J]. Journal of Physics: Conference Series, 2023, 2637(1): 012003.
[76] ZHONG Y, PIAO Y. Mix Transformer depth-wise separable convolution UNet for breast mass segmentation in mammographic[C]//Proceedings of the 5th International Conference on Artificial Intelligence and Computer Science. San Francisco: SPIE, 2023: 466-472.
[77] 霍璐, 胡晓欣, 肖勤, 等. 基于nnU-Net的乳腺DCE-MR图像中乳房和腺体自动分割[J]. 波谱学杂志, 2021, 38(3): 367-380.
HUO L, HU X X, XIAO Q, et al. Automatic segmentation of breast and fibroglandular tissues in DCE-MR images based on nnU-Net[J]. Chinese Journal of Magnetic Resonance, 2021, 38(3): 367-380.
[78] YAN Y, LIU Y, WU Y, et al. Accurate segmentation of breast tumors using AE U-net with HDC model in ultrasound images[J]. Biomedical Signal Processing and Control, 2022, 72: 103299.
[79] DAMANIA K, ANGEL ARUL JOTHI J. Graph convolutional neural networks for nuclei segmentation from histopathology images[C]//Proceedings of the 2022 International Conference on Soft Computing and Its Engineering Applications, Changa, Dec 9-10, 2022. Cham: Springer, 2022: 158-169.
[80] 谢远志, 闫士举, 魏高峰, 等. 基于U-Net++和对抗性学习网络的乳腺肿块分割[J]. 激光与光电子学进展, 2022, 59(16): 390-396.
XIE Y Z, YAN S J, WEI G F, et al. Breast mass segmentation based on U-Net++ and adversarial learning network [J]. Laser & Optoelectronics Progress, 2022, 59(16): 390-396.
[81] LIU L, CHENG J, QUAN Q, et al. A survey on U-shaped networks in medical image segmentations[J]. Neurocomputing, 2020, 409: ?244-258.
[82] 冯龙锋, 陈英, 周滔辉, 等. CT图像肺及肺病变区域分割方法综述[J]. 中国图象图形学报, 2022, 27(3): 722-749.
FENG L F, CHEN Y, ZHOU T H, et al. Review of human lung and lung lesion regions segmentation methods based on CT images[J]. Journal of Image and Graphics, 2022, 27(3): 722-749.
[83] ISAKSSON L J, SUMMERS P, MASTROLEO F, et al. Automatic segmentation with deep learning in radiotherapy[J]. Cancers, 2023, 15(17): 4389.
[84] MESE I, INAN N G, KO?ADAGLI O, et al. ChatGPT-assisted deep learning model for thyroid nodule analysis: beyond artifical intelligence[J]. Medical Ultrasonography, 2023, 25(4): 375-383.