Survey of Tongue Segmentation in Deep Learning

doi:10.3778/j.issn.1673-9418.2106080

Abstract

Abstract:

Tongue segmentation is an essential part of intelligent medicine diagnosis. The purpose is to generate an accurate contour of the tongue region by a precise mask. In recent years, deep learning methods have been widely applied in the field of image processing and have achieved impressive performance. With the increasing requirement of the performance for medical image segmentation, many scholars have employed deep learning to tongue segmentation. The methods of deep learning-based tongue segmentation are analyzed, classified and summarized. In the field of tongue segmentation applications, various tongue segmentation methods based on deep learning are divided into eight types: convolutional neural network (CNN), fully convolutional network (FCN), convolutional model with graphical model, encoder-decoder based model, regional convolutional network-based model, atrous convolutional model, transfer learning and other methods. This paper presents a comprehensive survey of the recently developed deep learning for tongue segmentation, and analyzes the strengths and weaknesses of these methods. The commonly used data sets and evaluation indexes of tongue segmentation based on deep learning are visually compared and quantitatively evaluated. This survey is concluded by discussing the potential development in future research work.

Key words: tongue segmentation, deep learning, convolutional neural network (CNN)

摘要：

舌体分割是智能医学诊断的重要组成部分，其目的是通过分割舌诊图像生成精准的舌体轮廓。近年来，深度学习方法在图像处理领域得到了广泛的应用并取得了较好的结果。随着医学图像分割对性能的要求越来越高，许多研究人员将深度学习运用到舌体分割中。主要对基于深度学习的舌体分割方法研究现状进行分析梳理和归纳总结。在舌体分割应用领域中，以各种深度学习方法作为研究对象，将基于深度学习的舌体分割方法划分为卷积神经网络（CNN）、全卷积网络（FCN）、卷积模型与图形模型、基于编解码器的模型、基于区域卷积网络模型、扩张卷积模型结构、迁移学习以及其他方法。在每类方法中，针对其改进和扩展的研究成果进行了全面的论述，总结分析其优势与不足；并对基于深度学习的舌体分割常用的数据集和评价指标进行了视觉比较与性能评估；最后讨论了未来研究工作中的发展潜力。

关键词: 舌体分割, 深度学习, 卷积神经网络（CNN）

LIU Huilin, FENG Yue, XU Hong, LUO Jianyi. Survey of Tongue Segmentation in Deep Learning[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(12): 2276-2291.

刘慧琳, 冯跃, 徐红, 罗坚义. 深度学习的舌体分割研究综述[J]. 计算机科学与探索, 2021, 15(12): 2276-2291.

References

[1] JI L Y, JIANG X Y, GAO Y B, et al. ADR-Net: context extraction network based on M-Net for medical image segmentation[J]. Medical Physics, 2020, 47(9): 4254-4264.
[2] ANASTASI J K, CHANG M, QUINN J, et al. Tongue inspection in TCM: observations in a study sample of patients living with HIV[J]. Medical Acupuncture, 2014, 26(1): 15-22.
[3] LIU W X, ZHOU C G, LI Z Y, et al. Patch-driven tongue image segmentation using sparse representation[J]. IEEE Access, 2020, 8: 41372-41383.
[4] XIANG L W. Research on image feature extraction improved algorithm for images of spleen deficiency tongue image recognition[D]. Ganzhou: Jiangxi University of Science and Technology, 2017.
项里伟. 脾虚舌象识别的图像特征提取优化算法研究[D]. 赣州: 江西理工大学, 2017.
[5] JIANG S, HU J, XIA C M, et al. A tongue image separation method based on Otsu threshold method and morphological adaptive correction[J]. Chinese High Technology Letters, 2017, 27(2): 150-155.
姜朔, 胡洁, 夏春明, 等. 基于Otsu阈值法与形态学自适应修正的舌像分割方法[J]. 高技术通讯, 2017, 27(2): 150-155.
[6] HUANG Z P, HAUNG Y S, YI F L, et al. An automatic tongue segmentation algorithm based on OTSU and region growing[J]. Lishizhen Medicine and Materia Medica Research, 2017, 28(12): 3062-3064.
黄展鹏, 黄益栓, 易法令, 等. 基于最大类间方差法和区域生长的舌体自动分割[J]. 时珍国医国药, 2017, 28(12): 3062-3064.
[7] YAN J J, XU Z, GUO R, et al. Research on tongue image segmentation based on active contour model of force field[J]. China Journal of Traditional Chinese Medicine and Pharmacy, 2019, 34(8): 3725-3727.
颜建军, 徐姿, 郭睿, 等. 基于力场活动轮廓模型的舌图像分割研究[J]. 中华中医药杂志, 2019, 34(8): 3725-3727.
[8] CHEN Z F, DENG X, LU Z T. A new and fast method for automatic tongue cancer image segmentation[J]. Chinese Journal of Medical Physics, 2020, 37(8): 1022-1029.
陈之锋, 邓旋, 卢振泰. 一种新的舌癌图像快速自动分割方法[J]. 中国医学物理学杂志, 2020, 37(8): 1022-1029.
[9] JIANG Z Y. A study of tongue segmentation from image and tongue recognition and classification algorithm[D]. Qinhuangdao: Yanshan University, 2017.
蒋再毅. 舌像分割与舌体识别及分类算法研究[D]. 秦皇岛: 燕山大学, 2017.
[10] LIANG F X, YANG F, LU L Y, et al. Review of brain tumor segmentation methods based on convolutional neural networks[J]. Computer Engineering and Applications, 2021, 57(7): 34-43.
梁芳烜, 杨锋, 卢丽云, 等. 基于卷积神经网络的脑肿瘤分割方法综述[J]. 计算机工程与应用, 2021, 57(7): 34-43.
[11] YANG P W, ZHOU Y H, XING G, et al. Applications of convolutional neural network in biomedical image[J]. Computer Engineering and Applications, 2021, 57(7): 44-58.
杨培伟, 周余红, 邢岗, 等. 卷积神经网络在生物医学图像上的应用进展[J]. 计算机工程与应用, 2021, 57(7): 44-58.
[12] MA Q P, XIE L B, PENG L. Application of improved convolutional neural network in medical image segmentation[J]. Laser & Optoeletronics Progress, 2020, 57(14): 190-196.
马其鹏, 谢林柏, 彭力. 改进的卷积神经网络在医学影像分割中的应用[J]. 激光与光电子学进展, 2020, 57(14): 190-196.
[13] LIAO D A, LU H, XU X P, et al. Image segmentation based on deep learning features[C]//Proceedings of the 2019 11th International Conference on Advanced Computational Intelligence, Guilin, Jun 7-9, 2019. Piscataway: IEEE, 2019: 296- 301.
[14] MINAEE S, BOYKOV Y, PORIKLI F, et al. Image segmentation using deep learning: a survey[J]. arXiv:2001.05566, 2020.
[15] ZHUANG S H, LI X, TIAN Z K, et al. Research on tongue instrument based on bibliometrics[J]. Modernization of Traditional Chinese Medicine and Materia Medica-World Science and Technology, 2020, 22(5): 1545-1552.
庄淑涵, 李馨, 田之魁, 等. 基于文献计量学的舌象仪研究[J]. 世界科学技术-中医药现代化, 2020, 22(5): 1545-1552.
[16] WANG D J, ZHU Q Q, YU J Y, et al. Atlas and visualization analysis of scientific knowledge in the field of tongue objectification research based on CiteSpace[J]. Modernization of Traditional Chinese Medicine and Materia Medica-World Science and Technology, 2021, 23(1): 283-291.
王东军, 朱青青, 余静寅, 等. 基于CiteSpace的舌诊客观化研究领域科学知识图谱与可视化分析[J]. 世界科学技术-中医药现代化, 2021, 23(1): 283-291.
[17] TIAN C Y, SUN X, ZHUANG S H, et al. Bibliometric analysis of tongue segmentation techniques[J]. Journal of Tianjin University of Traditional Chinese Medicine, 2019, 38(5): 461-467.
田春颖, 孙璇, 庄淑涵, 等. 舌象分割技术的文献计量学分析[J]. 天津中医药大学学报, 2019, 38(5): 461-467.
[18] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
[19] KRIZHEVSKY B A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90.
[20] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C]//Proceedings of the 3rd International Conference on Learning Representations, San Diego, May 7-9, 2015: 1-12.
[21] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE International Conference on Robotics and Automation, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 770-778.
[22] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 2261-2269.
[23] LIN B Q, XLE J W, LI C H, et al. Deeptongue: tongue segmentation via Resnet[C]//Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, Apr 15-20, 2018. Piscataway: IEEE, 2018: 1035-1039.
[24] PINHEIRO P H O, COLLOBERT R, DOLLáR P. Learning to segment object candidates[C]//Proceedings of the Annual Conference on Neural Information Processing Systems 2015, Montreal, Dec 7-12, 2015. Red Hook: Curran Associates, 2015: 1990-1998.
[25] LI J, XU B C, BAN X J, et al. A tongue image segmentation method based on enhanced HSV convolutional neural network[C]//LNCS 10451: Proceedings of the 14th International Conference on Cooperative Design, Visualization, and Engineering, Mallorca, Sep 17-20, 2017. Cham: Springer, 2017: 252-260.
[26] CAI Y Z, WANG T, LIU W, et al. A robust interclass and intraclass loss function for deep learning based tongue segmentation[J]. Concurrency and Computation: Practice and Experience, 2020, 32(22): 1-13.
[27] LIU Z H, CHEN L, TONG L, et al. Deep learning based brain tumor segmentation: a survey[J]. arXiv:2007.09479, 2020.
[28] YUAN W, LIU C. Cascaded CNN for real-time tongue segmentation based on key points localization[C]//Proceedings of the 4th IEEE International Conference on Big Data Analytics, Beijing, Mar 15-18, 2019. Washington: IEEE Computer Society, 2019: 303-307.
[29] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 39(4): 640-651.
[30] LI X L, YANG T, HU Y Y, et al. Automatic tongue image matting for remote medical diagnosis[C]//Proceedings of the 2017 IEEE International Conference on Bioinformatics and Biomedicine, Kansas City, Nov 13-16, 2017. Washington: IEEE Computer Society, 2017: 561-564.
[31] WANG L, HE X, TANG Y P, et al. Tongue semantic segmentation based on fully convolutional neural network[C]//Proceedings of the 2019 International Conference on Intelligent Computing, Automation and Systems, Chongqing, Dec 6-8, 2019. Piscataway: IEEE, 2019: 298-301.
[32] XUE Y S, LI X Q, WU P, et al. Automated tongue segmentation in Chinese medicine based on deep learning[C]//LNCS 11307: Proceedings of the 25th International Conference on Neural Information Processing, Siem Reap, Dec 13-16, 2018. Cham: Springer, 2018: 542-553.
[33] ROTHER C, KOLMOGOROV V, BLAKE A. GrabCut: interactive foreground extraction using iterated graph cuts[J]. ACM Transactions on Graphics, 2004, 23(3): 309-314.
[34] KR?HENBüHL P, KOLTUN V. Efficient inference in fully connected CRFs with Gaussian edge potentials[C]//Procee-dings of the 25th Annual Conference on Neural Information Processing Systems, Spain, Dec 12-17, 2011. Red Hook: Curran Associates, 2011: 109-117.
[35] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
[36] ZHANG X F, GUO Y T, CAI Y H, et al. Tongue image segmentation algorithm based on deep convolutional neural network and fully conditional random fields[J]. Journal of Beijing University of Aeronautics and Astronautics, 2019, 45(12): 2364-2374.
张新峰, 郭宇桐, 蔡轶珩, 等. 基于DCNN和全连接CRF的舌图像分割算法[J]. 北京航空航天大学学报, 2019, 45(12): 2364-2374.
[37] BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495.
[38] QU P L, ZHANG H, ZHUO L, et al. Automatic tongue image segmentation for traditional Chinese medicine using deep neural network[C]//LNCS 10361: Proceedings of the 13th International Conference on Intelligent Computing Theories and Application, Liverpool, Aug 7-10, 2017. Cham: Springer, 2017: 247-259.
[39] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//LNCS 9351: Proceedings of the 2015 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Oct 5-9, 2015. Cham: Springer, 2015: 234-241.
[40] XU Q, ZENG Y, TANG W J, et al. Multi-task joint learning model for segmenting and classifying tongue images using a deep neural network[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 14(8): 2481-2489.
[41] TRAJANOVSKI S, SHAN C, WEIJTMANS P J C, et al. Tongue tumor detection in hyperspectral images using deep learning semantic segmentation[J]. IEEE Transactions on Biomedical Engineering, 2021, 68(4): 1330-1340.
[42] LI L, LUO Z, ZHANG M, et al. An iterative transfer learning framework for cross-domain tongue segmentation[J]. Concurrency Computation, 2020, 32(14): 1-11.
[43] ZHOU J, ZHANG Q, ZHANG B, et al. TongueNet: a precise and fast tongue segmentation system using U-Net with a morphological processing layer[J]. Applied Sciences, 2019, 9(15): 3128-3147.
[44] YUAN Y H, CHEN X L, WANG J D. Object-contextual representations for semantic segmentation[C]//LNCS 12351: Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 173-190.
[45] MA L X, YANG H, SONG T T, et al. Research on segmentation algorithm of tongue image based on high resolution feature[J]. Computer Engineering, 2020, 46(10): 248-252.
马龙祥, 杨浩, 宋婷婷, 等. 基于高分辨率特征的舌象分割算法研究[J]. 计算机工程, 2020, 46(10): 248-252.
[46] SUN G Y, HUANG H, ZHANG A Z, et al. Fusion of multiscale convolutional neural networks for building extraction in very high-resolution images[J]. Remote Sensing, 2019, 11(3): 227.
[47] GIRSHICK R B, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Jun 23-28, 2014. Washington: IEEE Computer Society, 2014: 580-587.
[48] GIRSHICK R B. Fast R-CNN[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015. Washington: IEEE Computer Society, 2015: 1440-1448.
[49] REN S Q, HE K M, GIRSHICK R B, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149.
[50] HE K M, GKIOXARI G, DOLLáR P, et al. Mask R-CNN[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2): 386-397.
[51] GHOLAMI E, TABBAKH S R K, KHEIRABADI M. Proposing method to increase the detection accuracy of stomach cancer based on color and lint features of tongue using CNN and SVM[J]. arXiv:2011.09962, 2020.
[52] ZHOU C, FAN H Y, LI Z Y. TongueNet: accurate localization and segmentation for tongue images using deep neural networks[J]. IEEE Access, 2019, 7: 148779-148789.
[53] YAN J J, XU Z, GUO R, et al. Research on tongue image segmentation based on mask R-CNN[J]. Modernization of Traditional Chinese Medicine and Materia Medica-World Science and Technology, 2020, 22(5): 1532-1538.
颜建军, 徐姿, 郭睿, 等. 基于Mask R-CNN的舌图像分割研究[J]. 世界科学技术-中医药现代化, 2020, 22(5): 1532-1538.
[54] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs[J]. arXiv:1412.7062, 2014.
[55] CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethin-king atrous convolution for semantic image segmentation[J]. arXiv:1706.05587, 2017.
[56] ZHENG Y J, KAMBHAMETTU C. Learning based digital matting[C]//Proceedings of the 12th IEEE International Conference on Computer Vision, Kyoto, Sep 27-Oct 4, 2009. Washington: IEEE Computer Society, 2009: 889-896.
[57] LU Y X, LI X G, ZHANG H, et al. Review on tongue image segmentation technologies for traditional Chinese medicine: methodologies, performances and prospects[J]. Acta Automatica Sinica, 2021, 47(5): 1005-1016.
卢运西, 李晓光, 张辉, 等. 中医舌象分割技术研究进展: 方法、性能与展望[J]. 自动化学报, 2021, 47(5): 1005-1016.
[58] YIN Z, YIU V, HU X L, et al. End-to-end face parsing via interlinked convolutional neural networks[J]. Cognitive Neurodynamics, 2020, 15: 169-179.
[59] LIU Y L, SHI H L, SHEN H, et al. A new dataset and boundary-attention semantic segmentation for face parsing[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence, the 32nd Innovative Applications of Artificial Intelligence Conference, the 10th AAAI Symposium on Educational Advances in Artificial Intelligence, New York, Feb 7-12, 2020. Menlo Park: AAAI, 2020: 11637-11644.
[60] KHAN K, ATTIQUE M, KHAN R, et al. A multi-task framework for facial attributes classification through end-to-end face parsing and deep convolutional neural networks[J]. Sensors, 2020, 20: 328.
[61] WEI Z, LIU S, SUN Y, et al. Accurate facial image parsing at real-time speed[J]. IEEE Transactions on Image Processing, 2019, 28(9): 4659-4670.
[62] LIU Y L, SHI H L, SI Y, et al. A high-efficiency framework for constructing large-scale face parsing benchmark[J]. arXiv:1905.04830, 2019.
[63] LI X L, YANG D W, WANG Y, et al. TCMINet: face parsing for traditional Chinese medicine inspection via a hybrid neural network with context aggregation[J]. IEEE Access, 2020, 8: 93069-93082.
[64] SHAO Y W. Research on intelligent tongue diagnosis based on deep learning[D]. Xiamen: Xiamen University, 2018.
邵尤伟. 基于深度学习的智能舌诊方法研究[D]. 厦门: 厦门大学, 2018.
[65] ZENG X Y, ZHANG Q, CHEN J, et al. Boundary guidance hierarchical network for real-time tongue segmentation[J]. arXiv:2003.06529, 2020.
[66] LI Y T, LUO Y S, ZHU Z M. Tongue image analysis in traditional Chinese medicine based on deep learning[J]. Computer Science, 2020, 47(11): 148-158.
李渊彤, 罗裕升, 朱珍民. 基于深度学习的舌象特征分析[J]. 计算机科学, 2020, 47(11): 148-158.
[67] YANG K Q, LI J D, LI X Q. Unsupervised tongue segmentation using reference labels[C]//LNCS 12532: Proceedings of the 27th International Conference on Neural Information Processing, Bangkok, Nov 23-27, 2020. Cham: Springer, 2020: 603-615.
[68] ZHOU C, FAN H Y, ZHAO W, et al. Reconstruction enhanced probabilistic model for semisupervised tongue image segmentation[J]. Concurrency and Computation: Practice and Experience, 2020, 32(22): e5844.
[69] BIOHIT. BioHit tongue dataset[EB/OL]. [2021-06-21]. https://github.com/BioHit/TongeImageDataset.
[70] GUI M J, ZHANG X F, HU G Q, et al. A study on tongue image color description based on label distribution learning[C]//Proceedings of the 8th International Conference on BioMedical Engineering and Informatics, Shenyang, Oct 14-16, 2015. Piscataway: IEEE, 2015: 148-152.
[71] KALAKE L, WAN W G, HOU L. Analysis based on recent deep learning approaches applied in real-time multi-object tracking: a review[J]. IEEE Access, 2021, 9: 32650-32671.