基于生成对抗网络的舌象图像特征提取方法

doi:10.3778/j.issn.1673-9418.2408072

摘要/Abstract

摘要： 舌诊是中医特色诊法之一，《医镜》深刻阐释了舌象与人体健康状况、脏腑病变的密切关系。因此，现代中医智能舌诊的重要工作便是舌象图像特征处理，针对舌象图像特征分布规律的学习和特征提取困难的问题，提出一种基于生成对抗网络的舌象图像特征提取新方法（TongueIFE-GAN）。该方法通过对抗思想构建一种面向舌象图像潜在特征的提取网络，它包括判别器和生成器两个重要组成部分，藉此建立图像重构质量与舌象图像的特征提取效果映射关系。为增强深度算法的可解释性，在网络判别器中融入类激活映射机制，进一步优化编码器的特征处理性能，并对TongueIFE-GAN模型在提取特征时关注的图像敏感区域作出可视化解释。同时，在舌象图像分割、分类任务驱动下，新模型可自优化舌象特征提取的能力。通过多组实验，结果表明，基于TongueIFE-GAN模型的舌象分割和分类任务，其分割性能IoU与Dice指标值，以及分类准确率均优于基准模型和对比方法。TongueIFE-GAN利用对抗思想构建新型的特征提取以及注意力可视化机制的研究方法，可为舌象图像特征研究提供一种新的思路。

关键词: 生成对抗网络, 舌象图像, 类激活映射, 舌象图像分割, 舌象分类

Abstract: Tongue diagnosis is one of the characteristic diagnostic methods in traditional Chinese medicine (TCM). The book named Medical Mirror profoundly explains the intimate relationship between tongue images and human health status, as well as organ diseases. Therefore, the important task of modern intelligent tongue diagnosis in TCM is the processing for features of tongue image. To address these difficulties in learning the distribution rules of tongue image features and extracting these features, this paper proposes a novel method for extracting tongue image features based on the generative adversarial network (TongueIFE-GAN). This method constructs an extraction network for the latent features of tongue images through the adversarial idea. It consists of two important components, namely the discriminator and generator, thereby establishing a mapping relationship between the image reconstruction quality and the feature extraction effect of tongue images. To enhance the interpret ability of the deep algorithm, the class activation mapping mechanism (CAM) is integrated into the network discriminator to further optimize the feature processing performance of the encoder. Moreover, it provides a visual explanation of the sensitive image areas that the TongueIFE-GAN model focuses on during feature extraction. Meanwhile, driven by the tasks of tongue image segmentation and classification, the new model can self-optimize its ability to extract tongue features. Through various experiments, the experimental results demonstrate that the tongue image segmentation and classification task based on TongueIFE-GAN model has better segmentation performance and classification accuracy than the benchmark model and comparison methods on IoU and Dice indices. TongueIFE-GAN employs adversarial thinking to devise a novel feature extraction method and attention visualization mechanism, offering a fresh perspective on tongue image feature research.

Key words: generative adversarial network, tongue image, class activation mapping, tongue image segmentation, tongue classification

阮群生, 王硕诚, 吴清锋. 基于生成对抗网络的舌象图像特征提取方法[J]. 计算机科学与探索, 2025, 19(11): 3007-3022.

RUAN Qunsheng, WANG Shuocheng, WU Qingfeng. Method of Tongue Image Feature Extraction Based on Generative Adversarial Network[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(11): 3007-3022.

参考文献

[1] 江涛, 屠立平, 许家佗. 中医舌象智能诊断技术研究述评及展望[J]. 中国中医药信息杂志, 2024, 31(7): 182-187.
JIANG T, TU L P, XU J T. Review and prospects of research on artificial intelligence TCM tongue diagnosis technology[J]. Chinese Journal of Information on Traditional Chinese Medicine, 2024, 31(7): 182-187.
[2] 赵越, 杨玉强. 基于HIS色彩空间的水平集彩色图像分割算法[J]. 自动化技术与应用, 2017, 36(4): 25-27.
ZHAO Y, YANG Y Q. Algorithm for color image segmentation of level sets based on HIS color space[J]. Techniques of Automation and Applications, 2017, 36(4): 25-27.
[3] 张婷, 王琳. 中医舌诊客观化研究进展[J]. 实用中医内科杂志, 2024, 38(6): 82-85.
ZHANG T, WANG L. Research progress in objectification of tongue diagnosis[J]. Journal of Practical Traditional Chinese Internal Medicine, 2024, 38(6): 82-85.
[4] 陈恩纳, 李圣烨, 胡毓亲, 等. 中医舌诊图像采集与色彩信息分析研究进展[J]. 中华中医药杂志, 2024, 39(7): 3586-3589.
CHEN E N, LI S Y, HU Y Q, et al. Research progress on image acquisition and color information analysis of tongue diagnosis in traditional Chinese medicine[J]. China Journal of Traditional Chinese Medicine and Pharmacy, 2024, 39(7): 3586-3589.
[5] CHEN X, ZHAO J X, ISELIN K C, et al. Keratoconus detection of changes using deep learning of colour-coded maps[J]. BMJ Open Ophthalmology, 2021, 6(1): e000824.
[6] QURESHI I, YAN J H, ABBAS Q, et al. Medical image segmentation using deep semantic-based methods: a review of techniques, applications and emerging trends[J]. Information Fusion, 2023, 90: 316-352.
[7] SUN G C, DING S F, SUN T F, et al. SA-CapsGAN: using capsule networks with embedded self-attention for generative adversarial network[J]. Neurocomputing, 2021, 423: 399-406.
[8] 许家佗, 张志枫, 李蕾, 等. 基于图像分析的亚健康状态舌象颜色特征研究[J]. 辽宁中医杂志, 2010, 37(12): 2328-2330.
XU J T, ZHANG Z F, LI L, et al. Research on tongue image color features for sub-health state based on image analysis[J]. Liaoning Journal of Traditional Chinese Medicine, 2010, 37(12): 2328-2330.
[9] 张新峰, 焦月, 李欢欢, 等. 基于粒子群算法的Universum SVM参数选择[J]. 北京工业大学学报, 2013, 39(6): 840-845.
ZHANG X F, JIAO Y, LI H H, et al. Model parameter selection of the universum SVM based on particle swarm optimization[J]. Journal of Beijing University of Technology, 2013, 39(6): 840-845.
[10] 张翔, 胡广芹, 张新峰. 中医舌图像的质量评价研究[J]. 世界中西医结合杂志, 2017, 12(11): 1607-1611.
ZHANG X, HU G Q, ZHANG X F. Quality evaluation of tongue image in traditional Chinese medicine[J]. World Journal of Integrated Traditional and Western Medicine, 2017, 12(11): 1607-1611.
[11] 陆冠龙, 黄益栓, 张琦, 等. 融合舌象和形体特征的辅助中医体质辨识模型研究[J]. 时珍国医国药, 2019, 30(1): 244-246.
LU G L, HUANG Y S, ZHANG Q, et al. The study of auxiliary TCM constitution identification model based on tongue image and physical features[J]. Lishizhen Medicine and Materia Medica Research, 2019, 30(1): 244-246.
[12] QU P L, ZHANG H, ZHUO L, et al. Automatic analysis of tongue substance color and coating color using sparse representation-based classifier[C]//Proceedings of the 2016 International Conference on Progress in Informatics and Computing. Piscataway: IEEE, 2017: 289-294.
[13] 宋宁, 唐丽明, 杜红跃, 等. 基于聚类分析的糖尿病胃轻瘫中医证型分布研究[J]. 医学信息, 2022, 35(11): 97-101.
SONG N, TANG L M, DU H Y, et al. Distribution of TCM syndromes of diabetic gastroparesis based on cluster analysis[J]. Journal of Medical Information, 2022, 35(11): 97-101.
[14] 刘师师. 基于词典优化算法对舌象特征提取的研究[D]. 长沙: 长沙理工大学, 2016.
LIU S S. The research of tongue feature extraction based on optimization dictionary algorithm[D]. Changsha: Changsha University of Science & Technology, 2016.
[15] ZHOU Z W, RAHMAN SIDDIQUEE M M, TAJBAKHSH N, et al. UNet++: a nested U-Net architecture for medical image segmentation[C]//Proceedings of the 4th International Workshop on Deep Learning in Medical Image Analysis and the 8th International Workshop on Multimodal Learning for Clinical Decision Support. Cham: Springer, 2018: 3-11.
[16] CHEN L C, ZHU Y K, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of the 15th European Conference on Computer Vision. Cham: Springer, 2018: 833-851.
[17] ZHANG X M, ZHU X B, ZHANG X Y, et al. SegGAN: semantic segmentation with generative adversarial network[C]//Proceedings of the 4th IEEE International Conference on Multimedia Big Data. Piscataway: IEEE, 2018: 1-5.
[18] 肖庆新, 张菁, 张辉, 等. 基于轻型卷积神经网络的舌苔颜色分类方法[J]. 测控技术, 2019, 38(3): 26-31.
XIAO Q X, ZHANG J, ZHANG H, et al. Tongue coating color classification based on shallow convolutional neural network[J]. Measurement & Control Technology, 2019, 38(3): 26-31.
[19] SONG C, WANG B, XU J T. Classifying tongue images using deep transfer learning[C]//Proceedings of the 5th International Conference on Computational Intelligence and Applications. Piscataway: IEEE, 2020: 103-107.
[20] ZHANG D, ZHANG J H, WANG Z, et al. Tongue colour and coating prediction in traditional Chinese medicine based on visible hyperspectral imaging[J]. IET Image Processing, 2019, 13(12): 2265-2270.
[21] FAN Y, TANG X Y, WU X L, et al. A robust multi-environment tongue image segmentation method for computer-aided tongue diagnosis[C]//Proceedings of the Medical Imaging 2024: Computer-Aided Diagnosis, 2024: 119.
[22] WENG H, LI L, LEI H W, et al. A weakly supervised tooth-mark and crack detection method in tongue image[J]. Concurrency and Computation: Practice and Experience, 2021, 33(16): e6262.
[23] LI J, YUAN P, HU X J, et al. A tongue features fusion approach to predicting prediabetes and diabetes with machine learning[J]. Journal of Biomedical Informatics, 2021, 115: 103693.
[24] LIU Q, LI Y, YANG P, et al. A survey of artificial intelligence in tongue image for disease diagnosis and syndrome differentiation[J]. Digital Health, 2023, 9: 20552076231191044.
[25] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[C]//Advances in Neural Information Processing Systems 27, 2014: 2672-2680.
[26] DE SOUZA V L T, MARQUES B A D, BATAGELO H C, et al. A review on generative adversarial networks for image generation[J]. Computers & Graphics, 2023, 114: 13-25.
[27] PALADUGU P S, ONG J, NELSON N, et al. Generative adversarial networks in medicine: important considerations for this emerging innovation in artificial intelligence[J]. Annals of Biomedical Engineering, 2023, 51(10): 2130-2142.
[28] HAQUE A. EC-GAN: low-sample classification using semi-supervised algorithms and GANs[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(18): 15797-15798.
[29] TRAN N T, TRAN V H, NGUYEN N B, et al. On data augmentation for GAN training[J]. IEEE Transactions on Image Processing, 2021, 30: 1882-1897.
[30] CHAKRABORTY T, REDDY K S U, NAIK S M, et al. Ten years of generative adversarial nets (GANs): a survey of the state-of-the-art[J]. Machine Learning: Science and Technology, 2024, 5(1): 011001.
[31] ZHOU T, LI Q, LU H L, et al. GAN review: models and medical image fusion applications[J]. Information Fusion, 2023, 91: 134-148.
[32] 谢芳芳, 谢超群, 马健文, 等. 不同疲劳程度患者中医证候的舌象特征研究[J]. 中国中医药信息杂志, 2024, 31(11): 158-165.
XIE F F, XIE C Q, MA J W, et al. Study on tongue image characteristics of TCM symptoms in patients with different fatigue degree[J]. Chinese Journal of Information on Traditional Chinese Medicine, 2024, 31(11): 158-165.
[33] 徐安迎, 杨涛, 王天舒, 等. 中医望诊形态特征提取研究思路与方法[J]. 南京中医药大学学报, 2024, 40(12): 1315-1322.
XU A Y, YANG T, WANG T S, et al. Research ideas and methods of morphological feature extraction of traditional Chinese medicine inspection[J]. Journal of Nanjing University of Traditional Chinese Medicine, 2024, 40(12): 1315-1322.
[34] 惠世冀. 基于卷积神经网络和生成对抗网络舌象颜色校正[D]. 广州: 广州中医药大学, 2023.
HUI S J. Color correction with convolutional neural network and generative adversarial network for improving the performance of congested tongue recognition[D]. Guangzhou: Guangzhou University of Chinese Medicine, 2023.
[35] DAI S X, GUO X J, LIU S, et al. Application of intelligent tongue image analysis in conjunction with microbiomes in the diagnosis of MAFLD[J]. Heliyon, 2024, 10(7): e29269.
[36] YI Z H, LAI X S, SUN A N, et al. Tongue feature recognition to monitor rehabilitation: deep neural network with visual attention mechanism[J]. Frontiers in Bioengineering and Biotechnology, 2024, 12: 1392513.
[37] CHEN R F, HUANG W B, HUANG B H, et al. Reusing discriminators for encoding: towards unsupervised image-to-image translation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 8165-8174.
[38] ZHOU B L, KHOSLA A, LAPEDRIZA A, et al. Learning deep features for discriminative localization[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 2921-2929.
[39] KIM J, KIM M, KANG H, et al. U-GAT-IT: unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation[EB/OL]. [2024-06-25]. https://arxiv.org/abs/1907.10830.
[40] MIYATO T, KATAOKA T, KOYAMA M, et al. Spectral normalization for generative adversarial networks[EB/OL]. [2024-06-26]. https://arxiv.org/abs/1802.05957.
[41] YOSHIDA Y, MIYATO T. Spectral norm regularization for improving the generalizability of deep learning[EB/OL]. [2024-06-26]. https://arxiv.org/abs/1705.10941.
[42] HUANG X, BELONGIE S. Arbitrary style transfer in real-time with adaptive instance normalization[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2017: 1510-1519.
[43] MAO X D, LI Q, XIE H R, et al. Least squares generative adversarial networks[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2017: 2813-2821.
[44] SAGE A, BADURA P. Detection and segmentation of mouth region in stereo stream using YOLOv6 and DeepLab v3+ models for computer-aided speech diagnosis in children[J]. Applied Sciences, 2024, 14(16): 7146.
[45] ZHAO H S, SHI J P, QI X J, et al. Pyramid scene parsing network[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 6230-6239.
[46] ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 5967-5976.
[47] CAO S, WU Q F, MA L J. TongueSAM: an universal tongue segmentation model based on SAM with zero-shot[C]//Proceedings of the 2023 IEEE International Conference on Bioinformatics and Biomedicine. Piscataway: IEEE, 2023: 4520-4526.
[48] 赵亮, 刘晨, 王春艳. 位置信息增强的TransUnet医学图像分割方法[J]. 计算机科学与探索, 2025, 19(4): 976-988.
ZHAO L, LIU C, WANG C Y. Positional enhancement Trans-Unet for medical image segmentation[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(4): 976-988.