开放环境下无监督跨模态概念自动提取

doi:10.3778/j.issn.1673-9418.2505051

计算机科学与探索 ›› 2026, Vol. 20 ›› Issue (1): 154-168.DOI: 10.3778/j.issn.1673-9418.2505051

开放环境下无监督跨模态概念自动提取

海峻嘉1,2,3，景丽萍1,2,3+，刘华锋2,3，于剑2,3

1. 先进轨道交通自主运行全国重点实验室，北京 100044
2. 交通数据挖掘与具身智能北京市重点实验室，北京 100044
3. 北京交通大学计算机科学与技术学院，北京 100044
+ 通信作者 E-mail: lpjing@bjtu.edu.cn

收稿日期:2025-05-21 修回日期:2025-09-02 在线发布日期:2026-01-01 出版日期:2025-12-30
基金资助:
国家自然科学基金（62436001，62406019，62176020）；北京市自然科学基金（4244096）；北京交通大学人才基金（2024XKRC075）；教育部创新团队联合基金（8091B042235）；中央高校基本科研业务费专项资金（2019JBZ110）；北京交通大学轨道交通控制与安全国家重点实验室项目（RCS2023K006）。

Automatic Unsupervised Cross-Modal Concept Extraction in Open Environments

HAI Junjia1,2,3, JING Liping1,2,3+, LIU Huafeng2,3, YU Jian2,3

1. State Key Laboratory of Advanced Rail Autonomous Operation, Beijing 100044, China
2. Beijing Key Laboratory of Transportation Data Mining and Embodied Intelligence, Beijing 100044, China
3. School of Computer Science ＆ Technology, Beijing Jiaotong University, Beijing 100044, China
+ Corresponding author E-mail: lpjing@bjtu.edu.cn

Received:2025-05-21 Revised:2025-09-02 Online:2026-01-01 Published:2025-12-30
Supported by:
This work was supported by the National Natural Science Foundation of China (62436001, 62406019, 62176020), the Natural Science Foundation of Beijing (4244096), the Talent Foundation of Beijing Jiaotong University (2024XKRC075), the Joint Foundation of the Ministry of Education for Innovation Team (8091B042235), the Fundamental Research Funds for the Central Universities of China (2019JBZ110), and the Project of State Key Laboratory of Rail Traffic Control and Safety of Beijing Jiaotong University(RCS2023K006).

摘要/Abstract

摘要： 随着机器学习模型的复杂性不断增加，对其决策过程的可解释性需求也日益增长。概念学习作为一种能够提升模型透明度和可理解性的手段，在机器学习领域变得越来越重要，通过概念来帮助解释大语言模型等黑盒模型的推理过程也随之发展，如何准确、自动地提取概念是在这一解释过程中最为关键的一环。针对现有的概念提取方法中存在的依赖人工标注、粒度不一致、扩展性差等问题，设计了一套跨模态视觉概念自动提取框架。基于多模态大模型的问询方式，实现了图像中目标对象的自动提取，并通过CLIP模型和目标检测模型实现了区域对应机制，提高了提取的准确性。采用基于文本分割的大语言模型来提取目标对象的属性和关系，确保了概念与数据的一致性。引入ConceptNet概念网络扩展了提取到的概念的语义信息，增强了框架的灵活性和适用性。通过实现这些技术，展示了该框架在三种不同场景下的实际应用效果，证明了其在提升概念提取效率和准确性以及可扩展性方面的潜力。

关键词: 大语言模型, 可解释性, 概念提取, 自动化, 可扩展性

Abstract: As machine learning models become increasingly complex, the demand for interpretability in their decision-making processes continues to grow. Concept learning has become increasingly important in the field of machine learning as a means to enhance model transparency and comprehensibility. Consequently, the use of concepts to explain the reasoning processes of black-box models, such as large language models, has also developed. The ability to accurately and automatically extract concepts is a crucial step in this interpretability process. To address the issues in existing concept extraction methods, such as reliance on manual annotation, inconsistent granularity, and poor scalability, this paper proposes an automatic cross-modal visual concept extraction framework. This framework employs a multimodal large model-based inquiry method to achieve automatic extraction of target objects in images. The integration of the CLIP model and object detection models establishes a region-matching mechanism to enhance extraction accuracy. Additionally, a large language model based on text segmentation is utilized to extract attributes and relationships of target objects, ensuring consistency between concepts and data. Furthermore, ConceptNet is incorporated to enrich the semantic information of the extracted concepts, improving the flexibility and applicability of framework. Through the implementation of these technologies, this paper demonstrates the practical application of this framework in three different scenarios, showcasing its potential in improving the efficiency, accuracy, and scalability of concept extraction.

Key words: large language model, interpretability, concept extraction, automation, scalability

海峻嘉, 景丽萍, 刘华锋, 于剑. 开放环境下无监督跨模态概念自动提取[J]. 计算机科学与探索, 2026, 20(1): 154-168.

HAI Junjia, JING Liping, LIU Huafeng, YU Jian. Automatic Unsupervised Cross-Modal Concept Extraction in Open Environments[J]. Journal of Frontiers of Computer Science and Technology, 2026, 20(1): 154-168.

参考文献

[1] MURPHY K P. Machine learning: a probabilistic perspective [M]. Cambridge: MIT Press, 2012.
[2] BISHOP C M. Pattern recognition and machine learning[M]. New York: Springer, 2006.
[3] 周昊天, 傅小兰. 认知科学: 新千年的前沿领域[J]. 心理科学进展, 2005, 13(4): 388-397.
ZHOU H T, FU X L. Cognitive science: the scientific frontier of new millennium[J]. Advances in Psychological Science, 2005, 13(4): 388-397.
[4] 戴汝为, 张雷鸣. 思维(认知)科学在中国的创新与发展[J]. 自动化学报, 2010, 36(2): 193-198.
DAI R W, ZHANG L M. The creation and development of noetic (cognitive) science in China[J]. Acta Automatica Sinica, 2010, 36(2): 193-198.
[5] ZAHRA S A, GEORGE G. Absorptive capacity: a review, reconceptualization, and extension[J]. The Academy of Management Review, 2002, 27(2): 185-203.
[6] BENGIO Y. Learning deep architectures for AI[J]. Foundations and Trends? in Machine Learning, 2009, 2(1): 1-127.
[7] RENDLE S. Factorization machines[C]//Proceedings of the 2010 IEEE International Conference on Data Mining. Piscataway: IEEE, 2010: 995-1000.
[8] 尚俊杰, 王钰茹, 何奕霖. 探索学习的奥秘: 我国近五年学习科学实证研究[J]. 华东师范大学学报(教育科学版), 2020, 38(9): 162-178.
SHANG J J, WANG Y R, HE Y L. Exploring the mysteries of learning: a review of empirical research on learning sciences in recent five years in China[J]. Journal of East China Normal University (Educational Sciences), 2020, 38(9): 162-178.
[9] 郭衎, 曹一鸣. 学习动机对学习效果影响的深度解析: 基于大规模学生调查的实证研究[J]. 教育科学研究, 2019(3): 62-67.
GUO K, CAO Y M. The influence of learning motivation on students’ academic achievement: an empirical study based on large-scale student survey[J]. Educational Science Research, 2019(3): 62-67.
[10] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: ACL, 2019: 4171-4186.
[11] ACHIAM J, ADLER S, AGARWAL S, et al. GPT-4 technical report[EB/OL]. [2025-03-01]. https://arxiv.org/abs/2303.08774.
[12] LAGE I, DOSHI-VELEZ F. Learning interpretable concept-based models with human feedback[EB/OL]. [2025-02-28]. https://arxiv.org/abs/2012.02898.
[13] DIKTER M, BLAU T, BASKIN C. Conceptual learning via embedding approximations for reinforcing interpretability and transparency[EB/OL]. [2025-02-28]. https://arxiv.org/abs/2406.08840.
[14] KOH P W, NGUYEN T, TANG Y S, et al. Concept bottleneck models[C]//Proceedings of the 37th International Conference on Machine Learning, 2020: 5338-5348.
[15] YUKSEKGONUL M, WANG M, ZOU J. Post-hoc concept bottleneck models[C]//Proceedings of the 11th International Conference on Learning Representations, 2023.
[16] WANG W X, YUE T T, ZHANG Y S, et al. Unveiling parts beyond objects: towards finer-granularity referring expression segmentation[C]//Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2024: 12998-13008.
[17] ZHAO Z J, GUO L T, HE X J, et al. MAMO: fine-grained vision-language representations learning with masked multimodal modeling[C]//Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2023: 1528-1538.
[18] SPEER R, CHIN J, HAVASI C. ConceptNet 5.5: an open multilingual graph of general knowledge[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2017, 31(1).
[19] OIKARINEN T, DAS S, NGUYEN L M, et al. Label-free concept bottleneck models[EB/OL]. [2025-02-28]. https://arxiv.org/abs/2304.06129.
[20] KIM E, JUNG D, PARK S, et al. Probabilistic concept bottleneck models[C]//Proceedings of the 40th International Conference on Machine Learning, 2023: 16521-16540.
[21] SHANG C M, ZHOU S J, ZHANG H Y, et al. Incremental residual concept bottleneck models[C]//Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2024: 11030-11040.
[22] XU X Y, QIN Y, MI L, et al. Energy-based concept bottleneck models: unifying prediction, concept intervention, and probabilistic interpretations[C]//Proceedings of the 12th International Conference on Learning Representations, 2024.
[23] VANDENHIRTZ M, LAGUNA S, MARCINKEVI?S R, et al. Stochastic concept bottleneck models[C]//Advances in Neural Information Processing Systems 37, 2024: 51787-51810.
[24] DELFOSSE Q, SZTWIERTNIA S, ROTHERMEL M, et al. Interpretable concept bottlenecks to align reinforcement learning agents[C]//Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024: 66826-66855.
[25] LAGUNA S, MARCINKEVI?S R, VANDENHIRTZ M, et al. Beyond concept bottleneck models: how to make black boxes intervenable?[C]//Advances in Neural Information Processing Systems 37, 2024: 85006-85044.
[26] IENCO D, MARCOS D, PANOUSIS K. Coarse-to-fine concept bottleneck models[C]//Advances in Neural Information Processing Systems 37, 2024: 105171-105199.
[27] HURST A, LERER A, GOUCHER P A, et al. GPT-4o system card[EB/OL]. [2025-03-03]. https://arxiv.org/abs/2410.21276.
[28] GUO D Y, YANG D J, ZHANG H W, et al. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning[EB/OL]. [2025-03-25]. https://arxiv.org/abs/2501.12948.
[29] SHANG C M, ZHANG H Y, WEN H, et al. Understanding multimodal deep neural networks: a concept selection view[EB/OL]. [2025-03-03]. https://arxiv.org/abs/2404.08964.
[30] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the 38th International Conference on Machine Learning, 2021.
[31] REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[C]//Advances in Neural Information Processing Systems 28, 2015: 91-99.
[32] YANG Y, PANAGOPOULOU A, ZHOU S H, et al. Language in a bottle: language model guided concept bottlenecks for interpretable image classification[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 19187-19197.
[33] SRIVASTAVA D, WENG T W, YAN G. VLG-CBM: training concept bottleneck models with vision-language guidance[C]//Advances in Neural Information Processing Systems 37, 2024: 79057-79094.
[34] TAN Z, CHENG L, WANG S, et al. Interpreting pretrained language models via concept bottlenecks[C]//Advances in Knowledge Discovery and Data Mining. Singapore: Springer, 2024: 56-74.
[35] LAI S N, HU L J, WANG J X, et al. Faithful vision-language interpretation via concept bottleneck models[C]//Proceedings of the 11th International Conference on Learning Representations, 2023.
[36] RAFFEL C, SHAZEER N, ROBERTS A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer[J]. Journal of Machine Learning Research, 2020, 21(1): 5485-5551.
[37] MITCHELL T M. Machine learning[M]. New York: McGraw-Hill, 1997.
[38] QUINLAN J R. Induction of decision trees[M]. Boston: Kluwer Academic Publishers, 1986: 81-106.
[39] HOGAN A, BLOMQVIST E, COCHEZ M, et al. Knowledge graphs[J]. ACM Computing Surveys, 2022, 54(4): 1-37.
[40] BOLLACKER K, EVANS C, PARITOSH P, et al. Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. New York: ACM, 2008: 1247-1250.
[41] MIKOLOV T, CHEN K, CORRADO G S, et al. Efficient estimation of word representations in vector space[C]//Proceedings of the 1st International Conference on Learning Representations, 2013.
[42] PENNINGTON J, SOCHER R, MANNING C. GloVe: global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2014: 1532-1543.
[43] KRISHNA R, ZHU Y K, GROTH O, et al. Visual genome: connecting language and vision using crowdsourced dense image annotations[J]. International Journal of Computer Vision, 2017, 123(1): 32-73.
[44] CORDTS M, OMRAN M, RAMOS S, et al. The cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 3213-3223.
[45] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of the 13th European Conference on Computer Vision. Cham: Springer, 2014: 740-755.
[46] KUZNETSOVA A, ROM H, ALLDRIN N, et al. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale[J]. International Journal of Computer Vision, 2020, 128(7): 1956-1981.
[47] EVERINGHAM M, VAN GOOL L, WILLIAMS C K I, et al. The pascal visual object classes (VOC) challenge[J]. International Journal of Computer Vision, 2010, 88(2): 303-338.
[48] GUO K Y, LI J H, ZHANG X. Notes on the improvement of concept-cognitive learning accuracy[J]. International Journal of Approximate Reasoning, 2023, 156: 77-96.
[49] LIANG T J, LIN Y D, LI J J, et al. Incremental cognitive learning approach based on concept reduction[J]. International Journal of Approximate Reasoning, 2025, 179: 109359.
[50] SIMPSON E H. Measurement of diversity[M]. London: Nature Publishing Group, 1949: 688.

开放环境下无监督跨模态概念自动提取

Automatic Unsupervised Cross-Modal Concept Extraction in Open Environments

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	刘颖, 刘爽, 陆羽, 孙楠, 陈嘉茂. 基于LLM增强图结构的供应链金融信用风险评估[J]. 计算机科学与探索, 2026, 20(2): 522-532.
[2]	周昌镐, 孙靖超. 基于幻觉消除大模型的误导性信息检测方法研究[J]. 计算机科学与探索, 2026, 20(2): 546-560.
[3]	郭晋阳, 贺昌义, 杨戈, 刘祥龙. 大语言模型压缩综述[J]. 计算机科学与探索, 2026, 20(1): 1-20.
[4]	杨跃翔, 郑怀城, 刘学文, 涂新雨. 智能家居信息安全知识图谱构建研究[J]. 计算机科学与探索, 2026, 20(1): 169-181.
[5]	昂格鲁玛, 王斯日古楞, 斯琴图. 知识图谱补全研究综述[J]. 计算机科学与探索, 2025, 19(9): 2302-2318.
[6]	王劲滔, 孟琪翔, 高志霖, 卜凡亮. 基于大语言模型指令微调的案件信息要素抽取方法研究[J]. 计算机科学与探索, 2025, 19(8): 2161-2173.
[7]	张程, 曹京旭, 吕劲昕, 严冬梅. 自然语言视角下基于凸包的知识蒸馏可解释性研究[J]. 计算机科学与探索, 2025, 19(8): 2174-2187.
[8]	田崇腾, 刘静, 王晓燕, 李明. 大语言模型GPT在医疗文本中的应用综述[J]. 计算机科学与探索, 2025, 19(8): 2043-2056.
[9]	夏江镧, 李艳玲, 葛凤培. 基于大语言模型的实体关系抽取综述[J]. 计算机科学与探索, 2025, 19(7): 1681-1698.
[10]	时振普, 吕潇, 董彦如, 刘静, 王晓燕. 医学领域多模态知识图谱融合技术发展现状研究[J]. 计算机科学与探索, 2025, 19(7): 1729-1746.
[11]	崔健, 汪永伟, 李飞扬, 李强, 苏北荣, 张小健. 结合知识蒸馏的中文文本摘要生成方法[J]. 计算机科学与探索, 2025, 19(7): 1899-1908.
[12]	张欣, 孙靖超. 基于大语言模型的虚假信息检测框架综述[J]. 计算机科学与探索, 2025, 19(6): 1414-1436.
[13]	许德龙, 林民, 王玉荣, 张树钧. 基于大语言模型的NLP数据增强方法综述[J]. 计算机科学与探索, 2025, 19(6): 1395-1413.
[14]	傅洁, 郭帆. 覆盖导向的Android程序多入口测试方法[J]. 计算机科学与探索, 2025, 19(6): 1666-1680.
[15]	何静, 沈阳, 谢润锋. 大语言模型幻觉现象的分类识别与优化研究[J]. 计算机科学与探索, 2025, 19(5): 1295-1301.