Research on Question Answering System on Joint of Knowledge Graph and Large Language Models

doi:10.3778/j.issn.1673-9418.2308070

Abstract

Abstract: The large language model (LLM), including ChatGPT, has shown outstanding performance in understanding and responding to human instructions, and has a profound impact on natural language question answering (Q&A). However, due to the lack of training in the vertical field, the performance of LLM in the vertical field is not ideal. In addition, due to its high hardware requirements, training and deploying LLM remains difficult. In order to address these challenges, this paper takes the application of traditional Chinese medicine formulas as an example, collects the domain related data and preprocesses the data. Based on LLM and knowledge graph, a vertical domain Q&A system is designed. The system has the following capabilities: (1) Information filtering. Filter out vertical domain related questions and input them into LLM to answer. (2) Professional Q&A. Generate answers with more professional knowledge based on LLM and self-built knowledge base. Compared with the fine-tuning method of introducing professional data, using this technology can deploy large vertical domain models without the need for retraining. (3) Extract conversion. By strengthening the information extraction ability of LLM and utilizing its generated natural language responses, structured knowledge is extracted and matched with a professional knowledge graph for professional verification. At the same time, structured knowledge can be transformed into readable natural language, achieving a deep integration of large models and knowledge graphs. Finally, the effect of the system is demonstrated and the performance of the system is verified from both subjective and objective perspectives through two experiments of subjective evaluation of experts and objective evaluation of multiple choice questions.

Key words: large language model (LLM), knowledge graph, Q&A system, vertical field, traditional Chinese medicine

摘要： 大语言模型（large language model，LLM），包括ChatGPT，在理解和响应人类指令方面表现突出，对自然语言问答影响深远。然而，由于缺少针对垂直领域的训练，LLM在垂直领域的表现并不理想。此外，由于对硬件的高要求，训练和部署LLM仍然具有一定困难。为了应对这些挑战，以中医药方剂领域的应用为例，收集领域相关数据并对数据进行预处理，基于LLM和知识图谱设计了一套垂直领域的问答系统。该系统具备以下能力：（1）信息过滤，过滤出垂直领域相关的问题，并输入LLM进行回答；（2）专业问答，基于LLM和自建知识库来生成更具备专业知识的回答，相比专业数据的微调方法，该技术无需重新训练即可部署垂直领域大模型；（3）抽取转化，通过强化LLM的信息抽取能力，利用生成的自然语言回答，从中抽取出结构化知识，并和专业知识图谱匹配以进行专业验证，同时可以将结构化知识转化成易读的自然语言，实现了大模型与知识图谱的深度结合。最后展示了该系统的效果，并通过专家主观评估与选择题客观评估两个实验，从主客观两个角度验证了系统的性能。

关键词: 大语言模型（LLM）, 知识图谱, 问答系统, 垂直领域, 中医药方剂

ZHANG Heyi, WANG Xin, HAN Lifan, LI Zhao, CHEN Zirui, CHEN Zhe. Research on Question Answering System on Joint of Knowledge Graph and Large Language Models[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(10): 2377-2388.

张鹤译, 王鑫, 韩立帆, 李钊, 陈子睿, 陈哲. 大语言模型融合知识图谱的问答系统研究[J]. 计算机科学与探索, 2023, 17(10): 2377-2388.

References

[1] 陈子睿, 王鑫, 王林, 等. 开放领域知识图谱问答研究综述[J]. 计算机科学与探索, 2021, 15(10): 1843-1869.
CHEN Z R, WANG X, WANG L, et al. Survey of open-do-main knowledge graph question answering[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(10): 1843-1869.
[2] 萨日娜, 李艳玲, 林民. 知识图谱推理问答研究综述[J]. 计算机科学与探索, 2022, 16(8): 1727-1741.
SARINA, LI Y L, LIN M. Survey of question answering based on knowledge graph reasoning[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(8): 1727-1741.
[3] GUU K, LEE K, TUNG Z, et al. Retrieval augmented lan-guage model pre-training[C]//Proceedings of the 37th Inter-national Conference on Machine Learning, Vienna, Jul 12-18, 2020. New York: PMLR, 2020: 3929-3938.
[4] CHOWDHERY A, NARANG S, DEVLIN J, et al. PaLM: scaling language modeling with pathways[J]. arXiv:2204.02311, 2022.
[5] WEI J, TAY Y, BOMMASANI R, et al. Emergent abilities of large language models[J]. arXiv:2206.07682, 2022.
[6] OUYANG L, WU J, JIANG X, et al. Training language mo-dels to follow instructions with human feedback[C]//Ad-vances in Neural Information Processing Systems 35, 2022: 27730-27744.
[7] OPENAI. GPT-4 technical report[R]. arXiv:2303.08774, 2023.
[8] WANG Y, KORDI Y, MISHRA S, et al. Self-Instruct: alig-ning language model with self generated instructions[J]. arXiv:2212.10560, 2022.
[9] MAYNEZ J, NARAYAN S, BOHNET B, et al. On faithful-ness and factuality in abstractive summarization[J]. arXiv:2005.00661, 2020.
[10] TONEVA M, SORDONI A, COMBES R T, et al. An empi-rical study of example forgetting during deep neural net-work learning[J]. arXiv:1812.05159, 2018.
[11] DU Z, QIAN Y, LIU X, et al. GLM: general language mo-del pretraining with autoregressive blank infilling[C]//Pro-ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dub-lin, May 22-27, 2022. Stroudsburg: ACL, 2022: 320-335.
[12] LIU X, ZHENG Y, DU Z, et al. GPT understands, too[J]. arXiv:2103.10385, 2021.
[13] LIU X, JI K, FU Y, et al. P-tuning v2: prompt tuning can be comparable to fine-tuning universally across scales and tasks[J]. arXiv:2110.07602, 2021.
[14] WU C, ZHANG X, ZHANG Y, et al. PMC-LLaMA: further finetuning LLaMA on medical papers[J]. arXiv:2304.14454, 2023.
[15] SINGHAL K, AZIZI S, TU T, et al. Large language models encode clinical knowledge[J]. arXiv:2212.13138, 2022.
[16] YUNXIANG L, ZIHAN L, KAI Z, et al. ChatDoctor: a me-dical chat model fine-tuned on LLaMA model using medical domain knowledge[J]. arXiv:2303.14070, 2023.
[17] WANG H, LIU C, XI N, et al. HuaTuo: tuning LLaMA model with Chinese medical knowledge[J]. arXiv:2304.06975, 2023.
[18] XIONG H, WANG S, ZHU Y, et al. DoctorGLM: fine-tuning your Chinese doctor is not a herculean task[J]. arXiv:2304.01097, 2023.
[19] ZENG G, YANG W, JU Z, et al. MedDialog: large-scale me-dical dialogue datasets[C]//Proceedings of the 2020 Con-ference on Empirical Methods in Natural Language Pro-cessing, Nov 16-20, 2020. Stroudsburg: ACL, 2020: 9241-9250.
[20] ZHANG N, JIA Q, YIN K, et al. Conceptualized represen-tation learning for Chinese biomedical text mining[J]. arXiv:2008.10813, 2020.
[21] BASALDELLA M, LIU F, SHAREGHI E, et al. COMETA: a corpus for medical entity linking in the social media[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Nov 16-20, 2020. Stroud-sburg: ACL, 2020: 3122-3137.
[22] CUI Y, LIU T, CHE W, et al. A span-extraction dataset for Chinese machine reading comprehension[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Con-ference on Natural Language Processing, Hong Kong, China, Nov 3-7, 2019. Stroudsburg: ACL, 2019: 5883-5889.
[23] XIONG C, SU M. IARNN-based semantic-containing double-level embedding Bi-LSTM for question-and-answer mat-ching[J]. Computational Intelligence and Neuroscience, 2019. DOI:10.1155/2019/6074840.
[24] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understan-ding[J]. arXiv:1810.04805, 2018.