融合知识推理与相似度检索的民众诉求大模型构建与应用

doi:10.3778/j.issn.1673-9418.2406057

摘要/Abstract

摘要： 高效回复民众诉求是实现智能化管理、提升民众满意度的必要措施，将智能问答应用于民众诉求能有效节约人力和时间资源。然而，智能问答中基于规则和检索的模型依赖预设知识，当诉求超出预设知识范围时无法提供有效回复，在处理多轮对话时也无法保持对话连贯性。现有的大语言模型可以和用户流畅对话，但通用大语言模型缺乏诉求领域知识。由于训练数据中问答对的信息没有覆盖回答用户问题所需要的知识，导致通用大语言模型生成错误回复或答非所问，产生幻觉。针对上述问题，构建了面向民众诉求领域的智能问答大语言模型（PC-LLM）。设计基于BERT-BiLSTM-CRF的实体关系抽取模型获得诉求工单中实体及其关系，进而构建诉求知识图谱，使用BERT模型对诉求工单向量化并构建诉求工单向量索引库；回复生成阶段，抽取用户诉求的实体和关系，在诉求知识图谱中通过实体链接进行知识推理，获取潜在关系提示，同时在诉求工单向量索引库内对诉求进行快速检索，获取相似诉求并构建相似诉求提示；将潜在关系提示、相似诉求提示与用户诉求融合形成综合提示，引导大语言模型生成准确的回复。实验分析显示，该大语言模型在诉求数据集中的表现明显优于ChatGPT4o、文心一言、通义千问等大语言模型。

关键词: 大语言模型, 知识推理, 相似度检索, 民众诉求, 知识图谱

Abstract: Efficiently responding to public complaints is a necessary measure to realize intelligent management and enhance public satisfaction, and the use of intelligent question answering for public complaints can save time and human resources. However, rule-based and retrieval-based models in intelligent question answering rely on preset knowledge. Therefore, they cannot provide effective responses when complaints are out of the scope of knowledge, nor can they maintain the coherence of conversations when dealing with multiple rounds of dialogues. Existing large language models can communicate smoothly with users, but general-purpose large language models lack domain knowledge. Due to the fact that the correct answers in the training data will contain information not covered by the questions, the general large language model generates wrong responses or answers that are not the questions asked, resulting in hallucination. To address these issues, a large language model (PC-LLM) for intelligent question-and-answer in the domain of public complaints has been constructed. Firstly, an entity relationship extraction model based on BERT-BiLSTM-CRF is designed to extract entities and relationships in the complaint work order in order to construct the complaint knowledge graph. The BERT model is used to vectorize the complaint work order and construct the vector index library of the complaint work order. In the stage of reply generation, this paper extracts the entities and relationships of users’ complaints, conducts knowledge reasoning through entity links in the knowledge graph of complaints, obtains potential relationship tips, and uses the knowledge graph of complaints to perform knowledge reasoning to obtain potential relationship hints. Meanwhile, this paper performs quick search of complaints within the vector index library of complaint work orders, and obtains similar complaints. Finally, a more accurate response can be generated by integrating potential relationship prompts, similar complaint prompts and complaint into a large language model. Experimental analysis shows that the performance of this large language model on the complaints dataset is significantly better than that of ChatGPT4o, ERNIE Bot, Tongyi Qianwen, and other large language models.

Key words: large language model, knowledge reasoning, similarity retrieval, public complaints, knowledge graph

刘昕, 高会泉, 邵长恒, 陈子良, 卢文娟, 杨会如. 融合知识推理与相似度检索的民众诉求大模型构建与应用[J]. 计算机科学与探索, 2024, 18(11): 2940-2953.

LIU Xin, GAO Huiquan, SHAO Changheng, CHEN Ziliang, LU Wenjuan, YANG Huiru. Construction and Application of Large Language Model for Public Complaints with Knowledge Reasoning and Similarity Retrieval[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(11): 2940-2953.

参考文献

[1] 曹海军, 沈博仟. “枫桥经验”数智创新：技术平台与群众路线的融合机制研究——以Y市“政务热线+网格”模式为例[J]. 河南社会科学, 2024, 32(6): 117-124.
CAO H J, SHEN B Q. “Fengqiao Experience”digital in-telligence innovation: research on the integration mecha-nism of technology platform and mass line—taking the“Government Hotline + Grid”model of Y city as an example[J]. Henan Social Sciences, 2024, 32(6): 117-124.
[2] 娄文龙, 王晓萌, 王晓慧. 数字技术如何引发基层政府的行为策略?——基于B市C区12345政务服务便民热线的案例分析[J]. 天津行政学院学报, 2024, 26(3): 27-36.
LOU W L, WANG X M, WANG X H. How can digital technologies trigger the behavioral strategies of the grass-roots government?—Case analysis based on 12345 government service convenience hotline in C district of B city[J]. Journal of Tianjin Administration Institute, 2024, 26(3): 27-36.
[3] BAWDEN D, ROBINSON L. The dark side of information: over-load, anxiety and other paradoxes and pathologies[J]. Journal of Information Science, 2009, 35(2): 180-191.
[4] ABUJABAL A, YAHYA M, RIEDEWALD M, et al. Automated template generation for question answering over knowledge graphs[C]//Proceedings of the 26th International Conference on World Wide Web, Perth, Apr 3-7, 2017. New York: ACM, 2017: 1191-1200.
[5] CHEN B, SUN L, HAN X. Sequence-to-action: end-to-end semantic graph generation for semantic parsing[EB/OL]. [2024-04-15]. https://arxiv.org/abs/1809.00773.
[6] BORDES A, CHOPRA S, WESTON J. Question answering with subgraph embeddings[EB/OL]. [2024-04-15]. https://arxiv.org/abs/1406.3676.
[7] YANG H, LIU X Y, WANG C D. FinGPT: open-source financial large language models[EB/OL]. [2024-04-15]. https://arxiv.org/abs/2306.06031.
[8] 张金营, 王天堃, 么长英, 等. 基于大语言模型的电力知识库智能问答系统构建与评价[J/OL]. 计算机科学 [2024-05-31]. http://kns.cnki.net/kcms/detail/50.1075.TP.20240528. 0931.002.html.
ZHANG J Y, WANG T K, YAO C Y, et al. Construction and evaluation of intelligent question answering system for electric power knowledge base based on large language model[J/OL]. Computer Science [2024-05-31]. http://kns. cnki.net/kcms/detail/50.1075.TP.20240528.0931.002.html.
[9] 张鹤译, 王鑫, 韩立帆, 等. 大语言模型融合知识图谱的问答系统研究[J]. 计算机科学与探索, 2023, 17(10): 2377-2388.
ZHANG H Y, WANG X, HAN L F, et al. Research on question answering system on joint of knowledge graph and large language models[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(10): 2377-2388.
[10] 王珊珊. 智道奇点政务服务人工智能大模型亮相[N]. 中国建设报, 2024-05-30(2).
WANG S S. Smart Dodge Point government service artificial intelligence large model unveiled[N]. China Construction News, 2024-05-30(2).
[11] 舒抒. 窗口人员业务精湛，为何还要大模型[N]. 解放日报, 2024-06-26(2).
SHU S. Window staff business is superb, why do you want a large model[N]. Liberation Daily, 2024-06-26(2).
[12] LEWIS P, PEREZ E, PIKTUS A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks[C]//Advances in Neural Information Processing Systems 33, Dec 6-12, 2020: 9459-9474.
[13] CHEN H X, LI Y Q, SHI S Y, et al. Graph collaborative reasoning[C]//Proceedings of the 15th ACM International Conference on Web Search and Data Mining. New York: ACM, 2022: 75-84.
[14] 黄家豪, 王冰, 唐漾. 化工安全知识图谱的本体设计与基于规则推理的知识补全[J/OL]. 控制工程 [2024-08-03]. https://doi.org/10.14107/j.cnki.kzgc.20240077.
HUANG J H, WANG B, TANG Y. Ontology design of chemical safety knowledge graph and a rule-based inference approach for knowledge completion[J/OL]. Control Engineering of China [2024-08-03]. https://doi.org/10.14107/ j.cnki.kzgc.20240077.
[15] BORDES A, USUNIER N, GARCIA-DURAN A, et al. Translating embeddings for modeling multi-relational data[C]//Advances in Neural Information Processing Systems 26, Lake?Tahoe, Dec?5-8, 2013: 2787-2795.
[16] LI Z, LIU X, WANG X, et al. TransO: a knowledge-driven representation learning method with ontology information constraints[J]. World Wide Web, 2023, 26(1): 297-319.
[17] 彭鐄, 曾维新, 周杰, 等. 基于图神经网络的实体对齐表示学习方法比较研究[J]. 计算机科学与探索, 2023, 17(10): 2343-2357.
PENG H, ZENG W X, ZHOU J, et al. Contrast research of representation learning in entity alignment based on graph neural network[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(10): 2343-2357.
[18] VASHISHTH S, SANYAL S, NITIN V, et al. Composition-based multi-relational graph convolutional networks[EB/OL]. [2024-04-15]. https://arxiv.org/abs/1911.03082.
[19] ZHANG Y Q, YAO Q M. Knowledge graph reasoning with relational digraph[C]//Proceedings of the ACM Web Conference 2022. New York: ACM, 2022: 912-924.
[20] SAXENA A, TRIPATHI A, TALUKDAR P. Improving multi-hop question answering over knowledge graphs using know-ledge base embeddings[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2020: 4498-4507.
[21] 田侃, 曹新汶, 张浩然, 等. 结合图卷积模型和共享编码的知识图谱问答方法[J/OL]. 计算机工程与应用 [2024-08-03]. http://kns.cnki.net/kcms/detail/11.2127.TP.20240625.1053. 002.html.
TIAN K, CAO X W, ZHANG H R, et al. Knowledge graph question answering with shared encoding and graph convolution networks[J/OL]. Computer Engineering and Applications [2024-08-03]. http://kns.cnki.net/kcms/detail/11.2127. TP.20240625.1053.002.html.
[22] KASNECI E, SE?LER K, KüCHEMANN S, et al. ChatGPT for good? On opportunities and challenges of large language models for education[J]. Learning and Individual Differences, 2023, 103: 102274.
[23] 朱娜娜, 王航, 张家乐, 等. 基于预训练语言模型的政策识别研究[J]. 中文信息学报, 2022, 36(2): 104-110.
ZHU N N, WANG H, ZHANG J L, et al. Policy identification based on pretrained language model[J]. Journal of Chinese Information Processing, 2022, 36(2): 104-110.
[24] 华斌, 康月, 范林昊. 政策文本的知识建模与关联问答研究[J]. 数据分析与知识发现, 2022, 6(11): 79-92.
HUA B, KANG Y, FAN L H. Knowledge modeling and association Q&A for policy texts[J]. Data Analysis and Knowledge Discovery, 2022, 6(11): 79-92.
[25] 王昀, 胡珉, 塔娜, 等. 大语言模型及其在政务领域的应用[J]. 清华大学学报(自然科学版), 2024, 64(4): 649-658.
WANG Y, HU M, TA N, et al. Large language models and their application in government affairs[J]. Journal of Tsinghua University (Science and Technology), 2024, 64(4): 649-658.
[26] SHANAHAN M. Talking about large language models[J]. Communications of the ACM, 2024, 67(2): 68-79.
[27] BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[C]//Advances in Neural Information Processing Systems 33, Dec 6-12, 2020: 1877-1901.
[28] CHOWDHERY A, NARANG S, DEVLIN J, et al. Palm: scaling language modeling with pathways[J]. Journal of Machine Learning Research, 2023, 24.
[29] 杨波, 孙晓虎, 党佳怡, 等. 面向医疗问答系统的大语言模型命名实体识别方法[J]. 计算机科学与探索, 2023, 17(10): 2389-2402.
YANG B, SUN X H, DANG J Y, et al. Named entity recognition method of large language model for medical question answering system[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(10): 2389-2402.
[30] JI Z, LEE N, FRIESKE R, et al. Survey of hallucination in natural language generation[J]. ACM Computing Surveys, 2023, 55(12): 1-38.
[31] 李源, 马新宇, 杨国利, 等. 面向知识图谱和大语言模型的因果关系推断综述[J]. 计算机科学与探索, 2023, 17(10): 2358-2376.
LI Y, MA X Y, YANG G L, et al. Survey of causal inference for knowledge graphs and large language models[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(10): 2358-2376.
[32] RAJANI N F, MCCANN B, XIONG C, et al. Explain yourself! Leveraging language models for commonsense reasoning[EB/OL]. [2024-04-15]. https://arxiv.org/abs/1906.02361.
[33] WEI J, WANG X, SCHUURMANS D, et al. Chain-of-thought prompting elicits reasoning in large language models[C]//Advances in Neural Information Processing Systems 35, New Orleans, Nov 28-Dec 9, 2022: 24824-24837.
[34] LI S, GAO Y, JIANG H, et al. Graph reasoning for question answering with triplet retrieval[EB/OL]. [2024-04-15]. https://arxiv.org/abs/2305.18742.
[35] 文森, 钱力, 胡懋地, 等. 基于大语言模型的问答技术研究进展综述[J/OL]. 数据分析与知识发现 [2024-06-01]. http://kns.cnki.net/kcms/detail/10.1478.G2.20231110.1612.002.html.
WEN S, QIAN L, HU M D, et al. A survey of research progress of question answering technology based on large language model[J/OL]. Data Analysis and Knowledge Discovery [2024-06-01]. http://kns.cnki.net/kcms/detail/10.1478. G2.20231110.1612.002.html.
[36] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2024-04-15]. https://arxiv.org/abs/1810.04805.
[37] HU Z, DONG Y, WANG K, et al. Heterogeneous graph transformer[C]//Proceedings of the Web Conference 2020, Taipei, China, Apr 20-24, 2020. New York: ACM, 2020: 2704-2710.
[38] HU E J, SHEN Y, WALLIS P, et al. LoRA: low-rank adaptation of large language models[EB/OL]. [2024-04-15]. https://arxiv.org/abs/2106.09685.