关键词: 大模型, 检索增强生成, 大模型稳定性, 问答系统

Abstract: The question-answering systems based on large language models (LLMs) was affected by the issues of semantic inconsistency from LLMs, resulting in an "unstable outputs" phenomenon, which impaired the safety, robustness and credibility of a question-answering system and severely degraded the user experience. To address the above issue, this paper proposed a method of retrieval-augmented LLMs’ stability output for private question-answering systems. This approach optimized the use of prompt words, allowing the LLMs to first output num_k synonymous queries of the user’s query, and then output the final answer. This design aimed to refer to the first generated num_k synonymous queries during the generation of the final answer, thereby making the output of the LLMs more stable. To tackle issues such as unstable generation of equivalent queries and uninterpretable output formats due to the weak instruction comprehension of open-source LLMs, this paper proposed using data distillation to automatically construct an open-domain retrieval-augmented instruction dataset through utilizing a closed-source LLM. Then an open-source LLM was fine-tuned on this instruction dataset. In addition, an evaluation dataset was developed under a private question-answering scenario to validate the effectiveness of the proposed method. Experimental results on the evaluation dataset demonstrated that the proposed method significantly surpassed the baseline method in terms of both consistency and performance metrics. Notably, compared to the baseline method, the proposed method showed substantial improvement on the consistency metrics, with ROUGE-1, ROUGE-2, ROUGE-L, and BLEU scores improved by 18.9, 30.1, 24.5, and 30.6 respectively; as for performance metrics, the accuracy rate increased by 17.4%.

Key words: large language models, retrieval-augmented generation, stability of large language models, question- answering systems