关键词: 生成式语言模型, 基础知识问答, 大语言模型, Transformer, 冻结模型权重

Abstract: In the field of higher education, foundational knowledge question-answering (QA) systems play a crucial role in enhancing students’ academic performance and facilitating equitable distribution of educational resources. In recent years, question-answering techniques based on machine reading comprehension and text similarity matching have been developed atop pre-trained language models. However, when addressing complex natural language problems, these techniques still face challenges in answer quality and accuracy due to limitations like insufficient training data and restricted model generalization capabilities. This research aims to address the dual objective of reducing resource consumption while simultaneously enhancing the performance and accuracy of basic knowledge question-  answering systems in university settings. To achieve this goal, this paper proposes an efficient fine-tuning approach for large language model with low-rank encoding in the domain of fundamental knowledge. This method uses low-rank encoding to minimize computational costs during the training and prediction stages of large language models. It enhances research and analysis in our university’s basic knowledge question-answering field by employing the generative capabilities of these models, improving the quality, accuracy, and response speed of everyday queries. By freezing the weights of the pre-trained model and integrating university-specific knowledge into the Transformer architecture, along with a question-answering optimization module, this approach preserves generative abilities and achieves superior performance and accuracy in the university knowledge domain while reducing trainable parameters for downstream tasks.

Key words: generative language model, fundamental knowledge question-answering, large language model, Transformer, freezing model weights