计算机科学与探索

• 学术研究 •    

融合预训练模型和注意力的实体关系抽取方法

李智杰,韩瑞瑞,李昌华,张颉,石昊琦   

  1. 西安建筑科技大学 信息与控制工程学院,西安 710055

Entity relation extraction method integrating pre-trained model and attention

LI Zhijie, HAN Ruirui, LI Changhua, ZHANG Jie, SHI Haoqi   

  1. School of Information and Control Engineering, Xi’an University of Architectural Scinence and Technology, Xi’an 710055, China

摘要: 实体关系抽取旨在从无结构的文档中检测出实体和实体对的关系,是构建领域知识图谱的重要步骤。针对现有抽取模型语义表达能力差、重叠三元组抽取准确率低的情况,研究了融合预训练模型和注意力的实体关系联合抽取问题,将实体关系抽取任务分解为两个标记模块。头实体标记模块采用预训练模型对句子进行编码,为了进一步学习句子的内在特征,利用双向长短时记忆网络(bi-directional long-short term memory,BiLSTM)和自注意力机制(self-attention mechanism)组成特征加强层。采用二进制分类器作为模型的解码器,标记出头实体在句子中的起止位置。为了加深两个标记模块之间的联系,在尾实体标记任务前设置特征融合层,将头实体特征与句子向量通过卷积神经网络(convolutional neural networks, CNN)和注意力机制进行特征融合,通过多个相同且独立的二进制分类器判定实体间关系并标记尾实体,构建出融合预训练模型和注意力的联合抽取模型JPEA。实验结果表明,该方法能显著提升抽取的效果,对比不同预训练模型下抽取任务的性能,进一步说明了模型的优越性。

关键词: 领域知识图谱, 预训练模型, 自注意力机制, 特征融合

Abstract: Entity relationship extraction aims to detect the relationship between entities and entity pairs from unstructured text. It is an important step in constructing domain knowledge map. In view of the poor semantic expression ability of the existing extraction models and the low accuracy of overlapping triples extraction, this paper studies the joint extraction of entity relationships by integrating pre-trained model and attention, and divides the entity relationship extraction task into two tag modules. The head entity tagging module uses a pre-trained model to encode sentences. In order to further learn the internal characteristics of sentences, Bi-directional Long-short Term Memory and Self-attention Mechanism are used to form a feature enhancement layer. The binary classifier is used as the decoder of the model to mark the start and end positions of the head entity in the sentence. In order to deepen the relationship between the two marking modules, a feature fusion layer is set up before the tail entity marking task. The head entity features and sentence vectors are fused through Convolutional Neural Networks and attention mechanism. The relationship between entities is determined and the tail entity is marked through multiple identical and independent binary classifiers, A joint extraction model JPEA, which integrates pre-trained model and attention, is constructed. Experimental results show that this method can significantly improve the extraction effect, and compare the performance of extraction tasks under different pre-trained models, which further illustrates the superiority of the model.

Key words: domain knowledge graph, pre-trained model, self-attention mechanism, feature fusion