Abstract:

In recent years, the communication between human and computer has reached the inseparable degree, so the natural language processing as interaction technology between human and machine attracts more and more attention from researchers. Text generation is one of the common tasks of natural language processing. Currently, generative adversarial networks (GAN) is widely used in the field of text generation and the performance is excellent. To solve the sparse problem of scalar guided signals in traditional generative adversarial network discriminator and the limitation of learning only partial semantic information of text, a model based on the combination of multi-head self-attention leak generative adversarial networks (SALGAN) is proposed. Firstly, the feature vector is extracted by using the CNN model integrated with the multi-head self-attention mechanism as the feature extractor to enhance the feature extraction ability. Secondly, the features extracted by the discriminator are sent to the generator as step-by-step guidance signals to guide the generator to generate text, which makes the generated text more inclined to the reference text. Finally, the generator generates the text and passes it to the discriminator to determine whether it is true or not, in order to confirm whether the text meets the standards of human language. Experiments are carried out on two real datasets, COCO image captions and EMNLP2017 news, and the BLEU index is used for evaluation. The experimental results show that the text contains global semantic information after the multi-head self-attention mechanism is integrated into the CNN model, and the feature extraction performance of the CNN model is significantly improved.

Key words: generative adversarial networks (GAN), multi-head self-attention mechanism, text generation, deep learning, gated recurrent unit (GRU), natural language processing

摘要：

近年来,人类与计算机之间的通信已经达到密不可分的程度,因此自然语言处理作为人类与机器进行交互的技术越来越受到研究者的关注,文本生成是自然语言处理中的常见任务之一,目前生成对抗网络在文本生成领域广泛使用并且性能优异。针对传统生成对抗网络判别器的标量指导信号稀疏性问题和仅能学习文本局部语义信息的限制,提出了一种基于多头自注意力机制与LeakGAN结合的模型（SALGAN）。首先,采用融入多头自注意力机制的CNN模型作为特征提取器提取特征向量,以增强特征提取能力;其次,判别器提取的特征作为逐步指导信号发送给生成器以指导生成器生成文本,使得生成的文本更倾向于参考文本;最后,生成器完成文本生成后传递给判别器判断真假,确认文本是否符合人类语言标准。在两个真实数据集COCO图像字幕和EMNLP2017新闻上进行实验,采用BLEU指标进行评估。实验结果表明,将多头自注意力机制融入CNN模型后文本包含全局语义信息,CNN模型的特征提取性能有明显的提升。

关键词: 生成对抗网络（GAN）, 多头自注意力机制, 文本生成, 深度学习, 门控循环单元（GRU）, 自然语言处理