融合自注意力机制的长文本生成对抗网络模型

doi:10.3778/j.issn.1673-9418.2104038

计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (7): 1603-1610.DOI: 10.3778/j.issn.1673-9418.2104038

融合自注意力机制的长文本生成对抗网络模型

夏鸿斌¹^,², 肖奕飞¹^,⁺(), 刘渊¹^,²

1.江南大学人工智能与计算机学院,江苏无锡 214122
2.江苏省媒体设计与软件技术重点实验室,江苏无锡 214122

收稿日期:2021-04-12 修回日期:2021-07-08 出版日期:2022-07-01 发布日期:2021-07-20
作者简介:夏鸿斌（1972—）,男,江苏无锡人,博士,副教授,CCF会员,主要研究方向为个性化推荐、自然语言处理、网络优化。
XIA Hongbin, born in 1972, Ph.D., associate professor, member of CCF. His research interests include personalized recommendation, natural language processing and network optimization.
肖奕飞（1996—）,男,湖北黄石人,硕士研究生,主要研究方向为自然语言处理。
XIAO Yifei, born in 1996, M.S. candidate. His reaserch interest is natural language processing.
刘渊（1967—）,男,江苏无锡人,教授,CCF高级会员,主要研究方向为网络安全、社交网络。
LIU Yuan, born in 1967, professor, senior member of CCF. His research interests include network security and social network.
基金资助:
国家自然科学基金(61972182)

Long Text Generation Adversarial Network Model with Self-Attention Mechanism

XIA Hongbin¹^,², XIAO Yifei¹^,⁺(), LIU Yuan¹^,²

1. School of A.pngicial Intelligence and Computer, Jiangnan University, Wuxi, Jiangsu 214122, China
2. Jiangsu Key Laboratory of Media Design and Software Technology, Wuxi, Jiangsu 214122, China

Received:2021-04-12 Revised:2021-07-08 Online:2022-07-01 Published:2021-07-20
Supported by:
the National Natural Science Foundation of China(61972182)

摘要/Abstract

摘要：

近年来,人类与计算机之间的通信已经达到密不可分的程度,因此自然语言处理作为人类与机器进行交互的技术越来越受到研究者的关注,文本生成是自然语言处理中的常见任务之一,目前生成对抗网络在文本生成领域广泛使用并且性能优异。针对传统生成对抗网络判别器的标量指导信号稀疏性问题和仅能学习文本局部语义信息的限制,提出了一种基于多头自注意力机制与LeakGAN结合的模型（SALGAN）。首先,采用融入多头自注意力机制的CNN模型作为特征提取器提取特征向量,以增强特征提取能力;其次,判别器提取的特征作为逐步指导信号发送给生成器以指导生成器生成文本,使得生成的文本更倾向于参考文本;最后,生成器完成文本生成后传递给判别器判断真假,确认文本是否符合人类语言标准。在两个真实数据集COCO图像字幕和EMNLP2017新闻上进行实验,采用BLEU指标进行评估。实验结果表明,将多头自注意力机制融入CNN模型后文本包含全局语义信息,CNN模型的特征提取性能有明显的提升。

关键词: 生成对抗网络（GAN）, 多头自注意力机制, 文本生成, 深度学习, 门控循环单元（GRU）, 自然语言处理

Abstract:

In recent years, the communication between human and computer has reached the inseparable degree, so the natural language processing as interaction technology between human and machine attracts more and more attention from researchers. Text generation is one of the common tasks of natural language processing. Currently, generative adversarial networks (GAN) is widely used in the field of text generation and the performance is excellent. To solve the sparse problem of scalar guided signals in traditional generative adversarial network discriminator and the limitation of learning only partial semantic information of text, a model based on the combination of multi-head self-attention leak generative adversarial networks (SALGAN) is proposed. Firstly, the feature vector is extracted by using the CNN model integrated with the multi-head self-attention mechanism as the feature extractor to enhance the feature extraction ability. Secondly, the features extracted by the discriminator are sent to the generator as step-by-step guidance signals to guide the generator to generate text, which makes the generated text more inclined to the reference text. Finally, the generator generates the text and passes it to the discriminator to determine whether it is true or not, in order to confirm whether the text meets the standards of human language. Experiments are carried out on two real datasets, COCO image captions and EMNLP2017 news, and the BLEU index is used for evaluation. The experimental results show that the text contains global semantic information after the multi-head self-attention mechanism is integrated into the CNN model, and the feature extraction performance of the CNN model is significantly improved.

Key words: generative adversarial networks (GAN), multi-head self-attention mechanism, text generation, deep learning, gated recurrent unit (GRU), natural language processing

中图分类号:

TP391

夏鸿斌, 肖奕飞, 刘渊. 融合自注意力机制的长文本生成对抗网络模型[J]. 计算机科学与探索, 2022, 16(7): 1603-1610.

XIA Hongbin, XIAO Yifei, LIU Yuan. Long Text Generation Adversarial Network Model with Self-Attention Mechanism[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(7): 1603-1610.

图/表 8

参考文献 21

[1]	JONAS G, MICHAEL A, DAVID G, et al. A convolutional encoder model for neural machine translation[J]. arXiv:1611. 02344, 2016.
[2]	YANG Z, CHEN W, WANG F, et al. Improving neural machine translation with conditional sequence generative adversarial nets[J]. arXiv:1703. 04887, 2017.
[3]	LI J, MONROE W, SHI T, et al. Adversarial learning for neural dialogue generation[J]. arXiv:1701. 06547, 2017.
[4]	CHO K, VAN MERRIENBOER B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[J]. arXiv:1406. 1078, 2014.
[5]	GRAVES A. Generating sequences with recurrent neural networks[J]. arXiv:1308. 0850, 2013.
[6]	HOCHREITER S, SCHMIDHUBER J. Long short-term memory[J]. Neural Computation, 1997, 9(8): 1735-1780. DOI URL
[7]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[J]. arXiv:1706. 03762, 2017.
[8]	ZHANG X, LECUN Y. Text understanding from scratch[J]. arXiv:1502. 01710, 2015.
[9]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[C]// Advances in Neural Information Processing Systems 27, 2014: 2672-2680.
[10]	YU L, ZHANG W, WANG J, et al. SeqGAN: sequence generative adversarial nets with policy gradient[J]. arXiv:1609. 05473, 2016.
[11]	GUO J X, LU S D, CAI H, et al. Long text generation via adversarial training with leaked information[C]// Proceedings of the 32nd AAAI Conference on A.pngicial Intelligence, the 30th Innovative Applications of A.pngicial Intelligence, and the 8th AAAI Symposium on Educational Advances in A.pngicial Intelligence, New Orleans, Feb 2-7, 2018. Menlo Park: AAAI, 2018: 5141-5148.
[12]	DETHLEFS N, CUAYÁHUITL H. Hierarchical reinforcement learning for adaptive text generation[C]// Proceedings of the 6th International Natural Language Generation Conference, Trim, Jul 7-9, 2010. Stroudsburg: ACL, 2010: 1-9.
[13]	ALEXANDER S V, SIMON O, TOM S, et al. Feudal networks for hierarchical reinforcement learning[J]. arXiv:1703. 01161, 2017.
[14]	PENG B, LI X, LI L, et al. Composite task-completion dialogue system via hierarchical deep reinforcement learning[J]. arXiv:1704. 03084, 2017.
[15]	NIE W, NARODYTSKA N, PATEL A. RelGAN: relational generative adversarial networks for text generation[C]// Proceedings of the 7th International Conference on Learning Representations, New Orleans, May 6-9, 2019: 1462.
[16]	SUTTON R S, MCALLESTER D A, SINGH S P, et al. Policy gradient methods for reinforcement learning with function approximation[C]// Advances in Neural Information Processing Systems 12, Denver, Nov 29-Dec 4, 1999. Cam-bridge: MIT Press, 1999: 1057-1063.
[17]	THOMAS P S, BRUNSKILL E. Policy gradient methods for reinforcement learning with function approximation and action-dependent baselines[J]. arXiv:1706. 06643, 2017.
[18]	CHE T, LI Y, ZHANG R, et al. Maximum-likelihood augmented discrete generative adversarial networks[J]. arXiv: 1702. 07983, 2017.
[19]	PAPINENI K, ROUKOS S, WARD T, et al. Bleu: a method for automatic evaluation of machine translation[C]// Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Jul 6-12, 2002. Stroudsburg: ACL, 2002: 311-318.
[20]	NITISH S, HINTON G E, KRIZHEVSKY A, et al. Dropout: a simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research, 2014, 15(1): 1929-1958.
[21]	MIRZA M, OSINDERO S. Conditional generative adversarial nets[J]. arXiv:1411. 1784, 2014.

Length	MLE	SeqGAN	LeakGAN	RelGAN	SALGAN	Real
20	9.038	8.736	7.038	6.680	6.241	5.750
40	10.411	10.310	7.191	6.765	6.273	4.071

Length	MLE	SeqGAN	LeakGAN	RelGAN	SALGAN	Real
20	9.038	8.736	7.038	6.680	6.241	5.750
40	10.411	10.310	7.191	6.765	6.273	4.071

Method	BLEU-2	BLEU-3	BLEU-4	BLEU-5	P-value
MLE	0.731	0.497	0.305	0.189	<10^-6
SeqGAN	0.812	0.617	0.418	0.267	<10^-6
LeakGAN	0.833	0.635	0.423	0.277	<10^-6
SALGAN	0.863	0.721	0.524	0.354	<10^-6
RelGAN	0.849	0.687	0.502	0.331	<10^-6

Method	BLEU-2	BLEU-3	BLEU-4	BLEU-5	P-value
MLE	0.731	0.497	0.305	0.189	<10^-6
SeqGAN	0.812	0.617	0.418	0.267	<10^-6
LeakGAN	0.833	0.635	0.423	0.277	<10^-6
SALGAN	0.863	0.721	0.524	0.354	<10^-6
RelGAN	0.849	0.687	0.502	0.331	<10^-6

Method	BLEU-2	BLEU-3	BLEU-4	BLEU-5	P-value
MLE	0.768	0.473	0.240	0.126	<10^-6
SeqGAN	0.777	0.491	0.261	0.138	<10^-6
LeakGAN	0.826	0.645	0.437	0.272	<10^-6
SALGAN	0.876	0.654	0.404	0.233	<10^-6
RelGAN	0.868	0.686	0.478	0.301	<10^-6

融合自注意力机制的长文本生成对抗网络模型

Long Text Generation Adversarial Network Model with Self-Attention Mechanism

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 21

相关文章 15

编辑推荐

Metrics

Method	COCO IMAGE CAPTIONS	EMNLP2017 WMT NEWS
SeqGAN	(1) a man wearing a helmet and chairs in the middle of a kitchen holding a jug of milk. (2) a bathroom has a wall and shower.	(1) it takes a way for me and gives everyone think that I balance some more people don’t realize their parents. (2) i truly have been awared that the set of the supreme court failed to comment on the patient.
LeakGAN	(1) a red bathroom with a toilet and seat. (2) black and white photograph of a plane flying in the middle of a clear blue sky.	(1) i mean it is a birthday, I’ve had a long trip and got some hundred dollars, he said. (2) there’s a lot of people submitted with these issues and think about how to do to take the same experiences and recongnize their lives, the conpany said.
SALGAN	(1) a bird is sitting on a table. (2) a group of people sitting on the table with meat in front of a storefront.	(1) I don’t think he do not know what I think because there are things that millons of ohio are culture of the united states. (2) and later it might like the idea that it has to be a good person that in a certain time, that’s what we’re at the right thing by again, they should never be here, he said.
RelGAN	(1) a cat is looking on a man in a bathroom. (2) a man in a kitchen preparing food on a table.	(1) he has always vowed to give him a little more on for him, and I think he is very willing for the division. (2) i never thought I could put over the line but i counldn’t quite lose my job.

[1]	安凤平, 李晓薇, 曹翔. 权重初始化-滑动窗口CNN的医学图像分类[J]. 计算机科学与探索, 2022, 16(8): 1885-1897.
[2]	曾凡智, 许露倩, 周燕, 周月霞, 廖俊玮. 面向智慧教育的知识追踪模型研究综述[J]. 计算机科学与探索, 2022, 16(8): 1742-1763.
[3]	刘艺, 李蒙蒙, 郑奇斌, 秦伟, 任小广. 视频目标跟踪算法综述[J]. 计算机科学与探索, 2022, 16(7): 1504-1515.
[4]	赵小明, 杨轶娇, 张石清. 面向深度学习的多模态情感识别研究进展[J]. 计算机科学与探索, 2022, 16(7): 1479-1503.
[5]	韩毅, 乔林波, 李东升, 廖湘科. 知识增强型预训练语言模型综述[J]. 计算机科学与探索, 2022, 16(7): 1439-1461.
[6]	孙方伟, 李承阳, 谢永强, 李忠博, 杨才东, 齐锦. 深度学习应用于遮挡目标检测算法综述[J]. 计算机科学与探索, 2022, 16(6): 1243-1259.
[7]	刘雅芬, 郑艺峰, 江铃燚, 李国和, 张文杰. 深度半监督学习中伪标签方法综述[J]. 计算机科学与探索, 2022, 16(6): 1279-1290.
[8]	申瑞彩, 翟俊海, 侯璎真. 选择性集成学习多判别器生成对抗网络[J]. 计算机科学与探索, 2022, 16(6): 1429-1438.
[9]	林佳伟, 王士同. 用于无监督域适应的深度对抗重构分类网络[J]. 计算机科学与探索, 2022, 16(5): 1107-1116.
[10]	程卫月, 张雪琴, 林克正, 李骜. 融合全局与局部特征的深度卷积神经网络算法[J]. 计算机科学与探索, 2022, 16(5): 1146-1154.
[11]	钟梦圆, 姜麟. 超分辨率图像重建算法综述[J]. 计算机科学与探索, 2022, 16(5): 972-990.
[12]	裴利沈, 赵雪专. 群体行为识别深度学习方法研究综述[J]. 计算机科学与探索, 2022, 16(4): 775-790.
[13]	许嘉, 韦婷婷, 于戈, 黄欣悦, 吕品. 题目难度评估方法研究综述[J]. 计算机科学与探索, 2022, 16(4): 734-759.
[14]	朱伟杰, 陈莹. 双流时间域信息交互的微表情识别卷积网络[J]. 计算机科学与探索, 2022, 16(4): 950-958.
[15]	姜艺, 胥加洁, 柳絮, 朱俊武. 边缘指导图像修复算法研究[J]. 计算机科学与探索, 2022, 16(3): 669-682.