Research of Short Text Multi-intent Detection with Capsule Network

doi:10.3778/j.issn.1673-9418.1912058

Abstract

Abstract:

Intent detection is a key sub-task of spoken language understanding in human-machine dialogue system. Considering the problem of user??s multi-intent expressed, a multi-intent classifier based on single-intent marker is constructed by using capsule network to identify user??s multiple intents expressed. In order to ensure the feature quality of the intent text, the deep semantic information of intent text is extracted by adding convolution capsule layer in capsule network, at the same time feature capsules are dynamically allocated to intent capsule category by using dynamic routing in capsule network. The probability of multiple intents is determined by setting threshold value, thus completing the task of multi-intent detection. Experimental results show that the capsule network is better than the convolutional neural network in the multi-intent detection task, the capsule network with convolution capsule layer can improve the performance of multi-intent detection, and the macro average F1 values on Chinese and English datasets reach 77.3% and 94.7% respectively.

Key words: multi-intent detection, deep learning, spoken language understanding, dialogue system

摘要：

意图识别是人机对话系统中口语理解的关键子任务。考虑到当前用户表达存在多个意图的问题，主要采用胶囊网络构造基于单意图标记的多意图分类器对用户表达的多种意图进行识别。为了保证意图文本的特征质量，通过在胶囊网络中增加卷积胶囊层提取意图文本的深层次语义信息，同时利用胶囊网络中的动态路由将特征胶囊动态分配到意图胶囊类别中，通过设置阈值大小判别多种意图存在的概率，从而完成多意图识别任务。实验结果表明在多意图识别任务中，胶囊网络优于卷积神经网络，而增加卷积胶囊层的胶囊网络可以提升多意图识别的性能效果，在中文和英文数据集上的宏平均F1值分别达到77.3%和94.7%。

关键词: 多意图识别, 深度学习, 口语理解, 对话系统

LIU Jiao, LI Yanling, LIN Min. Research of Short Text Multi-intent Detection with Capsule Network[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(10): 1735-1743.

刘娇，李艳玲，林民. 胶囊网络用于短文本多意图识别的研究[J]. 计算机科学与探索, 2020, 14(10): 1735-1743.

References

[1] Chen H S, Liu X R, Yin D W, et al. A survey on dialogue sys-tems: recent advances and new frontiers[J]. SIGKDD Exp-lorations, 2017, 19(2): 25-35.
[2] Moldovan D, Pasca M, Marabagiu S, et al. Performance iss-ues and error analysis in an open-domain question answer-ing system[J]. ACM Transactions on Information Systems, 2003, 21(2): 133-154.
[3] Liu J, Li Y L, Lin M. Review of intent detection methods in human-machine dialogue system[J]. Computer Engineering and Applications, 2019, 55(12): 1-7.刘娇，李艳玲，林民. 人机对话系统中意图识别方法综述[J]. 计算机工程与应用，2019，55(12): 1-7.
[4] Xu X L. Research on multi-label short text classification based on deep learning[D]. Guilin: Guilin University of Elec-tronic Science and Technology, 2019.徐晓璐. 基于深度学习的多标签短文本分类方法研究[D]. 桂林: 桂林电子科技大学, 2019.
[5] Liu X H, Chen W S, Zhou A, et al. Multi-label text classifi-cation based on joint model[J]. Computer Engineering and Applications, 2020, 56(14): 111-117.刘心惠, 陈文实, 周爱, 等. 基于联合模型的多标签文本分类研究[J]. 计算机工程与应用, 2020, 56(14): 111-117.
[6] Li D Y, Luo F, Wang S G. A multi-label emotion classifica-tion method for Chinese text based on CNN and tag features[J]. Journal of Shanxi University (Natural Science Edition), 2020, 43(1): 65-71.李德玉, 罗锋, 王素格. 融合 CNN 和标签特征的中文文本情绪多标签分类[J]. 山西大学学报(自然科学版), 2020, 43(1): 65-71.
[7] Mu J P, Cai J, Yu M C, et al. Label-correlation based multi-label classification algorithm with label-specific features[J/OL]. Application Research of Computers [2019-09-25]. https://doi.org/10.19734/j.issn.1001-3695.2019.04.0118.牟甲鹏, 蔡剑, 余孟池, 等. 一种基于标签相关性的类属属性多标签分类算法[J/OL]. 计算机应用研究[2019-09-25]. https://doi.org/10.19734/j.issn.1001-3695.2019.04.0118.
[8] Xu P Y, Sarikaya R. Exploiting shared information for multi-intent natural language sentence classification[C]//Proceedings of the 14th Annual Conference of the International Speech Communication Association, Lyon, Aug 25-29, 2013: 3785-3789.
[9] Kim B, Ryu S, Gary G L. Two-stage multi-intent detection for spoken language understanding[J]. Multimedia Tools and Applications, 2017, 76(9): 1-14.
[10] Ratnaparkhi A. Maximum entropy models for natural lang-uage ambiguity resolution[M]. Philadelphia: University of Pen-nsylvania, 1998.
[11] Lafferty J, Mccallum A, Pereira F C N. Conditional random fields: probabilistic models for segmenting and labeling seq-uence data[C]//Proceedings of the 18th International Con-ference on Machine Learning, Williamstown, Jun 28-Jul 1, 2001. San Mateo: Morgan Kaufmann, 2001: 282-289.
[12] Yang C N, Feng C S. Multi-intention recognition model with combination of syntactic feature and convolution neural network[J]. Journal of Computer Applications, 2018, 38(7): 1839-1845.杨春妮, 冯朝胜. 结合句法特征和卷积神经网络的多意图识别模型[J]. 计算机应用, 2018, 38(7): 1839-1845.
[13] Kim Y. Convolutional neural networks for sentence classifi-cation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Oct 25-29, 2014. Stroudsburg: ACL, 2014: 1746-1751.
[14] Sabour S, Frosst N, Hinton G E. Dynamic routing between capsules[C]//Proceedings of the 2017 Annual Conference on Neural Information Processing Systems, Long Beach, Dec 4-9, 2017. Red Hook: Curran Associates, 2017: 3859-3869.
[15] Hinton G E, Krizhevsky A, Wang S D. Transforming auto-encoders[C]//LNCS 6791: Proceedings of the 21st International Conference on Artificial Neural Networks, Espoo, Jun 14-17, 2011. Berlin, Heidelberg: Springer, 2011: 44-51.
[16] Zhao W, Ye J B, Yang M, et al. Investigating capsule networks with dynamic routing for text classification[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Oct 31-Nov 4, 2018. Strouds-burg: ACL, 2018: 3110-3119.
[17] Xia C Y, Zhang C W, Yan C W, et al. Zero-shot user intent detection via capsule neural networks[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Lan-guage Processing, Brussels, Oct 31-Nov 4, 2018. Strouds-burg: ACL, 2018: 3090-3099.
[18] Renkens V, van Hamme H. Capsule networks for low resource spoken language understanding[C]//Proceedings of the 19th Annual Conference of the International Speech Communi-cation on Association, Hyderabad, Sep 2-6, 2018: 601-605.
[19] Renkens V, Janssens S, Ons B, et al. Acquisition of ordinal words using weakly supervised NMF[C]//Proceedings of the 2014 IEEE Spoken Language Technology Workshop, South Lake Tahoe, Dec 7-10, 2014. Piscataway: IEEE, 2014: 30-35.
[20] Yang J C, Han S J, Mao L, et al. Review of capsule network [J]. Journal of Shandong University (Engineering Science), 2019, 49(5): 1-8.杨巨成, 韩书杰, 毛磊, 等. 胶囊网络模型综述[J]. 山东大学学报（工学版）, 2019, 49(5): 1-8.
[21] Zhu Y Z, Hu Y M, Li M. Research on capsule network technology and development trend[J]. Guangdong Communi-cation Technology, 2018, 38(10): 51-54.朱应钊, 胡颖茂, 李嫚. 胶囊网络技术及发展趋势[J]. 广东通信技术, 2018, 38(10): 51-54.
[22] Zhang W N, Chen Z G, Che W X, et al. The first evaluation of Chinese human-computer dialogue technology[J]. arXiv:1709.10217, 2017.