Advance Research on Neural Machine Translation Integrating Linguistic Knowledge

doi:10.3778/j.issn.1673-9418.2012006

Abstract

Abstract:

Although neural machine translation has become the mainstream method and paradigm in the current research and application of machine translation, there are also some problems such as the fluent but not faithful of the translation results, difficult processing of rare words, poor performance of low-resource languages, poor cross-domain adaptability, and low prior knowledge utilization. Inspired by statistical machine translation research, incorporating linguistic information into neural machine translation models, using existing linguistic knowledge, alleviating the inherent difficulties faced by neural machine translation and improving translation quality has become a hot topic in the field of neural machine translation research. According to the grammatical unit??s classification system, the research in this area can be divided into three categories: neural machine translation incorporating character or word structure information, neural machine translation incorporating phrase structure information, and neural machine translation incorporating syntactic structure information. The current research also focuses on these three aspects. On the basis of sorting out the main challenges and reasons faced by neural machine translation, this paper focuses on each type of research to introduce its core ideas and functions, status and main results, problems and development trends. Finally, it summarizes the challenges that still exist in this field and looks forward to future research direction.

Key words: neural machine translation, linguistic knowledge, character or word structure information, phrase structure information, syntactic structure information

摘要：

尽管神经机器翻译已经成为目前机器翻译研究应用中的主流方法与范式，然而同时也存在译文流利但不够忠实、罕见词处理困难、低资源语言表现不佳、跨领域适应性差、先验知识利用率低等问题。受统计机器翻译研究启发，在神经机器翻译模型中融入语言学信息，利用已有的语言学知识，缓解神经机器翻译面临的固有困境，提升翻译质量，成为神经机器翻译研究领域的一个热门话题。根据语法单位分类体系，可以将这方面的研究分为三类：融合字词结构信息的神经机器翻译、融合短语结构的神经机器翻译和融合句法结构信息的神经机器翻译。目前的研究主要集中在这三方面。在梳理神经机器翻译面临的主要挑战及原因的基础上，重点介绍了每一类研究的核心思想与作用、现状与主要成果、面临的问题及发展趋势。最后总结归纳现有研究中面临的主要挑战，并对未来的研究方向进行展望。

关键词: 神经机器翻译, 语言学知识, 字词结构信息, 短语结构信息, 句法结构信息

GUO Wanghao, FAN Jiangwei, ZHANG Keliang. Advance Research on Neural Machine Translation Integrating Linguistic Knowledge[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(7): 1183-1194.

郭望皓, 范江威, 张克亮. 融合语言学知识的神经机器翻译研究进展[J]. 计算机科学与探索, 2021, 15(7): 1183-1194.

References

[1] KOEHN P, KNOWLES R. Six challenges for neural machine translation[C]//Proceedings of the 1st Workshop on Neural Machine Translation, Vancouver, Aug 4, 2017. Stroudsburg: ACL, 2017: 28-39.
[2] LIU Y. Recent advances in neural machine translation[J]. Journal of Computer Research and Development, 2017, 54(6): 1144-1149.
刘洋. 神经机器翻译前沿进展[J]. 计算机研究与发展, 2017, 54(6): 1144-1149.
[3] LI Y C, XIONG D Y, ZHANG M. A survey of neural machine translation[J]. Chinese Journal of Computers, 2018, 41(12): 2734-2755.
李亚超, 熊德意, 张民. 神经机器翻译综述[J]. 计算机学报, 2018, 41(12): 2734-2755.
[4] LUONG M T, SUTSKEVER I, LE Q, et al. Addressing the rare word problem in neural machine translation[C]//Proceedings of the 53rd Annual Meeting of the Association for Compu-tational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, Jul 26-31, 2015. Stroudsburg: ACL, 2015: 11-19.
[5] JEAN S, CHO K, MEMISEVIC R, et al. On using very large target vocabulary for neural machine translation[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, Jul 26-31, 2015. Stroudsburg: ACL, 2015: 1-10.
[6] BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate[C]//Proceedings of the 3rd International Conference on Learning Representations, San Diego, May 7-9, 2015. La Jolla: ICLR, 2015: 1-15.
[7] SENNRICH R, HADDOW B, BIRCH A. Improving neural machine translation models with monolingual data[C]//Pro-ceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Aug 7-12, 2016. Stroudsburg: ACL, 2016: 86-96.
[8] CHU C, DABRE R, KUROHASHI S. An empirical comparison of domain adaptation methods for neural machine translation[C]//Proceedings of the 55th Annual Meeting of the Asso-ciation for Computational Linguistics, Vancouver, Jul 30-Aug 4, 2017. Stroudsburg: ACL, 2017: 385-391.
[9] LI X, ZHANG J, ZONG C. Towards zero unknown word in neural machine translation[C]//Proceedings of the 25th Inter-national Joint Conference on Artificial Intelligence, New York, Jul 9-15, 2016. San Francisco: Morgan Kaufmann, 2016: 2852-2858.
[10] KIM Y, JERNITE Y, SONTAG D, et al. Character-aware neural language models[C]//Proceedings of the 30th AAAI Conference on Artificial Intelligence, Phoenix, Feb 12-17, 2016. Palo Alto: AAAI, 2016: 2741-2749.
[11] HAHN M, BARONI M. Tabula nearly rasa: probing the linguistic knowledge of character-level neural language models trained on unsegmented text[J]. arXiv:1906.07285, 2019.
[12] LING W, TRANCOSO I, DYER C, et al. Character-based neural machine translation[J]. arXiv:1511.04586, 2015.
[13] LEE J, CHO K, HOFMANN T. Fully character-level neural machine translation without explicit segmentation[J]. Trans-actions of the Association for Computational Linguistics, 2017, 5: 365-378.
[14] SENNRICH R, HADDOW B, BIRCH A. Neural machine translation of rare words with subword units[C]//Proceed-ings of the 54th Annual Meeting of the Association for Com-putational Linguistics, Berlin, Aug 7-12, 2016. Stroudsburg: ACL, 2016: 1715-1725.
[15] WU Y, SCHUSTER M, CHEN Z, et al. Google??s neural machine translation system: bridging the gap between human and machine translation[J]. arXiv:1609.08144, 2016.
[16] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, Dec 4-9, 2017. Cambridge: MIT Press, 2017: 6000-6010.
[17] COSTA-JUSSà M R, FONOLLOSA G A. Character-based neural machine translation[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics,Berlin, Aug 7-12, 2016. Stroudsburg: ACL, 2016: 357-361.
[18] CHUNG J, CHO K, BENGIO Y. A character-level decoder without explicit segmentation for neural machine translation[C]//Proceedings of the 54th Annual Meeting of the Associ-ation for Computational Linguistics, Berlin, Aug 7-12, 2016. Stroudsburg: ACL, 2016: 1693-1703.
[19] YANG Z, CHEN W, WANG F, et al. A character-aware encoder for neural machine translation[C]//Proceedings of the 26th International Conference on Computational Linguistics, Osaka, Dec 11-16, 2016. Cambridge: MIT Press, 2016: 3063-3070.
[20] SU J, TAN Z, XIONG D, et al. Lattice-based recurrent neural network encoders for neural machine translation[C]//Proceedings of the 31st AAAI Conference on Artificial Intelligence, San Francisco, Feb 4-9, 2017. Palo Alto: AAAI, 2017: 3302-3308.
[21] LUONG M T, MANNING C D. Achieving open vocabulary neural machine translation with hybrid word-character models[C]//Proceedings of the 54th Annual Meeting of the Associ-ation for Computational Linguistics, Berlin, Aug 7-12, 2016. Stroudsburg: ACL, 2016: 1054-1063.
[22] CHEN H, HUANG S, CHIANG D, et al. Combining character and word information in neural machine translation using a multi-level attention[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Com-putational Linguistics: Human Language Technologies, New Orleans, Jun 1-6, 2018. Stroudsburg: ACL, 2018: 1284-1293.
[23] WANG F, CHEN W, YANG Z, et al. Hybrid attention for Chinese character-level neural machine translation[J]. Neuro-computing, 2019, 358: 44-52.
[24] ZHANG L, KOMACHI M. Neural machine translation of logographic language using sub-character level information[C]//Proceedings of the 3rd Conference on Machine Trans-lation, Brussels, Oct 31-Nov 1, 2018. Stroudsburg: ACL, 2018: 17-25.
[25] KOEHN P, OCH F J, MARCU D. Statistical phrase-based translation[C]//Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, Edmonton, May 27-Jun 1, 2003. Stroudsburg: ACL, 2003: 48-54.
[26] CHIANG D, LOPEZ A, MADNANI N, et al. The Hiero machine translation system: extensions, evaluation, and analysis[C]//Proceedings of the 2005 Conference on Human Language Technology and Empirical Methods in Natural Language Processing, Vancouver, Oct 6-8, 2005. Stroudsburg: ACL, 2005: 779-786.
[27] KOEHN P. Statistical machine translation[M]. Cambridge: Cambridge University Press, 2009.
[28] WANG X, TU Z, XIONG D, et al. Translating phrases in neural machine translation[C]//Proceedings of the 2017 Con-ference on Empirical Methods in Natural Language Process-ing, Copenhagen, Sep 9-11, 2017. Stroudsburg: ACL, 2017: 1432-1441.
[29] TANG Y, MENG F, LU Z, et al. Neural machine translation with external phrase memory[J]. arXiv:1606.01792, 2016.
[30] DAHLMANN L, MATUSOV E, PETRUSHKOV P, et al. Neural machine translation leveraging phrase-based models in a hybrid search[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Sep 9-11, 2017. Stroudsburg: ACL, 2017: 1421-1431.
[31] RIKTERS M, BOJAR O. Paying attention to multi-word expressions in neural machine translation[J]. arXiv:1710.06313, 2017.
[32] LI Y C, XIONG D Y, ZHANG M. Neural machine translation with phrasal attention[C]//Proceedings of the 13th China Workshop on Machine Translation, Dalian, Sep 27-29, 2017. Berlin, Heidelberg: Springer, 2017: 1-8.
[33] ISHIWATARI S, YAO J, LIU S, et al. Chunk-based decoder for neural machine translation[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Jul 30-Aug 4, 2017. Stroudsburg: ACL, 2017: 1901-1912.
[34] ZHOU H, TU Z, HUANG S, et al. Chunk-based bi-scale decoder for neural machine translation[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Jul 30-Aug 4, 2017. Stroudsburg: ACL, 2017: 580-586.
[35] HUANG P S, WANG C, HUANG S, et al. Towards neural phrase-based machine translation[J]. arXiv:1706.05565, 2017.
[36] WANG C, WANG Y, HUANG P S, et al. Sequence modeling via segmentations[C]//Proceedings of the 34th International Conference on Machine Learning, Aug 6-11, 2017. New York: ACM, 2017: 3674-3683.
[37] LIU Y, LIU Q, LIN S. Tree-to-string alignment template for statistical machine translation[C]//Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Jul 17-21, 2006. Stroudsburg: ACL, 2006: 609-616.
[38] NGUYEN T P, SHIMAZU A, HO T B, et al. A tree-to-string phrase-based model for statistical machine translation[C]//Proceedings of the 12th Conference on Computational Natural Language Learning, Manchester, Aug 16-17, 2008. Stroudsburg: ACL, 2008: 143-150.
[39] MARTON Y, RESNIK P. Soft syntactic constraints for hierarchical phrased-based translation[C]//Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics,Columbus, Jun 15-20, 2008. Stroudsburg: ACL, 2008: 1003-1011.
[40] SHEN L, XU J, WEISCHEDEL R. A new string-to-dependency machine translation algorithm with a target dependency language model[C]//Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics, Columbus, Jun 15-20, 2008. Stroudsburg: ACL, 2008: 577-585.
[41] XIE J, MI H, LIU Q. A novel dependency-to-string model for statistical machine translation[C]//Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, Jul 27-29, 2011. Stroudsburg: ACL, 2011: 216-226.
[42] LI P, LIU Y, SUN M. Recursive autoencoders for ITG-based translation[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Oct 18-21, 2013. Stroudsburg: ACL, 2013: 567-577.
[43] ERIGUCHI A, HASHIMOTO K, TSURUOKA Y. Tree-to-sequence attentional neural machine translation[C]//Proceedings of the 54th Annual Meeting of the Association for Compu-tational Linguistics, Berlin, Aug 7-12, 2016. Stroudsburg: ACL, 2016: 823-833.
[44] CHEN H, HUANG S, CHIANG D, et al. Improved neural machine translation with a syntax-aware encoder and decoder[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Jul 30-Aug 4, 2017. Stroudsburg: ACL, 2017: 1936-1945.
[45] TU Z, LU Z, LIU Y, et al. Modeling coverage for neural machine translation[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Aug 7-12, 2016. Stroudsburg: ACL, 2016: 688-697.
[46] LI J, XIONG D, TU Z, et al. Modeling source syntax for neural machine translation[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics,Vancouver, Jul 30-Aug 4, 2017. Stroudsburg: ACL, 2017: 76-85.
[47] BASTINGS J, TITOV I, AZIZ W, et al. Graph convolutional encoders for syntax-aware neural machine translation[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Sep 9-11, 2017. Stroudsburg: ACL, 2017: 1957-1967.
[48] MA C, TAMURA A, UTIYAMA M, et al. Forest-based neural machine translation[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics,Melbourne, Jul 15-20, 2018. Stroudsburg: ACL, 2018: 1253-1263.
[49] XU K, WU L, WANG Z, et al. Graph2seq: graph to sequence learning with attention-based neural networks[J]. arXiv:1804.00823, 2018.
[50] SENNRICH R, HADDOW B. Linguistic input features improve neural machine translation[C]//Proceedings of the 1st Con-ference on Machine Translation: Volume 1, Research Papers, Berlin, Aug 7-12, 2016. Stroudsburg: ACL, 2016: 83-91.
[51] NADEJDE M, REDDY S, SENNRICH R, et al. Predicting target language CCG supertags improves neural machine translation[C]//Proceedings of the 2nd Conference on Machine Translation, Copenhagen, Sep 7-8, 2017. Stroudsburg: ACL, 2017: 68-79.
[52] AHARONI R, GOLDBERG Y. Towards string-to-tree neural machine translation[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Jul 30-Aug 4, 2017. Stroudsburg: ACL, 2017: 132-140.
[53] ERIGUCHI A, TSURUOKA Y, CHO K. Learning to parse and translate improves neural machine translation[C]//Pro-ceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Jul 30-Aug 4, 2017. Stroudsburg: ACL, 2017: 72-78.
[54] WU S, ZHANG D, YANG N, et al. Sequence-to-dependency neural machine translation[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics,Vancouver, Jul 30-Aug 4, 2017. Stroudsburg: ACL, 2017: 698-707.
[55] LE A N, MARTINEZ A, YOSHIMOTO A, et al. Improving sequence to sequence neural machine translation by utilizing syntactic dependency information[C]//Proceedings of the 8th International Joint Conference on Natural Language Proce-ssing, Taiwan, China, Nov 28-30, 2017. Stroudsburg: ACL, 2017: 21-29.