Entity Anomaly Recognition Method Based on GCNN and GRU

doi:10.3778/j.issn.1673-9418.2204015

Abstract

Abstract: Named entity recognition (NER) model can recognize the normal entities, but cannot provide any hints for the missing entity or the entity in the incorrect location, which cannot meet the extensive requirements on the text entities in the information processing and the archiving analysis field. Combined with the specific context characteristics of the entity anomaly recognition, a method for the entity location anomaly and the entity absence anomaly detection (NER-EAD) integrating GCNN (gated convolutional neural network) and GRU (gated recurrent unit) and its training data construction method are proposed, which is based on the structure of the named entity recognition model with the pre-trained language model. GCNN extracts more character context features to improve the model performance in identifying abnormal entities. The method integrates the semantic feature output of the convolutional neural network and the recurrent neural network to comprehensively extract the features of normal entities and the abnormal entities. Experiments show that NER-EAD reaches average F1 of 90.56%, 85.56% and 80.92% in the normal entity recognition, the entity location anomaly detection and the entity absence anomaly detection, respectively, surpassing the existing named entity recognition model architecture. Additionally, the ablation experiment proves the semantic feature extraction ability of the fusion network of GCNN and GRU.

Key words: named entity recognition (NER), gated convolutional neural network (GCNN), gated recurrent unit (GRU), abnormal detection

摘要： 当前的命名实体识别（NER）模型能够识别位于正确位置且符合语法表达的实体，却无法指出句子中的实体缺失与位于错误位置的实体，无法满足信息处理与归档分析中对于检测文本实体信息完整全面的要求。通过考察异常实体的识别依赖上下文相互联系语义特征的具体特点，提出以基于预训练语言模型的命名实体识别模型架构为基础，融合门控卷积神经网络（GCNN）与门控循环网络（GRU）的实体位置异常与实体缺失异常检测方法（NER-EAD）及其训练数据构造方法。其中门控卷积网络提取特定字符上下文特征联系以更好识别实体异常。融合卷积神经网络结构和门控循环神经网络的语义特征输出可全面提取正常实体与异常实体的特征，实现了正常、异常实体识别结果同时输出。实验表明NER-EAD在正常实体、实体位置异常和实体缺失异常的识别平均[F1]分别达到90.56%、85.56%和80.92%，超越了已有命名实体识别模型架构。最后通过消融实验证明了GCNN与GRU融合网络的语义特征提取能力。

关键词: 命名实体识别（NER）, 门控卷积神经网络（GCNN）, 门控循环网络（GRU）, 异常检测

YE Han, SUN Haichun, LI Xin. Entity Anomaly Recognition Method Based on GCNN and GRU[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(8): 1938-1948.

叶瀚, 孙海春, 李欣. 融合GCNN与GRU的异常实体识别方法[J]. 计算机科学与探索, 2023, 17(8): 1938-1948.

References

[1] BANERJEE P S, CHAKRABORTY B, TRIPATHI D, et al. A information retrieval based on question and answering and NER for unstructured information without using SQL[J]. Wireless Personal Communications, 2019, 108(3): 1909-1931.
[2] JIA Y, QI Y, SHANG H, et al. A practical approach to cons-tructing a knowledge graph for cybersecurity[J]. Cyber-security, 2018, 4(1): 53-60.
[3] TO H D, DO P. Extracting triples from Vietnamese text to create knowledge graph[C]//Proceedings of the 2020 12th International Conference on Knowledge and Systems Engi-neering, Can Tho, Nov 12-14, 2020. Piscataway: IEEE, 2020: 219-223.
[4] LI Y, CAO J, WANG Y. Implementation of intelligent question answering system based on basketball knowledge graph[C]//Proceedings of the 2019 IEEE 4th Advanced Informa-tion Technology, Electronic and Automation Control Confe-rence, Chengdu, Dec 20-22, 2019. Piscataway: IEEE, 2019: 2601-2604.
[5] COLLOBERT R, WESTON J, BOTTOU L, et al. Natural lan-guage processing (almost) from scratch[J]. The Journal of Machine Learning Research, 2011, 12: 2493-2537.
[6] ZHOU P, ZHENG S, XU J, et al. Joint extraction of multiple relations and entities by using a hybrid neural network[M]//Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data. Cham: Springer, 2017: 135-146.
[7] LI X, FENG J, MENG Y, et al. A unified MRC framework for named entity recognition[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Lin-guistics. Stroudsburg: ACL, 2020: 5849-5859.
[8] LI J, SUN A, HAN J, et al. A survey on deep learning for named entity recognition[J]. IEEE Transactions on Know-ledge and Data Engineering, 2022, 34(1): 50-70.
[9] MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estima-tion of word representations in vector space[C]//Procee-dings of the 1st International Conference on Learning Re-presentations, Scottsdale, May 2-4, 2013.
[10] PETERS M, NEUMANN M, IYYER M, et al. Deep contex-tualized word representations[C]//Proceedings of the 2018 Conference of the North American Chapter of the Asso-ciation for Computational Linguistics: Human Language Technologies, New Orleans, Jun 1-6, 2018. Stroudsburg: ACL, 2018: 2227-2237.
[11] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language under-standing[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa-tional Linguistics: Human Language Technologies, Minnea-polis, Jun 3-5, 2019. Stroudsburg: ACL, 2019: 4171-4186.
[12] MOORE C R, FARRAG A, ASHKIN E. Using natural lan-guage processing to extract abnormal results from cancer screening reports[J]. Journal of Patient Safety, 2017, 13(3): 138-143.
[13] ZHANG Y, LIU M, HU S, et al. Development and multi-center validation of chest X-ray radiography interpretations based on natural language processing[J]. Communications Medicine, 2021, 1(1): 43.
[14] GAN W, SU B, LI Y. An electric system abnormal analysis framework based on natural language processing[C]//Pro-ceedings of the 2020 8th International Conference on Ad-vanced Cloud and Big Data, Taiyuan, Dec 5-6, 2020. Pisca-taway: IEEE, 2020: 149-152.
[15] CHANG W, XU Z, ZHOU S, et al. Research on detection methods based on Doc2vec abnormal comments[J]. Future Generation Computer Systems, 2018, 86: 656-662.
[16] FELICE M, BRISCOE T. Towards a standard evaluation method for grammatical error detection and correction[C]//Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, May 31-Jun 5, 2015. Stroudsburg: ACL, 2015: 578-587.
[17] SOORAJ S, MANJUSHA K, ANAND KUMAR M, et al. Deep learning based spell checker for Malayalam language[J]. Journal of Intelligent & Fuzzy Systems, 2018, 34(3): 1427-1434.
[18] NGUYEN T T H, JATOWT A, NGUYEN N V, et al. Neural machine translation with BERT for post-OCR error detec-tion and correction[C]//Proceedings of the 2020 ACM/IEEE Joint Conference on Digital Libraries, New York, Aug 1-5, 2020. New York: ACM, 2020: 333-336.
[19] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, Dec 4-7, 2017: 6000-6010.
[20] HOCHREITER S, SCHMIDHUBER J. Long short-term me-mory[J]. Neural Computation, 1997, 9(8): 1735-1780.
[21] CHO K, VAN MERRI?NBOER B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Lan-guage Processing, Doha, Oct 26-28, 2014. Stroudsburg: ACL, 2014: 1724-1734.
[22] DAUPHIN Y N, FAN A, AULI M, et al. Language mode-ling with gated convolutional networks[C]//Proceedings of the 34th International Conference on Machine Learning, Sydney, Aug 7-9, 2017: 933-941.
[23] LAFFERTY J, MCCALLUM A, PEREIRA F C N. Condi-tional random fields: probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the 18th International Conference on Machine Learning, San Fran-cisco, Jun 28-Jul 1, 2001. San Francisco: Morgan Kaufmann Publishers Inc., 2001: 282-289.
[24] FORNEY G D. The viterbi algorithm[J]. Proceedings of the IEEE, 1973, 61(3): 268-278.
[25] ZHOU J T, ZHANG H, JIN D, et al. RoSeq: robust sequence labeling[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(7): 2304-2314.
[26] CHUNG J, GULCEHRE C, CHO K, et al. Empirical eva-luation of gated recurrent neural networks on sequence mo-deling[J]. arXiv:1412.3555, 2014.