Dual Features Local-Global Attention Model with BERT for Aspect Sentiment Analysis

doi:10.3778/j.issn.1673-9418.2210012

Abstract

Abstract: Aspect-based sentiment analysis aims to predict the sentiment polarity of a specific aspect in a sentence or document. Most of recent research uses attention mechanism to model the context. But there is a problem in that the context information needs to be considered according to different contexts when the BERT model is used to calculate the dependencies between representations to extract features by sentiment classification models, which leads to the lack of contextual knowledge of the modelled features. And the importance of aspect words is not given more attention, affecting the overall classification performance of the model. To address the problems above, this paper proposes a dual features local-global attention model with BERT (DFLGA-BERT). Local and global feature extraction modules are designed respectively to fully capture the semantic association between aspect words and context. Moreover, an improved quasi-attention mechanism is used in DFLGA-BERT, which leads to the model using minus attention in the fusion of attention to weaken the effect of noise on classification in the text. The feature fusion structure of local and global features is designed to better integrate regional and global features based on conditional layer normalization (CLN). Experiments are conducted on the SentiHood and SemEval 2014 Task 4 datasets. Experimental results show that the performance of the proposed model is significantly improved compared with the baselines after incorporating contextual features.

Key words: sentiment analysis, natural language understanding, quasi-attention mechanism, contextual attention

摘要： 方面情感分析旨在预测句子或文档中一个特定方面的情感极性，现阶段大部分的研究都是使用注意力机制对上下文进行建模。然而，目前情感分类模型在使用BERT模型计算表征之间的依赖关系抽取特征时，大多未根据不同的语境背景考虑上下文信息，导致建模后的特征缺乏上下文的语境信息。同时，方面词的重要性未能得到充分的重视而影响模型整体分类的性能。针对上述问题，提出双特征融合注意力方面情感分析模型（DFLGA-BERT），分别设计了局部与全局的特征抽取模块，充分捕捉方面词和上下文的语义关联。并将一种改进的“准”注意力添加到DFLGA-BERT的全局特征抽取器中，使模型学习在注意力的融合中使用减性注意力以削弱噪声产生的负面影响。基于条件层规泛化（CLN）设计了局部特征和全局特征的特征融合结构来更好地融合局部和全局特征。在SentiHood和SemEval 2014 Task 4数据集上进行了实验，实验结果表明，与基线模型相比该模型在融入了上下文语境特征后取得了较明显的性能提升。

关键词: 情感分析, 自然语言理解, 准注意力机制, 上下文注意

LI Jin, XIA Hongbin, LIU Yuan. Dual Features Local-Global Attention Model with BERT for Aspect Sentiment Analysis[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(1): 205-216.

李锦, 夏鸿斌, 刘渊. 基于BERT的双特征融合注意力的方面情感分析模型[J]. 计算机科学与探索, 2024, 18(1): 205-216.

References

[1] YADAV A, VISHWAKARMA D K. Sentiment analysis using deep learning architectures: a review[J]. Artificial Intelligence Review, 2020, 53(6): 4335-4385.
[2] JO Y, OH A H. Aspect and sentiment unification model for online review analysis[C]//Proceedings of the 4th ACM Inter-national Conference on Web Search and Data Mining, Hong Kong, China, Feb 9-12, 2011: 815-824.
[3] PONTIKI M, GALANIS D, PAPAGEORGIOU H, et al. SemEval-2016 task 5: aspect based sentiment analysis[C]//Proceedings of the 2016 International Workshop on Semantic Evaluation. Stroudsburg: ACL, 2016: 19-30.
[4] SAEIDI M, BOUCHARD G, LIAKATA M, et al. Sentihood: targeted aspect based sentiment analysis dataset for urban neighbourhoods[J]. arXiv:1610.03771, 2016.
[5] TANG D, QIN B, LIU T. Aspect level sentiment classifica-tion with deep memory network[J]. arXiv:1605.08900, 2016.
[6] WANG S, MAZUMDER S, LIU B, et al. Target-sensitive memory networks for aspect sentiment classification[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Jul 15-20, 2018.Stroudsburg: ACL, 2018: 957-967.
[7] LIU J, ZHANG Y. Attention modeling for targeted sentiment[C]//Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics.Stroudsburg: ACL, 2017: 572-577.
[8] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you needC]//Advances in Neural Information Processing Systems 30, Long Beach, Dec 4-9, 2017: 5998-6008.
[9] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[J]. arXiv:1810.04805, 2018.
[10] SUN C, HUANG L, QIU X. Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence[J]. arXiv:1903.09588, 2019.
[11] ZENG B, YANG H, XU R, et al. LCF: a local context focus mechanism for aspect-based sentiment classification[J]. Applied Sciences, 2019, 9(16): 3389.
[12] 吴仁彪, 乔晗, 贾云飞, 等. 基于胶囊网络的中长微博情感分析[J]. 信号处理, 2022, 38(8): 1632-1641.
WU R B, QIAO H, JIA Y F, et al. Sentiment analysis of mid-length microblog based on capsule network[J]. Journal of Signal Processing, 2022, 38(8): 1632-1641.
[13] 赵志影, 邵新慧, 林幸. 用于方面情感分析的结合图卷积神经网络的注意力模型[J]. 中文信息学报, 2022, 36(7): 154-163.
ZHAO Z Y, SHAO X H, LIN X. GCN-aware attention networks for aspect-based sentiment analysis[J]. Journal of Chinese Information Processing, 2022, 36(7): 154-163.
[14] WU Z, ONG D C. Context-guided BERT for targeted aspect-based sentiment analysis[C]//Proceedings of the 2021 AAAI Conference on Artificial Intelligence, Feb 2-9, 2021. Menlo Park: AAAI, 2021: 14094-14102.
[15] JIANG L, YU M, ZHOU M, et al. Target-dependent Twitter sentiment classification[C]//Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: ACL, 2011: 151-160.
[16] YANG B, LI J, WONG D F, et al. Context-aware self-attention networks[C]//Proceedings of the 2019 AAAI Con-ference on Artificial Intelligence, Hawaii, Jan 27-31, 2019.Stroudsburg: ACL, 2019: 387-394.
[17] TAY Y, LUU A T, ZHANG A, et al. Compositional de-attention networks[C]//Advances in Neural Information Processing Systems 32, Vancouver, Dec?8-14,?2019: 6132-6142.
[18] 张龙辉, 尹淑娟, 任飞亮, 等. BSLRel: 基于二元序列标注的级联关系三元组抽取模型[J]. 中文信息学报, 2021, 35(6): 74-84.
ZHANG L H, YIN S J, REN F L, et al. BSLRel: a binary sequence labeling based cascading relation triple extraction model[J]. Journal of Chinese Information Processing, 2021, 35(6): 74-84.
[19] BRITZ D, GOLDIE A, LUONG M T, et al. Massive explo-ration of neural machine translation architectures[J]. arXiv:1703.03906, 2017.
[20] RUDER S, GHAFFARI P, BRESLIN J G. A hierarchical model of reviews for aspect-based sentiment analysis[J]. arXiv:1609.02745, 2016.
[21] TANG D, QIN B, FENG X, et al. Effective LSTMs for target-dependent sentiment classification[J]. arXiv:1512.01100, 2015.
[22] MA Y, PENG H, CAMBRIA E. Targeted aspect-based senti-ment analysis via embedding commonsense knowledge into an attentive LSTM[C]//Proceedings of the 32nd AAAI Con-ference on Artificial Intelligence, the 30th Innovative Appli-cations of Artificial Intelligence, and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, Feb 2-7, 2018. Stroudsburg: ACL, 2018: 5876-5883.
[23] CAMBRIA E, PORIA S, BAJPAI R, et al. SenticNet 4: a semantic resource for sentiment analysis based on conceptual primitives[C]//Proceedings of the 26th International Confe-rence on Computational Linguistics: Technical Papers, Osaka, Dec 9-14, 2016: 2666-2677.
[24] LIU F, COHN T, BALDWIN T. Recurrent entity networks with delayed memory update for targeted aspect-based senti-ment analysis[J]. arXiv:1804.11019, 2018.
[25] HENAFF M, WESTON J, SZLAM A, et al. Tracking the world state with recurrent entity networks[J]. arXiv:1612.03969, 2016.
[26] PHAN M H, OGUNBONA P O. Modelling context and syntactical features for aspect-based sentiment analysis[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2020: 3211-3220.
[27] MAO Y, SHEN Y, YU C, et al. A joint training dual-MRC framework for aspect based sentiment analysis[J]. arXiv:2101.00816, 2021.
[28] LI R, CHEN H, FENG F, et al. Dual graph convolutional networks for aspect-based sentiment analysis[C]//Proceedings of the 59th Annual Meeting of the Association for Computa-tional Linguistics and the 11th International Joint Conference on Natural Language Processing. Stroudsburg: ACL, 2021: 6319-6329.
[29] BRUN C, POPA D N, ROUX C. XRCE: hybrid classification for aspect-based sentiment analysis[C]//Proceedings of the 8th International Workshop on Semantic Evaluation, Dublin, Aug 23-24, 2014. Stroudsburg: ACL, 2014: 838-842.
[30] KIRITCHENKO S, ZHU X, CHERRY C, et al. NRC-Canada-2014: detecting aspects and sentiment in customer reviews[C]//Proceedings of the 8th International Workshop on Seman-tic Evaluation, Dublin, Aug 23-24, 2014. Stroudsburg: ACL, 2014: 437-442.