Study on Predicting Psychological Traits of Online Text by BERT

doi:10.3778/j.issn.1673-9418.2007009

Abstract

Abstract:

With the rapid development and popularity of the Internet, an increasing number of people would like to use online platforms to express themselves and communicate with others. It is inevitable that a large number of online text data are constantly emerging with personal information, which often indicate individual real expression in different conditions and reflect personal inner psychological traits and personality tendency. Applying text mining techniques to analyzing psychological traits behind the online text is not only helpful for individuals to understand themselves, but also useful to avoid the motivation interfere while using traditional methods for psychological assessment. In recent years, the language model named bidirectional encoder representations from transformers (BERT) has greatly improved the performance of both the text classification task and the sentiment analysis task. In this paper, prediction models for psychological traits are constructed based on online text. Comprehensive semantic features and long dependency in the context are obtained by BERT. Considering that distinct algorithm frameworks of classifiers can lead to different classification results, the fully-connected layer of the BERTBASE model and the random forest algorithm are used in the downstream classification task to make comparison. The results show that psychological traits can be effectively predicted from text classification based on BERT, and the average accuracy, average precision and other indicators are more than 97%.

Key words: bidirectional encoder representations from transformers (BERT), psychological trait, attention mechanism, Transformer, text mining

摘要：

随着互联网的普及应用，通过网络平台进行表达和交流的用户越来越多，在此过程中不可避免地会留下与个人相关的大量网络文本数据和信息，这些非结构化的文本数据往往体现着不同场景下的真实表达，反映了人们内在的心理特质及人格倾向。利用文本挖掘相关技术基于网络文本数据分析心理特质可以弥补传统心理测量方法易受应试动机等因素影响的缺陷。近年来，BERT语言表示模型在文本分类、情感分析等任务上取得了很好的效果。针对网络文本数据构建心理特质预测模型，基于BERT获取完整的上下文语义特征和长距离的上下文依赖关系；同时考虑到分类器内部结构的差异可能会导致不同的分类效果，在下游分类任务中分别采用BERTBASE模型的全连接层和经典的随机森林算法作为两种不同的分类器进行模型效果对比。结果显示，基于BERT的文本分类模型能够有效实现心理特质的预测, 平均准确率、平均精准率等各项指标都在97%以上。

关键词: BERT, 心理特质, 注意力机制, Transformer, 文本挖掘

ZHANG Han, JIA Tianyuan, LUO Fang, ZHANG Sheng, WU Xia. Study on Predicting Psychological Traits of Online Text by BERT[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(8): 1459-1468.

张晗, 贾甜远, 骆方, 张生, 邬霞. 面向网络文本的BERT心理特质预测研究[J]. 计算机科学与探索, 2021, 15(8): 1459-1468.

References

[1] GOU L, ZHOU M X, YANG H H. KnowMe and ShareMe: understanding automatically discovered personality traits from social media and user sharing preferences[C]//Proceedings of the CHI Conference on Human Factors in Computing Systems, Toronto, Apr 26-May 1, 2014. New York: ACM, 2014: 955-964.
[2] TAUSCZIK Y R, PENNEBAKER J W. The psychological meaning of words: LIWC and computerized text analysis methods[J]. Journal of Language and Social Psychology, 2010, 29(1): 24-54.
[3] BARRICK M R, MOUNT M K. The big five personality dimensions and job performance: a meta-analysis[J]. Personnel Psychology, 1991, 44(1): 1-26.
[4] FORD J K. Brands laid bare: using market research for evidence-based brand management[M]. Hoboken: John Wiley & Sons, 2005.
[5] HINTON G E, OSINDERO S, TEH Y W. A fast learning algorithm for deep belief nets[J]. Neural Computation, 2006, 18(7): 1527-1554.
[6] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 770-778.
[7] XING J, HEEGER D J. Measurement and modeling of center-surround suppression and enhancement[J]. Vision Research, 2001, 41(5): 571-583.
[8] COLLOBERT R, WESTON J. A unified architecture for natural language processing: deep neural networks with multitask learning[C]//Proceedings of the 25th International Conference on Machine Learning, Helsinki, Jun 5-9, 2008. New York: ACM, 2008: 160-167.
[9] KIM Y. Convolutional neural networks for sentence classification[J]. arXiv:1408.5882, 2014.
[10] LAI S W, XU L H, LIU K, et al. Recurrent convolutional neural networks for text classification[C]//Proceedings of the 29th AAAI Conference on Artificial Intelligence, Austin, Jan 25-30, 2015. Menlo Park: AAAI, 2015: 2267-2273.
[11] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[J]. arXiv:1810.04805, 2018.
[12] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 30th Annual Conference on Neural Information Processing Systems, Long Beach, Dec 4-9, 2017. Red Hook: Curran Associates, 2017: 5998-6008.
[13] YANG Z C, YANG D Y, DYER C, et al. Hierarchical attention networks for document classification[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, Jun 12-17, 2016. Stroudsburg: ACL, 2016: 1480-1489.
[14] WANG B, ZHANG X, ZHOU X, et al. A gated dilated convolution with attention model for clinical cloze-style reading comprehension[J]. International Journal of Environmental Research and Public Health, 2020, 17: 1323.
[15] ADHIKARI A, RAM A, TANG R, et al. Docbert: BERT for document classification[J]. arXiv:1904.08398, 2019.
[16] KWANTES P J, DERBENTSEVA N, LAM Q, et al. Assessing the big five personality traits with latent semantic analysis[J]. Personality and Individual Differences, 2016, 102: 229-233.
[17] WEI H H, ZHANG F Z, YUAN N J, et al. Beyond the words: predicting user personality from heterogeneous information[C]//Proceedings of the 10th ACM International Conference on Web Search and Data Mining, Cambridge, Feb 6-10, 2017. New York: ACM, 2017: 305-314.
[18] MAJUMDER N, PORIA S, GELBUKH A, et al. Deep learning-based document modeling for personality detection from text[J]. IEEE Intelligent Systems, 2017, 32(2): 74-79.
[19] SUN C, QIU X, XU Y, et al. How to fine-tune BERT for text classification?[C]//LNCS 11856: Proceedings of the 18th China National Conference on Chinese Computational Linguistics, Kunming, Oct 18-20, 2019. Berlin, Heidelberg: Springer, 2019: 194-206.
[20] FERNáNDEZ-DELGADO M, CERNADAS E, BARRO S, et al. Do we need hundreds of classifiers to solve real world classification problems?[J]. The Journal of Machine Learning Research, 2014, 15(1): 3133-3181.