Short Video Behavior Recognition Combining Scene and Behavior Features

doi:10.3778/j.issn.1673-9418.1909044

Abstract

Abstract:

The behavior recognition method pays more attention to the action itself, but the short video contains less information. And it is necessary to utilize various feature information in the video as much as possible to improve the accuracy of behavioral recognition. Therefore, the short video behavior recognition method based on scene and behavior joint features is studied, and the scene information is used as context information to improve the effect of traditional single behavior recognition network. First, the scene features in the short video are extracted using a deep fusion network. Then, the behavioral features in the short video utilize the variable convolutional network for RGB features and flow features extraction. Finally, the dictionary learning method is used to sparsely represent the joint features, and more explanatory feature information is extracted for short video behavior recognition. The top-5 accuracy rate in the Charades test set is 33%. It is superior to the traditional single behavior recognition network, making the behavior recognition effect more accurate.

Key words: scene recognition, action recognition, dictionary learning, deep learning, video understanding

摘要：

目前行为识别方法更关注动作本身，但短视频中包含的信息比较少，需要利用视频中的多种特征信息，提高任务行为识别的准确率。因此，对基于场景和行为联合特征的短视频行为识别方法进行了研究，利用场景信息作为上下文信息，提高传统单一行为识别网络的效果。首先对短视频中的场景特征利用深度融合网络进行提取；然后对短视频中的行为特征利用可变卷积网络进行RGB特征和Flow特征提取；最后利用字典学习的方法对构建的联合特征进行稀疏表示，提取出更具解释性的特征信息。在Charades测试集top-5准确率为33%，优于传统单一行为识别网络，使行为识别效果更加准确。

关键词: 场景识别, 行为识别, 字典学习, 深度学习, 视频理解

DONG Xu, TAN Li, ZHOU Lina, SONG Yanyan. Short Video Behavior Recognition Combining Scene and Behavior Features[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(10): 1754-1761.

董旭，谭励，周丽娜，宋艳艳. 联合场景和行为特征的短视频行为识别[J]. 计算机科学与探索, 2020, 14(10): 1754-1761.

References

[1] Chen J Y, Song X M, Nie L Q, et al. Micro tells macro: predic-ting the popularity of micro-videos via a transductive model[C]//Proceedings of the 2016 ACM Conference on Multi-media Conference, Amsterdam, Oct 15-19, 2016. New York: ACM, 2016: 898-907.
[2] Liu M, Nie L Q, Wang X, et al. Online data organizer: micro-video categorization by structure-guided multimodal dictio-nary learning[J]. IEEE Transactions on Image Processing, 2019, 28(3): 1235-1247.
[3] Zhang J L, Nie L Q, Wang X, et al. Shorter-is-better: venue category estimation from micro-video[C]//Proceedings of the 2016 ACM Conference on Multimedia Conference, Ams-terdam, Oct 15-19, 2016. New York: ACM, 2016: 1415-1424.
[4] Liu J, Wang G, Duan L Y, et al. Skeleton based human action recognition with global context-aware attention LSTM net-works[J]. IEEE Transactions on Image Processing, 2018, 27(4): 1586-1599.
[5] Yu K, Yun F. Bilinear heterogeneous information machine for RGB-D action recognition[J]. International Journal of Com-puter Vision, 2017, 123(3): 350-371.
[6] Liu A A, Xu N, Nie W Z, et al. Benchmarking a multimodal and multiview and interactive dataset for human action recognition[J]. IEEE Transactions on Cybernetics, 2017, 47(7): 1781-1794.
[7] Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos[C]//Proceedings of the 2014 Annual Conference on Neural Information Processing Sys-tems, Montreal, Dec 8-13, 2014. Red Hook: Curran Associates, 2014: 568-576.
[8] Karpathy A, Toderici G, Shetty S, et al. Large-scale video classification with convolutional neural networks[C]//Pro-ceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Jun 23-28, 2014. Wash-ington: IEEE Computer Society, 2014: 1725-1732.
[9] Ji S W, Xu W, Yang M, et al. 3D convolutional neural net-works for human action recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(1): 221-231.
[10] Wang X H, Gao L L, Song J K, et al. Beyond frame-level CNN: saliency-aware 3D CNN with LSTM for video action recognition[J]. IEEE Signal Processing Letters, 2017, 24(4): 510-514.
[11] Peng B, Lei J J, Fu H Z, et al. Unsupervised video action clustering via motion-scene interaction constraint[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(1): 131-144.
[12] Zhang H B, Lei Q, Chen D S, et al. Probability-based method for Boosting human action recognition using scene context[J]. IET Computer Vision, 2016, 10(6): 528-536.
[13] Yang J, Shi Z K, Wu Z Y. Vision-based action recognition of construction workers using dense trajectories[J]. Advanced Engineering Informatics, 2016, 30(3): 327-336.
[14] Hou J Y, Wu X X, Sun Y C, et al. Content-attention rep-resentation by factorized action-scene network for action re-cognition[J]. IEEE Transactions on Multimedia, 2018, 20(6): 1537-1547.
[15] Dai J F, Qi H Z, Xiong Y W, et al. Deformable convolu-tional networks[C]//Proceedings of the 2017 IEEE Interna-tional Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 764-773.
[16] Sigurdsson G A, Russakovsky O, Gupta A. What actions are needed for understanding human actions in videos?[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 2156-2165.