Spatial Multiple-Temporal Graph Convolutional Neural Network for Human Action Recognition

doi:10.3778/j.issn.1673-9418.2106102

Abstract

Abstract: Spatial-temporal graph convolutional neural network (ST-GCN) based on skeleton data for human action recognition has the problem of monotonous and fixed temporal convolution architecture, which is difficult to cover all the stages of each action execution completely. In respect of the issues above, a novel multiple-temporal graph convolution layer with multiple convolution kernels and multiple architectures is proposed. Spatial multiple-temporal graph convolutional neural network (SMT-GCN) is built using different temporal graph convolution operations to extract and fuse temporal trajectory features of different scales. Further, a Transformer-Resnet (Tran-Res) module and a light attention module (convolutional block attention module，CBAM) are embedded into the SMT-GCN to enhance the human long-range connecting information and the spatial structual features, and spatial-attentive multiple-temporal graph convolutional neural network (SAMT-GCN) is built. Experments are performed on NTU RGB+D dataset and HDM05 dataset. The human action recognition accuracies are improved by both the proposed SMT-GCN and SAMT-GCN. In addition, the proposed multiple-temporal graph convolution modules are aggregated into other baselines to improve their performance. Moreover, an ablation study is performed for exploring the influence of multiple kernel sizes and architectures on the proposed algorithm. Experimental results show that the SAMT-GCN gets the best performance with convolution kernel size 1,5,9, and the accuracies of networks with densely architecture are better than those with serial and parallel architectures.

Key words: human action recognition, spatial-temporal graph convolutional neural network (ST-GCN), multiple- temporal graph convolution, Transformer-Resnet (Tran-Res), light attention module

摘要： 基于骨骼数据的时空图卷积人体动作识别网络（ST-GCN）存在时间卷积层结构单一、固定的问题，难以全面提取每个动作类别所需的全部重要阶段特征。针对这一问题，提出了包含多个不同尺度卷积核和多种结构的时间图卷积层，构造了多尺度时空图卷积网络（SMT-GCN），利用不同的时间图卷积操作抽取并融合不同尺度的时间轨迹特征。同时，为了强化人体长距离关联信息和空间结构化特征，在SMT-GCN中融合了构造的变换残差模块（Tran-Res）和轻量级注意力模块（CBAM），构造了多尺度时空图注意卷积网络（SAMT-GCN）。实验在NTU RGB+D数据集和HDM05数据集上进行，提出的SMT-GCN和SAMT-GCN均获得了识别精度的提升；另外，设计的多尺度时间图卷积模块可以融合于其他基线网络中并提高性能。为探究卷积核尺度及结构对算法的影响，设计了相应消融实验，实验结果表明卷积核大小为1、5、9的SAMT-GCN性能最优，并且具有稠密结构的网络识别精度要高于具有串行和并行结构的网络。

关键词: 人体动作识别, 时空图卷积网络（ST-GCN）, 多尺度时间图卷积, 变换残差模块（Tran-Res）, 轻量级注意力

ZHAO Dengge, ZHI Min. Spatial Multiple-Temporal Graph Convolutional Neural Network for Human Action Recognition[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(3): 719-732.

赵登阁, 智敏. 用于人体动作识别的多尺度时空图卷积算法[J]. 计算机科学与探索, 2023, 17(3): 719-732.

References

[1] STROUD J, ROSS D, SUN C, et al. D3D: distilled 3D networks for video action recognition[C]//Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision, Snowmass Village, Mar 1-5, 2020. Piscataway: IEEE, 2020: 614-623.
[2] LI C, ZHONG Q Y, XIE D, et al. Collaborative spatiotemporal feature learning for video action recognition[C]//Proceeding of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 7872-7881.
[3] YANG H, YUAN C F, LI B, et al. Asymmetric 3D convolutional neural networks for action recognition[J]. Pattern Recognition, 2019, 85: 1-12.
[4] PAUL S, ALI S, SHAH M. A 3-dimensional SIFT descriptor and its application to action recognition[C]//Proceedings of the 15th International Conference on Multimedia 2007, Augsburg, Sep 24-29, 2007. New York: ACM, 2007: 357-360.
[5] KL?SER A, MARSZALEK M, SCHMID C. A spatio-temporal descriptor based on 3D-gradients[C]//Proceedings of the British Machine Vision Conference 2008, Leeds, Sep 2008. Durham: BMVA, 2008: 1-10.
[6] LAPTEV I, MARSZALEK M, SCHMID C, et al. Learning realistic human actions from movies[C]//Proceeding of the 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Anchorage, Jun 24-26, 2008. Washington: IEEE Computer Society, 2008: 1-8.
[7] RAVANBAKHSH M, MOUSAVI H, RASTEGARI M, et al. Action recognition with image based CNN features[J].arXiv:1512.03980, 2015.
[8] LEV G, SADEH G, KLEIN B. RNN Fisher vectors for action recognition and image annotation[C]//LNCS 9910: Proceeding of the 14th European Conference on Computer Vision, Amsterdam, Oct 11-14, 2016. Cham: Springer, 2016: 833-850.
[9] ZHOU J, CUI G Q, ZHANG Z Y, et al. Graph neural networks:a review of methods and applications[J]. AI Open, 2020, 1: 57-81.
[10] KIPF T, WELLING M. Semi-supervised classification with graph convolutional networks[J]. arXiv:1609.02907, 2016.
[11] YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recogntion[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence, the 30th Innovative Applications of Artificial Intelligence, and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, Feb 2-7, 2018. Menlo Park: AAAI, 2018: 7444-7452.
[12] THAKKAR K, NARAYANAN P J. Part-based graph con-volutional network for action recognition[J]. arXiv:1809. 04983, 2018.
[13] ZHANG X K, XU C, TIAN X M, et al. Graph edge convolutional neural networks for skeleton-based action recognition[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(8): 3047-3060.
[14] CHENG K, ZHANG Y F, HE X Y, et al. Skeleton-based action recognition with shift graph convolutional network[C]//Proceeding of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 180-189.
[15] LIU Z Y, ZHANG H W, CHEN Z H, et al. Disentangling and unifying graph convolutions for skeleton-based action recognition[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 140-149.
[16] ZHANG X K, XU C, TAO D C. Context aware graph convolution for skeleton-based action recognition[C]//Pro-ceeding of the 2020 Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 1432-14330.
[17] 马利, 郑诗雨, 牛斌. 应用区域关联自适应图卷积的动作识别方法[J]. 计算机科学与探索, 2022, 16(4): 898-908.
MA L, ZHENG S Y, NIU B. Action recognition method on regional association adaptive graph convolution[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 898-908.
[18] SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//Proceeding of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 7-12, 2015. Washington: IEEE Computer Society, 2015: 1-9.
[19] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409. 1556, 2014.
[20] 李龙. 融合注意力机制的人体骨骼点动作识别方法研究[D]. 成都: 成都理工大学, 2019.
LI L. Research on the action recognition method of human skeletal point by integrating attention mechanism[D].Chengdu: Chengdu University Technology, 2019.
[21] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceeding of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 2261-2269.
[22] DOSOVITSKIY A, BEYER L, KOLSENIKOV A, et al. An image is worth [16×16] words: transformers for image recognition at scale[C]//Proceeding of the 9th International Conference on Learning Representations, Austria, May 3-7, 2021: 1-21.
[23] WOO S, PARK J, LEE J Y. CBAM: convolutional block attention module[C]//LNCS 11211: Proceeding of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 3-19.
[24] 刘芳, 乔建忠, 代钦, 等. 基于双流多关系GCNs的骨架动作识别方法[J]. 东北大学学报(自然科学版), 2021, 42(6): 768-774.
LIU F, QIAO J Z, DAI Q, et al. Skeleton-based action recognition method with two-stream multi-relational GCNs[J]. Journal of Northeastern University (Natural Science), 2021, 42(6): 768-774.
[25] SHAHROUDY A, LIU J, NG T T, et al. NTU RGB+D: a large scale dataset for 3D human activity analysis[C]//Pro-ceeding of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washing-ton: IEEE Computer Society, 2016: 1010-1019.
[26] MULLER M, RODER T, CLAUSEN M, et al. Documentation mocap database HDM05[R]. Computer Graphics Technical Reports, 2007: 2.
[27] SHI L, ZHANG Y F, CHENG J, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]//Proceeding of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 12026-12035.
[28] LI M S, CHEN S H, CHEN X, et al. Actional-structural graph convolutional networks for skeleton-based action recognition[C]//Proceeding of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 3595-3603.
[29] 王志华. 基于时空图卷积神经网络的人体动作识别[D]. 成都: 电子科技大学, 2020.
WANG Z H. Research on human action recognition based on spatio-temporal graph convolutional neural network[D]. Chengdu: University of Electronic Science and Technology of China, 2020.