预加权调制密集图卷积网络三维人体姿态估计

doi:10.3778/j.issn.1673-9418.2302065

摘要/Abstract

摘要： 图卷积网络（GCN）日益成为三维人体姿态估计（3D HPE）的主要研究热点之一，使用GCN对人体关节点之间的关系建模的方法使三维人体姿态估计获得了良好的性能。然而，基于GCN的三维人体姿态估计方法存在过平滑和未区分关节点与相邻关节点重要性的问题。为解决这些问题，设计了调制密集连接模块（MDC）和预加权图卷积模块，并基于这两个模块提出了预加权调制密集图卷积网络的三维人体姿态估计方法（WMDGCN）。针对过平滑问题，调制密集连接通过超参数[α]和[β]更好地实现特征重用（超参数[α]表示第[l]层和之前各层总特征的权重比例，超参数[β]表示之前各层特征到第[l]层的传播策略），从而有效地提高特征的表达能力。针对未区分关节点与相邻关节点重要性的问题，使用预加权图卷积为当前关节点赋予更高的权重，并对当前关节点及其相邻关节点使用不同的权重矩阵，更有效地捕获人体关节点特征。Human3.6M数据集上的对比实验结果表明，该方法在参数量和性能上均取得了最佳性能，WMDGCN的参数量、MPJPE和P-MPJPE值分别为0.27 MB、37.46 mm和28.85 mm。

关键词: 三维人体姿态估计, 图卷积网络, 预加权

Abstract: Graph convolutional networks (GCN) have increasingly become one of the main research hotspots in 3D human pose estimation. The method of modeling the relationship between human joint points by GCN has achieved good performance in 3D human pose estimation. However, the 3D human pose estimation method based on GCN has issues of over-smooth and indistinguishable importance between joint points and adjacent joint points. To address these issues, this paper designs a modulated dense connection (MDC) module and a pre-weighted graph convolutional module, and proposes a pre-weighted modulated dense graph convolutional network (WMDGCN) for 3D human pose estimation based on these two modules. For the problem of over-smoothing, the modulation dense connection can better realize feature reuse through hyperparameter [α] and [β] (hyperparameter [α] represents the weight proportion of features of layer L to previous layers, and hyperparameter [β] represents the propagation strategies of the features of previous layers to layer L), thus effectively improving the expression ability of features. To address the issue of not distinguishing the importance of the joint points and adjacent joint points, the pre-weighted graph convolution is used to assign higher weights to the joint point. Different weight matrices are used for the joint point and its adjacent joint points to capture human joint point features more effectively. Comparative experimental results on the Human3.6M dataset show that the proposed method achieves the best performance in terms of parameter number and performance. The parameter number, MPJPE and P-MPJPE values of WMDGCN are 0.27 MB, 37.46 mm and 28.85 mm, respectively.

Key words: 3D human pose estimation, graph convolution network, pre-weighted

马金林, 崔琦磊, 马自萍, 闫琦, 曹浩杰, 武江涛. 预加权调制密集图卷积网络三维人体姿态估计[J]. 计算机科学与探索, 2024, 18(4): 963-977.

MA Jinlin, CUI Qilei, MA Ziping, YAN Qi, CAO Haojie, WU Jiangtao. Pre-weighted Modulated Dense Graph Convolutional Networks for 3D Human Pose Estimation[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(4): 963-977.

参考文献

[1] WENG J, WENG C, YUAN J. Spatio-temporal naive-Bayes nearest-neighbor (ST-NBNN) for skeleton-based action recognition[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 4171-4180.
[2] YAN S, XIONG Y, LIN D. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence, New Orleans, Feb 2-7, 2018. Menlo Park: AAAI, 2018: 7444-7452.
[3] 邓淼磊, 高振东, 李磊, 等. 基于深度学习的人体行为识别综述[J]. 计算机工程与应用, 2022, 58(13): 14-26.
DENG M L, GAO Z D, LI L, et al. Overview of human behavior recognition based on deep learning[J]. Computer Engineering and Applications, 2022, 58(13): 14-26.
[4] 何坚, 郭泽龙, 刘乐园, 等. 基于滑动窗口和卷积神经网络的可穿戴人体活动识别技术[J]. 电子与信息学报, 2022, 44(1): 168-177.
HE J, GUO Z L, LIU L Y, et al. Human activity recognition technology based on sliding window and convolutional neural network[J]. Journal of Electronics & Information Technology, 2022, 44(1): 168-177.
[5] GAN J, WANG W. In air handwritten English word recognition using attention recurrent translator[J]. Neural Computing and Applications, 2019, 31(7): 3155-3172.
[6] HAGBI N, BERGIG O, BILLINGHURST M, et al. Shape recognition and pose estimation for mobile augmented reality[J]. Proceedings of the IEEE, 2011, 17(10): 1369-1379.
[7] HASSANIN M, KHAN S, TAHTALI M. Visual affordance and function understanding: a survey[J]. ACM Computing Surveys, 2021, 54(3): 1-35.
[8] GARCIA-SALGUERO M, GONZALEZ-JIMENEZ J, MORENO F A. Human 3D pose estimation with a tilting camera for social mobile robot interaction[J]. Sensors, 2019, 19(22): 4943.
[9] GUI L Y, ZHANG K, WANG Y X, et al. Teaching robots to predict human motion[C]//Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, Madrid, Oct 1-5, 2018. Piscataway: IEEE, 2018: 562-567.
[10] AGARWAL A, TRIGGS B. Recovering 3D human pose from monocular images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 28(1): 44-58.
[11] IONESCU C, LI F, SMINCHISESCU C. Latent structured models for human pose estimation[C]//Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Nov 6-13, 2011.Washington: IEEE Computer Society, 2011: 2220-2227.
[12] ROGEZ G, RIHAN J, RAMALINGAM S, et al. Randomized trees for human pose detection[C]//Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, Jun 24-26, 2008. Washington: IEEE Computer Society, 2008: 1-8.
[13] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553): 436-444.
[14] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Image-Net classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems 25, Lake Tahoe, Dec 3-6, 2012: 1106-1114.
[15] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
[16] TEKIN B, KATIRCIOGLU I, SALZMANN M, et al. Structured prediction of 3D human pose with deep neural net-works[C]//Proceedings of the British Machine Vision Conference 2016, York, Sep 19-22, 2016: 1-11.
[17] PAVLAKOS G, ZHOU X, DERPANIS K G, et al. Coarse-to-fine volumetric prediction for single-image 3D human pose[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 7025-7034.
[18] SUN X, SHANG J, LIANG S, et al. Compositional human pose regression[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 2602-2611.
[19] SUN X, XIAO B, WEI F, et al. Integral human pose regression[C]//Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 529-545.
[20] YANG W, OUYANG W, WANG X, et al. 3D human pose estimation in the wild by adversarial learning[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 5255-5264.
[21] ZHOU X, HUANG Q, SUN X, et al. Towards 3D human pose estimation in the wild: a weakly-supervised approach[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington:IEEE Computer Society, 2017: 398-407.
[22] MARTINEZ J, HOSSAIN R, ROMERO J, et al. A simple yet effective baseline for 3D human pose estimation[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 2640-2649.
[23] KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[C]//Proceedings of the 2017 International Conference on Learning Representations, Toulon, Apr 24-26, 2017: 1-14.
[24] FANG H S, XU Y, WANG W, et al. Learning pose grammar to encode human body configuration for 3D pose estimation[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence, New Orleans, Feb 2-7, 2018. Menlo Park: AAAI, 2018: 6821-6828.
[25] PAVLAKOS G, ZHOU X, DANIILIDIS K. Ordinal depth supervision for 3D human pose estimation[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 7307-7316.
[26] SHARMA S, VARIGONDA P T, BINDAL P, et al. Monocular 3D human pose estimation by generation and ordinal ranking[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 2325-2334.
[27] ZHAO L, PENG X, TIAN Y, et al. Semantic graph convolutional networks for 3D human pose regression[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 3425-3435.
[28] ZOU Z, LIU K, WANG L, et al. High-order graph convolutional networks for 3D human pose estimation[C]//Procee-dings of the 31st British Machine Vision Conference, Sep 7-10, 2020: 1-13.
[29] XU Y, WANG W, LIU T, et al. Monocular 3D pose estimation via pose grammar and data augmentation[J]. IEEE Tran-sactions on Pattern Analysis and Machine Intelligence, 2022, 44(10): 6327-6344.
[30] BANIK S, GARCíA A M, KNOLL A. 3D human pose regression using graph convolutional network[C]//Proceedings of the 2021 IEEE International Conference on Image Processing, Anchorage, Sep 19-22, 2021. Piscataway: IEEE, 2021: 924-928.
[31] QUAN J, HAMZA A B. Higher-order implicit fairing net-works for 3D human pose estimation[C]//Proceedings of the 32nd British Machine Vision Conference, Nov 22-25, 2021: 352.
[32] LI Q, HAN Z, WU X M. Deeper insights into graph convolutional networks for semi-supervised learning[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence, New Orleans, Feb 2-7, 2018. Menlo Park: AAAI, 2018: 3538-3545.
[33] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 770-778.
[34] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 2261-2269.
[35] ABU-EL-HAIJA S, PEROZZI B, KAPOOR A, et al. MixHop: higher-order graph convolutional architectures via sparsified neighborhood mixing[C]//Proceedings of the 36th International Conference on Machine Learning, Long Beach, Jun 9-15, 2019: 21-29.
[36] LIU K, ZOU Z, TANG W. Learning global pose features in graph convolutional networks for 3D human pose estimation[C]//Proceedings of the 2020 Asian Conference on Computer Vision, Kyoto, Nov 30-Dec 4, 2020. Cham: Spr-inger， 2020: 89-105.
[37] ZHAO W, TIAN Y, YE Q, et al. GraFormer: graph convolution transformer for 3D pose estimation[J]. arXiv:2109.08364, 2021.
[38] 邓辉, 徐杨. 融入注意力和密集连接的轻量型人体姿态估计[J]. 计算机工程与应用, 2022, 58(16): 265-273.
DENG H, XU Y. Lightweight human pose estimation based on attention and dense connection[J]. Computer Engineering and Applications, 2022, 58(16): 265-273.
[39] IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the 2015 International Conference on Machine Learning, Lille, Jul 6-11, 2015: 448-456.
[40] NAIR V, HINTON G E. Rectified linear units improve restricted Boltzmann machines[C]//Proceedings of the 27th International Conference on Machine Learning, Haifa, Jun 21-24, 2010. Madison: Omnipress, 2010: 807-814.
[41] IONESCU C, PAPAVA D, OLARU V, et al. Human3.6M: large scale datasets and predictive methods for 3D human sensing in natural environments[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 36(7): 1325-1339.
[42] MEHTA D, RHODIN H, CASAS D, et al. Monocular 3D human pose estimation in the wild using improved CNN supervision[C]//Proceedings of the 2017 International Conference on 3D Vision, Qingdao, Oct 10-12, 2017. Piscataway: IEEE, 2017: 506-516.
[43] KINGMA D, BA J. Adam: a method for stochastic optimization[C]//Proceedings of the 2015 International Conference on Learning Representations, San Diego, May 7-9, 2015: 1-15.
[44] CI H, WANG C, MA X, et al. Optimizing network structure for 3D human pose estimation[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 2262-2271.
[45] LI C, LEE G H. Generating multiple hypotheses for 3D human pose estimation with mixture density network[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 9887-9895.
[46] AZIZI N, POSSEGGER H, RODOLà E, et al. 3D human pose estimation using M?bius graph convolutional networks[C]//Proceedings of the 17th European Conference on Com-puter Vision, Tel Aviv, Oct 23-27, 2022. Cham: Springer, 2022: 160-178.
[47] LUO C, CHU X, YUILLE A. OriNet: a fully convolutional network for 3D human pose estimation[C]//Proceedings of the British Machine Vision Conference 2018, Newcastle, Sep 3-6, 2018: 92.
[48] ZENG A, SUN X, HUANG F, et al. SRNet: improving generalization in 3D human pose estimation with a split-and-recombine approach[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 507-523.
[49] WANG J, HUANG S, WANG X, et al. PONet: robust 3D human pose estimation via learning orientations only[J]. arXiv:2112.11153, 2021.
[50] WANDT B, RUDOLPH M, ZELL P, et al. Canonpose: self-supervised monocular 3D human pose estimation in the wild[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 19-25, 2021. Piscataway: IEEE, 2021: 13294-13304.
[51] ROY S K, CITRARO L, HONARI S, et al. On triangulation as a form of self-supervision for 3D human pose estimation[C]//Proceedings of the 10th International Conference on 3D Vision 2022, Prague, Sep 12-16, 2022. Piscataway: IEEE, 2022: 1-10.