多尺度全局自适应注意力图神经网络

doi:10.3778/j.issn.1673-9418.2208039

摘要/Abstract

摘要： 针对动态多尺度图神经网络的编解码网络中存在的身体部位内部关节点间关联度不高和感受野受限制导致运动预测误差偏高的问题，提出了一种用于人体运动预测的多尺度全局自适应注意力图神经网络，降低运动预测误差。提出了一种划分骨架关节点的多距离分区策略，用于提高身体部位关节点信息在时间和空间上的关联程度；提出了全局自适应注意力时空卷积神经网络，以动态地加强网络对某一动作有贡献的时空关节点的关注度；将上述两处改进集成到图卷积神经网络门控循环单元中，以增强解码网络的状态传播性能，并降低预测误差。实验表明，与最新方法相比，该方法在Human 3.6M、CMU Mocap和3DPW数据集上的预测误差都有所下降。

关键词: 运动预测, 多距离分区策略, 全局自适应注意力, 时空图卷积神经网络, 门控循环单元

Abstract: Dynamic multiscale graph neural networks have high motion prediction errors due to the low correlation between the internal joints of body parts and the limited perceptual fields. A multiscale global adaptive attention graph neural network for human motion prediction is proposed to reduce motion prediction errors. Firstly, a multi-distance partitioning strategy for dividing skeleton joint is proposed to improve the degree of temporal and spatial correlation of body joint information. Secondly, a global adaptive attention spatial temporal graph convolutional network is designed to dynamically enhance the network??s attention to the spatial temporal joints contributing to a motion in combination with global adaptive attention. Finally, this paper integrates the above two improvements into the graph convolutional neural network gate recurrent unit to enhance the state propagation performance of the decoding network and reduce prediction errors. Experimental results show that the prediction error of the proposed method is decreased on Human 3.6M dataset, CMU Mocap dataset and 3DPW dataset compared with state-of-the-art methods.

Key words: motion prediction, multi-distance partitioning strategy, global adaptive attention, spatial temporal graph convolution neural network, gated recurrent unit

苟茹茹, 杨文柱, 罗梓菲, 原云峰. 多尺度全局自适应注意力图神经网络[J]. 计算机科学与探索, 2023, 17(12): 3039-3051.

GOU Ruru, YANG Wenzhu, LUO Zifei, YUAN Yunfeng. Multiscale Global Adaptive Attention Graph Neural Network[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(12): 3039-3051.

参考文献

[1] PADEN B, ?AP M, YONG S Z, et al. A survey of motion planning and control techniques for self-driving urban vehicles[J]. IEEE Transactions on Intelligent Vehicles, 2016, 1(1): 33-55.
[2] SANG H F, CHEN Z Z, HE D K. Human motion prediction based on attention mechanism[J]. Multimediatools and Applications, 2020, 79(9): 5529-5544.
[3] YANG H, YUAN C, ZHANG L, et al. STA-CNN: convolutional spatial-temporal attention learning for action recognition[J]. IEEE Transactions on Image Processing, 2020, 29: 5783-5793.
[4] 邓辉, 徐杨. 融入注意力和密集连接的轻量型人体姿态估计[J]. 计算机工程与应用, 2022, 58(16): 265-273.
DENG H, XU Y. Lightweight human pose estimation based on attention and dense connection[J]. Computer Engineering and Applications, 2022, 58(16): 265-273.
[5] GUI L Y, WANG Y X, LIANG X, et al. Adversarial geometry-aware human motion prediction[C]//LNCS 11208: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 823-842.
[6] KUNDU J N, GOR M, BABU R V. BiHMP-GAN: bidirectional 3D human motion prediction GAN[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence, the 31st Innovative Applications of Artificial Intelligence Conference, the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, Honolulu, Jan 27-Feb 1, 2019. Menlo Park: AAAI, 2019: 8553-8560.
[7] DEFFERRARD M, BRESSON X, VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering[C]//Advances in Neural Information Processing Systems 29, Barcelona, Dec 5-10, 2016: 3844-3852.
[8] GUO X, CHOI J. Human motion prediction via learning local structure representations and temporal dependencies[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence, the 31st Innovative Applications of Artificial Intelligence Conference, the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, Honolulu, Jan 27-Feb 1, 2019. Menlo Park: AAAI, 2019: 2580-2587.
[9] MAO W, LIU M, SALZMANN M, et al. Learning trajectory dependencies for human motion prediction[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 9489-9497.
[10] MAO W, LIU M, SALZMANN M. History repeats itself: human motion prediction via motion attention[C]//LNCS 12359: Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 474-489.
[11] ZHOU H, GUO C, ZHANG H, et al. Learning multiscale correlations for human motion prediction[C]//Proceedings of the 2021 IEEE International Conference on Development and Learning, Beijing, Aug 23-26, 2021. Piscataway: IEEE, 2021: 1-7.
[12] MAO W, LIU M, SALZMANN M, et al. Multi-level motion attention for human motion prediction[J]. International Journal of Computer Vision, 2021, 129(9): 2513-2535.
[13] MAO W, LIU M, SALZMANN M. Generating smooth pose sequences for diverse human motion prediction[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021: 13289-13298.
[14] WANG H, HO E S L, SHUM H P H, et al. Spatio-temporal manifold learning for human motions via long-horizon modeling[J]. IEEE Transactions on Visualization and Computer Graphics, 2019, 27 (1): 216-227.
[15] LI M, CHEN S, ZHAO Y, et al. Dynamic multiscale graph neural networks for 3D skeleton based human motion prediction[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 214-223.
[16] DANG L, NIE Y, LONG C, et al. MSR-GCN: multi-scale residual graph convolution networks for human motion prediction[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021: 11447-11456.
[17] LI M, CHEN S, ZHAO Y, et al. Multiscale spatio-temporal graph neural networks for 3D skeleton-based motion prediction[J]. IEEE Transactions on Image Processing, 2021, 30: 7760-7775.
[18] YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence, the 30th Innovative Applications of Artificial Intelligence, and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, Feb 2-7, 2018. Menlo Park: AAAI, 2018: 7444-7452.
[19] 严春满, 王铖. 卷积神经网络模型发展及应用[J]. 计算机科学与探索, 2021, 15(1): 27-46.
YAN C M, WANG C. Development and application of convolutional neural network model[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(1): 27-46.
[20] 邓淼磊, 高振东, 李磊, 等. 基于深度学习的人体行为识别综述[J]. 计算机工程与应用, 2022, 58(13): 14-26.
DENG M L, GAO Z D, LI L, et al. Overview of human behavior recognition based on deep learning[J]. Computer Engineering and Applications, 2022, 58(13): 14-26.
[21] 何坚, 郭泽龙, 刘乐园, 等. 基于滑动窗口和卷积神经网络的可穿戴人体活动识别技术[J]. 电子与信息学报, 2022, 44(1): 168-177.
HE J, GUO Z L, LI L Y, et al. Human activity recognition technology based on sliding windowand convolutional neural network[J]. Journal of Electronics & Information Technology, 2022, 44(1): 168-177.
[22] WANG X L, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 7794-7803.
[23] 梁延禹, 李金宝. 多尺度非局部注意力网络的小目标检测算法[J]. 计算机科学与探索, 2020, 14(10): 1744-1753.
LIANG Y Y, LI J B. Small objects detection method based on multi-scale non-local attention network[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(10): 1744-1753.
[24] 盖杉, 王俊生. 基于深度学习的非局部注意力增强网络图像去雨算法研究[J]. 电子学报, 2020, 48(10): 1899-1908.
GAI S, WANG J S. Image raindrop algorithm research using nonlocal attention enhanced network based on deep learning[J]. Journal of Electronics & Information Technology, 2020, 48(10): 1899-1908.
[25] WANG B, ADELI E, CHIU H, et al. Imitation learning for human pose prediction[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 7124-7133.
[26] BUADES A, COLL B, MOREL J M. A non-local algorithm for image denoising[C]//Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, Jun 20-26, 2005. Washington: IEEE Computer Society, 2005: 60-65.
[27] VON MARCARD T, HENSCHEL R, BLACK M J, et al. Recovering accurate 3D human pose in the wild using IMUs and a moving camera[C]//LNCS 11214: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 614-631.
[28] MARTINEZ J, BLACK M J, ROMERO J. On human motion prediction using recurrent neural networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 2891-2900.
[29] LI C, ZHANG Z, LEE W S, et al. Convolutional sequence to sequence model for human dynamics[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 5226-5234.
[30] LI M, CHEN S, CHEN X, et al. Symbiotic graph neural networks for 3D skeleton-based human action recognition and motion prediction[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(6): 3316-3333.