基于Transformer-CVAE的三维人体动画生成方法

doi:10.3778/j.issn.1673-9418.2206060

摘要/Abstract

摘要： 三维人体动画生成技术是三维动画领域的核心技术。基于动作捕捉的人体动画生成方法通常制作流程较为繁琐、制作周期较长，无法快速生成人体动画，而现有数据驱动的方法生成的人体动画缺乏真实性，且生成人体运动的种类相对有限。基于此，提出了一种基于Transformer-CVAE的三维人体动画生成方法。首先，基于真实的人体运动构建人体运动数据集，并按照运动种类进行类别划分；其次，基于Transformer网络架构学习运动序列的时序依赖关系，进一步引入变分自编码器结构学习运动序列在隐空间上的概率分布；然后，在隐空间施加约束条件进而控制生成人体运动的效果；最后，在AMASS、HumanACT12、UESTC等数据集上进行实验，并从视觉效果与网络性能两方面对方法进行分析。实验结果表明，与现有方法相比，所提方法可生成种类丰富、真实细腻的人体动画，且在STED、RMSE等指标上具有明显的提升。

关键词: Transformer, 条件变分自编码器, 三维人体动画, 计算机图形学

Abstract: 3D human animation synthesis is a dominant technology in the domain of 3D animation. Traditional workflows depending on motion capture cannot generate human animation quickly due to complicated procedure and long authoring period. Existing data-driven methods have limited learning capability and therefore the gene-rated animations are lack of realism and the categories of the generation are relatively limited. To that end, this paper presents a 3D human animation synthesis method based on a Transformer-based conditional variation auto-encoder (Transformer-CVAE). Firstly, the motion dataset is constructed and classified by the motion category. Then, the temporal relationship between different frames in a common sequence is established by means of the Transformer architecture, and a variational autoencoder is further combined with the Transformer to infer the probabilistic distribution of human motions. Next, to control the desired body motion generated, the constraints are imposed on the latent space. Finally, a series of experiments are conducted on AMASS, HumanACT12 and UESTC datasets and the qualitative and quantitative evaluation is made from two aspects: the visual effect and the performance. Experimental results demonstrate that the method achieves superior performance in the metrics like STED, RMSE, etc. compared with the state-of-art, while capable of synthesizing various human animations with realism.

Key words: Transformer, conditional variational auto-encoder, 3D human animation, computer graphics

冯文科, 石敏, 朱登明, 李兆歆. 基于Transformer-CVAE的三维人体动画生成方法[J]. 计算机科学与探索, 2023, 17(9): 2137-2147.

FENG Wenke, SHI Min, ZHU Dengming, LI Zhaoxin. 3D Human Animation Synthesis with Transformer-CVAE[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(9): 2137-2147.

参考文献

[1] GUO C, ZUO X, WANG S, et al. Action2motion: condi-tioned generation of 3D human motions[C]//Proceedings of the 28th ACM International Conference on Multimedia, Seattle, Oct 12-16, 2020. New York: ACM, 2020: 2021-2029.
[2] CHUNG J, GULCEHRE C, CHO K, et al. Empirical eval-uation of gated recurrent neural networks on sequence modeling[J]. arXiv:1412.3555, 2014.
[3] PETROVICH M, BLACK M J, VAROL G. Action-condi-tioned 3D human motion synthesis with transformer VAE[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 11-17, 2021. Piscataway: IEEE, 2021: 10965-10975.
[4] LI R, YANG S, DAVID A R, et al. AI choreographer: music conditioned 3D dance generation with AIST++[C]//Proce-edings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 11-17, 2021. Piscataway: IEEE, 2021: 13381-13392.
[5] KANAZAWA A, BLACK M J, JACOBS D W, et al. End-to-end recovery of human shape and pose[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-23, 2018. Pisc-ataway: IEEE, 2018: 7122-7131.
[6] KOCABAS M, ATHANASIOU N, BLACK M J. Vibe: video inference for human body pose and shape estimation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 5252-5262.
[7] MAHMOOD N, GHORBANI N, TROJE N F, et al. AMASS: archive of motion capture as surface shapes[C]//Proce-edings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 5441-5450.
[8] LUO Z, GOLESTANEH S A, KITANI K M. 3D human motion estimation via motion compression and refinement[C]//Proceedings of the 15th Asian Conference on Computer Vision, Kyoto, Nov 30-Dec 4, 2020. Cham: Springer, 2020: 324-340.
[9] KINGMA D P, WELLING M. Auto-encoding variational Bayes[J]. arXiv:1312.6114, 2013.
[10] HUA G G, LI L H, LIU S G. Multipath affinage stacked-hourglass networks for human pose estimation[J]. Frontiers of Computer Science, 2020, 14(4): 144701.
[11] LIU S, LI Y, HUA G. Human pose estimation in video via structured space learning and halfway temporal evaluation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 29(7): 2029-2038.
[12] 吉斌, 潘烨, 金小刚, 等. 用于视频流人体姿态估计的时空信息感知网络[J]. 计算机辅助设计与图形学学报, 2022, 34(2): 189-197.
JI B, PAN Y, JIN X G, et al. Spatiotemporal neural network for video-based pose estimation[J]. Journal of Computer-Aided Design and Computer Graphics, 2022, 34(2): 189-197.
[13] BARSOUM E, KENDER J, LIU Z. HP-GAN: probabilistic 3D human motion prediction via GAN[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Pisca-taway: IEEE, 2018: 1499-1508.
[14] DUAN Y, SHI T, ZOU Z, et al. Single-shot motion comp-letion with transformer[J]. arXiv:2103.00776v1, 2021.
[15] ASHISH V, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems 30, Long Beach, Dec 4-9, 2017. Cambridge: MIT Press, 2017: 5999-6009.
[16] ZHANG Y, BLACK M J, TANG S. We are more than our joints: predicting how 3D bodies move[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, Jun 19-25, 2021. Piscataway: IEEE, 2021: 3371-3381.
[17] HYEMIN A, HA T, CHOI Y, et al. Text2action: generative adversarial synthesis from language to action[C]//Proce-edings of the 2018 IEEE International Conference on Robotics and Automation, Brisbane, May 21-25, 2018. Pisc-ataway: IEEE, 2018: 5915-5920.
[18] AHUJA C, MORENCY L P. Language2pose: natural language grounded pose forecasting[C]//Proceedings of the 2019 International Conference on 3D Vision, Queacutebec City, Sep 16-19, 2019. Piscataway: IEEE, 2019: 719-728.
[19] LEE H Y, YANG X, LIU M Y, et al. Dancing to music[C]//Advances in Neural Information Processing Systems 32, Vancouver, Dec 8-14, 2019. Cambridge: MIT Press, 2019: 1-11.
[20] LI J, YIN Y, CHU H, et al. Learning to generate diverse dance motions with transformer[J]. arXiv:2008.08171, 2020.
[21] LOPER M, MAHMOOD N, ROMERO J, et al. SMPL: a skinned multi-person linear model[J]. ACM Transactions on Graphics, 2015, 34(6): 1-16.
[22] ZHOU Y, BARNES C, LU J, et al. On the continuity of rotation representations in neural networks[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Pisc-ataway: IEEE, 2019: 5738-5746.
[23] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of North American Chapter of the Association for Compu-tational Linguistics, Minneapolis, Jun 2-7, 2019. Cambridge: MIT Press, 2019: 4171-4186.
[24] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recog-nition at scale[J]. arXiv:2010.11929, 2020.
[25] LE F, ZENG T, LIU C, et al. Transformer-based conditional variational autoencoder for controllable story generation[J]. arXiv:2101.00828, 2021.
[26] VASA L, SKALA V. A perception correlated comparison method for dynamic meshes[J]. IEEE Transactions on Visualization and Computer Graphics, 2010, 17(2): 220-230.
[27] JI Y, XU F, YANG Y, et al. A large-scale RGB-D database for arbitrary-view human action recognition[C]//Proceedings of the 2018 ACM International Conference on Multimedia, Seoul, Oct 22-26, 2018. New York: ACM, 2018: 1510-1518.