Pedestrian Trajectory Prediction Based on Transformer and Multi-relation Graph Convolutional Networks

doi:10.3778/j.issn.1673-9418.2405004

Abstract

Abstract: In the field of autonomous navigation, pedestrian trajectories are relatively complex, and accurately predicting pedestrian trajectories is crucial for ensuring safe travel and autonomous driving. Pedestrian trajectories are highly random, dynamic, and influenced by their surroundings, necessitating the effective modeling of their temporal and spatial interactions. To address this, a pedestrian trajectory prediction model combining Transformer and multi-relation graph convolutional network (GCN) is proposed. The model is composed of interaction capture module, anchor control module, and trajectory refinement module. The interaction capture module extracts motion features of each pedestrian on temporal and spatial sequences by using T-Transformer and GCN, while the anchor control module reduces errors by inferring intermediate destinations. The trajectory refinement module enhances predictions. Adding the inverse relationship when extracting features can obtain more optimized results, and using Gaussian pruning to reduce the generation of false paths can also improve the efficiency of the model. Experimental results on ETH and UCY datasets show superior performance in average displacement error (ADE) and final displacement error (FDE) compared with mainstream models. The excellent performance of the model on pedestrian trajectory prediction minimizes unnecessary trajectory changes and collision risks, offering a promising solution for pedestrian trajectory prediction applications.

Key words: T-Transformer, graph convolutional network (GCN), anchor control, pedestrian trajectory prediction

摘要： 在自动导航应用领域，行人轨迹相对复杂，有效且合理地预测行人未来轨迹对自动驾驶和出行安全至关重要。行人轨迹随机性和动态性极高且与交通环境有着复杂相互作用，因此需要对行人的时间依赖性和空间相互作用进行合理建模。为了解决该问题，提出了一种基于Transformer和多关系图卷积网络（GCN）的行人轨迹预测模型。该模型由交互捕获模块、锚点控制模块和轨迹修正补全模块构成。交互捕获模块由T-Transformer和多关系图卷积网络组成，分别提取每个行人在时间序列和空间序列上的运动特征，并结合锚点控制模块推断行人的中间目的地以减少递归累计误差，由修正补全模块进行最终轨迹细化。在提取特征时添加逆关系可得到更为优化的结果，使用高斯剪枝减少虚假路径的生成也可提高模型效率。在ETH与UCY数据集上的实验结果表明，在平均位移误差（ADE）和最终位移误差（FDE）方面，该模型具有比现有大部分主流模型更好的性能。由于该模型在行人轨迹预测上的出色性能，可避免不必要的轨迹变更和碰撞风险，为行人轨迹预测应用提供了更为可能的解决方案。

关键词: T-Transformer, 图卷积网络（GCN）, 锚点控制, 行人轨迹预测

LIU Guihong, ZHOU Zongrun, MENG Xiangfu. Pedestrian Trajectory Prediction Based on Transformer and Multi-relation Graph Convolutional Networks[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(5): 1353-1364.

刘桂红, 周宗润, 孟祥福. 基于Transformer和多关系图卷积网络的行人轨迹预测[J]. 计算机科学与探索, 2025, 19(5): 1353-1364.

References

[1] MONREALE A, PINELLI F, TRASARTI R, et al. Where N-next: a location predictor on trajectory pattern mining[C]//Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM, 2009: 637-646.
[2] WANG Z H, LIU S, ZHANG J H, et al. A spatio-temporal CRF for human interaction understanding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2017, 27(8): 1647-1660.
[3] BETANCOURT A, MORERIO P, REGAZZONI C S, et al. The evolution of first person vision methods: a survey[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(5): 744-760.
[4] JI Y L, YANG Y, SHEN F M, et al. A survey of human action analysis in HRI applications[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(7): 2114-2128.
[5] MEHRAN R, OYAMA A, SHAH M. Abnormal crowd behavior detection using social force model[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2009: 935-942.
[6] YAMAGUCHI K, BERG A C, ORTIZ L E, et al. Who are you with and where are you going?[C]//Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2011: 1345-1352.
[7] PELLEGRINI S, ESS A, VAN GOOL L. Improving data association by joint modeling of pedestrian trajectories and groupings[C]//Proceedings of the 11th European Conference on Computer Vision. Berlin, Heidelberg: Springer, 2010: 452-465.
[8] ALAHI A, GOEL K, RAMANATHAN V, et al. Social LSTM: human trajectory prediction in crowded spaces[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 961-971.
[9] GUPTA A, JOHNSON J, LI F F, et al. Social GAN: socially acceptable trajectories with generative adversarial networks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 2255-2264.
[10] VEMULA A, MUELLING K, OH J. Social attention: modeling attention in human crowds[C]//Proceedings of the 2018 IEEE International Conference on Robotics and Automation. Piscataway: IEEE, 2018: 4601-4607.
[11] SHI X D, SHAO X W, WU G M, et al. Social-DPF: socially acceptable distribution prediction of futures[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(3): 2550-2557.
[12] BAE I, JEON H G. Disentangled multi-relational graph convolutional network for pedestrian trajectory prediction[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(2): 911-919.
[13] 李文礼, 张祎楠, 王梦昕. 基于视野域机制的行人轨迹预测[J]. 计算机应用研究, 2023, 40(1): 80-85.
LI W L, ZHANG Y N, WANG M X. Pedestrian trajectory prediction based on field of view mechanism[J]. Application Research of Computers, 2023, 40(1): 80-85.
[14] MOHAMED A, QIAN K, ELHOSEINY M, et al. Social-STGCNN: a social spatio-temporal graph convolutional neural network for human trajectory prediction[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 14412-14420.
[15] MANGALAM K, GIRASE H, AGARWAL S, et al. It is not the journey but the destination: endpoint conditioned trajectory prediction[C]//Proceedings of the 16th European Conference on Computer Vision. Cham: Springer, 2020: 759-776.
[16] BAE I, PARK J H, JEON H G. Learning pedestrian group representations for multi-modal trajectory prediction[C]//Proceedings of the 17th European Conference on Computer Vision. Cham: Springer, 2022: 270-289.
[17] BARTOLI F, LISANTI G, BALLAN L, et al. Context-aware trajectory prediction[C]//Proceedings of the 2018 24th International Conference on Pattern Recognition. Piscataway: IEEE, 2018: 1941-1946.
[18] FERNANDO T, DENMAN S, SRIDHARAN S, et al. Soft + Hardwired attention: an LSTM framework for human trajectory prediction and abnormal event detection[J]. Neural Networks, 2018, 108: 466-478.
[19] SHI L, ZHANG Y F, CHENG J, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 12018-12027.
[20] LI J C, MA H B, TOMIZUKA M. Conditional generative neural system for probabilistic trajectory prediction[C]//Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE, 2019: 6150-6156.
[21] SHI X D, SHAO X W, FAN Z P, et al. Multimodal interaction-aware trajectory prediction in crowded space[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 11982-11989.
[22] KOTHARI P, SIFRINGER B, ALAHI A. Interpretable social anchors for human trajectory forecasting in crowds[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 15551-15561.
[23] XU C X, MAO W B, ZHANG W J, et al. Remember intentions: retrospective-memory-based trajectory prediction[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 6478-6487.
[24] 汤凌燕, 熊聪聪, 王嫄, 等. 基于深度学习的短文本情感倾向分析综述[J]. 计算机科学与探索, 2021, 15(5): 794-811.
TANG L Y, XIONG C C, WANG Y, et al. Review of deep learning for short text sentiment tendency analysis[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(5): 794-811.
[25] XUE H, HUYNH D Q, REYNOLDS M. SS-LSTM: a hierarchical LSTM model for pedestrian trajectory prediction[C]//Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision. Piscataway: IEEE, 2018: 1186-1194.
[26] 裴利沈, 赵雪专. 群体行为识别深度学习方法研究综述[J]. 计算机科学与探索, 2022, 16(4): 775-790.
PEI L S, ZHAO X Z. Survey of collective activity recognition based on deep learning[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(4): 775-790.
[27] SHAFIEE N, PADIR T, ELHAMIFAR E. Introvert: human trajectory prediction via conditional 3D attention[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 16810-16820.
[28] CHEN Y Y, LIU C C, SHI B E, et al. Robot navigation in crowds by graph convolutional networks with attention learned from human gaze[J]. IEEE Robotics and Automation Letters, 2020, 5(2): 2754-2761.
[29] TRAN K N, GALA A, KAKADIARIS I A, et al. Activity analysis in crowded environments using social cues for group discovery and human interaction modeling[J]. Pattern Recognition Letters, 2014, 44: 49-57.
[30] ZHAO T Y, XU Y F, MONFORT M, et al. Multi-agent tensor fusion for contextual trajectory prediction[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 12118-12126.
[31] KOSARAJU V, SADEGHIAN A, MARTíN-MARTíN R, et al. Social-BiGAT: multimodal trajectory forecasting using bicycle-GAN and graph attention networks[C]//Advances in Neural Information Processing Systems 32, 2019: 137-146.
[32] LIANG J W, JIANG L, NIEBLES J C, et al. Peeking into the future: predicting future person activities and locations in videos[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 5718-5727.
[33] SUN H, ZHAO Z Q, HE Z H. Reciprocal learning networks for human trajectory prediction[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 7414-7423.
[34] HUG R, HüBNER W, ARENS M. Introducing probabilistic Bézier curves for N-step sequence prediction[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(6): 10162-10169.
[35] 李明, 赵勋杰. 改进的基于高斯混合模型的运动目标检测算法[J]. 计算机工程与应用, 2011, 47(8): 204-206.
LI M, ZHAO X J. Improved moving objects detection algorithm based on Gaussian mixture model[J]. Computer Engineering and Applications, 2011, 47(8): 204-206.
[36] SADEGHIAN A, KOSARAJU V, SADEGHIAN A, et al. SoPhie: an attentive GAN for predicting paths compliant to social and physical constraints[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 1349-1358.
[37] LEE N, CHOI W, VERNAZA P, et al. DESIRE: distant future prediction in dynamic scenes with interacting agents[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2165-2174.
[38] IVANOVIC B, PAVONE M. The trajectron: probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 2375-2384.
[39] SALZMANN T, IVANOVIC B, CHAKRAVARTY P, et al. Trajectron++: dynamically-feasible trajectory forecasting with heterogeneous data[C]//Proceedings of the 16th European Conference on Computer Vision. Cham: Springer, 2020: 683-700.
[40] XU Y, WANG L C, WANG Y Z, et al. Adaptive trajectory prediction via transferable GNN[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 6510-6521.
[41] GU T P, CHEN G Y, LI J L, et al. Stochastic trajectory prediction via motion indeterminacy diffusion[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 17092-17101.
[42] LIU Z Y, ZHOU J. Graph attention networks[M]//Introduction to graph neural networks: synthesis lectures on artificial intelligence and machine learning. Cham: Springer, 2020: 39-41.
[43] HUANG Y F, BI H K, LI Z X, et al. STGAT: modeling spatial-temporal interactions for human trajectory prediction[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 6271-6280.
[44] SUN J H, JIANG Q H, LU C W. Recursive social behavior graph for trajectory prediction[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 657-666.
[45] REHDER E, KLOEDEN H. Goal-directed pedestrian prediction[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision Workshop. Piscataway: IEEE, 2015: 139-147.
[46] NGUYEN A D, TRAN N H, NGUYEN T T, et al. A hybrid multi-waypoints path planning system for robots with minimum turning radius constraint using GA-B-spline and Dubins interpolation[C]//Proceedings of the 2021 International Conference on Advanced Mechanical Engineering, Automation, and Sustainable Development. Cham: Springer, 2022: 906-917.
[47] DENDORFER P, O?EP A, LEAL-TAIXé L. Goal-GAN: multimodal trajectory prediction based on goal position estimation[C]//Proceedings of the 15th Asian Conference on Computer Vision. Cham: Springer, 2021: 405-420.
[48] DEO N, TRIVEDI M M. Trajectory forecasts in unknown environments conditioned on grid-based plans[EB/OL]. [2024-03-10]. https://arxiv.org/abs/2001.00735.
[49] MARCHETTI F, BECATTINI F, SEIDENARI L, et al. MANTRA: memory augmented networks for multiple trajectory prediction[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 7141-7150.
[50] DENDORFER P, ELFLEIN S, LEAL-TAIXé L. MG-GAN: a multi-generator model preventing out-of-distribution samples in pedestrian trajectory prediction[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 13138-13147.
[51] YU C J, MA X, REN J W, et al. Spatio-temporal graph transformer networks for pedestrian trajectory prediction[C]//Proceedings of the 16th European Conference on Computer Vision. Cham: Springer, 2020: 507-523.
[52] SHI L S, WANG L, LONG C J, et al. SGCN: sparse graph convolution network for pedestrian trajectory prediction[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 8990-8999.
[53] YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1): 7444-7452.
[54] LIAN J, REN W W, LI L H, et al. PTP-STGCN: pedestrian trajectory prediction based on a spatio-temporal graph convolutional neural network[J]. Applied Intelligence, 2023, 53(3): 2862-2878.
[55] LIU Y J, YAN Q, ALAHI A. Social NCE: contrastive learning of socially-aware motion representations[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 15098-15109.
[56] CHEN G Y, LI J L, LU J W, et al. Human trajectory prediction via counterfactual analysis[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 9804-9813.
[57] ZHANG P, OUYANG W L, ZHANG P F, et al. SR-LSTM: state refinement for LSTM towards pedestrian trajectory prediction[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 12077-12086.
[58] MONTI A, PORRELLO A, CALDERARA S, et al. How many observations are enough? Knowledge distillation for trajectory forecasting[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 6543-6552.