ART: Adaptive Relational Transformer for Pedestrian Trajectory Prediction with Temporal-Aware Relations

Ruochen Li; Ziyi Chang; Junyan Hu; Jiannan Li; Amir Atapour-Abarghouei; Hubert P. H. Shum

cs.CVApr 4, 2026

ART: Adaptive Relational Transformer for Pedestrian Trajectory Prediction with Temporal-Aware Relations

Authors: Ruochen Li, Ziyi Chang, Junyan Hu, Jiannan Li, Amir Atapour-Abarghouei, Hubert P. H. Shum

Organizations: Department of Computer Science, Durham University, UK · the School of Computing and Information Systems, Singapore Management University, Singapore

Abstract

Accurate prediction of real-world pedestrian trajectories is crucial for a wide range of robot-related applications. Recent approaches typically adopt graph-based or transformer-based frameworks to model interactions. Despite their effectiveness, these methods either introduce unnecessary computational overhead or struggle to represent the diverse and time-varying characteristics of human interactions. In this work, we present an Adaptive Relational Transformer (ART), which introduces a Temporal-Aware Relation Graph (TARG) to explicitly capture the evolution of pairwise interactions and an Adaptive Interaction Pruning (AIP) mechanism to reduce redundant computations efficiently. Extensive evaluations on ETH/UCY and NBA benchmarks show that ART delivers state-of-the-art accuracy with high computational efficiency.

Figures & tables

Fig. 1: Framework overview. Framework overview. The relation between pedestrians is inferred from the temporal evolution of their pairwise interactions over the observed history.

Fig. 2: Overview of ART. Left: Temporal-Aware Relation Graph (TARG) leverages pairwise attention to model agent interactions across time steps, assigning higher weights to informative moments. Right: Adaptive Interaction Pruning (AIP) uses top- p filtering to adaptively retain informative neighbors based on cumulative interaction strength, producing a sparsified graph for trajectory prediction.

ETH/UCY Dataset
Subset	GroupNet	MemoNet	MID	NPSN	EqMotion	ET	LED	SingularTraj	MART	Ours
	[ 33 ]	[ 34 ]	[ 12 ]	[ 4 ]	[ 35 ]	[ 3 ]	[ 26 ]	[ 5 ]	[ 17 ]
ETH	0.46/0.73	0.40/0.61	0.39/0.66	0.36 /0.59	0.40/0.61	0.36 /0.53	0.39/0.58	0.35 / 0.42	0.35 / 0.47	0.35 / 0.47
HOTEL	0.15/0.25	0.11 / 0.17	0.13/0.22	0.16/0.25	0.12 / 0.18	0.12 /0.19	0.11 / 0.17	0.13/0.19	0.14/0.22	0.13/0.21
UNIV	0.26/0.49	0.24/0.43	0.22 /0.45	0.23/ 0.39	0.23/0.43	0.24/0.43	0.26/0.43	0.25/0.44	0.25/0.45	0.24/0.43
ZARA1	0.21/0.39	0.18 /0.32	0.17 /0.30	0.18 /0.32	0.18 /0.32	0.19/0.33	0.18 / 0.26	0.19/0.32	0.17 /0.29	0.17 / 0.28

TABLE I: Quantitative result on the ETH/UCY dataset. Metrics are

\min\mathrm{ADE}_{20}

\min\mathrm{FDE}_{20}

. Best and second-best results are marked in bold and underline .

NBA Dataset
Time	STAR	GroupNet	MemoNet	MID	NPSN	DynGroupNet	LED	SingularTraj	MART	Ours
	[ 39 ]	[ 33 ]	[ 34 ]	[ 12 ]	[ 4 ]	[ 36 ]	[ 26 ]	[ 5 ]	[ 17 ]
1.0s	0.43/0.66	0.26/0.34	0.38/0.56	0.28/0.37	0.35/0.58	0.19/0.28	0.18 /0.27	0.28/0.44	0.18 / 0.26	0.17 / 0.25
2.0s	0.75/1.24	0.49/0.70	0.71/1.14	0.51/0.72	0.68/1.23	0.40/0.61	0.37 / 0.56	0.61/1.00	0.35 / 0.50	0.35 / 0.50
3.0s	1.03/1.51	0.73/1.02	1.00/1.57	0.71/0.98	1.01/1.76	0.65/0.90	0.58/ 0.84	0.96/1.47	0.54 / 0.71	0.53 / 0.71
4.0s	1.13/2.01	0.96/1.30	1.25/1.47	0.96/1.27	1.31/1.79	0.89/1.13	0.81/ 1.10	1.31/1.98	0.73 / 0.90	0.72 / 0.90

TABLE II: Quantitative result on the NBA dataset. Metrics are

\min\mathrm{ADE}_{20}

\min\mathrm{FDE}_{20}

. Best and second-best results are marked in bold and underline .

Fig. 3: Ablation study of Top- $p$ threshold on the ETH/UCY dataset.

Weighting Strategies	ETH/UCY Dataset
	$\min\mathrm{ADE}_{20}$	$\min\mathrm{FDE}_{20}$
Cosine Similarity	0.22	0.36
Random Weighting	0.23	0.37
Uniform Weighting	0.23	0.36
Ours	0.20	0.32

TABLE III: Ablation study of relation weighting strategies on the ETH/UCY dataset.

Fig. 4: Qualitative comparisons with MART [ 17 ] on the ETH/UCY dataset. Past trajectories are shown in blue, ground truth in red, and model predictions in green.

Fig. 5: Qualitative comparisons with MART [ 17 ] on the NBA dataset. Past trajectories are shown in blue, ground truth in red, and predictions in green.

Method	#Param.	MACs
STAR [ 39 ]	1.0 M	12.0G
MemoNet [ 34 ]	10.7M	6.0G
GroupNet [ 33 ]	2.2M	411.5M
MID [ 12 ]	9.0M	40.3G
EqMotion [ 35 ]	3.0M	147.1M
LED [ 26 ]	10.9M	15.0G

TABLE IV: Model complexity comparison. Best and second-best results are marked in bold and underline .

Fig. 6: Visualization of temporal-aware relation weights on the NBA dataset. Blue denotes historical trajectories, ground-truth trajectories are in red, and orange indicates the temporal evolution of attention scores.

Pedestrian Trajectory Prediction

Pedestrian trajectory prediction forecasts where people will move from their observed motion and surrounding context.

See all 36 papers

Three-Step Hierarchical Transformer for Multi-Pedestrian Trajectory Prediction
Jun 22, 2026Raphaël Delécluse, Hazem Wannous, Laurent Grisoni +1Pedestrian Behavior Modeling Pedestrian Trajectory Prediction
IMT Nord Europe, University of Lille, CNRS UMR 9189 - CRIStAL, F-59000 Lille, France · University of Lille, CNRS UMR 9189 - CRIStAL, F-59000 Lille, France · Explain, F-59000 Lille, France
TSCA-Net: Temporal-Spatial Clique Attention for Interpretable Multimodal Pedestrian Trajectory Prediction
Jul 11, 2026Md Mustafizur Rahman, Guangchao Yang, A F M Abdun Noor +3Pedestrian Behavior Modeling Pedestrian Trajectory Prediction
College of Computer Science, Chongqing University, Chongqing, China · Department of Software Engineering, Daffodil International University, Dhaka, Bangladesh
Social Graph Mamba: Forecasting Pedestrian Movements Based on Social Context
Aug 11, 2026Hong-Son Nguyen, Yen-Chen LiuPedestrian Trajectory Prediction Pedestrian Behavior Modeling
Department of Mechanical Engineering, National Cheng Kung University, Tainan 70101, Taiwan

Jun 22, 2026J/K move · Enter open · S save

Three-Step Hierarchical Transformer for Multi-Pedestrian Trajectory Prediction

Raphaël Delécluse, Hazem Wannous, Laurent Grisoni, Laurent Guimas

IMT Nord Europe, University of Lille, CNRS UMR 9189 - CRIStAL, F-59000 Lille, France · University of Lille, CNRS UMR 9189 - CRIStAL, F-59000 Lille, France · Explain, F-59000 Lille, France

Pedestrian Behavior Modeling Pedestrian Trajectory Prediction Multi-Agent Trajectory Prediction Multimodal Trajectory Prediction Trajectory Prediction

Open paper AskPDF arXiv

Pedestrian trajectory prediction requires modeling temporal dynamics, multimodal cues, and social interactions in crowded environments. Existing methods often address these factors separately or entangle them in costly attention blocks, limiting scalability, flexibility, and interpretability. We propose a three-step hierarchical Transformer that explicitly separates temporal encoding, multimodal fusion, and scene-level interaction reasoning. Lightweight GRU summaries enable efficient cross-modal attention, while social attention over time--agent tokens captures inter-pedestrian influences at manageable cost. Experiments on JTA, JRDB, and the Pedestrians and Cyclists in Road Traffic dataset show state-of-the-art performance on real-world datasets (JRDB, Urban) and competitive results on JTA. Ablation and qualitative analyses confirm the contribution of each stage and the model's ability to anticipate complex behaviors such as early turning.

Abstract

Pedestrian Trajectory Prediction

Explore similar work

Three-Step Hierarchical Transformer for Multi-Pedestrian Trajectory Prediction

TSCA-Net: Temporal-Spatial Clique Attention for Interpretable Multimodal Pedestrian Trajectory Prediction

Social Graph Mamba: Forecasting Pedestrian Movements Based on Social Context