SAIL: Scene-aware Adaptive Iterative Learning for Long-Tail Trajectory Prediction in Autonomous Vehicles
Authors: Bin Rao, Haicheng Liao, Chengyue Wang, Keqiang Li, Zhenning Li, Hai Yang
Organizations: State Key Laboratory of Internet of Things for Smart City and Department of Civil and Environmental Engineering, University of Macau, Macau SAR, China · State Key Laboratory of Internet of Things for Smart City and Department of Computer and Information Science, University of Macau, Macau SAR, China · Department of Automotive Engineering, Tsinghua University, Beijing, China · State Key Laboratory of Internet of Things for Smart City and Departments of Civil and Environmental Engineering and Computer and Information Science, University of Macau, Macau SAR, China · Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong, China
Autonomous vehicles (AVs) rely on accurate trajectory prediction for safe navigation in diverse traffic environments, yet existing models struggle with long-tail scenarios-rare but safety-critical events characterized by abrupt maneuvers, high collision risks, and complex interactions. These challenges stem from data imbalance, inadequate definitions of long-tail trajectories, and suboptimal learning strategies that prioritize common behaviors over infrequent ones. To address this, we propose SAIL, a novel framework that systematically tackles the long-tail problem by first defining and modeling trajectories across three key attribute dimensions: prediction error, collision risk, and state complexity. Our approach then synergizes an attribute-guided augmentation and feature extraction process with a highly adaptive contrastive learning strategy. This strategy employs a continuous cosine momentum schedule, similarity-weighted hard-negative mining, and a dynamic pseudo-labeling mechanism based on evolving feature clustering. Furthermore, it incorporates a focusing mechanism to intensify learning on hard-positive samples within each identified class. This comprehensive design enables SAIL to excel at identifying and forecasting diverse and challenging long-tail events. Extensive evaluations on the nuScenes and ETH/UCY datasets demonstrate SAIL's superior performance, achieving up to 28.8% reduction in prediction error on the hardest 1% of long-tail samples compared to state-of-the-art baselines, while maintaining competitive accuracy across all scenarios. This framework advances reliable AV trajectory prediction in real-world, mixed-autonomy settings.
Figures & tables
Figure 1: Analyzing vehicle trajectories from the perspectives of Prediction Error, Risk (inverse time-to-collision, 1/TTC), and Vehicle State reveals their intrinsic long-tail nature. The top 5% of data in each distribution corresponds to distinct real-world scenarios, which are often critical to ensuring the safe operation of autonomous vehicles.
Figure 2: The overall architecture of our proposed SAIL framework. The framework takes historical trajectories and HD map data as input and processes them through a multi-stage pipeline, including the Scene Representation Learning module and the Attribute-aware Trajectory Generator, to output multiple future trajectories. Panels (b), (c), and (d) provide detailed views of our key components: the Multi-dimensional Long-Tail Attributes definition, the Attribute Disentanglement and Prediction module, and the Attribute-aware Trajectory Generator, respectively.
Symbol
Description
Symbol
Description
A
Total number of agents
a
Index of the target agent
to
Observation horizon
tp
Prediction horizon
Ha={sat}
Full observed historical state sequence
sat
Kinematic state of agent a at time t
M={mn}
Set of map vectors
Nm
Number of map elements
T={ct∣t∈(to,to+tp]}
Ground-truth future trajectory
ct=(xt,yt)
Ground-truth coordinate at time t
T^={T^(k)}k=1K
Set of predicted future trajectories
K
Number of predicted modes
Table 1: Summary of main notations used throughout this paper.
Figure 3: Visualization of our Attribute-Guided Trajectory Augmentation strategies. Based on the identified long-tail attributes of a trajectory, AGTA applies a combination of targeted augmentations (Simplify, Shift, Mask, Subset) to create a diverse set of challenging positive samples for the subsequent contrastive learning stage.
Model
Top 1%
Top 2%
Top 3%
Top 4%
Top 5%
Rest
All
nuScenes Dataset
Traj++ EWTA [ 41 ]
1.73/4.43
1.36/3.54
1.17/3.03
1.04/2.68
0.95/2.41
0.16/0.26
0.22/0.39
Traj++ EWTA+contrastive [ 41 ]
1.28/2.85
0.97/2.15
0.83/1.83
0.76/1.64
0.70/1.48
0.15 /0.24
0.18/0.30
FEND [ 62 ]
1.21/2.50
0.92/1.88
0.79/1.61
0.72/1.43
0.66/1.31
0.14 / 0.20
0.17 / 0.26
TrACT [ 71 ]
1.23/2.65
0.98/2.11
0.85/1.82
0.78/1.64
0.72/1.49
-
0.19/0.31
CSD [ 13 ]
1.06 / 2.22
0.82 / 1.69
0.70 / 1.44
0.64 / 1.30
0.59 / 1.18
-
0.18/0.30
Table 2: Prediction results (minADE/minFDE) on nuScenes (2 s prediction) and ETH/UCY (4.8 s prediction) datasets. Samples are stratified based on their prediction error (FDE) to evaluate robustness, with the Top 1%-5% representing the highest error instances. Bold and underlined text indicate the best and second-best results, respectively. Cases marked with (’-’) indicate missing values. Improvement row shows the percentage improvement of our model over the second-best result.
Horizon (s)
Model
Top 1%
Top 2%
Top 3%
Top 4%
Top 5%
Rest
All
1
Q-EANet
0.18 /0.22
0.22/0.25
0.17 /0.21
0.15 /0.19
0.13 /0.17
0.08 / 0.10
0.13 /0.12
SAIL
0.19/ 0.21
0.20 / 0.23
0.18/ 0.20
0.16/ 0.18
0.14/ 0.16
0.13/0.11
0.14/ 0.12
2
Q-EANet
0.35/0.45
0.45/0.52
0.37/0.48
0.35/0.43
0.30/0.39
0.18/0.20
0.22/0.23
SAIL
0.32 / 0.40
0.42 / 0.48
0.36 / 0.45
0.33 / 0.40
0.28 / 0.37
0.16 / 0.19
0.18 / 0.23
3
Q-EANet
0.55/0.75
0.67/0.82
0.58/0.78
0.56/0.71
0.49/0.65
0.29 /0.33
0.34/0.38
SAIL
0.50 / 0.70
0.63 / 0.78
0.57 / 0.73
0.53 / 0.68
0.47 / 0.63
0.30/ 0.33
0.32 / 0.34
Table 3: Prediction results (minADE/minFDE) at different prediction horizons on the nuScenes dataset, based on Collision Risk. We compare our model with the baseline Q-EANet [ 6 ] . The Top 1%-5% intervals denote the most critical risk categories. Bold marks the best metric, while light gray background highlights cases where both metrics are simultaneously the best.
Figure 4: Heatmaps illustrating the performance improvements of the SAIL model relative to Q-EANet on the nuScenes dataset, categorized by collision risk levels and prediction horizons. Negative values indicate superior performance by SAIL. (a) Differences in minADE. (b) Differences in minFDE.
Horizon (s)
Model
Abrupt Acceleration
Abrupt Deceleration
Abrupt Lane Changes
High-Curvature Turns
Normal
All
1
Q-EANet
0.19 /0.17
0.22 /0.19
0.28/0.26
0.22/0.21
0.11 / 0.09
0.15/0.13
SAIL
0.20/ 0.16
0.23/ 0.18
0.26 / 0.24
0.21 / 0.19
0.12/0.10
0.14 / 0.12
2
Q-EANet
0.30 /0.29
0.36/0.34
0.42/0.44
0.34/0.37
0.18 / 0.18
0.23/0.23
SAIL
0.31/ 0.27
0.34 / 0.31
0.39 / 0.40
0.32 / 0.34
0.19/0.19
0.21 / 0.21
3
Q-EANet
0.42/0.45
0.49/0.50
0.56/0.62
0.47/0.56
0.29/0.32
0.35/0.38
SAIL
0.39 / 0.40
0.46 / 0.46
0.52 / 0.56
0.44 / 0.51
0.27 / 0.29
0.32 / 0.34
Table 4: Prediction results (minADE/minFDE) at different prediction horizons on the nuScenes dataset, based on vehicle state. We compare our model with the baseline Q-EANet [ 6 ] . Bold indicates the best result for each metric. Light gray background highlights cases where both metrics are simultaneously the best.
Model
Venue
Top 1%
Top 2%
Top 3%
Top 4%
Top 5%
All
PGP [ 10 ]
CoRL 2021
8.86/21.92
7.21/17.90
6.24/15.68
5.52/13.77
5.02/12.44
1.28/2.52
Q-EANet [ 6 ]
ITS 2024
7.55 / 18.78
6.15 / 15.58
5.44 / 13.76
4.94 / 12.49
4.55 / 11.49
1.20/2.45
LAformer [ 40 ]
CVPR 2024
8.19/19.03
6.73/15.81
5.89/13.90
5.33/12.60
4.90/11.61
1.19/ 2.42
UniTraj (MTR) [ 12 ]
ECCV 2024
7.84/21.69
6.44/18.06
5.69/15.95
5.18/14.49
4.78/13.37
1.15 /2.61
SAIL
-
6.84 / 17.21
5.80 / 14.57
5.05 / 12.88
4.58 / 11.68
4.22 / 10.72
1.18 / 2.42
Table 5: Model-agnostic worst-case performance comparison on the nuScenes dataset, reported in minADE5 / minFDE5 . The Top 1% to 5% subsets are dynamically defined for each model based on its own worst-performing samples, ranking by minFDE5 . The best results are in bold, and the second best are underlined.
Model
Venue
minADE 5
minADE 10
minFDE 1
minFDE 5
minFDE 10
MR 5
Trajectron++ [ 52 ]
ECCV 2020
1.88
1.51
9.52
5.63
-
0.70
LaPred [ 24 ]
CVPR 2021
1.47
1.12
8.12
3.37
2.39
0.53
MHA-JAM [ 44 ]
IV 2021
-
1.18
9.62
3.72
2.21
0.64
GoHome [ 16 ]
ICRA 2022
1.42
1.15
6.99
-
-
0.57
ContextVAE [ 64 ]
RAL 2023
1.59
-
8.24
3.28
-
-
AFormer-FLN [ 65 ]
CVPR 2024
1.83
1.32
-
3.78
2.86
-
Table 6: Quantitative evaluation of trajectory prediction performance on the full nuScenes dataset, with a prediction horizon of 6 s. Light gray background indicates the performance of our model.
Model
Venue
ETH
HOTEL
UNIV
ZARA1
ZARA2
AVG
PECNet [ 42 ]
ECCV 2020
0.54/0.87
0.18/0.24
0.22/0.39
0.17/0.30
0.35/0.60
0.29/0.48
AgentFormer [ 70 ]
ICCV 2021
0.45/0.75
0.14/0.22
0.25/0.45
0.18/0.30
0.14/0.24
0.23/0.39
Trajectron++ [ 52 ]
ECCV 2020
0.39/0.83
0.12/0.21
0.20/0.44
0.15 /0.33
0.11 /0.25
0.19/0.41
NPSN [ 2 ]
CVPR 2022
0.36/0.59
0.16/0.25
0.23/0.39
0.18/0.32
0.14/0.25
0.21/0.36
MID [ 18 ]
CVPR 2022
0.39/0.66
0.13/0.22
0.22/0.45
0.17/0.30
0.13/0.27
0.21/0.38
TUTR [ 54 ]
ICCV 2023
0.40/0.61
0.11/0.18
0.23/0.42
0.18/0.34
0.13/0.25
0.21/0.36
Table 7: Quantitative evaluation of trajectory prediction performance on the ETH/UCY dataset across all samples. Light gray background indicates the performance of our model.
Model
Components
Performance (minADE/minFDE)
AGTA
ADP
AMCL
EFC
FDCL
Top 1%
Top 2%
Top 3%
Top 4%
Top 5%
Rest
All
A
×
✓
✓
✓
✓
1.30/1.88
1.05/1.55
0.88/1.35
0.79/1.24
0.73/1.16
0.19/0.23
0.22/0.27
B
✓
×
✓
✓
✓
1.48/2.10
1.15/1.70
0.95/1.48
0.85/1.35
0.79/1.26
0.21/0.24
0.24/0.28
C
✓
✓
×
✓
✓
1.25/1.83
1.01/1.51
0.84/1.31
0.76/1.21
0.70/1.13
0.18/0.22
0.21/0.26
D
✓
✓
✓
×
✓
1.15/1.72
0.92/1.41
0.75/1.22
0.68/1.13
0.63/1.06
0.18/0.21
0.20/0.25
E
✓
✓
✓
✓
×
1.22/1.80
0.98/1.49
0.81/1.29
0.73/1.19
0.68/1.11
0.19/0.22
0.21/0.26
Table 8: Ablation analysis of key components within our method on the nuScenes benchmark. Each row from A to E represents a variant of our full model with one key component removed. AGTA: Attribute-Guided Trajectory Augmentation, ADP: Attribute Disentanglement and Prediction, AMCL: Adaptive Momentum Contrastive Learning, EFC: Evolving Feature Clustering, FDCL: Focused Decoupled Contrastive Learning. Bold text represents the best results.
Attribute Pair
Jaccard Similarity
Top 20%
Top 15%
Top 10%
Top 5%
Prediction Error vs. Collision Risk
10.7%
8.3%
5.2%
1.4%
Prediction Error vs. State Complexity
13.0%
10.3%
6.8%
4.8%
Collision Risk vs. State Complexity
17.0%
14.1%
11.4%
5.7%
Table 9: Jaccard similarity between attribute-specific long-tail subsets on the nuScenes validation set. The overlap between subsets identified by Prediction Error, Collision Risk, and State Complexity is measured under different tail thresholds, from the Top 20% to the Top 5% subsets.
Figure 5: UpSet visualization of intersections among attribute-specific long-tail subsets on the nuScenes validation set. Subfigures (a) to (d) correspond to the Top 20%, Top 15%, Top 10%, and Top 5% tail thresholds, respectively. In each subfigure, the bars indicate the number of samples in each subset intersection, while the connected dots denote the corresponding combination of Prediction Error, Collision Risk, and State Complexity. The percentages above the bars represent the proportion of each intersection among all samples selected under the given threshold.
Model
Inference Time (ms)
minADE5
minFDE1
MR5
MultiPath [ 5 ]
87
1.44
7.69
0.74
AgentFormer [ 70 ]
107
1.97
9.12
0.69
LAformer [ 40 ]
115
1.19
6.89
0.54
VisionTrap [ 46 ]
53
1.36
8.72
0.61
SAIL
18
1.18
6.45
0.50
Table 10: Comparative analysis of computational efficiency and predictive accuracy on the nuScenes dataset. We compare SAIL against SOTA baselines to demonstrate the trade-off between speed and performance. Bold and underlined text represent the best and second-best results, respectively.
Figure 6: Visualization of comparison results for inference time and error metrics on the nuScenes dataset. Panel (a) shows inference time, panel (b) depicts minADE5 , panel (c) illustrates minFDE1 , and panel (d) presents MR5 .
C
Top 1%
Top 2%
Top 3%
Top 4%
Top 5%
Rest
All
2
1.35/1.95
1.08/1.60
0.90/1.40
0.82/1.30
0.76/1.22
0.20/0.24
0.23/0.27
3
1.21/1.80
0.98/1.49
0.82/1.29
0.74/1.20
0.68/1.12
0.18/0.22
0.21/0.25
4
1.10/1.70
0.88/1.38
0.76/1.20
0.67/1.12
0.62/1.05
0.17/0.20
0.19/0.24
5
1.02 / 1.58
0.81 / 1.30
0.65 / 1.12
0.60 / 1.05
0.56 / 0.98
0.16 / 0.19
0.18 / 0.23
6
1.05/1.62
0.84/1.34
0.69/1.16
0.62/1.08
0.58/1.01
0.16/0.20
0.18/0.23
7
1.12/1.73
0.90/1.42
0.75/1.24
0.68/1.15
0.63/1.08
0.17/0.21
0.20/0.25
Table 11: Performance (minADE/minFDE) on nuScenes for varying numbers of clusters ( C ) in our EFC module.
Figure 7: Stability analysis of the Evolving Feature Clustering (EFC) process. The Adjusted Rand Index (ARI) between consecutive clustering updates is plotted against training epochs on the nuScenes dataset.
Figure 8: t-SNE visualization of samples drawn from the Top 5% subsets under the three-dimensional long-tail attributes of prediction error, collision risk, and state complexity. The three groups exhibit relatively distinct yet partially overlapping manifold structures, indicating that they capture different but correlated aspects of long-tail driving scenarios.
Figure 9: Qualitative results of long-tail trajectory prediction across diverse high-curvature driving scenarios on the nuScenes dataset. Panels (a) and (b) depict left-turn maneuvers, while panels (c) and (d) illustrate right-turn maneuvers. The others represent baseline model [ 6 ] predictions. Model B and Model C represent the variants of our model. The red line depicts the highest-probability trajectory, whereas the light red lines illustrate the multimodal predictions.
Figure 10: Qualitative results of long-tail trajectory prediction across various acceleration and deceleration scenarios on the nuScenes dataset. Panels (a) and (b) depict acceleration maneuvers, while panels (c) and (d) illustrate deceleration maneuvers. The others represent baseline model [ 6 ] predictions. Model B and Model C represent the variants of our model. The red line depicts the highest-probability trajectory, whereas the light red lines illustrate the multimodal predictions.
Figure 11: Representative failure cases under extreme long-tail scenarios. (a) Failure cases caused by severe visual occlusion. (b) Failure cases at open intersections without explicit traffic signal information. The red line depicts the highest-probability trajectory, whereas the light red lines illustrate the multimodal predictions.
Forecasting precise future motion of surrounding agents is essential for reliable autonomous vehicles. However, as the demand for longer prediction horizons increases, existing endpoint-completion or iterative-refine methods increasingly struggle with weak guidance and compounding errors. To tackle the long-horizon prediction challenge, we propose Pivot-Centric Trajectory Prediction (PCTP). By introducing pivots'' and focusing on predicting pivot points along extended trajectories, we divide the long-term prediction task into short-term sub-tasks at various scales. Specifically, PCTP decouples the long-term trajectory predicting process into two processes: pivot prediction and pivot-based trajectory refinement. The pivot prediction process aims to utilize global map context and agent-to-agent interactions to identify these pivot points'', while the pivot-based trajectory refinement process focuses on local map details and refines the short-term trajectory based on predicted ``pivot points''. Compared with existing methods, PCTP provides more intermediate guidance while reducing compounding errors. Moreover, PCTP is a flexible approach that can be integrated into most state-of-the-art trajectory prediction models. Experimental results show that PCTP improves the prediction accuracy of leading models on both Argoverse I and Argoverse II datasets with minimal impact on model size. Specifically, PCTP combined with QCNet outperforms all published ensemble-free methods on the Argoverse II leaderboard at submission.
Xiucong Zhao, Jindong Tian, Hao Miao
1Xi’an Jiaotong University · 2East China Normal University · 3Hong Kong Polytechnic University
Trajectory prediction is critical for autonomous driving, enabling safe and efficient planning in dense, dynamic traffic. Most existing methods optimize prediction accuracy under fixed-length observations. However, real-world driving often yields variable-length, incomplete observations, posing a challenge to these methods. A common strategy is to directly map features from incomplete observations to those from complete ones. This one-shot mapping, however, struggles to learn accurate representations for short trajectories due to significant information gaps. To address this issue, we propose a Progressive Retrospective Framework (PRF), which gradually aligns features from incomplete observations with those from complete ones via a cascade of retrospective units. Each unit consists of a Retrospective Distillation Module (RDM) and a Retrospective Prediction Module (RPM), where RDM distills features and RPM recovers previous timesteps using the distilled features. Moreover, we propose a Rolling-Start Training Strategy (RSTS) that enhances data efficiency during PRF training. PRF is plug-and-play with existing methods. Extensive experiments on datasets Argoverse 2 and Argoverse 1 demonstrate the effectiveness of PRF. Code is available at https://github.com/zhouhao94/PRF.
Hao Zhou, Lu Qi, Jason Li +5
Great Bay University · Tsinghua SIGS · Wuhan University +3
Accurate and reliable vehicle trajectory prediction is essential for safe autonomous driving. Recent studies have incorporated safety risk into trajectory prediction to quantify dangers posed by surrounding agents. However, most risk-aware approaches use past risk information as a secondary signal to help guide decisions, overlooking its future evolution and uncertainty. In this paper, we propose a risk horizon profiling (RHP) module that incorporates a continuous, learnable potential field model for risk-aware trajectory prediction. The RHP module calculates the spatial-temporal proximity of surrounding objects to profile risk distributions across future horizons, which supports better trajectory prediction by adaptively identifying what human drivers perceive as critical moments. We evaluate our method on two datasets from different driving settings, highD for highway corridors and SHRP2 for urban streets, which cover diverse risk scenarios including safe, near-crash, and crash events. Compared to the baseline methods, our framework achieves a 25.0% reduction in 5s RMSE on the highD dataset and a 29.1% reduction in 5s minFDE on SHRP2. These results indicate strong performance for both short and long horizon prediction and robust generalization across highway and urban scenarios. The proposed method enables more realistic AV path planning and strategic selection, thereby supporting safer autonomous driving and more advanced driver-assistance systems. The source code for this work is available at: https://github.com/bilab-nyu/RHP
Xinyi Ning, Zilin Bian, Dachuan Zuo +2
Department of Civil and Urban Engineering, New York University. · Department of Civil Engineering Technology and Environmental Management Safety, Rochester Institute of Technology.