Recent advancements in motion forecasting have been significantly accelerated by self-supervised pre-training techniques. However, current methods typically rely on extensive fine-tuning of pre-trained models for downstream tasks like trajectory prediction, a process that can be inefficient. This inefficiency primarily arises because traditional fine-tuning strategies fail to fully exploit the close alignment between motion prediction tasks and masked reconstruction objectives used during pre-training. To overcome these limitations, we introduce Forecast-PEFT, a parameter-efficient fine-tuning framework specifically designed for motion forecasting. Forecast-PEFT preserves most of the pre-trained model's parameters, introducing minimal yet effective adjustments through novel prompts and adapters. By doing so, Forecast-PEFT significantly reduces the number of parameters needing retraining while preserving essential pre-trained knowledge. Our method demonstrates robust generalization across multiple datasets, achieving higher accuracy with only about 17% of the parameters required by traditional full fine-tuning approaches. Additionally, our fully fine-tuned variant, Forecast-FT, further enhances prediction performance, achieving improvements of up to 9.6% over conventional baseline methods. These findings highlight Forecast-PEFT as a scalable, efficient, and effective alternative for fine-tuning pre-trained motion forecasting models. Code is available at https://github.com/csjfwang/Forecast-PEFT.
Figures & tables
Fig. 1: (a) Comparison of trainable parameters (percentage), performance (minFDE), and catastrophic forgetting (Reconstruction Error). The dashed line indicates the reconstruction error of the pre-trained baseline Forecast-MAE (PRE). The colored circles represent the full fine-tuned baseline Forecast-MAE (FT), and our presented PEFT methods, namely Forecast-PEFT (A), which uses only additive modules, and the default model Forecast-PEFT. (b) Traditional pretraining and fine-tuning on different datasets. (c) Our Forecast-PEFT method offers flexible cross-dataset fine-tuning, requiring just a single pre-training on a large dataset followed by efficient fine-tuning across various datasets. Its trainable parameters, acting as a plug-in module, require only 20% of the parameters for adaptation to each new dataset.
Fig. 2: Forecast-PEFT vs. traditional full finetuning : (a) Traditional methods, exemplified by Forecast-MAE [ 9 ] , use a pre-trained encoder and attach a randomly initialized new decoder. (b) Forecast-PEFT retains the pre-trained decoder, initially for masked motion/map reconstruction, adapting it for future motion forecasting by integrating Modality Control Prompts into the decoder, Contextual Embedding Prompts into the encoder, and incorporating Parallel Adapters.
Method
minADE ( ↓ )
minFDE ( ↓ )
MR ( ↓ )
b-minFDE ( ↓ )
SSL-Lanes [ 3 ]
0.83
1.67
0.25
2.35
GoReLa [ 11 ]
0.76
1.48
0.22
2.01
GANet [ 55 ]
0.73
1.35
0.17
1.97
Gnet [ 15 ]
0.69
1.34
0.18
1.90
Macformer [ 12 ]
0.70
1.38
0.19
1.90
ProphNet [ 56 ]
0.66
1.32
0.18
1.88
TABLE I: Quantitative Results on the AV2 test set. The percentage shows the improvement of Forecast-FT compared to Forecast-MAE.
Method
minADE ( ↓ )
minFDE ( ↓ )
MR ( ↓ )
b-minFDE ( ↓ )
THOMAS [ 17 ]
0.94
1.44
0.104
1.97
SSL-Lanes [ 3 ]
0.84
1.25
0.13
1.94
AutoBots [ 18 ]
0.89
1.41
0.16
2.06
GANet [ 55 ]
0.81
1.16
0.12
1.79
Wayformer [ 42 ]
0.77
1.16
0.12
1.74
Gnet [ 15 ]
0.79
1.16
0.12
1.75
TABLE II: Quantitative Results on the AV1 test set.
Fig. 3: Qualitative results of (a) pre-trained encoder and decoder of Forecast-MAE, (b) our proposed Forecast-PEFT, and (c) Forecast-FT. The history trajectories are in light blue, the forecasted future trajectories are in dark blue, and the ground truth is in gradient pink. The box in orange indicates the target agent, and the box in blue denotes neighbor agents.
Method
Params
minADE
minFDE
MR
Forecast-MAE [ 9 ]
1.9M
0.711
1.406
0.178
Multi-modal Decoder
246K
0.905
1.982
0.327
VPT Shallow [ 27 ]
252K
0.852
1.814
0.286
VPT Deep [ 27 ]
272K
0.843
1.793
0.281
LoRA (r=16) [ 25 ]
279K
0.828
1.762
0.26
LoRA (r=64) [ 25 ]
377K
0.786
1.638
0.235
TABLE III: Comparison with different PEFT methods on the AV2 validation set. The Multi-modal Decoder is the one we described in Equation 6 , which is the original decoder of our baseline [ 9 ] finetuned to get multi-modal output.
Fig. 4: Standardization techniques used for cross-dataset fine-tuning. (a) AV2 and AV1 datasets sampled at 10Hz provide detailed historical and future trajectories, while (b) nuScenes dataset sampled at 2Hz offers sparse trajectories. (c) Pre-training is performed on the AV2 dataset with masked trajectory inputs. (d-e) For fine-tuning on datasets with shorter trajectory lengths (AV1) or lower sampling rates (nuScenes), missing data points are represented as masked entries (zero padding with validity masks) to standardize inputs. (f) Forecast-PEFT effectively adapts the model across these diverse datasets without architectural changes, demonstrating robust generalization capabilities.
Eval on AV2 (6s future)
Eval on AV2 (3s future)
Eval on AV1
Eval on nu
Method (FT on)
Params
minADE
minFDE
MR
minADE
minFDE
MR
minADE
minFDE
MR
minADE
minFDE
MR
Forecast-MAE (AV1)
1.9M
14.7
38.8
0.967
0.76
1.69
0.248
0.64
1.07
0.098
-
-
-
Forecast-MAE (AV2)
1.9M
0.72
1.43
0.183
0.29
0.51
0.026
1.75
3.42
0.406
1.31
2.80
0.450
Forecast-MAE (nu)
1.9M
1.92
4.26
0.557
-
-
-
-
-
-
1.02
2.32
0.336
Forecast-PEFT (AV2/AV1/nu)
383K/383K/383K
0.71
1.39
0.181
0.29
0.51
0.024
0.64
1.06
0.096
0.99
2.20
0.313
TABLE IV: Cross-dataset fine-tuning results on the AV2, AV1 and nuScenes validation sets, with six prediction modes. Models are pre-trained on AV2 using a masking and reconstruction strategy. “FT on” and “Eval on” denote the datasets used for fine-tuning and evaluation, respectively. Forecast-MAE fully fine-tunes all 1.9M parameters for each dataset, while our Forecast-PEFT fine-tunes only the specified modules, totaling 383K parameters.
Figure 9
Fig. 6: Efficiency of using different percentages of the dataset for fine-tuning. All evaluation results are obtained on the AV2 validation set.
Fig. 7: Efficiency analysis comparing Forecast-PEFT, Forecast-PEFT (P), and Forecast-MAE across different batch sizes: (a) training GPU memory usage, (b) training throughput (samples per second), (c) inference GPU memory usage, and (d) inference speed measured in frames per second (FPS).
Fig. 8: Ablation study on Prompt Length and Adapter Rank. In our ablation study, we fixed the adapter rank at 64 when assessing prompt length and maintained a constant prompt length of 50 for evaluating adapter rank.
Prompt
Head
LN&Bias
Adapter
minADE
minFDE
MR
✓
0.775
1.56
0.223
✓
✓
0.749
1.49
0.209
✓
✓
0.753
1.50
0.205
✓
✓
0.704
1.37
0.181
✓
✓
✓
0.735
1.45
0.197
✓
✓
✓
0.701
1.37
0.180
TABLE VI: Ablation study for different modules. The top gray row represents our Forecast-PEFT (A) model, tuning only the additive modules, while the bottom gray row shows our complete Forecast-PEFT, tuning both additive and selective modules.
Fig. 9: Visualization of attention weights on road agents’ history and lane segments , illustrating the effectiveness of Contextual Embedding Prompts (CEPs). Each scenario represents typical driving maneuvers: (a) turning left or right at intersections, (b) proceeding straight through intersections, and (c) lane switching. For each case, the left column shows attention maps without CEP, while the right column shows attention maps with CEP, highlighting how CEPs modify the model’s attention toward spatially and behaviorally relevant agents and lane geometry.
Method
APD (m) ↑
FPD (m) ↑
Forecast-MAE [ 9 ]
2.774
7.158
Forecast-PEFT
3.053
7.968
TABLE VIII: Inter-mode diversity on the AV2 validation set. APD and FPD measure pairwise trajectory separation over all timesteps and at the final timestep, respectively.
Method
minADE ( ↓ )
minFDE ( ↓ )
MR ( ↓ )
b-minFDE ( ↓ )
AutoBots [ 18 ]
0.85
1.70
0.27
2.51
MTR [ 51 ]
0.85
1.68
0.30
2.08
Wayformer [ 42 ]
0.85
1.75
0.28
2.38
Wayformer (w.pretrain)
0.84
1.71
0.27
2.33
Forecast-PEFT (ours)
0.83
1.69
0.27
2.30
Forecast-FT (ours)
0.77
1.51
0.22
2.13
TABLE IX: Quantitative Results on the AV2 Validation set.
Eval on AV2
Eval on Nu
Method (FT on)
minADE
minFDE
MR
minADE
minFDE
MR
Wayformer (AV2)
0.84
1.71
0.27
1.25
2.64
0.41
Wayformer (Nu)
1.72
3.71
0.56
0.98
2.25
0.38
Forecast-PEFT (AV2/Nu)
0.83
1.69
0.27
0.98
2.17
0.34
TABLE X: Cross-dataset fine-tuning results on the AV2 and Nu validation sets, with six prediction modes. Models are pre-trained on AV2 using a masking and reconstruction strategy.
Accurate motion prediction is critical for safe autonomous driving, yet even strong forecasters leave residual errors in their multimodal proposals. Existing refiners compress complete proposals or update their segments sequentially, preventing the full proposed future from serving as explicit relational context for correction. We introduce FutureLift, a backbone-agnostic refiner that represents each multimodal proposal as a heterogeneous future graph, with spatially grounded temporal nodes connected to agent histories, map elements, alternative modes, and, for joint prediction, other agents future trajectories. Bidirectional temporal edges allow later evidence to revise earlier segments in a single pass, and the updated nodes predict residual coordinate and confidence corrections. Requiring only proposals, scores, and scene context, FutureLift provides a common refinement interface for structurally different forecasting backbones. On Argoverse2 and the Waymo Open Motion Dataset, it consistently improves frozen proposals and outperforms native and specialized refiners across marginal and joint forecasting. On the WOMD motion-prediction leaderboard, FutureLift ranks first in minADE (0.5049m) and minFDE (0.9942m) among the compared standard-input methods, without LiDAR or ensembling, while achieving a competitive 0.4874 Soft mAP.
Motion forecasting often requires trading interpretability for predictive accuracy. Standard anchor-based architectures rely on opaque latent queries that are highly prone to latent collapse, or naive trajectory sampling that limits multi-modal diversity. We propose an end-to-end differentiable framework that grounds predictions in a comprehensive "motion bank", a structured embedding space of physically realizable trajectories constructed via contrastive learning. Rather than regressing paths from a blank slate, our architecture dynamically retrieves explicit motion priors using a novel Anchor Retrieval Layer. This module adapts orthogonally initialized queries via a Dual-Level Gated Cross-Attention mechanism and executes discrete trajectory selection using a Straight-Through Gumbel-Softmax estimator to preserve continuous gradient flow. The retrieved semantically grounded anchors are then geometrically refined by a DETR-style decoder, optimized jointly with a Winner-Takes-All (WTA) kinematic Gaussian Mixture Model (GMM), a latent diversity penalty, and a soft-min weighted endpoint loss. By strictly conditioning the decoding phase on diverse, interpretable motion primitives, our approach eliminates the "black box" of standard latent queries while achieving competitive multi-modal accuracy on the Argoverse 2 and Waymo Open Motion datasets. Code is available at: https://github.com/abviv/recall2predict
Abhishek Vivekanandan, Ahmed Abouelazm, J. Marius Zöllner
FZI Forschungszentrum Informatik · Karlsruhe Institute of Technology (KIT) Karlsruhe, Germany
Human motion prediction combines the tasks of trajectory forecasting and human pose prediction. For each of the two tasks, specialized models have been developed. Combining these models for holistic human motion prediction is non-trivial, and recent methods have struggled to compete on established benchmarks for individual tasks. To address this, we propose a simple yet effective transformer-based model for human motion prediction. The model employs a stack of self-attention modules to effectively capture both spatial dependencies within a pose and temporal relationships across a motion sequence. This simple, streamlined, end-to-end model is sufficiently versatile to handle pose-only, trajectory-only, and combined prediction tasks without task-specific modifications. We demonstrate that this approach achieves state-of-the-art results across all tasks through extensive experiments on a wide range of benchmark datasets, including Human3.6M, AMASS, ETH-UCY, and 3DPW.