Forecast-PEFT: Parameter-Efficient Fine-Tuning for Pre-trained Motion Forecasting Models

Jifeng Wang; Kaouther Messaoud; Yuejiang Liu; Juergen Gall; Alexandre Alahi

cs.CVJul 28, 2024

Forecast-PEFT: Parameter-Efficient Fine-Tuning for Pre-trained Motion Forecasting Models

Authors: Jifeng Wang, Kaouther Messaoud, Yuejiang Liu, Juergen Gall, Alexandre Alahi

Organizations: University of Bonn, Bonn, Germany · VITA Laboratory, EPFL, Lausanne, Switzerland · Stanford University, Stanford, USA

Abstract

Recent advancements in motion forecasting have been significantly accelerated by self-supervised pre-training techniques. However, current methods typically rely on extensive fine-tuning of pre-trained models for downstream tasks like trajectory prediction, a process that can be inefficient. This inefficiency primarily arises because traditional fine-tuning strategies fail to fully exploit the close alignment between motion prediction tasks and masked reconstruction objectives used during pre-training. To overcome these limitations, we introduce Forecast-PEFT, a parameter-efficient fine-tuning framework specifically designed for motion forecasting. Forecast-PEFT preserves most of the pre-trained model's parameters, introducing minimal yet effective adjustments through novel prompts and adapters. By doing so, Forecast-PEFT significantly reduces the number of parameters needing retraining while preserving essential pre-trained knowledge. Our method demonstrates robust generalization across multiple datasets, achieving higher accuracy with only about 17% of the parameters required by traditional full fine-tuning approaches. Additionally, our fully fine-tuned variant, Forecast-FT, further enhances prediction performance, achieving improvements of up to 9.6% over conventional baseline methods. These findings highlight Forecast-PEFT as a scalable, efficient, and effective alternative for fine-tuning pre-trained motion forecasting models. Code is available at https://github.com/csjfwang/Forecast-PEFT.

Figures & tables

Fig. 1: (a) Comparison of trainable parameters (percentage), performance (minFDE), and catastrophic forgetting (Reconstruction Error). The dashed line indicates the reconstruction error of the pre-trained baseline Forecast-MAE (PRE). The colored circles represent the full fine-tuned baseline Forecast-MAE (FT), and our presented PEFT methods, namely Forecast-PEFT (A), which uses only additive modules, and the default model Forecast-PEFT. (b) Traditional pretraining and fine-tuning on different datasets. (c) Our Forecast-PEFT method offers flexible cross-dataset fine-tuning, requiring just a single pre-training on a large dataset followed by efficient fine-tuning across various datasets. Its trainable parameters, acting as a plug-in module, require only 20% of the parameters for adaptation to each new dataset.

Fig. 2: Forecast-PEFT vs. traditional full finetuning : (a) Traditional methods, exemplified by Forecast-MAE [ 9 ] , use a pre-trained encoder and attach a randomly initialized new decoder. (b) Forecast-PEFT retains the pre-trained decoder, initially for masked motion/map reconstruction, adapting it for future motion forecasting by integrating Modality Control Prompts into the decoder, Contextual Embedding Prompts into the encoder, and incorporating Parallel Adapters.

Method	minADE ( $\downarrow$ )	minFDE ( $\downarrow$ )	MR ( $\downarrow$ )	b-minFDE ( $\downarrow$ )
SSL-Lanes [ 3 ]	0.83	1.67	0.25	2.35
GoReLa [ 11 ]	0.76	1.48	0.22	2.01
GANet [ 55 ]	0.73	1.35	0.17	1.97
Gnet [ 15 ]	0.69	1.34	0.18	1.90
Macformer [ 12 ]	0.70	1.38	0.19	1.90
ProphNet [ 56 ]	0.66	1.32	0.18	1.88

TABLE I: Quantitative Results on the AV2 test set. The percentage shows the improvement of Forecast-FT compared to Forecast-MAE.

Method	minADE ( $\downarrow$ )	minFDE ( $\downarrow$ )	MR ( $\downarrow$ )	b-minFDE ( $\downarrow$ )
THOMAS [ 17 ]	0.94	1.44	0.104	1.97
SSL-Lanes [ 3 ]	0.84	1.25	0.13	1.94
AutoBots [ 18 ]	0.89	1.41	0.16	2.06
GANet [ 55 ]	0.81	1.16	0.12	1.79
Wayformer [ 42 ]	0.77	1.16	0.12	1.74
Gnet [ 15 ]	0.79	1.16	0.12	1.75

TABLE II: Quantitative Results on the AV1 test set.

Fig. 3: Qualitative results of (a) pre-trained encoder and decoder of Forecast-MAE, (b) our proposed Forecast-PEFT, and (c) Forecast-FT. The history trajectories are in light blue, the forecasted future trajectories are in dark blue, and the ground truth is in gradient pink. The box in orange indicates the target agent, and the box in blue denotes neighbor agents.

Method	Params	minADE	minFDE	MR
Forecast-MAE [ 9 ]	1.9M	0.711	1.406	0.178
Multi-modal Decoder	246K	0.905	1.982	0.327
VPT Shallow [ 27 ]	252K	0.852	1.814	0.286
VPT Deep [ 27 ]	272K	0.843	1.793	0.281
LoRA (r=16) [ 25 ]	279K	0.828	1.762	0.26
LoRA (r=64) [ 25 ]	377K	0.786	1.638	0.235

TABLE III: Comparison with different PEFT methods on the AV2 validation set. The Multi-modal Decoder is the one we described in Equation 6 , which is the original decoder of our baseline [ 9 ] finetuned to get multi-modal output.

Fig. 4: Standardization techniques used for cross-dataset fine-tuning. (a) AV2 and AV1 datasets sampled at 10Hz provide detailed historical and future trajectories, while (b) nuScenes dataset sampled at 2Hz offers sparse trajectories. (c) Pre-training is performed on the AV2 dataset with masked trajectory inputs. (d-e) For fine-tuning on datasets with shorter trajectory lengths (AV1) or lower sampling rates (nuScenes), missing data points are represented as masked entries (zero padding with validity masks) to standardize inputs. (f) Forecast-PEFT effectively adapts the model across these diverse datasets without architectural changes, demonstrating robust generalization capabilities.

Method (FT on)	Params	minADE	minFDE	MR	minADE	minFDE	MR	minADE	minFDE	MR	minADE	minFDE	MR
		Eval on AV2 (6s future)			Eval on AV2 (3s future)			Eval on AV1			Eval on nu
Forecast-MAE (AV1)	1.9M	14.7	38.8	0.967	0.76	1.69	0.248	0.64	1.07	0.098	-	-	-
Forecast-MAE (AV2)	1.9M	0.72	1.43	0.183	0.29	0.51	0.026	1.75	3.42	0.406	1.31	2.80	0.450
Forecast-MAE (nu)	1.9M	1.92	4.26	0.557	-	-	-	-	-	-	1.02	2.32	0.336
Forecast-PEFT (AV2/AV1/nu)	383K/383K/383K	0.71	1.39	0.181	0.29	0.51	0.024	0.64	1.06	0.096	0.99	2.20	0.313

TABLE IV: Cross-dataset fine-tuning results on the AV2, AV1 and nuScenes validation sets, with six prediction modes. Models are pre-trained on AV2 using a masking and reconstruction strategy. “FT on” and “Eval on” denote the datasets used for fine-tuning and evaluation, respectively. Forecast-MAE fully fine-tunes all 1.9M parameters for each dataset, while our Forecast-PEFT fine-tunes only the specified modules, totaling 383K parameters.

Fig. 6: Efficiency of using different percentages of the dataset for fine-tuning. All evaluation results are obtained on the AV2 validation set.

Fig. 7: Efficiency analysis comparing Forecast-PEFT, Forecast-PEFT (P), and Forecast-MAE across different batch sizes: (a) training GPU memory usage, (b) training throughput (samples per second), (c) inference GPU memory usage, and (d) inference speed measured in frames per second (FPS).

Fig. 8: Ablation study on Prompt Length and Adapter Rank. In our ablation study, we fixed the adapter rank at 64 when assessing prompt length and maintained a constant prompt length of 50 for evaluating adapter rank.

Prompt	Head	LN&Bias	Adapter	minADE	minFDE	MR
✓				0.775	1.56	0.223
✓	✓			0.749	1.49	0.209
✓		✓		0.753	1.50	0.205
✓			✓	0.704	1.37	0.181
✓	✓	✓		0.735	1.45	0.197
✓	✓		✓	0.701	1.37	0.180

TABLE VI: Ablation study for different modules. The top gray row represents our Forecast-PEFT (A) model, tuning only the additive modules, while the bottom gray row shows our complete Forecast-PEFT, tuning both additive and selective modules.

Fig. 9: Visualization of attention weights on road agents’ history and lane segments , illustrating the effectiveness of Contextual Embedding Prompts (CEPs). Each scenario represents typical driving maneuvers: (a) turning left or right at intersections, (b) proceeding straight through intersections, and (c) lane switching. For each case, the left column shows attention maps without CEP, while the right column shows attention maps with CEP, highlighting how CEPs modify the model’s attention toward spatially and behaviorally relevant agents and lane geometry.

Method	APD (m) $\uparrow$	FPD (m) $\uparrow$
Forecast-MAE [ 9 ]	2.774	7.158
Forecast-PEFT	3.053	7.968

TABLE VIII: Inter-mode diversity on the AV2 validation set. APD and FPD measure pairwise trajectory separation over all timesteps and at the final timestep, respectively.

Method	minADE ( $\downarrow$ )	minFDE ( $\downarrow$ )	MR ( $\downarrow$ )	b-minFDE ( $\downarrow$ )
AutoBots [ 18 ]	0.85	1.70	0.27	2.51
MTR [ 51 ]	0.85	1.68	0.30	2.08
Wayformer [ 42 ]	0.85	1.75	0.28	2.38
Wayformer (w.pretrain)	0.84	1.71	0.27	2.33
Forecast-PEFT (ours)	0.83	1.69	0.27	2.30
Forecast-FT (ours)	0.77	1.51	0.22	2.13

TABLE IX: Quantitative Results on the AV2 Validation set.

Method (FT on)	minADE	minFDE	MR	minADE	minFDE	MR
		Eval on AV2	Eval on Nu
Wayformer (AV2)	0.84	1.71	0.27	1.25	2.64	0.41
Wayformer (Nu)	1.72	3.71	0.56	0.98	2.25	0.38
Forecast-PEFT (AV2/Nu)	0.83	1.69	0.27	0.98	2.17	0.34

TABLE X: Cross-dataset fine-tuning results on the AV2 and Nu validation sets, with six prediction modes. Models are pre-trained on AV2 using a masking and reconstruction strategy.

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.

See all 609 papers

FutureLift: Lifting Trajectory Proposals into Bidirectional Future Graphs for Motion Forecasting
Oct 15, 2026Daniel Grimm, J. Marius ZöllnerMultimodal Trajectory Prediction Multi-Agent Trajectory Prediction
FZI Research Center for Information Technology
Recall to Predict: Grounding Motion Forecasting in Interpretable Motion Bank
May 2, 2026Abhishek Vivekanandan, Ahmed Abouelazm, J. Marius ZöllnerMulti-Agent Trajectory Prediction Multimodal Trajectory Prediction
FZI Forschungszentrum Informatik · Karlsruhe Institute of Technology (KIT) Karlsruhe, Germany
SimpliHuMoN: Simplifying Human Motion Prediction
Mar 4, 2026Aadya Agrawal, Alexander SchwingHuman Motion Prediction Human Trajectory Prediction
University of Illinois Urbana-Champaign, USA

Oct 15, 2026J/K move · Enter open · S save

FutureLift: Lifting Trajectory Proposals into Bidirectional Future Graphs for Motion Forecasting

Daniel Grimm, J. Marius Zöllner

FZI Research Center for Information Technology

Multimodal Trajectory Prediction Multi-Agent Trajectory Prediction Autonomous Driving Perception

Open paper AskPDF arXiv

Accurate motion prediction is critical for safe autonomous driving, yet even strong forecasters leave residual errors in their multimodal proposals. Existing refiners compress complete proposals or update their segments sequentially, preventing the full proposed future from serving as explicit relational context for correction. We introduce FutureLift, a backbone-agnostic refiner that represents each multimodal proposal as a heterogeneous future graph, with spatially grounded temporal nodes connected to agent histories, map elements, alternative modes, and, for joint prediction, other agents future trajectories. Bidirectional temporal edges allow later evidence to revise earlier segments in a single pass, and the updated nodes predict residual coordinate and confidence corrections. Requiring only proposals, scores, and scene context, FutureLift provides a common refinement interface for structurally different forecasting backbones. On Argoverse2 and the Waymo Open Motion Dataset, it consistently improves frozen proposals and outperforms native and specialized refiners across marginal and joint forecasting. On the WOMD motion-prediction leaderboard, FutureLift ranks first in minADE (0.5049m) and minFDE (0.9942m) among the compared standard-input methods, without LiDAR or ensembling, while achieving a competitive 0.4874 Soft mAP.

Abstract

Parameter-Efficient Fine-Tuning

Explore similar work

FutureLift: Lifting Trajectory Proposals into Bidirectional Future Graphs for Motion Forecasting

Recall to Predict: Grounding Motion Forecasting in Interpretable Motion Bank

SimpliHuMoN: Simplifying Human Motion Prediction