FutureLift: Lifting Trajectory Proposals into Bidirectional Future Graphs for Motion Forecasting

Daniel Grimm; J. Marius Zöllner

cs.CVOct 15, 2026

FutureLift: Lifting Trajectory Proposals into Bidirectional Future Graphs for Motion Forecasting

Authors: Daniel Grimm, J. Marius Zöllner

Organizations: FZI Research Center for Information Technology

Abstract

Accurate motion prediction is critical for safe autonomous driving, yet even strong forecasters leave residual errors in their multimodal proposals. Existing refiners compress complete proposals or update their segments sequentially, preventing the full proposed future from serving as explicit relational context for correction. We introduce FutureLift, a backbone-agnostic refiner that represents each multimodal proposal as a heterogeneous future graph, with spatially grounded temporal nodes connected to agent histories, map elements, alternative modes, and, for joint prediction, other agents future trajectories. Bidirectional temporal edges allow later evidence to revise earlier segments in a single pass, and the updated nodes predict residual coordinate and confidence corrections. Requiring only proposals, scores, and scene context, FutureLift provides a common refinement interface for structurally different forecasting backbones. On Argoverse2 and the Waymo Open Motion Dataset, it consistently improves frozen proposals and outperforms native and specialized refiners across marginal and joint forecasting. On the WOMD motion-prediction leaderboard, FutureLift ranks first in minADE (0.5049m) and minFDE (0.9942m) among the compared standard-input methods, without LiDAR or ensembling, while achieving a competitive 0.4874 Soft mAP.

Figures & tables

Figure 1: Refinement representations. Top: Trajectory-level methods compress proposal geometry into a shared representation. Middle: Causal segment-wise methods pass information forward, so later evidence cannot reach earlier segments within a forward-only sweep. Bottom: FutureLift lifts proposal segments into spatial nodes with bidirectional temporal relations, enabling later context to correct earlier segments in one graph pass. Social future relations are used only for coherent joint proposals.

Figure 2: FutureLift method overview. (a) Scene history and map context flow through an arbitrary frozen forecasting backbone, which exposes only multimodal trajectory proposals

\widehat{\mathbf{Y}}

and logits

\boldsymbol{\ell}

; FutureLift does not access backbone queries or intermediate features. (b) Each proposal is partitioned into

R

intervals and lifted into spatially anchored future nodes. For clarity, the panel visualizes one representative mode

k

of the

K

instantiated modes. Typed relations connect the full node set to agent histories, map context, other modes, and for coherent joint-scenes proposals also to other predicted agents, while bidirectional temporal edges expose the complete horizon before correction. (c) Relation-specific attention updates all future nodes on this fixed graph and simultaneously decodes coordinate residuals, uncertainty parameters, and confidence residuals. Thus, later evidence can revise earlier segments in one graph pass without a causal sweep or graph reconstruction.

Backbone	Refiner	b-minFDE 6 $\downarrow$	minFDE 6 $\downarrow$	minADE 6 $\downarrow$	MR 6 $\downarrow$
QCNet [ 33 ]	None	1.912	1.304	0.729	0.164
	Native	1.856	1.253	0.720	0.157
	SmartRefine †	1.852	1.258	0.718	0.157
	FutureLift	1.772	1.124	0.593	0.131
DenseTNT [ 6 ]	None	2.231	1.624	0.964	0.233
	SmartRefine	2.109	1.553	0.834	0.221

Table 1: Backbone-interface compatibility on the AV2 marginal validation split. Displacement errors are in meters; runtime is reported in Tab. 8 . † SmartRefine result using the authors’ matched-checkpoint export and backbone trajectory features.

Backbone	Refiner	avgBrier- minFDE 6 $\downarrow$	avgMin FDE 6 $\downarrow$	avgMin ADE 6 $\downarrow$	Actor MR 6 $\downarrow$
FJMP [ 17 ]	None	2.478	1.921	0.812	0.237
	SRefiner †	2.354	1.736	0.747	0.221
	FutureLift	2.114	1.512	0.699	0.187
Forecast- MAE [ 1 ]	None	2.412	1.642	0.713	0.195
	SRefiner †	2.251	1.477	0.658	0.183
	FutureLift	2.012	1.391	0.642	0.169

Table 2: Backbone-interface compatibility on the AV2 joint validation split. Displacement errors are in meters; runtime is reported in Tab. 8 . † Published result using the SRefiner authors’ matched-backbone proposal export and additional per-mode embeddings.

SD after $\pm$ is reported in units of $10^{-3}$ where shown.
Configuration	avgBrier- minFDE 6 $\downarrow$	avgMin FDE 6 $\downarrow$	avgMin ADE 6 $\downarrow$	Actor MR 6 $\downarrow$
No refinement	1.944	1.262	0.584	0.162
+ History grounding	1.924	1.253	0.581	0.158
+ Map grounding	1.886	1.216	0.572	0.154
+ Cross-mode grounding	1.862	1.204	0.570	0.153
+ Causal temp. reasoning x1	1.857	1.199 $\pm\mathrm{1.5}$	0.569	0.152 $\pm\mathrm{0.1}$

Table 4: FutureLift component build-up on the AV2 joint validation split using one frozen DONUT checkpoint. Context relations are added cumulatively; the temporal rows compare

J

repeated causal sweeps (x

J

) with one bidirectional sweep.

SD after $\pm$ is reported in units of $10^{-3}$ where shown.
Resolution	avgBrier- minFDE 6 $\downarrow$	avgMin FDE 6 $\downarrow$	avgMin ADE 6 $\downarrow$	Actor MR 6 $\downarrow$	Latency (ms)
$R=1$ (6 s / node)	1.884	1.221	0.599	0.161	59
$R=2$ (3 s / node)	1.827	1.162	0.574	0.153	76
$R=3$ (2 s / node)	1.816	1.142 $\pm 1.0$	0.552 $\pm 0.8$	0.146	90
$R=6$ (1 s / node)	1.810	1.139 $\pm 0.9$	0.551 $\pm 0.6$	0.144	136

Table 5: Temporal-resolution sensitivity on the AV2 joint validation split. The default setting is

R=3

Panel of Figure 3 — Figure 3: Representative failure cases. Left: The frozen proposal set omits the correct maneuver; post-hoc refinement cannot recover a mode absent from its inputs. Right: FutureLift produces excessive curvature because its residual decoder does not explicitly enforce vehicle dynamics.

Method	b-minFDE 6 $\downarrow$ (m)	minFDE 6 $\downarrow$ (m)	minADE 6 $\downarrow$ (m)	MR 6 $\downarrow$	minFDE 1 $\downarrow$ (m)	minADE 1 $\downarrow$ (m)
QCNet [ 33 ]	1.91	1.29	0.65	0.16	4.30	1.69
SmartRefine [ 32 ]	1.86	1.23	0.63	0.15	4.17	1.65
DeMo [ 29 ]	1.84	1.17	0.61	0.13	3.74	1.49
Polaris [ 28 ]	1.80	1.15	0.62	0.13	3.78	1.53
DONUT [ 10 ]	1.79	1.16	0.63	0.14	4.06	1.66
NDPNet [ 7 ]	1.83	1.17	0.61	0.14	4.08	1.62

Table 6: Additional AV2 marginal test-set context. Asterisks denote ensembles. FutureLift is evaluated locally with the released test annotations and official evaluator because the leaderboard is closed to new submissions. These results complement, but are distinct from, the controlled validation comparisons in Tab. 1 .

Method	avgBrier-minFDE 6 $\downarrow$ (m)	avgMinFDE 6 $\downarrow$ (m)	avgMinADE 6 $\downarrow$ (m)	ActorMR 6 $\downarrow$	avgMinFDE 1 $\downarrow$ (m)	avgMinADE 1 $\downarrow$ (m)
FJMP [ 17 ]	2.59	1.89	0.81	0.23	4.00	1.52
Forecast-MAE [ 1 ]	2.24	1.55	0.69	0.19	3.33	1.30
RealMotion [ 22 ]	2.01	1.32	0.62	0.18	2.87	1.14
DeMo [ 29 ]	1.93	1.24	0.58	0.16	2.78	1.12
Polaris [ 28 ]	1.86	1.18	0.56	0.15	2.67	1.07
FutureLift (DONUT)	1.82	1.14	0.55	0.15	2.63	1.07

Table 7: Additional AV2 joint test-set context. FutureLift is evaluated locally with the released test annotations and official evaluator because the leaderboard is closed to new submissions. These results complement, but are distinct from, the controlled validation comparisons in Tab. 2 .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Representative FutureLift corrections on the AV2 validation split. Each panel compares a frozen backbone proposal with the corresponding FutureLift-refined hypothesis and ground truth. FutureLift corrects lateral drift, curved-lane misalignment, and a local trajectory discontinuity.

Dataset / task	Backbone	Refiner	Params (M)	GFLOPs	Mean latency (ms)
AV2 marginal	QCNet	Native	7.7	55.8	95
AV2 marginal	QCNet	SmartRefine	8.0	56.2	112
AV2 marginal	QCNet	FutureLift	13.8	84.4	131
AV2 joint	FJMP	SRefiner	5.9	45.6	83
AV2 joint	FJMP	FutureLift	13.1	78.0	98
AV2 joint	DONUT	FutureLift	12.2	94.6	110

Appendix

Table 8: Refinement-only efficiency under the common protocol in Sec. A.4 . Latency includes graph construction and is averaged over 10 timed runs.

$J$	b-minFDE 6 $\downarrow$	minFDE 6 $\downarrow$	minADE 6 $\downarrow$	MR 6 $\downarrow$	Latency (ms) $\downarrow$
0	1.875	1.235	0.644	0.162	69
1	1.743	1.091	0.584	0.124	90
2	1.771	1.135	0.602	0.133	105
3	1.797	1.167	0.621	0.141	120

Appendix

Table 9: Effect of the number of refinement passes

J

on the AV2 marginal validation split. Displacement errors are in meters, and latency is measured on an NVIDIA H100 GPU.

Figure 5: Example scene

Figure 6: (point, belongs-to, lane) : All points belonging to the same lane, are connected to their respected lane node.

Figure 7: (lane, to, lane) : Lane nodes are connected to all neighbors within a radius of 150m.

Figure 8: (agent, self, agent) : Agent nodes are connected to up to 10 subsequent neighbors.

Figure 9: (lane, gives-traffic-info, agent) : Lanes are connected to all agents within a radius of 150m. For better visibility only edges to the green vehicle are depicted.

Figure 10: (agent, social, agent) : Agent nodes are connected to agent nodes from neighboring traffic participants at the concurrent timestamp within a 50m radius.

Figure 11: (agent, self, agent-future) : Agent-future nodes receive information from up to 30 previous agent nodes.

Figure 12: (agent-future, self, agent-future) : Agent-future nodes receive updates from other agent-future nodes belonging to the same trajectory, i.e. the same mode. For greater visibility only nodes for k = 0 are depicted.

Figure 13: (lane, gives-traffic-info, agent-future) : Lane nodes are connected to agent future nodes within a radius of 150m.

Figure 14: (point, gives-traffic-info, agent-future) : Point nodes are connected to agent-future nodes within a radius of 10m.

Figure 15: (agent-future, social, agent-future) : Agent-future nodes are connected to neighboring traffic participants in the same joint mode and future interval within a 150m radius. Only k = 0 is depicted for simplicity.

Figure 16: (agent-future, mode, agent-future) : Agent-future nodes belonging to the same future interval and traffic participant are connected across different modes.

Multimodal Trajectory Prediction

Multimodal trajectory prediction estimates several plausible future paths for people, vehicles, and other agents from motion and scene context.

See all 56 papers

PanguMotion: Continuous Driving Motion Forecasting with Pangu Transformers
Mar 17, 2026Quanhao Ren, Yicheng Li, Nan SongTrajectory Prediction Vehicle Trajectory Prediction
School of Data Science, Fudan University
LAMP: Lane-Aligned Motion Primitives for Feasible Trajectory Prediction
Jun 25, 2026Sangjin Han, Hoseong Jung, Jeongtae Her +2Multimodal Trajectory Prediction Trajectory Prediction
Seoul National University · Hyundai Motor Company, Republic of Korea
Recall to Predict: Grounding Motion Forecasting in Interpretable Motion Bank
May 2, 2026Abhishek Vivekanandan, Ahmed Abouelazm, J. Marius ZöllnerMulti-Agent Trajectory Prediction Multimodal Trajectory Prediction
FZI Forschungszentrum Informatik · Karlsruhe Institute of Technology (KIT) Karlsruhe, Germany

Mar 17, 2026J/K move · Enter open · S save

PanguMotion: Continuous Driving Motion Forecasting with Pangu Transformers

Quanhao Ren, Yicheng Li, Nan Song

School of Data Science, Fudan University

Trajectory Prediction Vehicle Trajectory Prediction Spatiotemporal Forecasting

Open paper AskPDF arXiv

Motion forecasting is a core task in autonomous driving systems, aiming to accurately predict the future trajectories of surrounding agents to ensure driving safety. Existing methods typically process discrete driving scenes independently, neglecting the temporal continuity and historical context correlations inherent in real-world driving environments. This paper proposes PanguMotion, a motion forecasting framework for continuous driving scenarios that integrates Transformer blocks from the Pangu-1B large language model as feature enhancement modules into autonomous driving motion prediction architectures. We conduct experiments on the Argoverse 2 datasets processed by the RealMotion data reorganization strategy, transforming each independent scene into a continuous sequence to mimic real-world driving scenarios.

Abstract

Multimodal Trajectory Prediction

Explore similar work

PanguMotion: Continuous Driving Motion Forecasting with Pangu Transformers

LAMP: Lane-Aligned Motion Primitives for Feasible Trajectory Prediction

Recall to Predict: Grounding Motion Forecasting in Interpretable Motion Bank