Accurate motion prediction is critical for safe autonomous driving, yet even strong forecasters leave residual errors in their multimodal proposals. Existing refiners compress complete proposals or update their segments sequentially, preventing the full proposed future from serving as explicit relational context for correction. We introduce FutureLift, a backbone-agnostic refiner that represents each multimodal proposal as a heterogeneous future graph, with spatially grounded temporal nodes connected to agent histories, map elements, alternative modes, and, for joint prediction, other agents future trajectories. Bidirectional temporal edges allow later evidence to revise earlier segments in a single pass, and the updated nodes predict residual coordinate and confidence corrections. Requiring only proposals, scores, and scene context, FutureLift provides a common refinement interface for structurally different forecasting backbones. On Argoverse2 and the Waymo Open Motion Dataset, it consistently improves frozen proposals and outperforms native and specialized refiners across marginal and joint forecasting. On the WOMD motion-prediction leaderboard, FutureLift ranks first in minADE (0.5049m) and minFDE (0.9942m) among the compared standard-input methods, without LiDAR or ensembling, while achieving a competitive 0.4874 Soft mAP.
Figures & tables
Figure 1: Refinement representations. Top: Trajectory-level methods compress proposal geometry into a shared representation. Middle: Causal segment-wise methods pass information forward, so later evidence cannot reach earlier segments within a forward-only sweep. Bottom: FutureLift lifts proposal segments into spatial nodes with bidirectional temporal relations, enabling later context to correct earlier segments in one graph pass. Social future relations are used only for coherent joint proposals.
Figure 2: FutureLift method overview. (a) Scene history and map context flow through an arbitrary frozen forecasting backbone, which exposes only multimodal trajectory proposals Y and logits ℓ ; FutureLift does not access backbone queries or intermediate features. (b) Each proposal is partitioned into R intervals and lifted into spatially anchored future nodes. For clarity, the panel visualizes one representative mode k of the K instantiated modes. Typed relations connect the full node set to agent histories, map context, other modes, and for coherent joint-scenes proposals also to other predicted agents, while bidirectional temporal edges expose the complete horizon before correction. (c) Relation-specific attention updates all future nodes on this fixed graph and simultaneously decodes coordinate residuals, uncertainty parameters, and confidence residuals. Thus, later evidence can revise earlier segments in one graph pass without a causal sweep or graph reconstruction.
Backbone
Refiner
b-minFDE 6 ↓
minFDE 6 ↓
minADE 6 ↓
MR 6 ↓
QCNet [ 33 ]
None
1.912
1.304
0.729
0.164
Native
1.856
1.253
0.720
0.157
SmartRefine †
1.852
1.258
0.718
0.157
FutureLift
1.772
1.124
0.593
0.131
DenseTNT [ 6 ]
None
2.231
1.624
0.964
0.233
SmartRefine
2.109
1.553
0.834
0.221
Table 1: Backbone-interface compatibility on the AV2 marginal validation split. Displacement errors are in meters; runtime is reported in Tab. 8 . † SmartRefine result using the authors’ matched-checkpoint export and backbone trajectory features.
Backbone
Refiner
avgBrier- minFDE 6 ↓
avgMin FDE 6 ↓
avgMin ADE 6 ↓
Actor MR 6 ↓
FJMP [ 17 ]
None
2.478
1.921
0.812
0.237
SRefiner †
2.354
1.736
0.747
0.221
FutureLift
2.114
1.512
0.699
0.187
Forecast- MAE [ 1 ]
None
2.412
1.642
0.713
0.195
SRefiner †
2.251
1.477
0.658
0.183
FutureLift
2.012
1.391
0.642
0.169
Table 2: Backbone-interface compatibility on the AV2 joint validation split. Displacement errors are in meters; runtime is reported in Tab. 8 . † Published result using the SRefiner authors’ matched-backbone proposal export and additional per-mode embeddings.
SD after ± is reported in units of 10−3 where shown.
Configuration
avgBrier- minFDE 6 ↓
avgMin FDE 6 ↓
avgMin ADE 6 ↓
Actor MR 6 ↓
No refinement
1.944
1.262
0.584
0.162
+ History grounding
1.924
1.253
0.581
0.158
+ Map grounding
1.886
1.216
0.572
0.154
+ Cross-mode grounding
1.862
1.204
0.570
0.153
+ Causal temp. reasoning x1
1.857
1.199 ±1.5
0.569
0.152 ±0.1
Table 4: FutureLift component build-up on the AV2 joint validation split using one frozen DONUT checkpoint. Context relations are added cumulatively; the temporal rows compare J repeated causal sweeps (x J ) with one bidirectional sweep.
SD after ± is reported in units of 10−3 where shown.
Resolution
avgBrier- minFDE 6 ↓
avgMin FDE 6 ↓
avgMin ADE 6 ↓
Actor MR 6 ↓
Latency (ms)
R=1 (6 s / node)
1.884
1.221
0.599
0.161
59
R=2 (3 s / node)
1.827
1.162
0.574
0.153
76
R=3 (2 s / node)
1.816
1.142 ±1.0
0.552 ±0.8
0.146
90
R=6 (1 s / node)
1.810
1.139 ±0.9
0.551 ±0.6
0.144
136
Table 5: Temporal-resolution sensitivity on the AV2 joint validation split. The default setting is R=3 .
Figure 3: Representative failure cases. Left: The frozen proposal set omits the correct maneuver; post-hoc refinement cannot recover a mode absent from its inputs. Right: FutureLift produces excessive curvature because its residual decoder does not explicitly enforce vehicle dynamics.
Method
b-minFDE 6 ↓ (m)
minFDE 6 ↓ (m)
minADE 6 ↓ (m)
MR 6 ↓
minFDE 1 ↓ (m)
minADE 1 ↓ (m)
QCNet [ 33 ]
1.91
1.29
0.65
0.16
4.30
1.69
SmartRefine [ 32 ]
1.86
1.23
0.63
0.15
4.17
1.65
DeMo [ 29 ]
1.84
1.17
0.61
0.13
3.74
1.49
Polaris [ 28 ]
1.80
1.15
0.62
0.13
3.78
1.53
DONUT [ 10 ]
1.79
1.16
0.63
0.14
4.06
1.66
NDPNet [ 7 ]
1.83
1.17
0.61
0.14
4.08
1.62
Table 6: Additional AV2 marginal test-set context. Asterisks denote ensembles. FutureLift is evaluated locally with the released test annotations and official evaluator because the leaderboard is closed to new submissions. These results complement, but are distinct from, the controlled validation comparisons in Tab. 1 .
Method
avgBrier-minFDE 6 ↓ (m)
avgMinFDE 6 ↓ (m)
avgMinADE 6 ↓ (m)
ActorMR 6 ↓
avgMinFDE 1 ↓ (m)
avgMinADE 1 ↓ (m)
FJMP [ 17 ]
2.59
1.89
0.81
0.23
4.00
1.52
Forecast-MAE [ 1 ]
2.24
1.55
0.69
0.19
3.33
1.30
RealMotion [ 22 ]
2.01
1.32
0.62
0.18
2.87
1.14
DeMo [ 29 ]
1.93
1.24
0.58
0.16
2.78
1.12
Polaris [ 28 ]
1.86
1.18
0.56
0.15
2.67
1.07
FutureLift (DONUT)
1.82
1.14
0.55
0.15
2.63
1.07
Table 7: Additional AV2 joint test-set context. FutureLift is evaluated locally with the released test annotations and official evaluator because the leaderboard is closed to new submissions. These results complement, but are distinct from, the controlled validation comparisons in Tab. 2 .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Representative FutureLift corrections on the AV2 validation split. Each panel compares a frozen backbone proposal with the corresponding FutureLift-refined hypothesis and ground truth. FutureLift corrects lateral drift, curved-lane misalignment, and a local trajectory discontinuity.
Dataset / task
Backbone
Refiner
Params (M)
GFLOPs
Mean latency (ms)
AV2 marginal
QCNet
Native
7.7
55.8
95
AV2 marginal
QCNet
SmartRefine
8.0
56.2
112
AV2 marginal
QCNet
FutureLift
13.8
84.4
131
AV2 joint
FJMP
SRefiner
5.9
45.6
83
AV2 joint
FJMP
FutureLift
13.1
78.0
98
AV2 joint
DONUT
FutureLift
12.2
94.6
110
Appendix
Table 8: Refinement-only efficiency under the common protocol in Sec. A.4 . Latency includes graph construction and is averaged over 10 timed runs.
J
b-minFDE 6 ↓
minFDE 6 ↓
minADE 6 ↓
MR 6 ↓
Latency (ms) ↓
0
1.875
1.235
0.644
0.162
69
1
1.743
1.091
0.584
0.124
90
2
1.771
1.135
0.602
0.133
105
3
1.797
1.167
0.621
0.141
120
Appendix
Table 9: Effect of the number of refinement passes J on the AV2 marginal validation split. Displacement errors are in meters, and latency is measured on an NVIDIA H100 GPU.
Figure 5: Example scene
Figure 6: (point, belongs-to, lane) : All points belonging to the same lane, are connected to their respected lane node.
Figure 7: (lane, to, lane) : Lane nodes are connected to all neighbors within a radius of 150m.
Figure 8: (agent, self, agent) : Agent nodes are connected to up to 10 subsequent neighbors.
Figure 9: (lane, gives-traffic-info, agent) : Lanes are connected to all agents within a radius of 150m. For better visibility only edges to the green vehicle are depicted.
Figure 10: (agent, social, agent) : Agent nodes are connected to agent nodes from neighboring traffic participants at the concurrent timestamp within a 50m radius.
Figure 11: (agent, self, agent-future) : Agent-future nodes receive information from up to 30 previous agent nodes.
Figure 12: (agent-future, self, agent-future) : Agent-future nodes receive updates from other agent-future nodes belonging to the same trajectory, i.e. the same mode. For greater visibility only nodes for k = 0 are depicted.
Figure 13: (lane, gives-traffic-info, agent-future) : Lane nodes are connected to agent future nodes within a radius of 150m.
Figure 14: (point, gives-traffic-info, agent-future) : Point nodes are connected to agent-future nodes within a radius of 10m.
Figure 15: (agent-future, social, agent-future) : Agent-future nodes are connected to neighboring traffic participants in the same joint mode and future interval within a 150m radius. Only k = 0 is depicted for simplicity.
Figure 16: (agent-future, mode, agent-future) : Agent-future nodes belonging to the same future interval and traffic participant are connected across different modes.
Motion forecasting is a core task in autonomous driving systems, aiming to accurately predict the future trajectories of surrounding agents to ensure driving safety. Existing methods typically process discrete driving scenes independently, neglecting the temporal continuity and historical context correlations inherent in real-world driving environments. This paper proposes PanguMotion, a motion forecasting framework for continuous driving scenarios that integrates Transformer blocks from the Pangu-1B large language model as feature enhancement modules into autonomous driving motion prediction architectures. We conduct experiments on the Argoverse 2 datasets processed by the RealMotion data reorganization strategy, transforming each independent scene into a continuous sequence to mimic real-world driving scenarios.
Motion forecasting is essential for autonomous driving systems to enable safe decision-making and planning in complex driving scenarios. While existing predictors excel at minimizing standard displacement errors, they often overlook the adherence to lane topology of multimodal predictions, particularly for lower-probability modes. Consequently, predicted trajectories may violate physical and logical constraints, making the prediction set unreliable for safety-critical planning. In this paper, we propose LAMP (Lane-Aligned Motion Primitives), a topology-aware forecasting framework that anchors multimodal prediction to structured motion primitives aligned with lane topology. Specifically, we use a VQ-VAE to learn shape-aware motion primitives as discrete intention queries, capturing spatiotemporal patterns beyond endpoint-based intentions. We further introduce a feasibility-aware intention selector trained with a lane-topology prior for filtering unreachable intention queries, guiding the decoder to prioritize topology-consistent intentions while preserving behavioral diversity. Extensive experiments on the Argoverse 2 dataset demonstrate that LAMP achieves prediction accuracy comparable to state-of-the-art baselines while outperforming them in feasibility and diversity metrics.
Sangjin Han, Hoseong Jung, Jeongtae Her +2
Seoul National University · Hyundai Motor Company, Republic of Korea
Motion forecasting often requires trading interpretability for predictive accuracy. Standard anchor-based architectures rely on opaque latent queries that are highly prone to latent collapse, or naive trajectory sampling that limits multi-modal diversity. We propose an end-to-end differentiable framework that grounds predictions in a comprehensive "motion bank", a structured embedding space of physically realizable trajectories constructed via contrastive learning. Rather than regressing paths from a blank slate, our architecture dynamically retrieves explicit motion priors using a novel Anchor Retrieval Layer. This module adapts orthogonally initialized queries via a Dual-Level Gated Cross-Attention mechanism and executes discrete trajectory selection using a Straight-Through Gumbel-Softmax estimator to preserve continuous gradient flow. The retrieved semantically grounded anchors are then geometrically refined by a DETR-style decoder, optimized jointly with a Winner-Takes-All (WTA) kinematic Gaussian Mixture Model (GMM), a latent diversity penalty, and a soft-min weighted endpoint loss. By strictly conditioning the decoding phase on diverse, interpretable motion primitives, our approach eliminates the "black box" of standard latent queries while achieving competitive multi-modal accuracy on the Argoverse 2 and Waymo Open Motion datasets. Code is available at: https://github.com/abviv/recall2predict
Abhishek Vivekanandan, Ahmed Abouelazm, J. Marius Zöllner
FZI Forschungszentrum Informatik · Karlsruhe Institute of Technology (KIT) Karlsruhe, Germany