Although existing candidate trajectory evaluators have substantially improved end-to-end planning accuracy, their underlying mechanism remains suboptimal. Model-based evaluators typically make scoring future-aware by explicitly predicting candidate-conditioned future states over the planning horizon, which we term rollout. However, rollout incurs high inference cost and accumulates prediction errors. We argue that planning does not require reconstructing a full future scene for each candidate, but only future-relevant information for reliable comparison. We therefore propose RFWorld, a Rollout-Free World model for end-to-end trajectory evaluation. RFWorld constructs a shared scene memory and shapes it into a future-readable representation via lightweight training-time temporal readout with semantic supervision, keeping training overhead low. At inference time, candidate trajectories directly query the shared memory via sparse spatial readout for scoring, reducing computation by roughly the planning horizon times the number of candidate trajectories, while avoiding recursive rollout errors. Experiments on NAVSIM and Bench2Drive show that RFWorld learns a more planning-aligned world representation and achieves state-of-the-art performance.
Figures & tables
Figure 1: Two paradigms for future-aware trajectory evaluation. Conventional methods evaluate each candidate by rolling out its future states separately. In contrast, RFWorld constructs a shared scene memory once and lets all candidate trajectories read from it directly for rollout-free scoring.
Figure 2: Overall framework of RFWorld. The current BEV representation is transformed into a shared memory. During training, temporal readout with semantic supervision shapes the memory into a future-aware representation. During inference, candidate trajectories directly query the shared memory for reward prediction, and the trajectory with highest score is selected as the final planning output.
Method
WM
Supervision
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
Human
–
–
100.0
100.0
99.9
100.0
87.5
94.8
Ego Status MLP
×
Human
93.0
77.3
83.6
100.0
62.8
65.6
VADv2 [ 3 ]
×
Human
97.2
89.1
91.6
100.0
76.0
80.9
UniAD [ 16 ]
×
Human
97.8
91.9
92.9
100.0
78.8
83.4
LTF [ 5 ]
×
Human
97.4
92.8
92.4
100.0
79.0
83.8
Transfuser [ 5 ]
×
Human
97.7
92.8
92.8
100.0
79.2
84.0
Table 1: Comparison with state-of-the-art end-to-end driving methods on NAVSIM. The best results are denoted in bold . WM indicates whether the method explicitly employs a world model. For fair comparison, we report the scores using the same ResNet-34 backbone.
Method
Success Rate (%) ↑
Driving Score ↑
AD-MLP [ 43 ]
0.00
18.05
UniAD [ 16 ]
16.36
45.81
VAD [ 19 ]
15.00
42.35
TCP [ 36 ]
30.00
59.90
RFWorld
31.36
62.38
Table 2: Closed-loop results on the official 220-route Bench2Drive [ 18 ] test set.
WM
Future Sup.
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
×
×
98.7
94.4
95.2
99.8
83.4
88.9
×
✓
98.8
97.1
95.4
100.0
83.7
89.4
✓
×
98.5
96.5
95.1
99.8
82.3
88.4
✓
✓
99.0
97.8
96.0
100.0
84.6
90.5
Table 3: Ablation study on world modeling and future supervision. “WM” denotes whether the world model branch is enabled, and “Future Sup.” denotes whether future readout features are explicitly supervised during training. The best results are shown in bold .
Method
#Traj.
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
Latency ↓
WoTE [ 24 ]
256
98.4
96.7
94.6
99.9
82.3
88.3
37.0 ms
WoTE [ 24 ]
512
98.3
96.3
94.5
99.8
83.4
88.4
41.2 ms
WoTE [ 24 ]
1024
98.4
97.1
94.8
100.0
82.5
88.7
49.2 ms
RFWorld
256
98.8
97.0
95.7
99.9
83.9
89.6
35.1 ms
RFWorld
512
98.9
97.7
95.5
100.0
84.4
90.2
36.2 ms
RFWorld
1024
99.0
97.8
96.0
100.0
84.6
90.5
37.5 ms
Table 4: Ablation study on inference-time candidate anchors. “#Traj.” denotes the number of candidate trajectories randomly sampled at test time. Latency is measured per sample on a single RTX 3090 GPU. The best results are shown in bold .
Figure 3: Qualitative comparison between WoTE [ 24 ] and RFWorld on NAVSIM-v1. Legend from DriveDPO [ 29 ] . (a) RFWorld avoids leaving the drivable area and remains within the lane boundaries. (b) RFWorld decelerates in advance and safely avoids the pedestrian. (c) RFWorld stays close to the lane centerline and avoids potential collision with the adjacent-lane vehicle. (d) RFWorld better accounts for oncoming traffic and yields before turning, thereby avoiding potential future risk.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
#Future Sup.
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
0
98.5
96.5
95.1
99.8
82.3
88.4
2
98.8
97.0
95.7
100.0
83.7
89.5
4
99.0
97.3
95.9
99.9
84.0
90.0
6
99.0
97.4
96.2
99.9
84.3
90.3
8
99.0
97.8
96.0
100.0
84.6
90.5
Appendix
Table 5: Ablation study on the number of supervised future timestamps. “#Future Sup.” denotes the number of future timestamps used for semantic supervision during training. “0” corresponds to the setting with world model but without future supervision. The best results are shown in bold .
Figure 4: Visualization of the readout semantic maps during training. The top row shows the semantic maps predicted from the time-conditioned readout features, and the bottom row shows the corresponding ground-truth semantic maps. From left to right, the columns correspond to four future timestamps at T=1s , 2s , 3s , and 4s . Red dashed boxes highlight representative dynamic regions to facilitate comparison of their temporal evolution.
Figure 5: Channel-wise analysis of temporal readout gates. Left: gate differences between each future timestamp and the current timestamp. Right: Top-20 channels ranked by gate value at each timestamp.
Figure 6: Distributional analysis of temporal readout gates. Left: aggregated statistics of gate values at each timestamp. Right: channel-wise variance of gate values at each timestamp.
World models in end-to-end autonomous driving predict future scene evolution to provide foresight for trajectory planning. Existing methods mainly study how to predict the future and how to use it, but less often ask which future representation is actually most useful for planning. To this end, we propose PlanWAM, a Planning-Shaped World Action Model. The key idea is to let the planning task shape the future-state representation, so that it retains the information most useful for planning. A latent world model then predicts this planning-shaped future latent representation from historical observations and uses it for planning, enabling foresighted planning. Specifically, we first use a Temporal Register Pyramid to compress multi-frame historical information in a recency-aware manner, learning a compact history representation oriented toward future reasoning and planning. We then introduce a privileged future posterior branch that observes ground-truth future frames, and shape its future latent representation with trajectory-planning objectives to obtain a planning-shaped future latent representation. Hindsight-to-Foresight Distillation trains a prior branch that depends only on history to predict this future latent representation. The predicted future latent representation serves as planning context and guides trajectory generation and selection. PlanWAM achieves 93.8 PDMS / 90.9 EPDMS on NAVSIM-v1/v2 navtest and reaches 38.7 HD-Score on closed-loop HUGSIM in a zero-shot setting, demonstrating leading planning performance across both open-loop and closed-loop evaluations. Extensive experiments further demonstrate that planning-shaped future representations provide an effective and deployable form of foresight for world-action models.
Jinchang Xu, Hongda Yu, Fengwei Dong +9
Southeast University · NIO · Nanyang Technological University +1
Autonomous driving requires choosing a safe and efficient plan as surrounding traffic evolves. Generate-and-select planners propose multiple trajectories and score them for execution, and they have outperformed representative direct-prediction baselines on NAVSIM. Their scorer must compare plans that were never executed. Driving logs record the future of only the executed trajectory, so matching the logged future can leave predictions for the alternatives unconstrained; a simulator, in contrast, can label the outcome of every candidate. We introduce World4Scorer, which builds the scorer as a trajectory-conditioned JEPA-style predictor: it predicts a state for each candidate and reads the candidate's scores from that state. Simulator outcome labels supervise the states of all candidates, and the observed future of the executed trajectory anchors the predictor to real scene evolution. Because one predictor produces every candidate's state, the anchor can constrain shared parameters used to score unexecuted plans, while the future itself is needed only during training. Generated candidates mostly score well, so a scene-matched bank adds low-scoring plans to the outcome supervision; framewise choices can conflict, so inertial re-ranking keeps consecutive selections consistent. World4Scorer achieves state-of-the-art NAVSIM-v2 performance and a strong adapted-system result on closed-loop Bench2Drive. With the LeWM world model and planning budget fixed, outcome-based scoring also improves manipulation planning on the OGBench-Cube benchmark.
Jieyuan Pei, Meiyi Lu, Sining Ang +12
Institute for AI Industry Research (AIR), Tsinghua University · HiThink Research · The Hong Kong University of Science and Technology (Guangzhou) +8
Driving world models learn rich predictive representations of the surrounding environment from visual observations, yet accurate visual prediction does not necessarily translate into effective trajectory planning. We argue that a key bottleneck lies in the mismatch between visual world states and raw geometric trajectories, which may limit the planner's ability to exploit action-relevant semantics encoded by the world model. To address this issue, we propose World-Model Alignment for Latent Trajectories (WALT), which learns a compact generative trajectory latent space by transferring information from a frozen pretrained driving world model without modifying the world model itself. Rather than directly generating raw waypoints, WALT maps them into compact representations through a dual-branch trajectory autoencoder and transfers semantic knowledge from the frozen visual world model into this trajectory space, encouraging the learned action representation to capture scene-level cues relevant to future motion and planning. Beyond our proposed formulation, we systematically study latent learning based on Joint-Embedding Predictive Architectures (JEPA) and feature alignment following Representation Alignment (REPA) to investigate how trajectory-only representation learning affects downstream planning. We evaluate WALT on the NAVSIM benchmarks. Relative to the raw-waypoint baseline, WALT improves PDMS from 89.4 to 89.8 on NAVSIMv1 and EPDMS from 87.3 to 87.9 on NAVSIMv2 while reducing trajectory planner FLOPs by 30.5%. These results suggest that preserving world representations while extracting action-relevant information provides an effective interface for world-model-based trajectory planning.
Mingkai Jia, Jiaxin Guo, Zhijian Shu +5
The Hong Kong University of Science and Technology. · Horizon Robotics. · The Chinese University of Hong Kong. +2