Plan-conditioned driving world models are used to compare candidate plans, so their forecast should change with the plan as the real scene would, yet widely used control metrics check only whether the camera follows the command. Recorded drives already contain real outcomes of plans that were never driven: retiming a log gives pose-matched images and LiDAR for other speeds, and a stereo rig records a laterally shifted view. Against these references, released world models move their camera but barely move the scene: their scene response, the share of the real change in distance that the forecast reproduces, reaches at most about a third of real video's, and the trajectory score cannot tell an executed speed command from an ignored one. Interventions inside the generator trace the failure to the plan's channel. The forecast geometry follows the context frames; once they carry the planned geometry, plan tokens add little. ReTime-Compose therefore delivers the plan as geometry, without training. It transports the frames of one logged-speed rollout to the planned cameras, and the frozen model completes only what no frame covers. Frozen generators then follow a 0.54 m lateral shift nearly as the real scene does on KITTI, nuScenes and Argoverse 2, and respond more than re-sampling to slow-down plans in three generators and to speed-up plans in two.
Figures & tables
Figure 2: The trajectory score does not rank the response to a plan. KITTI. Coloured: released world models (hollow: DriveLaW, exploratory adapter); grey: fine-tuned variants. Horizontal: Orbis 2’s scale-aligned ADE [ 26 ] ; vertical: scene response α (Eq. 2 ). Hatched yellow band: real pose-matched video; dash-dot line: rigid transport of the context depth, no learning. Red bracket: models with nearly the same score span the whole range of response. Bars: 95% drive-cluster intervals.
Figure 3: ReTime-Compose: the plan acts through transport, and the frozen generator completes only what transport cannot supply. (a) One frozen rollout at the logged speed. (b) Read depth and poses transport its frames from the rollout cameras (gray) to the planned ones (blue); the plan’s geometry enters only here. (c) Covered tokens are pinned, the rest generated; a refiner reduces artefacts.
Figure 4: How the scene response α is read. Half against logged speed, KITTI; each point is a near-field LiDAR point at one step (6000-point sample). α is the least-squares slope (lines: all points of 59 contexts); dashed: a forecast that changes as the world does. Real video reads .65, not 1, as the depth reader compresses depth.
Evaluation
compares
real outcome
ACT-Bench [ 1 ]
ego action vs. command
none
DrivingGen [ 65 ] , Orbis 2 [ 26 ]
ego path vs. command
none
CameraCtrl [ 15 ] , AC3D [ 3 ]
camera vs. command
none
WorldLens [ 23 ]
planner, geometry
logged plan
α (ours)
scene vs. LiDAR
retimed, stereo
Table 1: What control evaluations compare. Without the real outcome of an undriven plan, they check the camera, not the scene.
Figure 5: The scale-aligned score does not fall as a real video executes more of the command; the response does. 92 KITTI contexts. Left: response α against the executed fraction λ . Right: Orbis 2’s scale-aligned ADE (solid) and the unaligned error (dashed); App. C .
Figure 6: Under a slow-down plan, ReTime-Compose shows static content at the planned pose. Real pose-matched video, the released model, re-sampling and ReTime-Compose at steps 16 and 24 of the 0.5× plan; right: step 24 enlarged. Dashed line: the bottom of the real lead car; marks: whether the generated car ends there or closer (more: App. Fig. 8 ).
Figure 7: Completion takes ReTime-Compose off the response–noise trade-off (validation). Re-sampling lowers the noise, the composite raises the response, and completion brings the noise below re-sampling’s at both speeds (arrow: the change from re-sampling).
Change
Δα
ΔN
camera from the command
− .14 / − .14
+.64 / +.92
GEM, test sets 3–4 (41)
− .27 / –
− .22 / –
1 source, not 5
− .02 / +.02
+.19 / +.36
last real frame placed first
− .01 / +.06
+1.28 / +2.24
other vehicles on the plan clock
+.01 / +.02
+.00 / +.21
no completion (vs. unrefined)
+.13 / +.05
+.50 / +.73
Table 6: Undoing a step costs response or noise. Ablated minus full method at 0.5× / 1.5× ; negative Δα and positive ΔN are worse; bold: 95% drive-cluster interval excludes zero. Epona validation set unless noted (full table with intervals: App. Table 8 ).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Speed plans, one exploration context (chosen as the context whose slow-down gain over re-sampling is closest to the set’s median). Rows: real pose-matched video, re-sampling the rollout, ReTime-Compose before the refiner, and the final method, for the 0.5× and 1.5× plans at steps 8, 16, 24. The composite places static content at the planned pose; the refiner reduces the splat artefacts the completion would otherwise reproduce (image quality: Sec. 5.4 ).
Change
Failure it answers
Δα
ΔN
camera from the command, not read
executed = commanded
− .14 / − .14
+.64 [.14,1.80] / +.92 [.39,1.81]
1 source, not 5
per-source geometry error
− .02 / +.02
+.19 [.01,.50] / +.36 [.11,.82]
last real frame placed first
arm-specific magnification
− .01 / +.06
+1.28 [.52,2.88] / +2.24 [1.07,4.64]
other vehicles on the plan clock
agents keep physical time
+.01 / +.02
+.00 / +.21 [.04,.50]
no completion §
generated holes
+.13 / +.05
+.50 [.24,.91] / +.73 [.20,1.70]
no pinning
generator ignores the plan
− .02 / − .01
− .05 / +.34 [.01,.90]
Appendix
Table 8: Component and joint-safeguard ablations (final configuration; Epona validation set, 30 contexts; Orbis 2 validation set, 32 contexts). Change in response α and noise N at 0.5× / 1.5× , ablated minus full method, 95% drive-cluster intervals where they exclude zero; negative Δα and positive ΔN are worse. Each row undoes one design choice (“neither”: both protections of step 5); the second column names the failure that choice answers. † GEM test sets 3 and 4 (41 contexts, 9 drives), slow-down only (GEM’s scored plan). ‡ Paired with the frozen method on the same set: test set 3, 22 contexts, 8 drives; test set 5, 80 contexts, 25 drives. § The composite is unrefined, so it is paired with the full method before the refiner ( α / N .42/1.01 and .34/1.43); every other Epona row is paired with the refined output of Table 5 .
Set
Arm
0.5×
1.5×
Validation
re-sampling
.18 / 2.76
.23 / 3.49
ReTime-Compose
.57 / 1.54
.50 / 2.35
Test 1
re-sampling
.24 / 3.40
.15 / 3.20
ReTime-Compose
.58 / 2.27
.37 / 3.05
Test 2
re-sampling
.62 / 5.20
.28 / 3.56
ReTime-Compose
.80 / 4.00
.55 / 4.26
Appendix
Table 10: Second reader (Depth Anything V2 [ 60 ] for all arms; real video α .83–1.03); α / N , all steps.
Points
Arm
0.5×
1.5×
all static
real video
.92 / .24
.96 / .19
(87, 89 ctx)
re-sampling
.57 / 5.33
.54 / 2.98
ReTime-Compose
.86 / 1.29
.89 / 1.41
obstacle
real video
.96 / .05
.97 / .06
(99, 105 ctx)
re-sampling
.70 / .84
.52 / .38
ReTime-Compose
.76 / .29
.74 / .25
Appendix
Table 11: Reader-free speed response ( α / N from image displacement; all five evaluation sets pooled, contexts with points in range).
Method
0.6× : EF / α
1.6× : EF / α
Fixed transport
−.04/.46
−.27/.39
Measurement hybrid
.13/.54
−.10/.46
Source-view completion
.12/.59
−.14/.52
Target view, no depth prior
.59/.81
.50/.78
Bounded head (Eq. 4 )
.62/.78
.55/.74
Appendix
Table 18: Paired-change response of the geometric forecaster. Near-field common LiDAR support, KITTI five folds, seed 0, relative to the logged plan; pooled EF and slope (Eq. 2 ).
Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks. We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales: a high-level predictor that forecasts coarse scene structure over extended temporal horizons, and a low-level generator that produces detailed predictions conditioned on the high-level output. This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations. We further show that pretraining with a diffusion forcing objective yields substantially richer internal representations than the standard teacher forcing objective, while teacher forcing -- predicting only the next frame from clean context -- produces more stable autoregressive rollouts. We therefore introduce a generic two-stage training paradigm that pretrains the model with diffusion forcing and fine-tunes with teacher forcing, combining the representational benefits of the former with the rollout stability of the latter. Our approach achieves state-of-the-art results across the standard suite of driving world model evaluations on established benchmarks, including long-horizon generation fidelity, steering responsiveness evaluated on counterfactual scenarios, and internal representation quality. Project page with code, demo, checkpoints and qualitative results: https://lmb-freiburg.github.io/orbis2.github.io/
Sudhanshu Mittal, Arian Mousakhan, Silvio Galesso +5
Today's driving world models can generate remarkably realistic dash-cam videos, yet no single model excels universally. Some generate photorealistic textures but violate basic physics; others maintain geometric consistency but fail when subjected to closed-loop planning. This disconnect exposes a critical gap: the field evaluates how real generated worlds appear, but rarely whether they behave realistically. We introduce WorldLens, a unified benchmark that measures world-model fidelity across the full spectrum, from pixel quality and 4D geometry to closed-loop driving and human perceptual alignment, through five complementary aspects and 24 standardized dimensions. Our evaluation of six representative models reveals that no existing approach dominates across all axes: texture-rich models violate geometry, geometry-aware models lack behavioral fidelity, and even the strongest performers achieve only 2-3 out of 10 on human realism ratings. To bridge algorithmic metrics with human perception, we further contribute WorldLens-26K, a 26,808-entry human-annotated preference dataset pairing numerical scores with textual rationales, and WorldLens-Agent, a vision-language evaluator distilled from these judgments that enables scalable, explainable auto-assessment. Together, the benchmark, dataset, and agent form a unified ecosystem for assessing generated worlds not merely by visual appeal, but by physical and behavioral fidelity.
Recently, world-action models (WAM) have emerged to bridge vision-language-action (VLA) models and world models, unifying their reasoning and instruction-following capabilities and spatio-temporal world modeling. However, existing WAM approaches often focus on modeling 2D appearance or latent representations, with limited geometric grounding-an essential element for embodied systems operating in the physical world. We present DriveDreamer-Policy, a unified driving world-action model that integrates depth generation, future video generation, and motion planning within a single modular architecture. The model employs a large language model to process language instructions, multi-view images, and actions, followed by three lightweight generators that produce depth, future video, and actions. By learning a geometry-aware world representation and using it to guide both future prediction and planning within a unified framework, the proposed model produces more coherent imagined futures and more informed driving actions, while maintaining modularity and controllable latency. Experiments on the Navsim v1 and v2 benchmarks demonstrate that DriveDreamer-Policy achieves strong performance on both closed-loop planning and world generation tasks. In particular, our model reaches 89.2 PDMS on Navsim v1 and 88.7 EPDMS on Navsim v2, outperforming existing world-model-based approaches while producing higher-quality future video and depth predictions. Ablation studies further show that explicit depth learning provides complementary benefits to video imagination and improves planning robustness.