VectorWorld: Efficient Streaming World Model via Diffusion Flow on Vector Graphs

Chaokang Jiang; Desen Zhou; Jiuming Liu; Kevin Li Sun

cs.ROMar 18, 2026

VectorWorld: Efficient Streaming World Model via Diffusion Flow on Vector Graphs

Authors: Chaokang Jiang, Desen Zhou, Jiuming Liu, Kevin Li Sun

Organizations: Bosch, Shanghai, Shanghai, China · University of Cambridge, Cambridge, United Kingdom

Abstract

Closed-loop evaluation of autonomous-driving policies requires interactive simulation beyond log replay. However, existing generative world models often degrade in closed loop due to (i) history-free initialization that mismatches policy inputs, (ii) multi-step sampling latency that violates real-time budgets, and (iii) compounding kinematic infeasibility over long horizons. We propose VectorWorld, a streaming world model that incrementally generates ego-centric $64 \mathrm{m}\times 64\mathrm{m}$ lane--agent vector-graph tiles during rollout. VectorWorld aligns initialization with history-conditioned policies by producing a policy-compatible interaction state via a motion-aware gated VAE. It enables real-time outpainting via solver-free one-step masked completion with an edge-gated relational DiT trained with interval-conditioned MeanFlow and JVP-based large-step supervision. To stabilize long-horizon rollouts, we introduce $Δ$ Sim, a physics-aligned non-ego (NPC) policy with hybrid discrete--continuous actions and differentiable kinematic logit shaping. On Waymo open motion and nuPlan, VectorWorld improves map-structure fidelity and initialization validity, and supports stable, real-time $1\mathrm{km}+$ closed-loop rollouts (code).

Figures & tables

Figure 1: Comparison with Prior Works. Unlike prior pipelines that degrade in closed loop due to accumulated drift (autoregressive rollout) or history-free cold-start initialization, VectorWorldsupports kilometer-scale closed-loop simulation via streaming outpainting in our experiments. The engine operates in a closed loop: 0) An Ego Planner drives within the scene; 1) Initial Scene Generation creates the base tile; 2) Scene Outpainting dynamically extends the map frontier using edge-gated relational DiT; and 3) NPC Behavior ( $\Delta$ Sim) ensures reactive interactions. This mechanism enables consistent, kilometer-scale driving simulation by progressively outpainting the world ahead.

Figure 2: Interaction-state interface with motion-aware gated VAE. A motion-aware gate $g_{i}$ fuses static state $s_{i}$ and motion code $m_{i}$ , suppressing history noise for stationary agents while preserving dynamics for moving ones. A factorized scene transformer aggregates typed relations (L2L/L2A/A2A) and outputs Gaussian latents $z=(z_{\mathrm{lane}},z_{\mathrm{agent}})$ for downstream generation. Warm-started interaction states align initialization with history-conditioned policies.

Figure 3: Edge-gated relational DiT for vector-graph latent generation. The framework generates vector-graph latents via a factorized transformer backbone. To capture complex graph dependencies, we introduce a Relational Attention Module (Right). Unlike standard attention, this module conditions message passing on edge features $\mathbf{e}_{ij}$ via: 1) An additive bias $B_{ij}$ applied to attention scores to regulate connectivity; 2) A multiplicative gate $G_{ij}$ applied to value vectors to modulate feature aggregation. This design enables precise structural control across heterogeneous agents and map elements, compatible with various generative dynamics (DDPM, Flow, or MeanFlow).

Figure 4: $\Delta$ Sim: physics-aligned NPC policy. A single-pass return-to-go (RTG) control embedding modulates the decoder via FiLM $(\gamma,\beta)$ . Actions use a hybrid head (discrete k-disks token plus continuous residual) with differentiable kinematic logit shaping and DKAL regularization. This design reduces feasibility violations that compound under kilometer-scale closed-loop simulation.

Dataset	Method	Perceptual	Map Structure		Safety
Dataset	Method	FD $\downarrow$	Route Len. (m) $\uparrow$	Endpt. Dist. (m) $\downarrow$	Coll. Rate (%) $\downarrow$
nuPlan	SLEDGE-L ( Chitta et al., 2024 )	1.89	35.34 $\pm$ 8.63	0.470 $\pm$ 0.320	22.30
	SLEDGE-XL ( Chitta et al., 2024 )	1.44	35.83 $\pm$ 8.35	0.420 $\pm$ 0.290	21.20
	ScenDream-B ( Rowe et al., 2025 )	1.05	37.04 $\pm$ 10.21	0.320 $\pm$ 0.800	11.90
	ScenDream-L ( Rowe et al., 2025 )	0.67	36.87 $\pm$ 10.37	0.250 $\pm$ 0.710	9.30
	VectorWorld (Ours)	0.98	37.22 $\pm$ 11.40	0.078 $\pm$ 0.106	3.01

Table 1: Quantitative comparison of scene generation quality. We evaluate VectorWorld against prior vectorized scene generators on nuPlan and Waymo. VectorWorld achieves strong structure and safety with competitive perceptual alignment.

\downarrow

\uparrow

indicates lower/higher is better. Bold and underline denote the best and second-best results among non-privileged methods.

Figure 5: Qualitative comparison on Waymo and nuPlan. Visualizing decoded elements in $64\,\mathrm{m}\times 64\,\mathrm{m}$ tiles. VectorWorld demonstrates superior lane connectivity and fewer agent overlaps at initialization compared to prior vectorized generators.

Figure 6: Quality–latency trade-off for tile generation. Endpoint distance (EPD) and mean agent-JSD are plotted against wall-clock latency as the solver-step budget varies; the vertical marker indicates the streaming budget (including one VAE decode).

Simulator	NPC Policy	Ego Planner	Sys. Cap.		Scene Val.		Ego Performance				Dyn. Metric
Simulator	NPC Policy	Ego Planner	Online	Warm	FD $\downarrow$	S.Coll $\downarrow$	Coll.	Off.	Succ.	Route (m)	Value
I. Platform Stability & Consistency (Goal: Verify simulation realism using a standard IDM driver ( Treiber et al., 2000 ) .)
Waymo	Replay	IDM (Rule-based)	✗	✗	0.00	–	7.2	5.8	87.0	64	11.4 (Jerk)
ScenDream-L	Ctrl-sim	IDM (Rule-based)	✗	✗	1.38	4.80	12.4	5.9	81.7	200	15.5 (Jerk)
VectorWorld	$\Delta$ Sim	IDM (Rule-based)	✗	✗	1.05	4.69	7.5	1.5	91.0	200	10.4 (Jerk)
VectorWorld	$\Delta$ Sim	IDM (Rule-based)	✓	✗	1.19	7.39	45.5	6.9	47.6	1000	16.6 (Jerk)

Table 2: Closed-loop system evaluation on Waymo. Units: Scene Validity (FD, S.Coll) and Ego Performance (Coll, Offroad, Success) are in percentages (%) unless unitless. Route is in meters. Columns: NPC Policy denotes the background traffic controller; Ego Planner denotes the agent under test. Key: Warm = Motion-history warm start; Online = Streaming outpainting.

Method	Components			Acc.	Ctrl.	Effic.
Method	RTG	A+R	DKAL	ADE $\downarrow$	$\rho$ $\uparrow$	Lat. $\downarrow$	Fwd $\downarrow$
Ctrl-Sim ( Rowe et al., 2024 )	✗	✗	✗	2.80	-0.25	36.9	2.0
$\Delta$ Sim (w/o Refine)	✓	✗	✓	2.62	0.39	21.7	1.0
$\Delta$ Sim (w/o DKAL)	✓	✓	✗	2.10	0.24	26.1	1.0
$\Delta$ Sim (Ours)	✓	✓	✓	1.72	0.53	26.1	1.0

Table 3:

\Delta

Sim ablation on Waymo. Components: return-to-go conditioning (RTG), hybrid anchor+residual action head (A+R), and DKAL. Metrics: average displacement error (ADE;

\downarrow

), controllability rank-correlation (

\rho

;

\uparrow

), and inference cost (latency / forward passes;

\downarrow

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Type / shape	Meaning
$\mathcal{G}=(\mathcal{V},\mathcal{E})$	set-valued	Heterogeneous lane–agent vector graph for one ego-centric tile.
$\mathcal{V}_{\mathrm{lane}},\mathcal{V}_{\mathrm{agent}}$	sets	Lane nodes / agent nodes in $\mathcal{G}$ .
$N_{\ell}$	integer	Number of lane segments in a scene (capped by $N_{\ell}^{\max}$ ).
$N_{a}$	integer	Number of agents in a scene (capped by $N_{a}^{\max}$ ; includes ego).
$P$	integer	Number of points per lane polyline for the generator/autoencoder ( $P{=}20$ in Waymo generation).
$P_{\Delta}$	integer	Number of points per lane polyline for $\Delta$ Sim context ( $P_{\Delta}{=}50$ in Waymo behavior).

Appendix

Table 4: Core notation for the vectorized scene representation (per scene unless stated otherwise).

Symbol	Type / shape	Meaning
$\mathrm{Enc}_{\phi}$	function	VAE encoder mapping a scene graph to Gaussian posterior parameters.
$\mathrm{Dec}_{\psi}$	function	VAE decoder mapping sampled latents to reconstructed scene elements.
$\mathbf{z}^{\ell}$	$\mathbb{R}^{N_{\ell}\times d_{\ell}}$	Lane latents (Waymo: $d_{\ell}{=}24$ ).
$\mathbf{z}^{a}$	$\mathbb{R}^{N_{a}\times d_{a}}$	Agent latents (Waymo: $d_{a}{=}18$ ).
$\mathbf{z}$	$\mathbb{R}^{(\cdot)}$	Concatenated latents, $\mathbf{z}=(\mathbf{z}^{\ell},\mathbf{z}^{a})$ .
$\mathbf{z}_{t}$	$\mathbb{R}^{(\cdot)}$	Noisy latent on the rectified path at time $t$ : $\mathbf{z}_{t}=(1-t)\mathbf{z}+t\boldsymbol{\epsilon}$ .

Appendix

Table 5: Notation for latent variables and generative dynamics (VAE + diffusion/flow/MeanFlow).

Symbol	Type / shape	Meaning
$\pi_{\mathrm{ego}}$	policy	Ego planner/policy used in closed-loop rollout.
$\pi_{\Delta}$	policy	NPC policy implemented by $\Delta$ Sim.
$R$	scalar	Tile radius (ego-centered observation crop) used by the simulator.
$\tau$	scalar	Outpainting trigger distance to the tile boundary (Algorithm 2 ).
$\mathbf{V}_{\mathrm{kd}}$	$\mathbb{R}^{K_{\mathrm{kd}}\times 3}$	k-disks vocabulary of SE(2) deltas, $(\Delta x,\Delta y,\Delta\theta)$ , with $K_{\mathrm{kd}}{=}384$ .
$a_{k}$	integer	Discrete action token index at simulation step $k$ .

Appendix

Table 6: Notation for the

\Delta

Sim behavior model and closed-loop simulation.

Figure 7: Visualization of motion-aware gating (Waymo). Gate activation increases with motion magnitude, clustering static agents at low activation while assigning higher activation to dynamic agents. This supports that the motion-aware gated VAE suppresses motion-code noise for parked agents while encoding history for moving agents, improving warm-start consistency.

Hyperparameter	Motion-aware VAE	MeanFlow latent generator	$\Delta$ Sim behavior model
Optimizer	AdamW	AdamW	AdamW
Base learning rate	$1.0\times 10^{-4}$	$1.0\times 10^{-4}$	$6.0\times 10^{-5}$
$\beta_{1},\beta_{2}$	$(0.9,0.999)$	$(0.9,0.999)$	$(0.9,0.999)$
$\epsilon$	$1.0\times 10^{-7}$	$1.0\times 10^{-7}$	$1.0\times 10^{-7}$
Weight decay	$2.0\times 10^{-5}$	$1.0\times 10^{-5}$	$1.0\times 10^{-4}$
Training length	$60{,}000$ steps	$100{,}000$ steps	$10{,}000$ steps

Appendix

Table 7: Training hyperparameters on Waymo. Values are reported for the default setting used in our main experiments.

Dataset	Method	Topology $\downarrow$				Agent Distributional JSD $\downarrow$
Dataset	Method	Conn.	Dens.	Reach	Conve.	Near.	Lat.	Ang.	Len.	Wid.	Speed	Count
nuPlan	SLEDGE-L ( Chitta et al., 2024 )	2.34	2.43	0.70	2.44	0.53	0.63	3.60	11.84	10.16	0.46	0.109
	SLEDGE-XL ( Chitta et al., 2024 )	1.67	1.74	0.51	1.68	0.49	0.49	3.26	11.16	10.29	0.47	0.217
	ScenDream-B ( Rowe et al., 2025 )	0.28	0.45	0.07	0.14	0.12	0.15	0.17	0.22	0.14	0.07	0.023
	ScenDream-L ( Rowe et al., 2025 )	0.18	0.43	0.03	0.33	0.09	0.11	0.18	0.25	0.17	0.06	0.011
	Ours	0.09	0.41	0.09	0.79	0.26	0.31	0.18	0.35	0.22	0.07	0.004

Appendix

Table 8: Evaluation of map topology and agent distribution alignment. We report lane-graph Topology discrepancies (Conn., Dens., Reach, Conve.; lower is better) and Agent Distributional JSD for key attributes (lower is better). These metrics diagnose structural and density drifts that can be amplified under repeated inpainting/outpainting. Bold and underline denote the best and second-best results among non-privileged methods (excluding methods marked with * ).

VAE variant	Interface components				Autoencoding			Downstream init.
VAE variant	Motion code	Motion gate	Factorized scene tr.	Count prior	Static L1 $\downarrow$	Motion L1 $\downarrow$	KL $\downarrow$	EPD (m) $\downarrow$	Init coll. (%) $\downarrow$
Snapshot-only	✗	✗	✓	✓	0.012	–	5.23	0.120	9.45
Concat motion (no gate)	✓	✗	✓	✓	0.017	0.045	4.99	0.102	4.92
Gated	✓	✓	✓	✓	0.011	0.042	4.63	0.098	4.85
motion-aware gated VAE (ours)	✓	✓	✓	✓	0.008	0.038	4.37	0.094	4.69
Additional interface checks

Appendix

Table 9: Ablations of motion-aware gated VAE (Waymo). We evaluate the impact of the interaction-state interface on autoencoding fidelity and downstream initialization. Snapshot-only leads to high initialization collision rates due to the lack of motion history (cold start). Concat motion (naive) degrades static reconstruction by injecting noise into parked vehicles. Our Motion-aware Gated design achieves the best trade-off, selectively encoding momentum only for dynamic agents. Lower is better for all metrics.

Variant	Relation-aware modules				Perceptual	Map structure			Safety	Agent distributional JSD $\downarrow$
Variant	Rel. bias	Value gate	Cross-type bias	Global fusion	FD $\downarrow$	RouteLen. (m) $\uparrow$	Endpt. dist. (m) $\downarrow$	Route valid. (%) $\uparrow$	Coll. rate (%) $\downarrow$	Near.	Ang.	Len.	Wid.	Speed
Factorized DiT (Base)	✗	✗	✗	✗	1.53	35.26	0.301	87.36	6.53	0.12	0.09	0.51	0.29	0.39
+ Relational logit bias	✓	✗	✗	✗	1.35	37.10	0.185	92.10	5.80	0.11	0.09	0.42	0.25	0.35
+ Edge-gated values	✓	✓	✗	✗	1.20	37.85	0.142	94.50	5.12	0.10	0.08	0.30	0.18	0.28
+ Cross-type rel. bias	✓	✓	✓	✗	1.05	38.40	0.110	96.20	4.88	0.08	0.06	0.18	0.14	0.20
+ Global context (Ours)	✓	✓	✓	✓	0.94	39.03	0.094	98.10	4.69	0.07	0.06	0.13	0.11	0.15

Appendix

Table 10: Ablation of relation-aware components in edge-gated relational DiT (Waymo). We incrementally add relational modules to the factorized DiT backbone. Relational bias is critical for graph connectivity (Conn/Reach). Cross-type bias improves agent–lane geometric consistency (e.g., Ang. JSD) under the same map. Relational bias and value gating improve structural validity (e.g., route validity and endpoint distance) under large-step sampling. Our full model achieves the best structural integrity and safety.

Training recipe	Key design				Train GPU-h $\downarrow$	Steps at test	Waymo (same decoder)
Training recipe	MeanFlow ( $u_{\theta}$ )	JVP ( $\partial_{t}u$ )	Two-time ( $t,\Delta$ )	Distill ( 1-step )	Train GPU-h $\downarrow$	Steps at test	EPD (m) $\downarrow$	Agent JSD $\downarrow$	Time (ms) $\downarrow$	Valid (%) $\uparrow$
Flow matching	✗	✗	✗	✗	320	1-step	8.520	4.250	5.6	9.3
Flow matching	✗	✗	✗	✗	320	12-step	0.150	0.120	31.0	95.0
MeanFlow (Identity)	✓	✗	✗	✗	300	1-step	0.554	0.499	5.6	63.2
MeanFlow (Identity)	✓	✗	✗	✗	300	12-step	0.120	0.110	31.0	96.5
MeanFlow (Id + JVP)	✓	✓	✗	✗	400	1-step	0.293	0.312	5.6	68.0

Appendix

Table 11: Impact of the meanflow ( Geng et al., 2025 ) objective on solver-free sampling. We compare different training objectives under a fixed step budget. Training Cost is measured in GPU hours on 8

\times

H20. Standard Flow Matching (FM) fails completely at 1-step (EPD 8.52m). Distillation is fast but blurs details (High Agent JSD). Our MeanFlow (JVP+2-time) incurs higher training cost (+50%) but enables the only viable 1-step generation (EPD 0.269m) that preserves agent statistics (JSD 0.136), enabling real-time streaming.

Dynamics	Sampler / steps	Waymo			nuPlan
Dynamics	Sampler / steps	EPD (m) $\downarrow$	Agent JSD $\downarrow$	Time (ms) $\downarrow$	EPD (m) $\downarrow$	Agent JSD $\downarrow$	Time (ms) $\downarrow$
Flow (Rectified)	Flow-12	0.150	0.120	28	0.125	0.155	31
	Flow-24	0.110	0.105	55	0.095	0.140	55
	Flow-32	0.098	0.091	73	0.088	0.135	73
	Flow-48	0.089	0.084	110	0.082	0.130	110
MeanFlow	MeanFlow-1	0.269	0.136	5.6	0.224	0.171	6

Appendix

Table 12: Step-budget sensitivity of our latent generator (numeric companion). Detailed metrics for Figure 6 . While DDPM (100 steps) achieves the lowest EPD (Waymo: 0.080m; nuPlan: 0.072m), its latency (158ms) is the highest and limits online use. Our MeanFlow-5 provides a favorable balance between fidelity and latency, while MeanFlow-1 enables ultra-low latency streaming.

Method	Design toggles				Open-loop		Closed-loop feasibility (%) $\downarrow$				Efficiency
Method	RTG (1-pass)	Hybrid A+R	Logit shaping	DKAL loss	ADE (m) $\downarrow$	$\boldsymbol{\rho}$ $\uparrow$	Coll.	Yaw rate	Lat. acc.	Lon. jerk	Lat. (ms) $\downarrow$	Fwd $\downarrow$
Ctrl-Sim ( Rowe et al., 2024 )	✗	✗	✗	✗	2.80	-0.25	15.2	8.5	14.2	10.5	36.9	2.0
$\Delta$ Sim (discrete-only)	✓	✗	✓	✓	2.62	0.39	8.5	4.2	5.5	6.2	21.7	1.0
$\Delta$ Sim (hybrid A+R, no DKAL)	✓	✓	✓	✗	2.10	0.24	10.1	6.8	12.5	8.9	26.1	1.0
$\Delta$ Sim (hybrid A+R, no shaping)	✓	✓	✗	✓	1.95	0.45	5.2	2.5	3.1	4.5	26.1	1.0
$\Delta$ Sim (Ours)	✓	✓	✓	✓	1.72	0.53	2.9	1.2	2.1	3.2	26.1	1.0

Appendix

Table 13:

\Delta

Sim component analysis on accuracy, controllability, and feasibility (Waymo). We evaluate the impact of Hybrid Action Head (A+R) and Differentiable Kinematic Alignment Loss (DKAL). Ctrl-Sim baseline suffers from poor controllability (

\rho=-0.25

) and high latency. Our full

\Delta

Sim achieves the lowest ADE (1.72m) and highest controllability (

\rho=0.53

). Crucially, DKAL significantly reduces closed-loop physical violations (e.g., Lat. Acc. 12.5%

\to

2.1%), ensuring stable long-horizon rollouts.

Panel A: Fixed-horizon cross-backend evaluation (200 m).
Policy	Train backend	Train data	Train GPU-h	Eval: Log Replay		Eval: ScenDream-L (neg.)		Eval: VectorWorld (neg.)
Policy	Train backend	(#eps)		C/O/Succ (%)	Time (h)	C/O/Succ (%)	Time (h)	C/O/Succ (%)	Time (h)
PPO	Waymo Log	100k	64	29.3 / 6.9 / 63.8	0.4	59.0 / 9.0 / 32.1	5.8	65.5 / 9.5 / 25.0	1.2
PPO	ScenDream-L	100k	480	19.3 / 5.9 / 74.8	0.4	45.2 / 8.5 / 46.3	5.8	40.9 / 7.6 / 48.5	1.2
PPO	VectorWorld	100k	160	18.1 / 6.2 / 75.7	0.4	38.5 / 5.6 / 56.3	5.8	27.1 / 5.0 / 66.9	1.2
Panel B: Long-horizon evaluation (1 km $+$ ; VectorWorld only).

Appendix

Table 14: Policy training value with cross-backend evaluation (Waymo). To reduce the risk of simulator overfitting, we evaluate policies across different backends under matched horizons. A policy retrained in VectorWorld improves success not only in VectorWorld but also transfers to log replay and ScenDream backends (see Panel A). Panel B further evaluates long-horizon performance (

1\,\mathrm{km}+

) in VectorWorld.

Source of failures	Matching and reproduction diagnostics				Outcome agreement
Source of failures	#Fail episodes	Log match (%)	Init. state dist. (L2) $\downarrow$	Route match (%)	Fail repro. (%)	Type agreement (%)
Log replay (reference)	100	100.0	0.00	100.0	100.0	100.0
ScenDream-L (neg. tilt)	528	65.2	1.42	82.5	45.2	52.1
VectorWorld (neg. tilt)	655	88.5	0.92	95.4	72.0	78.5

Appendix

Table 15: Failure reproduction across backends. We verify if failures discovered in simulation correspond to realistic crash types. We match generated failure scenarios to the nearest log scenarios based on map and agent topology. VectorWorld failures show high Type Agreement (78.5%) with log data, indicating that our simulator exposes realistic safety-critical events rather than simulation artifacts.

Figure 8: Waymo qualitative comparison for initial scene generation (64m $\times$ 64m). We compare VectorWorld against representative vectorized scene generators and the Waymo reference. The visualization highlights lane connectivity at intersections, agent–lane geometric consistency, and the plausibility of the initialized interaction state (including short motion history when available).

Figure 9: nuPlan qualitative comparison with traffic-light semantics. Lane centerlines are colored by traffic-light state (green vs. red) to visualize control-relevant map semantics. We compare VectorWorld with strong baselines under the same vectorized rendering.

Figure 10: Agent controllability under fixed map conditioning. The map (lane graph) is held fixed as a condition, while the agent configuration is generated under different controllability settings. This visualization isolates controllability from map diversity by using the same conditioned lanes across all panels.

Figure 11: End-to-end generation latency (timeline view). Qualitative snapshots under increasing time budgets for a $64\,\mathrm{m}\times 64\,\mathrm{m}$ tile. VectorWorld reaches the final lane–agent vector scene with one solver-free MeanFlow step plus one VAE decode ( $\approx 6$ ms), whereas multi-step diffusion baselines only converge at substantially higher budgets (e.g., $\sim 160$ ms).

Figure 12: End-to-end generation latency. A different scene instance rendered with the same timing protocol (latent sampling + one VAE decode). The result is consistent: VectorWorld completes a full $64\,\mathrm{m}\times 64\,\mathrm{m}$ lane–agent tile in $\approx 6$ ms via a single MeanFlow update, while solver-based diffusion requires many iterative denoising calls.

Figure 13: Planner-relevant challenging scenarios sampled from VectorWorld. We render a subset of VectorWorld-generated scenes using the protocol in Section A.15 . These scenarios illustrate diverse intersection geometries, dense multi-agent interactions, and tight gaps that are valuable for stress testing and training an ego PPO planner in closed loop.

Figure 14: Streaming outpainting in closed-loop rollout (case study 1). The simulator repeatedly outpaints new lane–agent tiles near the frontier while clamping the observed region. The visualization highlights seam consistency and stable long-horizon progression.

Figure 15: Streaming outpainting with dynamic agent set and per-agent intent control (case study 2). The visualization illustrates stable long-horizon expansion with agent insertion/removal and controllable behaviors, enabled by the interaction-state interface .

Explore similar work

Jun 2, 2026

NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation

As autonomous vehicle capabilities advance, the safe evaluation of driving policies in long-tail scenarios remains a critical bottleneck. In closed-loop simulation, the driving policy model actively interacts with the environment, where its actions dynamically update the simulator state and directly influence the next set of generated sensor observations. While recent reconstruction-based neural simulators offer photorealism, they are fundamentally constrained by their initial captured data and struggle to generalize to highly dynamic or novel scenes. To overcome these limitations, we introduce OmniDreams, a foundation generative world model mid- and post-trained from the Cosmos diffusion model to autoregressively generate action-conditioned videos in real time. By leveraging the rich visual priors of Cosmos and mid- and post-training on 21k hours of driving scenarios, OmniDreams synthesizes complex, unobserved phenomena that are hard for traditional simulators to capture, such as extreme weather and unpredictable dynamic agent behaviors. Crucially, it autoregressively conditions its photorealistic sensor generation on past frames, the current simulator state, and immediate driving actions. Deployed in a closed-loop system with the Alpamayo 1 policy model and AlpaSim orchestrator, OmniDreams acts as a highly responsive, reactive environment, providing a scalable and comprehensive solution for training and evaluating next-generation autonomous driving policies. We additionally show preliminary results indicating that a world-action model (WAM) post-trained from OmniDreams achieves strong performance on the Physical AI Autonomous Vehicles NuRec dataset, surpassing the VLA-based Alpamayo 1.5 research policy model while using only 1/5 the total parameters. These results highlight the potential for a real-time world model like OmniDreams to also serve as a backbone for policy architectures.

Aarti Basant, Amlan Kar, Despoina Paschalidou +30

NVIDIA

Action-Conditioned Video Generation Video Diffusion Models Autoregressive Video Generation Autonomous Driving Simulation Autonomous Driving Scenario Generation Video World Models

Ask PDF

Jul 15, 2026

M $^\text{4}$ World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming

Driving-world generation has emerged as a core capability for scalable autonomous-driving simulation, yet existing methods remain limited in object-level controllability and long-horizon stability. We present M

^\text{4}

World, a Multi-view and Multimodal generative driving world model that synthesizes future surround-view video streams and synchronized LiDAR scans while supporting interactive object Manipulation and stable Minute-long streaming. Fine-grained object manipulation is realized through a flexible conditioning interface that supports explicit control over both the spatial layout and visual appearance of individual objects. Stable minute-long streaming, on the other hand, is achieved through a multi-stage training framework that enables online causal generation in only four denoising steps while maintaining coherent world dynamics throughout extended rollouts. Building on these components, we introduce an efficient few-clip post-training as well as a suite of visual reference-conditioned generation models, preserving general generation ability while allowing rare-case customization for long-tail controllability. To assess controllability beyond realism, we further introduce an automated VLM-based judging pipeline that evaluates scene-level condition adherence, view-wise object controllability, and cross-view object consistency. Comprehensive experiments show that M

^\text{4}

World consistently delivers high generation quality, precise controllability, and stable minute-long streaming. Together with downstream long-tail augmentation and scene editing, these results demonstrate the potential of M

^\text{4}

World for controllable, scalable driving simulation.

Ke Cheng, Hanqiao Ye, Lei Shi +8

1Meituan · Institute of Automation, Chinese Academy of Sciences · 3Beijing Institute of Technology

Multi-View Video Generation Long-Horizon Video Generation World Models for Autonomous Driving Controllable Video Generation Autonomous Driving Simulation Autonomous Driving Scenario Generation

Ask PDF

Jul 17, 2026

Orbis 2: A Hierarchical World Model for Driving

Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks. We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales: a high-level predictor that forecasts coarse scene structure over extended temporal horizons, and a low-level generator that produces detailed predictions conditioned on the high-level output. This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations. We further show that pretraining with a diffusion forcing objective yields substantially richer internal representations than the standard teacher forcing objective, while teacher forcing -- predicting only the next frame from clean context -- produces more stable autoregressive rollouts. We therefore introduce a generic two-stage training paradigm that pretrains the model with diffusion forcing and fine-tunes with teacher forcing, combining the representational benefits of the former with the rollout stability of the latter. Our approach achieves state-of-the-art results across the standard suite of driving world model evaluations on established benchmarks, including long-horizon generation fidelity, steering responsiveness evaluated on counterfactual scenarios, and internal representation quality. Project page with code, demo, checkpoints and qualitative results: https://lmb-freiburg.github.io/orbis2.github.io/

Sudhanshu Mittal, Arian Mousakhan, Silvio Galesso +4

University of Freiburg, Germany

World Models World Models for Autonomous Driving

Ask PDF

Abstract

Explore similar work

NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation

M4^\text{4}4World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming

Orbis 2: A Hierarchical World Model for Driving

M $^\text{4}$ World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming