Data-driven autonomous driving simulation has long been constrained by its heavy reliance on pre-recorded driving logs or spatial priors, such as HD maps. This fundamental dependency severely limits scalability, restricting open-ended generation capabilities to the finite scale of existing collected datasets. To break this bottleneck, we present OccSim, the first occupancy world model-driven 3D simulator. OccSim obviates the requirement for continuous logs or HD maps; conditioned only on a single initial frame and a sequence of future ego-actions, it can stably generate over 3,000 continuous frames, enabling the continuous construction of large-scale 3D occupancy maps spanning over 4 kilometers for simulation. This represents an >80x improvement in stable generation length over previous state-of-the-art occupancy world models. OccSim is powered by two modules: W-DiT based static occupancy world model and the Layout Generator. W-DiT handles the ultra-long-horizon generation of static environments by explicitly introducing known rigid transformations in architecture design, while the Layout Generator populates the dynamic foreground with reactive agents based on the synthesized road topology. With these designs, OccSim can synthesize massive, diverse simulation streams. Extensive experiments demonstrate its downstream utility: data collected directly from OccSim can pre-train 4D semantic occupancy forecasting models to achieve up to 67% zero-shot performance on unseen data, outperforming previous asset-based simulator by 11%. When scaling the OccSim dataset to 5x the size, the zero-shot performance increases to about 74%, while the improvement over asset-based simulators expands to 22.1%.
Figures & tables
Figure 1: Comparison of OccSim and workflow. We overcome all of the mentioned drawback of previous method, with only 1 single frame and future actions, the W-DiT we proposed is able to build a multi-kilometer consistent occupancy map for autonomous driving simulation. The agent initial poses are generated via the layout generator and then forward-simulated. OccSim is compatible with multi-agent forward simulation methods as plug-and-play modules; in our experiments, we use an IDM variant [ 16 ] .
Figure 2: Illustration of structure to generate road map of t+1 from condition at timestep t. Here, t represents the sequence frame index, and τ denotes the probability flow timestep. Different from classic temporal concatenation in the input, the core insight of this paradigm design is to transform temporal generation into scene completion at a single time point. This token-wise scale and shift condition injection method allow us can precisely control spatial rigid transformation during long-horizon generation. I(−) stand for rasterization process of given future trajectory Jt+1t+n . The latent representations zt and zt+1 ( ∈RH×W×Cz ) at bottom right corner are visualized as decoded BEV occupancy maps for intuitive illustration.
Figure 3: Agent initialization within generated static maps. Each panel displays a 200x200 voxel crop centered on the (unrendered) ego-vehicle from Mglobal . The first two panels show distinct generated scenes, the third and fourth panels demonstrate the model’s multimodality: two different plausible layouts generated for the same revisited location. Orange vehicles (second panel) are stationary.
Figure 4: Realism evaluation of generated 3D static occupancy. The left three columns report the conditional fidelity (MMD and KID) under a 30-frame constraint. The right two columns showcase the unconditional realism (KID and FID) over a 1000-frame long-horizon rollout. Our W-DiT-based method obtains the best long-horizon stability and conditional realism.
Figure 5: Long-horizon 2D realism under varying trajectories. FID and KID comparisons for 2D projections of the 3D generated occupancy. We evaluate three ego-actions: straight, closed-loop, and continuous turning. The dashed reference line indicates the “chaos” threshold characterizing scene collapse. In all scenarios, our model (red) remains strictly below this threshold, vastly outperforming the previous SOTA model(which exceed the “chaos” threshold in 35/40-th frame generated) and ensuring long-term structural integrity regardless of the driving action.
Figure 6: Generation diversity comparison. Pairwise mIoU diversity and Vendi scores evaluated over 100 timesteps using 10 random seeds. Solid lines and shaded areas denote the mean and the 25th-75th percentiles, respectively. Our methods performed the best diversity in the rollout among all tested generative method.
Model
Train Data
mIoU
IoU
IoUveh
Stoch.
1s
2s
3s
1s
2s
3s
OccWorld [ 44 ]
Uni-N
26.53
16.73
12.33
32.17
22.88
18.41
17.10
✗
OccWorld
Uni-C
11.79
8.35
6.75
18.31
14.09
12.03
6.58
✗
OccWorld†
OccSim
13.51
9.56
7.61
20.53
15.48
13.24
7.46
✗
OccWorld⋆
OccSim
19.54
12.31
9.08
23.87
16.92
13.47
12.41
✗
OccFM [ 20 ]
Uni-N
36.28
25.10
19.47
41.22
30.90
25.02
28.79
✓
Table 1: The performance of different semantic forecasting model trained with occupancy collected from UniOcc-Carla(Uni-C) or OccSim, test on UniOcc-NuScenes(Uni-N) validation set. mIOU measured over all background occupancy categories and car category for foreground. † : We collected the same amount of data on OccSim as that provided by UniOcc for Carla for fairness. ⋆ : Use ×5 amount of data collected from OccSim to train downstream models. Stoch.: whether the model is stochastic. The gray-shaded rows represent the performance upper bound, as the models are trained and tested on the same domain without domain shift.
Percep. Loss
Dual-head injection
SNR scaled
1st
5th
10th
30th
50th
100th
500th
✗
✗
✗
663.89
712.56
789.91
1023.51
1281.99
1587.21
1612.40
✗
✓
✗
637.52
708.91
924.11
992.72
1206.50
1387.55
1672.43
✓
✓
✗
676.98
652.91
711.66
782.25
801.67
852.30
901.57
✓
✓
✓
665.23
630.34
645.96
687.90
698.87
712.65
744.71
Table 2: W-DiT design ablation: we use 3D unconditional realism FID score. When not using the “local-global” form of feature aggregation we propose, we directly concatenate the masked condition with the input. The latent here is encoded by AE from UniScenes.
Method
WM
Input
#Params
secondFrames
frameGFLOPS
DOME
✓
4 frames sem. occ.&traj
444.07M
5.48
8891.98
COME
✓
4 frames sem. occ.&traj
204.53M
2.33
41333.17
InfiniCubes
✕
1 frames sem. occ.
831.75M
0.02
990927.28
OccSim (Ours)
✓
1 frames sem. occ. & traj
181.24M
1.47
15570.18
Table 3: Comparison of efficiency and computational cost among different methods. When evaluating the time taken by our method, we distribute the post-processing time evenly across each frame of the sequence generation. Each result was computed independently using an RTX4090. DOME and COME generate 6 frames together, ours is 1 single frame per step, for fair comparison, we keep measure the GFLOPS / inference time needed per single frame.
Figure 7: Qualitative comparison of two rollouts for two models: previous SOTA occupancy world model (first and third rows), and our model (second and fourth rows) at different horizons. Ours exhibits long-horizon stability under different trajectory input.
Figure 8: Visualization of trajectory shapes (Loop, Straight, and Turn) evaluated in Fig. 5. The green star indicates the starting point. The blue solid line denotes the initial ground-truth (GT) trajectory snippet (frames 1–40) used as the condition prompt, while the red dashed line represents the extended trajectory explicitly designed to guide the long-horizon generation. For visual clarity the displayed trajectories are truncated at 500 frames, whereas our actual synthesis spans up to 3000 frames.
Figure 9: Visualization of frames from the “chaos” baseline. We randomly visualize 20 frames from the 1000 samples pool. The semantic categories represented by colors here are consistent with those in Fig. 3 .
Simulation with realistic traffic agents is essential for validating autonomous driving systems. Existing data-driven simulators learn agent behavior from higher-level abstractions such as 3D bounding boxes and polylines, inferred by upstream perception pipelines. These lossy abstractions discard sensory context that directly shapes agent behavior, limiting the distributional realism that simulation aims to reproduce. To address this limitation, we propose AutoWorld, a traffic simulation framework that grounds agent behavior in raw sensor observations through a self-supervised world model trained on LiDAR occupancy data. Given world model samples, AutoWorld constructs a coarse-to-fine predictive scene context as input to a multi-agent motion generation model. Furthermore, we designed a motion-aware latent supervision objective that enriches AutoWorld's latent representation of scene dynamics. To better exploit this latent space during inference, AutoWorld employs a cascaded Determinantal Point Process framework to guide diversity-aware sampling across both the world model and motion model. Experiments on the Waymo Sim Agents Challenge (WOSAC) demonstrate that AutoWorld achieves competitive performance, with larger gains in partially-observed scenarios where trajectory abstractions are most limited. We further show that grounding simulation in raw LiDAR through AutoWorld scales better with additional data than trajectory-only and LiDAR-conditioning baselines. Ablations confirm the contribution of each component.
Mozhgan Pourkeshavarz, Tianran Liu, Nicholas Rhinehart
Generating high-fidelity and controllable synthetic data is critical for advancing end-to-end autonomous driving, particularly for addressing the long tail of rare safety-critical scenarios. Existing occupancy-guided methods typically rely on shallow conditioning mechanisms and reference-frame-dependent video synthesis, which limits fine-grained controllability from arbitrary BEV layouts and restricts their applicability for scalable simulation. In this paper, we propose AnyScene, a unified occupancy-centric framework for driving scene generation. AnyScene generates semantic occupancy sequences from BEV layouts through a Spatial-Temporal Occupancy Diffusion Transformer that jointly tokenizes BEV and occupancy features in an autoregressive manner. This design enables precise controllability from cross-dataset and user-defined BEV inputs while naturally supporting long-horizon generation. Building upon the generated occupancy, a Geometry-Grounded View Expansion module treats occupancy as the canonical spatial representation and synthesizes temporally consistent multi-view driving videos in a reference-free and autoregressive fashion, supporting flexible camera configurations at inference time. Extensive experiments demonstrate that AnyScene achieves state-of-the-art performance in both occupancy and video generation. It exhibits strong generalization to unseen and customized layouts, and provides measurable benefits for downstream tasks such as sparse-view 3D reconstruction.
Haiming Zhang, Junfei Zhou, Feng Jiang +6
Li Auto · Southwest Jiaotong University · Tsinghua University
As autonomous vehicle capabilities advance, the safe evaluation of driving policies in long-tail scenarios remains a critical bottleneck. In closed-loop simulation, the driving policy model actively interacts with the environment, where its actions dynamically update the simulator state and directly influence the next set of generated sensor observations. While recent reconstruction-based neural simulators offer photorealism, they are fundamentally constrained by their initial captured data and struggle to generalize to highly dynamic or novel scenes. To overcome these limitations, we introduce OmniDreams, a foundation generative world model mid- and post-trained from the Cosmos diffusion model to autoregressively generate action-conditioned videos in real time. By leveraging the rich visual priors of Cosmos and mid- and post-training on 21k hours of driving scenarios, OmniDreams synthesizes complex, unobserved phenomena that are hard for traditional simulators to capture, such as extreme weather and unpredictable dynamic agent behaviors. Crucially, it autoregressively conditions its photorealistic sensor generation on past frames, the current simulator state, and immediate driving actions. Deployed in a closed-loop system with the Alpamayo 1 policy model and AlpaSim orchestrator, OmniDreams acts as a highly responsive, reactive environment, providing a scalable and comprehensive solution for training and evaluating next-generation autonomous driving policies. We additionally show preliminary results indicating that a world-action model (WAM) post-trained from OmniDreams achieves strong performance on the Physical AI Autonomous Vehicles NuRec dataset, surpassing the VLA-based Alpamayo 1.5 research policy model while using only 1/5 the total parameters. These results highlight the potential for a real-time world model like OmniDreams to also serve as a backbone for policy architectures.