Autoregressive driving world models systematically suffer from structural drift, identity drift, and quality decay over long rollout horizons due to transient conditioning signals that lack persistent state. We present WAM-Sim, a driving world model anchored by a persistent hierarchical memory and optimized via online flow-matching RL. WAM-Sim structures persistent context across three temporal and semantic scales: pose-aligned structural tokens encoding persistent road topology, read-only scene tokens supplying geolocated background appearance, and write-once instance tokens preserving individual traffic participant identities across occlusions and chunk boundaries. To restore visual sharpness and physical plausibility degraded by multi-scale conditioning, we introduce an online post-training framework that optimizes self-generated rollouts via flow-matching RL guided by hierarchy-aligned visual rewards and a vision-language model physical judge. The invariant memory empirically confines policy optimization within the anchored context, helping suppress semantic collapse and reward hacking, while online RL systematically enforces physical commonsense and fine-grained visual fidelity. On the nuScenes benchmark, WAM-Sim achieves a superior generation fidelity (FID=3.47, FVD=25.72) and strong downstream perception performance (24.16 mAP, 71.23 road mIoU). On minute-scale rollouts of self-collected data, WAM-Sim substantially mitigates structural and identity drift, preserving high scene similarity and subject identity coherence across extended horizons. Project page: https://fudan-generative-vision.github.io/WAM-Sim.
Figures & tables
Figure 1: Overview of core capabilities in WAM-Sim. World anchored memory (top-left) anchors long-horizon rollouts, maintaining background consistency and persistent instance identities (e.g., tracked pedestrians, motor vehicles, and non-motorized vehicles) across extended horizons. Online RL Post-training (bottom-left) restores visual fidelity and enforces physical commonsense, synthesizing natural agent dynamics (e.g., pedestrian gait) and realistic vehicle motion. Furthermore, WAM-Sim generalizes across diverse weather conditions and environmental scenes (right).
Figure 2: Overview of the WAM-Sim architecture and world anchored memory. Persistent context is structured across three temporal and semantic scales: 1) Structure: pose-aligned HD-map tokens injected via spatio-temporal addition to anchor road geometry; 2) Scene: geolocated background tokens retrieved from a database and accessed via temporally routed scene cross-attention; and 3) Instance: object appearance crops stored in a write-once appearance bank and retrieved via instance-masked cross-attention to isolate participant identities. The hierarchical memory state conditions a 3D Diffusion Transformer (DiT) operating on noisy latents and history KV caches to synthesize autoregressive, consistent driving video chunks.
Figure 3: Overview of the Drive DiffusionNFT pipeline. (a) Data curation constructs a diverse RL training set with visible traffic participants and strong motion dynamics. (b) Multi-reward evaluation assesses scene quality, object quality, and physical plausibility. (c) NFT optimization normalizes and aggregates the rewards to update the driving world model.
Method
Synthesis Resolution
Fidelity
3D Detection
BEV Seg mIoU
FVD ↓
FID ↓
mAP ↑
NDS ↑
Road ↑
Vehicle ↑
Ori nuScenes
900×1600
-
-
35.54
41.21
73.66
34.82
(I) Image Generation Methods
BEVControl
224×400
-
24.85
-
-
60.80
26.80
PerLDiff
256×384
-
13.36
15.24
24.05
61.26
27.13
DrivePTS
224×480
-
11.45
15.37
25.49
63.95
27.82
Table 1: Comparisons of controllable driving scene generation on the nuScenes dataset, grouped by generation paradigm. FID is computed over keyframes corresponding to all nuScenes validation.
Method
Visual Fidelity
VBench Metrics
Persistence
Object & Physics
FID ↓
FVD ↓
Quality ↑
Subject ↑
Background ↑
Sub. Coh. ↑
Scene Sim. ↑
Sub. Fid. ↑
Phys. Plaus. ↑
(I) Inference Only
UniMLVG
44.21
753.16
75.75
83.43
89.77
0.591
0.694
39.19
46.57
OpenDWM
37.69
535.42
76.54
82.36
90.11
0.532
0.714
37.24
47.44
HorizonDrive
24.86
218.51
78.41
84.55
88.09
0.553
0.786
42.98
51.19
(II) SFT on Self-Collected Data
Table 2: Evaluation on the Self-Collected Dataset. All methods generate videos conditioned on ego trajectories and agent bounding boxes.
Table 6
Figure 4: Qualitative comparison results on nuScenes video generation.
Figure 8
Figure 7: Qualitative comparison of scene- and instance-level persistence. WAM-Sim better preserves background appearance and object identity over long rollouts.
Figure 8: Qualitative ablation of online RL. Left: comparison between the world-memory model without RL and the final model. Right: cumulative reward ablation on the same scenario, progressively adding Rscene , Robj , and Rphy .
Method
Long-range Coherence
Re-emergence
Avg. ↑
SC@1S ↑
SC@2S ↑
SC@3S ↑
Sim. ↑
Pass Rate ↑
Occ-ID@1S ↑
Occ-ID@2S ↑
Baseline (SFT)
0.605
0.579
0.546
0.780
0.461
0.781
0.743
0.642
ConsisDrive
0.634
0.604
0.597
0.805
0.510
0.799
0.812
0.680
Ours
0.653
0.620
0.613
0.810
0.535
0.804
0.833
0.695
Table 5: Comparison of instance identity preservation on the test set.
Rscene
Robj
Rphy
Img. ( Rscene ) ↑
Classi. ( Robj ) ↑
Phys. ( Rphy ) ↑
Sub. Fid. ↑
Phys. Plaus. ↑
–
–
–
59.55
-0.012
3.770
41.74
50.91
√
–
–
62.68
0.064
3.727
42.25
51.13
√
√
–
61.59
0.273
3.789
43.97
53.54
√
√
√
61.48
0.271
4.052
44.00
55.96
Table 6: Ablation of reward components in Drive DiffusionNFT post-training.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Pairwise Preference (%)
Mean Opinion Score
Method
Win
Tie
Loss
Fidelity
Identity
Physics
Location
UniMLVG
98.3
0.0
1.7
2.10
2.20
1.73
1.95
OpenDWM–CTSD3.5
98.3
0.0
1.7
1.80
1.80
1.38
1.87
HorizonDrive
71.7
1.7
26.7
4.10
4.00
4.00
3.90
OmniDreams
86.7
0.0
13.3
3.10
3.00
2.77
2.87
WAM-Sim (Ours)
–
–
–
4.30
4.40
4.50
4.50
Appendix
Table 7: Human evaluation on 60 driving scenarios assessed by 15 experts. Pairwise preference reports scenario-level win/tie/loss rates against WAM-Sim. MOS is reported on a 1–5 scale.
Dimension
UniMLVG
OpenDWM–CTSD3.5
HorizonDrive
OmniDreams
Fidelity
7.29×10−12
7.52×10−12
0.0546
2.40×10−10
Identity
7.58×10−12
7.28×10−12
7.65×10−4
2.58×10−10
Physics
7.52×10−12
5.96×10−12
4.32×10−8
6.49×10−12
Location
2.78×10−12
6.25×10−12
3.41×10−6
5.17×10−12
Appendix
Table 8: Statistical significance of scenario-level MOS differences. Two-sided Wilcoxon signed-rank tests compare WAM-Sim with each baseline using 60 paired scenario-level scores per dimension. Values below 0.05 meet the stated significance threshold.
Method
Resolution
Params (B)
Latency (s) ↓
Throughput (FPS) ↑
VRAM (GB) ↓
Horizondrive
384×768
1.3
3.51
4.31
20.11
UniMLVG
256×448
2.0
10.70
1.54
25.32
OpenDWM-CTSD3.5
256×448
2.5
14.98
1.08
32.67
Omnidreams
704×1280
2.0
0.65
8.00
45.71
Ours
256×448
3.1
0.60
18.63
45.21
Ours
704×1280
3.1
1.19
6.94
47.45
Appendix
Table 9: Efficiency comparison of driving world models during long-horizon rollouts.
Figure 9: Quality versus generation horizon. World memory reduces quality degradation from 10 s to 60 s.
Hyperparameter
Value
Group size G
16
Sampling steps
35
Guidance scale
1.0
Flow shift
5.0
NFT mixing coefficient β
1.0
Training timestep fraction
0.8
Appendix
Table 10: Training settings for Drive DiffusionNFT.
Configuration
Structural Memory
Scene Memory
Instance Memory
Shared Video Representation and Backbone
Input resolution
704×1280
Frames per window
93
Frame rate
10 FPS
Video VAE
Wan2.1 Video VAE
VAE latent channels
16
Appendix
Table 11: Model configuration. Shared backbone architecture and configurations of the structural, scene, and instance memory components.
Training Setting
Stage I: Structural
Stage II: Scene
Stage III: Instance
Stage IV: Reinforcement
Training target
Structural control
Scene memory
Instance memory
Generation quality
Initialized from
Pretrained backbone
Stage I
Stage II
Stage III
Trainable modules
Self-Attn
Scene.Cross-attn
Instance.Cross-Attn
Self-Attn
Training objective
Rectified flow
Rectified flow
Rectified flow
DiffusionNFT loss
Learning rate
1×10−5
3×10−5
3×10−4
1×10−5
Optimizer
AdamW
AdamW
AdamW
AdamW
Appendix
Table 12: Training configuration. Stage-wise training settings for structural control, scene memory, instance memory, and reinforcement fine-tuning.
Figure 10: Dataset construction pipeline.
Figure 12: Distribution of clip durations in the training set.
Driving world action models (WAMs) learn from paired trajectories and future observations. Post-training with reinforcement learning allows the policy to explore alternative trajectories and improve planning using driving rewards. However, driving logs contain future videos only for the recorded trajectories, leaving these alternatives without corresponding visual supervision. Thus, we introduce SimVTA, a simulation-grounded post-training framework that combines trajectory optimization with video supervision. First, we sample candidate trajectories for each observation and use a real-world simulator (e.g., World Engine) to render future videos for high-reward candidates in real-world scenes reconstructed with 3D Gaussian Splatting (3DGS). This provides trajectory-matched supervision independent of the model's own predictions. Under a limited rendering budget, we prioritize video supervision for higher-error futures among these rendered candidates. Finally, we optimize trajectory planning with Group Relative Policy Optimization (GRPO) and use a flow-matching loss on the matched videos to update shared representations. At inference time, the model predicts trajectories directly, without simulator calls. On NAVSIM v1, applying SimVTA to a WAM initialization improves PDMS from 86.2 to 91.3 (+5.1). It achieves 88.2 EPDMS on NAVSIM v2 and an average HD-Score of 38.8 in closed-loop evaluation on HUGSIM. Ablations show that updating shared representations with video supervision improves planning, while prioritizing higher-error futures among high-reward trajectories further improves performance.
Changhui Jing, Haochen Tian, Hongchen Li +8
Shanghai Innovation Institute · OpenDriveLab at The University of Hong Kong · Tongji University +2
Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks. We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales: a high-level predictor that forecasts coarse scene structure over extended temporal horizons, and a low-level generator that produces detailed predictions conditioned on the high-level output. This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations. We further show that pretraining with a diffusion forcing objective yields substantially richer internal representations than the standard teacher forcing objective, while teacher forcing -- predicting only the next frame from clean context -- produces more stable autoregressive rollouts. We therefore introduce a generic two-stage training paradigm that pretrains the model with diffusion forcing and fine-tunes with teacher forcing, combining the representational benefits of the former with the rollout stability of the latter. Our approach achieves state-of-the-art results across the standard suite of driving world model evaluations on established benchmarks, including long-horizon generation fidelity, steering responsiveness evaluated on counterfactual scenarios, and internal representation quality. Project page with code, demo, checkpoints and qualitative results: https://lmb-freiburg.github.io/orbis2.github.io/
Sudhanshu Mittal, Arian Mousakhan, Silvio Galesso +5
Closed-loop driving simulation requires real-time interaction beyond short offline clips, pushing current driving world models toward autoregressive (AR) rollout. Existing AR distillation approaches typically rely on frame sinks or student-side degradation training. The former transfers poorly to driving due to fast ego-motion and rapid scene changes, while the latter remains bounded by the teacher's single-pass output length and thus provides only a limited supervision horizon. A natural question is: can the teacher itself be extended via AR rollout to provide unbounded-horizon supervision at bounded memory cost? The key difficulty is that a standard teacher drifts under its own predictions, contaminating the supervision it provides. Our key insight is to make the teacher rollout-capable, ensuring reliable supervision from its own AR rollouts. This is instantiated as HorizonDrive, an anti-drifting training-and-distillation framework for AR driving simulation. First, scheduled rollout recovery (SRR) trains the base model to reconstruct ground-truth future clips from prediction-corrupted histories, yielding a teacher that remains stable across long AR rollouts. Second, the rollout-capable teacher is extended via AR rollout, providing long-horizon distribution-matching supervision under bounded memory, while a short-window student aligns to it with teacher rollout DMD (TRD) for efficient real-time deployment. HorizonDrive natively supports minute-scale AR rollout under bounded memory; on nuScenes, HorizonDrive reduces FID by 52% and FVD by 37%, and lowers ARE and DTW by 21% and 9% relative to the strongest long-horizon streaming baselines, while remaining competitive with single-pass driving video generators.
Conglang Zhang, Yifan Zhan, Qingjie Wang +10
Wuhan University · The University of Tokyo · Horizon Robotics +3