Autoregressive driving world models systematically suffer from structural drift, identity drift, and quality decay over long rollout horizons due to transient conditioning signals that lack persistent state. We present WAM-Sim, a driving world model anchored by a persistent hierarchical memory and optimized via online flow-matching RL. WAM-Sim structures persistent context across three temporal and semantic scales: pose-aligned structural tokens encoding persistent road topology, read-only scene tokens supplying geolocated background appearance, and write-once instance tokens preserving individual traffic participant identities across occlusions and chunk boundaries. To restore visual sharpness and physical plausibility degraded by multi-scale conditioning, we introduce an online post-training framework that optimizes self-generated rollouts via flow-matching RL guided by hierarchy-aligned visual rewards and a vision-language model physical judge. The invariant memory empirically confines policy optimization within the anchored context, helping suppress semantic collapse and reward hacking, while online RL systematically enforces physical commonsense and fine-grained visual fidelity. On the nuScenes benchmark, WAM-Sim achieves a superior generation fidelity (FID=3.47, FVD=25.72) and strong downstream perception performance (24.16 mAP, 71.23 road mIoU). On minute-scale rollouts of self-collected data, WAM-Sim substantially mitigates structural and identity drift, preserving high scene similarity and subject identity coherence across extended horizons. Project page: https://fudan-generative-vision.github.io/WAM-Sim.
Figures & tables
Figure 1: Overview of core capabilities in WAM-Sim. World anchored memory (top-left) anchors long-horizon rollouts, maintaining background consistency and persistent instance identities (e.g., tracked pedestrians, motor vehicles, and non-motorized vehicles) across extended horizons. Online RL Post-training (bottom-left) restores visual fidelity and enforces physical commonsense, synthesizing natural agent dynamics (e.g., pedestrian gait) and realistic vehicle motion. Furthermore, WAM-Sim generalizes across diverse weather conditions and environmental scenes (right).
Figure 2: Overview of the WAM-Sim architecture and world anchored memory. Persistent context is structured across three temporal and semantic scales: 1) Structure: pose-aligned HD-map tokens injected via spatio-temporal addition to anchor road geometry; 2) Scene: geolocated background tokens retrieved from a database and accessed via temporally routed scene cross-attention; and 3) Instance: object appearance crops stored in a write-once appearance bank and retrieved via instance-masked cross-attention to isolate participant identities. The hierarchical memory state conditions a 3D Diffusion Transformer (DiT) operating on noisy latents and history KV caches to synthesize autoregressive, consistent driving video chunks.
Figure 3: Overview of the Drive DiffusionNFT pipeline. (a) Data curation constructs a diverse RL training set with visible traffic participants and strong motion dynamics. (b) Multi-reward evaluation assesses scene quality, object quality, and physical plausibility. (c) NFT optimization normalizes and aggregates the rewards to update the driving world model.
Method
Synthesis Resolution
Fidelity
3D Detection
BEV Seg mIoU
FVD ↓
FID ↓
mAP ↑
NDS ↑
Road ↑
Vehicle ↑
Ori nuScenes
900×1600
-
-
35.54
41.21
73.66
34.82
(I) Image Generation Methods
BEVControl
224×400
-
24.85
-
-
60.80
26.80
PerLDiff
256×384
-
13.36
15.24
24.05
61.26
27.13
DrivePTS
224×480
-
11.45
15.37
25.49
63.95
27.82
Table 1: Comparisons of controllable driving scene generation on the nuScenes dataset, grouped by generation paradigm. FID is computed over keyframes corresponding to all nuScenes validation.
Method
Visual Fidelity
VBench Metrics
Persistence
Object & Physics
FID ↓
FVD ↓
Quality ↑
Subject ↑
Background ↑
Sub. Coh. ↑
Scene Sim. ↑
Sub. Fid. ↑
Phys. Plaus. ↑
(I) Inference Only
UniMLVG
44.21
753.16
75.75
83.43
89.77
0.591
0.694
39.19
46.57
OpenDWM
37.69
535.42
76.54
82.36
90.11
0.532
0.714
37.24
47.44
HorizonDrive
24.86
218.51
78.41
84.55
88.09
0.553
0.786
42.98
51.19
(II) SFT on Self-Collected Data
Table 2: Evaluation on the Self-Collected Dataset. All methods generate videos conditioned on ego trajectories and agent bounding boxes.
Table 6
Figure 4: Qualitative comparison results on nuScenes video generation.
Figure 8
Figure 7: Qualitative comparison of scene- and instance-level persistence. WAM-Sim better preserves background appearance and object identity over long rollouts.
Figure 8: Qualitative ablation of online RL. Left: comparison between the world-memory model without RL and the final model. Right: cumulative reward ablation on the same scenario, progressively adding Rscene , Robj , and Rphy .
Method
Long-range Coherence
Re-emergence
Avg. ↑
SC@1S ↑
SC@2S ↑
SC@3S ↑
Sim. ↑
Pass Rate ↑
Occ-ID@1S ↑
Occ-ID@2S ↑
Baseline (SFT)
0.605
0.579
0.546
0.780
0.461
0.781
0.743
0.642
ConsisDrive
0.634
0.604
0.597
0.805
0.510
0.799
0.812
0.680
Ours
0.653
0.620
0.613
0.810
0.535
0.804
0.833
0.695
Table 5: Comparison of instance identity preservation on the test set.
Rscene
Robj
Rphy
Img. ( Rscene ) ↑
Classi. ( Robj ) ↑
Phys. ( Rphy ) ↑
Sub. Fid. ↑
Phys. Plaus. ↑
–
–
–
59.55
-0.012
3.770
41.74
50.91
√
–
–
62.68
0.064
3.727
42.25
51.13
√
√
–
61.59
0.273
3.789
43.97
53.54
√
√
√
61.48
0.271
4.052
44.00
55.96
Table 6: Ablation of reward components in Drive DiffusionNFT post-training.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Pairwise Preference (%)
Mean Opinion Score
Method
Win
Tie
Loss
Fidelity
Identity
Physics
Location
UniMLVG
98.3
0.0
1.7
2.10
2.20
1.73
1.95
OpenDWM–CTSD3.5
98.3
0.0
1.7
1.80
1.80
1.38
1.87
HorizonDrive
71.7
1.7
26.7
4.10
4.00
4.00
3.90
OmniDreams
86.7
0.0
13.3
3.10
3.00
2.77
2.87
WAM-Sim (Ours)
–
–
–
4.30
4.40
4.50
4.50
Appendix
Table 7: Human evaluation on 60 driving scenarios assessed by 15 experts. Pairwise preference reports scenario-level win/tie/loss rates against WAM-Sim. MOS is reported on a 1–5 scale.
Dimension
UniMLVG
OpenDWM–CTSD3.5
HorizonDrive
OmniDreams
Fidelity
7.29×10−12
7.52×10−12
0.0546
2.40×10−10
Identity
7.58×10−12
7.28×10−12
7.65×10−4
2.58×10−10
Physics
7.52×10−12
5.96×10−12
4.32×10−8
6.49×10−12
Location
2.78×10−12
6.25×10−12
3.41×10−6
5.17×10−12
Appendix
Table 8: Statistical significance of scenario-level MOS differences. Two-sided Wilcoxon signed-rank tests compare WAM-Sim with each baseline using 60 paired scenario-level scores per dimension. Values below 0.05 meet the stated significance threshold.
Method
Resolution
Params (B)
Latency (s) ↓
Throughput (FPS) ↑
VRAM (GB) ↓
Horizondrive
384×768
1.3
3.51
4.31
20.11
UniMLVG
256×448
2.0
10.70
1.54
25.32
OpenDWM-CTSD3.5
256×448
2.5
14.98
1.08
32.67
Omnidreams
704×1280
2.0
0.65
8.00
45.71
Ours
256×448
3.1
0.60
18.63
45.21
Ours
704×1280
3.1
1.19
6.94
47.45
Appendix
Table 9: Efficiency comparison of driving world models during long-horizon rollouts.
Figure 9: Quality versus generation horizon. World memory reduces quality degradation from 10 s to 60 s.
Hyperparameter
Value
Group size G
16
Sampling steps
35
Guidance scale
1.0
Flow shift
5.0
NFT mixing coefficient β
1.0
Training timestep fraction
0.8
Appendix
Table 10: Training settings for Drive DiffusionNFT.
Configuration
Structural Memory
Scene Memory
Instance Memory
Shared Video Representation and Backbone
Input resolution
704×1280
Frames per window
93
Frame rate
10 FPS
Video VAE
Wan2.1 Video VAE
VAE latent channels
16
Appendix
Table 11: Model configuration. Shared backbone architecture and configurations of the structural, scene, and instance memory components.
Training Setting
Stage I: Structural
Stage II: Scene
Stage III: Instance
Stage IV: Reinforcement
Training target
Structural control
Scene memory
Instance memory
Generation quality
Initialized from
Pretrained backbone
Stage I
Stage II
Stage III
Trainable modules
Self-Attn
Scene.Cross-attn
Instance.Cross-Attn
Self-Attn
Training objective
Rectified flow
Rectified flow
Rectified flow
DiffusionNFT loss
Learning rate
1×10−5
3×10−5
3×10−4
1×10−5
Optimizer
AdamW
AdamW
AdamW
AdamW
Appendix
Table 12: Training configuration. Stage-wise training settings for structural control, scene memory, instance memory, and reinforcement fine-tuning.
Figure 10: Dataset construction pipeline.
Figure 12: Distribution of clip durations in the training set.