Breaking survivorship bias in driving world models
Authors: Zihao Sheng, Zilin Huang, Yuhao Luo, Zhengyang Wan, Boyue Wang, Soyoung Ahn, David A. Noyce, Sikai Chen
Organizations: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI, USA · Wisconsin Transportation Center, University of Wisconsin–Madison, Madison, WI, USA
World models allow autonomous vehicles to imagine the consequences of candidate actions before execution, but their reliability depends on the futures represented in their training data. We identify a survivorship bias in driving world models: because the futures recorded in prevailing driving datasets almost always end safely, existing models often imagine benign outcomes even when a collision is imminent. We introduce a neutral world model (NWM) that learns both routine and safety-critical futures from a mixture of routine driving and real accident videos with a bounding-box-guided objective. The NWM more faithfully anticipates imminent collisions, while maintaining a low false collision rate and competitive generation quality in routine driving. We further use the model as an imagination engine for decision-making. Given multiple ego trajectories, the NWM predicts the future associated with each candidate, and a reward function evaluates these imagined outcomes before selecting an action. On challenging NAVSIM scenarios, this procedure selects the better trajectory more reliably than the baseline world model. Zero-shot experiments with real-world pedestrian dummy crossing data collected using an autonomous research vehicle further demonstrate transfer to unseen sensor observations. These results show that correcting survivorship bias enables world models to more faithfully reason about futures and support safety-critical decision-making in autonomous driving.
Figures & tables
Figure 1: Overview of the neutral world model. a , Representative frames from routine driving datasets (nuScenes, OpenDV and NAVSIM) and from the accident dataset MM-AU. The neutral world model (NWM) is trained on both types of data (nuScenes and MM-AU). b , Architecture of the NWM, a video diffusion model conditioned on the initial frame, the ego trajectory and optional bounding boxes of surrounding agents. The bounding-box-guided loss Lbbox complements the diffusion loss LWM by emphasizing regions occupied by these agents. c , World-model-guided action selection. The NWM imagines the future for each of K candidate ego trajectories, and the candidate with the highest reward is selected. The procedure is evaluated on NAVSIM and on real-vehicle tests with a pedestrian dummy.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Stage 1: backbone adaptation
Stage 2: ControlNet
Initialization
Pretrained SVD, 25-frame XT checkpoint
Stage 1 at step 52,800; copied ControlNet blocks and zero-initialized residual projections
Trainable modules
Full spatiotemporal U-Net
ControlNet, trajectory projection and null embedding
Frozen modules
VAE and CLIP image encoder
VAE, CLIP image encoder and adapted U-Net
Training data
MM-AU and nuScenes
MM-AU and nuScenes
Clip dimensions
25 frames, 512×320
25 frames, 512×320
Optimizer
AdamW
AdamW
Appendix
Supplementary Table S1: Training configuration of the neutral world model. Settings for backbone adaptation and conditional training; batch sizes are measured in video clips. NWM, neutral world model; SVD, Stable Video Diffusion; VAE, variational autoencoder; CLIP, Contrastive Language–Image Pre-training; FP16, 16-bit floating point; EMA, exponential moving average; GPU, graphics processing unit.
Setting
Value
Sampler / denoising steps
Euler / 30
Noise schedule
Karras; σmin=0.002 , σmax=700 , ρ=7
Reference-model guidance
Linear increase from 1 to 9 across frames
ControlNet scale
1.0
Image noise augmentation
0.01
Motion bucket identifier
127
Appendix
Supplementary Table S2: NWM inference settings and computational cost. Latency is measured after two warm-up calls over five distinct inputs on the stated inference GPU. NWM, neutral world model; VAE, variational autoencoder; RGB, red–green–blue; FP16, 16-bit floating point; GPU, graphics processing unit; SD, sample standard deviation; GiB, 230 bytes.
Dataset
Training
Validation
Effective training
nuScenes
7,700
450
11,500
MM-AU (crash)
6,194
733
9,630
MM-AU (nominal)
1,745
219
2,381
MM-AU (total)
7,939
952
12,011
Combined total
15,639
1,402
23,511
Appendix
Supplementary Table S3: Dataset splits and training composition. Counts of extracted 25-frame clips and effective training samples after maneuver-based repetition.
Supplementary Fig. S8: Interface for manual assessment of generated outcomes. The interface presents generated videos for inspection with model names hidden and the presentation order randomized. Collision outcomes are assessed over the complete 25-frame clip.
Supplementary Fig. S9: Research van and real-world crossing configurations. a , Ford E-Transit research van operated by the Wisconsin Transportation Center and sensors. b , Pedestrian dummy and wheeled cart used for manually pulled crossings. c , Recorded views of the six configurations in Supplementary Table S4 , showing variations in snow presence, clothing and crossing origin relative to a parked vehicle.
Trial
Date
Snow present
Crossing origin
Dummy clothing
T1
February 23, 2026
No
In front of parked vehicle
Black
T2
February 23, 2026
No
In front of parked vehicle
Purple
T3
February 23, 2026
No
Behind parked vehicle
Black
T4
February 23, 2026
No
Behind parked vehicle
Purple
T5
March 17, 2026
Yes
In front of parked vehicle
Purple
T6
March 17, 2026
Yes
Behind parked vehicle
Purple
Appendix
Supplementary Table S4: Configurations of the six real-world trials. Crossing origin is specified relative to the parked vehicle. Snow presence describes the observed site conditions. Each trial contributes five Normal and five Response decision frames.
Supplementary Fig. S10: Recorded vehicle motion and decision-frame selection. Aligned displacement, speed, throttle and brake signals for the six trials. Markers identify the five Normal frames (A1–A5) and five Response frames (B1–B5) selected from each recording. Brake signals are shown in their recorded raw units.
Supplementary Fig. S11: Computation of reward from imagined futures. a , Candidate trajectories generate alternative futures from a shared current observation. b , Detection and temporal association yield target boxes, which are projected to approximate ground-plane positions and vehicle clearances. c , Observed proximity and warning times determine S and TTC ; reference context and candidate motion determine P and C . d , The weighted reward ranks the candidate actions.
Parameter
Meaning
Value
wT,wP,wC
Reward weights
5,5,2
H , Tsafe
Rollout horizon; warning-time normalization
4 s; 4 s
α
Proximity-event gate coefficient
0.5
dc , dw
Proximity and warning clearances
0.5 m; 1.0 m
a0 , j0
Acceleration and jerk scales
3 m / s 2 ; 5 m / s 3
λj
Relative jerk-cost weight
1
Appendix
Supplementary Table S5: Reward parameters and detection settings. Values for the reward implementation. IoU, intersection over union; NMS, non-maximum suppression.
Supplementary Fig. S12: Examples of reward-guided action selection. a , A routine-driving input illustrates the selection for continuing motion when progress is appropriate. b , A pedestrian-crossing input shows candidate futures generated by NWM and Vista, with tracked target detections. c , Clearance estimates show how the safety components are obtained.