Breaking survivorship bias in driving world models
Authors: Zihao Sheng, Zilin Huang, Yuhao Luo, Zhengyang Wan, Boyue Wang, Soyoung Ahn, David A. Noyce, Sikai Chen
Organizations: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI, USA · Wisconsin Transportation Center, University of Wisconsin–Madison, Madison, WI, USA
World models allow autonomous vehicles to imagine the consequences of candidate actions before execution, but their reliability depends on the futures represented in their training data. We identify a survivorship bias in driving world models: because the futures recorded in prevailing driving datasets almost always end safely, existing models often imagine benign outcomes even when a collision is imminent. We introduce a neutral world model (NWM) that learns both routine and safety-critical futures from a mixture of routine driving and real accident videos with a bounding-box-guided objective. The NWM more faithfully anticipates imminent collisions, while maintaining a low false collision rate and competitive generation quality in routine driving. We further use the model as an imagination engine for decision-making. Given multiple ego trajectories, the NWM predicts the future associated with each candidate, and a reward function evaluates these imagined outcomes before selecting an action. On challenging NAVSIM scenarios, this procedure selects the better trajectory more reliably than the baseline world model. Zero-shot experiments with real-world pedestrian dummy crossing data collected using an autonomous research vehicle further demonstrate transfer to unseen sensor observations. These results show that correcting survivorship bias enables world models to more faithfully reason about futures and support safety-critical decision-making in autonomous driving.
Figures & tables
Figure 1: Overview of the neutral world model. a , Representative frames from routine driving datasets (nuScenes, OpenDV and NAVSIM) and from the accident dataset MM-AU. The neutral world model (NWM) is trained on both types of data (nuScenes and MM-AU). b , Architecture of the NWM, a video diffusion model conditioned on the initial frame, the ego trajectory and optional bounding boxes of surrounding agents. The bounding-box-guided loss Lbbox complements the diffusion loss LWM by emphasizing regions occupied by these agents. c , World-model-guided action selection. The NWM imagines the future for each of K candidate ego trajectories, and the candidate with the highest reward is selected. The procedure is evaluated on NAVSIM and on real-vehicle tests with a pedestrian dummy.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Stage 1: backbone adaptation
Stage 2: ControlNet
Initialization
Pretrained SVD, 25-frame XT checkpoint
Stage 1 at step 52,800; copied ControlNet blocks and zero-initialized residual projections
Trainable modules
Full spatiotemporal U-Net
ControlNet, trajectory projection and null embedding
Frozen modules
VAE and CLIP image encoder
VAE, CLIP image encoder and adapted U-Net
Training data
MM-AU and nuScenes
MM-AU and nuScenes
Clip dimensions
25 frames, 512×320
25 frames, 512×320
Optimizer
AdamW
AdamW
Appendix
Supplementary Table S1: Training configuration of the neutral world model. Settings for backbone adaptation and conditional training; batch sizes are measured in video clips. NWM, neutral world model; SVD, Stable Video Diffusion; VAE, variational autoencoder; CLIP, Contrastive Language–Image Pre-training; FP16, 16-bit floating point; EMA, exponential moving average; GPU, graphics processing unit.
Setting
Value
Sampler / denoising steps
Euler / 30
Noise schedule
Karras; σmin=0.002 , σmax=700 , ρ=7
Reference-model guidance
Linear increase from 1 to 9 across frames
ControlNet scale
1.0
Image noise augmentation
0.01
Motion bucket identifier
127
Appendix
Supplementary Table S2: NWM inference settings and computational cost. Latency is measured after two warm-up calls over five distinct inputs on the stated inference GPU. NWM, neutral world model; VAE, variational autoencoder; RGB, red–green–blue; FP16, 16-bit floating point; GPU, graphics processing unit; SD, sample standard deviation; GiB, 230 bytes.
Dataset
Training
Validation
Effective training
nuScenes
7,700
450
11,500
MM-AU (crash)
6,194
733
9,630
MM-AU (nominal)
1,745
219
2,381
MM-AU (total)
7,939
952
12,011
Combined total
15,639
1,402
23,511
Appendix
Supplementary Table S3: Dataset splits and training composition. Counts of extracted 25-frame clips and effective training samples after maneuver-based repetition.
Supplementary Fig. S8: Interface for manual assessment of generated outcomes. The interface presents generated videos for inspection with model names hidden and the presentation order randomized. Collision outcomes are assessed over the complete 25-frame clip.
Supplementary Fig. S9: Research van and real-world crossing configurations. a , Ford E-Transit research van operated by the Wisconsin Transportation Center and sensors. b , Pedestrian dummy and wheeled cart used for manually pulled crossings. c , Recorded views of the six configurations in Supplementary Table S4 , showing variations in snow presence, clothing and crossing origin relative to a parked vehicle.
Trial
Date
Snow present
Crossing origin
Dummy clothing
T1
February 23, 2026
No
In front of parked vehicle
Black
T2
February 23, 2026
No
In front of parked vehicle
Purple
T3
February 23, 2026
No
Behind parked vehicle
Black
T4
February 23, 2026
No
Behind parked vehicle
Purple
T5
March 17, 2026
Yes
In front of parked vehicle
Purple
T6
March 17, 2026
Yes
Behind parked vehicle
Purple
Appendix
Supplementary Table S4: Configurations of the six real-world trials. Crossing origin is specified relative to the parked vehicle. Snow presence describes the observed site conditions. Each trial contributes five Normal and five Response decision frames.
Supplementary Fig. S10: Recorded vehicle motion and decision-frame selection. Aligned displacement, speed, throttle and brake signals for the six trials. Markers identify the five Normal frames (A1–A5) and five Response frames (B1–B5) selected from each recording. Brake signals are shown in their recorded raw units.
Supplementary Fig. S11: Computation of reward from imagined futures. a , Candidate trajectories generate alternative futures from a shared current observation. b , Detection and temporal association yield target boxes, which are projected to approximate ground-plane positions and vehicle clearances. c , Observed proximity and warning times determine S and TTC ; reference context and candidate motion determine P and C . d , The weighted reward ranks the candidate actions.
Parameter
Meaning
Value
wT,wP,wC
Reward weights
5,5,2
H , Tsafe
Rollout horizon; warning-time normalization
4 s; 4 s
α
Proximity-event gate coefficient
0.5
dc , dw
Proximity and warning clearances
0.5 m; 1.0 m
a0 , j0
Acceleration and jerk scales
3 m / s 2 ; 5 m / s 3
λj
Relative jerk-cost weight
1
Appendix
Supplementary Table S5: Reward parameters and detection settings. Values for the reward implementation. IoU, intersection over union; NMS, non-maximum suppression.
Supplementary Fig. S12: Examples of reward-guided action selection. a , A routine-driving input illustrates the selection for continuing motion when progress is appropriate. b , A pedestrian-crossing input shows candidate futures generated by NWM and Vista, with tracked target detections. c , Clearance estimates show how the safety components are obtained.
World-action models (WAMs) jointly model future traffic states and driving actions for end-to-end autonomous driving in intelligent transportation systems. However, their action learning remains dominated by expert-trajectory imitation, leaving unsafe alternatives close to expert trajectories insufficiently constrained. To improve WAM robustness in safety-critical scenarios, we propose NegWAM, a safety post-training framework for unified WAMs that learns explicit safety boundaries from near-expert failure trajectories. NegWAM constructs negative trajectories that are geometrically close to the expert trajectory yet carry safety risks, uses them as repulsive anchors to steer predictions away from dangerous behavior, and encodes them as scene-conditioned rider tokens to learn scene--trajectory compatibility. During training, NegWAM jointly optimizes the flow-matching and near-expert negative objectives, while the auxiliary rider branch is discarded at inference, incurring no additional overhead. On NAVSIM v2, NegWAM achieves 40.4 EPDMS on navhard, a 4.1% relative improvement over the strongest compared scorer-free WAM baseline, and achieves 93.3 EPDMS on navtest. These results demonstrate that NegWAM achieves strong performance in safety-critical scenarios while preserving routine-driving performance. Zero-shot closed-loop evaluation on HUGSIM further demonstrates cross-benchmark generalization. The code is available at https://github.com/Zizizi-hao/NegWAM.
Zihao Zhang, Xinyu Zhang, Bing Zeng +4
Southeast University · Shenzhen Loop Area Institute · Guangxi University +2
World models for autonomous driving should anticipate the consequences that matter for driving decisions rather than reconstruct every detail of future scenes. However, predictive representations alone are not guaranteed to distinguish candidate actions that lead to different planning outcomes.We propose DA-WAM, a decision-aligned world action model that couples a Joint-Embedding Predictive Architecture (JEPA) with candidate-level trajectory scoring. During joint training, the JEPA objective grounds future representations in observed scene dynamics, while the planning objectives encourage these representations to capture distinctions relevant to safety and utility. An action-conditioned predictor generates a distinct future latent for each candidate trajectory, and a multi-head trajectory scorer uses the corresponding latent to estimate planning factors and overall utility. In addition, safety-critical post-training with hard negatives further improves discrimination between geometrically similar trajectories with different safety outcomes. DA-WAM achieves state-of-the-art performance among learned camera-based planners on NAVSIM-v1, NAVSIM-v2, and Bench2Drive. Ablation studies demonstrate the effectiveness of each component in driving decision-making.
Ruiguo Zhong, Xiaolong Chen, Benshan Ma +5
The Hong Kong University of Science and Technology (Guangzhou) · Leapmotor · The Hong Kong University of Science and Technology
Vision-Language-Action (VLA) models have recently achieved notable progress in end-to-end autonomous driving by integrating perception, reasoning, and control within a unified multimodal framework. However, they often lack explicit modeling of temporal dynamics and global world consistency, which limits their foresight and safety. In contrast, world models can simulate plausible future scenes but generally struggle to reason about or evaluate the imagined future they generate. In this work, we present VLA-World, a simple yet effective VLA world model that unifies predictive imagination with reflective reasoning to improve driving foresight. VLA-World first uses an action-derived feasible trajectory to guide the generation of the next-frame image, capturing rich spatial and temporal cues that describe how the surrounding environment evolves. The model then reasons over this self-generated future imagined frame to refine the predicted trajectory, achieving higher performance and better interpretability. To support this pipeline, we curate nuScenes-GR-20K, a generative reasoning dataset derived from nuScenes, and employ a three-stage training strategy that includes pretraining, supervised fine-tuning, and reinforcement learning. Extensive experiments demonstrate that VLA-World consistently surpasses state-of-the-art VLA and world-model baselines on both planning and future-generation benchmarks. Project page: https://vlaworld.github.io
Guoqing Wang, Pin Tang, Xiangxuan Ren +3
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · Central Research Institute, Huawei