DriveReferee: Geometric Safety Verdicts Need Not Be Learned for Driving World-Action Models

Fengcheng Yu; Dhruv Parikh; Junjie Ye; Maulik Bhatt; Thang Vu; Igor Vasiljevic; Vitor Guizilini; Yue Wang

cs.CVSep 19, 2026

DriveReferee: Geometric Safety Verdicts Need Not Be Learned for Driving World-Action Models

Authors: Fengcheng Yu, Dhruv Parikh, Junjie Ye, Maulik Bhatt, Thang Vu, Igor Vasiljevic, Vitor Guizilini, Yue Wang

Organizations: University of Southern California · Woven by Toyota · Toyota Research Institute

Abstract

Generative world-action models (WAMs) jointly generate future video and vehicle actions, while their action branches remain primarily optimized by expert imitation. Yet imitation provides no explicit closed-loop geometric verdict for generated trajectories, making verification important during both training and deployment. Closed-loop evaluators can check collision and drivable-area violations, but require privileged scene state unavailable at deployment. Existing approaches often close this gap by learning a verifier from sensor features. For these geometric checks, the rule itself is explicit. For example, collision is determined by whether the rolled-out ego footprint overlaps occupied vehicle space. What is unavailable at deployment is the scene state needed to apply the rule. We introduce DriveReferee, which uses a learned geometry readout to predict the scene representation from camera observations and executes the geometric safety rule directly rather than learning it. The resulting analytic referee evaluates collision and drivable-area safety from a scene state and candidate trajectory. During training, it scores self-sampled trajectories on ground-truth state and distills the resulting preferences into the WAM policy. At deployment, the same referee evaluates generated trajectories on this predicted state and selects a safer alternative when needed. The analytic referee requires no verdict-specific training, and its decisions follow an explicit geometric rule. Under matched candidates and inference budgets, it matches or outperforms all learned-verifier and heuristic baselines. Given the same predicted state and trajectory, learning the verdict provides no measurable downstream gain despite requiring tens of thousands of evaluator-labeled training examples. On the full NAVSIM navtest, DriveReferee reaches 92.02 PDMS with single-camera visual input and no external training data.

Figures & tables

Panel of Figure 1 — Fig. 1 : DriveReferee recovers the state, instead of the rule. The geometric safety verdict is a computation whose input, not whose logic, is missing at deployment.

Fig. 2 : DriveReferee overview. The policy $\pi_{\theta}$ (frozen reasoner, trained generator) generates future video $\hat{V}_{k}$ and candidate plans $\tau_{k}$ ; the readout $g_{\phi}$ predicts the BEV scene state $\hat{M}=(\hat{M}_{\mathrm{drv}},\hat{M}_{\mathrm{veh}})$ , containing drivable-area and vehicle-occupancy maps, from frozen visual features and ego poses $P_{-H:0}$ . The analytic referee $R(\tau,M)$ , with no learned parameters, is placed twice: at training (green) on $M_{\mathrm{gt}}$ to form preference pairs $(\tau^{w},\tau^{l})$ distilled into the generator ( Eq. 3 ); at deployment (blue) on $\hat{M}$ to gate the default plan and, on alarm, rank $K{-}1$ further samples on the same map.

Method	Input	Video	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$	EPDMS $\uparrow$
DrivingGPT [ 27 ]	1 $\times$ C	✓	98.9	90.7	94.9	95.6	79.7	82.4	—
WoTE [ 9 ]	3 $\times$ C+L	✗	98.5	96.8	94.9	99.9	81.9	88.3	—
DriveVLA-W0 [ 28 ]	1 $\times$ C	✓ †	98.7	99.1	95.3	99.3	83.3	90.2	86.1
PWM [ 29 ]	1 $\times$ C	✓	98.6	95.9	95.4	100.0	81.8	88.1	—
DriveLaW ‡ [ 30 ]	1 $\times$ C	✓ †	99.0	97.1	96.7	100.0	81.3	89.1	—
CoPhy § [ 10 ]	C+L	✗	99.0	98.2	96.8	100.0	85.3	91.4	86.1

TABLE I : Comparison with representative world-model-based planners on navtest (12,146 scenes).

N{\times}

C denotes

N

cameras and +L denotes LiDAR. Video indicates whether video generation is part of the model. PDMS follows NAVSIM v1 and EPDMS follows v2. NC, DAC, TTC, Comf., and EP denote no at-fault collisions, drivable-area compliance, time-to-collision, comfort, and ego progress, respectively. Higher is better; “—” denotes unreported results.

Fig. 3 : Matched-budget verdict-source comparison. Per-scene paired $\Delta$ EPDMS relative to the imitation-only base on navtest ( $n{=}12{,}146$ ). All sources use the same pre-sampled $K{=}2$ candidates and are calibrated to an average sampling budget of about 1.35 $\times$ . Error bars are bootstrap 95% confidence intervals. Numbers at right report replaced plans.

Fig. 4 : Verdict-supervision scaling in the same-map setting. Only the amount of verdict-training data is varied; 100% corresponds to 66,385 training candidates. Points show the mean over two log-level draws, with bootstrap 95% confidence intervals. The analytic referee uses no verdict-training labels.

Fig. 5 : Deployment-time verification on self-predicted maps (base policy, $K{=}2$ ). (a) One intervention example. At the flagged state, the default plan crosses the predicted drivable boundary by 0.40 m, while the referee-selected plan keeps 1.26 m clearance. The signed-distance curve and projected boundaries show the same failure in BEV and video space; the referee’s clearance margin is clipped to zero once the boundary is crossed. (b) Four additional replacements found by automatic screening. In each case, the default plan leaves the ground-truth drivable area, while the selected plan remains inside both the predicted and ground-truth boundaries (local IoU $\geq 0.90$ ).

Variant	$\Delta$ EPDMS
Preference objective
$\beta{=}0$	$+0.56$
full preference objective	$\mathbf{+0.92}$
Winner construction
nearest to expert	$+0.63$
largest safety margin (ours)	$\mathbf{+0.92}$

TABLE II : Training design ablations. Each block changes one design choice while keeping the remaining settings fixed. Values are paired

\Delta

EPDMS over the imitation-only base on navtest (

n{=}12{,}146

Method	FVD $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	Frames	Resolution
Base (imitation only)	23.90	0.4166	18.75	9 @ 2Hz	832 $\times$ 480
+ referee distillation	24.25	0.4126	18.92	9 @ 2Hz	832 $\times$ 480
DriveDreamer-Policy [ 4 ]	53.59	—	—	9	144 $\times$ 256
PWM [ 29 ]	85.95	—	—	10	128 $\times$ 224
ForgeDrive [ 35 ]	69.2	—	—	8 @ 2Hz	512 $\times$ 1024
DrivingGPT [ 27 ]	142.6	—	—	12	288 $\times$ 512

TABLE III : Video generation quality. Our rows use the same navtest scenes, protocol, and sampling seeds. Prior-work FVD values are for context only because evaluation settings differ.

Autonomous Driving Safety Evaluation

Autonomous driving safety evaluation studies how to measure, test, and improve the safety of self-driving systems in routine and hazardous conditions.

See all 159 papers

GeoWAM: Visual Geometry World Action Models for Autonomous Driving
Aug 24, 2026Yiren Lu, Xin Ye, Jiaming Liu +103D Scene Prediction Autonomous Driving
Uber AV Labs · Case Western Reserve University
DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning
Apr 2, 2026Yang Zhou, Xiaofeng Wang, Hao Shao +8World Action Models World Models for Autonomous Driving
GigaAI · University of Toronto · CUHK MMLab
DriveVA: Video Action Models are Zero-Shot Drivers
Apr 5, 2026Mengmeng Liu, Diankun Zhang, Jiuming Liu +7Action-Conditioned Video Generation World Models
University of Twente, The Netherlands · Xiaomi EV, China · University of Cambridge, United Kingdom +1

Aug 24, 2026J/K move · Enter open · S save

GeoWAM: Visual Geometry World Action Models for Autonomous Driving

Yiren Lu, Xin Ye, Jiaming Liu, Philip Jacobson+9

Uber AV Labs · Case Western Reserve University

3D Scene Prediction Autonomous Driving Geometric Representation Learning World Models for Autonomous Driving Autonomous Driving Planning

Open paper AskPDF arXiv

World action models (WAMs) have recently gained increasing attention as a framework for jointly modeling scene evolution and ego actions in autonomous driving. Most existing WAMs learn scene dynamics in pixel space by combining a video-generation backbone for future-observation prediction with an action head for ego-trajectory prediction. Pixels, however, provide only an indirect representation of these dynamics: they entangle geometry and motion with appearance, texture, and illumination, forcing the model to infer three-dimensional transformations from two-dimensional observations. We argue that point-based geometry provides a more natural state space for driving. It explicitly captures spatial structure and both rigid and non-rigid scene dynamics while remaining aligned with the 3D space of driving actions. Building on this insight, we introduce GeoWAM, a visual geometry world action model for autonomous driving. Rather than predicting future images, GeoWAM is pretrained to forecast future scene geometry, yielding representations that jointly encode spatial structure and temporal evolution. A geometry-conditioned action head then leverages these learned geometric dynamics to predict future ego-trajectories. Extensive experiments show that GeoWAM outperforms image-based alternatives, achieving a combined EPDMS of 36.6 on navhard without PDMS supervision and strong zero-shot generalization to nuScenes, with a collision rate of 0.24%. Scaling geometry pretraining with unlabeled data further improves performance, increasing the navhard score by 8.2% to 39.6 and strengthening zero-shot transfer to nuScenes, where the collision rate is reduced by 50% to 0.12%. Together, these results establish geometry as an effective state representation for autonomous driving and geometry pretraining as a general, scalable strategy for downstream planning.

Abstract

Autonomous Driving Safety Evaluation

Explore similar work

GeoWAM: Visual Geometry World Action Models for Autonomous Driving

DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning

DriveVA: Video Action Models are Zero-Shot Drivers