When the Score Becomes the Target: Rethinking Metric Validity in Autonomous Driving

Morui Zhu; Deyuan Qu; Qi Chen; Kentaro Oguchi; Qing Yang

cs.CVSep 28, 2026

When the Score Becomes the Target: Rethinking Metric Validity in Autonomous Driving

Authors: Morui Zhu, Deyuan Qu, Qi Chen, Kentaro Oguchi, Qing Yang

Organizations: University of North Texas · Toyota InfoTech Labs

Abstract

Driving benchmark scores are increasingly used not only for evaluation but also as optimization targets. This raises a fundamental question: do score gains remain reliable evidence of driving improvement once the score itself is optimized? We address this question by examining how the scoring process responds to changes in driving behavior and whether the resulting gains persist under repeated execution and replanning. We decompose the process into execution, measurement, subscore mapping, and aggregation. Controlled interventions reveal substantial behavioral changes that receive little score response because distinctions are omitted, thresholded, or attenuated between requested and executed motion. Closed-loop comparisons further show that optimization gains can reverse when the execution interface changes, demonstrating their dependence on how requests are executed and returned as feedback. Together, these findings connect the behavioral distinctions preserved by a metric to the conditions under which its gains transfer. Metric validity under optimization therefore requires examining both what the scoring process measures and how the optimized behavior is executed.

Figures & tables

Figure 1: Optimization and score–behavior mismatch. (a) Conceptual distributions: optimizing $R$ changes $P_{0}$ to $P_{R}$ ; higher $S$ does not imply better $Q$ . (b) Group-mean score components $c_{k}$ and diagnostics $D_{k}$ for PDMS (Predictive Driver Model Score). Each axis is oriented so that a larger radius represents better compliance or a smaller behavioral error. Values are normalized relative to the logged-human mean, shown at radius 1, to place heterogeneous quantities on a common visual scale. The radial scale is nonlinear.

		Aggregate scores		EPDMS components			Behavioral diagnostics
Planner	Variant	PDMS $\uparrow$	EPDMS $\uparrow$	DDC v2 $\uparrow$	LK $\uparrow$	HC $\uparrow$	DDC v1 $\uparrow$	Jitter $\downarrow$	Smooth. $\downarrow$	Adh. $\downarrow$
Ego-status MLP	IL	$0.664$	$0.645$	$0.927$	$0.792$	$0.983$	$0.904$	$0.49$	$0.086$	$0.587$
TransFuser	IL	$0.839$	$0.814$	$0.992$	$0.848$	$0.983$	$0.980$	$0.50$	$0.092$	$0.603$
Latent TransFuser	IL	$0.835$	$0.810$	$0.992$	$0.853$	$0.983$	$0.979$	$0.53$	$0.091$	$0.624$
ReCogDrive 2B	IL	$0.863$	$0.834$	$0.993$	$0.860$	$0.983$	$0.980$	$0.50$	$0.105$	$0.575$
ReCogDrive 2B	RL	$0.905$ (+0.042)	$0.854$ (+0.019)	$0.982$ (−0.011)	$0.802$ (−0.058)	$0.977$ (−0.006)	$0.967$ (−0.013)	$1.69$ (+1.19)	$0.288$ (+0.183)	$0.893$ (+0.319)

Table 1: NAVSIM navtest : scores and behavioral changes. Absolute values with within-family changes below optimized entries, computed before rounding; green/red denotes improvement/deterioration. EPDMS rescores fixed outputs. DDC v2 excludes intersection positions. Jitter: rad; Smooth./Adh.: smoothing displacement/plan-adherence error [m]; LK: lane keeping; HC: history comfort. † marks changes not supported by the paired statistical criteria in Appendix A.4 .

Quantity	Role in CLS	$\Delta$ NR	$\Delta$ R
CLS $\uparrow$	Aggregate	$\mathbf{+0.034}$	$\mathbf{+0.067}$
Drivable-area compliance $\uparrow$	Gate	$\mathbf{+0.042}$	$\mathbf{+0.033}$
Planned heading jitter $\downarrow$	Absent	$\mathbf{-0.235}$	$\mathbf{-0.538}$
Absolute lon. jerk peak [m/s 3 ] $\downarrow$	Threshold	$\mathbf{+0.136}$	$\mathbf{+0.206}$
Absolute jerk peak [m/s 3 ] $\downarrow$	Threshold	$\mathbf{+0.068}$	$\mathbf{+0.145}$
Comfort term $\uparrow$	Weighted term	$\mathbf{-0.002^{\dagger}}$	$\mathbf{-0.009}$

Table 2: nuPlan val14 . Plan-R1 IL

\to

RL under non-reactive (NR) and reactive (R) traffic.

Figure 2: Illustrative trajectory comparison. WorldEngine case; labels: smoothing displacement (m, lower is better) / PDMS. The oscillatory v2 plan scores 0.979 versus v1 0.874 and Human 0.878.

		Stage 1 components				Stage 2 components				Overall
Planner	Variant	DAC $\uparrow$	DDC $\uparrow$	EP $\uparrow$	EC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	EP $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
ReCogDrive 2B	IL	$0.827$	$0.984$	$0.828$	$0.764$	$0.721$	$0.823$	$0.792$	$0.688$	$0.278$
ReCogDrive 2B	RL	$0.907$ (+0.080)	$0.956$ (−0.029)	$0.856$ (+0.028)	$0.329$ (−0.436)	$0.726$ (+0.005)	$0.812$ (−0.011)	$0.853$ (+0.061)	$0.310$ (−0.378)	$0.270$ (−0.008 $\dagger$ )
ReCogDrive 8B	IL	$0.842$	$0.990$	$0.834$	$0.769$	$0.721$	$0.821$	$0.808$	$0.680$	$0.263$
ReCogDrive 8B	RL	$0.911$ (+0.069)	$0.973$ (−0.017)	$0.860$ (+0.027)	$0.204$ (−0.564)	$0.728$ (+0.008)	$0.819$ (−0.002)	$0.864$ (+0.056)	$0.153$ (−0.527)	$0.255$ (−0.008 $\dagger$ )
Qwen-Drive	IL	$0.887$	$0.994$	$0.829$	$0.813$	$0.753$	$0.843$	$0.771$	$0.647$	$0.303$

Table 3: NAVSIM-v2 navhard-two-stage results. Selected EPDMS components on original (Stage 1) and synthetic (Stage 2) scenes, and combined score. RL entries show absolute values and parenthesized RL-minus-IL changes. EC: two-frame extended comfort.

	LQR + bicycle model			Playback			LQR $-$ PB		Interaction
Planner	Ref.	Opt.	$\Delta$	Ref.	Opt.	$\Delta$	Opt.	Ref.	$I_{\mathrm{LQR},\mathrm{PB}}$ [95% CI]
ReCogDrive-2B	0.580	0.798	$\mathbf{+0.218}$	0.688	0.452	$\mathbf{-0.236}$	$\mathbf{+0.346}$	$\mathbf{-0.108}$	$\mathbf{+0.454}\;[+0.377,+0.533]$
ReCogDrive-8B	0.628	0.786	$\mathbf{+0.158}$	0.690	0.490	$\mathbf{-0.200}$	$\mathbf{+0.295}$	$\mathbf{-0.062}$	$\mathbf{+0.358}\;[+0.274,+0.436]$
Qwen-Drive	0.697	0.834	$\mathbf{+0.136}$	0.787	0.398	$\mathbf{-0.388}$	$\mathbf{+0.435}$	$\mathbf{-0.089}$	$\mathbf{+0.525}\;[+0.448,+0.603]$
NoRD	0.462	0.648	$\mathbf{+0.186}$	0.516	0.672	$\mathbf{+0.156}$	$\mathbf{-0.024}$	$\mathbf{-0.054}$	$+0.030\;[-0.044,+0.102]$
TOAD	0.843	0.916	$\mathbf{+0.073}$	0.761	0.854	$\mathbf{+0.093}$	$\mathbf{+0.063}$	$\mathbf{+0.082}$	$-0.020\;[-0.062,+0.022]$

Table 4: Gain reversals across execution interfaces. Mean PDMS on matched NR scenes per family. Ref./Opt.: IL/base

\to

RL, except TOAD Frozen

\to

CEM;

\Delta

: Opt.

-

Ref. Interaction:

I_{\mathrm{LQR},\mathrm{PB}}=\Delta S_{\mathrm{LQR}}-\Delta S_{\mathrm{PB}}

(Eq. 7 ).

Figure 3: Qwen-Drive at the same junction. Historical WorldEngine non-reactive playback front-camera renders with timestamps and longitudinal speeds.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Comfort sensitivity and plan adherence on navhard . Left: Mean utilization of each comfort bound for DiffusionDrive v1, v2, and logged Human trajectories; a value of one denotes the bound. The lateral-acceleration check is constant by construction. Right: Empirical cumulative distribution of the maximum frame-wise distance between planned waypoints and LQR-executed positions, in meters.

	navhard				WorldEngine test set
Comfort term	DD	DDv2	Human	Retained	DD	DDv2	Human	Retained
Official (binary)	0.7044	0.7328	0.8992	100%	0.6540	0.6960	0.9325	100%
Worst-channel margin	0.6616	0.6801	0.8458	65%	0.6144	0.6458	0.8778	75%
Smooth sigmoid	0.7038	0.7315	0.8982	97%	0.6534	0.6948	0.9313	98%
Mean margin	0.6903	0.7138	0.8814	83%	0.6410	0.6777	0.9143	87%

Appendix

Table 5: Comfort rescoring on navhard and the WorldEngine test set. Mean PDMS (

\uparrow

) under the official and continuous comfort formulations, using fixed outputs. DD denotes DiffusionDrive and DDv2 denotes DiffusionDriveV2. Retained reports the DDv2–DD difference relative to the official binary difference, computed from unrounded means.

Figure 5: Planned and executed speed. (a) Mean planned speed relative to ego speed on navtest . (b) Mean speed spectrum after linear detrending (5 s, 10 Hz). (c) Mean longitudinal speed in WorldEngine NR simulation ( Li et al., 2026a ) ; matched sets: Appendix A . Shading: logged history; zero: first action.

	Smoothing displacement [m] $\downarrow$			Planned [m/s 2 ]		Simulated [m/s 2 ]
Checkpoint	Total	Along	Across	Acc.	Dec.	Acc.	Dec.
Qwen-Drive SFT	$0.116$	$0.103$	$0.049$	$0.67$	$0.66$	$0.32$	$0.41$
Qwen-Drive RL	$0.212$	$0.204$	$0.059$	$1.65$	$1.88$	$0.28$	$0.53$
Human	$0.099$	$0.081$	$0.050$	$0.37$	$0.43$	$0.32$	$0.43$

Appendix

Table 6: NAVSIM-v1 navtest : plan and simulated motion. Geometry and pace of the Qwen-Drive experts and the human trajectory: smoothing displacement and its components along and across the local path direction, planned peak acceleration and deceleration between consecutive 0.5 s segments, and the peak acceleration and deceleration of the simulated states that the scorer evaluates.

	$\max\|\Delta v\|/\Delta t$ [m/s 2 ] $\downarrow$		PDMS $\uparrow$
Checkpoint	Original	Retimed	Original	Retimed	$\Delta$ [95% CI]
Qwen-Drive SFT	$1.24$	$0.91$	$0.8791$	$0.8790$	$-.0002$ [ $-.0005$ , $.0002$ ]
Qwen-Drive RL	$2.46$	$1.31$	$0.9047$	$0.9045$	$-.0002$ [ $-.0008$ , $.0003$ ]
Human	$0.97$	$0.87$	$0.9455$	$0.9453$	$-.0002$ [ $-.0005$ , $.0000$ ]

Appendix

Table 7: NAVSIM-v1 navtest : retiming intervention. Re-timing the deployed trajectory along its own path: largest speed change per unit time between consecutive segments before and after, PDMS before and after, and the paired change with its bootstrap 95% interval.

navhard
	Aggregate scores		EPDMS components			Behavioral diagnostics
Pair	PDMS $\uparrow$	EPDMS $\uparrow$	DDC v2 $\uparrow$	LK $\uparrow$	HC $\uparrow$	DDC v1 $\uparrow$	Jitter $\downarrow$	Smooth. $\downarrow$	Adh. $\downarrow$
ReCog-2B IL $\to$ RL	$+.105$	$+.077$	$-.029$	$-.044$	$-.004$	$-.032$	$+0.83$	$+.154$	$+.385$
ReCog-8B IL $\to$ RL	$+.079$	$+.060$	$-.016$	$-.024$	$-.004$	$-.012$	$+0.71$	$+.164$	$+.399$
Qwen-Drive SFT $\to$ RL	$+.063$	$+.062$	$-.001$	$+.016$	$+.000$	$-.001$	$+.03$	$+.091$	$-.067$
DDv1 $\to$ v2	$+.028$	$-.000$	$-.017$	$-.011$	$-.096$	$-.024$	$+1.25$	$+.310$	$+.609$

Appendix

Table 8: Paired changes on safety-critical cases. Scores, EPDMS components, and behavioral diagnostics on fixed planner outputs from navhard and the WorldEngine test set. PDMS uses the v1 scorer; EPDMS and its components use the v2 scorer. DDC v1 has zero PDMS weight, whereas DDC v2 gates EPDMS and excludes intersection positions. Jitter is in radians; smoothing displacement (Smooth.) and plan-adherence error (Adh.) are in meters. Gray entries have unadjusted paired-test

p\geq 0.05

in these subset analyses; navtest adjusted tests and intervals appear in Tables 9 and 18 .

Pair	PDMS	DDC	Jitter	Smooth.	Adh.
ReCog-2B IL $\to$ RL	$<\!10^{-4}$	$<\!10^{-4}$	$<\!10^{-4}$	$<\!10^{-4}$	$<\!10^{-4}$
Scene CI	[ $.038$ , $.046$ ]	[ $-.015$ , $-.011$ ]	[ $1.15$ , $1.23$ ]	[ $.180$ , $.185$ ]	[ $.311$ , $.326$ ]
Log CI	[ $.035$ , $.050$ ]	[ $-.021$ , $-.008$ ]	[ $1.07$ , $1.31$ ]	[ $.176$ , $.189$ ]	[ $.287$ , $.351$ ]
ReCog-8B IL $\to$ RL	$<\!10^{-4}$	$<\!10^{-4}$	$<\!10^{-4}$	$<\!10^{-4}$	$<\!10^{-4}$
Scene CI	[ $.031$ , $.038$ ]	[ $-.007$ , $-.004$ ]	[ $.88$ , $.95$ ]	[ $.171$ , $.174$ ]	[ $.391$ , $.406$ ]
Log CI	[ $.029$ , $.040$ ]	[ $-.008$ , $-.003$ ]	[ $.84$ , $1.00$ ]	[ $.168$ , $.177$ ]	[ $.372$ , $.427$ ]

Appendix

Table 9: NAVSIM-v1 navtest : paired uncertainty. Holm-adjusted Wilcoxon

p

-values (first row), scene-bootstrap 95% intervals (second row), and driving-log-bootstrap 95% intervals (third row) for the five v1 quantities of Table 1 . Correction uses the original families specified in Appendix A.4 , including TOAD’s augmentation-only arm.

Figure 6: Additional illustrative trajectory comparisons. Six selected WorldEngine test scenes ( Li et al., 2026a ) . Each row pairs a cropped front-camera view with a bird’s-eye view; curves show planned waypoints. Values give smoothing displacement (m) / PDMS in legend order. In every scene, v2 scores above both v1 and Human while all three pass NC, DAC, TTC, and comfort after tracking. BEV scale bars indicate the per-scene zoom. These cases illustrate the score–geometry mismatch; Table 1 reports aggregate diagnostic changes.

Figure 7: TOAD search-depth sweep on navhard . Fixed DrivoR weights and 64 samples per CEM round are used throughout. Panels report official PDMS, EPDMS, DDC, and heading jitter (radians). Squares denote TOAD’s deployed trajectory; circles denote the candidate with the highest predicted PDMS among all evaluated candidates, starting from the augmented proposal set at round zero. Dashed lines show the planner’s argmax without augmentation or search. Candidate ranking uses the learned score; the plotted scores are obtained by evaluating the selected trajectories.

Planner	$\Delta S_{\mathrm{PB}}$	$\Delta S_{\mathrm{LQR}}$
ReCogDrive-2B	$-0.236$ [-0.300, -0.169]	$+0.218$ [+0.165, +0.273]
ReCogDrive-8B	$-0.200$ [-0.265, -0.132]	$+0.158$ [+0.101, +0.214]
Qwen-Drive	$-0.388$ [-0.457, -0.321]	$+0.136$ [+0.071, +0.198]
NoRD	$+0.156$ [+0.102, +0.210]	$+0.186$ [+0.128, +0.244]
TOAD	$+0.093$ [+0.043, +0.137]	$+0.073$ [+0.042, +0.104]

Appendix

Table 10: Paired PDMS gains within each execution interface. Mean [log-bootstrap 95% CI], the full scene set per row. Interaction estimates and intervals appear in the main comparison (Table 4 ). Each bootstrap resamples driving logs with all their scenes (2,000 draws, seed 0). Intervals are pointwise.

Planner	Interface	NC	DAC	EP	TTC	C	DDC
ReCogDrive-2B	PB	$0.92\!\to\!0.80$	$0.86\!\to\!0.67$	$0.59\!\to\!0.38$	$0.90\!\to\!0.80$	$1.00\!\to\!1.00$	$0.97\!\to\!0.83$
ReCogDrive-2B	LQR	$0.94\!\to\!0.94$	$0.66\!\to\!0.91$	$0.57\!\to\!0.76$	$0.90\!\to\!0.91$	$1.00\!\to\!1.00$	$0.93\!\to\!0.99$
ReCogDrive-8B	PB	$0.93\!\to\!0.84$	$0.85\!\to\!0.70$	$0.59\!\to\!0.40$	$0.91\!\to\!0.85$	$1.00\!\to\!1.00$	$0.98\!\to\!0.85$
ReCogDrive-8B	LQR	$0.95\!\to\!0.93$	$0.70\!\to\!0.90$	$0.63\!\to\!0.76$	$0.91\!\to\!0.91$	$1.00\!\to\!1.00$	$0.95\!\to\!0.99$
Qwen-Drive	PB	$0.96\!\to\!0.75$	$0.91\!\to\!0.61$	$0.68\!\to\!0.40$	$0.96\!\to\!0.74$	$1.00\!\to\!0.97$	$0.99\!\to\!0.86$
Qwen-Drive	LQR	$0.95\!\to\!0.98$	$0.77\!\to\!0.91$	$0.69\!\to\!0.76$	$0.93\!\to\!0.98$	$1.00\!\to\!1.00$	$0.99\!\to\!0.99$

Appendix

Table 11: Full-cohort component means. Each entry is reference

\to

optimized, on the same NR scenes per family and interface. NC/DAC are multiplier penalties, EP/TTC/C enter the weighted average, and DDC is diagnostic with zero PDMS weight. PB: playback. TOAD uses Frozen

\to

CEM.

Figure 9: Historical playback gain transfer on the WorldEngine test set ( Li et al., 2026a ) . Open-loop versus NR closed-loop PDMS changes (optimized minus baseline): IL

\to

RL for the first four panels, DrivoR

\to

TOAD for the last. Points: scenes; diamonds: paired means. Mean changes appear in Table 12 .

ReCogDrive	Open loop	First CL	All CL
	$n=286$	$n=286$	$n=2{,}288$
2B-IL	0.0%	0.0%	17.9%
2B-RL	6.3%	6.3%	36.2%
8B-IL	0.3%	0.3%	7.6%
8B-RL	1.4%	0.7%	35.4%

Appendix

Table 13: ReCogDrive predicted foldbacks. Historical WorldEngine playback with non-reactive traffic, 286 matched scenes; 0.05 m filter. CL: closed loop. Rates over

n

predictions (lower is better).

		navtest	WorldEngine test set (NR)
Planner	Variant	Full OL	Matched OL	First CL	All CL	Any CL
	Denominator	12,146	286	286	2,288	286
ReCogDrive-2B	IL	14 / 0	0 / 0	0 / 0	410 / 13	146 / 7
	RL	522 / 31	18 / 2	18 / 1	829 / 528	226 / 197
ReCogDrive-8B	IL	13 / 0	1 / 0	1 / 0	174 / 1	68 / 1
	RL	64 / 0	4 / 0	2 / 0	810 / 646	233 / 214

Appendix

Table 14: ReCogDrive foldbacks on navtest and the WorldEngine test set. Each cell gives counts at

\epsilon=0.05

m /

0.5

m. Column denominators are per configuration. The last column counts scenes with at least one qualifying prediction; the preceding columns count predictions. No outcome-based filtering or new model inference is used.

	Non-reactive		Reactive
Checkpoint	Published $\uparrow$	Ours $\uparrow$	Published $\uparrow$	Ours $\uparrow$
PDM-Closed	$92.84$	$92.84$	$92.12$	$92.13$
Plan-R1 imitation	–	$85.48$	–	$81.01$
Plan-R1 reinforcement	$88.98$	$88.89$	$87.69$	$87.71$

Appendix

Table 15: nuPlan val14 : score reproduction. CLS in score points (

\times 100

); a dash denotes no published value.

Quantity	IL	RL	$\Delta$	Scene CI	Log CI	$p_{\rm Holm}$
CLS $\uparrow$	$0.855$	$0.889$	$+0.034$	[ $0.018$ , $0.050$ ]	[ $0.019$ , $0.050$ ]	$<10^{-4}$
Drivable-area compliance $\uparrow$	$0.942$	$0.984$	$+0.042$	[ $0.030$ , $0.054$ ]	[ $0.030$ , $0.056$ ]	$<10^{-4}$
No-at-fault-collision gate $\uparrow$	$0.956$	$0.953$	$-0.004^{\dagger}$	[ $-0.016$ , $0.008$ ]	[ $-0.015$ , $0.008$ ]	$1.000$
Driving-direction gate $\uparrow$	$0.989$	$0.980$	$-0.009^{\dagger}$	[ $-0.016$ , $-0.002$ ]	[ $-0.017$ , $-0.002$ ]	$0.134$
Making-progress gate $\uparrow$	$0.986$	$0.999$	$+0.013$	[ $0.007$ , $0.021$ ]	[ $0.007$ , $0.020$ ]	$0.002$
Progress term $\uparrow$	$0.908$	$0.923$	$+0.015$	[ $0.007$ , $0.023$ ]	[ $0.006$ , $0.023$ ]	$0.022$

Appendix

Table 16: nuPlan val14 : non-reactive traffic (NR). Plan-R1 imitation and reinforcement means, paired changes and uncertainty. † does not pass all three statistical checks of Appendix A.4 .

Quantity	IL	RL	$\Delta$	Scene CI	Log CI	$p_{\rm Holm}$
CLS $\uparrow$	$0.810$	$0.877$	$+0.067$	[ $0.051$ , $0.083$ ]	[ $0.052$ , $0.083$ ]	$<10^{-4}$
Drivable-area compliance $\uparrow$	$0.948$	$0.981$	$+0.033$	[ $0.022$ , $0.045$ ]	[ $0.021$ , $0.047$ ]	$<10^{-4}$
No-at-fault-collision gate $\uparrow$	$0.934$	$0.944$	$+0.010^{\dagger}$	[ $-0.003$ , $0.022$ ]	[ $-0.002$ , $0.021$ ]	$0.598$
Driving-direction gate $\uparrow$	$0.988$	$0.983$	$-0.005^{\dagger}$	[ $-0.011$ , $0.001$ ]	[ $-0.012$ , $0.001$ ]	$0.598$
Making-progress gate $\uparrow$	$0.960$	$0.996$	$+0.037$	[ $0.026$ , $0.048$ ]	[ $0.026$ , $0.047$ ]	$<10^{-4}$
Progress term $\uparrow$	$0.851$	$0.895$	$+0.044$	[ $0.033$ , $0.055$ ]	[ $0.034$ , $0.055$ ]	$<10^{-4}$

Appendix

Table 17: nuPlan val14 : reactive traffic (R). Plan-R1 imitation and reinforcement means, paired changes and uncertainty. † does not pass all three statistical checks of Appendix A.4 .

	NAVSIM-v1	NAVSIM-v2 scorer
Pair	$\Delta$ PDMS	$\Delta$ EPDMS	$\Delta$ DDC	$\Delta$ LK	$\Delta$ HC
ReCog-2B IL $\to$ RL	$<\!10^{-4}$	$<\!10^{-4}$	$<\!10^{-4}$	$<\!10^{-4}$	$<\!10^{-4}$
Scene CI	[ $.038$ , $.046$ ]	[ $.015$ , $.024$ ]	[ $-.012$ , $-.009$ ]	[ $-.063$ , $-.052$ ]	[ $-.007$ , $-.004$ ]
Log CI	[ $.035$ , $.050$ ]	[ $.009$ , $.029$ ]	[ $-.017$ , $-.006$ ]	[ $-.073$ , $-.043$ ]	[ $-.008$ , $-.004$ ]
ReCog-8B IL $\to$ RL	$<\!10^{-4}$	$<\!10^{-4}$	$<\!10^{-4}$	$<\!10^{-4}$	$<\!10^{-4}$
Scene CI	[ $.031$ , $.038$ ]	[ $.013$ , $.021$ ]	[ $-.006$ , $-.004$ ]	[ $-.029$ , $-.020$ ]	[ $-.005$ , $-.003$ ]

Appendix

Table 18: EPDMS rescoring on navtest : paired statistical support. For each pair, rows give Holm-adjusted Wilcoxon

p

-values, scene-bootstrap 95% intervals, and driving-log-bootstrap 95% intervals for the mean changes reported in Table 1 . PDMS uses the v1 scorer; EPDMS, driving-direction compliance (DDC), lane keeping (LK), and history comfort (HC) use the v2 scorer on fixed outputs. Holm correction retains the original test families in Appendix A.4 , including TOAD’s augmentation-only arm, without recomputing tests for the displayed subset.

	Stage 2 component changes $\uparrow$
Pair	$\Delta$ NC	$\Delta$ DAC	$\Delta$ DDC	$\Delta$ EP	$\Delta$ TTC	$\Delta$ LK	$\Delta$ HC	$\Delta$ EC
ReCog-2B IL $\to$ RL	$-.001$	$+.005$	$-.011$	$+.061$	$-.001$	$+.006$	$-.000$	$-.378$
ReCog-8B IL $\to$ RL	$-.000$	$+.008$	$-.002$	$+.056$	$-.003$	$+.003$	$-.002$	$-.527$
Qwen-Drive SFT $\to$ RL	$+.005$	$+.029$	$+.008$	$+.006$	$+.005$	$+.005$	$+.008$	$-.296$
NoRD SFT $\to$ RL (vocab.)	$+.024$	$+.120$	$+.051$	$+.040$	$+.022$	$+.024$	$+.017$	$+.038$

Appendix

Table 19: NAVSIM-v2 navhard-two-stage : component analysis. Synthetic-stage paired changes of the EPDMS components. EC is the two-frame extended comfort.

Explore similar work

Do Open-Loop Metrics Predict Closed-Loop Driving? A Cross-Benchmark Correlation Study of NAVSIM and Bench2Drive
Apr 30, 2026Yiru Wang, Anqing Jiang, Shuo Wang +4Autonomous Driving Simulation Autonomous Driving
Bosch Corporate Research, Bosch (China) Investment Ltd., Shanghai, China
Driving on Memory
Aug 31, 2026Christian Löwens, Thorben Funke, Alexandru Paul ConduracheCausality-Aware End-To-End Autonomous Driving
Bosch Research · University of Lübeck · Automated Driving, Bosch
DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models
Jun 15, 2026Xinglong Sun, Kevin Xie, Jenny Schmalfuss +5Autonomous Driving Human Driving
NVIDIA · KE:SAI

Apr 30, 2026, cs.ROJ/K move · Enter open · S save

Do Open-Loop Metrics Predict Closed-Loop Driving? A Cross-Benchmark Correlation Study of NAVSIM and Bench2Drive

Yiru Wang, Anqing Jiang, Shuo Wang, Yuwen Heng+3

Bosch Corporate Research, Bosch (China) Investment Ltd., Shanghai, China

Autonomous Driving Simulation Autonomous Driving Autonomous Hard Drive Disassembly Closed Loop Performance Metric

Open paper AskPDF arXiv

Open-loop evaluation offers fast, reproducible assessment of autonomous driving planners, but its ability to predict real closed-loop driving performance remains questionable. Prior work has shown that traditional open-loop metrics such as Average Displacement Error (ADE) and Final Displacement Error (FDE) exhibit no reliable correlation with closed-loop Driving Score. In this paper, we ask whether the more recent, safety-aware open-loop metrics introduced by NAVSIM~v2 can bridge this gap. By systematically cross-referencing published results from 15 state-of-the-art methods across NAVSIM (open-loop) and Bench2Drive (closed-loop), we compile a paired dataset of open-loop sub-metrics and closed-loop performance, yielding 8 methods with complete paired data. Our analysis reveals three key findings: (1) the aggregate NAVSIM PDM Score shows a strong positive but non-monotonic correlation with Bench2Drive Driving Score, with clear ranking inversions; (2) among individual NAVSIM sub-metrics, Ego Progress (EP) is the strongest single predictor of closed-loop success, substantially exceeding the safety-critical collision metric NC; (3) the safety-progress trade-off manifests differently in open-loop and closed-loop: methods that maximize safety at the expense of progress rank highly in NAVSIM but underperform in closed-loop due to timeout and slow-driving penalties. We further demonstrate that a much simpler 3-metric formula matches the predictive power of the full 5-metric PDMS at the same Spearman

ρ{=}0.90

on our paired sample of

n{=}8

methods, suggesting that within current state-of-the-art methods -- where TTC and Comfort approach saturation -- these two sub-metrics add little marginal information for closed-loop ranking. Additionally, we identify the snowball effect -- where small open-loop deviations compound into closed-loop failures -- as a candidate mechanism for the residual gap.

Abstract

Explore similar work

Do Open-Loop Metrics Predict Closed-Loop Driving? A Cross-Benchmark Correlation Study of NAVSIM and Bench2Drive

Driving on Memory

DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models