Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol

Emad Aghajanzadeh; Thomas Schwarzfischer; Moritz Thoma; Nael Fasfous; Manoj Rohit Vemparala; Pierpaolo Mori; Daniel Mueller-Gritschneder; Ulf Schlichtmann

cs.ROOct 15, 2026

Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol

Authors: Emad Aghajanzadeh, Thomas Schwarzfischer, Moritz Thoma, Nael Fasfous, Manoj Rohit Vemparala, Pierpaolo Mori, Daniel Mueller-Gritschneder, Ulf Schlichtmann

Organizations: BMW Group, Munich, Germany · Technical University of Munich (TUM), Munich, Germany · Vienna University of Technology (TU Wien), Vienna, Austria

Abstract

Closed-loop evaluation is widely regarded as a more faithful way to assess end-to-end (E2E) driving models than open-loop evaluation, and Bench2Drive, built on the CARLA simulator, has become the dominant benchmark for it. We show that its evaluation protocol itself contains pitfalls that can misrepresent the models scored on it. Bench2Drive's comfort metric, adapted from nuPlan, has several implementation errors and a segmentation scheme that rewards standing still, so it mostly measures how long a vehicle stays stationary, correlating with stationary time at r = 0.97 across eight models. We propose a corrected, distance-based metric that cuts that correlation to r = 0.06. Under it, the strong driving-score/comfort tradeoff the original metric reports is no longer detectable once models are compared at matched driving speed, which suggests that comfort trades against how fast a model drives rather than against how well it drives. We also uncover two pitfalls in how the harness supplies navigation signals: the default controller can replace a model's own steering with an angle toward the route target point, and the target point supplied at evaluation time can differ substantially from what a model saw in training. We demonstrate both on UniAD and VAD, where their effects differ sharply, and argue that future closed-loop evaluations should check for them. We will release all fixes as standalone code, so corrected comfort scores can be obtained from logged trajectories without re-running closed-loop simulation.

Figures & tables

Model	SO	TPM	DS	SR (%)
Const. velocity	$\times$	–	48.57	20.0
UniAD	$\times$	$\times$	53.42	30.0
	$\checkmark$	$\times$	19.76	0.0
	$\checkmark$	$\checkmark$	26.06	10.0
VAD	$\times$	$\times$	52.37	30.0
	$\checkmark$	$\times$	52.49	30.0

Table I: Cumulative impact of removing steering overruling (SO) and aligning target-point sampling (TPM) on driving score (DS) and success rate (SR), on Dev10, against a constant-velocity baseline.

\checkmark

marks a fix applied. TPM is applied on top of SO, not independently.

Fig. 1: Illustration of two closed-loop evaluation pitfalls. In both panels blue marks the model’s own output: its predicted waypoints and the steering vector they imply. (a) Steering overruling: the red vector is the steering the controller substitutes, aimed at the navigation target point ( red pin). This can mask a model’s failure to compensate its own steering error (left) and can override a maneuver around an obstacle (right). (b) Target-point mismatch: because the evaluation route planner samples the route differently than the planner that collected the training data, the navigation command switches at a different place along the route at test time ( faded red pins) than during training ( red pins). Here the ego receives turn right early and begins to rotate, then receives straight at the junction: the command calls for the straight continuation ( purple ) while its heading and the road geometry support completing the right turn ( green ), a command-context combination absent from training. Interpolating between the two can produce dangerous behavior. SO: steering overruling; TPM: target point mismatch.

Panel of Figure 3 — Fig. 2: Target-point distance supplied to UniAD in training and at evaluation time, on a logarithmic axis, with statistics ( $\mathrm{m}$ ) for both models below. The shaded band is the training p5 to p95 range, $4.06\text{\,}\mathrm{m}6.60\text{\,}\mathrm{m}$ , shared by both models. The original planner places UniAD’s target at a median of $64.6\text{\,}\mathrm{m}$ , over an order of magnitude beyond the $4.94\text{\,}\mathrm{m}$ training median, and behind the ego in $13.7\text{\,}\mathrm{\%}$ of frames. Ours restores the median for both models, to $6.28\text{\,}\mathrm{m}$ and $6.09\text{\,}\mathrm{m}$ . For UniAD it also removes the tail (p95 $130.2\text{\,}\mathrm{m}$ to $20.0\text{\,}\mathrm{m}$ ) and the behind-ego frames ( $13.7\text{\,}\mathrm{\%}$ to $0.9\text{\,}\mathrm{\%}$ ); for VAD it does not, and we report that residual in section III-B .

Method	Traditional Comfort	+ fixed implementation	+ distance-based segmentation (ours)	Time spent stationary (%)	Efficiency	Success Rate (%)	Driving Score
UniAD [ 9 ]	39.28	79.06	72.93	45.67	135.20	10.96	37.58
VAD [ 11 ]	45.95	56.84	37.21	47.64	152.69	17.89	46.18
SparseDriveV2 [ 14 ]	7.75	23.50	50.28	7.08	211.98	70.00	87.17
TFv6 [ 6 ]	19.58	25.97	29.20	19.83	207.17	85.91	94.47
Orion [ 15 ]	14.76	70.24	75.98	23.07	151.69	51.14	75.33
SimLingo [ 3 ]	31.71	36.80	25.86	32.16	225.91	71.23	87.68

Table II: Closed-loop results on 220 Bench2Drive routes: the traditional comfort metric, a fixed implementation of it, and our distance-based metric. UniAD and VAD use the corrected protocol (SO removed, TPM fixed; sections III-A and III-B ).

Panel of Figure 5 — Fig. 3: Sample route of SimLingo [ 3 ] , showing driving speed over time together with the cumulative comfort scores of the different implementations. We re-project our distance-based implementation on the time axis. The traditional metric considers driving comfortable when and only when the ego vehicle is standing still, marked by the red shaded bands. Even the slight movement from seconds 35 to 38 is deemed uncomfortable. When we apply our fixes, the metric correctly scores segments when the vehicle is driving comfortably, but still incentivizes standing still. Further applying our distance-based segmentation fixes both problems, not changing the comfort score when the ego vehicle is stationary.

Panel of Figure 6 — Fig. 4: Traditional comfort (vermillion circles) and our distance-based comfort (blue triangles), one point per model ( $n=8$ ), least-squares fit per series. Left: comfort against efficiency; ours recovers the negative relationship the traditional metric misses. Right: the same against driving score with speed removed from both axes, each value being a model’s score minus what its efficiency alone predicts, so zero means exactly as expected for that speed. At matched speed the traditional metric stays strongly negative, as expected of a score that tracks stationary time, while ours leaves no relationship; ours is dashed because at $n=8$ a tilt that size appears by chance about one time in three ( $p=0.32$ ).

Autonomous Driving Benchmarks

Autonomous driving benchmarks evaluate how well vehicles perceive, predict, plan, and act safely across varied roads and conditions.

See all 140 papers

Do Open-Loop Metrics Predict Closed-Loop Driving? A Cross-Benchmark Correlation Study of NAVSIM and Bench2Drive
Apr 30, 2026Yiru Wang, Anqing Jiang, Shuo Wang +4Autonomous Driving Benchmarks Autonomous Driving Safety Evaluation
Bosch Corporate Research, Bosch (China) Investment Ltd., Shanghai, China
Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models
Apr 1, 2026Xiaosong Jia, Yuqian Shao, Zhenjie Yang +3Autonomous Driving Benchmarks VLMs for Autonomous Driving
Institute of Trustworthy Embodied AI, Fudan University and Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, 200441, China. · School of Computer Science & School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, 200240, China.
Bench2Drive-Robust: Benchmarking Closed-Loop Autonomous Driving under Deployment Perturbations
May 18, 2026Zhiyuan Zhang, Zhenghao Jin, Yanlun Peng +8Autonomous Driving Benchmarks End-to-End Autonomous Driving
Sch. of Computer Science & Sch. of Artificial Intelligence, Shanghai Jiao Tong University · Institute of Trustworthy Embodied AI (TEAI), Fudan University · Great Wall Motor +2

Apr 30, 2026J/K move · Enter open · S save

Do Open-Loop Metrics Predict Closed-Loop Driving? A Cross-Benchmark Correlation Study of NAVSIM and Bench2Drive

Yiru Wang, Anqing Jiang, Shuo Wang, Yuwen Heng+3

Bosch Corporate Research, Bosch (China) Investment Ltd., Shanghai, China

Autonomous Driving Benchmarks Autonomous Driving Safety Evaluation Autonomous Driving Autonomous Driving Planning

Open paper AskPDF arXiv

Open-loop evaluation offers fast, reproducible assessment of autonomous driving planners, but its ability to predict real closed-loop driving performance remains questionable. Prior work has shown that traditional open-loop metrics such as Average Displacement Error (ADE) and Final Displacement Error (FDE) exhibit no reliable correlation with closed-loop Driving Score. In this paper, we ask whether the more recent, safety-aware open-loop metrics introduced by NAVSIM~v2 can bridge this gap. By systematically cross-referencing published results from 15 state-of-the-art methods across NAVSIM (open-loop) and Bench2Drive (closed-loop), we compile a paired dataset of open-loop sub-metrics and closed-loop performance, yielding 8 methods with complete paired data. Our analysis reveals three key findings: (1) the aggregate NAVSIM PDM Score shows a strong positive but non-monotonic correlation with Bench2Drive Driving Score, with clear ranking inversions; (2) among individual NAVSIM sub-metrics, Ego Progress (EP) is the strongest single predictor of closed-loop success, substantially exceeding the safety-critical collision metric NC; (3) the safety-progress trade-off manifests differently in open-loop and closed-loop: methods that maximize safety at the expense of progress rank highly in NAVSIM but underperform in closed-loop due to timeout and slow-driving penalties. We further demonstrate that a much simpler 3-metric formula matches the predictive power of the full 5-metric PDMS at the same Spearman

ρ{=}0.90

on our paired sample of

n{=}8

methods, suggesting that within current state-of-the-art methods -- where TTC and Comfort approach saturation -- these two sub-metrics add little marginal information for closed-loop ranking. Additionally, we identify the snowball effect -- where small open-loop deviations compound into closed-loop failures -- as a candidate mechanism for the residual gap.

Abstract

Autonomous Driving Benchmarks

Explore similar work

Do Open-Loop Metrics Predict Closed-Loop Driving? A Cross-Benchmark Correlation Study of NAVSIM and Bench2Drive

Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models

Bench2Drive-Robust: Benchmarking Closed-Loop Autonomous Driving under Deployment Perturbations