cs.RONov 4, 2026

Beyond Infraction Scores: Failure-Boundary Profiles for Closed-Loop Evaluation of Driving and Robot Navigation Agents

Authors: Milan Bulat, Gino Ardelean, Ellington Kirby

Organizations: Social Chaos Lab

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Closed-loop driving benchmarks in CARLA rank agents by a driving score that multiplies route completion by an infraction penalty. This score does not distinguish an agent that stops in front of an obstacle from one that pushes past it. In a corridor closed by two parked cars, for example, one agent that drives through with contact averages 52.2 on the blocked widths, while the agents that stop average between 37.8 and 40.6. We propose failure-boundary profiles, an evaluation in which each scenario family varies one measured factor, such as corridor width or the time gap in oncoming traffic, and each variant is certified from measurement as one where the agent should proceed or one where it should stop. Along this axis we estimate two curves per agent, commitment (the probability of attempting the passage) and competence (the probability of passing without contact), and locate where each crosses one half, with intervals that account for the simulator's run-to-run variance. The certified labels also give a summary metric, decision accuracy, which credits clean passes on passable variants and stops on blocked ones. We evaluate seven driving agents on seven CARLA families. The rule-based expert reaches a decision accuracy of 0.74 and TFv6, the highest-scoring learned agent, 0.60. Agents with similar driving scores fail in different ways, and agents that reliably stop at a blocked corridor still rarely wait for a gap in oncoming traffic. We also apply the method, unchanged, to a warehouse robot in Isaac Sim. The navigation policy we test stops at blocked gaps in only one of three families, and two fine-tuned versions of it pass more gaps cleanly but no longer stop at all.

Figures & tables

Figure 1: The two ends of the Charger gauntlet. Left: the widest variant, with two cars parked on the shoulders and a 3.64 m free corridor between them. Right: the narrowest variant, where the two cars close the corridor completely.
Figure 2: Driving score and commitment on the Charger gauntlet. TFv6 and BridgeDrive, three traffic seeds each, against the minimum free corridor width. Top: driving score of each run (dots) and the mean per width (lines); the circled pair is the same width and seed, the fully closed corridor at seed 2, where TFv6 stops (39.1) and BridgeDrive continues past the blockage with contact (54.5). Bottom: commitment per width; the diamond marks TFv6’s commitment boundary with its bootstrap interval. Solid line: body width, 1.84 m; dashed line: policy floor, 2.30 m.
 Mean DSBlocked widths
AgentAllPass.Block.nnStopContact
PlanT 2.0 ∗72.586.744.01236
HiP-AD71.587.739.212111
SimLingo67.875.552.212012
TF++65.678.140.612120
TFv664.376.939.112120
Table 1: Driving score split by certified passability. Mean driving score on the Charger gauntlet (three traffic seeds) over all runs, over the eight passable widths and over the four blocked widths (0.00 to 2.07 m), with the outcomes of the runs on the blocked widths. Stop: the agent stops before the blockage. Contact: any contact with the blockage, whether the agent continued past it or not. Rows are ordered by the mean over all runs. ∗ Checkpoint 1; its other 3 blocked runs complete the 2.07 m width without contact.
Figure 3: The ten scenario families. One picture per family, cut from the recordings of the campaigns the paper reports: (a) to (g) the front cameras of TFv6 in the CARLA families, (h) to (j) the top-down view of the warehouse families. (a) The Charger gauntlet at its closed end, the two parked cars meeting across the lane. (b) The vending machine at the lane edge, the shallowest intrusion of the sweep. (c) The narrowest corridor: two street barriers in a row on the right, opposite the vending machine on the left curb. (d) The identity ladder with the slide in place. (e) The cluttered intrusion, the machine and two bins at the lane edge. (f) The slowest oncoming flow and (g) a mid-range spacing, the car waiting at the junction to turn left across the oncoming lane. (h) The slab pincer at a gap certified blocked, (i) the floor stock and (j) the composed stacks at their narrowest passable gaps, the robot approaching the row.
Figure 4: The Social Chaos Lab scenario-generation pipeline. A source scenario and a natural-language intent enter the pipeline; an agent planner turns the intent into a declarative plan, a deterministic solver realizes and validates each variant, and the accepted variants form a scenario family whose placed objects are countersigned in the simulator before campaigns run; campaigns are then run, scored and certified.
AgentAvg. DSAvg. DA [95% CI]
Driving (CARLA)
Expert (reference)79.80.74 [0.67, 0.81]
TFv675.50.60 [0.53, 0.68]
PlanT 2.0 (ckpt 1)72.60.56 [0.50, 0.61]
PlanT 2.0 (ckpt 3)71.40.50 [0.50, 0.50]
TF++59.50.49 [0.41, 0.57]
Table 2: Driving score and decision accuracy per agent. Both averaged with equal weight over the certified families that contain variants where the correct call is to stop. Driving: the Charger gauntlet (four certified blocked widths) and the two oncoming-flow sweeps (wait windows labeled by the gap certificate; the defective shortest spacing variant is excluded). Warehouse: the slab pincer, floor stock and composed stacks, which have no driving score. Avg. DS: mean CARLA driving score. Avg. DA: decision accuracy, the balanced mean of the clean-pass rate on passable or usable variants and the stop rate on blocked or wait variants, with a 95% bootstrap interval over runs, of zero width where every run on each side has the same outcome. Rows are ordered by Avg. DA within each block; bold marks the best learned driving agent in each column.
Figure 5: Failure-boundary profiles on the Charger gauntlet. Twelve widths, three traffic seeds per agent. Each panel is one agent. Black: commitment, the share of runs in which the agent attempts the passage. Blue: competence, the share of runs that pass without contact. Orange: widths where the agent commits but does not pass cleanly. Grey: widths certified blocked, below the 2.30 m policy floor; the dashed line marks the 1.84 m body width. A diamond marks each located boundary at one half. Panel titles give each boundary with its 95% bootstrap interval or, where the outcomes are separated or the fit does not converge, its bracket ( b ): the pair of adjacent widths between which the observed rate crosses one half. A title reads “flat” where every run has the same outcome, “not located in range” where the fitted curve crosses one half outside the tested range, and “below tested range” where the observed rate stays above one half at every tested value. PlanT 2.0: checkpoint 1 is drawn; the panel title gives the boundaries of checkpoints 2 and 3 where located.
Figure 6: Failure-boundary profiles on the corridor-width family. Ten widths, three traffic seeds; every variant is certified passable. Panels and colors as in Figure 5 . The expert does not register these barriers and was not run.
Figure 7: Failure-boundary profiles on the machine intrusion. Twelve intrusion depths, three traffic seeds; every variant is certified passable, the deepest leaving 2.32 m of free corridor. Each panel is one agent. Black: commitment, the share of runs in which the agent attempts the passage. Blue: competence, the share of runs that pass without contact. Orange: depths where the agent commits but does not pass cleanly. Panel titles give each boundary with its interval.
ObjectExpertTFv6TF++BridgeD.SimLingoHiP-AD
Vending machine3/31/3 ⋅\cdot 2R3/31/3 ⋅\cdot 2L3/33/3
Plastic chair3/31/3 ⋅\cdot 2R3/33/33/33/3
12 other objects36/3636/3636/3636/3636/3636/36
Table 3: Identity ladder. Fourteen roadside objects at one certified intrusion (0.5 m, about 8 m free corridor), three traffic seeds each. Cells give clean passes out of runs; R: refusal, L: late commit; shading marks cells with fewer clean passes than runs. The twelve other objects are the dog house, street barrier, traffic warning, mailbox, playground slide, plastic table, traffic cone, trampoline, barbecue grill, iron plank, waste bin and cardboard box.
Figure 8: Failure-boundary profiles on the cluttered intrusion. Twelve intrusion depths, three traffic seeds; every variant is certified passable. Panels and colors as in Figure 5 ; intrusion grows to the right. Grey: the 0.99 m variant, a scenario defect (the vending machine tips over the moment it is placed, before any car arrives), excluded from every score.
Figure 9: Failure-boundary profiles on the oncoming-flow speed sweep. Eleven crossing time gaps, three traffic seeds; larger gaps are easier. Panels and colors as in Figure 5 . A go is entry into the oncoming lane while the flow is active.
Figure 10: Failure-boundary profiles on the oncoming-flow spacing sweep. Eleven crossing time gaps, three traffic seeds; larger gaps are easier. Panels and colors as in Figure 5 . A go is entry into the oncoming lane while the flow is active. Grey: the shortest spacing variant, at 0.45 s, a scenario defect excluded from every score.
Figure 11: Failure-boundary profiles in the warehouse. Three warehouse families in Isaac Sim, each narrowing a robot’s aisle to a single gap from 2.5 m to a closed aisle, with the neighboring aisle open as a detour; twelve certified gaps on the slab pincer and the floor stock, ten on the composed stacks, five policy samples each. Rows: families. Columns: the released X-Mobility checkpoint and its two fine-tunes, both trained on demonstrations from the slab pincer and the floor stock; the composed stacks are held out. The released checkpoint and the fine-tunes were measured on the same gaps of each family. Black: commitment, the share of samples that attempt the gap. Blue: competence, the share that pass through it without contact. Orange: gaps the robot attempts but does not pass cleanly. Grey: gaps certified blocked. A diamond marks each located boundary at one half. Panel titles give the go and clean-pass counts on passable gaps, the stop count on blocked gaps, and the competence boundary with its interval or bracket ( b ).
Figure 12: X-Mobility before and after fine-tuning, on the held-out composed stacks. Five policy samples per certified gap for the released checkpoint (left) and the whole-model fine-tune (right), on the same ten gaps. Top: commitment (black), the share of samples that decide to go, and competence (blue), the share that pass cleanly, against the certified minimum free gap; diamonds mark the located boundaries, orange the gaps where the robot commits but does not pass cleanly, and grey the gaps certified blocked. Panel titles give the go and clean-pass counts on passable gaps, the stop count on blocked gaps, and the competence boundary with its bracket. Bottom: the outcome of each sample at each gap: commit clean, a passage without contact; timeout clean, the robot neither passes nor touches anything before the time limit, its way of stopping; timeout contact, the time limit reached after a contact.
More in this topic

Autonomous Driving Safety Evaluation

Autonomous driving safety evaluation studies how to measure, test, and improve the safety of self-driving systems in routine and hazardous conditions.

See all 166 papers

Explore similar work

CardsList
Apr 9, 2026

Fail2Drive: Benchmarking Closed-Loop Driving Generalization

Generalization under distribution shift remains a central bottleneck for closed-loop autonomous driving. Although simulators like CARLA enable safe and scalable testing, existing benchmarks rarely measure true generalization: they typically reuse training scenarios at test time. Success can therefore reflect memorization rather than robust driving behavior. We introduce Fail2Drive, the first paired-route benchmark for closed-loop generalization in CARLA, with 200 routes and 17 new scenario classes spanning appearance, layout, behavioral, and robustness shifts. Each shifted route is matched with an in-distribution counterpart, isolating the effect of the shift and turning qualitative failures into quantitative diagnostics. Evaluating multiple state-of-the-art models reveals consistent degradation, with an average success-rate drop of 22.8%. Our analysis uncovers unexpected failure modes, such as ignoring objects clearly visible in the LiDAR and failing to learn the fundamental concepts of free and occupied space. To accelerate follow-up work, Fail2Drive includes an open-source toolbox for creating new scenarios and validating solvability via a privileged expert policy. Together, these components establish a reproducible foundation for benchmarking and improving closed-loop driving generalization. We open-source all code, data, and tools at https://github.com/autonomousvision/fail2drive .

Simon Gerstenecker, Andreas Geiger, Katrin Renz

University of Tübingen Tübingen AI Center

Autonomous Driving BenchmarksAutonomous DrivingAutonomous Driving SimulationAutonomous Driving Scenario Generation
AskPDF
Oct 15, 2026

Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol

Closed-loop evaluation is widely regarded as a more faithful way to assess end-to-end (E2E) driving models than open-loop evaluation, and Bench2Drive, built on the CARLA simulator, has become the dominant benchmark for it. We show that its evaluation protocol itself contains pitfalls that can misrepresent the models scored on it. Bench2Drive's comfort metric, adapted from nuPlan, has several implementation errors and a segmentation scheme that rewards standing still, so it mostly measures how long a vehicle stays stationary, correlating with stationary time at r = 0.97 across eight models. We propose a corrected, distance-based metric that cuts that correlation to r = 0.06. Under it, the strong driving-score/comfort tradeoff the original metric reports is no longer detectable once models are compared at matched driving speed, which suggests that comfort trades against how fast a model drives rather than against how well it drives. We also uncover two pitfalls in how the harness supplies navigation signals: the default controller can replace a model's own steering with an angle toward the route target point, and the target point supplied at evaluation time can differ substantially from what a model saw in training. We demonstrate both on UniAD and VAD, where their effects differ sharply, and argue that future closed-loop evaluations should check for them. We will release all fixes as standalone code, so corrected comfort scores can be obtained from logged trajectories without re-running closed-loop simulation.

Emad Aghajanzadeh, Thomas Schwarzfischer, Moritz Thoma +5

BMW Group, Munich, Germany · Technical University of Munich (TUM), Munich, Germany · Vienna University of Technology (TU Wien), Vienna, Austria

Autonomous Driving BenchmarksBenchmark DesignAutonomous DrivingAutonomous Driving Simulation
AskPDF
Sep 22, 2026

NavSafe-∞\infty: Benchmarking Closed-Loop Driving Safety in Photorealistic Environments

End-to-end (E2E) driving policies have advanced rapidly on open-loop (OL) benchmarks, yet OL evaluation cannot reveal whether a policy can withstand compounding errors, recover from failures, or interact safely with surrounding actors. We introduce NavSafe-∞\infty, a photorealistic closed-loop (CL) benchmark comprising 280 scenarios spanning 28 event types, each with success and failure criteria defined within a structured traffic-safety taxonomy, yielding category-level capability scores for Traffic Crashes, Vulnerable Road User Crashes, Traffic Violations, and Traffic Incidents. After evaluating 20 E2E policies, we find that OL gains do not reliably transfer to CL safety. Analysis of two common remedies reveals that (1) passive demonstration perturbation helps mainly when CL rollouts stay near their perturbed training states, and (2) OL reinforcement-learning fine-tuning exhibits reward hacking by trading safety margin for ego progress, which CL feedback amplifies into compounding safety-critical errors. Together, these results demonstrate the blind spot of OL benchmarks indicating CL safety success. The benchmark and an extensible toolbox for customizable event curation and policy diagnosis will be open-sourced to facilitate future research.

Yuxin Bao, Hongwei Ruan, Luobin Wang +7

UCLA · UCSD · Toyota Research Institute

Autonomous Driving BenchmarksEnd-to-End Autonomous DrivingAutonomous Driving Safety EvaluationAutonomous DrivingAutonomous Driving Simulation
AskPDF