End-to-end driving models perform well on standard benchmarks but collide frequently in safety-critical scenarios. We present a failure-mode analysis on NeuroNCAP which shows that 95.9% of DiffusionDrive's collisions are planning failures despite reliable detection, pinpointing the bottleneck in planning rather than perception. We attribute this to a fundamental data imbalance, as safety-critical interactions are rare in real driving logs, so planners are hardly ever trained on them. We close this gap with a novel model-aware framework that generates safety-critical training scenes against a specific planner. Each planner is attacked at the bird's-eye-view (BEV) level through a differentiable surrogate distilled from its own planning head. This keeps the expensive video renderer out of the attack loop. We extract 50 scenes per planner, each pairing an attack with a safe escape trajectory that serves as the fine-tuning label. A fully automated quality-diversity search generates both the attacks and the escapes, so that their variety is not specified by hand. Only these scenes are rendered to fine-tune the planner, which reduces the computational cost significantly. Fine-tuning on these scenes roughly halves the closed-loop collision rate of two planners on NeuroNCAP (SparseDrive 28.0% to 12.2%, DiffusionDrive 37.5% to 19.2%) while largely preserving normal driving. A cross-planner transfer experiment shows that the gain comes from targeting the specific planner. Each planner improves most on the scenes generated against itself, and DiffusionDrive fine-tuned on SparseDrive's scenes retains only 6.7 of its 18.3-point collision-rate reduction.
Figures & tables
Fig. 1: Model-aware adversarial fine-tuning. We distill an end-to-end planner into a differentiable surrogate, optimize adversarial scenarios against it, and render them to fine-tune that planner. The renderer never enters the optimization loop. Bottom: closed-loop collision rate on NeuroNCAP. Each planner improves most on the scenes generated against itself.
Fig. 2: Method. (A) Only the adapter Aθ trains; the planner’s own planning head fhead is copied and frozen, so gradients through π^ point at the planner’s failure modes. (B) Adversary controls (a,δ) are optimized against the collision objective J through the surrogate, and a quality-diversity grid keeps the attack closest to the ego per cell, so attack types are found rather than specified. (C) The escape is solved by geometry with the attacker held fixed, and scenes with no valid escape are rejected. (D) Surviving scenes are rendered and only parameters inside the planning head of π are updated.
Scenario
Det.+Cr.
Det.+No Cr.
Miss+Cr.
Miss+No Cr.
Recall
Stationary
6.8%
82.9%
0.0%
10.2%
0.822
Frontal
52.5%
41.7%
4.5%
1.2%
0.861
Side
50.0%
50.0%
0.0%
0.0%
0.915
Overall
27.8%
65.3%
1.2%
5.7%
0.866
TABLE I: Failure decomposition on NeuroNCAP (stock DiffusionDrive). An agent counts as detected at recall@5–35 m ≥0.5 on the model’s own detections. Rates weight every episode equally, unlike NeuroNCAP’s family-weighted collision rate.
NeuroNCAP (closed-loop)
Challenger
nuScenes
Planner
Training data
score ↑
CR (%) ↓
stat. ↑
front. ↑
side ↑
CR (%) ↓
L2 ↓
CR (%) ↓
L2 ↓
SparseDrive
none (stock)
4.009
28.0
4.935
3.692
3.399
1.00
1.66
0.094
0.609
recovery
4.144 ± 0.07
23.8 ± 2.5
4.999
3.815
3.619
0.94 ± 0.05
1.66 ± 0.06
0.114 ± 0.006
0.636 ± 0.013
rule-based
4.373 ± 0.13
17.9 ± 1.0
4.843
4.134
4.141
0.91 ± 0.05
1.60 ± 0.05
0.116 ± 0.014
0.652 ± 0.003
Challenger-style
4.276 ± 0.04
19.1 ± 0.9
4.839
4.282
3.708
0.87 ± 0.08
1.55 ± 0.03
0.135 ± 0.029
0.689 ± 0.070
ours
4.639 ± 0.04
12.2 ± 0.8
5.000
4.010
4.907
0.86 ± 0.03
1.68 ± 0.01
0.088 ± 0.020
0.668 ± 0.070
TABLE II: Main results. Closed-loop NeuroNCAP decides; stat. / front. / side are its per-scenario scores. Challenger is open-loop adversarial replay, and clean nuScenes checks that normal driving is preserved. ours denotes our model-aware scenes. Stock rows evaluate the released checkpoints without fine-tuning; all other rows are means over 3 seeds.
Fig. 3: Generated training scenes, one per planner. Six cameras at the moment the ego must react, attacker boxed in red, with the BEV trail below, where footprints darken with time and the escape clears the attacker by the marked distance. Left: DiffusionDrive-targeted in-path attack. Right: SparseDrive-targeted front-brake at night, where the ego swerves around the near-stationary attacker.
NeuroNCAP (closed-loop)
Challenger
Expert label
score ↑
CR (%) ↓
CR (%) ↓
L2 ↓
none (stock)
3.624
37.5
1.67
2.37
CEM-Diffusion
3.759
34.3
0.78
1.38
brake-always
3.833
33.1
1.32
1.95
solved (ours)
3.939
30.8
1.00
1.75
TABLE III: Choice of expert label. Scenes, budget and recipe are identical; only the label changes. CEM-Diffusion refines samples from a nuPlan-trained trajectory diffusion prior [ 2 , 28 ] , and brake-always applies maximum deceleration. DiffusionDrive, mean over 3 seeds.
SparseDrive
DiffusionDrive
π vs. GT, L2 (the bar)
0.609
0.575
π^ vs. π , L2
0.620
0.643
ratio to the bar
1.02 ×
1.12 ×
collision incidence , π
1.60%
1.33%
collision incidence , π^
1.52%
1.07%
precision
0.603
0.673
TABLE IV: Surrogate fidelity on the 5,119 nuScenes val samples with annotated futures. Incidence is the fraction of samples with a collision within 3 s. It is stricter than the nuScenes collision rate of Table II , which averages over timesteps.
data ↓ / model →
SparseDrive
DiffusionDrive
SparseDrive-targeted
4.639 / 12.2%
3.837 / 30.8%
DiffusionDrive-targeted
4.460 / 15.7%
4.394 / 19.2%
premium (own − cross)
+0.18
+0.56
TABLE V: Transfer and targeting premium. Each planner fine-tuned on data generated against itself (diagonal) and against the other (off-diagonal). Cells give closed-loop NeuroNCAP score / collision rate, mean over 3 seeds.
Fig. 4: Attacks discovered against each planner differ. Those against DiffusionDrive are predominantly cut-ins and almost never head-on threats, while those against SparseDrive also include head-on and front-brake attacks.