What Matters for Scalable and Robust Learning in End-to-End Driving Planners?

David Holtz; Niklas Hanselmann; Simon Doll; Marius Cordts; Bernt Schiele

cs.ROMar 16, 2026

What Matters for Scalable and Robust Learning in End-to-End Driving Planners?

Authors: David Holtz, Niklas Hanselmann, Simon Doll, Marius Cordts, Bernt Schiele

Organizations: Mercedes-Benz AG · Max-Planck-Institute for Informatics, SIC

Abstract

End-to-end autonomous driving has gained significant attention for its potential to learn robust behavior in interactive scenarios and scale with data. Popular architectures often build on separate modules for perception and planning connected through latent representations, such as bird's eye view feature grids, to maintain end-to-end differentiability. This paradigm emerged mostly on open-loop datasets, with evaluation focusing not only on driving performance, but also intermediate perception tasks. Unfortunately, architectural advances that excel in open-loop often fail to translate to scalable learning of robust closed-loop driving. In this paper, we systematically re-examine the impact of common architectural patterns on closed-loop performance: (1) high-resolution perceptual representations, (2) disentangled trajectory representations, and (3) generative planning. Crucially, our analysis evaluates the combined impact of these patterns, revealing both unexpected limitations as well as underexplored synergies. Building on these insights, we introduce BevAD, a novel lightweight and highly scalable end-to-end driving architecture. BevAD achieves 72.7% success rate on the Bench2Drive benchmark and demonstrates strong data-scaling behavior using pure imitation learning. Our code and models are publicly available here: https://dmholtz.github.io/bevad/

Figures & tables

Panel of Figure 1 — Figure 1: Architectural Patterns. (a) High-resolution BEV features facilitate perception tasks, but promote overfitting the planner. (b) Closed-loop methods prefer path over trajectory representations due to robust steering. (c) Point-estimates interpolate between trajectory modes that diffusion-based sampling can breed.

Panel of Figure 2 — Figure 2: Analysis Framework. (a) We build our analysis framework on ParaDrive [ 52 ] . (b) We introduce a scene tokenizer to reduce the spatial resolution of the BEV features. The design of our planning head is based on a diffusion transformer [ 41 ] . Crucially, the choice of the planning queries determines whether the planner is modeled as a point estimator or by diffusion.

Panel of Figure 3 — Figure 3: Qualitative visualization of the planning queries’ cross-attention to BEV features. Fig. 3(a) . The planner attends to distant BEV cells. Despite strong attention on the traffic light, the autonomous vehicle runs the red light. Fig. 3(b) : There are numerous attention spikes to random BEV cells, but barely no attention to the oncoming traffic. Fig. 3(c) : The attention map significantly simplifies and exhibits fewer attention outliers.

Mask	Patch Size	Scene Tokens	DS $\uparrow$	SR $\uparrow$	L1 (m) $\downarrow$
✗	1	$100\times 100$	66.86	36.36	1.45
✓ †	1	$100\times 60$	71.79	41.37	1.45
✓	1	$100\times 60$	72.40	41.97	1.53
✓	2	$50\times 30$	74.98	48.18	1.43
✓	4	$25\times 15$	82.62	57.43	1.43
✓	5	$20\times 12$	66.44	40.91	1.73

Table 1: Impact of Tokenizer Design. Masking : Restricting planning queries’ attention benefits closed-loop driving. Patchifying : Aggregating

p\times p

BEV features into scene tokens significantly enhances closed-loop driving for

p\leq 4

. Notably, these closed-loop improvements are not reflected in the open-loop L1 trajectory error metric. Legend:

\dagger

indicates test-time masking.

Model	Repr.	DS $\uparrow$	SR $\uparrow$	$\text{IR}_{s}$ $\downarrow$	$\text{IR}_{d}$ $\downarrow$
PE	T	77.2 $\pm 0.6$	51.7 $\pm 0.7$	0.185	0.505
PE	P+S	82.6 $\pm 1.0$	57.4 $\pm 0.6$	0.055	0.447
DI	T	80.7 $\pm 3.0$	56.2 $\pm 4.7$	0.147	0.391
DI	P+S	81.8 $\pm 1.5$	59.4 $\pm 1.0$	0.094	0.423

Table 2: Comparison of modeling and planning representation. Legend: PE: point-estimator, DI: diffusion, T: trajectory (entangled), P+S: path and speed (disentangled) representation.

Figure 4: Scaling Properties. Diffusion demonstrates superior performance over point estimators when scaled with sufficient training data, despite initially underperforming with limited data.

Method	Details			Overall
Method	Expert	Sensors	Labels	Driving Score $\uparrow$	Success Rate $\uparrow$
VAD [ 29 ]	Think2Drive	6x CAM	O, M	42.35	15.00
UniAD [ 24 ]	Think2Drive	6x CAM	O, M	45.81	16.36
ThinkTwice [ 27 ]	Think2Drive	6x CAM, LiDAR	O	62.44	31.23
DriveAdapter [ 26 ]	Think2Drive	6x CAM, LiDAR	O	64.22	33.08
Hydra-NeXt [ 34 ]	Think2Drive	2x CAM	-	73.86	50.00

Table 3: Closed-loop Results on Bench2Drive. Despite its simpler design BevAD outperforms previous modular baselines UniAD and VAD by a large margin, reaching SOTA-level performance. We highlight that BevAD can gain further substantial driving performance by uniformly scaling up training data. Legend: O: 3D Object Detection, M: Map, S: Semantic Segmentation, D: Depth, L: Language,

\dagger

: concurrent work. If available, we report mean and standard deviation over three seeds to account for the randomness in CARLA.

Panel of Figure 5 — Figure 5: Yield to Emergency Vehicle. By increasing the training dataset size, BevAD-M learns to yield to emergency vehicles (red) on highways by safely merging into slower traffic. This cability is absent at smaller data scales (BevAD-S) and in prior leading closed-loop methods [ 43 , 48 ] .

Method	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
UniAD [ 24 ]	97.8	91.9	92.9	78.8	83.4
ParaDrive [ 52 ]	97.9	92.4	93.0	79.3	84.0
BevAD (ours)	98.1	95.3	94.5	80.5	86.6
BevAD w/o mask	97.8	94.9	93.7	80.3	85.9 (-0.7)
BevAD w/o patchifying	97.9	94.4	94.4	79.7	85.6 (-1.0)

Table 4: Real-world Results on NAVSIM. Performance comparison of BevAD against baseline methods on the real-world NAVSIM benchmark ( navtest ), including key ablations. Legend: NC: no at-fault collision, DAC: drivable area compliance, TTC: time-to-collsion, EP: ego progress, PDMS: PDM score.

Figure 6: Label Flaws. Bench2Drive exhibits incomplete and erroneous 3D bounding box annotations in various scenes.

Panel of Figure 7 — Figure 7: Visualization of Trigger Point Selection Criteria. (a) This scenario requires a signalized junction with a left-turn exit relative to the ego lane. (b) The VehicleOpensDoorTwoWays scenario requires a right-side parking lane for the adversary and an adjacent left-side lane for oncoming traffic. (c) The Accident scenario demands a right-side shoulder lane for the group of blocking vehicles and an adjacent left-side lane with the same traffic flow.

Method	BEV	FPS (train) $\uparrow$
UniAD-tiny [ 24 , 28 ]	frozen (stage-2)	2.5
BevAD (ours)	frozen	89.0
BevAD (ours)	end-to-end	55.0

Table 5: Training Efficiency. BevAD achieves substantial training speed-up relative to UniAD-tiny. Measured on 8xA100-80GB.

Figure 8: adaLN-Zero Attention. We employ an adaLN-Zero transformation to condition the transformer on

\mathbf{c}

. For self-attention, we set

\mathcal{K}=\mathcal{V}=\mathcal{Q}_{\text{Plan}}

and for cross-attention, we set

\mathcal{K}=\mathcal{V}=\mathcal{F}_{\text{Scene}}

Augmentation	DS $\uparrow$	SR $\uparrow$	$\text{IR}_{s}$ $\downarrow$	L1 (m) $\downarrow$
✓	$\mathbf{82.62}$	$\mathbf{57.43}$	$\mathbf{0.055}$	$\mathbf{1.43}$
✗	$66.60$	$33.64$	$0.314$	$1.47$

Table 6: Ablation of camera augmentation. The absence of camera augmentation significantly degrades closed-loop performance, despite minimal impact on open-loop L1 trajectory deviation. This underscores the contribution of our BEV-based augmentation to robust driving and covariate shift mitigation.

BEV	Scene Tokens	DS $\uparrow$	SR $\uparrow$	$\text{IR}_{s}$ $\downarrow$
$100\times 100$	$25\times 15$	$\mathbf{82.62}$	$\mathbf{57.43}$	$\mathbf{0.055}$
$25\times 25$	$25\times 15$	$72.18$	$40.36$	$0.409$

Table 7: BEV Resolution and Tokenization. High-resolution BEV generation, compressed via our tokenizer, yields superior closed-loop driving performance compared to direct low-resolution BEV generation.

$S$	DS $\uparrow$	SR $\uparrow$	FPS $\uparrow$
10	88.11	72.73	4.2
5	88.33	72.72	5.8
2	88.53	72.72	7.5

Table 8: Impact of Denoising Iterations. Reducing denoising steps significantly boosts inference FPS while preserving closed-loop driving performance, enabling real-time applications. FPS measured on a Quadro RTX 8000.

Method	Ability (%) $\uparrow$
Method	Merging	Overtaking	Emergency Brake	Give Way	Traffic Sign	Mean
VAD [ 29 ]	8.11	24.44	18.64	20.00	19.15	18.07
UniAD-Base [ 24 ]	14.10	17.78	21.67	10.00	14.21	15.55
ThinkTwice [ 27 ]	27.38	18.42	35.82	50.00	54.23	37.17
DriveAdapter [ 26 ]	28.82	26.38	48.76	50.00	56.43	42.08
Hydra-NeXt [ 34 ]	40.00	64.44	61.67	50.00	50.00	53.22

Table 9: Multi-Ability Evaluation. BevAD consistenly achieves high results across all skills, dominating the mean score score. Notably, it significantly surpasses prior work in the Merging and Give Way skills. However, it exhibits comparatively lower performance in Overtaking , Emergency Brake , and Traffic Sign skills. Legend:

\dagger

: concurrent work.

Figure 9: Emerging Skills. All skills remain underdeveloped with fewer than 4,000 training episodes. As the volume of training data increases, the skills progressively and uniformly emerge.

Figure 10: Merging. BevAD merges from a parallel parking space into traffic. It yields to rear-end flow of vehicles, identifies a safe gap, and accelerates for seamless merging.

Figure 11: Overtaking (1). BevAD executes an overtaking maneuver when the route is blocked by a construction vehicle. It waits for clear oncoming traffic, then steers into the opposing lane, accelerates to quickly pass the obstacle, and subsequently decelerates when returning to its original lane.

Figure 12: Overtaking (2). BevAD approaches a group of cyclists. It executes a safe left lane change to overtake them. After the maneuver, BevAD returns to its original lane, maintaining a safe distance to the cyclists.

Figure 13: Emergency Brake. BevAD brakes at a green-light intersection due to a pedestrian crossing its left-turn path. Driving resumes upon pedestrian clearance.

Figure 14: Give Way. BevAD yields to an oncoming vehicle encroaching on the ego lane. It performs a controlled rightward deviation from the lane center, staying within road limits, and re-centers once the oncoming traffic has passed.

Figure 15: Traffic Sign. BevAD stops at a stop sign at an intersection with cross-traffic. It waits for a safe gap, then executes a right turn and merges into the traffic flow.

End-to-End Autonomous Driving

End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.

See all 227 papers

Planning-Oriented End-to-End Autonomous Driving: Architectures, Evaluation, and Emerging Paradigms
Aug 20, 2026Yanchen Guan, Xingcheng Liu, Bin Rao +7End-to-End Autonomous Driving Autonomous Driving Planning
State Key Laboratory of Internet of Things for Smart City, University of Macau, Macau SAR, 999078, China · College of Mechanical and Vehicle Engineering, Chongqing University, Chongqing, 400044, China · Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong, 999077, China +2
Closed-Loop Aware Planning via Imitation-Free Speed Scoring for End-to-End Autonomous Driving
Oct 26, 2026Emad Aghajanzadeh, Thomas Schwarzfischer, Marc Kaufeld +8Autonomous Driving Planning End-to-End Autonomous Driving
Technical University of Munich, Germany · BMW Group, Germany · TU Wien, Austria
DriveZero: End-to-End Driving Beyond Human Demonstrations
Sep 5, 2026Hao He, Chengcheng Hu, Zirun Su +17End-to-End Autonomous Driving RL for Autonomous Driving
Xiaomi EV AD & Robotics, L3 Team

Aug 20, 2026J/K move · Enter open · S save

Planning-Oriented End-to-End Autonomous Driving: Architectures, Evaluation, and Emerging Paradigms

Yanchen Guan, Xingcheng Liu, Bin Rao, Chengyue Wang+6

State Key Laboratory of Internet of Things for Smart City, University of Macau, Macau SAR, 999078, China · College of Mechanical and Vehicle Engineering, Chongqing University, Chongqing, 400044, China · Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong, 999077, China+2

End-to-End Autonomous Driving Autonomous Driving Planning Autonomous Driving Safety Evaluation Autonomous Driving Benchmarks

Open paper AskPDF arXiv

End-to-end autonomous driving has evolved from camera-to-control regression toward planning-oriented systems that use structured representations, trajectory-level outputs, and increasingly realistic evaluation protocols. This survey reviews this transition across behavior cloning, conditional imitation learning, privileged distillation, BEV and vectorized planning, unified perception-prediction-planning architectures, world-model-based planners, and vision-language-action systems. We argue that the key distinction in modern end-to-end driving is not whether intermediate representations are used, but whether they are learned, supervised, and evaluated to support safe, feasible, and route-compliant planning. To organize the literature, we synthesize existing methods along four axes: input representation, planning output, supervision signal, and evaluation protocol. We further examine the benchmark shift from open-loop trajectory matching to closed-loop simulation, non-reactive real-log evaluation, long-tail testing, and human-preference-aware metrics. Our analysis highlights that architectural progress is difficult to interpret without benchmark-consistent evaluation, and that displacement-based open-loop metrics alone provide limited evidence for safe and human-aligned driving. We conclude with open challenges in uncertainty-aware planning, learner-expert mismatch, runtime safety assurance, language-action grounding, world-model validation, and reproducible benchmarking.

Abstract

End-to-End Autonomous Driving

Explore similar work

Planning-Oriented End-to-End Autonomous Driving: Architectures, Evaluation, and Emerging Paradigms

Closed-Loop Aware Planning via Imitation-Free Speed Scoring for End-to-End Autonomous Driving

DriveZero: End-to-End Driving Beyond Human Demonstrations