Planning-Oriented End-to-End Autonomous Driving: Architectures, Evaluation, and Emerging Paradigms

Yanchen Guan; Xingcheng Liu; Bin Rao; Chengyue Wang; Guofa Li; Yunjian Li; Lishengsa Yue; Zhiyong Cui

cs.ROAug 20, 2026

Planning-Oriented End-to-End Autonomous Driving: Architectures, Evaluation, and Emerging Paradigms

Authors: Yanchen Guan, Xingcheng Liu, Bin Rao, Chengyue Wang, Guofa Li, Yunjian Li, Lishengsa Yue, Zhiyong Cui,

Organizations: State Key Laboratory of Internet of Things for Smart City, University of Macau, Macau SAR, 999078, China · College of Mechanical and Vehicle Engineering, Chongqing University, Chongqing, 400044, China · Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong, 999077, China · School of Transportation Engineering, Tongji University, No. 4800 Cao’an Road, Shanghai, 201804, China ·

Abstract

End-to-end autonomous driving has evolved from camera-to-control regression toward planning-oriented systems that use structured representations, trajectory-level outputs, and increasingly realistic evaluation protocols. This survey reviews this transition across behavior cloning, conditional imitation learning, privileged distillation, BEV and vectorized planning, unified perception-prediction-planning architectures, world-model-based planners, and vision-language-action systems. We argue that the key distinction in modern end-to-end driving is not whether intermediate representations are used, but whether they are learned, supervised, and evaluated to support safe, feasible, and route-compliant planning. To organize the literature, we synthesize existing methods along four axes: input representation, planning output, supervision signal, and evaluation protocol. We further examine the benchmark shift from open-loop trajectory matching to closed-loop simulation, non-reactive real-log evaluation, long-tail testing, and human-preference-aware metrics. Our analysis highlights that architectural progress is difficult to interpret without benchmark-consistent evaluation, and that displacement-based open-loop metrics alone provide limited evidence for safe and human-aligned driving. We conclude with open challenges in uncertainty-aware planning, learner-expert mismatch, runtime safety assurance, language-action grounding, world-model validation, and reproducible benchmarking.

Figures & tables

Existing survey line	Typical scope	Less emphasized issue	This survey’s angle
Broad E2E-AD surveys [ 7 , 24 ]	General E2E driving motivations, architectures, datasets, and challenges.	The post-2024 benchmark mismatch around NAVSIM, Bench2Drive, WOD-E2E, and cross-protocol correlation.	Planning-oriented taxonomy centered on output spaces, supervision, and evaluation validity.
Vision-language driving surveys [ 25 , 26 ]	VLM/VLA perception, reasoning, instruction following, and data generation.	Whether language improves closed-loop planning rather than only QA or explanation.	Action-grounded VLA analysis: language as annotation, instruction, teacher, or action generator.
World-model surveys [ 27 , 28 ]	Generative prediction, scene evolution, simulation, and latent dynamics.	The difference between realistic generation and safe trajectory selection.	Consequence-aware planning view: representation pretraining, future prediction, trajectory evaluation, and reactive simulation.
Foundation-model surveys for AD [ 29 ]	Large models for perception, simulation, reasoning, and decision making.	Planning-specific benchmark evidence and deployment constraints.	Connects foundation-model roles to trajectory output, latency, safety envelopes, and reproducibility.
Benchmark and dataset surveys [ 30 , 31 ]	Dataset scale, sensor modalities, annotation types, and task definitions.	Method taxonomy and how metric choices change E2E claims.	Benchmark-consistent comparison of open-loop, non-reactive, closed-loop, long-tail, and preference-aware evaluation.

TABLE I: Related Surveys and This Survey’s Positioning.

Fig. 1: Search-and-screen workflow used for this survey. The workflow records the main sources, search terms, citation tracing, screening criteria, and final categorization of the reviewed literature.

Fig. 2: Conceptual roadmap of planning-oriented E2E-AD. The field has evolved from direct control imitation toward structured planning interfaces, consequence-aware models, action-grounded language systems, and evaluation protocols that assess planning quality beyond trajectory similarity.

Fig. 3: A planning-oriented view of end-to-end autonomous driving. The key distinction is whether the structure is learned and optimized for downstream planning and evaluated under protocols that reflect driving quality.

Axis	Main options	Representative works	Planning implication
Input representation	Front-view image; multi-view image; LiDAR/radar fusion; BEV or occupancy; object/vector tokens; latent world state; VLM/VLA token space	PilotNet [ 5 ] , TransFuser [ 53 ] , UniAD [ 8 ] , VAD [ 9 ] , LAW [ 62 ] , SpaceDrive [ 63 ]	Determines geometry, interaction modeling, map usage, and whether the policy can reason about unseen or rare agents.
Output space	Steering/throttle; waypoints; trajectory; multi-modal trajectory distribution; scene rollout; language-conditioned action tokens	CIL [ 6 ] , TCP [ 34 ] , VADv2 [ 58 ] , DiffusionDrive [ 64 ] , WAM-Flow [ 21 ]	Determines controllability, uncertainty representation, controller dependence, and metric compatibility.
Supervision	Human BC; synthesized perturbations; privileged teacher; rule-based or RL expert; auxiliary perception/prediction; world-model prediction; language QA; rater preference	ChauffeurNet [ 38 ] , Learning by Cheating [ 40 ] , Hydra-MDP [ 65 ] , LEAD [ 66 ] , DriveLM [ 67 ] , WOD-E2E [ 17 ]	Determines whether the policy learns recovery, safety margins, semantic intent, and long-tail preferences.
Evaluation protocol	Open-loop log replay; non-reactive simulation; reactive closed-loop simulation; real-world closed-loop testing; long-tail and preference-aware evaluation; cross-benchmark correlation	nuScenes [ 68 ] , nuPlan [ 14 ] , NAVSIM [ 16 ] , Bench2Drive [ 15 ] , Fail2Drive [ 69 ] , WOD-E2E [ 17 ]	Determines whether claims concern imitation fidelity, scalable proxy planning, interaction, generalization, or human-aligned judgment.

TABLE II: Four-Axis Taxonomy of Planning-Oriented E2E-AD.

Method	Year	Planning Interface	Training Signal	Evaluation Evidence	Public Resource
PilotNet [ 5 ]	2016	Front camera $\rightarrow$ Image features $\rightarrow$ Steering	BC from demonstrations	Real-road closed-loop demonstration	Partial
CIL [ 6 ]	2018	Camera + command $\rightarrow$ Image features $\rightarrow$ Control	Command-conditioned BC	CARLA closed-loop routes	Code
ChauffeurNet [ 38 ]	2019	Rasterized scene $\rightarrow$ Context features $\rightarrow$ Trajectory	BC with perturbation and auxiliary losses	Internal/RSS-style limited evidence	Paper-only
Learning by Cheating [ 40 ]	2020	Camera student $\rightarrow$ Distilled waypoints/control	Privileged teacher-student IL	CARLA closed-loop routes	Code
TransFuser [ 53 ]	2021	Camera + LiDAR $\rightarrow$ Fusion transformer $\rightarrow$ Waypoints/control	BC with sensor fusion	CARLA closed-loop routes	Code
TCP [ 34 ]	2022	Camera $\rightarrow$ Shared features $\rightarrow$ Trajectory and control	BC with coupled planning/control	CARLA closed-loop routes	Code

TABLE III: Representative Planning-Oriented E2E-AD Methods.

Design choice	What it improves	What it risks	Useful diagnostic
Direct control output	Minimal engineering and fast inference.	Weak interpretability, unstable actuation, hard safety checks, poor multi-modality.	Compare closed-loop recovery and comfort against waypoint or trajectory variants.
Waypoint or trajectory output	Better compatibility with controllers, constraints, and planning metrics.	Controller choices can hide policy weaknesses; open-loop trajectory loss can over-penalize valid alternatives.	Report controller details and evaluate both trajectory error and closed-loop infractions.
Dense BEV/occupancy representation	Strong geometry, map alignment, and interaction modeling.	High compute, sensitivity to perception labels, possible overfitting to static map priors.	Ablate maps, history length, and sensor modality under closed-loop routes.
Sparse object/vector tokens	Efficient interaction reasoning and clearer object-level structure.	May miss unmodeled context, road texture, small obstacles, or non-canonical objects.	Stress test unusual agents, free-space reasoning, and mapless scenarios.
Privileged teacher	Strong supervision and scalable simulated data.	Teacher may rely on information unavailable to the student.	Measure learner-expert asymmetry and student performance under partial observability.
Generative or probabilistic planner	Represents multiple valid futures and avoids mode averaging.	Hard to rank samples; can generate plausible but unsafe trajectories.	Report sample diversity, best-of-N versus selected trajectory, and safety-progress trade-offs.

TABLE IV: Design Trade-Offs in Planning-Oriented E2E-AD.

Training signal	Representative works	Benefit	Main risk
Human behavior cloning	PilotNet [ 5 ] , CIL [ 6 ]	Simple, scalable, log-based supervision.	Covariate shift, route ambiguity, multimodal futures collapsed into one label.
Perturbation and data aggregation	ChauffeurNet [ 38 ] , data aggregation [ 81 ] , feedback-guided AD [ 102 ]	Exposes policy to recovery states and corrective examples.	Quality depends on perturbation design and feedback reliability.
Privileged teacher-student learning	Learning by Cheating [ 40 ] , LEAD [ 66 ]	Uses simulator or privileged state to produce strong supervision.	Expert may rely on information unavailable to the deployed student.
Auxiliary perception/prediction tasks	UniAD [ 8 ] , InterFuser [ 57 ] , VAD [ 9 ]	Stabilizes learning and improves interpretability.	May overfit to intermediate metrics rather than planning quality.
World-model prediction	DriveWorld [ 82 ] , LAW [ 62 ] , WoTE [ 18 ] , World4Drive [ 11 ]	Learns consequences and supports candidate trajectory evaluation.	Latent futures may be poorly calibrated in long-tail scenes.
Language and preference supervision	DriveLM [ 67 ] , SimLingo [ 19 ] , WOD-E2E [ 17 ]	Adds reasoning labels, action alignment, and human preference signals.	Risk of explanation-action mismatch and annotation subjectivity.

TABLE V: Training Signals for Planning-Oriented E2E-AD.

Method	World-model role	Planning connection	Open concern
MILE [ 56 ]	Model-based imitation	Learns latent dynamics for urban driving and connects model learning with policy learning.	Closed-loop realism depends on the learned latent dynamics and training distribution.
GenAD [ 110 ]	Large-scale predictive video model	Can be adapted toward action-conditioned prediction or planning.	Video prediction quality does not directly certify safe trajectory choice.
DriveWorld [ 82 ]	4D representation pretraining	Improves downstream perception, forecasting, occupancy, and planning through spatio-temporal pretraining.	Benefit depends on transfer to planning metrics, not only representation benchmarks.
Drive-WM [ 111 ]	Multiview visual forecasting	Generates multiple futures and selects trajectories through image-based rewards.	Requires reliable future generation under ego interventions.
LAW [ 62 ]	Latent world model	Enhances E2E driving through self-supervised future modeling.	Latent prediction errors can be hard to diagnose.
WoTE [ 18 ]	BEV world-model evaluator	Predicts future BEV states to evaluate candidate trajectories online.	Needs calibration between BEV future quality and closed-loop safety.

TABLE VI: World Models and Generative Planning.

Method	Language role	Action interface	Main limitation or open question
DriveLM [ 67 ]	Graph VQA and planning-oriented reasoning labels.	Language supervises perception-prediction-planning questions rather than directly controlling the vehicle.	Needs faithful link between QA correctness and driving quality.
LMDrive [ 105 ]	Language-guided closed-loop driving.	Uses language instruction as part of the driving input.	Generalization to ambiguous and unseen instructions remains difficult.
DriveVLM [ 106 ]	Hierarchical scene description, analysis, and planning.	VLM supports planning decisions with structured reasoning.	Real-time deployment and verification are challenging.
OmniDrive [ 114 ]	3D-aware multimodal LLM-agent and counterfactual reasoning.	Couples 3D perception, reasoning, and planning tasks.	Benchmarking reasoning improvements against closed-loop driving is still immature.
EMMA [ 85 ]	Generalist multimodal model with driving outputs represented in text.	Generates planner trajectories, objects, and road graph elements through task prompts.	Textual coordinate/action representation is expensive and can be awkward for precise control.
DiMA [ 108 ]	Distills MLLM knowledge into a vision planner.	Language model can be removed or reduced at inference.	Distillation may lose rare semantic reasoning capabilities.

TABLE VII: Language, VLM, and VLA Driving Systems.

Resource	Type	Main contribution	Planning-oriented caveat
CARLA [ 119 ]	Closed-loop simulator	Enables interactive route following, scenario testing, and leaderboards.	Simulation realism and route randomness can affect comparability.
nuScenes [ 68 ]	Real-world multimodal dataset	Standard perception and planning-related dataset with cameras, LiDAR, radar, maps, and annotations.	Open-loop planning on logged scenes does not guarantee closed-loop behavior.
nuPlan [ 14 ]	Real-world planning benchmark	Introduces large-scale planning evaluation with closed-loop simulator and planning metrics.	Metric implementation and simulator settings must be reported carefully.
Bench2Drive [ 15 ]	Closed-loop E2E benchmark	Provides standardized training data and multi-ability CARLA evaluation across interactive scenarios.	Still inherits CARLA sim-to-real limitations.
NAVSIM [ 16 ]	Non-reactive real-log benchmark	Scalable real-sensor evaluation through PDMS and pseudo-simulation.	Background agents do not react to ego behavior.
DriveLM [ 67 ]	Language/reasoning dataset	Adds graph VQA and planning-oriented language supervision.	Language correctness must be tied to action quality.

TABLE VIII: Datasets and Evaluation Resources.

Protocol	Examples	What it measures well	What it cannot establish alone
Open-loop log replay	nuScenes planning splits, WOD-E2E trajectory prediction	Imitation fidelity, fast iteration, large-scale real-log evaluation.	Recovery, interaction, compounding errors, and whether a different safe trajectory is preferable.
Safety-aware open-loop or pseudo-simulation	NAVSIM, NAVSIM-v2 style metrics	Scalable planning proxies using real sensor inputs and safety/progress/comfort submetrics.	Full reactivity of other agents and closed-loop recovery after ego-induced deviations.
Reactive closed-loop simulation	CARLA Leaderboard, Bench2Drive, nuPlan closed-loop	Interaction, route completion, infractions, comfort, and recovery under the policy’s own state distribution.	Sim-to-real validity, rare real-world semantics, and human preference in ambiguous scenes.
Generative reactive evaluation	Bench2Drive-R, emerging world-model simulators	Potential bridge between real data and reactive closed-loop testing.	Rendering and behavior model fidelity must be validated separately.
Long-tail stress tests	WOD-E2E, Fail2Drive, hazardous scenario generation	Rare events, distribution shift, paired generalization drops, and preference-sensitive decisions.	Average driving competence across normal routes.
Cross-benchmark correlation	NAVSIM–Bench2Drive correlation studies [ 121 ]	Whether proxy metrics predict closed-loop ranking.	True safety; correlations can be sample-dependent and non-monotonic.

TABLE IX: Evaluation Protocols and Supported Claims.

Fig. 4: Why open-loop and closed-loop evaluations can disagree. Open-loop trajectory matching tests similarity under the logged state distribution, while closed-loop evaluation tests the policy under the states it actually induces.

Mechanism	How it is used	Strength	Limitation
Structured intermediate features	BEV, occupancy, vectors, object tokens, or map elements remain visible to the planner.	Supports debugging and compatibility with rule checks.	Intermediate correctness does not guarantee final action safety.
Safety-aware losses	Penalize collisions, off-road motion, traffic-rule violations, or uncomfortable acceleration.	Easy to train and benchmark.	Loss weights can overfit to metrics or produce overly conservative behavior.
Runtime filters	Emergency braking, drivable-area filters, occupancy constraints, or fallback policies.	Provides a last line of empirical defense.	Can mask planner weaknesses and may cause deadlock or abrupt behavior.
Uncertainty estimates	Probabilistic plans, diffusion samples, ensemble uncertainty, or calibrated risk scores.	Helps identify ambiguous scenes and multi-modal futures.	Uncertainty can be poorly calibrated under distribution shift.
Causal diagnostics	Counterfactual scene edits and interventions [ 122 ] .	Reveals shortcut learning and failure causes.	Expensive to scale; counterfactual validity is hard.
Language rationales	DriveLM, LMDrive, Drive-R1, EMMA, SimLingo-style explanations [ 67 , 105 , 85 , 19 , 123 ] .	Improves human inspection and instruction following.	Explanations may not be faithful to the trajectory generator.

TABLE X: Safety and Interpretability Mechanisms.

Item	Minimum reporting requirement	Why it matters
Benchmark version	Dataset split, simulator version, metric version, leaderboard date, and official-server versus local evaluation.	Avoids hidden protocol mismatch and stale leaderboard comparisons.
Sensor setup	Camera count, resolution, history length, LiDAR/radar use, ego state, map inputs, and route or command interface.	Determines whether two methods solve the same observation problem.
Output and controller	Direct control, waypoint sequence, deterministic or probabilistic trajectory, action tokens, and PID/MPC/learned controller details.	Separates planner quality from controller effects and post-processing.
Training signal	Human BC, simulator expert, privileged distillation, auxiliary tasks, RL, world-model loss, language QA, preference labels, or feedback correction.	Explains whether gains come from architecture, supervision, or expert design.
Data curation	Filtering, hard-scenario mining, perturbations, augmentation, pseudo-expert generation, and long-tail sampling.	Makes robustness claims auditable rather than anecdotal.
Runtime structure	Safety filter, emergency brake, drivable-area constraint, uncertainty threshold, fallback policy, or post-planning optimizer.	Clarifies whether reported safety comes from the learned planner or external assurance.

TABLE XI: Reproducibility Checklist.

Resource	Reproducibility role	Useful for	Main caution
CARLA [ 119 ]	Simulator platform	Closed-loop route following, scenario design, expert data generation.	Simulator version, route set, traffic manager, and random seed can change results.
TransFuser/TCP lineage [ 53 , 34 ]	Public baselines and strong CARLA agents	Comparing sensor fusion, trajectory-control coupling, and closed-loop behavior.	Results are sensitive to route protocol and controller details.
UniAD [ 8 ]	Unified planning-oriented stack	Studying perception-prediction-planning coupling and multi-task losses.	Heavy training and nuScenes-centered evaluation can limit easy replication.
VAD/VADv2 [ 9 , 58 ]	Vectorized and probabilistic planning baselines	Testing vector representation and uncertainty-aware planning.	Open-loop performance needs closed-loop triangulation.
Bench2Drive [ 15 ]	Standardized closed-loop E2E benchmark	Multi-ability evaluation, fixed training data, CARLA closed-loop comparison.	Sim-to-real gap and leaderboard variance remain.
NAVSIM [ 16 ]	Scalable non-reactive real-log benchmark	Fast proxy evaluation with safety/progress/comfort metrics.	Background agents are not reactive; ranking inversions with closed-loop can occur.

TABLE XII: Public Resources and Reproducibility Levels.

Problem	Why it is unsolved	Promising direction	What evidence is needed
Metric validity	Open-loop, non-reactive, closed-loop, and preference metrics test different properties.	Cross-benchmark correlation and causal analysis of metric failures.	Paired evaluations across NAVSIM, Bench2Drive, WOD-E2E, and real-world or high-fidelity simulation.
Long-tail robustness	Rare events are too sparse for ordinary imitation and too diverse for fixed scenario suites.	Data mining, preference labels, hazardous scenario generation, and paired generalization tests.	Degradation curves, not just average scores.
Learner-expert asymmetry	Expert demonstrations may use unobservable or unrealistically precise state.	Student-aware expert design, multi-teacher distillation, uncertainty-aware imitation.	Controlled experiments holding architecture and data scale fixed.
World-model calibration	Generated futures may look plausible but fail under intervention.	Uncertainty-aware world models, action-conditioned rollouts, future-aware rewarders.	Calibration, rare-case prediction, and closed-loop plan-selection tests.
Language-action grounding	VLMs reason in semantic space while vehicles execute metric trajectories.	Spatial tokens, action-aligned language supervision, distilled planners, structured VLA decoders.	Ablations showing trajectory improvement, not only better explanations.
Formal and empirical safety	Public E2E work mostly reports benchmark mitigation rather than guarantees.	Runtime monitors, verified envelopes, uncertainty thresholds, fallback planning.	Failure-rate confidence intervals, monitor recall, and adversarial or counterfactual testing.

TABLE XIII: Open Problems and Needed Evidence.

End-to-End Autonomous Driving

End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.

See all 227 papers

What Matters for Scalable and Robust Learning in End-to-End Driving Planners?
Mar 16, 2026David Holtz, Niklas Hanselmann, Simon Doll +2End-to-End Autonomous Driving Autonomous Driving Planning
Mercedes-Benz AG · Max-Planck-Institute for Informatics, SIC
Post-Training in End-to-End Autonomous Driving
Jul 9, 2026Ruining Yang, Muxing Wang, Yixiao Chen +8End-to-End Autonomous Driving Autonomous Driving
Northeastern University, Boston, USA · Purdue University, West Lafayette, USA
Closed-Loop Aware Planning via Imitation-Free Speed Scoring for End-to-End Autonomous Driving
Oct 26, 2026Emad Aghajanzadeh, Thomas Schwarzfischer, Marc Kaufeld +8Autonomous Driving Planning End-to-End Autonomous Driving
Technical University of Munich, Germany · BMW Group, Germany · TU Wien, Austria

Mar 16, 2026J/K move · Enter open · S save

What Matters for Scalable and Robust Learning in End-to-End Driving Planners?

David Holtz, Niklas Hanselmann, Simon Doll, Marius Cordts+1

Mercedes-Benz AG · Max-Planck-Institute for Informatics, SIC

End-to-End Autonomous Driving Autonomous Driving Planning Trajectory Representation Learning

Open paper AskPDF arXiv

End-to-end autonomous driving has gained significant attention for its potential to learn robust behavior in interactive scenarios and scale with data. Popular architectures often build on separate modules for perception and planning connected through latent representations, such as bird's eye view feature grids, to maintain end-to-end differentiability. This paradigm emerged mostly on open-loop datasets, with evaluation focusing not only on driving performance, but also intermediate perception tasks. Unfortunately, architectural advances that excel in open-loop often fail to translate to scalable learning of robust closed-loop driving. In this paper, we systematically re-examine the impact of common architectural patterns on closed-loop performance: (1) high-resolution perceptual representations, (2) disentangled trajectory representations, and (3) generative planning. Crucially, our analysis evaluates the combined impact of these patterns, revealing both unexpected limitations as well as underexplored synergies. Building on these insights, we introduce BevAD, a novel lightweight and highly scalable end-to-end driving architecture. BevAD achieves 72.7% success rate on the Bench2Drive benchmark and demonstrates strong data-scaling behavior using pure imitation learning. Our code and models are publicly available here: https://dmholtz.github.io/bevad/

Abstract

End-to-End Autonomous Driving

Explore similar work

What Matters for Scalable and Robust Learning in End-to-End Driving Planners?

Post-Training in End-to-End Autonomous Driving

Closed-Loop Aware Planning via Imitation-Free Speed Scoring for End-to-End Autonomous Driving