Planning-Oriented End-to-End Autonomous Driving: Architectures, Evaluation, and Emerging Paradigms
Organizations: State Key Laboratory of Internet of Things for Smart City, University of Macau, Macau SAR, 999078, China · College of Mechanical and Vehicle Engineering, Chongqing University, Chongqing, 400044, China · Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong, 999077, China · School of Transportation Engineering, Tongji University, No. 4800 Cao’an Road, Shanghai, 201804, China ·
Abstract
End-to-end autonomous driving has evolved from camera-to-control regression toward planning-oriented systems that use structured representations, trajectory-level outputs, and increasingly realistic evaluation protocols. This survey reviews this transition across behavior cloning, conditional imitation learning, privileged distillation, BEV and vectorized planning, unified perception-prediction-planning architectures, world-model-based planners, and vision-language-action systems. We argue that the key distinction in modern end-to-end driving is not whether intermediate representations are used, but whether they are learned, supervised, and evaluated to support safe, feasible, and route-compliant planning. To organize the literature, we synthesize existing methods along four axes: input representation, planning output, supervision signal, and evaluation protocol. We further examine the benchmark shift from open-loop trajectory matching to closed-loop simulation, non-reactive real-log evaluation, long-tail testing, and human-preference-aware metrics. Our analysis highlights that architectural progress is difficult to interpret without benchmark-consistent evaluation, and that displacement-based open-loop metrics alone provide limited evidence for safe and human-aligned driving. We conclude with open challenges in uncertainty-aware planning, learner-expert mismatch, runtime safety assurance, language-action grounding, world-model validation, and reproducible benchmarking.
Figures & tables
| Existing survey line | Typical scope | Less emphasized issue | This survey’s angle |
|---|---|---|---|
| Broad E2E-AD surveys [ 7 , 24 ] | General E2E driving motivations, architectures, datasets, and challenges. | The post-2024 benchmark mismatch around NAVSIM, Bench2Drive, WOD-E2E, and cross-protocol correlation. | Planning-oriented taxonomy centered on output spaces, supervision, and evaluation validity. |
| Vision-language driving surveys [ 25 , 26 ] | VLM/VLA perception, reasoning, instruction following, and data generation. | Whether language improves closed-loop planning rather than only QA or explanation. | Action-grounded VLA analysis: language as annotation, instruction, teacher, or action generator. |
| World-model surveys [ 27 , 28 ] | Generative prediction, scene evolution, simulation, and latent dynamics. | The difference between realistic generation and safe trajectory selection. | Consequence-aware planning view: representation pretraining, future prediction, trajectory evaluation, and reactive simulation. |
| Foundation-model surveys for AD [ 29 ] | Large models for perception, simulation, reasoning, and decision making. | Planning-specific benchmark evidence and deployment constraints. | Connects foundation-model roles to trajectory output, latency, safety envelopes, and reproducibility. |
| Benchmark and dataset surveys [ 30 , 31 ] | Dataset scale, sensor modalities, annotation types, and task definitions. | Method taxonomy and how metric choices change E2E claims. | Benchmark-consistent comparison of open-loop, non-reactive, closed-loop, long-tail, and preference-aware evaluation. |
| Axis | Main options | Representative works | Planning implication |
|---|---|---|---|
| Input representation | Front-view image; multi-view image; LiDAR/radar fusion; BEV or occupancy; object/vector tokens; latent world state; VLM/VLA token space | PilotNet [ 5 ] , TransFuser [ 53 ] , UniAD [ 8 ] , VAD [ 9 ] , LAW [ 62 ] , SpaceDrive [ 63 ] | Determines geometry, interaction modeling, map usage, and whether the policy can reason about unseen or rare agents. |
| Output space | Steering/throttle; waypoints; trajectory; multi-modal trajectory distribution; scene rollout; language-conditioned action tokens | CIL [ 6 ] , TCP [ 34 ] , VADv2 [ 58 ] , DiffusionDrive [ 64 ] , WAM-Flow [ 21 ] | Determines controllability, uncertainty representation, controller dependence, and metric compatibility. |
| Supervision | Human BC; synthesized perturbations; privileged teacher; rule-based or RL expert; auxiliary perception/prediction; world-model prediction; language QA; rater preference | ChauffeurNet [ 38 ] , Learning by Cheating [ 40 ] , Hydra-MDP [ 65 ] , LEAD [ 66 ] , DriveLM [ 67 ] , WOD-E2E [ 17 ] | Determines whether the policy learns recovery, safety margins, semantic intent, and long-tail preferences. |
| Evaluation protocol | Open-loop log replay; non-reactive simulation; reactive closed-loop simulation; real-world closed-loop testing; long-tail and preference-aware evaluation; cross-benchmark correlation | nuScenes [ 68 ] , nuPlan [ 14 ] , NAVSIM [ 16 ] , Bench2Drive [ 15 ] , Fail2Drive [ 69 ] , WOD-E2E [ 17 ] | Determines whether claims concern imitation fidelity, scalable proxy planning, interaction, generalization, or human-aligned judgment. |
| Method | Year | Planning Interface | Training Signal | Evaluation Evidence | Public Resource |
|---|---|---|---|---|---|
| PilotNet [ 5 ] | 2016 | Front camera Image features Steering | BC from demonstrations | Real-road closed-loop demonstration | Partial |
| CIL [ 6 ] | 2018 | Camera + command Image features Control | Command-conditioned BC | CARLA closed-loop routes | Code |
| ChauffeurNet [ 38 ] | 2019 | Rasterized scene Context features Trajectory | BC with perturbation and auxiliary losses | Internal/RSS-style limited evidence | Paper-only |
| Learning by Cheating [ 40 ] | 2020 | Camera student Distilled waypoints/control | Privileged teacher-student IL | CARLA closed-loop routes | Code |
| TransFuser [ 53 ] | 2021 | Camera + LiDAR Fusion transformer Waypoints/control | BC with sensor fusion | CARLA closed-loop routes | Code |
| TCP [ 34 ] | 2022 | Camera Shared features Trajectory and control | BC with coupled planning/control | CARLA closed-loop routes | Code |
| Design choice | What it improves | What it risks | Useful diagnostic |
|---|---|---|---|
| Direct control output | Minimal engineering and fast inference. | Weak interpretability, unstable actuation, hard safety checks, poor multi-modality. | Compare closed-loop recovery and comfort against waypoint or trajectory variants. |
| Waypoint or trajectory output | Better compatibility with controllers, constraints, and planning metrics. | Controller choices can hide policy weaknesses; open-loop trajectory loss can over-penalize valid alternatives. | Report controller details and evaluate both trajectory error and closed-loop infractions. |
| Dense BEV/occupancy representation | Strong geometry, map alignment, and interaction modeling. | High compute, sensitivity to perception labels, possible overfitting to static map priors. | Ablate maps, history length, and sensor modality under closed-loop routes. |
| Sparse object/vector tokens | Efficient interaction reasoning and clearer object-level structure. | May miss unmodeled context, road texture, small obstacles, or non-canonical objects. | Stress test unusual agents, free-space reasoning, and mapless scenarios. |
| Privileged teacher | Strong supervision and scalable simulated data. | Teacher may rely on information unavailable to the student. | Measure learner-expert asymmetry and student performance under partial observability. |
| Generative or probabilistic planner | Represents multiple valid futures and avoids mode averaging. | Hard to rank samples; can generate plausible but unsafe trajectories. | Report sample diversity, best-of-N versus selected trajectory, and safety-progress trade-offs. |
| Training signal | Representative works | Benefit | Main risk |
|---|---|---|---|
| Human behavior cloning | PilotNet [ 5 ] , CIL [ 6 ] | Simple, scalable, log-based supervision. | Covariate shift, route ambiguity, multimodal futures collapsed into one label. |
| Perturbation and data aggregation | ChauffeurNet [ 38 ] , data aggregation [ 81 ] , feedback-guided AD [ 102 ] | Exposes policy to recovery states and corrective examples. | Quality depends on perturbation design and feedback reliability. |
| Privileged teacher-student learning | Learning by Cheating [ 40 ] , LEAD [ 66 ] | Uses simulator or privileged state to produce strong supervision. | Expert may rely on information unavailable to the deployed student. |
| Auxiliary perception/prediction tasks | UniAD [ 8 ] , InterFuser [ 57 ] , VAD [ 9 ] | Stabilizes learning and improves interpretability. | May overfit to intermediate metrics rather than planning quality. |
| World-model prediction | DriveWorld [ 82 ] , LAW [ 62 ] , WoTE [ 18 ] , World4Drive [ 11 ] | Learns consequences and supports candidate trajectory evaluation. | Latent futures may be poorly calibrated in long-tail scenes. |
| Language and preference supervision | DriveLM [ 67 ] , SimLingo [ 19 ] , WOD-E2E [ 17 ] | Adds reasoning labels, action alignment, and human preference signals. | Risk of explanation-action mismatch and annotation subjectivity. |
| Method | World-model role | Planning connection | Open concern |
|---|---|---|---|
| MILE [ 56 ] | Model-based imitation | Learns latent dynamics for urban driving and connects model learning with policy learning. | Closed-loop realism depends on the learned latent dynamics and training distribution. |
| GenAD [ 110 ] | Large-scale predictive video model | Can be adapted toward action-conditioned prediction or planning. | Video prediction quality does not directly certify safe trajectory choice. |
| DriveWorld [ 82 ] | 4D representation pretraining | Improves downstream perception, forecasting, occupancy, and planning through spatio-temporal pretraining. | Benefit depends on transfer to planning metrics, not only representation benchmarks. |
| Drive-WM [ 111 ] | Multiview visual forecasting | Generates multiple futures and selects trajectories through image-based rewards. | Requires reliable future generation under ego interventions. |
| LAW [ 62 ] | Latent world model | Enhances E2E driving through self-supervised future modeling. | Latent prediction errors can be hard to diagnose. |
| WoTE [ 18 ] | BEV world-model evaluator | Predicts future BEV states to evaluate candidate trajectories online. | Needs calibration between BEV future quality and closed-loop safety. |
| Method | Language role | Action interface | Main limitation or open question |
|---|---|---|---|
| DriveLM [ 67 ] | Graph VQA and planning-oriented reasoning labels. | Language supervises perception-prediction-planning questions rather than directly controlling the vehicle. | Needs faithful link between QA correctness and driving quality. |
| LMDrive [ 105 ] | Language-guided closed-loop driving. | Uses language instruction as part of the driving input. | Generalization to ambiguous and unseen instructions remains difficult. |
| DriveVLM [ 106 ] | Hierarchical scene description, analysis, and planning. | VLM supports planning decisions with structured reasoning. | Real-time deployment and verification are challenging. |
| OmniDrive [ 114 ] | 3D-aware multimodal LLM-agent and counterfactual reasoning. | Couples 3D perception, reasoning, and planning tasks. | Benchmarking reasoning improvements against closed-loop driving is still immature. |
| EMMA [ 85 ] | Generalist multimodal model with driving outputs represented in text. | Generates planner trajectories, objects, and road graph elements through task prompts. | Textual coordinate/action representation is expensive and can be awkward for precise control. |
| DiMA [ 108 ] | Distills MLLM knowledge into a vision planner. | Language model can be removed or reduced at inference. | Distillation may lose rare semantic reasoning capabilities. |
| Resource | Type | Main contribution | Planning-oriented caveat |
|---|---|---|---|
| CARLA [ 119 ] | Closed-loop simulator | Enables interactive route following, scenario testing, and leaderboards. | Simulation realism and route randomness can affect comparability. |
| nuScenes [ 68 ] | Real-world multimodal dataset | Standard perception and planning-related dataset with cameras, LiDAR, radar, maps, and annotations. | Open-loop planning on logged scenes does not guarantee closed-loop behavior. |
| nuPlan [ 14 ] | Real-world planning benchmark | Introduces large-scale planning evaluation with closed-loop simulator and planning metrics. | Metric implementation and simulator settings must be reported carefully. |
| Bench2Drive [ 15 ] | Closed-loop E2E benchmark | Provides standardized training data and multi-ability CARLA evaluation across interactive scenarios. | Still inherits CARLA sim-to-real limitations. |
| NAVSIM [ 16 ] | Non-reactive real-log benchmark | Scalable real-sensor evaluation through PDMS and pseudo-simulation. | Background agents do not react to ego behavior. |
| DriveLM [ 67 ] | Language/reasoning dataset | Adds graph VQA and planning-oriented language supervision. | Language correctness must be tied to action quality. |
| Protocol | Examples | What it measures well | What it cannot establish alone |
|---|---|---|---|
| Open-loop log replay | nuScenes planning splits, WOD-E2E trajectory prediction | Imitation fidelity, fast iteration, large-scale real-log evaluation. | Recovery, interaction, compounding errors, and whether a different safe trajectory is preferable. |
| Safety-aware open-loop or pseudo-simulation | NAVSIM, NAVSIM-v2 style metrics | Scalable planning proxies using real sensor inputs and safety/progress/comfort submetrics. | Full reactivity of other agents and closed-loop recovery after ego-induced deviations. |
| Reactive closed-loop simulation | CARLA Leaderboard, Bench2Drive, nuPlan closed-loop | Interaction, route completion, infractions, comfort, and recovery under the policy’s own state distribution. | Sim-to-real validity, rare real-world semantics, and human preference in ambiguous scenes. |
| Generative reactive evaluation | Bench2Drive-R, emerging world-model simulators | Potential bridge between real data and reactive closed-loop testing. | Rendering and behavior model fidelity must be validated separately. |
| Long-tail stress tests | WOD-E2E, Fail2Drive, hazardous scenario generation | Rare events, distribution shift, paired generalization drops, and preference-sensitive decisions. | Average driving competence across normal routes. |
| Cross-benchmark correlation | NAVSIM–Bench2Drive correlation studies [ 121 ] | Whether proxy metrics predict closed-loop ranking. | True safety; correlations can be sample-dependent and non-monotonic. |
| Mechanism | How it is used | Strength | Limitation |
|---|---|---|---|
| Structured intermediate features | BEV, occupancy, vectors, object tokens, or map elements remain visible to the planner. | Supports debugging and compatibility with rule checks. | Intermediate correctness does not guarantee final action safety. |
| Safety-aware losses | Penalize collisions, off-road motion, traffic-rule violations, or uncomfortable acceleration. | Easy to train and benchmark. | Loss weights can overfit to metrics or produce overly conservative behavior. |
| Runtime filters | Emergency braking, drivable-area filters, occupancy constraints, or fallback policies. | Provides a last line of empirical defense. | Can mask planner weaknesses and may cause deadlock or abrupt behavior. |
| Uncertainty estimates | Probabilistic plans, diffusion samples, ensemble uncertainty, or calibrated risk scores. | Helps identify ambiguous scenes and multi-modal futures. | Uncertainty can be poorly calibrated under distribution shift. |
| Causal diagnostics | Counterfactual scene edits and interventions [ 122 ] . | Reveals shortcut learning and failure causes. | Expensive to scale; counterfactual validity is hard. |
| Language rationales | DriveLM, LMDrive, Drive-R1, EMMA, SimLingo-style explanations [ 67 , 105 , 85 , 19 , 123 ] . | Improves human inspection and instruction following. | Explanations may not be faithful to the trajectory generator. |
| Item | Minimum reporting requirement | Why it matters |
|---|---|---|
| Benchmark version | Dataset split, simulator version, metric version, leaderboard date, and official-server versus local evaluation. | Avoids hidden protocol mismatch and stale leaderboard comparisons. |
| Sensor setup | Camera count, resolution, history length, LiDAR/radar use, ego state, map inputs, and route or command interface. | Determines whether two methods solve the same observation problem. |
| Output and controller | Direct control, waypoint sequence, deterministic or probabilistic trajectory, action tokens, and PID/MPC/learned controller details. | Separates planner quality from controller effects and post-processing. |
| Training signal | Human BC, simulator expert, privileged distillation, auxiliary tasks, RL, world-model loss, language QA, preference labels, or feedback correction. | Explains whether gains come from architecture, supervision, or expert design. |
| Data curation | Filtering, hard-scenario mining, perturbations, augmentation, pseudo-expert generation, and long-tail sampling. | Makes robustness claims auditable rather than anecdotal. |
| Runtime structure | Safety filter, emergency brake, drivable-area constraint, uncertainty threshold, fallback policy, or post-planning optimizer. | Clarifies whether reported safety comes from the learned planner or external assurance. |
| Resource | Reproducibility role | Useful for | Main caution |
|---|---|---|---|
| CARLA [ 119 ] | Simulator platform | Closed-loop route following, scenario design, expert data generation. | Simulator version, route set, traffic manager, and random seed can change results. |
| TransFuser/TCP lineage [ 53 , 34 ] | Public baselines and strong CARLA agents | Comparing sensor fusion, trajectory-control coupling, and closed-loop behavior. | Results are sensitive to route protocol and controller details. |
| UniAD [ 8 ] | Unified planning-oriented stack | Studying perception-prediction-planning coupling and multi-task losses. | Heavy training and nuScenes-centered evaluation can limit easy replication. |
| VAD/VADv2 [ 9 , 58 ] | Vectorized and probabilistic planning baselines | Testing vector representation and uncertainty-aware planning. | Open-loop performance needs closed-loop triangulation. |
| Bench2Drive [ 15 ] | Standardized closed-loop E2E benchmark | Multi-ability evaluation, fixed training data, CARLA closed-loop comparison. | Sim-to-real gap and leaderboard variance remain. |
| NAVSIM [ 16 ] | Scalable non-reactive real-log benchmark | Fast proxy evaluation with safety/progress/comfort metrics. | Background agents are not reactive; ranking inversions with closed-loop can occur. |
| Problem | Why it is unsolved | Promising direction | What evidence is needed |
|---|---|---|---|
| Metric validity | Open-loop, non-reactive, closed-loop, and preference metrics test different properties. | Cross-benchmark correlation and causal analysis of metric failures. | Paired evaluations across NAVSIM, Bench2Drive, WOD-E2E, and real-world or high-fidelity simulation. |
| Long-tail robustness | Rare events are too sparse for ordinary imitation and too diverse for fixed scenario suites. | Data mining, preference labels, hazardous scenario generation, and paired generalization tests. | Degradation curves, not just average scores. |
| Learner-expert asymmetry | Expert demonstrations may use unobservable or unrealistically precise state. | Student-aware expert design, multi-teacher distillation, uncertainty-aware imitation. | Controlled experiments holding architecture and data scale fixed. |
| World-model calibration | Generated futures may look plausible but fail under intervention. | Uncertainty-aware world models, action-conditioned rollouts, future-aware rewarders. | Calibration, rare-case prediction, and closed-loop plan-selection tests. |
| Language-action grounding | VLMs reason in semantic space while vehicles execute metric trajectories. | Spatial tokens, action-aligned language supervision, distilled planners, structured VLA decoders. | Ablations showing trajectory improvement, not only better explanations. |
| Formal and empirical safety | Public E2E work mostly reports benchmark mitigation rather than guarantees. | Runtime monitors, verified envelopes, uncertainty thresholds, fallback planning. | Failure-rate confidence intervals, monitor recall, and adversarial or counterfactual testing. |
End-to-End Autonomous Driving
End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.
See all 227 papers