Generative AI for Autonomous Driving: A Review

Katharina Winter; Abhishek Vivekanandan; Rupert Polley; Yinzhe Shen; Christian Schlauch; Mohamed-Khalil Bouzidi; Bojan Derajic; Natalie Grabowsky

cs.CVMay 21, 2025

Generative AI for Autonomous Driving: A Review

Authors: Katharina Winter, Abhishek Vivekanandan, Rupert Polley, Yinzhe Shen, Christian Schlauch, Mohamed-Khalil Bouzidi, Bojan Derajic, Natalie Grabowsky,

Organizations: Munich University of Applied Sciences, Intelligent Vehicles Lab (IVL), 80335 Munich, Germany · FZI Research Center for Information Technology, 76131 Karlsruhe, Germany · Continental Automotive Technologies GmbH, Guericke Str.7, 60488 Frankfurt a. M., Germany · Technical University of Berlin, Institute of Mathematics, Straße des 17.Juni 136, 10587 Berlin, Germany · German Aerospace Center (DLR), Institute of Transportation Systems, Lilienthalplatz 7, 38108 Braunschweig, Germany · Aptiv Services Deutschland GmbH, Am Technologiepark 1, 42119 Wuppertal, Germany · AI Lab, ZF Friedrichshafen AG, Uni-Campus D 5 1, 66123 Saarbruecken, Germany · DENSO AUTOMOTIVE Deutschland GmbH, Freisinger Str. 21, 85386 Eching, Germany · Karlsruhe Institute of Technology, Scientific Computing Center, Zirkel 2, 76131 Karlsruhe, Germany

Abstract

Generative AI (GenAI) is rapidly advancing the field of Autonomous Driving (AD), extending beyond traditional applications in text, image, and video generation. We explore how generative models can enhance automotive tasks, such as static map creation, dynamic scenario generation, trajectory forecasting, and vehicle motion planning. By examining multiple generative approaches ranging from Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) to Generative Transformers (GTs) and Diffusion Models (DMs), we highlight and compare their capabilities and limitations for AD-specific applications. Additionally, we discuss hybrid methods integrating conventional techniques with generative approaches, and emphasize their improved adaptability and robustness. We also identify relevant datasets and outline open research questions to guide future developments in GenAI. Finally, we discuss five core challenges: safety, interpretability, realtime capabilities, training and benchmarking. We also present recommendations for image generation, dynamic scenario generation, and planning.

Figures & tables

Survey	Published	Application	Features
[ 57 ]	02/2022	ScenarioGen	Methods and tools for safety-critical scenario gen.
[ 268 ]	12/2023	ScenarioGen	Percept., pred., sim. and decision-making for traffic gen.
[ 245 ]	12/2024	ScenarioGen	Non-adaptive and adaptive scenario gen. for testing.
[ 74 ] *	06/2025	ScenarioGen	LLM -, DM -, world model- based Foundation models for scenario gen.
[ 202 ] *	08/2025	ScenarioGen	Distribution GenAI models used in ADS testing.
[ 52 ] *	09/2025	Prediction	LLM-based Foundation models for trajectory prediction.

TABLE I : Overview of existing GenAI surveys. * indicates surveys published after submission. AD Stack includes SceneGen, ScenarioGen, Prediction, Planning, and related tasks.

Model class	Distributional	Inference	Training	Time
	Accuracy	Speed	Stability	Series
Energy Based	–	$\times$	$\times$	$\times$
Transport impl.	–	$\checkmark$	$\times$	–
Transport expl.	$\checkmark$	–	$\checkmark$	–
Diffusion	$\checkmark$	$\times$	$\checkmark$	–
Autoregressive	$\checkmark$	–	$\checkmark$	$\checkmark$

TABLE II : Properties of generative learning methods. The symbol “–” is used when the model class shows mixed performance w.r.t. the criterion. The markings show tendencies, but exceptions occur.

Fig. 2 : Taxonomy of generative learning approaches. Categories and subcategories are shown in green, generative modeling approaches in orange. Hybrid forms may exist but are not listed. Sections IV to IX discuss applications of these methods to the AD stack.

Panel of Figure 4 — Fig. 3 : Comparison of prediction types in a traffic intersection scenario. Blue and orange colors represent different prediction modes. The subfigure (a) shows marginal prediction for the red vehicle, (b) illustrates the conditional prediction of the red vehicle dependent on the ego vehicle (shown in blue), and (c) shows joint prediction, where blue and orange colors represent an entire scene independently.

Model	Year	Approach	Category	Benchmarks	Features
Neural Turtle Graphics [ 45 ]	2019	Autoregressive (encoder-decoder RNN )	S	RoadNet/SpaceNet	Generates plausible road network graphs
HDMapGen [ 160 ]	2021	Autoregressive (graph)	S	Argoverse	Hierarchical graph model for lane-level HD maps
SLEDGE [ 43 ]	2024	Diffusion (latent)	S	nuPlan	Generates lane graph and agents; integrated traffic simulation
DriveSceneGen [ 208 ]	2024	Diffusion (image) + graph	S	Waymo	Two-stage: rasterized scene generation and agent simulation
PolyDiffuse [ 36 ]	2023	Diffusion (set)	S	nuScenes/Argoverse2	Guided set diffusion for HD map reconstruction from sensor data
TrafficGen [ 68 ]	2023	Autoregressive (encoder-decoder)	D	Waymo	Autoregressive vehicle and trajectory generation

TABLE III : Overview of scene and scenario generation methods, with focus on static maps (S), dynamic scenarios (D) and world models (W)

Panel of Figure 6 — Fig. 4 : Demonstration of scenario generation starting from an initial position and a tag. Images were taken from [ 56 ] .

Fig. 5 : Demonstration of text-prompt to scenario generation. Image was taken from [ 213 ] . Prompt: The scene is very dense. There are only vehicles on the left side of the center car. Most cars are moving at high speeds. The ego-vehicle turns right.

Category	Metric	Description
Distribution	Topology fidelity (MMD) [ 160 ]	Graph degree, Laplacian stats
	Geometry fidelity (Fréchet) [ 160 ]	Lane length and orientation stats
	Urban planning (Fréchet) [ 45 ] [ 160 ]	Density, connectivity, reach, convenience
	Map diversity (Chamfer) [ 45 ] [ 160 ]	Novelty and internal diversity of graphs
	Perceptual FID [ 91 ] [ 45 ]	Inception features on rendered maps
Reconstruction	GEO metric [ 43 ]	Point match precision, recall, F1

TABLE IV : Metrics for static scene generation

Model	Year	Approach	Interaction Type	Benchmarks	Features
MFP [ 216 ]	2019	Autoregressive	joint	NGSIM, Argoverse	parallel per-agent RNNs with shared latent variable attention
LB-EBM [ 178 ]	2021	EBM	joint	Stanford Drone Dataset, ETH-UCY	inverse reinforcement learning with latent beliefs
SEEM [ 236 ]	2022	EBM	joint	ETH-UCY	generator and energy network architecture
EPD [ 143 ]	2022	EBM & Diffusion	joint	ETH-UCY	hybrid energy and diffusion prediction model
LatentFormer [ 1 ]	2022	Autoregressive	joint	nuScenes	transformer encoder-decoder conditioned on learned latent intention variables
MotionLM [ 194 ]	2023	Autoregressive	joint	Waymo, Argoverse 2	transformer encoder-decoder predicting over discrete heuristic motion token vocabulary

TABLE V : Overview of Generative Trajectory Forecasting Methods

Category	Metric	Description
Displacement	minADE@ $K$ [ 104 ]	Average Displacement
	minFDE@ $K$ [ 104 ]	Displacement of final step
	MissRate@ $\delta$ [ 104 ]	Existence of prediction within bound
Probabilistic	Negative Log Likelihood (NLL) [ 104 ]	Probability of ground truth
	Expected Calibration Error [ 177 ]	Reliability of predicted confidences
Multi-Modality	Pairwise-Spread [ 198 ]	Spread of predicted modes

TABLE VI : Trajectory Forecasting Metrics

Method	Year	Approach	Benchmarks	Features
Performer-MPC [ 260 ]	2022	Transformer + MPC	Custom Sim/Real World	Learns cost function from vision embeddings; robust to constraint violations.
GAN-MPC [ 24 ]	2023	GAN + MPC	DeepMind Control suite	Minimizes the Jensen-Shannon divergence between the state-trajectory distributions.
LanguageMPC [ 196 ]	2023	LLM + MPC	SUMO + IDSim	LLM generates MPC parameters and cost weights from text prompts.
Transformer-MPC [ 148 ]	2024	Transformer + MPC	Real World	Replaces BADGR’s LSTM with Transformer; improves uncertainty-aware control.
Diffusion SDE MPC [ 58 ]	2025	Diffusion + MPC	Real World	Physics-informed conditional diffusion for stochastic vehicle dynamics.
D-MPC [ 289 ]	2024	Diffusion + MPC	D4RL	Generates initial trajectory; MPC refines under dynamic constraints.

TABLE VII : Overview of Hybrid Motion Planning Methods

Category	Metric	Description
Safety	Collision rate [ 119 ]	Fraction of scenarios with any collision.
	Time-to-collision (TTC) [ 119 ]	Minimum predicted time to impact.
	Drivable area compliance [ 119 ]	Fraction of trajectory within drivable space.
	Safe following distance (SFD) [ 2 ]	Safe gap behind vehicles to react and stop safely.
Compliance	Lane deviation [ 2 ]	Lateral deviation from the assigned lane center.
	Speed-limit compliance [ 119 ]	Fraction of time that ego remains within speed limits.

TABLE VIII : Overview of Motion Planning Metrics

Model	Year	Approach	OL / CL	Benchmarks	Features
DiffusionDrive [ 136 ]	2025	Diffusion	OL	nuScenes, NAVSIM	Diffusion with anchored Gaussian priors and transformer-based decoder for conditioning.
GenAD [ 287 ]	2024	Transport, Autoregressive	OL	nuScenes	Encodes scene features into a latent trajectory space, then predicts&plans future trajectories from it.
DriveGPT4 [ 267 ]	2024	Autoregressive	OL	BDD-X	Predicts vehicle control and driving reasoning.
OmniDrive [ 239 ]	2025	Autoregressive	OL	nuScenes	Leverages 2D VLM priors with 3D position encoding and counterfactual reasoning.
SimLingo [ 188 ]	2025	Autoregressive	CL	Carla, Bench2Drive	Maps split front-view image into path and waypoint representations.
DriveLM [ 221 ]	2024	Autoregressive	OL	nuScenes, SUP-AD	Performs scene-level reasoning for improved planning.

TABLE IX : Overview of End-to-End Methods

	Argoverse [ 33 ]	INTER. [ 280 ]	LYFT [ 97 ]	Waymo [ 67 ]	NuScenes [ 25 ]	Yandex [ 155 ]	Argoverse2 [ 250 ]
# Scenarios	$324$ k	$-$	$170$ k	$104$ k	$41$ k	$600$ k	$250$ k
# Unique Tracks	$11.7$ M	$40$ k	$53.4$ M	$7.64$ M	$-$	$17.4$ M	$13.9$ M
Average Track Length	$2.48$ s	$19.8$ s	$1.84$ s	$7.04$ s	$-$	$-$	$5.16$ s
Total Time	$320$ h	$16.5$ h	$1118$ h	$574$ h	$5.5$ h	$1667$ h	$763$ h
Scenario Duration	$5$ s	$-$	$25$ s	$9.1$ s	$8$ s	$10$ s	$11$ s
Test Forecast Horizon	$3$ s	$3$ s	$5$ s	$8$ s	$6$ s	$5$ s	$6$ s

TABLE X : Overview of Motion Datasets [ 250 ]

Fig. 8 : Connection between taxonomy of generative learning approaches in Figure 2 and AD tasks Figure . The ribbon width is proportional to the frequency of generative learning approaches contributing to the tasks as noted in Tables III , V , VII and IX .

Autonomous Driving

Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.

See all 536 papers

Generative AI for Autonomous Driving: Frontiers and Opportunities
May 13, 2025Yuping Wang, Shuo Xing, Cui Can +46Autonomous Driving Autonomous Driving Scenario Generation
EvoDrive: Pareto Evolution for Safety-Critical Autonomous Driving via Self-Improving LLM Agents
Jun 2, 2026Tong Nie, Yuewen Mei, Yihong Tang +4Multi-Objective Evolutionary Optimization Autonomous Driving Safety Evaluation
The Hong Kong Polytechnic University · Tongji University · McGill University +1
TrafficAlign: Aligning Large Language Models for Traffic Scenario Generation
Jun 27, 2026Zhi Tu, Liangkun Niu, Tianyi ZhangLLM Alignment Autonomous Driving Safety Evaluation
Purdue University

May 13, 2025J/K move · Enter open · S save

Generative AI for Autonomous Driving: Frontiers and Opportunities

Yuping Wang, Shuo Xing, Cui Can, Renjie Li+45

Autonomous Driving Autonomous Driving Scenario Generation

Open paper AskPDF arXiv

Generative Artificial Intelligence (GenAI) constitutes a transformative technological wave that reconfigures industries through its unparalleled capabilities for content creation, reasoning, planning, and multimodal understanding. This revolutionary force offers the most promising path yet toward solving one of engineering's grandest challenges: achieving reliable, fully autonomous driving, particularly the pursuit of Level 5 autonomy. This survey delivers a comprehensive and critical synthesis of the emerging role of GenAI across the autonomous driving stack. We begin by distilling the principles and trade-offs of modern generative modeling, encompassing VAEs, GANs, Diffusion Models, and Large Language Models (LLMs). We then map their frontier applications in image, LiDAR, trajectory, occupancy, video generation as well as LLM-guided reasoning and decision making. We categorize practical applications, such as synthetic data workflows, end-to-end driving strategies, high-fidelity digital twin systems, smart transportation networks, and cross-domain transfer to embodied AI. We identify key obstacles and possibilities such as comprehensive generalization across rare cases, evaluation and safety checks, budget-limited implementation, regulatory compliance, ethical concerns, and environmental effects, while proposing research plans across theoretical assurances, trust metrics, transport integration, and socio-technical influence. By unifying these threads, the survey provides a forward-looking reference for researchers, engineers, and policymakers navigating the convergence of generative AI and advanced autonomous mobility. An actively maintained repository of cited works is available at https://github.com/taco-group/GenAI4AD.

Abstract

Autonomous Driving

Explore similar work

Generative AI for Autonomous Driving: Frontiers and Opportunities

EvoDrive: Pareto Evolution for Safety-Critical Autonomous Driving via Self-Improving LLM Agents

TrafficAlign: Aligning Large Language Models for Traffic Scenario Generation