Autonomous Driving Scenario Generation

Follow this topic Ask about this topic

This field studies how to create driving scenes and agent behaviors that help train and evaluate autonomous vehicles, including rare hazards that are difficult to capture in ordinary driving data. Approaches range from traffic simulators, self-play, and adversarial search to diffusion and world models, scene reconstruction, and language- or rule-based tools that turn specifications into executable tests. Researchers assess scenarios using measures such as realism, physical feasibility, controllability, diversity, and whether they expose failures in closed-loop driving benchmarks. A central challenge is generating genuinely difficult situations without sacrificing natural behavior, while ensuring results transfer beyond the simulator and standard benchmark conditions.

Latest papers

111 papers17 papers in the last four weeks, up 143% on the four weeks before. 0.1% of all new papers.

Model-Aware Generation of Safety-Critical Scenes for Robust End-to-End Driving via Distilled Planner Surrogates
Oct 26, 2026Khaled M. Seyam, Julian Wiederer, Samar Fathalah +2End-to-End Autonomous Driving Autonomous Driving Scenario Generation
University of Stuttgart, Germany · Mercedes-Benz AG, Germany
LAST-V2X: Language-Driven Adaptive Scenario Synthesis and Testing for Communication-Induced Failures in V2X Safety Applications
Oct 22, 2026Keshu Wu, Hao Zhang, Rui Gan +3Scenario-Based Testing Autonomous Driving Scenario Generation
Texas A&M University · University of Wisconsin-Madison · University of Wyoming
FaultGene: LLM-Assisted Fault-Augmented Test Generation for Hardware-in-the-Loop Validation of Autonomous Driving Systems
Oct 21, 2026Chaoran Yuan, Fadi KurdahiAutomated Test Generation Autonomous Driving Safety Evaluation
Center for Embedded and Cyber-Physical Systems, University of California, Irvine, Irvine, USA
HorizonComposer: Spatiotemporally Consistent Driving Video Editing with Enriched Traffic Semantics
Oct 20, 2026Mauricio Soroco, Yuqiu Liu, Zaid Tasneem +5Temporal Consistency in Video Generation Physical Consistency in Video Generation
Simon Fraser University · NEC Labs America
Drive2Gauss: FeedForward 4D Gaussian Reconstruction from Multimodal Generative Driving Videos
Oct 19, 2026Tong Zhao, Zhiyuan Han, Lin Chen +10Multimodal Video Generation Dynamic Scene Reconstruction
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China. · Tongji University, Shanghai, China. · Beijing Technology and Business University, Beijing, China. +8
BodyBridge: Grounded Body Layers for Sim-to-Real Translation of Vulnerable Road Users
Oct 17, 2026Mark Azer, Fabian B. FlohrSim-to-Real Transfer Video Diffusion Models
Intelligent Vehicles Lab Munich University of Applied Sciences
DriveWorld3D Evidence Constrained 3D World Modeling and Generative Simulation for Real-World Autonomous Driving
Oct 15, 2026Wei Wang, Yiru Veronika Wang, Yihan Wang +2World Models for Autonomous Driving Autonomous Driving Simulation
Robostreet research team Cambridge, MA, USA
SHRED: Self-Play for Heterogeneous Road-Users without Expert Demonstrations
Oct 14, 2026Wei-Jer Chang, Zhouchonghao Wu, Akshay Rangesh +3Self-Play RL Autonomous Driving Simulation
Applied Intuition · University of California, Berkeley
Uncertainty-Driven Self-Play for Robust Multi-Agent Driving Policies
Oct 14, 2026Sajad Marvi, Julian Schmidt, Alessandro Canevaro +2Uncertainty-Aware Exploration Autonomous Driving Planning
Mercedes-Benz AG, Germany. · Department of Computer Science, University of Freiburg, Germany.
Assessing the Quality of Concrete Scenarios Generated from Scenario Classes for Automated Driving
Oct 13, 2026Lina Putze, Thies de Graaff, Jan Steffen Becker +1Autonomous Driving Scenario Generation Automated Evaluation
German Aerospace Center (DLR) e.V. Institute of Systems Engineering for Future Mobility, Oldenburg, Germany
Lamarck's Driving School: Discovering Autonomous Driving Training Strategies through Evolutionary Competition
Oct 8, 2026Yichun Ye, He Zhang, Ye Tian +1Curriculum RL Autonomous Driving Scenario Generation
Department of Traffic Engineering and the Key Laboratory of Road and Traffic Engineering, Ministry of Education, Tongji University, Shanghai, China · School of Transportation and Logistics, Southwest Jiaotong University, Chengdu, Sichuan, China
Controllable and Photorealistic Pedestrian Risky Motion Generation for End-to-End Driving Safety Evaluation
Oct 5, 2026Siyuan Liu, Miao Li, Haibao Yu +4Pedestrian Behavior Modeling End-to-End Autonomous Driving
School of Vehicle and Mobility, Tsinghua University · Carnegie Mellon University · State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University
How corner is a corner case? Percentile control for highway scenario generation
Oct 4, 2026Jiaxi Liu, Hang Zhou, Hangyu Li +5Autonomous Driving Safety Evaluation Adversarial Scenario Generation
Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, 53706, WI, USA
Human Behavior-Informed Crash Scenario Generation with Real-World Crash Priors for Autonomous Vehicle Safety Evaluation
Oct 3, 2026Mingxing Peng, Xusen Guo, Long Chen +3Autonomous Driving Scenario Generation Autonomous Driving Safety Evaluation
The Hong Kong University of Science and Technology (Guangzhou), China · Institute of Automation Chinese Academy of Sciences, Beijing, China · The University of Hong Kong, Hong Kong, China +1
TerrainForge: Physics-Grounded Road Geometry Editing for Counterfactual Autonomous Driving
Oct 2, 2026Yang Chen, Yicheng Zhu, zhenning Li +2Autonomous Driving Scenario Generation Autonomous Driving Simulation
Rochester Institute of Technology · University of Macau · City University of Hong Kong
Evaluating Physical Consistency and Plausibility in Generative Scenario Models for Autonomous Driving
Oct 1, 2026Manasa Mariam Mammen, Zafer Kayatas, Stefan WagnerAutonomous Driving Safety Evaluation Autonomous Driving Scenario Generation
Technical University of Munich, Heilbronn, Germany · Mercedes Benz AG, Sindelfingen, Germany
TrafficSignBench: Rule-Centric Closed-Loop Evaluation of Traffic-Sign Compliance in Autonomous Driving
Sep 29, 2026Victoria Smirnova, Viktoriia Zinkovich, Gregorii Bukhtuev +4Autonomous Driving Benchmarks Autonomous Driving Safety Evaluation
Lomonosov Moscow State University · FusionBrain Lab · Moscow Institute of Physics and Technology +3
ExceptionDrive: A Planning-Oriented Counterfactual Corner-Case Benchmark for Autonomous Driving
Sep 29, 2026Ziyi Luo, Zhe Sun, Yehao Lu +5Autonomous Driving Benchmarks Autonomous Driving Safety Evaluation
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Yinwang Intelligent Technology Co., Ltd., Shenzhen, China
HelloWorld: Towards Practical Applications of Generative Driving World Models
Sep 24, 2026Fan Lu, Hanshi Wang, Zijing Wang +20Multi-View Video Generation World Models for Autonomous Driving
Teach-to-Crash: A Closed-Loop Student-Teacher LLM Framework for Collision-Inducing Test Scenario Generation
Sep 23, 2026Zaid Ghazal, Khouloud Gaaloul, Bruce MaximAutonomous Driving Safety Evaluation Adversarial Scenario Generation
University of Michigan-Dearborn Dearborn, MI, USA
Safety-Critical Scenarios Emerge from Initial Scenes
Sep 17, 2026Yin Wu, Jiarong Wei, Carl Esselborn +4Adversarial Scenario Generation Autonomous Driving Simulation
PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving
Sep 8, 2026Yuan Gao, Sebastian Müller, Mattia Piccinini +5Autonomous Driving Safety Evaluation LLM Agent Orchestration
Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich, 85748 Garching, Germany · University College London, London, United Kingdom
Collision Snapshot Guided Time-Reversed Safety-Critical Scenario Generation
Sep 6, 2026Taehyung Kim, Jongeun ChoiAutonomous Driving Scenario Generation Adversarial Scenario Generation
Yonsei University, Seoul, South Korea
One Diffusion Model, Two Roles: Guided Trajectory Planning and Safety-Critical Scenario Generation in Closed-Loop Simulation
Sep 4, 2026Arka Pal, Rajesh Kumar, Hannes Eriksson +4Diffusion-Based Planning Diffusion Model Guidance
KTH Royal Institute of Technology, Sweden · Zenseact AB, Sweden · Chalmers University of Technology, Sweden
CrashDiffuser: VLM-Guided Collision Intent Reasoning for Fine-Grained Safety-Critical Traffic Scenario Generation
Sep 2, 2026Shucheng Zhang, Yuang Zhang, Bingzhang Wang +3Autonomous Driving Scenario Generation Adversarial Scenario Generation
Department of Civil and Environmental Engineering, University of Washington Seattle, WA 98195, USA
SIREN-Bench: Behavior-Driven Generation and Evaluation of Emergency-Vehicle Interactions
Aug 25, 2026Yicheng Zhu, Tianmu Zhao, Haoxin Leng +3Autonomous Driving Benchmarks Autonomous Driving Safety Evaluation
Rochester Institute of Technology · CVS Health · City University of Hong Kong +1
BehaviorWorldGen: Closing the Loop between Action Models and World Simulators via Controllable Behavior-Aware Structured World Generation
Aug 23, 2026Jiaqi Wang, Zhuo Zhang, Haining Guan +15Action-Conditioned World Models World Models for Autonomous Driving
AFARI World Model Team
Think Inside the Chunk: RegulaRAG for Regulation-Compliant Scenario Generation using LLMs: A Case Study of UN Regulation No. 152
Aug 17, 2026Vahid Zolfaghari, Nenad Petrovic, AndrÉ Schamschurko +1Retrieval-Augmented Generation Autonomous Driving Scenario Generation
Chair of Robotics, Artificial Intelligence (AI) and Embedded Systems, Technical University of Munich, Munich, Germany
Video2Track: From Real-World Interaction Videos to Steerable Adversarial Closed-Track Testing for Automated Driving Systems
Aug 12, 2026Mengjie Tian, Xinrui Zhang, Tianyu Li +6Autonomous Driving Safety Evaluation Autonomous Driving
College of Automotive and Energy Engineering, Tongji University, Shanghai 201804, China · Department of Civil and Environmental Engineering, National University of Singapore, 119077, Singapore
Top-down Traffic Scenario Generation via Joint Initial-Goal Diffusion and Trajectory Infilling
Aug 11, 2026Da Saem Lee, Yash Vardhan Pant, Sebastian FischmeisterAutonomous Driving Simulation Autonomous Driving Scenario Generation
Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, Canada

Oct 26, 2026J/K move · Enter open · S save

Model-Aware Generation of Safety-Critical Scenes for Robust End-to-End Driving via Distilled Planner Surrogates

Khaled M. Seyam, Julian Wiederer, Samar Fathalah, Markus Braun+1

University of Stuttgart, Germany · Mercedes-Benz AG, Germany

End-to-End Autonomous Driving Autonomous Driving Scenario Generation Adversarial Scenario Generation Autonomous Driving Safety Evaluation

Open paper AskPDF arXiv

End-to-end driving models perform well on standard benchmarks but collide frequently in safety-critical scenarios. We present a failure-mode analysis on NeuroNCAP which shows that 95.9% of DiffusionDrive's collisions are planning failures despite reliable detection, pinpointing the bottleneck in planning rather than perception. We attribute this to a fundamental data imbalance, as safety-critical interactions are rare in real driving logs, so planners are hardly ever trained on them. We close this gap with a novel model-aware framework that generates safety-critical training scenes against a specific planner. Each planner is attacked at the bird's-eye-view (BEV) level through a differentiable surrogate distilled from its own planning head. This keeps the expensive video renderer out of the attack loop. We extract 50 scenes per planner, each pairing an attack with a safe escape trajectory that serves as the fine-tuning label. A fully automated quality-diversity search generates both the attacks and the escapes, so that their variety is not specified by hand. Only these scenes are rendered to fine-tune the planner, which reduces the computational cost significantly. Fine-tuning on these scenes roughly halves the closed-loop collision rate of two planners on NeuroNCAP (SparseDrive 28.0% to 12.2%, DiffusionDrive 37.5% to 19.2%) while largely preserving normal driving. A cross-planner transfer experiment shows that the gain comes from targeting the specific planner. Each planner improves most on the scenes generated against itself, and DiffusionDrive fine-tuned on SparseDrive's scenes retains only 6.7 of its 18.3-point collision-rate reduction.