Adversarial Scenario Generation

Follow this topic Ask about this topic

The field studies how to construct difficult but meaningful situations that reveal where AI systems fail, from autonomous vehicles and vision models to language and tool-using agents. Methods include simulator-based search, reinforcement learning, diffusion and other generative models, controlled perturbations, and LLM-driven red teaming with iterative feedback. Researchers evaluate generated cases using measures such as collision rates, attack success, realism, feasibility, diversity, and whether failures are avoidable. A central challenge is making scenarios adversarial enough to expose weaknesses while keeping them realistic, interpretable, and useful for training or safety evaluation.

Latest papers

39 papers5 papers in the last four weeks, down 29% on the four weeks before. 0.0% of all new papers.

Model-Aware Generation of Safety-Critical Scenes for Robust End-to-End Driving via Distilled Planner Surrogates
Oct 26, 2026Khaled M. Seyam, Julian Wiederer, Samar Fathalah +2End-to-End Autonomous Driving Autonomous Driving Scenario Generation
University of Stuttgart, Germany · Mercedes-Benz AG, Germany
CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat
Oct 13, 2026Bangya Liu, Jianhong Tu, Yueyang Pan +19AI Agent Security Benchmarks Cybersecurity
2077AI Open Source Foundation · University of Wisconsin–Madison · UC Santa Cruz +8
A Local Agentic AI Framework to Generate Edge Case Scenarios for Safety Assurance
Oct 9, 2026Victoria J. Hodge, Colin Paterson, Rob AlexanderAI Assurance Adversarial Scenario Generation
Department of Computer Science, University of York, York, YO10 5DD, UK
MADS: Multi-Agent Dialog Simulation for Robust Conversational Agents
Oct 8, 2026Arkajyoti Chakraborty, Adam Zernik, Ishika Kulkarni +6Task-Oriented Dialogue Systems Adversarial Robustness of Language Models
University of California, Santa Cruz · Uniphore
How corner is a corner case? Percentile control for highway scenario generation
Oct 4, 2026Jiaxi Liu, Hang Zhou, Hangyu Li +5Autonomous Driving Safety Evaluation Adversarial Scenario Generation
Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, 53706, WI, USA
Penumbra: Sample-Efficient Adversarial Search for Regulatory Obligations
Oct 3, 2026Anthony RhodesAutomated Red Teaming AI Regulatory Compliance
Confidential Core AI
Adversarial Closed-Loop Curriculum for Evolving Role-Playing Agents
Sep 23, 2026Zheng Zhang, Liu Liu, Qi Chai +4Large Language Model-Based Role-Play Simulation Adversarial Scenario Generation
The Hong Kong University of Science and Technology (Guangzhou) · Tencent · Shanghai Jiao Tong University
Teach-to-Crash: A Closed-Loop Student-Teacher LLM Framework for Collision-Inducing Test Scenario Generation
Sep 23, 2026Zaid Ghazal, Khouloud Gaaloul, Bruce MaximAutonomous Driving Safety Evaluation Adversarial Scenario Generation
University of Michigan-Dearborn Dearborn, MI, USA
Safety-Critical Scenarios Emerge from Initial Scenes
Sep 17, 2026Yin Wu, Jiarong Wei, Carl Esselborn +4Adversarial Scenario Generation Autonomous Driving Simulation
Discovering Natural Transformation Vulnerabilities in Black-Box Vision Models
Sep 7, 2026Dongsu Song, DaeYun GO, Jay Hoon JungAdversarial Attacks Adversarial Examples
Collision Snapshot Guided Time-Reversed Safety-Critical Scenario Generation
Sep 6, 2026Taehyung Kim, Jongeun ChoiAutonomous Driving Scenario Generation Adversarial Scenario Generation
Yonsei University, Seoul, South Korea
One Diffusion Model, Two Roles: Guided Trajectory Planning and Safety-Critical Scenario Generation in Closed-Loop Simulation
Sep 4, 2026Arka Pal, Rajesh Kumar, Hannes Eriksson +4Diffusion-Based Planning Diffusion Model Guidance
KTH Royal Institute of Technology, Sweden · Zenseact AB, Sweden · Chalmers University of Technology, Sweden
CrashDiffuser: VLM-Guided Collision Intent Reasoning for Fine-Grained Safety-Critical Traffic Scenario Generation
Sep 2, 2026Shucheng Zhang, Yuang Zhang, Bingzhang Wang +3Autonomous Driving Scenario Generation Adversarial Scenario Generation
Department of Civil and Environmental Engineering, University of Washington Seattle, WA 98195, USA
ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents
Aug 12, 2026Yutao Mou, Pengfei Yang, Zhe Yin +6AI Agent Evaluation Adversarial Scenario Generation
National Engineering Research Center for Software Engineering, Peking University · Harbin Institute of Technology · Beijing University of Posts and Telecommunications +1
Threat-guided Policy-aware Scene Perturbation for Safe Autonomous Driving with Online Reinforcement Learning
Aug 11, 2026Xincong Hu, Lei Ou, Maosen Li +3Adversarial Scenario Generation Safe RL
Nanjing University · Yinwang Intelligent Technology Co., Ltd
Do LLMs Know Their Vulnerable Scenarios?
Jul 26, 2026Ziheng Peng, Huiqi Deng, Haoran Jing +5LLM Interpretability Adversarial Scenario Generation
Xi’an Jiaotong University Shanghai, China · Shanghai Artificial Intelligence Laboratory Shanghai, China
Automatic Hard Example Synthesis with Multi-Level Agentic Data Curation
Jul 15, 2026Genglin Liu, Muye Zhang, Krishnamurthy Viswanathan +5Multimodal Robustness Adversarial Scenario Generation
1UCLA · *Work done as a student researcher at Google. · 2Google
Seduced by the Narrative: Assessing Rule Adherence in Semi-Open Textual Sandboxes
Jul 2, 2026Weiying Chen, Junlong Shen, Zhanyuan Guo +1LLM Safety Benchmarks Adversarial Prompt Generation
University of Alberta, Edmonton, AB, Canada · N-Dice Association · Qilu University of Technology, Shandong, China
AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability
Jun 23, 2026Khanak KhandelwalAdversarial Prompt Generation Adversarial Attacks on LLMs
Indian Institute of Technology Jodhpur
PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Models
Jun 23, 2026Simone Gallivanone, Hossein Khodadadi, Mauro Dore +2VLM Evaluation VLM Robustness
The Italian Institute of Artificial Intelligence (AI4I), Turin, Italy · HikmaAI S.r.l., Pula, Italy
SPHINX: First Explain, Then Explore
Jun 16, 2026Nguyen Do, Tue M. Cao, Tien Van Do +3Autonomous Driving Safety Evaluation Adversarial Robustness
University of Florida, Gainesville, Florida, USA · Budapest University of Technology and Economics, Department of Networked Systems and Services, Budapest, Hungary · University of Debrecen, Debrecen, Hungary
Learning of Robot Safety Policies via Adversarial Synthetic Scenarios
Jun 4, 2026Nikolai Dorofeev, Alexey Odinokov, Rostislav YavorskiyRobot Policy Learning Adversarial Scenario Generation
SafePI.ai Grenoble, France · SafePI.ai Belgrade, Serbia · SafePI.ai Madrid, Spain
EvoDrive: Pareto Evolution for Safety-Critical Autonomous Driving via Self-Improving LLM Agents
Jun 2, 2026Tong Nie, Yuewen Mei, Yihong Tang +4Multi-Objective Evolutionary Optimization Autonomous Driving Safety Evaluation
The Hong Kong Polytechnic University · Tongji University · McGill University +1
ScenePilot: Controllable Boundary-Driven Critical Scenario Generation for Autonomous Driving
May 20, 2026Qiyu Ruan, Yuxuan Wang, He Li +2Autonomous Driving Safety Evaluation Adversarial Scenario Generation
State Key Laboratory of Internet of Things for Smart City (SKL-IOTSC), University of Macau, Macau, China · Faculty of Science and Technology, University of Macau, Macau, China
KG-ASG: Collision-Knowledge-Guided Closed-Loop Adversarial Scenario Generation With Primary-Support Attribution
May 17, 2026Cheng Wang, Chen Xiong, Ziwen Wang +2Autonomous Driving Safety Evaluation Adversarial Scenario Generation
Guangdong Provincial Key Laboratory of Intelligent Transportation System, School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China
PCASim: Promptable Closed-loop Adversarial Simulation for Urban Traffic Environment
May 15, 2026Chuancheng Zhang, Zhenhao Wang, Kaizheng Li +3Autonomous Driving Safety Evaluation Synthetic Data Generation
Shenzhen Research Institute of Shandong University, Shenzhen, 518057, Guangdong, China · School of Airspace Science and Engineering, Shandong University, 264209 Weihai, China. · School of Computer Science and Technology, Shandong University of Finance and Economics, 250014, Jinan, China.
Learning Responsibility-Attributed Adversarial Scenarios for Testing Autonomous Vehicles
May 13, 2026Yizhuo Xiao, Haotian Yan, Ying Wang +5AI Accountability Autonomous Driving Safety Evaluation
School of Engineering and Physical Sciences, Heriot-Watt University, Edinburgh, U.K. · State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University, Shanghai, China · College of Computer Science and Technology, Jilin University, Changchun, China +3
MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety
May 3, 2026Jialin Song, Xiaodong Liu, Weiwei Yang +4LLM Safety Benchmarks Adversarial Prompt Generation
Simon Fraser University · Microsoft Research
Reverse Constitutional AI: A Framework for Controllable Toxic Data Generation via Probability-Clamped RLAIF
Apr 20, 2026Yuan Fang, Yiming Luo, Aimin Zhou +1Language Model Safety Evaluation Adversarial Scenario Generation
East China Normal University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China
Evaluation as Evolution: Transforming Adversarial Diffusion into Closed-Loop Curricula for Autonomous Vehicles
Apr 8, 2026Yicheng Guo, Jiaqi Liu, Chengkai Xu +2Autonomous Driving Safety Evaluation Adversarial Scenario Generation
College of Transportation, Tongji University, Shanghai 201804, China · Department of Computer Science, University of North Carolina at Chapel Hill, United States

Oct 26, 2026J/K move · Enter open · S save

Model-Aware Generation of Safety-Critical Scenes for Robust End-to-End Driving via Distilled Planner Surrogates

Khaled M. Seyam, Julian Wiederer, Samar Fathalah, Markus Braun+1

University of Stuttgart, Germany · Mercedes-Benz AG, Germany

End-to-End Autonomous Driving Autonomous Driving Scenario Generation Adversarial Scenario Generation Autonomous Driving Safety Evaluation

Open paper AskPDF arXiv

End-to-end driving models perform well on standard benchmarks but collide frequently in safety-critical scenarios. We present a failure-mode analysis on NeuroNCAP which shows that 95.9% of DiffusionDrive's collisions are planning failures despite reliable detection, pinpointing the bottleneck in planning rather than perception. We attribute this to a fundamental data imbalance, as safety-critical interactions are rare in real driving logs, so planners are hardly ever trained on them. We close this gap with a novel model-aware framework that generates safety-critical training scenes against a specific planner. Each planner is attacked at the bird's-eye-view (BEV) level through a differentiable surrogate distilled from its own planning head. This keeps the expensive video renderer out of the attack loop. We extract 50 scenes per planner, each pairing an attack with a safe escape trajectory that serves as the fine-tuning label. A fully automated quality-diversity search generates both the attacks and the escapes, so that their variety is not specified by hand. Only these scenes are rendered to fine-tune the planner, which reduces the computational cost significantly. Fine-tuning on these scenes roughly halves the closed-loop collision rate of two planners on NeuroNCAP (SparseDrive 28.0% to 12.2%, DiffusionDrive 37.5% to 19.2%) while largely preserving normal driving. A cross-planner transfer experiment shows that the gain comes from targeting the specific planner. Each planner improves most on the scenes generated against itself, and DiffusionDrive fine-tuned on SparseDrive's scenes retains only 6.7 of its 18.3-point collision-rate reduction.