Human Behavior-Informed Crash Scenario Generation with Real-World Crash Priors for Autonomous Vehicle Safety Evaluation
Authors: Mingxing Peng, Xusen Guo, Long Chen, Xintao Yan, Siyu Teng, Jun Ma
Organizations: The Hong Kong University of Science and Technology (Guangzhou), China · Institute of Automation Chinese Academy of Sciences, Beijing, China · The University of Hong Kong, Hong Kong, China · Shenzhen University, Shenzhen, China
Reliable safety evaluation of autonomous vehicles (AVs) is essential to improving road safety, yet it depends critically on realistic simulation of rare crashes. Existing crash scenario generation methods can increase collision occurrence, but often fail to realistically reproduce how crashes evolve before impact or the distribution of crash types observed in the real world. Here, we present CrashSim, a human behavior-informed crash scenario generation framework that uses real-world crash priors to guide generative multi-agent traffic simulation for more reliable AV safety evaluation. These priors capture how real-world crashes evolve before impact and how different crash types are distributed, allowing limited crash data to guide realistic and scalable scenario generation across naturalistic driving contexts. We evaluate CrashSim against competing methods, showing that it more closely reproduces real-world pre-impact behavior, collision dynamics, collision geometry and crash-type distributions. We further use CrashSim to construct nuCrash dataset, containing over 4,000 crash and near-crash scenarios. Closed-loop evaluation of five AV planners shows that nuCrash more effectively exposes differences in planner safety capabilities than nuScenes. An LLM-assisted evaluation agent further analyzes planner failures to provide capability-level diagnoses and targeted improvement guidance. Together, CrashSim enables realistic and scalable crash generation for more informative AV safety evaluation.
Figures & tables
Figure 1: Challenges in autonomous vehicle safety evaluation and overview of CrashSim. a Limitations of current autonomous vehicle safety evaluation, including the sparsity of real-world crashes and the behavioral, kinematic, and distributional biases of existing safety-critical simulation. Statistics in the real-world testing funnel are derived from ref. [ 32 ] . b Real-world crash processes evolve from intention formation through risk awareness and emergency response to the eventual collision outcome whereas existing safety-critical simulation often overlooks the emergency response and leads to aggressive collisions. c CrashSim retrieves real-world crash priors based on the current scene context and uses them to provide human behavior-informed guidance for generative multi-agent simulation and autonomous vehicle safety evaluation.
Figure 2: CrashSim generates realistic human behavior-informed collision processes. a Temporal composition of intention formation, risk awareness, emergency response and collision in real-world crashes and generated scenarios. CrashSim retains the complete four-stage progression and more closely reproduces the stage proportions observed in real-world crashes. b Time-resolved probability heatmaps showing the evolution of risk awareness and emergency response before collision. CrashSim captures the gradual development of risk awareness followed by the emergence of emergency response. c Temporal distributions of longitudinal acceleration during the final four seconds preceding collision. CrashSim reproduces the predominantly decelerative response and its temporal evolution in real-world crashes. d Distributions of reaction time response time and braking severity across real-world crashes and generated scenarios. Together a–d show that CrashSim captures key human behavioral dynamics underlying crash progression including risk development response timing and emergency braking. e Spatial distributions of collision impact locations across the front, rear, left and right regions of the ego vehicle. CrashSim generates behaviorally realistic ego-at-fault collisions, characterized by a predominance of frontal impacts, thereby enabling more effective evaluation of ego-planner safety. f Percentages of adversarial and other non-ego vehicles that leave the drivable area. The low off-road rates produced by CrashSim indicate physically plausible pre-collision interactions and stronger compliance with road boundaries. Together, e-f demonstrate the behavioral plausibility of CrashSim-generated collision scenarios.
Figure 3: CrashSim preserves realistic crash scenario distributions. a Distribution of fine-grained crash scenarios across real-world crashes and generated datasets. b Distribution of five high-level crash categories, including longitudinal car-following conflicts, lane-change and lateral intrusion conflicts, intersection crossing and turning conflicts, opposing-direction conflicts, and other conflicts. c Distributional discrepancy from real-world crashes, quantified using total variation distance for the five high-level categories and Jensen-Shannon divergence for the sixteen fine-grained crash types. Across both levels of semantic granularity, CrashSim most closely reproduces real-world crash statistics and achieves the lowest distributional discrepancy among the compared methods.
Figure 4: nuCrash constructs a realistic long-tail crash dataset. a Comparison of crash rate and near-crash rate among nuScenes, nuCrash, and SimEngine. b Temporal evolution of the emergency response stage before collision in real-world crashes, nuCrash, and SimEngine. c Distribution of high-level crash categories in real-world crashes, nuCrash, and SimEngine. d Distributional discrepancy from real-world crashes, measured using Total Variation distance and Jensen-Shannon divergence. a shows that nuCrash substantially increases the prevalence of realized collisions relative to the naturalistic nuScenes dataset, whereas SimEngine is dominated by near-crash cases and contains relatively few actual collisions. b-d demonstrate that nuCrash more faithfully reproduces the temporal dynamics and scenario distributions observed in real-world crashes than the existing long-tail dataset. Collectively, these results show that nuCrash provides a more realistic and challenging dataset for autonomous vehicle safety evaluation.
Figure 5: Interpretable and reliable LLM-based evaluation of autonomous driving planners. a Planner performance across five safety and driving-quality metrics on nuScenes and nuCrash. Values are normalized within each metric for visual comparison. b Performance range and between-planner variance on the two datasets, showing substantially stronger planner differentiation on nuCrash. c LLM-based capability assessment across six planner capabilities together with planner-level diagnostic synthesis that translates quantitative evaluation evidence into interpretable planner strengths and limitations. d Alignment between capability assessments and quantitative evidence. e Planner-level audit results quantifying the evidence sufficiency, scenario specificity, and diagnostic depth. Together these analyses demonstrate that the LLM-based evaluation agent can provide interpretable and reliable planner assessments.
Figure 6: Qualitative case studies of LLM-based planner evaluation. a Learning-based IL Planner. b Rule-based Lane-graph Planner. The LLM-based evaluation agent synthesizes cross-dataset planner behavior across naturalistic and long-tail scenarios. It identifies performance degradation and possible failure mechanisms. It further translates the diagnostic findings into planner refinement guidance and highlights representative performance-limiting scenarios. These cases illustrate the ability of the LLM-based evaluation to provide interpretable and actionable planner assessments across distinct planner paradigms.
Figure 7: Ablation studies of the crash-prior retrieval module and the VLM agent. a Temporal composition of the collision process in real-world crashes, CrashSim and CrashSim without the crash-prior retrieval module. Removing the crash-prior retrieval module increases the discrepancy from the real-world composition, indicating that retrieved crash knowledge helps preserve realistic collision-process characteristics. b Temporal evolution of risk awareness and emergency response probabilities for the same settings. The complete framework more closely reproduces the temporal progression observed in real-world crashes, demonstrating that the crash-prior retrieval module improves the realism of pre-collision behavioral dynamics. c Total variation and Jensen-Shannon divergence between generated and real-world crash distributions for CrashSim and CrashSim without the VLM-based scene-understanding agent. The complete framework achieves consistently lower distances, indicating that incorporating real-world crash-type distribution priors into scene-level reasoning improves the alignment of generated scenario types with real-world crash statistics.
Figure 8: Human behavior-informed crash generation and interpretable safety evaluation with CrashSim. a Human behavior-informed adversarial crash scenario generation. CrashSim first interprets the current driving context through VLM-based scene understanding and retrieves relevant real-world crash priors through hierarchical matching. The retrieved crash prior provides human behavior-informed anchor guidance for a diffusion-based generator to produce safety-critical crash scenarios. b Interpretable evaluation and actionable insights. Generated crash scenarios form the long-tail nuCrash dataset which complements the naturalistic nuScenes dataset for closed-loop planner evaluation. An LLM-based evaluation agent performs both per-dataset and cross-dataset evaluation to diagnose failure mechanisms, assess planner capabilities, and provide actionable guidance for planner improvement.
Evaluation of autonomous vehicle (AV) planners in safety-critical closed-loop simulation is essential for real-world deployment. However, generating controllable safety-critical scenarios remains challenging. Existing approaches use soft guidance that provides only probabilistic preferences and cannot guarantee the satisfaction of geometric and severity constraints associated with specific collision types. We introduce Collision-Constrained Flow Matching (CCFM), a novel framework that guarantees precise collision control through hard physical constraints. CCFM consists of three key components: (i) a heuristic collision selector that optimally identifies an adversarial agent and collision type via composite scoring; (ii) structured hard constraints that explicitly define four collision types (rear-end, side, cut-in, head-on) through contact point, heading, and severity requirements; and (iii) a collision-constrained flow matching sampler that enforces the constraints via Gauss-Newton manifold projection. CCFM achieves collision rate up to 46.4% on nuScenes and 83.1% on nuPlan, significantly outperforming baselines while preserving realistic driving behavior. By enabling controllable collision characteristics in safety-critical scenario generation, CCFM provides a reliable foundation for AV safety evaluation and sim-to-real crash data generation. The code and implementation details are available at https://github.com/KELISBU/CCFM.
Ke Li, Kaidi Liang, Yuxin Ding +3
Stony Brook University, Stony Brook, NY 11794, USA · Pennsylvania State University, University Park, PA 16802, USA
Validating Autonomous Vehicles (AVs) requires exposure to rare, safety-critical scenarios, infrequent in routine driving data. Existing benchmarks address this by generating synthetic conflicts or mapping accident descriptions to abstract road geometries, failing to capture the topological complexity of real-world crashes. We introduce TRACE , a pipeline that automates the reconstruction of NHTSA crash reports into high-fidelity CARLA simulations by (1) retrieving site-specific OpenStreetMap data to preserve exact road topology, (2) leveraging Large Language Models to infer vehicles' initial state from road geometry and pre-crash maneuvers, and (3) generating simulation trajectories from semi-structured report data. Using this pipeline, we curated a benchmark of 52 diverse accident scenarios covering varied collision types, road topologies, and pre-crash maneuvers, providing a challenging open source resource for testing AV systems against real-world failures.
Nahian Salsabil, Sebastian Elbaum
University of Virginia Charlottesville, Virginia, USA
Safety-critical scenarios are essential for the development of autonomous vehicles (AVs) but are rare in real-world driving data. While simulation offers a way to generate such scenarios, manually designed test cases lack scalability, and adversarial optimization often produces unrealistic behaviors. In this work, we introduce a conditional latent flow matching approach for scalable and realistic safety-critical scenario generation. Our method uses distribution matching to transform nominal scenes into safety-critical rollouts. Furthermore, we demonstrate that incorporating both simulation and real-world data enables our framework to efficiently generate diverse, data-driven scenarios. Experimental results highlight that our approach is able to more consistently and realistically generate novel safety-critical scenarios, making it a valuable tool for training and benchmarking AV systems.
Zimu Gong, Brian Zhaoning Zhang, Chris Zhang +2
University of Michigan-Ann Arbor · University of Waterloo · 1Waabi Innovation Inc +1