cs.ROMar 22, 2026

CounterScene: Counterfactual Causal Reasoning in Generative World Models for Safety-Critical Closed-Loop Evaluation

Authors: Bowen Jing, Ruiyang Hao, Weitao Zhou, Haibao Yu

Organizations: Tuojing Intelligence · King’s College London · Tsinghua University · The University of Hong Kong

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Generating safety-critical driving scenarios requires understanding why dangerous interactions arise, rather than merely forcing collisions. However, existing methods rely on heuristic adversarial agent selection and unstructured perturbations, lacking explicit modeling of interaction dependencies and thus exhibiting a realism--adversarial trade-off. We present CounterScene, a framework that endows closed-loop generative BEV world models with structured counterfactual reasoning for safety-critical scenario generation. Given a safe scene, CounterScene asks: what if the causally critical agent had behaved differently? To answer this, we introduce causal adversarial agent identification to identify the critical agent and classify conflict types, and develop a conflict-aware interactive world model in which a causal interaction graph is used to explicitly model dynamic inter-agent dependencies. Building on this structure, stage-adaptive counterfactual guidance performs minimal interventions on the identified agent, removing its spatial and temporal safety margins while allowing risk to emerge through natural interaction propagation. Extensive experiments on nuScenes demonstrate that CounterScene achieves the strongest adversarial effectiveness while maintaining superior trajectory realism across all horizons, improving long-horizon collision rate from 12.3% to 22.7% over the strongest baseline with better realism (ADE 1.88 vs.2.09). Notably, this advantage further widens over longer rollouts, and CounterScene generalizes zero-shot to nuPlan with state-of-the-art realism.

Figures & tables

Figure 1: Counterfactual causal reasoning for safety-critical generative BEV world model. Given an observed traffic scene where the critical agent waits and the ego vehicle passes safely, we ask the counterfactual question: What if the critical agent did not wait? By intervening on the agent trajectory, CounterScene constructs a counterfactual world that induces safety-critical interactions while preserving realistic traffic dynamics. This is achieved through causal adversarial agent identification, causal interaction graph modeling, and counterfactual guidance in diffusion.
MethodAgent SelectionInteraction ModelingCounterfactual Intervention
STRIVE Rempe et al. [2022]NoneMotion prior✗ Trajectory optimization
CTG Zhong et al. [2023b]NoneImplicit✗ Rule-guided diffusion
CTG++ Zhong et al. [2023a]NoneImplicit✗ Language-guided diffusion
SafeSim Chang et al. [2024]Distance-basedNone✗ Adversarial diffusion guidance
CCDiff Lin et al. [2024]Manual/TTC-basedCompositional causal modeling✗ Structured diffusion guidance
CounterScene (Ours)Causal SelectionCausal Interaction Graph (CIG)✓ Counterfactual Guided Diffusion
Table 1: Comparison of representative safety-critical generative simulation methods. We compare key design dimensions, including adversarial agent selection, interaction modeling, and counterfactual intervention. Existing approaches typically rely on heuristic agent selection and lack explicit causal interaction modeling, which limits their ability to construct controllable high-risk interactions. In contrast, CounterScene combines causal adversarial agent identification, causal interaction graph modeling, and counterfactual diffusion guidance to generate realistic yet safety-critical closed-loop interactions.
Figure 2: Overview of CounterScene. The framework consists of four modules. (1) Causal adversarial agent selection identifies the most critical interacting agent using geometric conflict analysis and danger scoring. (2) Causal Interaction Graph (CIG) encodes conflict-aware interaction relationships among agents. (3) A diffusion-based interactive BEV world model with a SceneTransformer denoiser generates multi-agent trajectories. (4) Counterfactual guidance perturbs the adversarial agent trajectory during denoising to construct safety-critical interactions while maintaining realistic traffic dynamics.
Method1–4s5–7s8–10s
RealismAdversarialRealismAdversarialRealismAdversarial
ADE ↓\downarrowFDE ↓\downarrowORR ↓\downarrowHBR ↑\uparrowCR ↑\uparrowADE ↓\downarrowFDE ↓\downarrowORR ↓\downarrowHBR ↑\uparrowCR ↑\uparrowADE ↓\downarrowFDE ↓\downarrowORR ↓\downarrowHBR ↑\uparrowCR ↑\uparrow
CTG0.5261.1820.2%1.7%0.0%1.5533.7680.3%1.5%2.3%2.4806.1430.2%1.4%2.0%
VAE0.6381.4580.4%0.4%1.3%1.8874.5500.6%0.2%7.7%3.0867.4331.0%0.2%13.3%
STRIVE0.5021.1710.3%0.1%1.3%1.5654.0180.5%0.1%7.7%2.7227.0600.8%0.1%15.3%
CTG++0.5481.2590.2%1.3%0.8%1.7304.4160.3%1.2%2.3%2.9637.5250.2%1.3%3.7%
Table 2: Main results across prediction horizons. Realism (ADE, FDE, ORR ↓\downarrow ) and adversarial effectiveness (HBR, CR ↑\uparrow ) averaged over short (1–4 s), mid (5–7 s), and long (8–10 s) horizons. CounterScene is the only method that simultaneously achieves superior realism and the highest collision rate, with the adversarial advantage widening at longer horizons. Best results are highlighted in bold and underline.
SelectionADE ↓\downarrowFDE ↓\downarrowORR ↓\downarrowHBR ↑\uparrowCR ↑\uparrow
Random1.0242.6961.4%1.8%10.0%
CCDiff1.0362.7081.4%1.7%8.0%
SafeSim1.0042.6431.3%1.7%9.5%
Ours0.7211.8981.1%1.8%11.0%
Table 3: Agent-selection comparison under identical backbone and guidance. CounterScene’s causal selection achieves the best realism and highest CR, while CCDiff’s TTC-based selection underperforms even random sampling. Best results are highlighted in bold and underline.
#VariantADE ↓\downarrowFDE ↓\downarrowORR ↓\downarrowHBR ↑\uparrowCR ↑\uparrow
1Full0.7471.9780.9%1.8%11.0%
2No Jerk0.7842.0981.0%1.6%10.5%
3No Conflict Aware0.7682.0521.0%1.6%9.0%
4No Progressive0.7752.0721.0%1.6%9.0%
5No Adaptive0.7532.0000.9%1.7%7.5%
6Minimal0.7982.1571.2%1.4%6.5%
Table 4: Ablation study (averaged across 3 s and 7 s). Adaptive temporal compression has the largest CR impact (31.8% relative drop); jerk regularization and progressive scheduling primarily preserve realism. The minimal variant degrades all metrics. Best results are highlighted in bold and underline.
Method3 s5 s7 s
RealismAdversarialRealismAdversarialRealismAdversarial
ADE ↓\downarrowFDE ↓\downarrowORR ↓\downarrowHBR ↑\uparrowCR ↑\uparrowADE ↓\downarrowFDE ↓\downarrowORR ↓\downarrowHBR ↑\uparrowCR ↑\uparrowADE ↓\downarrowFDE ↓\downarrowORR ↓\downarrowHBR ↑\uparrowCR ↑\uparrow
CTG0.8011.9590.4%7.3%5.1%1.6094.0750.6%7.0%10.1%2.5136.4621.0%6.5%18.2%
STRIVE0.6721.6290.4%4.5%4.0%1.3323.4270.6%4.3%16.2%2.1135.5851.0%4.2%23.2%
CTG++1.1992.6360.2%4.5%15.2%2.2825.2330.1%4.8%13.1%3.4298.1490.1%4.5%15.2%
CCDiff0.6881.7340.8%17.2%14.1%1.5644.1721.3%16.3%28.3%2.5346.7891.5%15.6%40.2%
Table 5: Zero-shot cross-dataset transfer on nuPlan. All models are trained on nuScenes only. CounterScene achieves the best realism across all horizons. At short horizons, it generates realistic near-miss interactions (high HBR) rather than forced collisions; at longer horizons, these interactions naturally evolve into collisions, matching the highest CR while maintaining substantially better trajectory quality. Best results are bold , second best are underlined .
Figure 3: Qualitative comparison in a merging scenario (Scene 103, 10-second closed-loop rollout at 10 Hz). Factual (Original): In the recorded safe scene, the merging vehicle from the side road maintains a temporal safety margin by yielding, allowing the ego vehicle to pass the intersection safely. CounterScene (Ours): By causally identifying the yielding vehicle and progressively compressing its spatiotemporal margin, CounterScene generates a highly realistic safety-critical counterfactual: the adversarial vehicle aggressively cuts into the main road, forcing the ego vehicle into an emergency hard brake (matching our high HBR metric) to avoid a severe collision. Baselines: In contrast, baseline methods fail to capture the causal interaction structure. They either preserve the safe yielding behavior (failing to induce adversarial risk) or exhibit varying degrees of unstructured trajectory distortion, highlighting their inability to generate structurally valid and physically plausible critical interactions.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter NameValueParameter NameValue
Step length0.1 sMap encoderResNet-18
History steps31Map feature dim.128
Prediction horizon52History feature dim.64
Future horizon5.2 sAttention heads16
Learning rate1×10−41\times 10^{-4}Temporal decoder layers2
OptimizerAdamSocial decoder layers/block4
Appendix
Table 6: Hyper-parameters used in CounterScene experiments.
 EarlyMidLate
Normalized progress pp[0,0.3)[0,0.3)[0.3,0.7)[0.3,0.7)[0.7,1.0][0.7,1.0]
Stage multiplier m(p)m(p)0.20.2linear 0.2→1.50.2\rightarrow 1.5linear 1.5→3.01.5\rightarrow 3.0
Primary objectivecoarse spatial attractionstronger attraction and timing alignmentprecise collision shaping
Arrival-time compressionoffactive only for p≥0.5p\geq 0.5on
Jerk regularizationon if enabledon if enabledon if enabled
Conflict-aware base weightsfixed by typefixed by typefixed by type
Appendix
Table 7: Progressive guidance configuration in CounterScene.
HorizonMetricCTGVAESTRIVECTG++CCDiffCounterScene
1sADE0.1780.2060.1700.1850.1250.089
 FDE0.3480.4150.3260.3610.2470.174
 OOR0.2%0.3%0.2%0.2%0.2%0.2%
 HBR1.9%0.5%0.2%1.5%1.4%1.3%
 CR0.0%0.0%0.0%0.0%0.0%0.0%
2sADE0.3930.4670.3590.4090.2630.184
Appendix
Table 8: Full per-horizon results on nuScenes (10 Hz). OOR, HBR, and CR are reported in percentage.
HorizonSettingPGCWATCJRADE ↓\downarrowFDE ↓\downarrowORR ↓\downarrowCR ↑\uparrowHBR ↑\uparrow
3sFull (Ours)✓✓✓✓0.3380.8140.5%4.0%1.7%
3sNo Jerk✓✓✓ 0.3620.8750.6%4.0%1.5%
3sNo Conflict Aware✓ ✓✓0.3500.8450.6%3.0%1.5%
3sNo Progressive ✓✓✓0.3550.8550.6%3.0%1.5%
3sNo Adaptive✓✓ ✓0.3400.8200.5%2.0%1.6%
3sMinimal    0.3720.8950.7%2.0%1.3%
Appendix
Table 9: Ablation results on nuScenes at 3s, 7s, and 10s (10 Hz). PG: progressive guidance, CW: conflict-aware weighting, ATC: adaptive arrival-time compression, JR: jerk regularization.
Figure 4: Qualitative comparison of a merging collision in Scene 110. Factual (Original): In the recorded scene, the merging vehicle from the side road actively maintains safety by yielding, providing a sufficient spatiotemporal margin for the ego vehicle to pass. CounterScene (Ours): By accurately identifying the yielding vehicle as the causal variable and surgically stripping its safety margin, CounterScene generates a highly realistic collision. The adversarial agent aggressively merges into the main road without yielding, resulting in a physically plausible crash while the rest of the scene evolves naturally. Baselines: Lacking structural causal reasoning, baseline methods completely fail to generate meaningful adversarial pressure. They either passively preserve the original yielding behavior (resulting in zero risk) or introduce unstructured trajectory noise that fails to culminate in a realistic collision.
Figure 5: Qualitative comparison of a rear-end collision in Scene 905. Factual (Original): In the recorded safe scene, the trailing vehicle actively maintains a safe longitudinal distance, preserving a temporal margin that prevents a collision as the ego vehicle navigates the lane. CounterScene (Ours): By causally identifying the trailing vehicle and surgically compressing its longitudinal safety margin, our framework generates a highly realistic rear-end crash. The adversarial agent aggressively accelerates into the ego vehicle, demonstrating how targeted temporal compression naturally evolves into a high-risk scenario without distorting the surrounding traffic dynamics. Baselines: Lacking structured causal reasoning, baseline methods fail to capture the nature of this longitudinal interaction. They either maintain the original safe following distance (generating zero adversarial pressure) or apply unstructured perturbations that cause the vehicle to veer off-course, ultimately failing to induce a physically plausible rear-end collision.
Figure 6: Qualitative comparison of a head-on collision induced by a subtle trajectory shift in Scene 913. Factual (Original): In the recorded scene, the vehicle on the other path strictly maintains its nominal trajectory, preserving a safe spatial margin as the ego vehicle approaches. CounterScene (Ours): By identifying this vehicle as the causal variable, CounterScene applies a minimal, targeted spatial intervention. A slight, physically plausible deviation in the steering angle surgically strips the safety margin, seamlessly redirecting the vehicle into the ego’s path and culminating in a severe head-on crash. Baselines: Lacking structured causal reasoning, baseline methods overlook this subtle vulnerability. They either passively maintain the original safe trajectory (generating zero risk) or apply unstructured noise that results in unrealistic meandering, failing to orchestrate a precise head-on collision.
Figure 7: Qualitative comparison of a high-speed rear-end collision in Scene 915. Factual (Original): In the recorded scene, the trailing vehicle decelerates appropriately, actively maintaining a safe following distance and temporal margin behind the ego vehicle. CounterScene (Ours): By identifying the trailing vehicle as the causal variable, CounterScene specifically suppresses its natural braking behavior. The adversarial agent maintains a high velocity and fails to decelerate in time, effectively compressing the temporal safety margin to zero. This targeted temporal intervention culminates in a severe, physically plausible rear-end crash. Baselines: Lacking structured causal reasoning, baseline methods fail to orchestrate this direct velocity-based conflict. They either passively execute the original safe deceleration (generating zero adversarial pressure) or apply unstructured noise that disrupts the vehicle’s heading without achieving a precise longitudinal impact.
More in this topic

Autonomous Driving Scenario Generation

Autonomous driving scenario generation creates realistic, controllable traffic situations for training and testing vehicles, especially in rare and safety-critical cases.

See all 111 papers

Explore similar work

CardsList
Sep 17, 2026

Safety-Critical Scenarios Emerge from Initial Scenes

Safety-critical driving scenario generation has largely focused on manipulating the behavior of surrounding agents while starting from an initial scene from driving data. This assumption can limit the space of discoverable failures, since driving data can provide little opportunity for meaningful interaction. For example, in the Waymo Open Motion Dataset, 20.44% of recorded slices feature a stationary ego vehicle that never moves, and 30.39% of initial frames contain no nearby traffic participants within 10 meters. We instead study safety-critical scenario generation as an initialization problem: given agnostic black-box driving policies, we learn to generate realistic initial scenes that are more likely to evolve into critical interactions. We propose AdvScene, a conditional latent diffusion model that is trained in two stages. Starting from pretraining on naturalistic driving data, we post-train the adversarial-agent generation branch using reinforcement learning with feedback from closed-loop simulator rollouts. Conditioning on ego driving displacement prevents the ego from remaining static, and RL finetuning induces criticality directly with non-differentiable safety-critical metrics. Experiments on the Waymo dataset across 12 combinations of ego and traffic policies show that our AdvScene substantially increases the rate of ego-fault collision events and TTC<3s events.

Yin Wu, Jiarong Wei, Carl Esselborn +4

Adversarial Scenario GenerationAutonomous Driving SimulationAutonomous Driving Scenario GenerationDiffusion Models
AskPDF
Jun 4, 2026

RiskFlow: Fast and Faithful Safety-Critical Traffic Scenario Generation

Safety-critical traffic scenario generation is essential for evaluating autonomous driving systems under rare but high-risk interactions. Existing diffusion-based methods offer strong controllability in closed-loop generation, but their iterative denoising process is computationally expensive and may accumulate sampling and guidance errors over long rollouts, causing unrealistic motion artifacts such as jitter, abnormal acceleration, and off-road behavior. To address these issues, we propose RiskFlow, a closed-loop safety-critical multi-agent traffic generation framework that formulates future trajectory generation as transport in the action space. Instead of relying on iterative denoising, RiskFlow learns an average velocity field over a finite interval to transform Gaussian action sequences into future acceleration and yaw-rate commands with a single forward pass, using a JVP-based objective for efficient and stable training. At test time, RiskFlow applies output-space guidance to the generated actions, steering selected critical agents toward risky interactions while regularizing off-road behavior, and reconstructs physically feasible trajectories through vehicle dynamics. Experiments on nuScenes with tbsim closed-loop evaluation show that RiskFlow achieves a strong adversariality-realism trade-off across multi-agent and long-horizon settings. Compared with representative baselines, RiskFlow consistently improves realism while maintaining competitive safety-critical generation capability, and substantially reduces inference time for evaluation.

Qi Lan, Yining Tang, Yu Shen +4

College of Mechanical and Vehicle Engineering Chongqing University, Chongqing, China

Flow MatchingAutonomous Driving Safety EvaluationGenerative ModelingAutonomous Driving SimulationAutonomous Driving Scenario Generation
AskPDF
May 15, 2026

PCASim: Promptable Closed-loop Adversarial Simulation for Urban Traffic Environment

Real-world autonomous driving, particularly in urban environments with numerous corner cases, requires rigorous testing to ensure product safety and robustness. However, few studies have explored integrating adversarial scenario generation with the training of safety agents in closed-loop testing, enabling efficient co-evolution and mutual enhancement of both. To address this challenge, an adversarial behavior knowledge repository is constructed by applying rule-based filtering to an open-source dataset, combined with knowledge retrieval modules tailored for simulation environments. A large language model (LLM) is employed to integrate knowledge-, data-, and adversarial-driven approaches, generating safety-critical traffic scenarios customized to user needs. Additionally, while evaluating the generated scenarios, we employ reinforcement learning models to train the behaviors of different types of vehicles, thereby enriching scenario diversity beyond existing datasets while preserving realism. Experimental results demonstrate that the proposed framework improves the accuracy of domain-specific language generation by 12%. Moreover, the success rate of newly generated scenario transformations increases by 8%, while obstacle-avoidance capability is enhanced by 30%. For the complete manuscript, please refer to: https://zhenhaooo.github.io/PCASim.github.io/

Chuancheng Zhang, Zhenhao Wang, Kaizheng Li +3

Shenzhen Research Institute of Shandong University, Shenzhen, 518057, Guangdong, China · School of Airspace Science and Engineering, Shandong University, 264209 Weihai, China. · School of Computer Science and Technology, Shandong University of Finance and Economics, 250014, Jinan, China.

Autonomous Driving Safety EvaluationSynthetic Data GenerationAdversarial Scenario GenerationAutonomous DrivingAutonomous Driving SimulationAutonomous Driving Scenario Generation
AskPDF