cs.ROMar 22, 2026
SaveCounterScene: Counterfactual Causal Reasoning in Generative World Models for Safety-Critical Closed-Loop Evaluation
Organizations: Tuojing Intelligence · King’s College London · Tsinghua University · The University of Hong Kong
Abstract
Generating safety-critical driving scenarios requires understanding why dangerous interactions arise, rather than merely forcing collisions. However, existing methods rely on heuristic adversarial agent selection and unstructured perturbations, lacking explicit modeling of interaction dependencies and thus exhibiting a realism--adversarial trade-off. We present CounterScene, a framework that endows closed-loop generative BEV world models with structured counterfactual reasoning for safety-critical scenario generation. Given a safe scene, CounterScene asks: what if the causally critical agent had behaved differently? To answer this, we introduce causal adversarial agent identification to identify the critical agent and classify conflict types, and develop a conflict-aware interactive world model in which a causal interaction graph is used to explicitly model dynamic inter-agent dependencies. Building on this structure, stage-adaptive counterfactual guidance performs minimal interventions on the identified agent, removing its spatial and temporal safety margins while allowing risk to emerge through natural interaction propagation. Extensive experiments on nuScenes demonstrate that CounterScene achieves the strongest adversarial effectiveness while maintaining superior trajectory realism across all horizons, improving long-horizon collision rate from 12.3% to 22.7% over the strongest baseline with better realism (ADE 1.88 vs.2.09). Notably, this advantage further widens over longer rollouts, and CounterScene generalizes zero-shot to nuPlan with state-of-the-art realism.
Figures & tables
Figure 1: Counterfactual causal reasoning for safety-critical generative BEV world model. Given an observed traffic scene where the critical agent waits and the ego vehicle passes safely, we ask the counterfactual question: What if the critical agent did not wait? By intervening on the agent trajectory, CounterScene constructs a counterfactual world that induces safety-critical interactions while preserving realistic traffic dynamics. This is achieved through causal adversarial agent identification, causal interaction graph modeling, and counterfactual guidance in diffusion.
| Method | Agent Selection | Interaction Modeling | Counterfactual Intervention |
| STRIVE Rempe et al. [2022] | None | Motion prior | ✗ Trajectory optimization |
| CTG Zhong et al. [2023b] | None | Implicit | ✗ Rule-guided diffusion |
| CTG++ Zhong et al. [2023a] | None | Implicit | ✗ Language-guided diffusion |
| SafeSim Chang et al. [2024] | Distance-based | None | ✗ Adversarial diffusion guidance |
| CCDiff Lin et al. [2024] | Manual/TTC-based | Compositional causal modeling | ✗ Structured diffusion guidance |
| CounterScene (Ours) | Causal Selection | Causal Interaction Graph (CIG) | ✓ Counterfactual Guided Diffusion |
Table 1: Comparison of representative safety-critical generative simulation methods. We compare key design dimensions, including adversarial agent selection, interaction modeling, and counterfactual intervention. Existing approaches typically rely on heuristic agent selection and lack explicit causal interaction modeling, which limits their ability to construct controllable high-risk interactions. In contrast, CounterScene combines causal adversarial agent identification, causal interaction graph modeling, and counterfactual diffusion guidance to generate realistic yet safety-critical closed-loop interactions.
Figure 2: Overview of CounterScene. The framework consists of four modules. (1) Causal adversarial agent selection identifies the most critical interacting agent using geometric conflict analysis and danger scoring. (2) Causal Interaction Graph (CIG) encodes conflict-aware interaction relationships among agents. (3) A diffusion-based interactive BEV world model with a SceneTransformer denoiser generates multi-agent trajectories. (4) Counterfactual guidance perturbs the adversarial agent trajectory during denoising to construct safety-critical interactions while maintaining realistic traffic dynamics.
| Method | 1–4s | 5–7s | 8–10s | ||||||||||||
| Realism | Adversarial | Realism | Adversarial | Realism | Adversarial | ||||||||||
| ADE | FDE | ORR | HBR | CR | ADE | FDE | ORR | HBR | CR | ADE | FDE | ORR | HBR | CR | |
| CTG | 0.526 | 1.182 | 0.2% | 1.7% | 0.0% | 1.553 | 3.768 | 0.3% | 1.5% | 2.3% | 2.480 | 6.143 | 0.2% | 1.4% | 2.0% |
| VAE | 0.638 | 1.458 | 0.4% | 0.4% | 1.3% | 1.887 | 4.550 | 0.6% | 0.2% | 7.7% | 3.086 | 7.433 | 1.0% | 0.2% | 13.3% |
| STRIVE | 0.502 | 1.171 | 0.3% | 0.1% | 1.3% | 1.565 | 4.018 | 0.5% | 0.1% | 7.7% | 2.722 | 7.060 | 0.8% | 0.1% | 15.3% |
| CTG++ | 0.548 | 1.259 | 0.2% | 1.3% | 0.8% | 1.730 | 4.416 | 0.3% | 1.2% | 2.3% | 2.963 | 7.525 | 0.2% | 1.3% | 3.7% |
Table 2: Main results across prediction horizons. Realism (ADE, FDE, ORR ) and adversarial effectiveness (HBR, CR ) averaged over short (1–4 s), mid (5–7 s), and long (8–10 s) horizons. CounterScene is the only method that simultaneously achieves superior realism and the highest collision rate, with the adversarial advantage widening at longer horizons. Best results are highlighted in bold and underline.
| Selection | ADE | FDE | ORR | HBR | CR |
| Random | 1.024 | 2.696 | 1.4% | 1.8% | 10.0% |
| CCDiff | 1.036 | 2.708 | 1.4% | 1.7% | 8.0% |
| SafeSim | 1.004 | 2.643 | 1.3% | 1.7% | 9.5% |
| Ours | 0.721 | 1.898 | 1.1% | 1.8% | 11.0% |
Table 3: Agent-selection comparison under identical backbone and guidance. CounterScene’s causal selection achieves the best realism and highest CR, while CCDiff’s TTC-based selection underperforms even random sampling. Best results are highlighted in bold and underline.
| # | Variant | ADE | FDE | ORR | HBR | CR |
| 1 | Full | 0.747 | 1.978 | 0.9% | 1.8% | 11.0% |
| 2 | No Jerk | 0.784 | 2.098 | 1.0% | 1.6% | 10.5% |
| 3 | No Conflict Aware | 0.768 | 2.052 | 1.0% | 1.6% | 9.0% |
| 4 | No Progressive | 0.775 | 2.072 | 1.0% | 1.6% | 9.0% |
| 5 | No Adaptive | 0.753 | 2.000 | 0.9% | 1.7% | 7.5% |
| 6 | Minimal | 0.798 | 2.157 | 1.2% | 1.4% | 6.5% |
Table 4: Ablation study (averaged across 3 s and 7 s). Adaptive temporal compression has the largest CR impact (31.8% relative drop); jerk regularization and progressive scheduling primarily preserve realism. The minimal variant degrades all metrics. Best results are highlighted in bold and underline.
| Method | 3 s | 5 s | 7 s | ||||||||||||
| Realism | Adversarial | Realism | Adversarial | Realism | Adversarial | ||||||||||
| ADE | FDE | ORR | HBR | CR | ADE | FDE | ORR | HBR | CR | ADE | FDE | ORR | HBR | CR | |
| CTG | 0.801 | 1.959 | 0.4% | 7.3% | 5.1% | 1.609 | 4.075 | 0.6% | 7.0% | 10.1% | 2.513 | 6.462 | 1.0% | 6.5% | 18.2% |
| STRIVE | 0.672 | 1.629 | 0.4% | 4.5% | 4.0% | 1.332 | 3.427 | 0.6% | 4.3% | 16.2% | 2.113 | 5.585 | 1.0% | 4.2% | 23.2% |
| CTG++ | 1.199 | 2.636 | 0.2% | 4.5% | 15.2% | 2.282 | 5.233 | 0.1% | 4.8% | 13.1% | 3.429 | 8.149 | 0.1% | 4.5% | 15.2% |
| CCDiff | 0.688 | 1.734 | 0.8% | 17.2% | 14.1% | 1.564 | 4.172 | 1.3% | 16.3% | 28.3% | 2.534 | 6.789 | 1.5% | 15.6% | 40.2% |
Table 5: Zero-shot cross-dataset transfer on nuPlan. All models are trained on nuScenes only. CounterScene achieves the best realism across all horizons. At short horizons, it generates realistic near-miss interactions (high HBR) rather than forced collisions; at longer horizons, these interactions naturally evolve into collisions, matching the highest CR while maintaining substantially better trajectory quality. Best results are bold , second best are underlined .
Figure 3: Qualitative comparison in a merging scenario (Scene 103, 10-second closed-loop rollout at 10 Hz). Factual (Original): In the recorded safe scene, the merging vehicle from the side road maintains a temporal safety margin by yielding, allowing the ego vehicle to pass the intersection safely. CounterScene (Ours): By causally identifying the yielding vehicle and progressively compressing its spatiotemporal margin, CounterScene generates a highly realistic safety-critical counterfactual: the adversarial vehicle aggressively cuts into the main road, forcing the ego vehicle into an emergency hard brake (matching our high HBR metric) to avoid a severe collision. Baselines: In contrast, baseline methods fail to capture the causal interaction structure. They either preserve the safe yielding behavior (failing to induce adversarial risk) or exhibit varying degrees of unstructured trajectory distortion, highlighting their inability to generate structurally valid and physically plausible critical interactions.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter Name | Value | Parameter Name | Value |
| Step length | 0.1 s | Map encoder | ResNet-18 |
| History steps | 31 | Map feature dim. | 128 |
| Prediction horizon | 52 | History feature dim. | 64 |
| Future horizon | 5.2 s | Attention heads | 16 |
| Learning rate | Temporal decoder layers | 2 | |
| Optimizer | Adam | Social decoder layers/block | 4 |
Table 6: Hyper-parameters used in CounterScene experiments.
| Early | Mid | Late | |
| Normalized progress | |||
| Stage multiplier | linear | linear | |
| Primary objective | coarse spatial attraction | stronger attraction and timing alignment | precise collision shaping |
| Arrival-time compression | off | active only for | on |
| Jerk regularization | on if enabled | on if enabled | on if enabled |
| Conflict-aware base weights | fixed by type | fixed by type | fixed by type |
Table 7: Progressive guidance configuration in CounterScene.
| Horizon | Metric | CTG | VAE | STRIVE | CTG++ | CCDiff | CounterScene |
|---|---|---|---|---|---|---|---|
| 1s | ADE | 0.178 | 0.206 | 0.170 | 0.185 | 0.125 | 0.089 |
| FDE | 0.348 | 0.415 | 0.326 | 0.361 | 0.247 | 0.174 | |
| OOR | 0.2% | 0.3% | 0.2% | 0.2% | 0.2% | 0.2% | |
| HBR | 1.9% | 0.5% | 0.2% | 1.5% | 1.4% | 1.3% | |
| CR | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | |
| 2s | ADE | 0.393 | 0.467 | 0.359 | 0.409 | 0.263 | 0.184 |
Table 8: Full per-horizon results on nuScenes (10 Hz). OOR, HBR, and CR are reported in percentage.
| Horizon | Setting | PG | CW | ATC | JR | ADE | FDE | ORR | CR | HBR |
| 3s | Full (Ours) | ✓ | ✓ | ✓ | ✓ | 0.338 | 0.814 | 0.5% | 4.0% | 1.7% |
| 3s | No Jerk | ✓ | ✓ | ✓ | 0.362 | 0.875 | 0.6% | 4.0% | 1.5% | |
| 3s | No Conflict Aware | ✓ | ✓ | ✓ | 0.350 | 0.845 | 0.6% | 3.0% | 1.5% | |
| 3s | No Progressive | ✓ | ✓ | ✓ | 0.355 | 0.855 | 0.6% | 3.0% | 1.5% | |
| 3s | No Adaptive | ✓ | ✓ | ✓ | 0.340 | 0.820 | 0.5% | 2.0% | 1.6% | |
| 3s | Minimal | 0.372 | 0.895 | 0.7% | 2.0% | 1.3% |
Table 9: Ablation results on nuScenes at 3s, 7s, and 10s (10 Hz). PG: progressive guidance, CW: conflict-aware weighting, ATC: adaptive arrival-time compression, JR: jerk regularization.
Figure 4: Qualitative comparison of a merging collision in Scene 110. Factual (Original): In the recorded scene, the merging vehicle from the side road actively maintains safety by yielding, providing a sufficient spatiotemporal margin for the ego vehicle to pass. CounterScene (Ours): By accurately identifying the yielding vehicle as the causal variable and surgically stripping its safety margin, CounterScene generates a highly realistic collision. The adversarial agent aggressively merges into the main road without yielding, resulting in a physically plausible crash while the rest of the scene evolves naturally. Baselines: Lacking structural causal reasoning, baseline methods completely fail to generate meaningful adversarial pressure. They either passively preserve the original yielding behavior (resulting in zero risk) or introduce unstructured trajectory noise that fails to culminate in a realistic collision.
Figure 5: Qualitative comparison of a rear-end collision in Scene 905. Factual (Original): In the recorded safe scene, the trailing vehicle actively maintains a safe longitudinal distance, preserving a temporal margin that prevents a collision as the ego vehicle navigates the lane. CounterScene (Ours): By causally identifying the trailing vehicle and surgically compressing its longitudinal safety margin, our framework generates a highly realistic rear-end crash. The adversarial agent aggressively accelerates into the ego vehicle, demonstrating how targeted temporal compression naturally evolves into a high-risk scenario without distorting the surrounding traffic dynamics. Baselines: Lacking structured causal reasoning, baseline methods fail to capture the nature of this longitudinal interaction. They either maintain the original safe following distance (generating zero adversarial pressure) or apply unstructured perturbations that cause the vehicle to veer off-course, ultimately failing to induce a physically plausible rear-end collision.
Figure 6: Qualitative comparison of a head-on collision induced by a subtle trajectory shift in Scene 913. Factual (Original): In the recorded scene, the vehicle on the other path strictly maintains its nominal trajectory, preserving a safe spatial margin as the ego vehicle approaches. CounterScene (Ours): By identifying this vehicle as the causal variable, CounterScene applies a minimal, targeted spatial intervention. A slight, physically plausible deviation in the steering angle surgically strips the safety margin, seamlessly redirecting the vehicle into the ego’s path and culminating in a severe head-on crash. Baselines: Lacking structured causal reasoning, baseline methods overlook this subtle vulnerability. They either passively maintain the original safe trajectory (generating zero risk) or apply unstructured noise that results in unrealistic meandering, failing to orchestrate a precise head-on collision.
Figure 7: Qualitative comparison of a high-speed rear-end collision in Scene 915. Factual (Original): In the recorded scene, the trailing vehicle decelerates appropriately, actively maintaining a safe following distance and temporal margin behind the ego vehicle. CounterScene (Ours): By identifying the trailing vehicle as the causal variable, CounterScene specifically suppresses its natural braking behavior. The adversarial agent maintains a high velocity and fails to decelerate in time, effectively compressing the temporal safety margin to zero. This targeted temporal intervention culminates in a severe, physically plausible rear-end crash. Baselines: Lacking structured causal reasoning, baseline methods fail to orchestrate this direct velocity-based conflict. They either passively execute the original safe deceleration (generating zero adversarial pressure) or apply unstructured noise that disrupts the vehicle’s heading without achieving a precise longitudinal impact.
More in this topic
Autonomous Driving Scenario Generation
Autonomous driving scenario generation creates realistic, controllable traffic situations for training and testing vehicles, especially in rare and safety-critical cases.
See all 111 papers