Generating safety-critical driving scenarios is crucial for evaluating and improving autonomous driving systems, but long-tail risky situations are rarely observed in real-world data and difficult to specify through manual scenario design. Existing generative approaches typically treat risk as an after-the-fact label and struggle to maintain geometric consistency in multi-view driving scenes. We present RiskMV-DPO, a general and systematic pipeline for physically-informed, risk-controllable multi-view scenario generation. By integrating target risk levels with physically-grounded risk modeling, we synthesize diverse and high-stakes dynamic trajectories that serve as explicit geometric anchors for a diffusion-based video generator. To ensure spatial-temporal coherence and geometric fidelity, we introduce a geometry-appearance alignment module and a region-aware direct preference optimization (RA-DPO) strategy with motion-aware masking to focus learning on localized dynamic regions. Experiments on the nuScenes dataset show that RiskMV-DPO can freely generate a wide spectrum of diverse scenarios while maintaining visual quality, improving 3D detection mAP from 18.17 to 30.50 and reducing FID to 15.70. Our work shifts the role of world models from passive environment prediction to proactive, risk-controllable synthesis, providing a scalable toolchain for the development of embodied intelligence.
Figures & tables
Figure 1: Examples of mined potential risks in typical left-turn scenarios using the proposed per-frame risk quantification. The numbers indicate the risk coefficient of the ego vehicle at that location. (a) Unprotected left turn at an unsignalized intersection with an oncoming straight-moving vehicle. (b) Parallel left turns with tight lateral clearance. (c) Left turn followed by a nearby parking-lot exit that creates a local blind spot.
Figure 2: Risk-conditioned motion generation. The target risk r is encoded by an MLP and injected into GENAD via FiLM-modulated BEV features. The model predicts trajectories and 3D boxes for the ego vehicle and surrounding agents, followed by risk-aware selection to obtain the final motion set.
Figure 3: Overview of the proposed RiskMV-DPO framework. Given multi-view observations and scenario context, trajectories and 3D bounding boxes generated by the risk control module at a specified risk level are used as structured motion conditions. A multimodal encoder embeds view tokens and motion cues, which are then injected into a diffusion backbone composed of spatial and temporal STDiT3 blocks. Region-aware DPO further aligns the generation toward localized dynamic regions, producing temporally coherent multi-view driving videos consistent with the specified risk-conditioned motion.
Method
FID ↓
FVD ↓
mAP ↑
DriveDreamer-2
25.00
105.10
–
Panacea
16.96
139.00
–
MagicDriveV2
20.91
94.84
18.17
RiskMV-DPO (ours)
15.70
87.65
30.50
Table 1: Comparison with representative driving video generators on nuScenes.
Configuration
FID ↓
FVD ↓
MV-SSIM ↑
Depth AbsRel ↓
Fg-FID ↓
DINO-FID ↓
(A) Baseline
20.91
94.84
0.812
0.250
35.0
28.4
(B) + Random Mask DPO
20.91
93.01
0.818
0.243
33.6
27.9
(C) + Motion-Aware Mask
19.42
91.67
0.825
0.236
32.1
27.4
(D) + 3D MV Mask
19.33
89.51
0.841
0.228
31.0
26.8
(E) + VGGT-GAM (no Align)
16.79
88.70
0.848
0.205
30.2
26.3
(F) RiskMV-DPO (ours)
15.70
87.65
0.856
0.204
29.5
25.8
Table 2: Ablation study on nuScenes.
Figure 4: Qualitative examples generated by RiskMV-DPO at different target risk quantiles. The quantiles indicate the relative position of the induced risk coefficient among all generated scenarios. (a) Higher risk: an unsignalized intersection with a fast left-turning heavy truck at very close distance under partial occlusion. (b) Medium risk: the same interaction with slower turning and larger clearance, reducing near-collision risk. (c) Lower risk: a common straight-driving scenario with no immediate conflict.
Autonomous driving scenario generation creates realistic, controllable traffic situations for training and testing vehicles, especially in rare and safety-critical cases.
Generating high-fidelity and controllable synthetic data is critical for advancing end-to-end autonomous driving, particularly for addressing the long tail of rare safety-critical scenarios. Existing occupancy-guided methods typically rely on shallow conditioning mechanisms and reference-frame-dependent video synthesis, which limits fine-grained controllability from arbitrary BEV layouts and restricts their applicability for scalable simulation. In this paper, we propose AnyScene, a unified occupancy-centric framework for driving scene generation. AnyScene generates semantic occupancy sequences from BEV layouts through a Spatial-Temporal Occupancy Diffusion Transformer that jointly tokenizes BEV and occupancy features in an autoregressive manner. This design enables precise controllability from cross-dataset and user-defined BEV inputs while naturally supporting long-horizon generation. Building upon the generated occupancy, a Geometry-Grounded View Expansion module treats occupancy as the canonical spatial representation and synthesizes temporally consistent multi-view driving videos in a reference-free and autoregressive fashion, supporting flexible camera configurations at inference time. Extensive experiments demonstrate that AnyScene achieves state-of-the-art performance in both occupancy and video generation. It exhibits strong generalization to unseen and customized layouts, and provides measurable benefits for downstream tasks such as sparse-view 3D reconstruction.
Haiming Zhang, Junfei Zhou, Feng Jiang +6
Li Auto · Southwest Jiaotong University · Tsinghua University
Autonomous driving systems require comprehensive evaluation in safety-critical scenarios to ensure safety and robustness. However, such scenarios are rare and difficult to collect from real-world driving data, necessitating simulation-based synthesis. Yet, existing methods often exhibit limitations in both controllability and realism. From a capability perspective, LLMs excel at controllable generation guided by natural language instructions, while diffusion models are better suited for producing trajectories consistent with realistic driving distributions. Leveraging their complementary strengths, we propose AnchorDrive, a two-stage safety-critical scenario generation framework. In the first stage, we deploy an LLM as a driver agent within a closed-loop simulation, which reasons and iteratively outputs control commands under natural language constraints; a plan assessor reviews these commands and provides corrective feedback, enabling semantically controllable scenario generation. In the second stage, the LLM extracts key anchor points from the first-stage trajectories as guidance objectives, which jointly with other guidance terms steer the diffusion model to regenerate complete trajectories with improved realism while preserving user-specified intent. Experiments on the highD dataset demonstrate that AnchorDrive achieves superior overall performance in criticality, realism, and controllability, validating its effectiveness for generating controllable and realistic safety-critical scenarios.
Zhulin Jiang, Zetao Li, Cheng Wang +2
School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen, China.
Safety-critical traffic scenario generation is essential for evaluating autonomous driving systems under rare but high-risk interactions. Existing diffusion-based methods offer strong controllability in closed-loop generation, but their iterative denoising process is computationally expensive and may accumulate sampling and guidance errors over long rollouts, causing unrealistic motion artifacts such as jitter, abnormal acceleration, and off-road behavior. To address these issues, we propose RiskFlow, a closed-loop safety-critical multi-agent traffic generation framework that formulates future trajectory generation as transport in the action space. Instead of relying on iterative denoising, RiskFlow learns an average velocity field over a finite interval to transform Gaussian action sequences into future acceleration and yaw-rate commands with a single forward pass, using a JVP-based objective for efficient and stable training. At test time, RiskFlow applies output-space guidance to the generated actions, steering selected critical agents toward risky interactions while regularizing off-road behavior, and reconstructs physically feasible trajectories through vehicle dynamics. Experiments on nuScenes with tbsim closed-loop evaluation show that RiskFlow achieves a strong adversariality-realism trade-off across multi-agent and long-horizon settings. Compared with representative baselines, RiskFlow consistently improves realism while maintaining competitive safety-critical generation capability, and substantially reduces inference time for evaluation.
Qi Lan, Yining Tang, Yu Shen +4
College of Mechanical and Vehicle Engineering Chongqing University, Chongqing, China