A major challenge in autonomous driving is the "long tail" of safety-critical edge cases, which often emerge from unusual combinations of common traffic elements. Synthesizing these scenarios is crucial, yet current controllable generative models provide incomplete or entangled guidance, preventing the independent manipulation of scene structure, object identity, and ego actions. We introduce CompoSIA, a compositional driving video simulator that disentangles these traffic factors, enabling fine-grained control over diverse adversarial driving scenarios. To support controllable identity replacement of scene elements, we propose a noise-level identity injection, allowing pose-agnostic identity generation across diverse element poses, all from a single reference image. Furthermore, a hierarchical dual-branch action control mechanism is introduced to improve action controllability. Such disentangled control enables adversarial scenario synthesis-systematically combining safe elements into dangerous configurations that entangled generators cannot produce. Extensive comparisons demonstrate superior controllable generation quality over state-of-the-art baselines, with a 17% improvement in FVD for identity editing and reductions of 30% and 47% in rotation and translation errors for action control. Furthermore, downstream stress-testing reveals substantial planner failures: across editing modalities, the average collision rate of 3s increases by 173%.
Figures & tables
Type
Method
Control Modes
Structure
Element ID
Scene ID
Ego-Action
General Controllable Generation
ReCamMaster [ 2 ]
✓
Wan-Move [ 9 ]
✓
✓
✓
LoRA-Edit [ 12 ]
✓
✓
TTM [ 55 ]
✓
✓
✓
Driving Model
Vista [ 16 ]
✓
✓
Table 1: Comparison of control capabilities across general and driving-specific models. Through disentangled conditioning, CompoSIA enables more comprehensive and fine-grained control and editing over adversarial scenario generation.
Figure 2: Overview of the CompoSIA framework. During training (top), structure, identity, and ego-action signals are explicitly decomposed and injected into a Flow Matching–based DiT backbone in a disentangled yet compositional manner. During sampling (bottom), different combinations of conditions enable controllable driving video generation, supporting multiple editing applications.
Figure 3: Design of hierarchical dual-branch action Conditioning. Local residual modulation accelerates early convergence, while global PRoPE embedding improves overall accuracy.
Figure 4: Identity editing for the first frame. We preserve the background to anchor scene identity while editing the foreground. If the reference area is smaller than the original area, the intermediate region is treated as an inpainting area during denoising.
Table 2: Quantitative comparison of different methods. We evaluate multi-view scene following, identity control, and action control tasks.
Figure 6: Comparison on identity control . CompoSIA enables precise identity injection across diverse element poses, categories, and in-the-wild reference images.
Figure 7: Comparison on action control . CompoSIA faithfully executes diverse action commands with precise frame-wise controllability.
Figure 8: Ablation studies on structure, action, and identity injection. (a) Structure ablation.(b) Action ablation.(c) Effect of Tid .(d) Identity injection improves cross-frame consistency under complex illumination.
Figure 10: More examples of identity editing. Given a single in-the-wild reference image of the target object, our method replaces the identity of a scene element while preserving the original scene structure and motion. Despite large variations in element pose, categories, and appearance between the reference image and the generated sequence, the edited object is rendered with temporally consistent and realistic identity.
Figure 11: Additional results on compositional structure-and-action control. We present more examples that jointly modify scene structure and ego action, including cut-in vehicles, ego stopping, sudden braking, lane changes, and skidding events. The generated videos remain temporally coherent while following the intended compositional interventions across diverse traffic scenes. The instruction column is included only for human-readable description of the applied edits; no real text prompt is used during the editing process.
Generating high-fidelity and controllable synthetic data is critical for advancing end-to-end autonomous driving, particularly for addressing the long tail of rare safety-critical scenarios. Existing occupancy-guided methods typically rely on shallow conditioning mechanisms and reference-frame-dependent video synthesis, which limits fine-grained controllability from arbitrary BEV layouts and restricts their applicability for scalable simulation. In this paper, we propose AnyScene, a unified occupancy-centric framework for driving scene generation. AnyScene generates semantic occupancy sequences from BEV layouts through a Spatial-Temporal Occupancy Diffusion Transformer that jointly tokenizes BEV and occupancy features in an autoregressive manner. This design enables precise controllability from cross-dataset and user-defined BEV inputs while naturally supporting long-horizon generation. Building upon the generated occupancy, a Geometry-Grounded View Expansion module treats occupancy as the canonical spatial representation and synthesizes temporally consistent multi-view driving videos in a reference-free and autoregressive fashion, supporting flexible camera configurations at inference time. Extensive experiments demonstrate that AnyScene achieves state-of-the-art performance in both occupancy and video generation. It exhibits strong generalization to unseen and customized layouts, and provides measurable benefits for downstream tasks such as sparse-view 3D reconstruction.
Haiming Zhang, Junfei Zhou, Feng Jiang +6
Li Auto · Southwest Jiaotong University · Tsinghua University
Controllable driving scene generation is critical for realistic and scalable autonomous driving simulation, yet existing approaches struggle to jointly achieve photorealism and precise control. We introduce HorizonForge, a unified framework that reconstructs scenes as editable Gaussian Splats and Meshes, enabling fine-grained 3D manipulation and language-driven vehicle insertion. Edits are rendered through a noise-aware video diffusion process that enforces spatial and temporal consistency, producing diverse scene variations in a single feed-forward pass without per-trajectory optimization. To standardize evaluation, we further propose HorizonSuite, a comprehensive benchmark spanning ego- and agent-level editing tasks such as trajectory modifications and object manipulation. Extensive experiments show that Gaussian-Mesh representation delivers substantially higher fidelity than alternative 3D representations, and that temporal priors from video diffusion are essential for coherent synthesis. Combining these findings, HorizonForge establishes a simple yet powerful paradigm for photorealistic, controllable driving simulation, achieving an 83.4% user-preference gain and a 25.19% FID improvement over the second best state-of-the-art method. Project page: https://horizonforge.github.io/ .
Yifan Wang, Francesco Pittaluga, Zaid Tasneem +3
NEC Labs America · Stony Brook University · UC San Diego
Generating closed-loop traffic scenarios that are both realistic and controllable is crucial for evaluating autonomous driving systems, especially under rare safety-critical interactions. Existing learning-based methods often struggle to balance controllability and realism, offering either limited fine-grained control over traffic behavior or controllable scenarios at the expense of behavioral plausibility. This paper presents E2E-CDiff, an end-to-end conditional diffusion framework for controllable and realistic scenario generation. Conditioned on front-view visual observations, E2E-CDiff jointly denoises future motion states and executable low-level controls for route-interacting background vehicles. This unified state-action generation mitigates the planning-control mismatch in conventional two-stage trajectory-then-controller pipelines. Differentiable guidance further regulates speed, enforces drivable-area compliance, and supports collision-avoidance or collision-seeking behaviors, enabling both naturalistic and safety-critical scenario generation. Experiments on Bench2Drive show that E2E-CDiff achieves a favorable controllability-realism trade-off compared with representative reinforcement- and imitation-learning baselines, while its collision-guided variant induces challenging interactions across multiple autonomous driving systems. E2E-CDiff also performs competitively as a learning-based ego planner, demonstrating the generality of end-to-end state-action diffusion.
Jingzheng Li, Yufei Ge, Zhijun Chen +8
Zhongguancun Laboratory, Beijing 100094, China · Tianjin University, Tianjin 300073, China · Beihang University, Beijing 100091, China +4