Ensuring safety in autonomous driving requires scalable generation of realistic, controllable driving scenes beyond what real-world testing provides. Yet existing instruction guided image editors, trained on object-centric or artistic data, struggle with dense, safety-critical driving layouts. We propose HorizonWeaver, which tackles three fundamental challenges in driving scene editing: (1) multi-level granularity, requiring coherent object- and scene-level edits in dense environments; (2) rich high-level semantics, preserving diverse objects while following detailed instructions; and (3) ubiquitous domain shifts, handling changes in climate, layout, and traffic across unseen environments. The core of HorizonWeaver is a set of complementary contributions across data, model, and training: (1) Data: Large-scale dataset generation, where we build a paired real/synthetic dataset from Boreas, nuScenes, and Argoverse2 to improve generalization; (2) Model: Language-Guided Masks for fine-grained editing, where semantics-enriched masks and prompts enable precise, language-guided edits; and (3) Training: Content preservation and instruction alignment, where joint losses enforce scene consistency and instruction fidelity. Together, HorizonWeaver provides a scalable framework for photorealistic, instruction-driven editing of complex driving scenes, collecting 255K images across 13 editing categories and outperforming prior methods in L1, CLIP, and DINO metrics, achieving +46.4% user preference and improving BEV segmentation IoU by +33%. Project page: https://msoroco.github.io/horizonweaver/
Figures & tables
Figure 1: In each example, from left to right are the input image, LangMasks, and output image. Masks for global edits are blank. The careful design of HorizonWeaver across data (Section 3.2 ), model (Section 3.3 ), and training (Section 3.4 ) addresses three critical challenges in driving scene editing: 1) mult-level granularity (rows 2, 3); 2) rich high-level semantics (rows 1, 2); 3) ubiquitous domains shifts (Tab. 1 ).
Figure 2: Dataset Construction . Real-world data (Sec. 3.2.1 ) are paired by camera pose and annotated using an image descriptor pipeline and passed to an LLM to produce instructions. Pseudo-data (Sec 3.2.2 ) for Local edits crop an annotated object before VLM filtering; global edits apply VLM filtering to full images. Our dataset is composed of image pairs, global editing instructions, and masks indicating fine-grained edits to perform.
Figure 3: LangMask Generation and Training . Left: To provide fine-grained instructions with rich semantics, we insert CLIP text features into binary masks (Section 3.3 ). Right: To support LangMasks, we copy and expand the VAE. It is trained end to end with the editing model.
Figure 4: Training language-guided driving scene image editing . Our training pipeline supports both supervised training for paired images and unsupervised training for unpaired ones (e.g. downstream unseen real scenarios). We include three training objectives: supervised fine-tuning Lsft (Section 3.4.1 ), cycle consistency Lcycle (Section 3.4.2 ), and Lclip (Section 3.4.3 ).
Figure 5: HorizonWeaver Editing. Rows 1,2 Local edits: the masks (projected as binary images and stated in text for reference) enable modifications to traffic. Rows 3, 4 Global edits: the text prompt informs the appearance of the scene. For brevity, only the portions relevant to the shown edits are displayed. Rows 5, 6 Compound edits: The masks (projected as binary images) enable modifications to traffic while the text prompt informs the desired global appearance. We compare to Qwen [ 60 ] , OmniGen2 [ 61 ] , UltraEdit [ 72 ] , and BAGEL [ 13 ] .
Fine-Grained (Full Image)
Fine-Grained (Crop)
Global
Model
L1 ↓
L2 ↓
CLIP ↑
DINO ↑
L1 ↓
L2 ↓
CLIP ↑
DINO ↑
L1 ↓
L2 ↓
CLIP ↑
DINO ↑
Bagel
0.1548
0.0600
0.8381
0.7200
0.1645
0.0569
0.7498
0.5075
0.2925
0.1370
0.7952
0.6361
OmniGen2
0.2406
0.1109
0.7689
0.5539
0.2039
0.0735
0.7043
0.3383
0.2797
0.1250
0.7908
0.6365
Qwen
0.1577
0.0581
0.8387
0.7302
0.1505
0.0463
0.7688
0.5289
0.2618
0.1136
0.8196
0.7073
UltraEdit
0.0341
0.0047
0.9278
0.8993
0.0964
0.0239
0.7451
0.4791
0.1881
0.0635
0.8086
0.7374
Ours
0.0301
0.0033
0.9473
0.9298
0.0727
0.0164
0.8113
0.6563
0.1025
0.0197
0.9189
0.9110
Table 1: Fine-grained, Global, and OOD Editing Results. Each model is evaluated across fine-grained (full image), fine-grained (crop), and global editing tasks on nuPlan which is outside of the training dataset distribution. We report local-editing performance both over the full image and an edited-region crop to evaluate both content preservation and multi-scale instruction alignment.
Model
L1 (↓)
L2 (↓)
CLIP (↑)
DINO (↑)
Bagel
0.2258
0.0899
0.8224
0.6918
OmniGen2
0.2313
0.0948
0.8385
0.7212
Qwen
0.1868
0.0603
0.8640
0.7896
UltraEdit
0.2208
0.0848
0.8667
0.7838
Ours
0.1955
0.0708
0.8740
0.8232
Table 2: Compound Editing. We evaluate across real-world local and global edits simultaneously.
Model
L1 (↓)
L2 (↓)
CLIP (↑)
DINO (↑)
UltraEdit
0.2208
0.0848
0.8667
0.7838
UltraEdit-Text
0.2324
0.0996
0.8565
0.7323
UltraEdit-Mask
0.2278
0.0880
0.8726
0.7849
UltraEdit-Mask-UFT
0.1733
0.0540
0.8758
0.8003
Table 3: LangMasks and Training Objectives. Masks describing pixel level changes and unsupervised training objectives promote instruction alignment and content preservation on compound edits.
Model
Bagel
Omnigen2
Qwen
UltraEdit
Ours
Pref. (%)
9.0
2.0
14.7
13.2
61.1
Win
9.0
2.0
14.7
13.2
61.3
Table 4: User study results : overall preference distribution (“Pref.” in %) and win rates including local, global, and compound edits.
Setting
Model
L1 (↓)
L2 (↓)
CLIP (↑)
DINO (↑)
Full Image
UltraEdit-Mask-UFT
0.1851
0.0585
0.7464
0.5521
Ours
0.0301
0.0033
0.9473
0.9298
Crop
UltraEdit-Mask-UFT
0.1858
0.0616
0.7001
0.3071
Ours
0.0727
0.0164
0.8113
0.6563
Global
UltraEdit-Mask-UFT
0.2534
0.1067
0.7278
0.5560
Ours
0.1025
0.0197
0.9189
0.9110
Table 5: Diverse Environment Editing. Each model is evaluated across fine-grained (full image), fine-grained (crop), and global editing tasks on nuPlan.
Drivable
Ped. Cross.
Walkway
Stop Line
Carpark
Divider
Mean
Original
0.5834
0.0533
0.1626
0.0609
0.0981
0.1569
0.1859
Augmented
0.6448
0.1147
0.2251
0.1059
0.1776
0.2180
0.2477
Table 6: BEV Map Segmentation . Intersection-over-Union (IoU) across 6 classes and the class-averaged IoU on nuScenes.
Edit Type
Count
Percentage
Road Conditions
146,159
15%
Time of Day
235,710
24%
Traffic
263,064
27%
Traffic Light
55,998
6%
Weather
269,715
28%
Table 7: Compound Dataset Statistics (Edit Types). Distribution of edit types across the compound editing dataset.
Partition by Number of Subedits
1 Edit Type
16,623
4.86%
2 Edit Types
95,295
27.88%
3 Edit Types
160,413
46.93%
4 Edit Types
64,385
18.84%
5 Edit Types
4,922
1.44%
Total Examples
341,796
–
Table 8: Compound Dataset Statistics (Subedit Counts). Distribution of the number of edit types per example of the compound editing dataset.
Editing Type
Count
Action
Count
vehicle
3,584
modify
2,035
pedestrian
1,879
replace
1,559
trafficlight
41
delete/insert
1,910
Total
5,504
Table 9: nuScenes — Editing Statistics. Editing types and editing actions for the nuScenes dataset.
Editing Type
Count
Action
Count
vehicle
5,729
modify
2,070
pedestrian
96
delete/insert
2,957
trafficlight
144
replace
942
Total
5,969
Table 10: Boreas — Editing Statistics. Editing types and editing actions for the Boreas dataset.
Editing Type
Count
Action
Count
vehicle
2,558
modify
1,815
trafficlight
229
replace
1,503
pedestrian
2,031
delete/insert
1,500
Total
4,818
Table 11: Argoverse — Editing Statistics. Editing types and editing actions for the Argoverse dataset.
Editing Type
Count
Action
Count
vehicle
50
delete/insert
33
trafficlight
23
replace
31
pedestrian
48
modify
57
Total
121
Table 12: nuPlan — Editing Statistics. Editing types and editing actions for the nuPlan dataset.
Edit Type
Count
Percentage
Season
3,033
33%
Weather
3,116
33%
Time of Day
3,162
34%
Total
9,311
–
Table 13: nuScenes Global Editing Type. Distribution of global editing types in the nuScenes dataset.
Edit Type
Count
Percentage
Season
3,267
33%
Weather
3,288
33%
Time of Day
3,409
34%
Total
9,964
–
Table 14: Argoverse2 Global Editing Types. Distribution of global editing types in the Argoverse2 dataset.
Edit Type
Count
Percentage
Season
112
35%
Weather
100
31%
Time of Day
108
34%
Total
320
–
Table 15: nuPlan Global Editing Types. Distribution of global editing types in the nuPlan dataset.
Figure 7: Example annotation produced by the annotation pipeline.
Figure 10: HorizonWeaver Editing. Local edits: the masks (projected as binary images and stated in text for reference) enable modifications to traffic. We compare to Qwen [ 60 ] , OmniGen2 [ 61 ] , UltraEdit [ 72 ] , and BAGEL [ 13 ] .
Figure 11: HorizonWeaver Editing. Global edits: the text prompt informs the appearance of the scene. For brevity, only the portions relevant to the shown edits are displayed. We compare to Qwen [ 60 ] , OmniGen2 [ 61 ] , UltraEdit [ 72 ] , and BAGEL [ 13 ] .
Figure 12: HorizonWeaver Editing. Compound edits: The masks (projected as binary images) enable modifications to traffic while the text prompt informs the desired global appearance. We compare to Qwen [ 60 ] , OmniGen2 [ 61 ] , UltraEdit [ 72 ] , and BAGEL [ 13 ] .
Figure 13: Driving specific editor. MagicDrive [ 15 ] , cannot preserve critical driving scene components (appearance of vehicles, construction infrastructure, road signs, etc.) Long-Trail Editing our edits can extend to rare scenarios, e.g., crosswalks, road signs, and specialized vehicles (cement mixer) which are not preserved by MagicDrive.
Figure 14: Temporal Consistency. Videos produced from an edited initial image by HorizonWeaver, UltraEdit, and Qwen (top to bottom). UltraEdit leaves the initial image unchanged. Prompt: Change the season to summer.
UltraEdit
Qwen
Ours
Sem. Cons.
5.603
8.250
8.191
Temp. Cons.
6.588
6.471
6.765
Perceptual
2.603
6.191
6.265
Table 16: Quantitative comparison of semantic consistency, temporal consistency, and perceptual quality.
Fine-Grained (Full)
Fine-Grained (Crop)
Setting
L1↓
L2↓
CLIP↑
DINO↑
L1↓
L2↓
CLIP↑
DINO↑
Prefilter
0.0300
0.0032
0.9423
0.9263
0.0720
0.0157
0.8214
0.6646
Postfilter
0.0321
0.0040
0.9450
0.9267
0.0755
0.0176
0.8206
0.6536
Global
Compound
Prefilter
0.1010
0.0196
0.9193
0.9112
0.2030
0.0740
0.8708
0.8247
Postfilter
0.0983
0.0184
0.9174
0.9074
0.2026
0.0768
0.8683
0.8374
Table 17: Effect of VLM and algorithmic filtering.
Figure 15: Driving scenes are dense; thus it is difficult to describe rich semantics for multiple object-oriented changes solely with natural language.
Figure 16: Overlapping Edits , HorizonWeaver will apply insertion edits to empty spaces in crowded regions
Controllable driving scene generation is critical for realistic and scalable autonomous driving simulation, yet existing approaches struggle to jointly achieve photorealism and precise control. We introduce HorizonForge, a unified framework that reconstructs scenes as editable Gaussian Splats and Meshes, enabling fine-grained 3D manipulation and language-driven vehicle insertion. Edits are rendered through a noise-aware video diffusion process that enforces spatial and temporal consistency, producing diverse scene variations in a single feed-forward pass without per-trajectory optimization. To standardize evaluation, we further propose HorizonSuite, a comprehensive benchmark spanning ego- and agent-level editing tasks such as trajectory modifications and object manipulation. Extensive experiments show that Gaussian-Mesh representation delivers substantially higher fidelity than alternative 3D representations, and that temporal priors from video diffusion are essential for coherent synthesis. Combining these findings, HorizonForge establishes a simple yet powerful paradigm for photorealistic, controllable driving simulation, achieving an 83.4% user-preference gain and a 25.19% FID improvement over the second best state-of-the-art method. Project page: https://horizonforge.github.io/ .
Yifan Wang, Francesco Pittaluga, Zaid Tasneem +3
NEC Labs America · Stony Brook University · UC San Diego
We present ReinDriveGen, a framework that enables full controllability over dynamic driving scenes, allowing users to freely edit actor trajectories to simulate safety-critical corner cases such as front-vehicle collisions, drifting cars, vehicles spinning out of control, pedestrians jaywalking, and cyclists cutting across lanes. Our approach constructs a dynamic 3D point cloud scene from multi-frame LiDAR data, introduces a vehicle completion module to reconstruct full 360° geometry from partial observations, and renders the edited scene into 2D condition images that guide a video diffusion model to synthesize realistic driving videos. Since such edited scenarios inevitably fall outside the training distribution, we further propose an RL-based post-training strategy with a pairwise preference model and a pairwise reward mechanism, enabling robust quality improvement under out-of-distribution conditions without ground-truth supervision. Extensive experiments demonstrate that ReinDriveGen outperforms existing approaches on edited driving scenarios and achieves state-of-the-art results on novel ego viewpoint synthesis.
Generating high-fidelity and controllable synthetic data is critical for advancing end-to-end autonomous driving, particularly for addressing the long tail of rare safety-critical scenarios. Existing occupancy-guided methods typically rely on shallow conditioning mechanisms and reference-frame-dependent video synthesis, which limits fine-grained controllability from arbitrary BEV layouts and restricts their applicability for scalable simulation. In this paper, we propose AnyScene, a unified occupancy-centric framework for driving scene generation. AnyScene generates semantic occupancy sequences from BEV layouts through a Spatial-Temporal Occupancy Diffusion Transformer that jointly tokenizes BEV and occupancy features in an autoregressive manner. This design enables precise controllability from cross-dataset and user-defined BEV inputs while naturally supporting long-horizon generation. Building upon the generated occupancy, a Geometry-Grounded View Expansion module treats occupancy as the canonical spatial representation and synthesizes temporally consistent multi-view driving videos in a reference-free and autoregressive fashion, supporting flexible camera configurations at inference time. Extensive experiments demonstrate that AnyScene achieves state-of-the-art performance in both occupancy and video generation. It exhibits strong generalization to unseen and customized layouts, and provides measurable benefits for downstream tasks such as sparse-view 3D reconstruction.
Haiming Zhang, Junfei Zhou, Feng Jiang +6
Li Auto · Southwest Jiaotong University · Tsinghua University