HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes

Mauricio Soroco; Francesco Pittaluga; Zaid Tasneem; Abhishek Aich; Bingbing Zhuang; Wuyang Chen; Manmohan Chandraker; Ziyu Jiang

cs.CVApr 6, 2026

HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes

Authors: Mauricio Soroco, Francesco Pittaluga, Zaid Tasneem, Abhishek Aich, Bingbing Zhuang, Wuyang Chen, Manmohan Chandraker, Ziyu Jiang

Organizations: Simon Fraser University · NEC Labs America

Abstract

Ensuring safety in autonomous driving requires scalable generation of realistic, controllable driving scenes beyond what real-world testing provides. Yet existing instruction guided image editors, trained on object-centric or artistic data, struggle with dense, safety-critical driving layouts. We propose HorizonWeaver, which tackles three fundamental challenges in driving scene editing: (1) multi-level granularity, requiring coherent object- and scene-level edits in dense environments; (2) rich high-level semantics, preserving diverse objects while following detailed instructions; and (3) ubiquitous domain shifts, handling changes in climate, layout, and traffic across unseen environments. The core of HorizonWeaver is a set of complementary contributions across data, model, and training: (1) Data: Large-scale dataset generation, where we build a paired real/synthetic dataset from Boreas, nuScenes, and Argoverse2 to improve generalization; (2) Model: Language-Guided Masks for fine-grained editing, where semantics-enriched masks and prompts enable precise, language-guided edits; and (3) Training: Content preservation and instruction alignment, where joint losses enforce scene consistency and instruction fidelity. Together, HorizonWeaver provides a scalable framework for photorealistic, instruction-driven editing of complex driving scenes, collecting 255K images across 13 editing categories and outperforming prior methods in L1, CLIP, and DINO metrics, achieving +46.4% user preference and improving BEV segmentation IoU by +33%. Project page: https://msoroco.github.io/horizonweaver/

Figures & tables

Figure 1: In each example, from left to right are the input image, LangMasks, and output image. Masks for global edits are blank. The careful design of HorizonWeaver across data (Section 3.2 ), model (Section 3.3 ), and training (Section 3.4 ) addresses three critical challenges in driving scene editing: 1) mult-level granularity (rows 2, 3); 2) rich high-level semantics (rows 1, 2); 3) ubiquitous domains shifts (Tab. 1 ).

Figure 2: Dataset Construction . Real-world data (Sec. 3.2.1 ) are paired by camera pose and annotated using an image descriptor pipeline and passed to an LLM to produce instructions. Pseudo-data (Sec 3.2.2 ) for Local edits crop an annotated object before VLM filtering; global edits apply VLM filtering to full images. Our dataset is composed of image pairs, global editing instructions, and masks indicating fine-grained edits to perform.

Figure 3: LangMask Generation and Training . Left: To provide fine-grained instructions with rich semantics, we insert CLIP text features into binary masks (Section 3.3 ). Right: To support LangMasks, we copy and expand the VAE. It is trained end to end with the editing model.

Figure 4: Training language-guided driving scene image editing . Our training pipeline supports both supervised training for paired images and unsupervised training for unpaired ones (e.g. downstream unseen real scenarios). We include three training objectives: supervised fine-tuning $\mathcal{L}_{\text{sft}}$ (Section 3.4.1 ), cycle consistency $\mathcal{L}_{\text{cycle}}$ (Section 3.4.2 ), and $\mathcal{L}_{\text{clip}}$ (Section 3.4.3 ).

Panel of Figure 5 — Figure 5: HorizonWeaver Editing. Rows 1,2 Local edits: the masks (projected as binary images and stated in text for reference) enable modifications to traffic. Rows 3, 4 Global edits: the text prompt informs the appearance of the scene. For brevity, only the portions relevant to the shown edits are displayed. Rows 5, 6 Compound edits: The masks (projected as binary images) enable modifications to traffic while the text prompt informs the desired global appearance. We compare to Qwen [ 60 ] , OmniGen2 [ 61 ] , UltraEdit [ 72 ] , and BAGEL [ 13 ] .

	Fine-Grained (Full Image)				Fine-Grained (Crop)				Global
Model	L1 ↓	L2 ↓	CLIP ↑	DINO ↑	L1 ↓	L2 ↓	CLIP ↑	DINO ↑	L1 ↓	L2 ↓	CLIP ↑	DINO ↑
Bagel	0.1548	0.0600	0.8381	0.7200	0.1645	0.0569	0.7498	0.5075	0.2925	0.1370	0.7952	0.6361
OmniGen2	0.2406	0.1109	0.7689	0.5539	0.2039	0.0735	0.7043	0.3383	0.2797	0.1250	0.7908	0.6365
Qwen	0.1577	0.0581	0.8387	0.7302	0.1505	0.0463	0.7688	0.5289	0.2618	0.1136	0.8196	0.7073
UltraEdit	0.0341	0.0047	0.9278	0.8993	0.0964	0.0239	0.7451	0.4791	0.1881	0.0635	0.8086	0.7374
Ours	0.0301	0.0033	0.9473	0.9298	0.0727	0.0164	0.8113	0.6563	0.1025	0.0197	0.9189	0.9110

Table 1: Fine-grained, Global, and OOD Editing Results. Each model is evaluated across fine-grained (full image), fine-grained (crop), and global editing tasks on nuPlan which is outside of the training dataset distribution. We report local-editing performance both over the full image and an edited-region crop to evaluate both content preservation and multi-scale instruction alignment.

Model	L1 (↓)	L2 (↓)	CLIP (↑)	DINO (↑)
Bagel	0.2258	0.0899	0.8224	0.6918
OmniGen2	0.2313	0.0948	0.8385	0.7212
Qwen	0.1868	0.0603	0.8640	0.7896
UltraEdit	0.2208	0.0848	0.8667	0.7838
Ours	0.1955	0.0708	0.8740	0.8232

Table 2: Compound Editing. We evaluate across real-world local and global edits simultaneously.

Model	L1 (↓)	L2 (↓)	CLIP (↑)	DINO (↑)
UltraEdit	0.2208	0.0848	0.8667	0.7838
UltraEdit-Text	0.2324	0.0996	0.8565	0.7323
UltraEdit-Mask	0.2278	0.0880	0.8726	0.7849
UltraEdit-Mask-UFT	0.1733	0.0540	0.8758	0.8003

Table 3: LangMasks and Training Objectives. Masks describing pixel level changes and unsupervised training objectives promote instruction alignment and content preservation on compound edits.

Model	Bagel	Omnigen2	Qwen	UltraEdit	Ours
Pref. (%)	9.0	2.0	14.7	13.2	61.1
Win	9.0	2.0	14.7	13.2	61.3

Table 4: User study results : overall preference distribution (“Pref.” in %) and win rates including local, global, and compound edits.

Setting	Model	L1 (↓)	L2 (↓)	CLIP (↑)	DINO (↑)
Full Image	UltraEdit-Mask-UFT	0.1851	0.0585	0.7464	0.5521
Full Image	Ours	0.0301	0.0033	0.9473	0.9298
Crop	UltraEdit-Mask-UFT	0.1858	0.0616	0.7001	0.3071
Crop	Ours	0.0727	0.0164	0.8113	0.6563
Global	UltraEdit-Mask-UFT	0.2534	0.1067	0.7278	0.5560
Global	Ours	0.1025	0.0197	0.9189	0.9110

Table 5: Diverse Environment Editing. Each model is evaluated across fine-grained (full image), fine-grained (crop), and global editing tasks on nuPlan.

	Drivable	Ped. Cross.	Walkway	Stop Line	Carpark	Divider	Mean
Original	0.5834	0.0533	0.1626	0.0609	0.0981	0.1569	0.1859
Augmented	0.6448	0.1147	0.2251	0.1059	0.1776	0.2180	0.2477

Table 6: BEV Map Segmentation . Intersection-over-Union (IoU) across 6 classes and the class-averaged IoU on nuScenes.

Edit Type	Count	Percentage
Road Conditions	146,159	15%
Time of Day	235,710	24%
Traffic	263,064	27%
Traffic Light	55,998	6%
Weather	269,715	28%

Table 7: Compound Dataset Statistics (Edit Types). Distribution of edit types across the compound editing dataset.

Partition by Number of Subedits
1 Edit Type	16,623	4.86%
2 Edit Types	95,295	27.88%
3 Edit Types	160,413	46.93%
4 Edit Types	64,385	18.84%
5 Edit Types	4,922	1.44%
Total Examples	341,796	–

Table 8: Compound Dataset Statistics (Subedit Counts). Distribution of the number of edit types per example of the compound editing dataset.

Editing Type	Count	Action	Count
vehicle	3,584	modify	2,035
pedestrian	1,879	replace	1,559
trafficlight	41	delete/insert	1,910
Total	5,504

Table 9: nuScenes — Editing Statistics. Editing types and editing actions for the nuScenes dataset.

Editing Type	Count	Action	Count
vehicle	5,729	modify	2,070
pedestrian	96	delete/insert	2,957
trafficlight	144	replace	942
Total	5,969

Table 10: Boreas — Editing Statistics. Editing types and editing actions for the Boreas dataset.

Editing Type	Count	Action	Count
vehicle	2,558	modify	1,815
trafficlight	229	replace	1,503
pedestrian	2,031	delete/insert	1,500
Total	4,818

Table 11: Argoverse — Editing Statistics. Editing types and editing actions for the Argoverse dataset.

Editing Type	Count	Action	Count
vehicle	50	delete/insert	33
trafficlight	23	replace	31
pedestrian	48	modify	57
Total	121

Table 12: nuPlan — Editing Statistics. Editing types and editing actions for the nuPlan dataset.

Edit Type	Count	Percentage
Season	3,033	33%
Weather	3,116	33%
Time of Day	3,162	34%
Total	9,311	–

Table 13: nuScenes Global Editing Type. Distribution of global editing types in the nuScenes dataset.

Edit Type	Count	Percentage
Season	3,267	33%
Weather	3,288	33%
Time of Day	3,409	34%
Total	9,964	–

Table 14: Argoverse2 Global Editing Types. Distribution of global editing types in the Argoverse2 dataset.

Edit Type	Count	Percentage
Season	112	35%
Weather	100	31%
Time of Day	108	34%
Total	320	–

Table 15: nuPlan Global Editing Types. Distribution of global editing types in the nuPlan dataset.

Figure 7: Example annotation produced by the annotation pipeline.

Panel of Figure 10 — Figure 10: HorizonWeaver Editing. Local edits: the masks (projected as binary images and stated in text for reference) enable modifications to traffic. We compare to Qwen [ 60 ] , OmniGen2 [ 61 ] , UltraEdit [ 72 ] , and BAGEL [ 13 ] .

Panel of Figure 11 — Figure 11: HorizonWeaver Editing. Global edits: the text prompt informs the appearance of the scene. For brevity, only the portions relevant to the shown edits are displayed. We compare to Qwen [ 60 ] , OmniGen2 [ 61 ] , UltraEdit [ 72 ] , and BAGEL [ 13 ] .

Panel of Figure 12 — Figure 12: HorizonWeaver Editing. Compound edits: The masks (projected as binary images) enable modifications to traffic while the text prompt informs the desired global appearance. We compare to Qwen [ 60 ] , OmniGen2 [ 61 ] , UltraEdit [ 72 ] , and BAGEL [ 13 ] .

Figure 13: Driving specific editor. MagicDrive [ 15 ] , cannot preserve critical driving scene components (appearance of vehicles, construction infrastructure, road signs, etc.) Long-Trail Editing our edits can extend to rare scenarios, e.g., crosswalks, road signs, and specialized vehicles (cement mixer) which are not preserved by MagicDrive.

Panel of Figure 14 — Figure 14: Temporal Consistency. Videos produced from an edited initial image by HorizonWeaver, UltraEdit, and Qwen (top to bottom). UltraEdit leaves the initial image unchanged. Prompt: Change the season to summer.

	UltraEdit	Qwen	Ours
Sem. Cons.	5.603	8.250	8.191
Temp. Cons.	6.588	6.471	6.765
Perceptual	2.603	6.191	6.265

Table 16: Quantitative comparison of semantic consistency, temporal consistency, and perceptual quality.

	Fine-Grained (Full)				Fine-Grained (Crop)
Setting	L1↓	L2↓	CLIP↑	DINO↑	L1↓	L2↓	CLIP↑	DINO↑
Prefilter	0.0300	0.0032	0.9423	0.9263	0.0720	0.0157	0.8214	0.6646
Postfilter	0.0321	0.0040	0.9450	0.9267	0.0755	0.0176	0.8206	0.6536
	Global				Compound
Prefilter	0.1010	0.0196	0.9193	0.9112	0.2030	0.0740	0.8708	0.8247
Postfilter	0.0983	0.0184	0.9174	0.9074	0.2026	0.0768	0.8683	0.8374

Table 17: Effect of VLM and algorithmic filtering.

Figure 15: Driving scenes are dense; thus it is difficult to describe rich semantics for multiple object-oriented changes solely with natural language.

Panel of Figure 16 — Figure 16: Overlapping Edits , HorizonWeaver will apply insertion edits to empty spaces in crowded regions

Explore similar work

HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles
Feb 24, 2026Yifan Wang, Francesco Pittaluga, Zaid Tasneem +3Autonomous Driving Scenario Generation 3D Scene Editing
NEC Labs America · Stony Brook University · UC San Diego
ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generation
Apr 1, 2026Hao Zhang, Lue Fan, Weikang Bian +4Autonomous Driving Scenario Generation Controllable Video Generation
MMLab, CUHK · CASIA · SenseTime Research
AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond
May 25, 2026Haiming Zhang, Junfei Zhou, Feng Jiang +6Long-Horizon Video Generation 3D Scene Generation
Li Auto · Southwest Jiaotong University · Tsinghua University

Feb 24, 2026J/K move · Enter open · S save

HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles

Yifan Wang, Francesco Pittaluga, Zaid Tasneem, Chenyu You+2

NEC Labs America · Stony Brook University · UC San Diego

Autonomous Driving Scenario Generation 3D Scene Editing Autonomous Driving Benchmarks Gaussian Splatting

Open paper AskPDF arXiv

Controllable driving scene generation is critical for realistic and scalable autonomous driving simulation, yet existing approaches struggle to jointly achieve photorealism and precise control. We introduce HorizonForge, a unified framework that reconstructs scenes as editable Gaussian Splats and Meshes, enabling fine-grained 3D manipulation and language-driven vehicle insertion. Edits are rendered through a noise-aware video diffusion process that enforces spatial and temporal consistency, producing diverse scene variations in a single feed-forward pass without per-trajectory optimization. To standardize evaluation, we further propose HorizonSuite, a comprehensive benchmark spanning ego- and agent-level editing tasks such as trajectory modifications and object manipulation. Extensive experiments show that Gaussian-Mesh representation delivers substantially higher fidelity than alternative 3D representations, and that temporal priors from video diffusion are essential for coherent synthesis. Combining these findings, HorizonForge establishes a simple yet powerful paradigm for photorealistic, controllable driving simulation, achieving an 83.4% user-preference gain and a 25.19% FID improvement over the second best state-of-the-art method. Project page: https://horizonforge.github.io/ .

Abstract

Explore similar work

HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles

ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generation

AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond