HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles

Yifan Wang; Francesco Pittaluga; Zaid Tasneem; Chenyu You; Manmohan Chandraker; Ziyu Jiang

cs.CVFeb 24, 2026

HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles

Authors: Yifan Wang, Francesco Pittaluga, Zaid Tasneem, Chenyu You, Manmohan Chandraker, Ziyu Jiang

Organizations: NEC Labs America · Stony Brook University · UC San Diego

Abstract

Controllable driving scene generation is critical for realistic and scalable autonomous driving simulation, yet existing approaches struggle to jointly achieve photorealism and precise control. We introduce HorizonForge, a unified framework that reconstructs scenes as editable Gaussian Splats and Meshes, enabling fine-grained 3D manipulation and language-driven vehicle insertion. Edits are rendered through a noise-aware video diffusion process that enforces spatial and temporal consistency, producing diverse scene variations in a single feed-forward pass without per-trajectory optimization. To standardize evaluation, we further propose HorizonSuite, a comprehensive benchmark spanning ego- and agent-level editing tasks such as trajectory modifications and object manipulation. Extensive experiments show that Gaussian-Mesh representation delivers substantially higher fidelity than alternative 3D representations, and that temporal priors from video diffusion are essential for coherent synthesis. Combining these findings, HorizonForge establishes a simple yet powerful paradigm for photorealistic, controllable driving simulation, achieving an 83.4% user-preference gain and a 25.19% FID improvement over the second best state-of-the-art method. Project page: https://horizonforge.github.io/ .

Figures & tables

Figure 2: Overview of the HorizonForge framework. With original video and trajectory, we will firstly extract corresponding 3D assets according to the manipulated novel trajectories, then feed them into our rendering model for final generation results.

Figure 3: A demonstration of 3D Meshes Harvesting pipeline.

Figure 4: A demonstration of constructing data pairs for Gaussian Splats.

Figure 5: A demonstration of mesh-Gaussian training data pairs. The top frames are the original Gaussian Splats and the bottom ones are Gaussian with mesh vehicle replacements

Figure 6: A demonstration for trajectory modification in our proposed HorizonSuite .

(a) Ego Car Manipulation
Method	Speed Change				Lane Change				Direction Change
Method	FID $\downarrow$	FVD $\downarrow$	VIMS $\uparrow$	BAS $\uparrow$	FID $\downarrow$	FVD $\downarrow$	VIMS $\uparrow$	BAS $\uparrow$	FID $\downarrow$	FVD $\downarrow$	VIMS $\uparrow$	BAS $\uparrow$
StreetCrafter [ 59 ]	150.11	1232.13	0.9623	0.8754	134.65	1347.02	0.9568	0.8612	199.28	1846.96	0.9573	0.8497
Difix3D [ 55 ]	146.11	1019.25	0.9643	0.8357	135.80	1049.53	0.9586	0.8191	203.11	1416.44	0.9590	0.8052
OmniRe [ 11 ]	46.89	548.22	0.9619	0.9121	54.02	667.70	0.9541	0.9102	80.60	1237.23	0.9517	0.8719
Ours	39.92	583.78	0.9657	0.9326	50.33	729.76	0.9615	0.9303	144.83	1175.52	0.9585	0.8947

Table 1: Quantitative comparison on HorizonSuite with different methods. Metrics include FID

\downarrow

, FVD

\downarrow

, Vehicle Identity Matching Score (VIMS)

\uparrow

, Background Alignment Score (BAS)

\uparrow

and Operation Success Rate (OSR)

\uparrow

. Lower is better for

\downarrow

and higher is better for

\uparrow

. The best results are highlighted in bold , and the second best results are underlined .

Figure 7: Qualitative Comparison among different methods. From top to bottom is Original scene, StreetCrafter [ 59 ] , Difix3D [ 55 ] , OmniRe [ 11 ] and Ours . It’s a operation of turning ego direction to left.

(a) Ego Car Manipulation
Method	Speed Change				Lane Change				Direction Change
Method	FID $\downarrow$	FVD $\downarrow$	VIMS $\uparrow$	BAS $\uparrow$	FID $\downarrow$	FVD $\downarrow$	VIMS $\uparrow$	BAS $\uparrow$	FID $\downarrow$	FVD $\downarrow$	VIMS $\uparrow$	BAS $\uparrow$
Point Cloud	62.16	869.06	0.9633	0.9051	81.65	877.87	0.9599	0.8946	152.10	1427.25	0.9593	0.8751
BBox	108.54	2105.98	0.9491	0.7986	125.19	1677.12	0.9469	0.7744	228.06	2662.92	0.9475	0.7663
Image DM	82.19	920.29	0.9655	0.8353	89.77	967.54	0.9607	0.8188	114.11	1180.18	0.9608	0.8063
Ours	39.92	583.78	0.9657	0.9326	50.33	729.76	0.9615	0.9303	144.83	1175.52	0.9595	0.8947

Table 2: Ablation study on HorizonSuite with different settings. Metrics include FID

\downarrow

, FVD

\downarrow

, Vehicle Identity Matching Score (VIMS)

\uparrow

, Background Alignment Score (BAS)

\uparrow

and Operation Success Rate (OSR)

\uparrow

. Lower is better for

\downarrow

and higher is better for

\uparrow

. The best results are highlighted in bold , and the second best results are underlined .

Method	Wins	Total	Win Rate
StreetCrafter [ 59 ]	2	501	0.40%
Difix3D [ 55 ]	4	501	0.80%
OmniRe [ 11 ]	39	501	7.78%
Ours	456	501	91.02%

Table 3: Comparison of win rates among different methods.

Object Type	Manipulation Type	# Samples
Ego Vehicle	Direction Change	10
	Lane Change	15
	Speed Change	14
Other Vehicles	Direction Change	8
	Lane Change	13
	Speed Change	15

Table 4: Statistics of manipulation categories in HorizonSuite . The benchmark with in total 109 samples covers two object types (Ego vehicle and Other vehicles) and five manipulation types, providing diverse scene editing scenarios for controllable driving generation evaluation.

Training Data	[email protected]	[email protected]	[email protected]
Org	0.1211	0.0533	0.0103
Org + Ins	0.1343	0.0635	0.0125
Org + Rem	0.1359	0.0613	0.0134

Table 5: Downstream 3D detection results of BEVFormer [ 26 ] on the Waymo validation set under a front-camera-only configuration. Org denotes the original

8000

real frames. Ins and Rem denote additional frames generated using our method with random vehicle insertion and removal, respectively.

Method	NTA-IoU ( $\uparrow$ )		NTL-IoU ( $\uparrow$ )		FID ( $\downarrow$ )
Method	Lane Shift @ 3m	Lane Shift @ 6m	Lane Shift @ 3m	Lane Shift @ 6m	Lane Shift @ 3m	Lane Shift @ 6m
DriveDreamer4D	0.340	0.121	51.32	49.28	129.05	210.37
ReconDreamer	0.539	0.467	54.58	52.58	93.56	149.19
Ours	0.552	0.523	56.73	54.01	49.47	53.61

Table 6: Comparison on large-offset manipulation.

Figure 8: Visualization under challenging conditions.

Metric	Ours		Difix3D		OmniRe		StreetCrafter
Metric	Ins.	Rem.	Ins.	Rem.	Ins.	Rem.	Ins.	Rem.
OSR	5.86	7.00	4.71	6.89	4.23	6.42	4.62	6.51
Human	7.50	7.25	3.79	4.40	3.58	4.75	3.12	3.25

Table 7: Comparison of OSR and Human Scores.

Explore similar work

Real2Sim: A Physics-driven and Editable Gaussian Splatting Framework for Autonomous Driving Scenes
May 13, 2026Kaicong Huang, Talha Azfar, Weisong Shi +13D Gaussian Splatting Synthetic Data Generation
Department of Civil and Environmental Engineering, Rensselaer Polytechnic Institute, 110 Eighth Street, Troy, NY USA 12180. · Department of Computer and Information Sciences, University of Delaware, Newark, DE USA 19716.
HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes
Apr 6, 2026Mauricio Soroco, Francesco Pittaluga, Zaid Tasneem +5Text-Guided Image Editing Autonomous Driving Scenario Generation
Simon Fraser University · NEC Labs America
GA-Drive: Geometry-Appearance Decoupled Modeling for Free-viewpoint Driving Scene Generation
Feb 24, 2026Hao Zhang, Lue Fan, Qitai Wang +5Autonomous Driving Simulation Dynamic Novel View Synthesis
MMLab, CUHK · CASIA · Shanghai Jiaotong University +1

May 13, 2026J/K move · Enter open · S save

Real2Sim: A Physics-driven and Editable Gaussian Splatting Framework for Autonomous Driving Scenes

Kaicong Huang, Talha Azfar, Weisong Shi, Ruimin Ke

Department of Civil and Environmental Engineering, Rensselaer Polytechnic Institute, 110 Eighth Street, Troy, NY USA 12180. · Department of Computer and Information Sciences, University of Delaware, Newark, DE USA 19716.

3D Gaussian Splatting Synthetic Data Generation Dynamic Scene Reconstruction Differentiable Physics Physics-Based Simulation Autonomous Driving Simulation+1

Open paper AskPDF arXiv

Reliable autonomous driving relies on large-scale, well-labeled data and robust models. However, manual data collection is resource-intensive, and traditional simulation suffers from a persistent reality gap. While recent generative frameworks and radiance-field methods improve visual fidelity, they still struggle with temporal and spatial consistency and cannot ensure physics-aware behavior, limiting their applicability to driving scenario generation. To address these challenges, we propose Real2Sim, an unified framework that combines 4D Gaussian Splatting (4DGS) with a differentiable Material Point Method (MPM) solver. Real2Sim explicitly reconstructs dynamic driving scenes as temporally continuous Gaussian primitives, supports instance-level editing, and simulates realistic object-object and object-environment interactions. This framework enables physics-aware, high-fidelity synthesis of diverse, editable scenarios, including challenging corner cases such as collisions and post-impact trajectories. Experiments on the Waymo Open Dataset validate Real2Sim's capabilities in rendering, reconstruction, editing, and physics simulation, demonstrating its potential as a scalable tool for data generation in downstream tasks such as perception, tracking, trajectory prediction, and end-to-end policy learning.

Abstract

Explore similar work

Real2Sim: A Physics-driven and Editable Gaussian Splatting Framework for Autonomous Driving Scenes

HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes

GA-Drive: Geometry-Appearance Decoupled Modeling for Free-viewpoint Driving Scene Generation