GA-Drive: Geometry-Appearance Decoupled Modeling for Free-viewpoint Driving Scene Generation

Hao Zhang; Lue Fan; Qitai Wang; Wenbo Li; Zehuan Wu; Lewei Lu; Zhaoxiang Zhang; Hongsheng Li

cs.CVFeb 24, 2026

GA-Drive: Geometry-Appearance Decoupled Modeling for Free-viewpoint Driving Scene Generation

Authors: Hao Zhang, Lue Fan, Qitai Wang, Wenbo Li, Zehuan Wu, Lewei Lu, Zhaoxiang Zhang, Hongsheng Li

Organizations: MMLab, CUHK · CASIA · Shanghai Jiaotong University · SenseTime Research

Abstract

A free-viewpoint, editable, and high-fidelity driving simulator is crucial for training and evaluating end-to-end autonomous driving systems. In this paper, we present GA-Drive, a novel simulation framework capable of generating camera views along user-specified novel trajectories through Geometry-Appearance Decoupling and Diffusion-Based Generation. Given a set of images captured along a recorded trajectory and the corresponding scene geometry, GA-Drive synthesizes novel pseudo-views using geometry information. These pseudo-views are then transformed into photorealistic views using a trained video diffusion model. In this way, we decouple the geometry and appearance of scenes. An advantage of such decoupling is its support for appearance editing via state-of-the-art video-to-video editing techniques, while preserving the underlying geometry, enabling consistent edits across both original and novel trajectories. Extensive experiments demonstrate that GA-Drive substantially outperforms existing methods in terms of NTA-IoU, NTL-IoU, and FID scores.

Figures & tables

Figure 2: The overview framework of GA-Drive. Pseudo views are created by casting rays from novel poses into 3D space with rendered depth maps to generate a 3D point cloud, which is then projected onto recorded views to sample color information. These pseudo views are subsequently transformed into photorealistic frames using our trained video diffusion model (referred to as Generation). The black curved line indicates the recorded camera trajectory. By iteratively applying the video diffusion model to each segment, our method can generate photorealistic novel views of unlimited length.

Figure 3: (a) illustrates the 2D version of the visibility check. Since the rendered depth $<$ the point depth, the point is occluded by a surface and should be masked. (b) shows an incorrect pseudo view without applying the visibility check, while (c) presents the correct result with visibility properly handled.

Figure 4: The architecture of our video diffusion model. Our segment-wise video diffusion model generates videos conditioned on the novel pseudo-views.

Figure 5: The pseudo-view simulation pipeline. We design this pseudo-view simulation pipeline to simulate the characteristic patterns of the novel pseudo-views.

Figure 6: Qualitative comparison of novel trajectories with a 3-meter lane shift. In the comparison, the fourth row is shifted to the right, while the others are shifted to the left.

Models	PVG	DD4D w/. PVG	Recon. w/. PVG	$S^{3}$ Gauss.	DD4D w/. $S^{3}$ Gauss.	Recon. w/. $S^{3}$ Gauss.	Deform.-GS	DD4D w/. Deform.-GS	Recon. w/. Deform.-GS	Ours
NTA-IoU $\uparrow$	0.256	0.438	0.464	0.175	0.495	0.413	0.240	0.335	0.443	0.558
NTL-IoU $\uparrow$	50.70	53.06	53.21	49.05	53.42	51.62	51.62	52.93	53.78	56.83
FID $\downarrow$	105.29	71.52	74.32	124.90	66.93	123.61	92.24	77.32	76.24	49.77

Table 1: Quantitative comparison in a lane-change scenario where the trajectory gradually shifts 4 m to the left.

Figure 7: We conduct an ablation study on Dense Depth Supervision and Depth Distortion Regularization. Specifically, setting (a) refers to training OmniRe with the original configuration, (b) adds Dense Depth Supervision, and (c) includes both Dense Depth Supervision and Depth Distortion Regularization. Setting (c) produces the cleanest geometry.

Figure 8: Ablation study on segment-wise video generation: (a) last frame of the previous segment; (b) first pseudo-novel view of the next segment; (c) view generated by the ablation model; (d) view generated by our default model. Compared to (c), (d) provides a much more coherent transition with (a).

Corruption	NTA-IoU $\uparrow$	NTL-IoU $\uparrow$	FID $\downarrow$
–	0.558	56.83	49.77
$-20\%$ Gaussians	0.555	56.54	50.67
$-40\%$ Gaussians	0.542	56.02	51.75
$-80\%$ Gaussians	0.535	54.79	50.86
$\pm 0.1$ m noise	0.545	56.62	52.13
$\pm 0.2$ m noise	0.538	56.24	50.97

Table 2: Performance under different corruptions. The first row is our default model without corruption

Explore similar work

HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles
Feb 24, 2026Yifan Wang, Francesco Pittaluga, Zaid Tasneem +3Autonomous Driving Scenario Generation 3D Scene Editing
NEC Labs America · Stony Brook University · UC San Diego
Real2Sim: A Physics-driven and Editable Gaussian Splatting Framework for Autonomous Driving Scenes
May 13, 2026Kaicong Huang, Talha Azfar, Weisong Shi +13D Gaussian Splatting Synthetic Data Generation
Department of Civil and Environmental Engineering, Rensselaer Polytechnic Institute, 110 Eighth Street, Troy, NY USA 12180. · Department of Computer and Information Sciences, University of Delaware, Newark, DE USA 19716.
DriveCtrl: Conditioned Sim-to-Real Driving Video Generation
May 14, 2026Haonan Zhao, Yiting Wang, Jingkun Chen +3Autonomous Driving Datasets Synthetic Data Augmentation
University of Warwick · 2Northwestern Polytechnical University · 3Queen Mary University of London

Feb 24, 2026J/K move · Enter open · S save

HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles

Yifan Wang, Francesco Pittaluga, Zaid Tasneem, Chenyu You+2

NEC Labs America · Stony Brook University · UC San Diego

Autonomous Driving Scenario Generation 3D Scene Editing Autonomous Driving Benchmarks Gaussian Splatting

Open paper AskPDF arXiv

Controllable driving scene generation is critical for realistic and scalable autonomous driving simulation, yet existing approaches struggle to jointly achieve photorealism and precise control. We introduce HorizonForge, a unified framework that reconstructs scenes as editable Gaussian Splats and Meshes, enabling fine-grained 3D manipulation and language-driven vehicle insertion. Edits are rendered through a noise-aware video diffusion process that enforces spatial and temporal consistency, producing diverse scene variations in a single feed-forward pass without per-trajectory optimization. To standardize evaluation, we further propose HorizonSuite, a comprehensive benchmark spanning ego- and agent-level editing tasks such as trajectory modifications and object manipulation. Extensive experiments show that Gaussian-Mesh representation delivers substantially higher fidelity than alternative 3D representations, and that temporal priors from video diffusion are essential for coherent synthesis. Combining these findings, HorizonForge establishes a simple yet powerful paradigm for photorealistic, controllable driving simulation, achieving an 83.4% user-preference gain and a 25.19% FID improvement over the second best state-of-the-art method. Project page: https://horizonforge.github.io/ .

Abstract

Explore similar work

HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles

Real2Sim: A Physics-driven and Editable Gaussian Splatting Framework for Autonomous Driving Scenes

DriveCtrl: Conditioned Sim-to-Real Driving Video Generation