PRG-Fusion: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis

Sipeng He; Jialei Chen; Zhen Fang; Dongchun Ren; Feng Zhao

cs.CVSep 7, 2026

PRG-Fusion: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis

Authors: Sipeng He, Jialei Chen, Zhen Fang, Dongchun Ren, Feng Zhao

Organizations: University of Science and Technology of China · Yootta

Abstract

Synthesizing photorealistic driving videos along specified trajectories is essential for scalable closed-loop simulation. Reconstruction-based methods leverage neural rendering to synthesize geometrically consistent views, but often exhibit diverse artifacts and missing content when the viewpoint deviates from the training trajectory. In contrast, generative models can synthesize realistic views along arbitrary trajectories from vehicle sensor data, yet often struggle to maintain temporal and geometric consistency across frames. To combine the strengths of both, we propose PRG-Fusion, a framework for driving view synthesis that uses reconstruction evidence to orchestrate generative priors across regions. Specifically, we extract region-wise degradation evidence from reconstructed driving scenes and convert it into Preserve, Repair, and Generate (PRG) labels. At inference, these labels serve as a unified routing policy for region-aware spatiotemporal synthesis, orchestrating 3DGS appearance preservation, LiDAR-guided structural correction, and video-prior-driven content completion across Preserve, Repair, and Generate regions, respectively. We then follow a two-stage training paradigm, first establish geometric control from sparse LiDAR projections and subsequently learning appearance control from dense 3DGS renderings. Extensive experiments on Waymo demonstrate that PRG-Fusion achieves state-of-the-art overall performance in novel trajectory video synthesis, with superior visual quality and geometric fidelity while maintaining competitive view consistency under large trajectory shifts.

Figures & tables

Figure 1: Motivation of PRG-Fusion. Trajectory deviation causes spatially varying degradation in 3DGS renderings, ranging from reliable appearance to geometric artifacts and missing content. PRG-Fusion models these regions as Preserve, Repair, and Generate, and accordingly coordinates 3DGS appearance, LiDAR geometry, and the video prior.

Figure 2: PRG-Fusion pipeline. Given a recorded driving log, we reconstruct a 3DGS scene and colorize the aggregated LiDAR sweeps. Along a target trajectory, 3DGS renderings and LiDAR projections form aligned visual and geometric conditions, while reconstruction reliability and support are used to derive pixel-wise PRG labels. The two conditions are encoded by separate control branches, and the PRG labels route their residuals into the frozen video DiT for novel-trajectory synthesis.

Figure 3: Visualization of PRG evidence maps. Unreliability and reconstruction support jointly determine Preserve, Repair, and Generate regions.

Figure 4: Qualitative comparison under lateral trajectory shifts. Note that GT is the reference view from the recorded trajectory, not the ground truth for the shifted target view.

	Shift 3 m			Shift 6 m
Method	NTA-IoU $\uparrow$	NTL-IoU $\uparrow$	FID $\downarrow$	NTA-IoU $\uparrow$	NTL-IoU $\uparrow$	FID $\downarrow$
OmniRe [ 7 ]	0.424	51.73	188.42	0.423	49.08	191.00
FreeVS [ 29 ]	0.505	56.84	104.23	0.465	55.37	121.44
StreetCrafter [ 35 ]	0.540	55.54	92.29	0.483	54.87	116.71
ReconDreamer [ 24 ]	0.539	54.58	93.56	0.467	52.58	149.19
ReconDreamer++ [ 43 ]	0.572	57.06	72.02	0.489	56.57	111.92

Table 1: Quantitative comparison under 3 m and 6 m lateral trajectory shifts. The best and second-best results are shown in bold and underlined, respectively.

Method	FVD $\downarrow$	CLIP-V $\uparrow$
StreetCrafter [ 35 ]	41.78	98.11
FreeVS [ 29 ]	96.62	97.24
Difix3D [ 31 ]	41.30	97.68
Ours	40.66	98.36

Table 2: Quantitative comparison of view consistency.

Figure 5: Qualitative comparison of view consistency. Note that the officially released FreeVS model is trained and evaluated at a cropped resolution that excludes the sky region to reduce computation and avoid regions with sparse LiDAR observations.

Figure 6: Qualitative ablation of PRG-Fusion. We progressively introduce LiDAR geometry, 3DGS rendering, and PRG-guided routing into the Cosmos Transfer 2.5 baseline. PRG routing enables region-adaptive coordination of geometric and visual controls, improving structural fidelity and scene appearance.

Variant	NTA-IoU $\uparrow$	NTL-IoU $\uparrow$
Baseline	0.449	53.06
LiDAR only	0.476	57.85
Render only	0.513	55.68
LiDAR + Render	0.539	56.39
Ours (+ PRG)	0.543	57.94

Table 3: Ablation of input evidence and PRG gating.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Video input	$93$ frames, $10$ fps, $960\times 640$
LiDAR aggregation	5 sweeps, 5 cameras
Masked LiDAR rows	Top 256
Iterations per stage	5,000
Hardware / batch size	8 H100s / 1 per GPU
Optimizer	AdamW

Appendix

Table 4: Training and inference settings.

Waymo segment ID	Frames
10359308928573410754_720_000_740_000	120–159
12820461091157089924_5202_916_5222_916	0–39
15021599536622641101_556_150_576_150	0–39
16767575238225610271_5185_000_5205_000	0–39
17152649515605309595_3440_000_3460_000	60–99
17860546506509760757_6040_000_6060_000	90–129

Appendix

Table 5: Evaluation scenes. Unique identifiers of the selected Waymo validation segments and their inclusive frame ranges.

Figure 7: Additional qualitative comparisons across novel trajectories. Rows show representative examples from lane-change, 3 m lateral-shift, and 6 m lateral-shift settings. The column marked GT is the recorded-view reference from the original camera trajectory, rather than pixel-aligned RGB ground truth at the target novel viewpoint. FreeVS is shown at its officially released cropped resolution, which excludes the sky region.

Dynamic Novel View Synthesis

Dynamic novel view synthesis creates time-consistent images or videos from unseen camera positions in scenes that change over time.

See all 40 papers

GeoV2V: Geometry-Grounded Video Diffusion Model for Driving Scene Generation
Oct 18, 2026Yuchen Xi, Yuhan Guo, Chenwei Hou +4Novel View Synthesis Trajectory-Guided Video Generation
Global Institute of Future Technology, Shanghai Jiao Tong University, China · Global College, Shanghai Jiao Tong University, China · School of Mechanical Engineering, Shanghai Jiao Tong University, China +1
Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis
Jun 1, 2026Zhengfei Kuang, Adam Sun, Liyuan Zhu +7Camera-Conditioned Video Generation Video Diffusion Models
1Stanford Univerity · 2NVIDIA
ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generation
Apr 1, 2026Hao Zhang, Lue Fan, Weikang Bian +4Autonomous Driving Scenario Generation Controllable Video Generation
MMLab, CUHK · CASIA · SenseTime Research

Oct 18, 2026J/K move · Enter open · S save

GeoV2V: Geometry-Grounded Video Diffusion Model for Driving Scene Generation

Yuchen Xi, Yuhan Guo, Chenwei Hou, Zixu Liu+3

Global Institute of Future Technology, Shanghai Jiao Tong University, China · Global College, Shanghai Jiao Tong University, China · School of Mechanical Engineering, Shanghai Jiao Tong University, China+1

Novel View Synthesis Trajectory-Guided Video Generation Geometric Consistency in Video Generation Video Diffusion Models Autonomous Driving Simulation Autonomous Driving Datasets

Open paper AskPDF arXiv

Novel view synthesis under trajectory changes is essential for autonomous driving simulation, yet existing methods struggle to generate consistent videos when extrapolating to unseen viewpoints such as lane shifts. Reconstruction-based approaches maintain geometric consistency but degrade in visual quality under large viewpoint changes, while reconstruct-then-restore methods often produce visually plausible frames yet suffer from geometric and temporal inconsistencies. We present GeoV2V, a geometry-grounded video-to-video diffusion framework that synthesizes temporally coherent driving videos along novel trajectories by integrating LiDAR and depth-derived geometric priors with full-video conditioning in the Wan 2.1 backbone. A key challenge in learning cross-lane transformations is the lack of synchronized multi-trajectory supervision in real-world datasets. To address this, we also introduce Para4D, a synthetic dataset capturing synchronized multi-lane driving videos within dynamic scenes, which provides explicit supervision for lane-shift generation and serves as a strong training prior for geometry-consistent view synthesis. Experiments on Waymo, nuScenes, and Para4D demonstrate state-of-the-art performance in visual quality, geometric consistency, and robustness to complex lighting dynamics under large trajectory shifts. Project page: https://xiyuche.github.io/GeoV2V/

Abstract

Dynamic Novel View Synthesis

Explore similar work

GeoV2V: Geometry-Grounded Video Diffusion Model for Driving Scene Generation

Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis

ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generation