cs.CVSep 7, 2026

PRG-Fusion: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis

Authors: Sipeng He, Jialei Chen, Zhen Fang, Dongchun Ren, Feng Zhao

Organizations: University of Science and Technology of China · Yootta

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Synthesizing photorealistic driving videos along specified trajectories is essential for scalable closed-loop simulation. Reconstruction-based methods leverage neural rendering to synthesize geometrically consistent views, but often exhibit diverse artifacts and missing content when the viewpoint deviates from the training trajectory. In contrast, generative models can synthesize realistic views along arbitrary trajectories from vehicle sensor data, yet often struggle to maintain temporal and geometric consistency across frames. To combine the strengths of both, we propose PRG-Fusion, a framework for driving view synthesis that uses reconstruction evidence to orchestrate generative priors across regions. Specifically, we extract region-wise degradation evidence from reconstructed driving scenes and convert it into Preserve, Repair, and Generate (PRG) labels. At inference, these labels serve as a unified routing policy for region-aware spatiotemporal synthesis, orchestrating 3DGS appearance preservation, LiDAR-guided structural correction, and video-prior-driven content completion across Preserve, Repair, and Generate regions, respectively. We then follow a two-stage training paradigm, first establish geometric control from sparse LiDAR projections and subsequently learning appearance control from dense 3DGS renderings. Extensive experiments on Waymo demonstrate that PRG-Fusion achieves state-of-the-art overall performance in novel trajectory video synthesis, with superior visual quality and geometric fidelity while maintaining competitive view consistency under large trajectory shifts.

Figures & tables

Figure 1: Motivation of PRG-Fusion. Trajectory deviation causes spatially varying degradation in 3DGS renderings, ranging from reliable appearance to geometric artifacts and missing content. PRG-Fusion models these regions as Preserve, Repair, and Generate, and accordingly coordinates 3DGS appearance, LiDAR geometry, and the video prior.
Figure 2: PRG-Fusion pipeline. Given a recorded driving log, we reconstruct a 3DGS scene and colorize the aggregated LiDAR sweeps. Along a target trajectory, 3DGS renderings and LiDAR projections form aligned visual and geometric conditions, while reconstruction reliability and support are used to derive pixel-wise PRG labels. The two conditions are encoded by separate control branches, and the PRG labels route their residuals into the frozen video DiT for novel-trajectory synthesis.
Figure 3: Visualization of PRG evidence maps. Unreliability and reconstruction support jointly determine Preserve, Repair, and Generate regions.
Figure 4: Qualitative comparison under lateral trajectory shifts. Note that GT is the reference view from the recorded trajectory, not the ground truth for the shifted target view.
 Shift 3 mShift 6 m
MethodNTA-IoU ↑\uparrowNTL-IoU ↑\uparrowFID ↓\downarrowNTA-IoU ↑\uparrowNTL-IoU ↑\uparrowFID ↓\downarrow
OmniRe [ 7 ]0.42451.73188.420.42349.08191.00
FreeVS [ 29 ]0.50556.84104.230.46555.37121.44
StreetCrafter [ 35 ]0.54055.5492.290.48354.87116.71
ReconDreamer [ 24 ]0.53954.5893.560.46752.58149.19
ReconDreamer++ [ 43 ]0.57257.0672.020.48956.57111.92
Table 1: Quantitative comparison under 3 m and 6 m lateral trajectory shifts. The best and second-best results are shown in bold and underlined, respectively.
MethodFVD ↓\downarrowCLIP-V ↑\uparrow
StreetCrafter [ 35 ]41.7898.11
FreeVS [ 29 ]96.6297.24
Difix3D [ 31 ]41.3097.68
Ours40.6698.36
Table 2: Quantitative comparison of view consistency.
Figure 5: Qualitative comparison of view consistency. Note that the officially released FreeVS model is trained and evaluated at a cropped resolution that excludes the sky region to reduce computation and avoid regions with sparse LiDAR observations.
Figure 6: Qualitative ablation of PRG-Fusion. We progressively introduce LiDAR geometry, 3DGS rendering, and PRG-guided routing into the Cosmos Transfer 2.5 baseline. PRG routing enables region-adaptive coordination of geometric and visual controls, improving structural fidelity and scene appearance.
VariantNTA-IoU ↑\uparrowNTL-IoU ↑\uparrow
Baseline0.44953.06
LiDAR only0.47657.85
Render only0.51355.68
LiDAR + Render0.53956.39
Ours (+ PRG)0.54357.94
Table 3: Ablation of input evidence and PRG gating.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

SettingValue
Video input9393 frames, 1010 fps, 960×640960\times 640
LiDAR aggregation5 sweeps, 5 cameras
Masked LiDAR rowsTop 256
Iterations per stage5,000
Hardware / batch size8 H100s / 1 per GPU
OptimizerAdamW
Appendix
Table 4: Training and inference settings.
Waymo segment IDFrames
10359308928573410754_720_000_740_000120–159
12820461091157089924_5202_916_5222_9160–39
15021599536622641101_556_150_576_1500–39
16767575238225610271_5185_000_5205_0000–39
17152649515605309595_3440_000_3460_00060–99
17860546506509760757_6040_000_6060_00090–129
Appendix
Table 5: Evaluation scenes. Unique identifiers of the selected Waymo validation segments and their inclusive frame ranges.
Figure 7: Additional qualitative comparisons across novel trajectories. Rows show representative examples from lane-change, 3 m lateral-shift, and 6 m lateral-shift settings. The column marked GT is the recorded-view reference from the original camera trajectory, rather than pixel-aligned RGB ground truth at the target novel viewpoint. FreeVS is shown at its officially released cropped resolution, which excludes the sky region.
More in this topic

Dynamic Novel View Synthesis

Dynamic novel view synthesis creates time-consistent images or videos from unseen camera positions in scenes that change over time.

See all 40 papers

Explore similar work

CardsList
Oct 18, 2026

GeoV2V: Geometry-Grounded Video Diffusion Model for Driving Scene Generation

Novel view synthesis under trajectory changes is essential for autonomous driving simulation, yet existing methods struggle to generate consistent videos when extrapolating to unseen viewpoints such as lane shifts. Reconstruction-based approaches maintain geometric consistency but degrade in visual quality under large viewpoint changes, while reconstruct-then-restore methods often produce visually plausible frames yet suffer from geometric and temporal inconsistencies. We present GeoV2V, a geometry-grounded video-to-video diffusion framework that synthesizes temporally coherent driving videos along novel trajectories by integrating LiDAR and depth-derived geometric priors with full-video conditioning in the Wan 2.1 backbone. A key challenge in learning cross-lane transformations is the lack of synchronized multi-trajectory supervision in real-world datasets. To address this, we also introduce Para4D, a synthetic dataset capturing synchronized multi-lane driving videos within dynamic scenes, which provides explicit supervision for lane-shift generation and serves as a strong training prior for geometry-consistent view synthesis. Experiments on Waymo, nuScenes, and Para4D demonstrate state-of-the-art performance in visual quality, geometric consistency, and robustness to complex lighting dynamics under large trajectory shifts. Project page: https://xiyuche.github.io/GeoV2V/

Yuchen Xi, Yuhan Guo, Chenwei Hou +4

Global Institute of Future Technology, Shanghai Jiao Tong University, China · Global College, Shanghai Jiao Tong University, China · School of Mechanical Engineering, Shanghai Jiao Tong University, China +1

Novel View SynthesisTrajectory-Guided Video GenerationGeometric Consistency in Video GenerationVideo Diffusion ModelsAutonomous Driving SimulationAutonomous Driving Datasets
AskPDF
Jun 1, 2026

Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis

Modern vehicle platforms are equipped with a rich sensor suite, including LiDAR, calibrated multi-camera rigs, and accurate ego-motion, that in principle offers strong signal for re-rendering a driving scene from novel viewpoints. A growing line of recent work leverages video diffusion models for this task, using their generative priors to synthesize plausible novel views from sparse vehicle observations. In practice, however, existing methods exploit only a fragment of this signal, and their quality tends to degrade as the target trajectory departs from the recorded driving path. We argue that this is fundamentally a multi-sensor fusion problem: sparse LiDAR reprojections supply accurate but incomplete metric geometry, surround-view reference imagery supplies dense appearance but no metric depth, and camera poses tie the two together across views. We introduce StreetNVS, a video diffusion framework that jointly conditions on all three signals through a Reference-Enhanced Camera Attention module based on a relative ray-level positional encoding. We develop a two-stage curriculum training strategy that gradually exposes the model to increasingly sparse LiDAR. On the Waymo Open Dataset, StreetNVS substantially outperforms state-of-the-art baselines under sparse LiDAR conditioning, matches methods that rely on 10-100 times denser point clouds. We further show capabilities of synthesizing coherent videos along extreme out-of-trajectory paths such as elevation, lane-shift, pullback, and rotation. Our website: https://streetnvs.github.io

Zhengfei Kuang, Adam Sun, Liyuan Zhu +7

1Stanford Univerity · 2NVIDIA

Camera-Conditioned Video GenerationVideo Diffusion ModelsAutonomous DrivingDynamic Novel View SynthesisNovel View SynthesisMultimodal Sensor Fusion
AskPDF
Apr 1, 2026

ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generation

We present ReinDriveGen, a framework that enables full controllability over dynamic driving scenes, allowing users to freely edit actor trajectories to simulate safety-critical corner cases such as front-vehicle collisions, drifting cars, vehicles spinning out of control, pedestrians jaywalking, and cyclists cutting across lanes. Our approach constructs a dynamic 3D point cloud scene from multi-frame LiDAR data, introduces a vehicle completion module to reconstruct full 360° geometry from partial observations, and renders the edited scene into 2D condition images that guide a video diffusion model to synthesize realistic driving videos. Since such edited scenarios inevitably fall outside the training distribution, we further propose an RL-based post-training strategy with a pairwise preference model and a pairwise reward mechanism, enabling robust quality improvement under out-of-distribution conditions without ground-truth supervision. Extensive experiments demonstrate that ReinDriveGen outperforms existing approaches on edited driving scenarios and achieves state-of-the-art results on novel ego viewpoint synthesis.

Hao Zhang, Lue Fan, Weikang Bian +4

MMLab, CUHK · CASIA · SenseTime Research

Autonomous Driving Scenario GenerationControllable Video GenerationRL for Generative Models
AskPDF