A fundamental bottleneck in Novel View Synthesis (NVS) for autonomous driving is the inherent supervision gap on novel trajectories: models are tasked with synthesizing unseen views during inference, yet lack ground truth images for these shifted poses during training. In this paper, we propose VisionNVS, a camera-only framework that fundamentally reformulates view synthesis from an ill-posed extrapolation problem into a self-supervised inpainting task. By introducing a ``Virtual-Shift'' strategy, we use monocular depth proxies to simulate occlusion patterns and map them onto the original view. This paradigm shift allows the use of raw, recorded images as pixel-perfect supervision, effectively eliminating the domain gap inherent in previous approaches. Furthermore, we address spatial consistency through a Pseudo-3D Seam Synthesis strategy, which integrates visual data from adjacent cameras during training to explicitly model real-world photometric discrepancies and calibration errors. Experiments demonstrate that VisionNVS achieves superior geometric fidelity and visual quality compared to LiDAR-dependent baselines, offering a robust solution for scalable driving simulation.
Figures & tables
Fig. 1: Overview of VisionNVS. Our camera-only framework reformulates novel view synthesis as self-supervised inpainting, trained via Flow Matching with a DiT backbone and the Wan2.2 [ 22 ] streaming VAE for efficient encoding. As shown in the bottom data pipeline, we construct a “Pseudo Image” condition by masking the raw image based on virtual shift occlusions and filling gaps with neighbor views. The model (top) then learns to recover the original, artifact-free raw image from this degraded condition.
Method
Shift ± 1m
Shift ± 2m
Shift ± 4m
FID ↓
FVD ↓
FID ↓
FVD ↓
FID ↓
FVD ↓
PVG [ 3 ]
48.15
246.74
60.44
356.23
84.50
501.16
EmerNeRF [ 24 ]
37.57
171.47
52.03
294.55
76.11
497.85
StreetGaussain [ 10 ]
32.12
153.45
43.24
256.91
67.44
429.98
OmniRe [ 4 ]
31.48
152.01
43.31
254.52
67.36
428.20
FreeVS [ 23 ]
51.26
431.99
62.04
497.37
77.14
556.14
TABLE I: Comparison with state-of-the-art methods on the nuScenes. bold and underlined , respectively. ”-” indicates that the metric was not reported.
Virtual-Shift Strategy
Pseudo-3D Seam Synthesis
FID↓
FVD↓
21.76
56.80
✓
14.34
46.68
✓
✓
11.98
46.27
TABLE II: Ablation study of component effectiveness.
Fig. 2: Comparison of computational efficiency. We profile memory usage and inference time under different input lengths N .
Fig. 3: Qualitative comparison of temporal consistency. We visualize consecutive frames ( t−2 to t+2 ) synthesized by DiST-4D [ 7 ] and Ours. The orange and red boxes highlight specific regions of interest. DiST-4D exhibits severe temporal instability, with building structures distorting (top) and tree details flickering or vanishing (bottom) across frames. In contrast, Ours maintains remarkable geometric stability and visual coherence, preserving object details without artifacts.
Fig. 4: Visual comparison with state-of-the-art DiST-4D [ 7 ] . Despite DiST-4D utilizing expensive LiDAR priors, dense metric depth supervision, and complex Cycle Consistency (SCC) strategies, our camera-only VisionNVS achieves comparable visual fidelity and geometric consistency without employing any of these additional tricks.
3D Gaussian Splatting (3DGS) has emerged as a powerful technique for real-time LiDAR and camera synthesis in autonomous driving simulation. However, simulating LiDAR with 3DGS remains challenging for extrapolated views beyond the training trajectory, as existing methods are typically trained on single-traversal sensor scans, suffer from severe overfitting and poor generalization to novel ego-vehicle paths. To enable reliable simulation of LiDAR along unseen driving trajectories without external multi-pass data, we present LiDAR-EVS, a lightweight framework for robust extrapolated-view LiDAR simulation in autonomous driving. Designed to be plug-and-play, LiDAR-EVS readily extends to diverse LiDAR sensors and neural rendering baselines with minimal modification. Our framework comprises two key components: (1) pseudo extrapolated-view point cloud supervision with multi-frame LiDAR fusion, view transformation, occlusion curling, and intensity adjustment; (2) spatially-constrained dropout regularization that promotes robustness to diverse trajectory variations encountered in real-world driving. Extensive experiments demonstrate that LiDAR-EVS achieves SOTA performance on extrapolated-view LiDAR synthesis across three datasets, making it a promising tool for data-driven simulation, closed-loop evaluation, and synthetic data generation in autonomous driving systems.
Extrapolative novel view synthesis can reduce camera-rig dependency in autonomous driving by generating standardized virtual views from heterogeneous sensors. Existing methods degrade outside recorded trajectories because extrapolated poses provide weak geometric support and no dense target-view supervision. The key is to explicitly expose the model to out-of-trajectory condition defects during training. We propose Geo-EVS, a geometry-conditioned framework under sparse supervision. Geo-EVS has two components. Geometry-Aware Reprojection (GAR) uses fine-tuned VGGT to reconstruct colored point clouds and reproject them to observed and virtual target poses, producing geometric condition maps. This design unifies the reprojection path between training and inference. Artifact-Guided Latent Diffusion (AGLD) injects reprojection-derived artifact masks during training so the model learns to recover structure under missing support. For evaluation, we use a LiDAR-Projected Sparse-Reference (LPSR) protocol when dense extrapolated-view ground truth is unavailable. On Waymo, Geo-EVS improves sparse-view synthesis quality and geometric accuracy, especially in high-angle and low-coverage settings. It also improves downstream 3D detection.
Yatong Lan, Rongkui Tang, Lei He
School of Vehicle and Mobility, Tsinghua University, Beijing, China · State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University, Beijing, China · School of Science, Minzu University, Beijing, China +1
Modern vehicle platforms are equipped with a rich sensor suite, including LiDAR, calibrated multi-camera rigs, and accurate ego-motion, that in principle offers strong signal for re-rendering a driving scene from novel viewpoints. A growing line of recent work leverages video diffusion models for this task, using their generative priors to synthesize plausible novel views from sparse vehicle observations. In practice, however, existing methods exploit only a fragment of this signal, and their quality tends to degrade as the target trajectory departs from the recorded driving path. We argue that this is fundamentally a multi-sensor fusion problem: sparse LiDAR reprojections supply accurate but incomplete metric geometry, surround-view reference imagery supplies dense appearance but no metric depth, and camera poses tie the two together across views. We introduce StreetNVS, a video diffusion framework that jointly conditions on all three signals through a Reference-Enhanced Camera Attention module based on a relative ray-level positional encoding. We develop a two-stage curriculum training strategy that gradually exposes the model to increasingly sparse LiDAR. On the Waymo Open Dataset, StreetNVS substantially outperforms state-of-the-art baselines under sparse LiDAR conditioning, matches methods that rely on 10-100 times denser point clouds. We further show capabilities of synthesizing coherent videos along extreme out-of-trajectory paths such as elevation, lane-shift, pullback, and rotation. Our website: https://streetnvs.github.io