VisionNVS: Self-Supervised Inpainting for Novel View Synthesis under the Virtual-Shift Paradigm
Organizations: Shanghai Jiao Tong University · COWARobot Co. Ltd. · Hohai University
Abstract
A fundamental bottleneck in Novel View Synthesis (NVS) for autonomous driving is the inherent supervision gap on novel trajectories: models are tasked with synthesizing unseen views during inference, yet lack ground truth images for these shifted poses during training. In this paper, we propose VisionNVS, a camera-only framework that fundamentally reformulates view synthesis from an ill-posed extrapolation problem into a self-supervised inpainting task. By introducing a ``Virtual-Shift'' strategy, we use monocular depth proxies to simulate occlusion patterns and map them onto the original view. This paradigm shift allows the use of raw, recorded images as pixel-perfect supervision, effectively eliminating the domain gap inherent in previous approaches. Furthermore, we address spatial consistency through a Pseudo-3D Seam Synthesis strategy, which integrates visual data from adjacent cameras during training to explicitly model real-world photometric discrepancies and calibration errors. Experiments demonstrate that VisionNVS achieves superior geometric fidelity and visual quality compared to LiDAR-dependent baselines, offering a robust solution for scalable driving simulation.
Figures & tables
| Method | Shift 1m | Shift 2m | Shift 4m | |||
|---|---|---|---|---|---|---|
| FID | FVD | FID | FVD | FID | FVD | |
| PVG [ 3 ] | 48.15 | 246.74 | 60.44 | 356.23 | 84.50 | 501.16 |
| EmerNeRF [ 24 ] | 37.57 | 171.47 | 52.03 | 294.55 | 76.11 | 497.85 |
| StreetGaussain [ 10 ] | 32.12 | 153.45 | 43.24 | 256.91 | 67.44 | 429.98 |
| OmniRe [ 4 ] | 31.48 | 152.01 | 43.31 | 254.52 | 67.36 | 428.20 |
| FreeVS [ 23 ] | 51.26 | 431.99 | 62.04 | 497.37 | 77.14 | 556.14 |
| Virtual-Shift Strategy | Pseudo-3D Seam Synthesis | ||
|---|---|---|---|
| 21.76 | 56.80 | ||
| ✓ | 14.34 | 46.68 | |
| ✓ | ✓ | 11.98 | 46.27 |