Wide View Synthesis from Overlapping Perspective Cameras in Autonomous Driving

Kaixuan Zhang; Zhipeng Xiong; Minxian Li; Mingwu Ren; Xiatian Zhu

cs.CVNov 4, 2026

Wide View Synthesis from Overlapping Perspective Cameras in Autonomous Driving

Authors: Kaixuan Zhang, Zhipeng Xiong, Minxian Li, Mingwu Ren, Xiatian Zhu

Organizations: Nanjing University of Science and Technology · University of Surrey

Abstract

Multi-perspective-camera systems are widely adopted since they provide broad visual coverage for autonomous driving, motivating the synthesis of a unified wide view from spatially separated observations. In this work, we formulate wide view synthesis (WVS), which aims to synthesize a geometrically consistent wide view at a specified virtual viewpoint from multiple perspective cameras. Unlike conventional image stitching, WVS requires reasoning about 3D geometry and visibility to handle depth-dependent parallax and occlusions arising from spatially separated camera centers. While vanilla 3D Gaussian Splatting (3DGS) enables high-fidelity view synthesis, its reliance on per-scene optimization limits its applicability to driving scenes. Furthermore, WVS introduces additional challenges, e.g., peripheral stretching under perspective projection and photometric inconsistencies across cameras. To address these challenges, we propose Wide-View Gaussian Splatting (WV-GS), a feed-forward Gaussian-splatting framework that synthesizes wide views without per-scene optimization. Specifically, we introduce Piecewise Perspective Rendering (PPR) to mitigate peripheral stretching through local perspective projections sharing a common camera center, and Spatial Affine Photometric Modeling (SAPM) to account for camera-dependent photometric variations during training. We further introduce WideDrive, a synthetic benchmark providing dense ground-truth wide views and adapt a LiDAR-projected sparse-reference protocol to real-world datasets. Extensive experiments on multiple datasets demonstrate the effectiveness of WV-GS for WVS. Dataset and code will be publicly released.

Figures & tables

Figure 2: Stitching images of different methods. (a)–(c) Input images captured by three perspective cameras with limited overlap. (d)–(g) Results obtained using REwarp [ 21 ] , DSFN [ 19 ] , RopStitch [ 33 ] , and UniStitch [ 28 ] , respectively, showing severe geometric distortions and visible artifacts.

Figure 3: WF-GS overview. A feed-forward backbone predicts $\mathcal{G}_{t}$ from the input context $\mathcal{C}_{t}$ . (a) Training: SAPM maps canonical renders $\widehat{\mathbf{I}}_{i,t}$ to corrected renders $\widetilde{\mathbf{I}}_{i,t}$ , which are compared with observed images $\mathbf{I}_{i,t}$ . Reference depths $D^{\mathrm{ref}}_{i,t}$ supervise predicted depths $\widehat{D}_{i,t}$ through $\mathcal{L}_{\mathrm{dep}}$ , while $\mathcal{L}_{\mathrm{con}}$ couples predicted and rasterized depths $\widehat{D}^{\mathrm{rend}}_{i,t}$ . (b) Inference: PPR composes local canonical renders into the wide-FOV image $\widehat{\mathbf{I}}^{\mathrm{ppr}}_{w,t}$ .

Figure 4: Architecture of SAPM. SAPM maps canonical renders

\widehat{\mathbf{I}}_{i,t}

to camera-specific predictions

\widetilde{\mathbf{I}}_{i,t}

during training. Blue and orange blocks denote

3\times 3

and

1\times 1

convolutions, respectively. Concat: channel concatenation; Up: bilinear upsampling;

\odot

and

\oplus

: element-wise multiplication and addition.

Figure 5: Illustration of PPR. Conceptual schematic of PPR illustrated with

M=3

local perspective planes

\Pi_{m}

(

m\in\{1,2,3\}

). All planes share the target optical center

\mathbf{o}_{w,t}

and cover adjacent angular sub-regions. Their canonical renders

\widehat{\mathbf{I}}^{m}_{w,t}

are concatenated horizontally into the wide-FOV output

\widehat{\mathbf{I}}^{\mathrm{ppr}}_{w,t}

Figure 6: Visual comparison of WVS on WideDrive. (a)–(c) Input images from three overlapping perspective cameras. (d)–(l) Wide-FOV outputs from StabStitch++ [ 32 ] , UDIS2 [ 31 ] , DSFN [ 19 ] , APAP [ 39 ] , REwarp [ 21 ] , RopStitch [ 33 ] , AutoStitch [ 2 ] , UniStitch [ 28 ] , and WV-GS (ours), respectively.

Non-learnable
Method	HM-PSNR $\uparrow$	HM-SSIM $\uparrow$	HM-MAE $\downarrow$	HM-RMSE $\downarrow$	Success (%) $\uparrow$
AutoStitch	15.92	0.898	39.62	53.40	57.69
APAP	14.56	0.876	47.95	59.26	0.01
Learnable
UDIS2	17.04	0.649	37.47	54.96	0.02
REwarp	14.67	0.796	53.81	66.59	8.73

Table 1: Results on nuScenes [ 3 ] with the best and second-best results highlighted.

Non-learnable
Method	HM-PSNR $\uparrow$	HM-SSIM $\uparrow$	HM-MAE $\downarrow$	HM-RMSE $\downarrow$	Success (%) $\uparrow$
AutoStitch	13.85	0.865	39.48	60.59	96.11
APAP	11.21	0.776	69.88	90.00	0.24
Learnable
UDIS2	12.42	0.819	55.44	76.43	72.78
REwarp	10.53	0.759	72.39	92.90	7.46

Table 2: Results on Lyft [ 26 ] with the best and second-best results highlighted.

Non-learnable
Method	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	LPIPS $\downarrow$	Success (%) $\uparrow$
AutoStitch	11.04	0.494	69.97	89.79	0.463	79.62
APAP	11.96	0.539	48.96	68.87	0.441	37.62
Learnable
UDIS2	15.21	0.605	28.97	46.12	0.366	98.20
REwarp	6.67	0.420	83.98	123.03	0.558	19.28

Table 3: Results on WideDrive with the best and second-best results highlighted.

Figure 7: Visual comparison of WVS on nuScenes [ 3 ] . (a)–(c) Input images from three overlapping perspective cameras. (d)–(j) Wide-FOV outputs from RopStitch [ 33 ] , REwarp [ 21 ] , UniStitch [ 28 ] , StabStitch++ [ 32 ] , DSFN [ 19 ] , AutoStitch [ 2 ] , and WV-GS (ours), respectively.

Figure 8: Visual comparison of the SAPM ablation. (a)–(c): input images. (d)–(f) correspond to WV-GS w/o SAPM, WV-GS w/o SAPM + OpenCV and full WV-GS. Without SAPM, the synthesized wide views inherit the per-camera color and exposure offsets, which remain visible as brightness discontinuities and color seams in the overlapping regions, while the OpenCV-based preprocessing only partially suppresses them. SAPM instead models the camera-dependent photometric response during training and yields a consistent appearance across the whole wide field of view.

Settings	HM-PSNR $\uparrow$	HM-SSIM $\uparrow$	HM-MAE $\downarrow$	HM-RMSE $\downarrow$
w/o SAPM	23.32	0.979	9.66	18.47
w/o SAPM + OpenCV	23.45	0.978	9.71	18.31
w/o PPR	22.42	0.969	11.96	20.97
Full (Ours)	23.76	0.977	9.62	17.69

Table 4: Ablation on core modules. “w/o SAPM”: trained without photometric modeling; “w/o SAPM + OpenCV”: SAPM replaced by OpenCV-based color preprocessing; “w/o PPR”: a single wide-angle perspective projection replaces PPR.

M	$\theta$	HM-PSNR $\uparrow$	HM-SSIM $\uparrow$	HM-MAE $\downarrow$	HM-RMSE $\downarrow$
3	$\sim 80^{\circ}$	23.76	0.977	9.62	17.69
5	$\sim 45^{\circ}$	23.42	0.972	10.91	19.19
7	$\sim 35^{\circ}$	23.42	0.971	11.10	19.37
9	$\sim 25^{\circ}$	23.48	0.971	11.12	19.35

Table 5: Ablation on the number of projection planes

M

in PPR.

L	HM-PSNR $\uparrow$	HM-SSIM $\uparrow$	HM-MAE $\downarrow$	HM-RMSE $\downarrow$
1	23.35	0.976	10.20	18.46
2	23.50	0.977	9.94	18.14
3	23.76	0.977	9.62	17.69
4	23.58	0.977	9.83	17.97
5	23.54	0.977	9.87	18.06

Table 6: Ablation on the temporal context length

L

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Dataset composition of WideDrive. WideDrive contains both daytime and night time scenes for better diversity. At each timestamp, we capture three perspective images (a)–(c)/(e)–(g) from spatially separated cameras, together with a wide-view image (d)/(h) that serves as the ground truth. Images (a)/(c)/(e)/(g) are additionally rendered with an exposure offset of $-1$ EV, emulating the auto-exposure mismatch of real cameras. LiDAR point clouds are also recorded but are omitted here for clarity.

Dataset	Type	Source cameras	Input FOV (overlap)	Exposure mismatch	Wide-view GT	Eval. protocol
nuScenes [ 3 ]	Real	3 persp. ( $1600\times 900$ )	$\sim 140^{\circ}$ ( $\sim 20^{\circ}$ )	Natural	$\times$ (Sparse LiDAR)	LPSR (Sparse)
Lyft [ 26 ]	Real	3 persp. ( $1920\times 1080$ )	$\sim 140^{\circ}$ ( $\sim 30^{\circ}$ )	Natural	$\times$ (Sparse LiDAR)	LPSR (Sparse)
WideDrive (Ours)	Synthetic	3 persp. ( $1920\times 1080$ )	$132^{\circ}$ ( $24^{\circ}$ / 40%)	Controlled ( $-1$ EV)	✓ ( $120^{\circ}$ Dense)	Dense (Full Image)

Appendix

Table 7: Comparison of benchmark datasets for WF-NVS. Unlike existing real-world driving benchmarks that lack ground-truth wide-angle perspective captures, WideDrive provides native, dense, seam-free wide-view ground-truth images alongside calibrated perspective source views and realistic cross-camera exposure offsets.

Figure 10: Illustration of the LPSR protocol. Given the captured perspective images (a)–(c) and their calibrated LiDAR sweeps, we project the colored LiDAR point cloud onto (d) a single wide-angle perspective imaging plane and (e) our $M$ piecewise perspective planes of PPR, yielding a sparse reference panel and its validity mask on which all metrics are computed. Only pixels hit by projected points are evaluated; black regions carry no LiDAR evidence and are excluded from the scores.

Figure 11: Visual comparison of WVS on Lyft [ 26 ] . (a)–(c) Input images from three overlapping perspective cameras. (d)–(k) Wide-view outputs from DSFN [ 19 ] , RopStitch [ 33 ] , UDIS2 [ 31 ] , REwarp [ 21 ] , StabStitch++ [ 32 ] , AutoStitch [ 2 ] , UniStitch [ 28 ] , and WF-GS (ours), respectively. APAP [ 39 ] fails on this scene.

Figure 12: Rendering results of WVS with different number of projection planes $M$ in PPR. (a)–(c) Input images. (d)–(g) Images rendered by WF-GS with $M=3/5/7/9$ .

Multi-View Image Synthesis

See all 31 papers

VisionNVS: Self-Supervised Inpainting for Novel View Synthesis under the Virtual-Shift Paradigm
Mar 18, 2026Hongbo Lu, Liang Yao, Chenghao He +4Novel View Synthesis Autonomous Driving Simulation
Shanghai Jiao Tong University · COWARobot Co. Ltd. · Hohai University
Quo Vadis, Driving Scene Reconstruction? Occlusion-Aware Gaussian Densification
Nov 4, 2026Dung Do Tien, Kien Nguyen Do Trung, Kiet Dang Vu +23D Reconstruction 3D Gaussian Splatting
AITech Lab., Ho Chi Minh City University of Technology (HCMUT) · Vietnam National University Ho Chi Minh City (VNUHCM)
ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction
Mar 8, 2026Haibao Yu, Kuntao Xiao, Jiahang Wang +7Dynamic Scene Reconstruction 3D Gaussian Splatting
The University of Hong Kong · Tuojing Intelligence · King’s College London +2

Mar 18, 2026J/K move · Enter open · S save

VisionNVS: Self-Supervised Inpainting for Novel View Synthesis under the Virtual-Shift Paradigm

Hongbo Lu, Liang Yao, Chenghao He, Fan Liu+3

Shanghai Jiao Tong University · COWARobot Co. Ltd. · Hohai University

Novel View Synthesis Autonomous Driving Simulation Autonomous Driving

Open paper AskPDF arXiv

A fundamental bottleneck in Novel View Synthesis (NVS) for autonomous driving is the inherent supervision gap on novel trajectories: models are tasked with synthesizing unseen views during inference, yet lack ground truth images for these shifted poses during training. In this paper, we propose VisionNVS, a camera-only framework that fundamentally reformulates view synthesis from an ill-posed extrapolation problem into a self-supervised inpainting task. By introducing a ``Virtual-Shift'' strategy, we use monocular depth proxies to simulate occlusion patterns and map them onto the original view. This paradigm shift allows the use of raw, recorded images as pixel-perfect supervision, effectively eliminating the domain gap inherent in previous approaches. Furthermore, we address spatial consistency through a Pseudo-3D Seam Synthesis strategy, which integrates visual data from adjacent cameras during training to explicitly model real-world photometric discrepancies and calibration errors. Experiments demonstrate that VisionNVS achieves superior geometric fidelity and visual quality compared to LiDAR-dependent baselines, offering a robust solution for scalable driving simulation.

Abstract

Multi-View Image Synthesis

Explore similar work

VisionNVS: Self-Supervised Inpainting for Novel View Synthesis under the Virtual-Shift Paradigm

Quo Vadis, Driving Scene Reconstruction? Occlusion-Aware Gaussian Densification

ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction