PIS3R: Very Large Parallax Image Stitching via Deep 3D Reconstruction
Authors: Muhua Zhu, Xinhao Jin, Chengbo Wang, Yongcong Zhang, Yifei Xue, Tie Ji, Yizhen Lao
Organizations: M. Zhu, X. Jin, C. Wang, Y. Xue, T. Ji, and Y. Lao are with Hunan University, Changsha 410082, China. · Y. Zhang is with Institut Pascal, Université Clermont Auvergne / CNRS, Clermont-Ferrand, France.
Image stitching aim to align two images taken from different viewpoints into one seamless, wider image. However, when the 3D scene contains depth variations and the camera baseline is significant, noticeable parallax occurs-meaning the relative positions of scene elements differ substantially between views. Most existing stitching methods struggle to handle such images with large parallax effectively. To address this challenge, in this paper, we propose an image stitching solution called PIS3R that is robust to very large parallax based on the novel concept of deep 3D reconstruction. First, we apply visual geometry grounded transformer to two input images with very large parallax to obtain both intrinsic and extrinsic parameters, as well as the dense 3D scene reconstruction. Subsequently, we reproject reconstructed dense point cloud onto a designated reference view using the recovered camera parameters, achieving pixel-wise alignment and generating an initial stitched image. Finally, to further address potential artifacts such as holes or noise in the initial stitching, we propose a point-conditioned image diffusion module to obtain the refined result.Compared with existing methods, our solution is very large parallax tolerant and also provides results that fully preserve the geometric integrity of all pixels in the 3D photogrammetric context, enabling direct applicability to downstream 3D vision tasks such as SfM. Experimental results demonstrate that the proposed algorithm provides accurate stitching results for images with very large parallax, and outperforms the existing methods qualitatively and quantitatively.
Figures & tables
Figure 1: PIS3R for very large parallax image stitching. (a) The image stitching task can be classified (Sec. C in Appendix) into three cases based on the baseline between target images: pure rotation , slight parallax , and very large parallax . In this paper, we propose a novel image stitching solution named PIS3R, specifically for stitching images with very large parallax . (b) Representative existing image stitching methods [ 45 , 15 , 29 ] are unable to handle stitching tasks involving very large parallax . Instead, our solution is achieving promising results in such challenging scenario.
Method
Adaptive Warping
Seam Driven
Deep Learning
APAP [ 45 ]
DFW [ 16 ]
ELA [ 15 ]
OBJ-GSP [ 6 ]
PTIS [ 46 ]
SEAGULL [ 19 ]
UDIS [ 28 ]
UDIS++ [ 29 ]
PIS3R
TPAMI’14
ICIP’17
TMM’18
AAAI’25
CVPR’14
ECCV’16
TIP’21
ICCV’23
Pure rotation
✓
✓
✓
✓
✓
✓
✓
✓
✓
Slight parallax
✓
✓
✓
✓
✓
✓
✓
✓
Very large parallax
✓
Table 1: Comparison of the proposed method PIS3R, vs. the state-of-the-art parallax-tolerate solutions. Although existing methods can generate stitched images when processing VLP cases, they suffer from significant distortions and artifacts (Fig. 1 ). As a result, they fail to achieve visual pleasantness and maintain structural integrity. Therefore, we argue that current approaches cannot effectively handle VLP scenarios.
Figure 2: The relationship between quality of stitched image and parallax variation. Our PIS3R method maintains significantly higher stability than UDIS++ across all three metrics (PSNR, SSIM, and LPIPS). As Parallax level PL (Sec. C, Appendix) increases, stitched image quality from UDIS++ degrades substantially, whereas PIS3R demonstrates superior robustness to parallax variations. Moreover, PIS3R delivers comparable visual quality to state-of-the-art methods under pure rotation and slight parallax.
Figure 3: Overview of the PIS3R pipeline. Given a sparse set of input images, we first construct a dense point cloud representation Pw using a feed-forward deep 3D reconstruction model. Subsequently, the point cloud is reprojected onto the input camera poses pi to generate preliminary stitching images S^i . While these images preserve the majority of the scene’s structural information, they concurrently introduce substantial noise artifacts. To address this limitation, we train a denoising diffusion model to restore image fidelity, ultimately producing refined stitching images Si .
Figure 4: Visual Comparison of 3D Reconstruction Results between COLMAP and VGGT. Reconstruction is performed under three distinct camera pose scenarios: pure rotation, slight parallax, and very large parallax. It can be observed that COLMAP produces significantly sparser point clouds across all scenarios, capturing limited structural information. In contrast, VGGT generates dense point clouds that preserve the majority of scene details.
Figure 5: Qualitative comparison of different image restoration diffusion models. We evaluated the performance of several state-of-the-art image restoration diffusion models in refining reprojected images, including DiffIR [ 43 ] , DDRM [ 10 ] , and RDDM [ 23 ] .
Figure 6: The results of MetaShape reconstruction using stitched images. When used as inputs for 3D reconstruction in MetaShape, images stitched with APAP and ELA resulted in a large number of surface holes. Images stitched with UDIS++ and OBJ-GSP exhibited significant distortions to geometric structure. In contrast, reconstructions generated from our stitched images consistently exhibited high geometric consistency and completeness within MetaShape. This enhanced structural integrity suggests that PIS3R is more suitable for downstream 3D reconstruction tasks.
Figure 7: Sample images from the experimental dataset. (a) Synthetic dataset rendered by Blender [ 5 ] . (b) Real-world dataset captured with a mobile phone. For each camera pose, the limited-FoV input image is center-cropped from its corresponding wide-FoV GT image and therefore shares the same camera extrinsics. Each input pair, however, consists of images acquired at two different camera poses. Consequently, the parallax is caused by relative camera translation and scene-depth variation rather than by cropping or FoV differences.
Figure 8: Qualitative comparison of stitched images on synthetic dataset. Our PIS3R generates images with significantly more complete structures and fewer severe distortions compared to existing image stitching algorithms APAP, ELA, OBJ-GSP and UDIS++ on the synthetic datasets. Ref./Tar. Img. refers to Reference and Target Images.
Figure 9: Qualitative comparison of stitched images on real-world dataset. Our PIS3R generates images with significantly more complete structures and fewer severe distortions compared to existing image stitching algorithms APAP, ELA, OBJ-GSP and UDIS++ on the real-world datasets. Ref./Tar. Img. refers to Reference and Target Images.
Figure 10: Qualitative comparison of stitched images on UDIS-D dataset. Even without training on the UDIS-D dataset, PIS3R remains visually comparable to various state-of-the-art methods.
Dataset
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Synthetic Dataset
APAP
19.116
0.659
0.169
ELA
19.813
0.670
0.158
UDIS++
19.858
0.690
0.139
OBJ-GSP
16.961
0.538
0.205
PIS3R
22.026
0.782
0.110
Real World Dataset
APAP
16.764
0.525
0.246
Table 2: Quantitative comparison of image stitching on synthetic dataset and real world dataset. PIS3R outperforms the other four methods across PSNR, SSIM and LPIPS when all image pairs are included in the evaluation.
Dataset
Method
PIQE ↓
NIQE ↓
UDIS-D
APAP
44.58
4.383
ELA
47.87
4.152
UDIS++
43.79
4.098
OBJ-GSP
56.96
4.885
PIS3R
44.13
4.059
Table 3: Quantitative comparison of image stitching on the UDIS-D [ 28 ] dataset. PIS3R performs comparably to UDIS++ and both outperform other methods.
APAP
ELA
UDIS++
OBJ-GSP
PIS3R (w/o Refine)
PIS3R(Full)
dSam2↓
22.78
38.97
12.94
16.43
2.87
4.12
Table 4: Sampson epipolar errors on the synthetic dataset. PIS3R (Full) denotes the final output after RDDM refinement, whereas PIS3R w/o Refine denotes the directly reprojected result.
Dataset
Model
PSNR ↑
SSIM ↑
LPIPS ↓
Synthetic Dataset
Dust3R
21.872
0.769
0.134
Mast3R
21.814
0.780
0.126
VGGT
22.754
0.794
0.121
Real World Dataset
Dust3R
18.134
0.617
0.239
Mast3R
18.965
0.636
0.221
VGGT
20.134
0.652
0.207
Table 5: Ablation of feed-forward 3D reconstruction models on the synthetic and real-world datasets. Metrics are computed on preliminary reprojected images before diffusion refinement.
Figure 11: Qualitative comparison of different feed-forward deep 3D reoconstruction models including VGGT [ 39 ] , MAST3R [ 14 ] and DUST3R [ 40 ] . VGGT demonstrates visually superior reconstruction results.
Dataset
Model
PSNR ↑
SSIM ↑
LPIPS ↓
Synthetic Dataset
w/o Refine
22.754
0.794
0.121
DiffIR
21.361
0.740
0.120
DDRM
20.894
0.706
0.132
RDDM
22.026
0.782
0.110
Real World Dataset
w/o Refine
20.134
0.652
0.207
DiffIR
18.744
0.624
0.193
Table 6: Ablation study of diffusion models on the synthetic and real-world datasets. RDDM improves perceptual similarity, as measured by LPIPS, while reducing PSNR and SSIM, indicating a trade-off between pixel-level fidelity and perceptual quality.
Resolution
1280×720
720×480
512×512
APAP 1
3.46
1.63
1.29
ELA 1
5.69
2.91
2.61
OBJ-GSP 1
4.26
2.10
1.61
UDIS++ 2
0.35
0.20
0.18
PIS3R 2
6.41
3.38
2.91
Table 7: Runtime (s) comparison at three input resolutions. 1: tested with Intel i9-14900KF 3.20GHZ CPU; 2: tested with NVIDIA RTX 4090 GPU.
Figure 12: User-study preference rates from ten participants evaluating ten image pairs per dataset. GT was additionally shown in the “with GT” settings. (a) Comparison with existing methods. (b) Refinement ablation.
Prior panorama stitching approaches heavily rely on pairwise feature correspondences and are unable to leverage geometric consistency across multiple views. This leads to severe distortion and misalignment, especially in challenging scenes with weak textures, large parallax, and repetitive patterns. Given that multi-view geometric correspondences can be directly constructed in 3D space, making them more accurate and globally consistent, we extend the 2D alignment task to the 3D photogrammetric space. We adopt a novel transformer-based architecture to achieve 3D awareness and aggregate global information across all views. It directly utilizes camera poses to guide image warping for global alignment in 3D space and employs a multi-feature joint optimization strategy to compute the seams. Additionally, to establish an evaluation benchmark and train our network, we constructed a large-scale dataset of real-world scenes. Extensive experiments show that our method significantly outperforms existing alternatives in alignment accuracy and perceptual quality.
Zhengdong Zhu, Weiyi Xue, Zuyuan Yang +2
South China University of Technology · Tongji University · Guangdong University of Technology
Panoramic imagery offers a full 360° field of view and is increasingly common in consumer devices. However, it introduces non-pinhole distortions that challenge joint pose estimation and 3D reconstruction. Existing feed-forward models, built for perspective cameras, generalize poorly to this setting. We propose PanoVGGT, a permutation-equivariant Transformer framework that jointly predicts camera poses, depth maps, and 3D point clouds from one or multiple panoramas in a single forward pass. The model incorporates spherical-aware positional embeddings and a panorama-specific three-axis SO(3) rotation augmentation, enabling effective geometric reasoning in the spherical domain. To resolve inherent global-frame ambiguity, we further introduce a stochastic anchoring strategy during training. In addition, we contribute PanoCity, a large-scale outdoor panoramic dataset with dense depth and 6-DoF pose annotations. Extensive experiments on PanoCity and standard benchmarks demonstrate that PanoVGGT achieves competitive accuracy, strong robustness, and improved cross-domain generalization. Code and dataset will be released.
Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view. A natural extension is 3D reconstruction from panoramas, with the exciting prospect of recovering a full 360-degree scene from a single panoramic image. In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain. Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images, in a single forward pass. By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360-degree scenes from single panoramas. We extensively test our method in both indoor and outdoor environments and find that it delivers state-of-the-art performance and excellent zero-shot performance across a wide range of scenes. Code, data and models are available here.