PanoVGGT: Feed-Forward 3D Reconstruction from Panoramic Imagery

Yijing Guo; Mengjun Chao; Luo Wang; Tianyang Zhao; Haizhao Dai; Yingliang Zhang; Jingyi Yu; Yujiao Shi

cs.CVMar 18, 2026

PanoVGGT: Feed-Forward 3D Reconstruction from Panoramic Imagery

Authors: Yijing Guo, Mengjun Chao, Luo Wang, Tianyang Zhao, Haizhao Dai, Yingliang Zhang, Jingyi Yu, Yujiao Shi

Organizations: ShanghaiTech University · Sudo

Abstract

Panoramic imagery offers a full 360° field of view and is increasingly common in consumer devices. However, it introduces non-pinhole distortions that challenge joint pose estimation and 3D reconstruction. Existing feed-forward models, built for perspective cameras, generalize poorly to this setting. We propose PanoVGGT, a permutation-equivariant Transformer framework that jointly predicts camera poses, depth maps, and 3D point clouds from one or multiple panoramas in a single forward pass. The model incorporates spherical-aware positional embeddings and a panorama-specific three-axis SO(3) rotation augmentation, enabling effective geometric reasoning in the spherical domain. To resolve inherent global-frame ambiguity, we further introduce a stochastic anchoring strategy during training. In addition, we contribute PanoCity, a large-scale outdoor panoramic dataset with dense depth and 6-DoF pose annotations. Extensive experiments on PanoCity and standard benchmarks demonstrate that PanoVGGT achieves competitive accuracy, strong robustness, and improved cross-domain generalization. Code and dataset will be released.

Figures & tables

Dataset	#Images	Scene	Pose	Depth	Limitations / Characteristics
PanoSUNCG [ 26 ]	$\sim$ 25K	Indoor	✓ (6DoF)	✓ (8-bit)	Deprecated and no longer legally available
Matterport3D [ 5 ]	$\sim$ 10.8K	Indoor	✓ (6DoF)	✓ (16-bit)	Discrete panoramas with limited overlap
Structured3D [ 40 ]	$\sim$ 12K	Indoor	✓ (3DoF)	✓ (16-bit)	Lack rotation annotations and no image overlap
Stanford2D3D [ 3 ]	$\sim$ 1.4K	Indoor	✓ (6DoF)	✓ (16-bit)	Small-scale dataset with limited overlap
Pano3D [ 2 ]	$\sim$ 42K	Mixed	×	✓ (16-bit)	No camera pose annotations
IndoorLoc [ 15 ]	$\sim$ 0.7K	Indoor	✓ (6DoF)	✓ (16-bit)	RGB–depth inconsistencies and small scale

Table 1: Comparison of PanoCity with prior panoramic datasets (✓: available; ×: unavailable). Unlike existing datasets that are limited in scale, annotation completeness, or multi-view continuity, our dataset provides large-scale, high-fidelity, trajectory-based multi-view panoramas with complete 6DoF pose and 16-bit depth supervision.

Figure 2: Overview of the proposed PanoVGGT framework. Given one or multiple panoramic images, PanoVGGT extracts spherical patch tokens using a ViT backbone, augments them via $SO(3)$ geometric augmentation, and encodes them with shared and branch-adapted positional embeddings. A multi-branch attention aggregator jointly reasons about local geometry, global structure, and camera motion. The model outputs dense depth maps, camera poses (under a randomly selected global anchor), and local/global point maps for final 3D reconstruction.

Method	Matterport3D (Indoor)					Stanford2D3D (Indoor)					PanoCity (Outdoor)
Method	AUC@30 $\uparrow$	Rotation ( ${}^{\circ}\!\downarrow$ )		Translation ( ${}^{\circ}\!\downarrow$ )		AUC@30 $\uparrow$	Rotation ( ${}^{\circ}\!\downarrow$ )		Translation ( ${}^{\circ}\!\downarrow$ )		AUC@30 $\uparrow$	Rotation ( ${}^{\circ}\!\downarrow$ )		Translation ( ${}^{\circ}\!\downarrow$ )
		Mean	Med	Mean	Med		Mean	Med	Mean	Med		Mean	Med	Mean	Med
Bifusev2 [ 28 ]	0.007	66.049	60.924	82.038	88.794	0.030	57.568	43.296	45.933	45.460	0.833	1.655	0.173	5.044	4.731
VGGT [ 29 ]	0.034	66.510	56.329	47.857	48.149	0.047	71.035	61.112	41.540	40.011	0.205	7.659	1.609	35.867	28.796
$\pi^{3}$ [ 35 ]	0.047	63.286	54.292	44.201	42.750	0.076	58.727	45.378	39.567	35.652	0.571	7.669	1.020	16.780	7.768
$\pi^{3}$ *	0.305	25.205	25.205	31.550	31.550	0.274	19.491	24.173	33.516	27.871	0.682	0.525	0.474	22.379	15.727

Table 2: Camera pose estimation results. Bold = best, underline = second best.

Model	Input	Matterport3D (Indoor)		Stanford2D3D (Indoor)		Structured3D (Indoor)		PanoCity (Outdoor)
		Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$
UniFuse [ 11 ]	Monocular	0.1367	0.8314	0.1390	0.8270	0.0387	0.9804	1.0374	0.4365
BiFuse++ [ 28 ]	Monocular	0.1076	0.8846	0.1120	0.8805	0.0293	0.9840	0.0200	0.9843
HoHoNet [ 24 ]	Monocular	0.1217	0.8576	0.1192	0.8872	0.0401	0.9759	0.0297	0.9885
EGFormer [ 37 ]	Monocular	0.0987	0.9082	0.0929	0.8890	0.0273	0.9889	0.0363	0.9807
Elite360D [ 1 ]	Monocular	0.1263	0.8534	0.1273	0.8642	0.1019	0.9180	0.3622	0.4171

Table 3: Monocular Depth estimation performance. Bold = best, underline = second best.

Figure 3: Multi-view point-cloud reconstructions on the Matterport3D dataset from two unordered panoramic inputs. From left to right: $\pi^{3}$ , $\pi^{3}$ *, $\pi^{3}{}^{\dagger}$ , and PanoVGGT. Here, $\dagger$ denotes the original pinhole-only $\pi^{3}$ applied to panoramic inputs via MoGe’s dodecahedral projection protocol, which decomposes each panorama into 12 perspective views. PanoVGGT produces sharper and more structurally consistent indoor reconstructions.

Method	Abs Rel ↓	RMSE ↓	$\delta_{1}$ ↑	$\delta_{2}$ ↑
VGGT † [ 29 ]	0.1266	0.3402	0.8650	0.9816
$\pi^{3}$ †	0.1196	0.3145	0.8811	0.9855
MoGe † [ 31 ]	0.1236	0.3353	0.8589	0.9797
MoGeV2 † [ 32 ]	0.1264	0.3429	0.8578	0.9799
PanDA [ 4 ]	0.3577	0.7506	0.4069	0.6245
Unik3D [ 18 ]	0.1568	0.3107	0.7989	0.9636

Table 4: Zero-shot depth estimation on Pano3D (GibsonV2) under scale-only (top) and scale+shift (bottom) alignments.

\dagger

denotes pinhole-based models evaluated on panoramas using the dodecahedral projection protocol of MoGe.

Method	Acc $\downarrow$		Comp $\downarrow$		Overall $\downarrow$
Method	Mean	Med	Mean	Med	Mean	Med
$\pi^{3}$ [ 35 ]	1.404	0.905	11.737	6.002	6.570	3.454
$\pi^{3}$ † [ 35 ]	1.261	0.995	15.182	9.578	8.222	5.287
$\pi^{3}$ *	1.111	0.707	1.037	0.592	1.074	0.650
PanoVGGT (global points)	0.768	0.179	0.705	0.149	0.737	0.164
PanoVGGT (local points)	0.744	0.180	0.756	0.204	0.750	0.192

Table 5: Point cloud reconstruction performance on PanoCity.

\dagger

denotes the perspective-based model evaluated using the dodecahedral projection protocol of MoGe.

Components	Pose		Depth
Components	AUC@30↑	R ${}_{\text{mean}}$ /T ${}_{\text{mean}}$ ↓	AbsRel↓	$\delta$ ¡1.25↑
Baseline ( $\pi^{3}$ *)	0.228	34.39 / 33.32	0.194	0.709
+ PanoCity	0.256	27.49 / 31.52	0.105	0.896
+ Tri-axis aug	0.285	22.05 / 35.04	0.116	0.871
+ Global pt. fusion	0.380	24.15 / 26.34	0.103	0.894
+ Pos. embedding (Complete)	0.427	24.53 / 18.42	0.098	0.908

Table 6: Ablation study of PanoVGGT components on Matterport3D. Bold = best, underline = second best.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Input	Matterport3D (Indoor)		Stanford2D3D (Indoor)		Structured3D (Indoor)		PanoCity (Outdoor)
		Abs Rel $\downarrow$	$\delta{<}1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta{<}1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta{<}1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta{<}1.25$ $\uparrow$
$\pi^{3}$ * [ 35 ]	Monocular	0.0940	0.9142	0.0852	0.9291	0.0652	0.9649	0.0834	0.9161
PanoVGGT (Ours)	Monocular	0.0884	0.9157	0.0711	0.9392	0.0438	0.9728	0.0312	0.9713
PanoVGGT (Ours)	Multi-view	0.0840	0.9266	0.0778	0.9323	0.0400	0.9870	0.0196	0.9812

Appendix

Table 7: Monocular depth estimation performance including the retrained panoramic variant

\pi^{3}

* [ 35 ] on Matterport3D [ 5 ] , Stanford2D3D [ 3 ] , Structured3D [ 40 ] , and PanoCity . Bold = best, underline = second best.

Method	Abs Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}$ $\uparrow$	$\delta_{2}$ $\uparrow$
$\pi^{3}$ * [ 35 ]	0.1879	0.3743	0.9085	0.9717
PanoVGGT (ours)	0.0869	0.3069	0.9223	0.9859
$\pi^{3}$ * [ 35 ]	0.1777	0.3639	0.9197	0.9759
PanoVGGT (ours)	0.0833	0.3015	0.9299	0.9894

Appendix

Table 8: Zero-shot depth results on Pano3D (GibsonV2) [ 2 , 36 ] with

\pi^{3}

* (retrained panoramic variant) [ 35 ] . The top part reports results using Scale-only alignment; the bottom part uses Scale+Shift alignment.

Method	Acc $\downarrow$		Comp $\downarrow$		Overall $\downarrow$
Method	Mean	Med	Mean	Med	Mean	Med
$\pi^{3}$ [ 35 ]	0.4027	0.3853	0.9964	0.8803	0.6995	0.6328
$\pi^{3}$ * [ 35 ]	0.2661	0.2199	0.3351	0.2378	0.3006	0.2288
$\pi^{3}$ † [ 35 ]	0.1843	0.1420	0.2647	0.1857	0.2245	0.1638
PanoVGGT (global points)	0.1743	0.1231	0.1530	0.0890	0.1636	0.1060
PanoVGGT (local points)	0.1750	0.1242	0.1530	0.0932	0.1640	0.1087

Appendix

Table 9: Point-cloud results on Matterport3D [ 5 ] .

\dagger

denotes the perspective-based model evaluated on panoramas using the dodecahedral projection protocol of MoGe.

Method	Acc $\downarrow$		Comp $\downarrow$		Overall $\downarrow$
Method	Mean	Med	Mean	Med	Mean	Med
$\pi^{3}$ [ 35 ]	0.5087	0.4667	0.9394	0.9103	0.7241	0.6885
$\pi^{3}$ * [ 35 ]	0.2590	0.2021	0.3143	0.2245	0.2866	0.2133
$\pi^{3}$ † [ 35 ]	0.2359	0.1942	0.3668	0.2870	0.3013	0.2406
PanoVGGT (global points)	0.2087	0.1752	0.2624	0.1943	0.2355	0.1848
PanoVGGT (local points)	0.2109	0.1786	0.2590	0.1943	0.2349	0.1865

Appendix

Table 10: Point-cloud results on Stanford2D3D [ 3 ] .

\dagger

denotes the perspective-based model evaluated on panoramas using the dodecahedral projection protocol of MoGe.

Figure 4: Multi-view point-cloud reconstructions on the Stanford2D3D dataset [ 3 ] using two unordered panoramic inputs. From left to right: $\pi^{3}$ [ 35 ] , $\pi^{3}$ * [ 35 ] , $\pi^{3}$ † [ 35 ] , and PanoVGGT. Our method achieves higher geometric accuracy and cross-view consistency than the baselines.

Figure 5: Multi-view point-cloud reconstructions on the PanoCity dataset using ten unordered panoramic inputs. From left to right: $\pi^{3}$ [ 35 ] , $\pi^{3}$ * [ 35 ] , $\pi^{3}$ † [ 35 ] , and PanoVGGT. The baseline methods struggle to learn accurate geometry on this long-trajectory setup, whereas PanoVGGT reconstructs large-scale outdoor scenes with coherent structure and accurate alignment across views.

Figure 6: Overview of the five core modalities in the PanoCity dataset. Row 1: representative urban scenes from three captured cities or regions (City-1, City-2, City-3), annotated with their coverage areas ( $15\text{km}^{2}$ , $113\text{km}^{2}$ , $6\text{km}^{2}$ ). Row 2: example panoramic RGB images collected along the acquisition routes, showing diverse lighting, weather, and façade structures. Row 3: corresponding ground-truth 3D point clouds for the same scenes. Row 4: ground-truth panoramic depth maps derived from the point clouds, with depth encoded by color. Row 5: camera trajectories in bird’s-eye view, illustrating long-range paths with wide baselines and varied viewpoints.

Explore similar work

Unified Panoramic Geometry Estimation via Multi-View Foundation Models
May 25, 2026Vukasin Bozic, Isidora Slavkovic, Dominik Narnhofer +43D Foundation Models 3D ViTs
ETH Zürich · Google · Athlence Sports +1
Pano3DComposer: Feed-Forward Compositional 3D Scene Generation from Single Panoramic Image
Mar 6, 2026Zidian Qiu, Ancong Wu3D Scene Generation 3D Scene Layout Generation
Sun Yat-sen University
VGGT-360: Geometry-Consistent Zero-Shot Panoramic Depth Estimation
Mar 19, 2026Jiayi Yuan, Haobo Jiang, De Wen Soh +1Depth Estimation
Singapore University of Technology and Design · Nanyang Technological University

May 25, 2026J/K move · Enter open · S save

Unified Panoramic Geometry Estimation via Multi-View Foundation Models

Vukasin Bozic, Isidora Slavkovic, Dominik Narnhofer, Nando Metzger+3

ETH Zürich · Google · Athlence Sports+1

3D Foundation Models 3D ViTs Panoramic 3D Reconstruction 3D Reconstruction Depth Estimation Single-View 3D Reconstruction

Open paper AskPDF arXiv

Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view. A natural extension is 3D reconstruction from panoramas, with the exciting prospect of recovering a full 360-degree scene from a single panoramic image. In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain. Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images, in a single forward pass. By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360-degree scenes from single panoramas. We extensively test our method in both indoor and outdoor environments and find that it delivers state-of-the-art performance and excellent zero-shot performance across a wide range of scenes. Code, data and models are available here.

Abstract

Explore similar work

Unified Panoramic Geometry Estimation via Multi-View Foundation Models

Pano3DComposer: Feed-Forward Compositional 3D Scene Generation from Single Panoramic Image

VGGT-360: Geometry-Consistent Zero-Shot Panoramic Depth Estimation