PanoSeg3R: Feed-Forward 3D Semantic Segmentation for Panoramic Images with an Automatic Data Curation Pipeline

Heechan Yoon; Dongki Jung; Phuc Nguyen; Ming Lin; Dinesh Manocha

cs.CVSep 19, 2026

PanoSeg3R: Feed-Forward 3D Semantic Segmentation for Panoramic Images with an Automatic Data Curation Pipeline

Authors: Heechan Yoon, Dongki Jung, Phuc Nguyen, Ming Lin, Dinesh Manocha

Organizations: University of Maryland, College Park, USA

Abstract

We present PanoSeg3R, a feed-forward framework for 3D panoramic semantic segmentation. Unlike existing methods designed for perspective inputs, PanoSeg3R jointly predicts 3D geometry and multi-view semantic segmentation in one single forward pass. Built upon a pretrained reconstruction backbone that supports panoramic images, our approach extends feed-forward 3D reconstruction with a query-based mask decoder. Furthermore, we introduce an automatic panorama data curation pipeline that leverages the complementary strengths of off-the-shelf foundation models to generate reliable pseudo semantic annotations, substantially expanding the training data and improving zero-shot generalization. PanoSeg3R achieves state-of-the-art performance on panoramic 3D semantic segmentation, improving 3D mIoU by up to 16.02 on ScanNet++, while the curated training data further improves zero-shot performance by up to 4.26 and 43.28 mIoU on Stanford2D3D and ToF-360, respectively. Website: https://harryyoon777.github.io/PanoSeg3R/

Figures & tables

Figure 1: PanoSeg3R predicts globally consistent 3D semantic segmentation from multi-view panoramic images, outperforming existing feed-forward methods ( Zust et al. 2025 ; Li et al. 2025 ) that rely on cubemap projections.

Figure 2: Overview of the semantic panorama data curation pipeline. Given an RGB ERP image, we first generate overlapping perspective images and then divide them into crops. SAM3 ( Carion et al. 2025 ) generates class-agnostic masks, and PanSt3R ( Zust et al. 2025 ) produces coarse semantic labels from the perspective crops. Their predictions are fused through mask aggregation, followed by VLM-based verification ( Bai et al. 2025 ) to remove inconsistent predictions and produce the final annotations.

Figure 3: Model architecture. Multi-view ERP images are processed by the pretrained Wid3R backbone to extract features. These features are refined by a query-based decoder, while a DPT head produces dense feature maps. An image stem injects low-level features into the feature maps to complement the decoded representations. We reconstruct 3D geometry through the ray-based modules and project the predicted semantic masks to obtain consistent 3D semantic segmentation.

	Dataset	GT Semantic	Scenes	Images	Classes
Train	ScanNet++	✓	856	4,670	100
	Matterport3D	✓	61	7,829	20
	Realsee3D	$\times$	1,000	24,263	100
	OmniScenes	$\times$	28	2,995	100
Eval	ScanNet++	✓	50	243	100
	Matterport3D	✓	18	2,014	20

Table 1: Statistics of training and evaluation datasets. “GT Semantic” denotes official ground-truth annotations. Datasets lacking GT use pseudo-labels generated via our semantic panorama data curation pipeline following 100 ScanNet++ categories ( Yeshwanth et al. 2023 ) to boost performance.

Figure 4: Qualitative results on ScanNet++ ( Yeshwanth et al. 2023 ) and Stanford2D3D ( Armeni et al. 2017 ) . Our method demonstrates superior 3D geometric reconstruction and semantic segmentation performance on panoramic images compared to existing approaches.

Figure 5: Qualitative evaluation of 2D semantic segmentation on ToF-360 ( Kanayama et al. 2025 ) .

Method	Input Modalities	Matterport3D		ScanNet++
		2D		2D		3D
		mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$
Single Panorama Input
PanoSAMic	RGB-D	45.22	61.09	26.69	51.86	–	–
SFSS-MMSI	RGB-D	49.85	61.41	33.86	42.90	–	–
PanoSAMic	RGB	44.33	63.12	26.74	54.27	–	–

Table 2: Comparison of 2D and scene-level 3D semantic segmentation performance on Matterport3D ( Teng et al. 2024 ) and ScanNet++ ( Yeshwanth et al. 2023 ) . † denotes perspective foundation models evaluated on panoramic inputs via six cubemap faces. Our method achieves competitive 2D segmentation performance while substantially outperforming perspective-based approaches in 3D semantic segmentation.

Method	Input Modalities	Stanford2D3D				ToF-360
		2D		3D		2D
		mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$
Single Panorama Input
PanoSAMic	RGB	55.31	63.54	–	–	37.99	57.26
SFSS-MMSI		54.30	60.54	–	–	44.98	52.14
MTPano		65.61	68.92	–	–	61.05	71.04

Table 3: Zero-shot 2D and 3D semantic segmentation on Stanford2D3D ( Armeni et al. 2017 ) and ToF-360 ( Kanayama et al. 2025 ) . ToF-360 is evaluated only in 2D since camera poses are unavailable for 3D reconstruction. † denotes perspective foundation models evaluated on panoramic inputs via six cubemap faces.

Setting	Stanford2D3D				ToF-360
	2D		3D		2D
	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$
(a) Training Data
Base	60.89	67.87	33.79	38.10	13.23	20.87
+ OmniScenes	61.66	67.84	34.22	38.28	37.24	43.16
+ RealSee3D	60.84	68.61	34.09	38.48	51.94	61.36

Table 4: Ablation study on zero-shot evaluation on Stanford2D3D ( Armeni et al. 2017 ) and ToF-360 ( Kanayama et al. 2025 ) . We analyze the effects of the proposed panorama data curation pipeline and the image stem. The curated pseudo semantic datasets substantially improve cross-dataset generalization, with the image stem also contributing to the overall performance.

Method	Stanford2D3D			ToF-360
Method	mIoU $\uparrow$	Wrong removed $\uparrow$	Correct removed $\downarrow$	mIoU $\uparrow$	Wrong removed $\uparrow$	Correct removed $\downarrow$
Baseline
SAM3 + SigLIP	15.22	–	–	19.18	–	–
SAM3 + text prompts	47.76	–	–	29.15	–	–
Ours
PanSt3R raw	53.85	–	–	42.77	–	–

Table 5: Ablation study of our data curation pipeline, including pseudo-annotation quality. For VLM filtering evaluation, we report correctly removed mislabeled masks over total mislabeled masks and mistakenly removed correct masks over total correct masks, together with their respective percentages.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Details of the image stem. Conv is a convolutional layer and CircConv is a convolutional layer with circular padding, k is the kernel size, s is the stride, and p is the padding. The numbers in the parentheses denote the number of filters. $\oplus$ denotes element wise addition.

#Crops	$c_{w}\times c_{h}$	$o_{w}$	$o_{h}$	ScanNet++		Stanford2D3D		ToF-360
#Crops	$c_{w}\times c_{h}$	$o_{w}$	$o_{h}$	2D		2D		2D
				mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$
8 (4 $\times$ 2)	$256\times 256$	82	96	29.37	36.28	52.52	55.13	39.32	43.47
12 (4 $\times$ 3)	$256\times 192$	82	64	29.00	35.66	51.01	53.51	40.03	44.10
16 (4 $\times$ 4)	$256\times 160$	82	64	26.66	33.88	48.79	51.70	37.41	41.92
20 (4 $\times$ 5)	$256\times 144$	82	64	22.36	29.27	45.22	48.98	33.82	38.48

Appendix

Table 6: Ablation study of crop configurations on ScanNet++ ( Yeshwanth et al. 2023 ) , Stanford2D3D ( Armeni et al. 2017 ) , and ToF-360 ( Kanayama et al. 2025 ) . We vary the number of crops, crop size

c_{w}\times c_{h}

, and horizontal and vertical overlap

o_{w}

and

o_{h}

to evaluate their effects on the PanSt3R ( Zust et al. 2025 ) predictions used in our curation pipeline. 15 (5

\times

3) denotes the default configuration used in our experiments.

Setting	$\#\mathcal{R}_{s}$	Pad	$\mathcal{R}$	Stanford2D3D				ToF-360
				2D		3D		2D
				mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$
A (w/o stem)	–	–	–	59.90	68.02	34.85	39.28	49.89	57.59
B	1	zero	O	58.64	66.22	34.37	39.02	53.36	60.88
C	1	circ	O	63.86	70.28	34.35	39.29	54.72	60.98
D	1	circ	X	60.48	70.13	34.24	38.82	56.02	64.45

Appendix

Table 7: Ablation study of the image stem on Stanford2D3D ( Armeni et al. 2017 ) and ToF-360 ( Kanayama et al. 2025 ) . We vary the number of residual convolutional blocks

\#\mathcal{R}_{s}

, the padding strategy used in

\mathcal{R}_{s}

, and the use of the final residual convolutional unit

\mathcal{R}

. Setting A denotes the baseline without the image stem, while Setting F denotes the full configuration used in our experiments.

Method	Input	#Images	#Params (M)	Mem. (GB)	Time (s)	3D mIoU $\uparrow$
Ours	ERP image	8	1123.7	9.56	1.096	17.28
PanSt3R	Cube	48	906.3	9.28	24.938	9.26
PanSt3R	15 Crops	120	906.3	18.07	118.781	5.18
IGGT	Cube	48	1617.3	27.67	4.638	2.30
IGGT	15 Crops	120	1617.3	OOM	OOM	–
Uni3R	Cube	48	2754.9	26.65	4.249	1.26

Appendix

Table 8: Efficiency comparison on a scene with eight covisible panoramaic images. We benchmark the number of input images, parameter count, peak memory, model inference time, and 3D mIoU on ScanNet++ ( Yeshwanth et al. 2023 ) .

Figure 7: Failure cases on Matterport3D ( Teng et al. 2024 ) . Our model frequently misclassifies or fails to recognize outdoor objects because it is trained primarily on indoor datasets.

Figure 8: Visualization of the semantic annotations generated by our data curation pipeline. Each example consists of an input RGB ERP image and its corresponding semantic annotation.

Figure 9: Additional 2D qualitative comparisons with other methods.

Figure 10: Additional 3D qualitative comparisons with other methods.

Explore similar work

Jun 12, 2026

Pano3D: Unified 3D Reconstruction and Panoptic Segmentation

Recent advances in 3D feedforward reconstruction neural networks have achieved remarkable success in dense reconstruction from images without any camera parameters. Yet, equipping these models with robust semantic understanding remains an open problem. Here we introduce an approach that performs 3D reconstruction and 3D panoptic segmentation in a unified framework. We build on existing 3D reconstruction models and augment them with a set-based mask decoder. The approach is jointly trained with a geometric and semantic loss, which are shown to be mutually beneficial. More precisely, the features are initialized from the geometric information and then finetuned to capture jointly geometry and semantics. We demonstrate the generality of our approach by successfully applying our framework both to online and all-to-all attention reconstruction backbones. Our method achieves state-of-the-art performance in 3D panoptic segmentation across ScanNet, ScanNet200, and ScanNet++ datasets. Ablation studies show that such joint training of a unified model equips 3D feedforward reconstruction neural networks with panoptic segmentation and yields mutually beneficial improvements.

Victor Barberteguy, Ahmet Iscen, Mathilde Caron +3

Google DeepMind · LIGM, École des Ponts, IP Paris, Univ Gustave Eiffel, CNRS

Panoptic Segmentation 3D Semantic Segmentation Feed-Forward 3D Reconstruction 3D Instance Segmentation

Ask PDF

Jun 8, 2026

EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation

This paper introduces EPS3D, a new end-to-end feed-forward framework for open-vocabulary 3D panoptic segmentation. Unlike existing methods relying on additional preprocessing, we design an end-to-end architecture, with a distillation-based training strategy on diverse 3D scenes to predict 3D-aware semantic and instance features from multi-view images, improving 3D consistency and avoiding error accumulation. We further propose a mutual enhancement module to enforce inherent semantic-instance consistency. By aligning semantics within instances (Ins2Sem) and refining instance features with semantic guidance (Sem2Ins), we achieve more coherent 3D scene understanding. Ultimately, EPS3D outperforms SOTA baselines on two benchmarks (e.g., +13% mIoU for semantics on Replica) with high efficiency (e.g., 1s per scene), supporting tasks like robotic manipulation and 3D scene editing.

Runsong Zhu, Jiaxin Guo, Xiaoyang Guo +9

The Chinese University of Hong Kong, HK SAR, China · Hong Kong Centre for Logistics Robotics, HK SAR, China · Horizon Robotics, China +2

Panoptic Segmentation Open-Vocabulary 3D Segmentation 3D Semantic Segmentation 3D Instance Segmentation

Ask PDF

Mar 18, 2026

PanoVGGT: Feed-Forward 3D Reconstruction from Panoramic Imagery

Panoramic imagery offers a full 360° field of view and is increasingly common in consumer devices. However, it introduces non-pinhole distortions that challenge joint pose estimation and 3D reconstruction. Existing feed-forward models, built for perspective cameras, generalize poorly to this setting. We propose PanoVGGT, a permutation-equivariant Transformer framework that jointly predicts camera poses, depth maps, and 3D point clouds from one or multiple panoramas in a single forward pass. The model incorporates spherical-aware positional embeddings and a panorama-specific three-axis SO(3) rotation augmentation, enabling effective geometric reasoning in the spherical domain. To resolve inherent global-frame ambiguity, we further introduce a stochastic anchoring strategy during training. In addition, we contribute PanoCity, a large-scale outdoor panoramic dataset with dense depth and 6-DoF pose annotations. Extensive experiments on PanoCity and standard benchmarks demonstrate that PanoVGGT achieves competitive accuracy, strong robustness, and improved cross-domain generalization. Code and dataset will be released.

Yijing Guo, Mengjun Chao, Luo Wang +5

ShanghaiTech University · Sudo

Panoramic 3D Reconstruction Feed-Forward 3D Reconstruction Camera Pose Estimation 3D ViTs

Ask PDF