AdapToPASS: Ambiguity-aware Adaptive Spherical Transformer for Panoramic Semantic Segmentation

Soumyaratna Debnath; Weiming Zhang; Shriram Damodaran; Dingwen Xiao; Addison Lin Wang

cs.CVAug 29, 2026

AdapToPASS: Ambiguity-aware Adaptive Spherical Transformer for Panoramic Semantic Segmentation

Authors: Soumyaratna Debnath, Weiming Zhang, Shriram Damodaran, Dingwen Xiao, Addison Lin Wang

Organizations: HKUST

Abstract

Spherical Transformers have emerged as a promising framework for panoramic semantic segmentation (PASS) by operating directly on spherical geometry and alleviating projection-induced distortions. However, existing architectures often assume canonical spherical structure and stable viewpoints, which are frequently violated in real-world imagery due to unconstrained camera motion, introducing contextual and geometric ambiguity. Consequently, they lack adaptive mechanisms to handle such ambiguity, limiting robustness to unseen spherical transformations. In contrast, biological perception is inherently ambiguity-aware, adapting to fluctuations in cue reliability caused by geometric and contextual variations to maintain stable interpretation under complex transformations. Motivated by this, we first systematically analyze existing PASS architectures under various unseen spherical transformations. We then introduce AdapToPASS, a novel bio-inspired Spherical Transformer that adaptively models contextual and geometric ambiguities for robust PASS. At its core, Adaptive Spherical Attention (AdaSpA) blocks dynamically modulate attention according to local contextual ambiguity, mimicking adaptive, context-driven biological perception. To address geometric ambiguity, AdapToPASS employs Bifocal Spherical Representation to balance field of view and spatial resolution, together with boundary supervision inspired by the boundary-sensitive nature of biological vision. Across indoor and outdoor semantic segmentation, AdapToPASS consistently outperforms prior state-of-the-art methods. Under unseen spherical transformations, it surpasses the next-best method by +13.38% relative mIoU on Stanford2D3D and +18.77% on WildPASS. We further introduce AdapToPASS-Swift, a lightweight variant with fewer than 2M parameters, which surpasses compact baselines while retaining robustness to spherical transformations.

Figures & tables

Figure 1: (a) Spherical perception in current methods degrades under spherical transformations due to geometric and contextual ambiguity, while biological vision remains stable via adaptive perception; AdapToPASS models this adaptivity. (b) Improved segmentation performance of AdapToPASS on Stanford2D3D and WildPASS. (c) AdapToPASS exhibits higher and more consistent confidence, indicating stronger geometric and contextual robustness. Click here to visit the project page.

Figure 2: Qualitative comparison of AdapToPASS and SphereUFormer on Stanford2D3D under unseen spherical transformations. We show segmentation predictions and confidence maps, where confidence is defined as the inverse of ambiguity $\mathrm{Conf}=1-\mathrm{Amb}$ and visualized using the maximum softmax probability. SphereUFormer shows a clear confidence drop under spherical transformations, whereas AdapToPASS maintains higher confidence and sharper semantic boundaries.

Figure 3: Overview of the AdapToPASS framework. Zoom in for a better view .

Method	Mild					Robust					Stress
Method	Rot.	Scale	Ori.	Trans.	View.	Rot.	Scale	Ori.	Trans.	View.	Rot.	Scale	Ori.	Trans.	View.
SFSS	50.40	55.47	47.02	59.45	48.51	35.65	47.84	34.14	52.84	33.00	24.66	42.39	24.20	46.11	20.26
SGAT4PASS	59.92	62.62	56.27	65.05	57.19	37.59	55.83	36.11	56.64	32.61	18.56	49.22	18.74	47.68	12.50
Elite360D	54.21	58.92	48.23	64.17	51.51	31.37	47.02	28.08	56.00	29.26	17.01	38.12	15.05	47.77	14.52
HEAL-SWIN	36.94	40.71	33.13	43.61	32.18	25.42	34.08	24.23	34.60	20.54	17.01	30.39	15.61	28.65	12.40
SphereUFormer	64.01	69.33	59.16	67.23	55.27	36.29	60.13	33.89	57.21	30.76	18.96	51.93	17.61	48.29	14.66

Table 1: mIoU comparison under unseen spherical transformations on the Stanford2D3D benchmark. Results are reported across mild, robust, and stress settings, including rotation (Rot.), Scale, Orientation Shifts (Ori.), Translation (Trans.), and Viewpoint Shifts (View.). Higher is better .

Model	Params	Stanford2D3D			WildPASS
Model	(M)	mAcc. $\uparrow$	mIoU $\uparrow$	Conf. $\uparrow$	mAcc. $\uparrow$	mIoU $\uparrow$	Conf. $\uparrow$
SFSS	15.28	74.22	63.29	85.97	77.85	65.27	81.06
SGAT4PASS	15.22	73.13	64.75	91.76	72.15	56.77	82.35
Elite360D	15.41	76.57	67.42	88.10	74.36	65.40	59.65
HEAL-SWIN	14.06	72.60	43.46	88.01	64.64	48.20	81.70
SphereUFormer	14.77	78.13	68.60	90.34	79.15	63.38	74.89

Table 2: Comparison with state-of-the-art panoramic semantic segmentation methods on Stanford2D3D and WildPASS. We report model parameters, mean accuracy (mAcc), mean IoU (mIoU), and mean prediction confidence (Conf.).

Panel of Figure 5 — Figure 5: (a) Analysis on Stanford2D3D under unseen spherical transformations. (i) Difficulty-wise mIoU comparison across difficulties. (ii) Transformation-wise mIoU comparison under Rotation, Scale, Orientation Shift (Ori.), Translation, and Viewpoint Shift (View). (b) Comparison of AdapToPASS-Swift (our 2M param variant) with lightweight panoramic segmentation baselines.

Configuration	mIoU $\uparrow$	Conf. $\uparrow$
AdapToPASS	70.80	96.42
AdapToPASS w/o Context Ambiguity	64.97	88.47
AdapToPASS w/o Acuity Stream	66.50	92.20
AdapToPASS w/o Boundary Supervision	63.70	91.90

Table 3: Ablation study evaluating the contribution of key components in AdapToPASS.

Figure 6: Qualitative comparison on Stanford2D3D. We compare segmentation predictions and their corresponding confidence maps across representative panoramic segmentation methods. AdapToPASS produces more accurate semantic regions with sharper boundaries while maintaining high confidence (SUF: SphereUFormer, SO3UF: SO3UFormer). Zoom in for a better view .

Figure 7: Failure cases of AdapToPASS on Stanford2d3d under Stress setting.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Icosphere representations at different ranks. The same panoramic scene is shown using different icosphere subdivision levels, ranging from Rank 2 (162 vertices) to Rank 7 (163,842 vertices). Higher ranks provide finer spherical resolution and preserve more visual detail.

Type	Parameter	Mild	Robust	Stress
Rotation	Yaw	$0^{\circ},45^{\circ},90^{\circ},135^{\circ},180^{\circ}$	$0^{\circ}{:}30^{\circ}{:}330^{\circ}$	$0^{\circ}{:}30^{\circ}{:}330^{\circ}$
	Pitch	$-10^{\circ},-5^{\circ},0^{\circ},5^{\circ},10^{\circ}$	$-30^{\circ},-20^{\circ},-10^{\circ},0^{\circ},10^{\circ},20^{\circ},30^{\circ}$	$-60^{\circ},-45^{\circ},-30^{\circ},0^{\circ},30^{\circ},45^{\circ},60^{\circ}$
	Roll	$-10^{\circ},0^{\circ},10^{\circ}$	$-30^{\circ},-15^{\circ},0^{\circ},15^{\circ},30^{\circ}$	$-45^{\circ},0^{\circ},45^{\circ}$
Scale	Zoom	$0.9,1.0,1.1,1.25,1.5$	$0.8,1.0,1.2,1.5,2.0$	$0.6,0.8,1.0,1.5,2.0,3.0$
Translation	$t_{x}/t_{y}$	$-0.08,-0.05,0,0.05,0.08$	$-0.18,-0.12,-0.08,0,0.08,0.12,0.18$	$-0.30,-0.22,-0.16,-0.10,0,0.10,0.16,0.22,0.30$
Translation	$t_{z}$	$-0.04,0,0.04$	$-0.08,-0.05,0,0.05,0.08$	$-0.12,-0.08,0,0.08,0.12$

Appendix

Table 4: Difficulty levels for unseen spherical transformations. Each level defines progressively stronger perturbations for rotation, scale, translation, orientation shift, and viewpoint shift.

Figure 9: Examples of unseen spherical transformations.

Panel of Figure 10 — Figure 10: Class distribution for WildPASS dataset.

Method	Mild			Robust			Stress
Method	Rot.	Scale	Ori.	Rot.	Scale	Ori.	Rot.	Scale	Ori.
SFSS	56.53	58.43	53.46	49.21	56.58	47.59	46.33	53.78	46.85
SGAT4PASS	64.32	66.60	62.91	58.80	65.88	56.67	51.53	63.88	52.87
Elite360D	53.14	55.09	50.00	41.26	51.17	39.55	33.02	47.44	34.23
HEAL-SWIN	50.56	52.52	48.23	46.33	50.04	44.41	43.65	48.27	42.63
SphereUFormer	61.75	63.88	61.28	55.04	62.15	53.46	42.72	58.98	41.22

Appendix

Table 5: mIoU comparison under unseen spherical transformations on the WildPASS benchmark. Results are reported across mild, robust, and stress settings, including rotation (Rot.), Scale, and Orientation Shifts (Ori.). Higher is better.

Figure 11: Robustness analysis on WildPASS under unseen spherical transformations. (a) Difficulty-wise average mIoU comparison across mild, robust, and stress settings. (b) Transformation-wise average mIoU comparison across rotation, scale, and orientation shift (Ori.).

Figure 12: Qualitative comparison on Stanford2D3D under unseen spherical transformations. We compare segmentation predictions and confidence maps for different methods across the original image and transformed variants, including rotation, scale, orientation shift, translation, and viewpoint shift under Robust setting. Each row pair shows the predicted semantic map and the corresponding confidence map for the same input. Zoom in for a better view. (SUF:SphereUFormer, SO3F: SO3UFormer)

Figure 13: Qualitative comparison on WildPASS under unseen spherical transformations. We compare segmentation predictions and confidence maps for different methods across the original image and transformed variants, including rotation, scale, and orientation shift under Robust setting. Each row pair shows the predicted semantic map and the corresponding confidence map for the same input. Zoom in for a better view. (SUF:SphereUFormer, SO3F: SO3UFormer)

Figure 14: Qualitative ablation study of AdapToPASS. Removing contextual ambiguity modeling, the acuity stream, or boundary supervision degrades semantic consistency and boundary quality. The full AdapToPASS model produces more coherent predictions that better align with the ground truth.

Model	Params	Stanford2D3D
Model	(M)	Acc. $\uparrow$	mIoU $\uparrow$	Conf. $\uparrow$
HexRUNet	1.99	59.79	44.55	75.24
SphereUFormer	2.04	69.42	55.10	82.45
Trans4PASS	1.99	70.60	54.50	83.25
AdapToPASS-Swift	1.99	73.20	57.85	88.23

Appendix

Table 6: Comparison of lightweight panoramic semantic segmentation models on Stanford2D3D. We report average class accuracy (Acc.), mean IoU (mIoU), and mean prediction confidence (Conf.)

Figure 15: Overview of the AdaSpX Module. AdaSpX replaces the AdaSpA blocks in AdapToPASS to enable efficient spherical semantic segmentation.

Method	Mild					Robust					Stress
Method	Rot.	Scale	Ori.	Trans.	View.	Rot.	Scale	Ori.	Trans.	View.	Rot.	Scale	Ori.	Trans.	View.
HexRUNet	37.84	39.01	35.91	38.29	34.01	28.94	34.62	27.53	33.39	24.60	19.98	31.06	18.45	28.67	15.68
SphereUFormer	37.40	53.00	27.10	35.50	19.80	30.40	27.20	23.40	50.70	19.80	26.40	38.10	30.30	13.30	19.10
Trans4PASS	57.40	56.83	54.42	53.78	54.08	40.10	50.49	38.63	45.94	34.12	25.42	44.31	25.66	38.44	18.87
AdapToPASS-Swift (Ours)	62.20	61.80	57.80	60.10	55.20	47.80	54.00	43.80	50.50	38.90	31.80	47.00	28.60	41.70	25.30

Appendix

Table 7: mIoU comparison of lightweight panoramic semantic segmentation models under unseen spherical transformations on the Stanford2D3D benchmark. Results are reported across mild, robust, and stress settings, including rotation (Rot.), Scale, Orientation Shifts (Ori.), Translation (Trans.), and Viewpoint Shifts (View.). Higher is better.

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 416 papers

Mar 26, 2026

Denoise and Align: Towards Source-Free UDA for Robust Panoramic Semantic Segmentation

Panoramic semantic segmentation is pivotal for comprehensive 360° scene understanding in critical applications like autonomous driving and virtual reality. However, progress in this domain is constrained by two key challenges: the severe geometric distortions inherent in panoramic projections and the prohibitive cost of dense annotation. While Unsupervised Domain Adaptation (UDA) from label-rich pinhole-camera datasets offers a viable alternative, many real-world tasks impose a stricter source-free (SFUDA) constraint where source data is inaccessible for privacy or proprietary reasons. This constraint significantly amplifies the core problems of domain shift, leading to unreliable pseudo-labels and dramatic performance degradation, particularly for minority classes. To overcome these limitations, we propose the DAPASS framework. DAPASS introduces two synergistic modules to robustly transfer knowledge without source data. First, our Panoramic Confidence-Guided Denoising (PCGD) module generates high-fidelity, class-balanced pseudo-labels by enforcing perturbation consistency and incorporating neighborhood-level confidence to filter noise. Second, a Contextual Resolution Adversarial Module (CRAM) explicitly addresses scale variance and distortion by adversarially aligning fine-grained details from high-resolution crops with global semantics from low-resolution contexts. DAPASS achieves state-of-the-art performances on outdoor (Cityscapes-to-DensePASS) and indoor (Stanford2D3D) benchmarks, yielding 55.04% (+2.05%) and 70.38% (+1.54%) mIoU, respectively.

Yaowen Chang, Zhen Cao, Xu Zheng +2

Wuhan University · HKUST (Guangzhou) · Wuhan University of Technology

Semantic Segmentation Source-Free Domain Adaptation Unsupervised Domain Adaptation Pseudo-Labeling Domain-Adversarial Learning

Ask PDF

Mar 16, 2026

Seeing Beyond: Extrapolative Domain Adaptive Panoramic Segmentation

Cross-domain panoramic semantic segmentation has attracted growing interest as it enables comprehensive 360° scene understanding for real-world applications. However, it remains particularly challenging due to severe geometric Field of View (FoV) distortions and inconsistent open-set semantics across domains. In this work, we formulate an open-set domain adaptation setting, and propose Extrapolative Domain Adaptive Panoramic Segmentation (EDA-PSeg) framework that trains on local perspective views and tests on full 360° panoramic images, explicitly tackling both geometric FoV shifts across domains and semantic uncertainty arising from previously unseen classes. To this end, we propose the Euler-Margin Attention (EMA), which introduces an angular margin to enhance viewpoint-invariant semantic representation, while performing amplitude and phase modulation to improve generalization toward unseen classes. Additionally, we design the Graph Matching Adapter (GMA), which builds high-order graph relations to align shared semantics across FoV shifts while effectively separating novel categories through structural adaptation. Extensive experiments on four benchmark datasets under camera-shift, weather-condition, and open-set scenarios demonstrate that EDA-PSeg achieves state-of-the-art performance, robust generalization to diverse viewing geometries, and resilience under varying environmental conditions. The code is available at https://github.com/zyfone/EDA-PSeg.

Yuanfan Zheng, Kunyu Peng, Xu Zheng +1

Hunan University · Karlsruhe Institute of Technology · INSAIT, Sofia University “St. Kliment Ohridski” +1

Semantic Segmentation Domain Adaptation

Ask PDF

Jun 26, 2026

Panoramic Scene Understanding: A Survey from Distortion-Aware Engineering to Sphere-Native Modeling

Panoramic images capture the full visual sphere in a frame, offering context unavailable to conventional cameras. Yet this completeness has an unavoidable geometric cost: the 2-sphere cannot be faithfully mapped to the plane, and every projection introduces distortions that challenge standard vision architectures. This survey traces panoramic scene understanding from projection-based adaptation and distortion-aware engineering to sphere-native modeling, reflecting increasing commitment to spherical geometry. Foundation models form a fourth family, geometry-aware tokenization, which adapts the input interface while reusing perspective-pretrained weights. We review these approaches across five task families: dense prediction, unified multi-task understanding, open-world perception, vision-language reasoning, and dynamic video analysis. Across tasks, the same shift toward spherical geometry recurs. In practice, however, the field has converged not on the strongest sphere-native operators, which are exactly rotation-equivariant but cannot reuse perspective-pretrained backbones and thus have not scaled, but on a compatibility-preserving middle ground combining moderate geometric awareness with large pretrained models. This commitment is uneven: deepest in dense prediction and shallowest in dynamic perception, where methods are spatially sphere-aware yet temporally planar. Foundation-model adaptation has advanced panoramic depth fastest, while layout, surface-normal, and video-level understanding remain largely unexplored. No panoramic foundation model has yet been pretrained on spherical data. We identify five evaluation gaps: spherical-area-weighted metrics, seam-consistency tests, polar-robustness stratification, cross-projection generalization, and standardized open-world protocols. We conclude with a six-point roadmap toward general-purpose panoramic intelligence.

Qinfeng Zhu, Lei Fan

University of Liverpool, United Kingdom. · Xi’an Jiaotong-Liverpool University, China.

Vision Foundation Models Panoramic Depth Estimation 3D Scene Understanding

Ask PDF

Abstract

Semantic Segmentation

Explore similar work

Denoise and Align: Towards Source-Free UDA for Robust Panoramic Semantic Segmentation

Seeing Beyond: Extrapolative Domain Adaptive Panoramic Segmentation

Panoramic Scene Understanding: A Survey from Distortion-Aware Engineering to Sphere-Native Modeling