Authors: Mengfei Duan, Yuheng Zhang, Yihong Cao, Fei Teng, Kai Luo, Jiaming Zhang, Kailun Yang, Zhiyong Li
Organizations: The authors are with the School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410082, China
Panoramic imaging enables capturing 360° images with an ultra-wide Field-of-View (FoV) for dense omnidirectional perception, which is critical to applications such as autonomous driving and augmented reality. However, current panoramic semantic segmentation methods fail to identify outliers, and pinhole out-of-distribution segmentation (pinhole-OoS) models perform unsatisfactorily in the panoramic domain due to pixel distortions and background clutter. To address these issues, we introduce a new task, Panoramic Out-of-Distribution Segmentation (PanOoS), with the aim of achieving comprehensive and safe scene understanding. Furthermore, we propose the first solution, POS, which adapts to the characteristics of panoramic images through text-guided prompt distribution learning. Specifically, POS integrates a disentanglement strategy designed to materialize the cross-domain generalization capability of CLIP. The proposed Prompt-based Restoration Attention (PRA) optimizes semantic decoding by prompt guidance and self-adaptive correction, while Bilevel Prompt Distribution Learning (BPDL) refines the manifold of per-pixel mask embeddings via semantic prototype supervision. Besides, to compensate for the scarcity of PanOoS datasets, we establish two benchmarks: DenseOoS, which features diverse outliers in complex environments, and QuadOoS, captured by a quadruped robot with a panoramic annular lens system. Extensive experiments demonstrate the effectiveness of POS, with AuPRC increasing and FPR95 decreasing by 34.25 and 21.42 percentage points, respectively, compared with RbA on DenseOoS, outperforming state-of-the-art pinhole-OoS methods. Moreover, POS achieves leading closed-set segmentation capabilities and advances the development of panoramic understanding. Code and datasets will be available at https://github.com/MengfeiD/PanOoS.
Figures & tables
Fig. 1 : PanOoS enhances comprehensive and safe omnidirectional scene perception. \scriptsize{1}⃝: pinhole image with narrow FoV; \scriptsize{2}⃝: 360∘ panoramic image; \scriptsize{3}⃝: segmentation result. The tiger in the image is an outlier on the street.
Fig. 2 : Comparison of panoramic semantic and out-of-distribution segmentation paradigms. (a) Panoramic semantic segmentation methods adapt the model from pinhole to panoramas through domain adaptation, yet remain limited to in-distribution dense predictions; (b) Pinhole- OoS methods combine semantic segmentation with OoD scoring to detect outliers, but are restricted to narrow pinhole FoV; (c) POS achieves segmentation in the distribution space of panoramic images via prompt distribution learning. The pinhole images denote the training-domain samples, including labeled inlier data and auxiliary outlier-exposure data, while the panoramic images denote the target-domain evaluation data.
Fig. 3 : Impact of pixel distortions and background clutter on state-of-the-art pinhole- OoS methods. The white dashed boxes highlight representative challenging regions, while the red solid contours indicate anomalous objects or regions. Existing pinhole- OoS methods, e.g. , Mask2Anomaly and RbA, fail to adapt to pixel distortions and background clutter, resulting in a higher number of false positives. In contrast, the proposed POS effectively enhances semantic perception of panoramic pixels.
Fig. 4 : Overview of POS. POS consists of an image encoder, a text encoder, a pixel decoder, and a Transformer decoder equipped with Prompt-based Restoration Attention (PRA). During outlier exposure fine-tuning, Bilevel Prompt Distribution Learning (BPDL) regularizes the embedding manifold of per-pixel mask embeddings with class-level and distribution-level prompt priors. See Fig. 6 for details of BPDL. During inference, POS jointly processes image features and prompt embeddings to predict masks and class probabilities, which are further integrated into per-pixel class logits for final OoD scoring.
Fig. 5 : Prompt-based Restoration Attention continuously employs masked attention and cross attention between image features and prompt embeddings while refining semantic clarity through query semantic self-adaptive correction.
Fig. 6 : Bilevel Prompt Distribution Learning. The distribution prompts, text prompts for each class, and pixel embeddings are denoted by stars, diamonds, and circles, respectively. The dashed circles indicate equidistant lines from the semantic center, and the dashed lines represent the decision boundary separating in- and out-of-distribution regions.
Fig. 7 : In-distribution Learning. Enhancing intra-class compactness: the similarity of pixel embeddings within the same class is reinforced, resulting in a more compact intra-class distribution. Improving inter-class separation: the embedding distance between different classes is expanded, thereby strengthening the discriminability of inter-class features.
Fig. 8 : Distribution Learning. Isolation of out-of-distribution pixels: A distinct boundary is established in the feature space to separate out-of-distribution regions from known class regions, thereby mitigating overconfident predictions on outliers. Aggregation of out-of-distribution pixels: Through semantic guidance, out-of-distribution pixel embeddings are clustered into specific semantic spaces, preventing their dispersion and interference with the feature distribution of known classes.
Datasets
Year
Domain
Annotation
Test Images
OoD realism
No shift
Road Anomaly [ 59 ]
2019
Pinhole
outlier
60
✓
✘
Fishyscapes L&F [ 51 ]
2019
outlier
275
✓
(✓)
Fishyscapes Static [ 51 ]
2019
outlier
1000
✘
✓
StreetHazards [ 56 ]
2019
all
1500
(✓)
✘
SMIYC RA-21 [ 50 ]
2021
outlier
100
✓
✘
SMIYC RO-21 [ 50 ]
2021
outlier
327
✓
✘
TABLE I : Comparison of different OoS datasets. Referring to POC [ 84 ] , we qualitatively compare datasets on two main axes, and score them as good ( ✓ ), medium ( (✓) ) or bad ( ✘ ).
Fig. 9 : Distribution details of DenseOoS. DenseOoS contains 30 distinct outliers, divided into 5 main categories: Animals, Garbage, Furniture, Vehicles & Tools, and Others.
Fig. 10 : Examples from the DenseOoS dataset. Image backgrounds are highly cluttered, and distortions are severe. Outliers appear in semantically plausible regions. Zoom in for a better view.
Fig. 11 : (a) Quadruped robot platform with a panoramic annular camera; (b) is the data statistics; (c) Y-axis pixel positions of quadruped robot motion trajectories. While in motion, the quadruped robot experiences significant vertical jitter.
Fig. 12 : Examples from the QuadOoS dataset. The images come with challenges, including uneven exposure, color inconsistencies due to the broader FoV, and increased motion blur. Numerous small outliers are scattered across the scene.
Method
DenseOoS
QuadOoS
Mean
AuPRC ↑
FPR 95 ↓
mIoU ↑
AuPRC ↑
FPR 95 ↓
AuPRC ↑
FPR 95 ↓
PEBAL [ 31 ]
16.39
7.59
46.29
11.34
39.25
13.87
23.42
DenseHybrid [ 32 ]
24.44
7.74
45.19
19.09
33.86
21.77
20.80
EAM [ 23 ]
17.97
6.46
61.58
33.05
79.66
25.51
43.06
Mask2Anomaly [ 25 ]
36.28
8.79
48.10
25.93
92.55
31.10
50.67
RbA [ 24 ]
51.31
21.87
57.54
34.12
90.34
42.71
56.11
TABLE II : Panoramic out-of-distribution segmentation quantitative evaluation. POS shows significant improvement over baseline per-pixel and mask-transformer-based pinhole- OoS methods on PanOoS . Higher values for AuPRC and mIoU are better, whereas for FPR 95 lower values are better. The best and second best results are bold and underlined , respectively, and the line in-between the table divides the per-pixel architectures and mask-transformer-based methods.
Method
mIoU
DAFormer [ 33 ]
44.53
Trans4PASS (Tiny) [ 10 ]
44.93
Trans4PASS (Small) [ 10 ]
49.35
Trans4PASS+ (Tiny) [ 1 ]
49.29
Trans4PASS+ (Small) [ 1 ]
50.11
360SFUDA++ [ 48 ]
50.46
TABLE III : Quantitative closed-set segmentation results on DenseOoS. POS achieves the highest mIoU among the compared methods.
Method
AuPRC
FPR 95
PEBAL [ 31 ]
62.37
28.29
DenseHybrid [ 32 ]
-
-
EAM [ 23 ]
69.40
7.70
Mask2Anomaly [ 25 ]
79.70
13.45
RbA [ 24 ]
85.42
6.92
Ours
89.39
6.25
TABLE IV : Pinhole out-of-distribution segmentation results on Road Anomaly. The best and second best results are bold and underlined , respectively; “-” indicates the unavailability of benchmark results.
PRA
BPDL
AuPRC
FPR 95
mIoU
76.88
0.84
61.73
✓
83.94
0.46
62.50
✓
✓
85.56
0.45
63.30
TABLE V : Component-wise ablation study of POS. PRA: Prompt-based Restoration Attention; BPDL: Bilevel Prompt Distribution Learning.
MA
CA
SAC
AuPRC
FPR 95
mIoU
✓
81.24
0.67
58.03
✓
✓
81.98
0.53
62.48
✓
✓
✓
85.56
0.45
63.30
TABLE VI : Structural ablation of Prompt-based Restoration Attention. MA: Masked Attention; CA: Cross Attention; SAC: Self-adaptive Correction.
Fig. 13 : Attention maps of queries in PRA after different query stages, resembling a confusion matrix in which each query has the highest semantic similarity with itself. When querying only with image features, the semantic information of queries is relatively vague. However, after further interaction with the prompt embeddings, the semantics become clearer, and the self-adaptive correction further mitigates the impact of pixel distortion on semantic decoding.
Lpixel
Lind
LinDir
Loutlier
AuPRC
FPR 95
mIoU
✓
81.72
0.54
62.69
✓
✓
✓
82.61
0.50
62.67
✓
✓
✓
83.61
0.49
63.04
✓
✓
✓
83.37
0.49
63.06
✓
✓
✓
83.08
0.50
62.89
✓
✓
✓
✓
85.56
0.45
63.30
TABLE VII : Ablation study of different loss contributions in bilevel prompt distribution learning loss Lbpdl .
Fig. 14 : Sensitivity analysis of s , d , α and λbpdl .
TABLE VIII : Ablation study on outlier exposure fine-tuning. Lower NLL and ECE indicate better probabilistic prediction quality and calibration, respectively. Params denotes the number of parameters updated during OE fine-tuning.
Fig. 15 : OoD score statistics on DenseOoS and QuadOoS. (a) and (b) show the relative-frequency distributions of ID and OoD pixel scores. (c) shows the relationship between the logarithm of GT OoD-component area and mean OoD score, together with the Spearman correlation ρ .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Class
ID
Number
stroller
19
19
trolley
20
20
garbage bag
21
55
wheelie bin
22
41
suitcase
23
16
skateboard
24
13
Appendix
TABLE B.1 : Details of the DenseOoS dataset.
Fig. B.1 : Examples of outliers included in the QuadOoS dataset. Including but not limited to the above types.
Method
road
sidewalk
building
wall
fence
pole
traffic light
traffic sign
vegetation
terrain
sky
person
rider
car
truck
bus
train
motorcycle
bicycle
Metric
DAFormer [ 33 ]
48.94
31.78
84.12
26.51
31.06
24.92
6.83
7.73
71.86
40.63
91.23
44.13
2.1
78.02
55.02
25.24
84.89
61.37
29.59
44.53
Trans4PASS (S) [ 10 ]
74.28
30.68
84.79
28.45
36.60
27.90
15.40
15.11
78.99
33.71
93.28
49.69
6.79
82.00
46.42
51.79
72.84
68.34
40.63
49.35
Trans4PASS+ (S) [ 1 ]
74.74
34.17
85.43
23.46
42.78
28.45
17.14
18.08
79.64
29.41
93.00
48.74
5.40
82.76
50.54
44.91
91.49
61.49
40.20
50.11
360SFUDA++ [ 48 ]
72.29
45.24
84.91
32.11
38.18
29.83
19.28
17.98
76.02
27.23
92.01
53.14
18.74
81.35
44.42
45.38
82.84
60.93
36.92
50.46
Ours
87.67
59.56
89.68
50.88
45.10
42.62
34.02
22.73
81.21
45.19
94.59
72.24
14.42
86.62
64.88
69.16
95.27
82.98
63.92
63.30
Appendix
TABLE C.1 : Panoramic semantic segmentation results on the DenseOoS benchmark. The per-class results are reported as IoU, and the metric is mIoU.
Features
Layer
AuPRC
FPR 95
mIoU
f4
1
85.56
0.45
63.30
3
71.18
0.68
57.72
6
76.46
0.64
57.33
f2 , f3 , f4
3
72.38
0.94
58.96
6
63.60
1.01
58.66
9
65.91
1.06
58.79
Appendix
TABLE C.2 : Comparison of different features in Transformer Decoder. The f2 and f3 are feature maps with stride 8(c8) and 16(c16) processed by DeformAttn layers, respectively.
Method
Efficiency Metrics
DenseOoS
Input Res.
Total Params (M) ↓
Trainable Params (M) ↓
TFLOPs ↓
FPS ↑
Mem. (GB) ↓
AuPRC ↑
FPR95 ↓
mIoU ↑
Ours
400×2048
361.27
9.49
3.25
12.92
4.21
85.56
0.45
63.30
Appendix
TABLE C.3 : Efficiency analysis of POS when inferring with RTX 3090 on DenseOoS images.
Dataset
Size
Inst.
Median Area (%)
AuPRC
FPR 95
Inst. Recall ↑
DenseOoS
Small
250
0.142
50.75
0.35
96.22
Medium
500
0.353
77.05
0.43
95.16
Large
250
0.743
76.56
0.49
94.93
Overall
1000
0.353
85.56
0.45
95.37
QuadOoS
Small
184
0.014
0.16
97.18
74.41
Medium
367
0.061
10.38
90.51
88.10
Appendix
TABLE C.4 : Performance of POS across different outlier-instance sizes on DenseOoS and QuadOoS. Instances are grouped independently for each dataset according to their relative GT areas: Small (bottom 25% ), Medium (middle 50% ), and Large (top 25% ). Median Area denotes the median instance area relative to the full panoramic image. Inst. Recall is the macro-averaged per-instance recall at the dataset-level threshold corresponding to 95% OoD pixel recall. Overall reports performance on the complete dataset.
Category
Images
Median Area (%)
AuPRC
FPR 95
Inst. Recall ↑
Animals
392
0.275
85.65
0.34
97.97
Garbage
145
0.354
80.21
0.63
91.24
Furniture
159
0.363
90.65
0.29
95.76
Vehicles & Tools
141
0.585
83.62
0.64
89.40
Others
163
0.373
92.47
0.23
97.58
Overall
1000
0.353
85.56
0.45
95.37
Appendix
TABLE C.5 : Category-wise OoD detection performance of POS on DenseOoS. The five super-categories follow the DenseOoS taxonomy. Pixel-level metrics are computed using all valid ID and OoD pixels from images belonging to each category.
Fig. D.1 : Qualitative comparisons of PanOoS . When existing pinhole- OoS methods are transferred to PanOoS , they tend to misidentify complex background regions as outliers. Although EAM [ 23 ] has a low false positive rate, it incorrectly identifies bicycles as outliers and nearly overlooks the actual outliers. In contrast, Mask2Anomaly [ 25 ] and RbA [ 24 ] suffer from significant false positives in the background regions, whereas POS delivers more precise PanOoS across the 360∘ .
Fig. D.2 : Visualization results of PanOoS on DenseOoS. POS better distinguishes outliers from inliers and produces smoother outlier maps with fewer false positives. Zoom in for a better view.
Fig. D.3 : Visualization results of PanOoS on QuadOoS. Uneven exposure, defocus, and motion blur caused by dense motion pose extra challenges to all methods. The pinhole- OoS methods have serious omissions and false positives. However, POS achieves precise segmentation of outliers. Zoom in for a better view.
Fig. E.1 : Failure case analysis in PanOoS . Brighter colors indicate higher OoD confidence. The white dashed boxes highlight challenging regions, while the red solid contours indicate anomalous objects or regions.
Cross-domain panoramic semantic segmentation has attracted growing interest as it enables comprehensive 360° scene understanding for real-world applications. However, it remains particularly challenging due to severe geometric Field of View (FoV) distortions and inconsistent open-set semantics across domains. In this work, we formulate an open-set domain adaptation setting, and propose Extrapolative Domain Adaptive Panoramic Segmentation (EDA-PSeg) framework that trains on local perspective views and tests on full 360° panoramic images, explicitly tackling both geometric FoV shifts across domains and semantic uncertainty arising from previously unseen classes. To this end, we propose the Euler-Margin Attention (EMA), which introduces an angular margin to enhance viewpoint-invariant semantic representation, while performing amplitude and phase modulation to improve generalization toward unseen classes. Additionally, we design the Graph Matching Adapter (GMA), which builds high-order graph relations to align shared semantics across FoV shifts while effectively separating novel categories through structural adaptation. Extensive experiments on four benchmark datasets under camera-shift, weather-condition, and open-set scenarios demonstrate that EDA-PSeg achieves state-of-the-art performance, robust generalization to diverse viewing geometries, and resilience under varying environmental conditions. The code is available at https://github.com/zyfone/EDA-PSeg.
Yuanfan Zheng, Kunyu Peng, Xu Zheng +1
Hunan University · Karlsruhe Institute of Technology · INSAIT, Sofia University “St. Kliment Ohridski” +1
Panoramic semantic segmentation is pivotal for comprehensive 360° scene understanding in critical applications like autonomous driving and virtual reality. However, progress in this domain is constrained by two key challenges: the severe geometric distortions inherent in panoramic projections and the prohibitive cost of dense annotation. While Unsupervised Domain Adaptation (UDA) from label-rich pinhole-camera datasets offers a viable alternative, many real-world tasks impose a stricter source-free (SFUDA) constraint where source data is inaccessible for privacy or proprietary reasons. This constraint significantly amplifies the core problems of domain shift, leading to unreliable pseudo-labels and dramatic performance degradation, particularly for minority classes. To overcome these limitations, we propose the DAPASS framework. DAPASS introduces two synergistic modules to robustly transfer knowledge without source data. First, our Panoramic Confidence-Guided Denoising (PCGD) module generates high-fidelity, class-balanced pseudo-labels by enforcing perturbation consistency and incorporating neighborhood-level confidence to filter noise. Second, a Contextual Resolution Adversarial Module (CRAM) explicitly addresses scale variance and distortion by adversarially aligning fine-grained details from high-resolution crops with global semantics from low-resolution contexts. DAPASS achieves state-of-the-art performances on outdoor (Cityscapes-to-DensePASS) and indoor (Stanford2D3D) benchmarks, yielding 55.04% (+2.05%) and 70.38% (+1.54%) mIoU, respectively.
Yaowen Chang, Zhen Cao, Xu Zheng +2
Wuhan University · HKUST (Guangzhou) · Wuhan University of Technology
We present PanoSeg3R, a feed-forward framework for 3D panoramic semantic segmentation. Unlike existing methods designed for perspective inputs, PanoSeg3R jointly predicts 3D geometry and multi-view semantic segmentation in one single forward pass. Built upon a pretrained reconstruction backbone that supports panoramic images, our approach extends feed-forward 3D reconstruction with a query-based mask decoder. Furthermore, we introduce an automatic panorama data curation pipeline that leverages the complementary strengths of off-the-shelf foundation models to generate reliable pseudo semantic annotations, substantially expanding the training data and improving zero-shot generalization. PanoSeg3R achieves state-of-the-art performance on panoramic 3D semantic segmentation, improving 3D mIoU by up to 16.02 on ScanNet++, while the curated training data further improves zero-shot performance by up to 4.26 and 43.28 mIoU on Stanford2D3D and ToF-360, respectively. Website: https://harryyoon777.github.io/PanoSeg3R/