SAP: Segment Any 4K Panorama

Lutao Jiang; Zidong Cao; Weikai Chen; Xu Zheng; Yuanhuiyi Lyu; Zhenyang Li; Zeyu HU; Yingda Yin

cs.CVMar 13, 2026

SAP: Segment Any 4K Panorama

Authors: Lutao Jiang, Zidong Cao, Weikai Chen, Xu Zheng, Yuanhuiyi Lyu, Zhenyang Li, Zeyu HU, Yingda Yin,

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · LIGHTSPEED · The University of Hong Kong · The Hong Kong University of Science and Technology

Abstract

Promptable instance segmentation is widely adopted in embodied and AR systems, yet the performance of foundation models trained on perspective imagery often degrades on 360° panoramas. In this paper, we introduce Segment Any 4K Panorama (SAP), a foundation model for 4K high-resolution panoramic instance-level segmentation. We reformulate panoramic segmentation as fixed-trajectory perspective video segmentation, decomposing a panorama into overlapping perspective patches sampled along a continuous spherical traversal. This memory-aligned reformulation preserves native 4K resolution while restoring the smooth viewpoint transitions required for stable cross-view propagation. To enable large-scale supervision, we synthesize 183,440 4K-resolution panoramic images with instance segmentation labels using the InfiniGen engine. Trained under this trajectory-aligned paradigm, SAP generalizes effectively to real-world 360° images, achieving +17.2 zero-shot mIoU gain over vanilla SAM2 of different sizes on real-world 4K panorama benchmark.

Figures & tables

Figure 1: The main challenges of equirectangular panoramic segmentation include (i) severe distortions compared to standard perspective images, (ii) discontinuities at the left-right seam and near the poles, and (iii) ultra-high resolution that is rarely addressed by conventional segmentation methods. Our key idea is to convert a panorama into a fixed-trajectory perspective scanning video to avoid these issues and to provide high-quality synthetic data for fine-tuning segmentation models.

Figure 2: Overview of our pipeline. We reformulate panoramic segmentation as perspective scanning video segmentation. A panorama with the prompt points is first projected into multiple continuous perspective frames with a fixed trajectory. Next, the scanning video is processed with the fine-tuned SAM2 to generate video segmentation results, which are merged back to the ERP plane to obtain the final segmentation.

Figure 3: The visualization of different trajectories.

	1-click				3-click Error Correction
Method	Overall	Small	Medium	Large	Overall	Small	Medium	Large
SAM2-tiny	51.6	46.3	55.5	49.1	82.2	61.7	79.6	88.8
SAM2-tiny+scan	65.1	49.6	63.4	70.0	83.0	65.8	82.3	87.2
SAP-tiny	75.8	53.9	76.6	79.7	84.8	68.7	84.7	88.4
$\Delta$ (SAP-SAM2)	+24.2	+7.6	+21.1	+30.6	+2.6	+7.0	+5.1	-0.4
SAM2-small	54.0	45.8	59.7	50.6	83.1	60.8	80.3	90.3

Table 1: Zero-shot Segmentation Mask mIoU on PAV-SOD [ 35 ] benchmark.

	1-click				3-click Error Correction
Method	Overall	Small	Medium	Large	Overall	Small	Medium	Large
SAM2-tiny	68.6	77.6	76.7	56.0	84.7	82.9	85.9	84.4
SAM2-tiny+scan	46.7	40.0	52.9	43.1	78.7	75.5	81.0	77.7
SAP-tiny	75.0	73.3	80.8	69.6	85.7	83.5	85.2	87.3
$\Delta$ (SAP-SAM2)	+6.4	-4.3	+4.1	+13.6	+1.0	+0.6	-0.7	+2.9
SAM2-small	68.7	77.7	76.7	56.1	86.3	82.6	86.2	87.9

Table 2: Zero-shot Segmentation Mask mIoU on HunyuanWorld-1.0 [ 25 ] benchmark.

	1-click				3-click Error Correction
Method	Overall	Small	Medium	Large	Overall	Small	Medium	Large
SAM2-tiny	59.7	55.3	69.0	57.5	80.2	67.7	84.4	90.0
SAM2-tiny+scan	59.4	60.4	60.8	57.0	83.8	78.4	86.6	87.7
SAP-tiny	77.3	64.9	82.1	87.6	87.4	79.4	90.7	93.8
$\Delta$ (SAP-SAM2)	+17.6	+9.6	+13.1	+30.1	+7.2	+11.7	+6.3	+3.8
SAM2-small	59.8	57.2	68.0	56.6	80.5	67.5	84.7	91.0

Table 3: Segmentation mask mIoU on InfiniGen [ 18 ] benchmark.

Figure 4: Qualitative Comparison based on PAV-SOD [ 35 ] dataset.

Figure 5: Qualitative Comparison based on HunyuanWorld-1.0 [ 25 ] dataset. The red circles indicate segmentation errors.

Training Data		PAV-SOD				HunyuanWorld-1.0
SAM2	Ours	Overall	Small	Medium	Large	Overall	Small	Medium	Large
$\checkmark$		60.0	58.9	60.6	59.7	47.4	46.8	51.5	43.3
	$\checkmark$	67.3	44.2	59.4	79.2	71.8	63.8	77.8	69.1
$\checkmark$	$\checkmark$	77.3	61.1	76.2	81.7	77.4	77.1	82.9	71.5

Table 4: The ablation studies on the training sets.

Strategy		PAV-SOD				HunyuanWorld-1.0
Scan	Overlap	Overall	Small	Medium	Large	Overall	Small	Medium	Large
		73.0	50.0	72.5	78.3	76.1	79.8	80.2	70.2
$\checkmark$		66.5	59.4	65.6	68.6	72.0	73.2	79.8	62.5
$\checkmark$	$\checkmark$	77.3	61.1	76.2	81.7	77.4	77.1	82.9	71.5

Table 5: The ablation studies of the training strategies.

Method	PAV-SOD				HunyuanWorld-1.0
Method	Overall	Small	Medium	Large	Overall	Small	Medium	Large
Direct Patchify	72.6	61.2	72.6	74.9	76.4	79.7	82.8	68.0
Perspective Transform	77.3	61.1	76.2	81.7	77.4	77.1	82.9	71.5

Table 6: The ablation studies on the perspective transformation.

Method	SAM2	SAM2+scan	SAP
mIoU	46.1	53.8	68.2

Table 7: Seam/Pole subset evaluation.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

$\Delta$	$N_{\text{pitch}}$	$N_{\text{yaw}}$	$N$	Overlap Ratio
90 ∘	2	4	8	0.0
45 ∘	3	8	24	0.5
30 ∘	4	12	48	2/3
22.5 ∘	5	16	80	0.75
18 ∘	6	20	120	0.8

Appendix

Table 8: The feasible parameters for the trajectory of pseudo-video.

Figure 6: Demos of our collected dataset.

Instance Segmentation

Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.

See all 159 papers

PanoWorld: Towards Spatial Supersensing in 360 $^\circ$ Panorama World
May 13, 2026Changpeng Wang, Xin Lin, Junhan Liu +53D Spatial Reasoning 3D Scene Understanding
1Zhejiang University · University of California, San Diego · University of California, Irvine +1
PanoSeg3R: Feed-Forward 3D Semantic Segmentation for Panoramic Images with an Automatic Data Curation Pipeline
Sep 19, 2026Heechan Yoon, Dongki Jung, Phuc Nguyen +23D Semantic Segmentation
University of Maryland, College Park, USA
Unified Panoramic Geometry Estimation via Multi-View Foundation Models
May 25, 2026Vukasin Bozic, Isidora Slavkovic, Dominik Narnhofer +43D Foundation Models 3D ViTs
ETH Zürich · Google · Athlence Sports +1

May 13, 2026J/K move · Enter open · S save

PanoWorld: Towards Spatial Supersensing in 360 $^\circ$ Panorama World

Changpeng Wang, Xin Lin, Junhan Liu, Yuheng Liu+4

1Zhejiang University · University of California, San Diego · University of California, Irvine+1

3D Spatial Reasoning 3D Scene Understanding 3D VLMs

Open paper AskPDF arXiv

Multimodal large laboratory models (MLLMs) still struggle with spatial understanding under the dominant perspective-image paradigm, which inherits the narrow field of view of human-like perception. For navigation, robotic search, and 3D scene understanding, 360-degree panoramic sensing offers a form of supersensing by capturing the entire surrounding environment at once. However, existing MLLM pipelines typically decompose panoramas into multiple perspective views, leaving the spherical structure of equirectangular projection (ERP) largely implicit. In this paper, we study pano-native understanding, which requires an MLLM to reason over an ERP panorama as a continuous, observer-centered space. To this end, we first define the key abilities for pano-native understanding, including semantic anchoring, spherical localization, reference-frame transformation, and depth-aware 3D spatial reasoning. We then build a large-scale metadata construction pipeline that converts mixed-source ERP panoramas into geometry-aware, language-grounded, and depth-aware supervision, and instantiate these signals as capability-aligned instruction tuning data. On the model side, we introduce PanoWorld with Spherical Spatial Cross-Attention, which injects spherical geometry into the visual stream. We further construct PanoSpace-Bench, a diagnostic benchmark for evaluating ERP-native spatial reasoning. Experiments show that PanoWorld substantially outperforms both proprietary and open-source baselines on PanoSpace-Bench, H* Bench, and R2R-CE Val-Unseen benchmarks. These results demonstrate that robust panoramic reasoning requires dedicated pano-native supervision and geometry-aware model adaptation. All source code and proposed data will be publicly released.

Abstract

Instance Segmentation

Explore similar work

PanoWorld: Towards Spatial Supersensing in 360∘^\circ∘ Panorama World

PanoSeg3R: Feed-Forward 3D Semantic Segmentation for Panoramic Images with an Automatic Data Curation Pipeline

Unified Panoramic Geometry Estimation via Multi-View Foundation Models

PanoWorld: Towards Spatial Supersensing in 360 $^\circ$ Panorama World