PanoSAM2: Lightweight Distortion- and Memory-aware Adaptions of SAM2 for 360 Video Object Segmentation

Dingwen Xiao; Weiming Zhang; Shiqi Wen; Lin Wang

cs.CVApr 9, 2026

PanoSAM2: Lightweight Distortion- and Memory-aware Adaptions of SAM2 for 360 Video Object Segmentation

Authors: Dingwen Xiao, Weiming Zhang, Shiqi Wen, Lin Wang

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University

Abstract

360 video object segmentation (360VOS) aims to predict temporally-consistent masks in 360 videos, offering full-scene coverage, benefiting applications, such as VR/AR and embodied AI. Learning 360VOS model is nontrivial due to the lack of high-quality labeled dataset. Recently, Segment Anything Models (SAMs), especially SAM2 -- with its design of memory module -- shows strong, promptable VOS capability. However, directly using SAM2 for 360VOS yields implausible results as 360 videos suffer from the projection distortion, semantic inconsistency of left-right sides, and sparse object mask information in SAM2's memory. To this end, we propose PanoSAM2, a novel 360VOS framework based on our lightweight distortion- and memory-aware adaptation strategies of SAM2 to achieve reliable 360VOS while retaining SAM2's user-friendly prompting design. Concretely, to tackle the projection distortion and semantic inconsistency issues, we propose a Pano-Aware Decoder with seam-consistent receptive fields and iterative distortion refinement to maintain continuity across the 0/360 degree boundary. Meanwhile, a Distortion-Guided Mask Loss is introduced to weight pixels by distortion magnitude, stressing stretched regions and boundaries. To address the object sparsity issue, we propose a Long-Short Memory Module to maintain a compact long-term object pointer to re-instantiate and align short-term memories, thereby enhancing temporal coherence. Extensive experiments show that PanoSAM2 yields substantial gains over SAM2: +5.6 on 360VOTS and +6.7 on PanoVOS, showing the effectiveness of our method.

Figures & tables

Figure 2: Overview of our PanoSAM2 framework . Compared with SAM2 [ 31 ] , it has two architectural contributions: Pano-Aware (PA) Decoder and Long-Short Memory Module (LSMM).

Figure 3: Overview of our LSMM framework.

Figure 4: Distortion-guided 360 mask weight calculation.

VOS Tracker	Backbone	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	Memory (GB)	GFLOPs	FPS
Perspective-Video-Specified
CFBI [ 41 ]	ResNet-101	46.3	41.2	51.3	-	-	-
CFBI+ [ 43 ]	ResNet-101	48.0	42.8	53.2	-	-	-
TarVis [ 1 ]	Swin-L	36.8	32.5	41.1	-	-	-
GMVOS [ 19 ]	ResNet-50	47.7	43.1	52.2	-	-	-
UNICORN [ 38 ]	ConvNeXt-L	40.4	33.9	46.9	-	-	-

Table 1: Comparisons between our method and prior arts on the 360VOTS test dataset. Our PanoSAM2 achieves a new state-of-the-art performance. The best results are shown in bold , and * means the method is re-implemented and reproduced.

Perspective-Video-Specified
VOS Tracker	Backbone	PanoVOS Validation					PanoVOS Test
VOS Tracker	Backbone	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}_{s}$	$\mathcal{F}_{s}$	$\mathcal{J}_{u}$	$\mathcal{F}_{u}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}_{s}$	$\mathcal{F}_{s}$	$\mathcal{J}_{u}$	$\mathcal{F}_{u}$
CFBI [ 41 ]	ResNet-101	35.8	34.6	44.8	24.2	39.7	19.1	18.2	26.1	12.2	19.8
CFBI+ [ 41 ]	ResNet-101	41.3	38.0	47.9	32.5	46.9	30.9	30.8	42.7	21.4	28.5
AFB-URR [ 16 ]	ResNet-50	34.3	34.8	42.8	24.9	34.5	34.2	28.2	38.8	32.9	36.8
STCN [ 6 ]	ResNet-50	52.0	51.2	60.8	41.5	54.5	50.8	43.6	56.5	49.3	53.7

Table 2: Comparisons between our method and existing approaches on PanoVOS validation and test datasets. The best results are shown in bold . Evaluation metric subscript

s

and

u

denote scores in seen and unseen categories compared to the training dataset.

Method	PanoVOS Validation			PanoVOS Test
Method	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
PerSAM [ 45 ]	19.1	14.9	23.5	19.5	15.6	23.3
SAM-PT [ 30 ]	47.5	41.4	53.7	41.0	35.7	46.4
SAM-I2V [ 21 ]	48.9	42.8	55.0	40.5	34.9	46.1
SAM2 [ 31 ]	70.6	63.4	77.8	66.3	59.4	73.2
XMem [ 5 ]	52.1	46.9	57.2	44.4	39.6	49.1

Table 3: Comparisons of generalization ability between our method pretrained on 360VOTS and prior arts on PanoVOS validation and test dataset. The best results are shown in bold .

Figure 5: Qualitative comparison between PanoSAM2 and other VOS models. The orange boxes highlight the areas where the target object straddles the 0°/360° seam, showing PanoSAM2’s improved segmentation precision in complex scenes at different time steps.

Figure 6: Visualization of zero-shot results from PanoSAM2 and other VOS models on the PanoVOS dataset.

PA Decoder	LSMM	$\mathcal{L}_{mask}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
			59.4	53.9	64.9
✓			62.7	57.6	67.8
	✓		61.6	57.4	65.8
		✓	60.1	54.7	65.5
✓	✓	✓	64.3	59.2	69.3

Table 4: Ablation study on key components of PanoSAM2. The best result is shown in bold .

Table 11

Figure 9: Examples of failure cases. Orange bounding boxes highlight and zoom in on the small mask region.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Table 13

Model	Backbone	LVOSv2 val	SA-V val	SA-V test
SAM2	Hiera-T	77.3	75.2	76.5
SAM2 ${}^{\text{\textdaggerdbl}}$	Hiera-T	70.1	64.5	65.4
PanoSAM2 ${}^{\text{\textdaggerdbl}}$	Hiera-T	68.7	63.5	64.1

Appendix

Table 9:

\mathcal{J}\&\mathcal{F}

of SAM2-based tackers on perspective VOS datasets, where

{}^{\text{\textdaggerdbl}}

stands for fine-tuned on 360VOTS.

Figure 11: More qualitative comparison between PanoSAM2 and other VOS models. Orange bounding boxes highlight and zoom in on the small mask region.

Figure 12: More ablation visualizations of proposed modules. Orange bounding boxes highlight and zoom in on the small mask region.

Figure 13: More visual results on self-collected open-world 360 scene.

Video Object Segmentation

Video object segmentation assigns pixel-level masks to objects across frames while preserving their identities through motion, occlusion, and change.

See all 114 papers

SAM2Dual: Training-Free, Dual Memory for Long-Term Video Object Segmentation
Aug 19, 2026JeongRae Kim, Changwon LimVideo Object Segmentation Memory-Augmented Video Understanding
Department of Applied Statistics, Chung-Ang University, Seoul 06974, Republic of Korea
SAM3Dual: A 3rd Place Solution to the MOSEv2 Track, 8th LSVOS Challenge
Aug 23, 2026JeongRae Kim, Chaehyun Kim, Changwon LimVideo Object Segmentation
Chung-Ang University, Seoul, Republic of Korea
When Predicting Nothing Beats SAM 3: Revisiting Evaluation in Video Object Segmentation
Oct 2, 2026Jihwan Hong, Woohyeon Park, Jaeik Kim +1Video Object Segmentation Visual Object Tracking
AIDAS Laboratory, 1IPAI & 2ECE, Seoul National University

Aug 19, 2026J/K move · Enter open · S save

SAM2Dual: Training-Free, Dual Memory for Long-Term Video Object Segmentation

JeongRae Kim, Changwon Lim

Department of Applied Statistics, Chung-Ang University, Seoul 06974, Republic of Korea

Video Object Segmentation Memory-Augmented Video Understanding

Open paper AskPDF arXiv

Long-term video object segmentation (VOS) remains challenging due to error accumulation under extended occlusions, re-appearance, and scene changes. Although SAM2 provides strong zero-shot performance, its streaming memory can amplify drift over long horizons when recent, unreliable predictions dominate the memory state. We propose SAM2Dual, a training-free, plug-and-play inference-time enhancement that improves long-video robustness without updating model weights. SAM2Dual introduces a Dual Memory design that explicitly separates (i) short-term memory for rapid local adaptation and (ii) long-term memory built via interval-based sampling to preserve global identity cues, combined through a gated fusion strategy. In addition, we present Text-Aware Memory (TAM), which extracts a compact word-level cue from early frames and uses text embeddings to reweight memory contributions based on semantic compatibility, supporting identity preservation when visual evidence becomes weak or ambiguous. Across long-term benchmarks, SAM2Dual consistently improves stability on long videos, raising J&F from 49.33 to 50.65 on MOSEv2 and achieving consistent gains on LVOSv2.

Abstract

Video Object Segmentation

Explore similar work

SAM2Dual: Training-Free, Dual Memory for Long-Term Video Object Segmentation

SAM3Dual: A 3rd Place Solution to the MOSEv2 Track, 8th LSVOS Challenge

When Predicting Nothing Beats SAM 3: Revisiting Evaluation in Video Object Segmentation