360 video object segmentation (360VOS) aims to predict temporally-consistent masks in 360 videos, offering full-scene coverage, benefiting applications, such as VR/AR and embodied AI. Learning 360VOS model is nontrivial due to the lack of high-quality labeled dataset. Recently, Segment Anything Models (SAMs), especially SAM2 -- with its design of memory module -- shows strong, promptable VOS capability. However, directly using SAM2 for 360VOS yields implausible results as 360 videos suffer from the projection distortion, semantic inconsistency of left-right sides, and sparse object mask information in SAM2's memory. To this end, we propose PanoSAM2, a novel 360VOS framework based on our lightweight distortion- and memory-aware adaptation strategies of SAM2 to achieve reliable 360VOS while retaining SAM2's user-friendly prompting design. Concretely, to tackle the projection distortion and semantic inconsistency issues, we propose a Pano-Aware Decoder with seam-consistent receptive fields and iterative distortion refinement to maintain continuity across the 0/360 degree boundary. Meanwhile, a Distortion-Guided Mask Loss is introduced to weight pixels by distortion magnitude, stressing stretched regions and boundaries. To address the object sparsity issue, we propose a Long-Short Memory Module to maintain a compact long-term object pointer to re-instantiate and align short-term memories, thereby enhancing temporal coherence. Extensive experiments show that PanoSAM2 yields substantial gains over SAM2: +5.6 on 360VOTS and +6.7 on PanoVOS, showing the effectiveness of our method.
Figures & tables
Figure 2: Overview of our PanoSAM2 framework . Compared with SAM2 [ 31 ] , it has two architectural contributions: Pano-Aware (PA) Decoder and Long-Short Memory Module (LSMM).
Table 1: Comparisons between our method and prior arts on the 360VOTS test dataset. Our PanoSAM2 achieves a new state-of-the-art performance. The best results are shown in bold , and * means the method is re-implemented and reproduced.
VOS Tracker
Backbone
PanoVOS Validation
PanoVOS Test
J&F
Js
Fs
Ju
Fu
J&F
Js
Fs
Ju
Fu
Perspective-Video-Specified
CFBI [ 41 ]
ResNet-101
35.8
34.6
44.8
24.2
39.7
19.1
18.2
26.1
12.2
19.8
CFBI+ [ 41 ]
ResNet-101
41.3
38.0
47.9
32.5
46.9
30.9
30.8
42.7
21.4
28.5
AFB-URR [ 16 ]
ResNet-50
34.3
34.8
42.8
24.9
34.5
34.2
28.2
38.8
32.9
36.8
STCN [ 6 ]
ResNet-50
52.0
51.2
60.8
41.5
54.5
50.8
43.6
56.5
49.3
53.7
Table 2: Comparisons between our method and existing approaches on PanoVOS validation and test datasets. The best results are shown in bold . Evaluation metric subscript s and u denote scores in seen and unseen categories compared to the training dataset.
Method
PanoVOS Validation
PanoVOS Test
J&F
J
F
J&F
J
F
PerSAM [ 45 ]
19.1
14.9
23.5
19.5
15.6
23.3
SAM-PT [ 30 ]
47.5
41.4
53.7
41.0
35.7
46.4
SAM-I2V [ 21 ]
48.9
42.8
55.0
40.5
34.9
46.1
SAM2 [ 31 ]
70.6
63.4
77.8
66.3
59.4
73.2
XMem [ 5 ]
52.1
46.9
57.2
44.4
39.6
49.1
Table 3: Comparisons of generalization ability between our method pretrained on 360VOTS and prior arts on PanoVOS validation and test dataset. The best results are shown in bold .
Figure 5: Qualitative comparison between PanoSAM2 and other VOS models. The orange boxes highlight the areas where the target object straddles the 0°/360° seam, showing PanoSAM2’s improved segmentation precision in complex scenes at different time steps.
Figure 6: Visualization of zero-shot results from PanoSAM2 and other VOS models on the PanoVOS dataset.
Figure 9
PA Decoder
LSMM
Lmask
J&F
J
F
59.4
53.9
64.9
✓
62.7
57.6
67.8
✓
61.6
57.4
65.8
✓
60.1
54.7
65.5
✓
✓
✓
64.3
59.2
69.3
Table 4: Ablation study on key components of PanoSAM2. The best result is shown in bold .
Table 11
Figure 9: Examples of failure cases. Orange bounding boxes highlight and zoom in on the small mask region.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Table 13
Model
Backbone
LVOSv2 val
SA-V val
SA-V test
SAM2
Hiera-T
77.3
75.2
76.5
SAM2 ‡
Hiera-T
70.1
64.5
65.4
PanoSAM2 ‡
Hiera-T
68.7
63.5
64.1
Appendix
Table 9: J&F of SAM2-based tackers on perspective VOS datasets, where ‡ stands for fine-tuned on 360VOTS.
Table 15
Figure 11: More qualitative comparison between PanoSAM2 and other VOS models. Orange bounding boxes highlight and zoom in on the small mask region.
Figure 12: More ablation visualizations of proposed modules. Orange bounding boxes highlight and zoom in on the small mask region.
Figure 13: More visual results on self-collected open-world 360 scene.
Long-term video object segmentation (VOS) remains challenging due to error accumulation under extended occlusions, re-appearance, and scene changes. Although SAM2 provides strong zero-shot performance, its streaming memory can amplify drift over long horizons when recent, unreliable predictions dominate the memory state. We propose SAM2Dual, a training-free, plug-and-play inference-time enhancement that improves long-video robustness without updating model weights. SAM2Dual introduces a Dual Memory design that explicitly separates (i) short-term memory for rapid local adaptation and (ii) long-term memory built via interval-based sampling to preserve global identity cues, combined through a gated fusion strategy. In addition, we present Text-Aware Memory (TAM), which extracts a compact word-level cue from early frames and uses text embeddings to reweight memory contributions based on semantic compatibility, supporting identity preservation when visual evidence becomes weak or ambiguous. Across long-term benchmarks, SAM2Dual consistently improves stability on long videos, raising J&F from 49.33 to 50.65 on MOSEv2 and achieving consistent gains on LVOSv2.
JeongRae Kim, Changwon Lim
Department of Applied Statistics, Chung-Ang University, Seoul 06974, Republic of Korea
We present SAM3Dual, our third-place solution to the MOSEv2 track of the 8th Large-scale Video Object Segmentation (LSVOS) Challenge at ECCV 2026. SAM3Dual is a training-free inference extension of pretrained SAM 3 that explicitly separates temporal memory into a short-term branch for recent observations and a long-term branch for interval-sampled historical representations. The two memory responses are combined using a deterministic sequence-relative fusion schedule and conservatively modulated by the previous-frame object confidence. All pretrained SAM 3 parameters remain frozen, requiring no task-specific training, fine-tuning, test-time training, or online parameter optimization. The complete system achieved an official J&F score of 64.37 and ranked third in the MOSEv2 track. This result highlights the potential of reorganizing temporal memory entirely at inference time to obtain competitive long-term VOS performance while preserving the pretrained model.
Video Object Segmentation (VOS) in complex and long videos is increasingly important for real-world applications, where target objects often appear only intermittently within long temporal horizons. However, existing benchmarks largely focus on temporally salient objects that remain visible for most of the video. To address this gap, we introduce FaVOS (A Benchmark for Video Object Segmentation with Fractional Temporal Visibility), a benchmark designed to evaluate VOS methods under low temporal visibility. We show that, in this regime, the standard J&F metric can collapse VOS evaluation into absence classification, because empty predictions receive high rewards on target-absent frames. Consequently, even a trivial empty-mask predictor can outperform strong models such as SAM 3, revealing a fundamental mismatch between current metrics and practical VOS performance. To mitigate this issue, we propose Volumetric J&F, which evaluates mask sequences as spatio-temporal volumes and reduces the dominance of target-absence rewards while preserving sensitivity to segmentation quality and temporal structure. Project page: https://aidaslab.github.io/FaVOS.
Jihwan Hong, Woohyeon Park, Jaeik Kim +1
AIDAS Laboratory, 1IPAI & 2ECE, Seoul National University