Long-term video object segmentation (VOS) remains challenging due to error accumulation under extended occlusions, re-appearance, and scene changes. Although SAM2 provides strong zero-shot performance, its streaming memory can amplify drift over long horizons when recent, unreliable predictions dominate the memory state. We propose SAM2Dual, a training-free, plug-and-play inference-time enhancement that improves long-video robustness without updating model weights. SAM2Dual introduces a Dual Memory design that explicitly separates (i) short-term memory for rapid local adaptation and (ii) long-term memory built via interval-based sampling to preserve global identity cues, combined through a gated fusion strategy. In addition, we present Text-Aware Memory (TAM), which extracts a compact word-level cue from early frames and uses text embeddings to reweight memory contributions based on semantic compatibility, supporting identity preservation when visual evidence becomes weak or ambiguous. Across long-term benchmarks, SAM2Dual consistently improves stability on long videos, raising J&F from 49.33 to 50.65 on MOSEv2 and achieving consistent gains on LVOSv2.
Figures & tables
Fig. 1: A qualitative comparison on a long sequence (over 1,300 frames) shows that, under the same first-frame mask initialization, SAM2 progressively drifts and loses the target in later frames (e.g., Frames 1016 and 1300) under large scale and pose variations, whereas the proposed approach maintains consistent segmentation throughout the video, effectively suppressing long-term drift.
Fig. 2: Overall architecture of the proposed method. Given video frames, the model reads from short-term (ST) and long-term (LT) memories, fuses the readouts, and decodes masks for each frame. Notation: m : long-term update interval; α : ST/LT fusion gate; λword : text modulation strength; g : text gate.
Component
Setting
Input
long side 1024 , mask@t1
Dual Memory
K=7 , L=7 , m=10 , α=0.63
TAM
N=5 , Kword=50 , λword=0.1 , gate K&V
TABLE I: Reference setting used for all experiments.
Dataset / split
Role
Used in
DAVIS_train [ 8 , 10 ]
HP tuning
tuning only
DAVIS_dev [ 8 , 10 ]
Appendix eval
Appendix only
YouTube-VOS_val [ 11 ]
Appendix eval
Appendix only
MOSEv2 [ 12 ]
evaluation
main & Appendix
LVOSv1_val [ 13 ]
Appendix eval
Appendix only
LVOSv2_train [ 14 ]
HP tuning
tuning only
TABLE II: Datasets/splits and their roles in our study (all inference-only).
Statistic
MOSEv2
LVOSv2_val
PUMaVOS_val
Year
2025
2024
2023
Avg. frames/video
154
472
883
Videos
433
140
24
Objects
575
239
26
TABLE III: Summary statistics of the main evaluation benchmarks used for the main results.
Method
MOSEv2
LVOSv2
PUMaVOS
DeAOT [ 4 ]
OOM
OOM
OOM
XMem++ [ 15 ]
36.20
62.6
62.3
Cutie [ 16 ]
41.90
77.7
65.9
SAM2 [ 1 ]
49.33
82.1
81.0
SAMURAI [ 17 ]
47.82
82.1
79.4
HiM2SAM [ 18 ]
45.90
81.7
78.5
TABLE IV: J&F (%) on long-term VOS benchmarks (MOSEv2, LVOSv2, and PUMaVOS; higher is better). Best in bold and second best underlined ; OOM indicates out-of-memory.
Fig. 3: Per-sequence F scores ordered by sequence length.
Fig. 4: Per-sequence J scores ordered by sequence length.
Model / Setting
MOSEv2
LVOSv2_val
PUMaVOS_val
Dual only
50.4
82.9
80.7
TAM only
49.33
82.3
81.0
Dual + TAM
50.65
83.1
80.6
TABLE V: J&F (%) on MOSEv2, LVOSv2_val, and PUMaVOS_val. Best in bold , second best underlined .
Fig. 5: Qualitative ablation on a long elephant sequence. Rows (top to bottom): SAM2, SAM2 + Dual Memory, SAM2 + TAM, and SAM2Dual. SAM2 progressively erodes the mask and loses the target in later frames, whereas SAM2Dual maintains stable target coverage over time.
Fig. 6: Qualitative ablation on a long car sequence. Rows (top to bottom): SAM2, SAM2 + Dual Memory, SAM2 + TAM, and SAM2Dual. SAM2 progressively erodes the mask and loses the target in later frames, whereas SAM2Dual maintains stable target coverage over time.
λword
α
m=5
m=10
m=15
0.10
0.50
92.20
92.30
92.10
0.63
92.50
92.80
92.40
0.75
92.20
92.20
92.00
0.05
0.50
92.40
92.30
92.50
0.63
92.00
91.60
92.40
0.75
92.20
91.70
92.20
TABLE VI: Hyperparameter sweep on LVOSv2 train (J&F, %). We vary λword∈{0.10,0.05,0.01} , α∈{0.50,0.63,0.75} , and m∈{5,10,15} . Best is in bold .
λword
α
m=5
m=10
m=15
0.10
0.50
90.60
90.80
90.60
0.63
90.90
90.90
90.60
0.75
90.90
90.60
90.50
0.05
0.50
90.60
90.80
90.60
0.63
90.60
90.90
90.50
0.75
90.90
90.60
90.50
TABLE VII: Hyperparameter sweep on DAVIS train (J&F, %). We vary λword∈{0.10,0.05,0.01} , α∈{0.50,0.63,0.75} , and m∈{5,10,15} . Best is in bold .
Fig. 7: Representative failure case on PUMaVOS with a fine-grained, part-level target (shoe), where SAM2Dual drifts and switches identity to the full person in later frames.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Method
DAVIS
YouTube-VOS
MOSEv2
LVOSv1
LVOSv2
PUMaVOS
DeAOT [ 4 ]
82.19
86.25
OOM
OOM
OOM
OOM
XMem++ [ 15 ]
80.94
84.00
36.20
44.13
62.6
62.3
Cutie [ 16 ]
86.37
86.15
41.90
66.65
77.7
65.9
SAM2 [ 1 ]
88.59
88.81
49.33
76.72
82.1
81.0
SAMURAI [ 17 ]
88.98
88.51
47.82
75.90
82.1
79.4
HiM2SAM [ 18 ]
88.77
88.64
45.90
67.31
81.7
78.5
Appendix
TABLE VIII: J&F (%) comparison on VOS benchmarks under a previous setting.