SAM2Dual: Training-Free, Dual Memory for Long-Term Video Object Segmentation
Organizations: Department of Applied Statistics, Chung-Ang University, Seoul 06974, Republic of Korea
Abstract
Long-term video object segmentation (VOS) remains challenging due to error accumulation under extended occlusions, re-appearance, and scene changes. Although SAM2 provides strong zero-shot performance, its streaming memory can amplify drift over long horizons when recent, unreliable predictions dominate the memory state. We propose SAM2Dual, a training-free, plug-and-play inference-time enhancement that improves long-video robustness without updating model weights. SAM2Dual introduces a Dual Memory design that explicitly separates (i) short-term memory for rapid local adaptation and (ii) long-term memory built via interval-based sampling to preserve global identity cues, combined through a gated fusion strategy. In addition, we present Text-Aware Memory (TAM), which extracts a compact word-level cue from early frames and uses text embeddings to reweight memory contributions based on semantic compatibility, supporting identity preservation when visual evidence becomes weak or ambiguous. Across long-term benchmarks, SAM2Dual consistently improves stability on long videos, raising J&F from 49.33 to 50.65 on MOSEv2 and achieving consistent gains on LVOSv2.
Figures & tables
| Component | Setting |
|---|---|
| Input | long side , mask@t1 |
| Dual Memory | , , , |
| TAM | , , , gate K&V |
| Dataset / split | Role | Used in |
|---|---|---|
| DAVIS_train [ 8 , 10 ] | HP tuning | tuning only |
| DAVIS_dev [ 8 , 10 ] | Appendix eval | Appendix only |
| YouTube-VOS_val [ 11 ] | Appendix eval | Appendix only |
| MOSEv2 [ 12 ] | evaluation | main & Appendix |
| LVOSv1_val [ 13 ] | Appendix eval | Appendix only |
| LVOSv2_train [ 14 ] | HP tuning | tuning only |
| Statistic | MOSEv2 | LVOSv2_val | PUMaVOS_val |
|---|---|---|---|
| Year | 2025 | 2024 | 2023 |
| Avg. frames/video | 154 | 472 | 883 |
| Videos | 433 | 140 | 24 |
| Objects | 575 | 239 | 26 |
| Method | MOSEv2 | LVOSv2 | PUMaVOS |
|---|---|---|---|
| DeAOT [ 4 ] | OOM | OOM | OOM |
| XMem++ [ 15 ] | 36.20 | 62.6 | 62.3 |
| Cutie [ 16 ] | 41.90 | 77.7 | 65.9 |
| SAM2 [ 1 ] | 49.33 | 82.1 | 81.0 |
| SAMURAI [ 17 ] | 47.82 | 82.1 | 79.4 |
| HiM2SAM [ 18 ] | 45.90 | 81.7 | 78.5 |
| Model / Setting | MOSEv2 | LVOSv2_val | PUMaVOS_val |
|---|---|---|---|
| Dual only | 50.4 | 82.9 | 80.7 |
| TAM only | 49.33 | 82.3 | 81.0 |
| Dual + TAM | 50.65 | 83.1 | 80.6 |
| 0.10 | 0.50 | 92.20 | 92.30 | 92.10 |
| 0.63 | 92.50 | 92.80 | 92.40 | |
| 0.75 | 92.20 | 92.20 | 92.00 | |
| 0.05 | 0.50 | 92.40 | 92.30 | 92.50 |
| 0.63 | 92.00 | 91.60 | 92.40 | |
| 0.75 | 92.20 | 91.70 | 92.20 |
| 0.10 | 0.50 | 90.60 | 90.80 | 90.60 |
| 0.63 | 90.90 | 90.90 | 90.60 | |
| 0.75 | 90.90 | 90.60 | 90.50 | |
| 0.05 | 0.50 | 90.60 | 90.80 | 90.60 |
| 0.63 | 90.60 | 90.90 | 90.50 | |
| 0.75 | 90.90 | 90.60 | 90.50 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Method | DAVIS | YouTube-VOS | MOSEv2 | LVOSv1 | LVOSv2 | PUMaVOS |
|---|---|---|---|---|---|---|
| DeAOT [ 4 ] | 82.19 | 86.25 | OOM | OOM | OOM | OOM |
| XMem++ [ 15 ] | 80.94 | 84.00 | 36.20 | 44.13 | 62.6 | 62.3 |
| Cutie [ 16 ] | 86.37 | 86.15 | 41.90 | 66.65 | 77.7 | 65.9 |
| SAM2 [ 1 ] | 88.59 | 88.81 | 49.33 | 76.72 | 82.1 | 81.0 |
| SAMURAI [ 17 ] | 88.98 | 88.51 | 47.82 | 75.90 | 82.1 | 79.4 |
| HiM2SAM [ 18 ] | 88.77 | 88.64 | 45.90 | 67.31 | 81.7 | 78.5 |
Video Object Segmentation
Video object segmentation assigns pixel-level masks to objects across frames while preserving their identities through motion, occlusion, and change.
See all 110 papers