MESAM: Enhancing SAM2 for Few-Shot Segmentation through Semantic-Visual Memory Generation and Refinement
Authors: Tianyu Zou, Shengwu Xiong, Rui Wu, Ruilin Yao, Yi Rong
Organizations: Sanya Science and Education Innovation Park, Wuhan University of Technology Sanya, Hainan, China · School of Artificial Intelligence, Wuhan University of Technology Wuhan, Hubei, China · Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology Wuhan, Hubei, China · Institute of Automation, Chinese Academy of Sciences Beijing, China
This paper investigates how to leverage the powerful capabilities of the Segment Anything Model 2 (SAM2) to tackle few-shot segmentation (FSS) tasks. One major challenge lies in the substantial appearance differences between the support and query images, which may diminish the effectiveness of the memory-matching process in SAM2. Through systematic analysis, we find that the memory generated based on the categorical semantic information is robust to this appearance divergence and is more compatible with SAM2. Based on these observations, we propose a new Memory-Enhanced SAM2 (MESAM) method, which first introduces a Semantic-Visual Memory Generation (SVMG) module to separately construct a query semantic memory, a query foreground visual memory, and a support adaptive visual memory. Then, in our designed Interactive Semantic Memory Refinement (ISMR) module, we further enrich the obtained semantic memory with fine-grained visual details, and suppress its misatcivated background responses. Extensive experiments on the PASCAL-5i, COCO-20i, and LVIS-92i datasets demonstrate that MESAM consistently outperforms existing FSS methods, achieving new state-of-the-art segmentation performance. Our source code is available at: https://github.com/tianyu-zou/MESAM.
Figures & tables
Figure 1. Illustration of the large support-query appearance divergence challenge. With only partial visual cues provided by the support image, existing methods segment only visually similar parts in the query, instead of the entire object. In contrast, with the refined semantic memory, our MESAM method yields better segmentation results. The orange and blue circles indicate foreground (FG) misses and background (BG) misactivations, respectively.
Figure 2. The priors and the segmentation results of two visual memories (i.e., FG, DISC) and our semantic memory.
Memory
FG
DISC
Semantic
PASCAL- 5i
81.1
78.1
82.8
COCO- 20i
55.9
56.0
58.9
Table 1. Performance comparison between the visual memories constructed in FSSAM and our semantic memory.
PASCAL- 5i
COCO- 20i
Memory
FP
FN
FP
FN
FG
0.0865
0.0436
0.1271
0.0381
DISC
0.0186
0.1056
0.0324
0.0820
Semantic
0.0139
0.1207
0.0155
0.0915
Table 2. The false positive (FP) and false negative (FN) rates of the prior masks used for generating FG and DISC visual memories, and our semantic memory.
Figure 3. Performance comparison between different memories obtained under varying dilation and erosion ratios.
Figure 4. Illustration of our MESAM method. (a) The SVMG module generates three initial memories Memqs , Memqv , and Memsv . (b) In the ISMR module, Memqs is enhanced by interacting with Memqv through the cross-attention operation, while Memsv is used to suppress the BG noise in the attention scores. Subsequently, the refined semantic memory Memqs′ is processed by SAM2’s memory attention module and mask decoder to produce the final segmentation mask.
1-shot
5-shot
Methods
Backbone
Venue
50
51
52
53
Mean
FB-IoU
50
51
52
53
Mean
FB-IoU
ABCBNet ( Zhu et al., 2024 )
ResNet50
CVPR’24
72.9
76.0
69.5
64.0
70.6
-
74.4
78.0
73.9
68.3
73.6
-
PAHNet ( Zou et al., 2025 )
ResNet50
ICCV‘25
72.1
76.4
71.0
66.6
71.5
82.4
75.8
78.6
76.9
73.2
76.1
85.7
OCNet ( Wen et al., 2025 )
ResNet50
ICCV‘25
73.5
75.9
71.1
64.9
71.4
82.2
75.9
77.1
74.1
70.9
74.5
84.7
Matcher ( Liu et al., 2024b )
SAM-H
ICLR’24
67.7
70.7
66.9
67.0
68.1
-
71.4
77.5
74.1
72.8
74.0
-
VRP-SAM ( Sun et al., 2024 )
SAM-H
CVPR’24
73.9
78.3
70.6
65.1
71.9
-
-
-
-
-
-
-
Table 3. Performance comparisons on PASCAL- 5i . “ 5i ” shows the mIoU scores of 5 novel classes in fold i , “Mean” denotes the averaged mIoU score across all four folds. The best and second-best results are highlighted in bold and underlined , respectively.
1-shot
5-shot
Methods
Backbone
Venue
200
201
202
203
Mean
FB-IoU
200
201
202
203
Mean
FB-IoU
ABCBNett ( Zhu et al., 2024 )
ResNet50
CVPR’24
44.2
54.0
52.1
49.8
50.0
-
50.5
59.1
57.0
53.6
55.1
-
PAHNet ( Zou et al., 2025 )
ResNet50
ICCV‘25
45.2
58.2
54.4
51.2
52.3
75.2
53.0
64.1
60.8
58.7
59.2
78.1
OCNet ( Wen et al., 2025 )
ResNet50
ICCV‘25
45.9
56.9
52.9
50.4
51.5
73.7
52.7
63.1
57.4
54.8
57.0
76.8
Matcher ( Liu et al., 2024b )
SAM-H
ICLR’24
52.7
53.5
52.6
52.1
52.7
-
60.1
62.7
60.9
59.2
60.7
-
VRP-SAM ( Sun et al., 2024 )
SAM-H
CVPR’24
56.0
61.3
57.9
58.8
58.5
-
61.4
69.4
65.9
64.9
65.4
-
Table 4. Performance comparisons on COCO- 20i .“ 20i ” shows the mIoU scores of 20 novel classes in fold i , “Mean” denotes the averaged mIoU score across all four folds. The best and second-best results are highlighted in bold and underlined , respectively.
Methods
920
921
922
923
924
925
926
927
928
929
Mean
FB-IoU
Matcher ( Liu et al., 2024b )
31.4
30.9
33.7
38.1
30.5
32.5
35.9
34.2
33.0
29.7
33.0
66.2
SINE ( Liu et al., 2024a )
28.3
31.0
31.9
34.6
30.0
31.9
32.2
33.7
30.6
27.8
31.2
63.5
FSSAM ( Xu et al., 2025b )
34.7
37.8
37.2
41.1
33.9
38.1
40.6
38.9
36.9
33.8
37.3
68.4
MESAM (Ours)
38.0
44.7
43.5
44.8
38.2
41.4
44.0
41.2
44.0
39.4
41.9
70.9
Table 5. Performance comparisons on LVIS- 92i . “ 92i ” shows the mIoU scores of 92 novel classes in fold i , “Mean” denotes the averaged mIoU score across all ten folds. The 1-shot results are reported, where the best results are highlighted in bold .
Figure 5. Qualitative comparisons between FSSAM and MESAM on PASCAL- 5i , COCO- 20i and LVIS- 92i under the 1-shot setting.
Settings
200
201
202
203
Mean
A: Only Memqs
51.2
67.1
58.4
58.7
58.9
B: Memqs+Memqv
62.1
68.9
65.1
66.9
65.8
C: Memqs+Memqv+Memsv(Ms)
61.9
69.9
64.7
68.1
66.2
D: Memqs+Memqv+Memsv
63.9
70.4
66.2
68.6
67.3
Table 6. Ablation study on the effect of different memories.
200
201
202
203
Mean
Cosine over K
61.7
70.1
66.7
67.1
66.4
FG Enhancement
62.0
69.8
67.2
67.7
66.7
Memory Swap
61.1
70.5
67.8
67.9
66.8
ISMR in MESAM
63.9
70.4
66.2
68.6
67.3
Table 7. Ablation study on the effect of different ISMR.
Figure 6. The cross-attention weights and the segmentation results obtained without and with the support calibration.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Input: Query ground-truth mask Mq , target ratio r ,
kernel size k , padding parameter pad
Output: Dilated mask M
Stage 1: Initialization
M←Mq ;
kernal← ones( k,k );
initial_area← sum of non-zero pixels in M ;
Appendix
Algorithm 1 Mask Dilation
Figure 7. Qualitative comparisons between dilation, vanilla-erosion, and skeleton-erosion operations with varying target ratios.
Figure 8. The prior masks and the segmentation results of visual memories (FG, DISC) and semantic memory on PASCAL- 5i .
Figure 9. The prior masks and the segmentation results of visual memories (FG, DISC) and semantic memory on COCO- 20i .
Input: Query ground-truth mask Mq , target ratio r ,
kernel size k , padding parameter pad
Output: Eroded mask M
Stage 1: Initialization
M←Mq ;
kernal← ones( k,k );
initial_area← sum of non-zero pixels in M ;
Appendix
Algorithm 2 Mask Erosion
1-shot
5-shot
Seed
50
51
52
53
Mean
50
51
52
53
Mean
default
83.70
88.93
83.74
83.70
85.02
84.21
90.08
86.21
87.50
87.00
0
83.75
89.29
83.88
83.38
85.08
83.65
89.57
86.49
87.69
86.85
1
83.44
88.75
83.47
83.46
84.78
83.95
89.91
86.38
87.45
86.92
2
83.43
89.18
83.38
83.30
84.82
83.92
89.77
86.03
87.87
86.90
3
83.38
89.23
83.82
83.33
84.94
84.11
89.66
86.62
87.15
86.89
Appendix
Table 8. Performance of MESAM on PASCAL- 5i with five random seeds {321 (default), 0, 1, 2, 3}. 1,000 episodes are randomly sampled for evaluation. “Average” and “Std” denote the mean mIoU scores and the standard deviations across different random seeds.
1-shot
5-shot
Seed
200
201
202
203
Mean
200
201
202
203
Mean
default
63.92
70.41
66.17
68.63
67.28
67.22
74.13
66.01
73.32
70.17
0
62.97
70.59
65.13
67.48
66.54
65.59
72.84
68.16
72.64
69.81
1
63.55
70.33
65.46
67.21
66.64
68.52
74.45
63.31
72.79
69.77
2
63.38
70.12
62.33
67.59
65.86
68.16
73.45
66.96
72.70
70.32
3
62.65
70.60
64.94
68.07
66.57
65.61
73.65
67.25
71.85
69.59
Appendix
Table 9. Performance of MESAM on COCO- 20i with five random seeds {321 (default), 0, 1, 2, 3}. 1,000 episodes are randomly sampled for evaluation. “Average” and “Std” denote the mean mIoU scores and the standard deviations across different random seeds.
1-shot
5-shot
#Episodes
200
201
202
203
Mean
Time (s)
200
201
202
203
Mean
Time (s)
1000
63.92
70.41
66.17
68.63
67.28
143.1
67.22
74.13
66.01
73.32
70.17
218.7
4000
65.38
69.86
68.68
66.57
67.62
568.2
66.75
73.81
72.52
72.32
71.35
881.4
10000
63.66
70.35
67.27
65.43
66.68
1406.5
66.38
73.73
70.73
71.84
70.67
2183.2
20000
63.80
70.55
67.41
65.54
66.83
2799.9
66.57
73.87
70.69
71.87
70.75
4425.7
Average
64.19
70.29
67.38
66.54
67.10
1229.4
66.73
73.89
69.99
72.34
70.74
1927.3
Appendix
Table 10. Performance of MESAM on COCO- 20i evaluated with varying numbers of testing episodes {1,000, 4,000, 10,000, 20,000}. “Time” denotes the total inference time for the evaluation process. “Average” and “Std” denote the mean mIoU scores and the standard deviations across different numbers of testing episodes.
Figure 10. Ablation study of the scaling factor α on COCO- 200 .
Backbones
Size (M)
200
201
202
203
Mean
SAM2-T
38.9
62.8
70.6
66.4
69.4
67.3
SAM2-S
46.0
63.9
70.4
66.2
68.6
67.3
SAM2-B
80.8
63.0
69.8
65.6
67.7
66.5
SAM2-L
224.4
62.4
71.1
68.7
68.4
67.7
Appendix
Table 11. Performance comparisons between different MESAM models built upon four SAM2 versions.
Figure 11. Additional qualitative comparisons between FSSAM and MESAM on LVIS- 92i under the 1-shot setting.
Methods
50
51
52
53
Mean
HSNet ( Min et al., 2021 )
47.0
65.2
67.1
77.1
64.1
VAT ( Hong et al., 2022 )
68.3
64.9
67.5
79.8
65.1
PMNet ( Chen et al., 2024 )
71.0
72.3
66.6
63.8
68.4
GF-SAM ( Zhang et al., 2024a )
71.1
75.7
69.2
73.3
72.1
SANSA ( Cuttano et al., 2025 )
67.1
70.9
73.5
78.6
72.5
FSSAM ( Xu et al., 2025b )
79.5
85.7
85.4
76.5
81.8
Appendix
Table 12. Performance comparisons under COCO- 20i→ PASCAL- 5i cross-dataset setting. The 1-shot results are reported, where the best results are highlighted in bold .
Figure 12. Additional qualitative comparisons between FSSAM and MESAM on PASCAL- 5i and COCO- 20i under the 1-shot setting.
Few-Shot Semantic Segmentation (FSS) focuses on segmenting novel object categories from only a handful of annotated examples. Most existing approaches rely on extensive episodic training to learn transferable representations, which is both computationally demanding and sensitive to distribution shifts. In this work, we revisit FSS from the perspective of modern vision foundation models and explore the potential of Segment Anything Model 3 (SAM3) as a training-free solution. By repurposing its Promptable Concept Segmentation (PCS) capability, we adopt a simple spatial concatenation strategy that places support and query images into a shared canvas, allowing a fully frozen SAM3 to perform segmentation without any fine-tuning or architectural changes. Experiments on PASCAL-5i and COCO-20i show that this minimal design already achieves state-of-the-art performance, outperforming many heavily engineered methods. Beyond empirical gains, we uncover that negative prompts can be counterproductive in few-shot settings, where they often weaken target representations and lead to prediction collapse despite their intended role in suppressing distractors. These findings suggest that strong cross-image reasoning can emerge from simple spatial formulations, while also highlighting limitations in how current foundation models handle conflicting prompt signals. Code at: https://github.com/WongKinYiu/FSS-SAM3
Yi-Jen Tsai, Yen-Yu Lin, Chien-Yao Wang
National Yang Ming Chiao Tung University · Institute of Information Science, Academia Sinica, Taiwan
Few-shot segmentation (FSS) aims to segment target objects with limited annotated examples. Recent studies have begun to explore large foundation models to improve generalization for this task. In particular, SAM 3 provides strong visual representations and supports text-guided segmentation. However, its performance remains limited in specialized domains, where semantic prompts and visual representations are often poorly aligned. To address this challenge, we propose ViTA-SAM, a framework built upon SAM 3 that integrates visual memory and textual features through cross-modal alignment. Specifically, we construct a visual memory module that encodes support features together with their corresponding masks, providing support-aware visual guidance for query prediction. In parallel, we leverage the text encoder of SAM 3 to extract semantic features as complementary priors. To bridge the gap between semantic and visual representations, we introduce a cross-modal alignment module that conditions textual features on memory-enhanced visual representations, yielding more consistent and visually grounded predictions. In addition, we propose a pseudo-label refinement strategy that updates the memory with predicted query masks, enabling query-adaptive representation learning. Extensive experiments on multiple benchmarks show that ViTA-SAM achieves the best overall performance among the compared methods, outperforming the strongest competing method by over 5% in average mIoU.
Tianqi Wei, Zhi Chen, Xin Yu +2
The University of Queensland, Brisbane, Australia · University of Southern Queensland, Toowoomba, Australia · Adelaide University, Adelaide, Australia
Few-shot segmentation (FSS) aims to segment unseen object categories with a few (e.g., one or five) labeled examples, enabling efficient adaptation to novel classes. Conventional models typically rely on appearance-based visual matching between support and query images for segmentation. While straightforward, these methods often struggle to handle significant appearance discrepancies and occlusions in the query image due to insufficient target knowledge. To mitigate this, we introduce a novel framework that mines target knowledge using the strong reasoning capacity of Multimodal Large Language Models (MLLMs) and employs it to enhance FSS. Specifically, building on SAM 2, our method, named MK-FSS, exploits two forms of complementary knowledge derived from a query image by an MLLM for FSS, including spatial knowledge, which provides a spatial prior indicating the potential target location, and semantic knowledge, which describes the target using text. The spatial knowledge is first encoded into a memory representation, and then resulting memory is integrated with the support-guided memory feature from query image through a carefully designed dual-memory debate-fusion (DMDF) module, yielding a more robust target memory feature. In parallel, the semantic knowledge is encoded into the textual feature, which is fused with multi-scale query features via a progressive cross-modal prompt generator (PCPG), producing a target-aware multimodal prompt for segmentation. Working together, the dual-memory feature and the multimodal prompt provide a comprehensive representation of the target, enabling more robust segmentation. In our extensive experiments, MK-FSS shows promising results and largely surpasses existing methods. Code will be released.