MESAM: Enhancing SAM2 for Few-Shot Segmentation through Semantic-Visual Memory Generation and Refinement
Authors: Tianyu Zou, Shengwu Xiong, Rui Wu, Ruilin Yao, Yi Rong
Organizations: Sanya Science and Education Innovation Park, Wuhan University of Technology Sanya, Hainan, China · School of Artificial Intelligence, Wuhan University of Technology Wuhan, Hubei, China · Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology Wuhan, Hubei, China · Institute of Automation, Chinese Academy of Sciences Beijing, China
This paper investigates how to leverage the powerful capabilities of the Segment Anything Model 2 (SAM2) to tackle few-shot segmentation (FSS) tasks. One major challenge lies in the substantial appearance differences between the support and query images, which may diminish the effectiveness of the memory-matching process in SAM2. Through systematic analysis, we find that the memory generated based on the categorical semantic information is robust to this appearance divergence and is more compatible with SAM2. Based on these observations, we propose a new Memory-Enhanced SAM2 (MESAM) method, which first introduces a Semantic-Visual Memory Generation (SVMG) module to separately construct a query semantic memory, a query foreground visual memory, and a support adaptive visual memory. Then, in our designed Interactive Semantic Memory Refinement (ISMR) module, we further enrich the obtained semantic memory with fine-grained visual details, and suppress its misatcivated background responses. Extensive experiments on the PASCAL-5i, COCO-20i, and LVIS-92i datasets demonstrate that MESAM consistently outperforms existing FSS methods, achieving new state-of-the-art segmentation performance. Our source code is available at: https://github.com/tianyu-zou/MESAM.
Figures & tables
Figure 1. Illustration of the large support-query appearance divergence challenge. With only partial visual cues provided by the support image, existing methods segment only visually similar parts in the query, instead of the entire object. In contrast, with the refined semantic memory, our MESAM method yields better segmentation results. The orange and blue circles indicate foreground (FG) misses and background (BG) misactivations, respectively.
Figure 2. The priors and the segmentation results of two visual memories (i.e., FG, DISC) and our semantic memory.
Memory
FG
DISC
Semantic
PASCAL- 5i
81.1
78.1
82.8
COCO- 20i
55.9
56.0
58.9
Table 1. Performance comparison between the visual memories constructed in FSSAM and our semantic memory.
PASCAL- 5i
COCO- 20i
Memory
FP
FN
FP
FN
FG
0.0865
0.0436
0.1271
0.0381
DISC
0.0186
0.1056
0.0324
0.0820
Semantic
0.0139
0.1207
0.0155
0.0915
Table 2. The false positive (FP) and false negative (FN) rates of the prior masks used for generating FG and DISC visual memories, and our semantic memory.
Figure 3. Performance comparison between different memories obtained under varying dilation and erosion ratios.
Figure 4. Illustration of our MESAM method. (a) The SVMG module generates three initial memories Memqs , Memqv , and Memsv . (b) In the ISMR module, Memqs is enhanced by interacting with Memqv through the cross-attention operation, while Memsv is used to suppress the BG noise in the attention scores. Subsequently, the refined semantic memory Memqs′ is processed by SAM2’s memory attention module and mask decoder to produce the final segmentation mask.
1-shot
5-shot
Methods
Backbone
Venue
50
51
52
53
Mean
FB-IoU
50
51
52
53
Mean
FB-IoU
ABCBNet ( Zhu et al., 2024 )
ResNet50
CVPR’24
72.9
76.0
69.5
64.0
70.6
-
74.4
78.0
73.9
68.3
73.6
-
PAHNet ( Zou et al., 2025 )
ResNet50
ICCV‘25
72.1
76.4
71.0
66.6
71.5
82.4
75.8
78.6
76.9
73.2
76.1
85.7
OCNet ( Wen et al., 2025 )
ResNet50
ICCV‘25
73.5
75.9
71.1
64.9
71.4
82.2
75.9
77.1
74.1
70.9
74.5
84.7
Matcher ( Liu et al., 2024b )
SAM-H
ICLR’24
67.7
70.7
66.9
67.0
68.1
-
71.4
77.5
74.1
72.8
74.0
-
VRP-SAM ( Sun et al., 2024 )
SAM-H
CVPR’24
73.9
78.3
70.6
65.1
71.9
-
-
-
-
-
-
-
Table 3. Performance comparisons on PASCAL- 5i . “ 5i ” shows the mIoU scores of 5 novel classes in fold i , “Mean” denotes the averaged mIoU score across all four folds. The best and second-best results are highlighted in bold and underlined , respectively.
1-shot
5-shot
Methods
Backbone
Venue
200
201
202
203
Mean
FB-IoU
200
201
202
203
Mean
FB-IoU
ABCBNett ( Zhu et al., 2024 )
ResNet50
CVPR’24
44.2
54.0
52.1
49.8
50.0
-
50.5
59.1
57.0
53.6
55.1
-
PAHNet ( Zou et al., 2025 )
ResNet50
ICCV‘25
45.2
58.2
54.4
51.2
52.3
75.2
53.0
64.1
60.8
58.7
59.2
78.1
OCNet ( Wen et al., 2025 )
ResNet50
ICCV‘25
45.9
56.9
52.9
50.4
51.5
73.7
52.7
63.1
57.4
54.8
57.0
76.8
Matcher ( Liu et al., 2024b )
SAM-H
ICLR’24
52.7
53.5
52.6
52.1
52.7
-
60.1
62.7
60.9
59.2
60.7
-
VRP-SAM ( Sun et al., 2024 )
SAM-H
CVPR’24
56.0
61.3
57.9
58.8
58.5
-
61.4
69.4
65.9
64.9
65.4
-
Table 4. Performance comparisons on COCO- 20i .“ 20i ” shows the mIoU scores of 20 novel classes in fold i , “Mean” denotes the averaged mIoU score across all four folds. The best and second-best results are highlighted in bold and underlined , respectively.
Methods
920
921
922
923
924
925
926
927
928
929
Mean
FB-IoU
Matcher ( Liu et al., 2024b )
31.4
30.9
33.7
38.1
30.5
32.5
35.9
34.2
33.0
29.7
33.0
66.2
SINE ( Liu et al., 2024a )
28.3
31.0
31.9
34.6
30.0
31.9
32.2
33.7
30.6
27.8
31.2
63.5
FSSAM ( Xu et al., 2025b )
34.7
37.8
37.2
41.1
33.9
38.1
40.6
38.9
36.9
33.8
37.3
68.4
MESAM (Ours)
38.0
44.7
43.5
44.8
38.2
41.4
44.0
41.2
44.0
39.4
41.9
70.9
Table 5. Performance comparisons on LVIS- 92i . “ 92i ” shows the mIoU scores of 92 novel classes in fold i , “Mean” denotes the averaged mIoU score across all ten folds. The 1-shot results are reported, where the best results are highlighted in bold .
Figure 5. Qualitative comparisons between FSSAM and MESAM on PASCAL- 5i , COCO- 20i and LVIS- 92i under the 1-shot setting.
Settings
200
201
202
203
Mean
A: Only Memqs
51.2
67.1
58.4
58.7
58.9
B: Memqs+Memqv
62.1
68.9
65.1
66.9
65.8
C: Memqs+Memqv+Memsv(Ms)
61.9
69.9
64.7
68.1
66.2
D: Memqs+Memqv+Memsv
63.9
70.4
66.2
68.6
67.3
Table 6. Ablation study on the effect of different memories.
200
201
202
203
Mean
Cosine over K
61.7
70.1
66.7
67.1
66.4
FG Enhancement
62.0
69.8
67.2
67.7
66.7
Memory Swap
61.1
70.5
67.8
67.9
66.8
ISMR in MESAM
63.9
70.4
66.2
68.6
67.3
Table 7. Ablation study on the effect of different ISMR.
Figure 6. The cross-attention weights and the segmentation results obtained without and with the support calibration.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Input: Query ground-truth mask Mq , target ratio r ,
kernel size k , padding parameter pad
Output: Dilated mask M
Stage 1: Initialization
M←Mq ;
kernal← ones( k,k );
initial_area← sum of non-zero pixels in M ;
Appendix
Algorithm 1 Mask Dilation
Figure 7. Qualitative comparisons between dilation, vanilla-erosion, and skeleton-erosion operations with varying target ratios.
Figure 8. The prior masks and the segmentation results of visual memories (FG, DISC) and semantic memory on PASCAL- 5i .
Figure 9. The prior masks and the segmentation results of visual memories (FG, DISC) and semantic memory on COCO- 20i .
Input: Query ground-truth mask Mq , target ratio r ,
kernel size k , padding parameter pad
Output: Eroded mask M
Stage 1: Initialization
M←Mq ;
kernal← ones( k,k );
initial_area← sum of non-zero pixels in M ;
Appendix
Algorithm 2 Mask Erosion
1-shot
5-shot
Seed
50
51
52
53
Mean
50
51
52
53
Mean
default
83.70
88.93
83.74
83.70
85.02
84.21
90.08
86.21
87.50
87.00
0
83.75
89.29
83.88
83.38
85.08
83.65
89.57
86.49
87.69
86.85
1
83.44
88.75
83.47
83.46
84.78
83.95
89.91
86.38
87.45
86.92
2
83.43
89.18
83.38
83.30
84.82
83.92
89.77
86.03
87.87
86.90
3
83.38
89.23
83.82
83.33
84.94
84.11
89.66
86.62
87.15
86.89
Appendix
Table 8. Performance of MESAM on PASCAL- 5i with five random seeds {321 (default), 0, 1, 2, 3}. 1,000 episodes are randomly sampled for evaluation. “Average” and “Std” denote the mean mIoU scores and the standard deviations across different random seeds.
1-shot
5-shot
Seed
200
201
202
203
Mean
200
201
202
203
Mean
default
63.92
70.41
66.17
68.63
67.28
67.22
74.13
66.01
73.32
70.17
0
62.97
70.59
65.13
67.48
66.54
65.59
72.84
68.16
72.64
69.81
1
63.55
70.33
65.46
67.21
66.64
68.52
74.45
63.31
72.79
69.77
2
63.38
70.12
62.33
67.59
65.86
68.16
73.45
66.96
72.70
70.32
3
62.65
70.60
64.94
68.07
66.57
65.61
73.65
67.25
71.85
69.59
Appendix
Table 9. Performance of MESAM on COCO- 20i with five random seeds {321 (default), 0, 1, 2, 3}. 1,000 episodes are randomly sampled for evaluation. “Average” and “Std” denote the mean mIoU scores and the standard deviations across different random seeds.
1-shot
5-shot
#Episodes
200
201
202
203
Mean
Time (s)
200
201
202
203
Mean
Time (s)
1000
63.92
70.41
66.17
68.63
67.28
143.1
67.22
74.13
66.01
73.32
70.17
218.7
4000
65.38
69.86
68.68
66.57
67.62
568.2
66.75
73.81
72.52
72.32
71.35
881.4
10000
63.66
70.35
67.27
65.43
66.68
1406.5
66.38
73.73
70.73
71.84
70.67
2183.2
20000
63.80
70.55
67.41
65.54
66.83
2799.9
66.57
73.87
70.69
71.87
70.75
4425.7
Average
64.19
70.29
67.38
66.54
67.10
1229.4
66.73
73.89
69.99
72.34
70.74
1927.3
Appendix
Table 10. Performance of MESAM on COCO- 20i evaluated with varying numbers of testing episodes {1,000, 4,000, 10,000, 20,000}. “Time” denotes the total inference time for the evaluation process. “Average” and “Std” denote the mean mIoU scores and the standard deviations across different numbers of testing episodes.
Figure 10. Ablation study of the scaling factor α on COCO- 200 .
Backbones
Size (M)
200
201
202
203
Mean
SAM2-T
38.9
62.8
70.6
66.4
69.4
67.3
SAM2-S
46.0
63.9
70.4
66.2
68.6
67.3
SAM2-B
80.8
63.0
69.8
65.6
67.7
66.5
SAM2-L
224.4
62.4
71.1
68.7
68.4
67.7
Appendix
Table 11. Performance comparisons between different MESAM models built upon four SAM2 versions.
Figure 11. Additional qualitative comparisons between FSSAM and MESAM on LVIS- 92i under the 1-shot setting.
Methods
50
51
52
53
Mean
HSNet ( Min et al., 2021 )
47.0
65.2
67.1
77.1
64.1
VAT ( Hong et al., 2022 )
68.3
64.9
67.5
79.8
65.1
PMNet ( Chen et al., 2024 )
71.0
72.3
66.6
63.8
68.4
GF-SAM ( Zhang et al., 2024a )
71.1
75.7
69.2
73.3
72.1
SANSA ( Cuttano et al., 2025 )
67.1
70.9
73.5
78.6
72.5
FSSAM ( Xu et al., 2025b )
79.5
85.7
85.4
76.5
81.8
Appendix
Table 12. Performance comparisons under COCO- 20i→ PASCAL- 5i cross-dataset setting. The 1-shot results are reported, where the best results are highlighted in bold .
Figure 12. Additional qualitative comparisons between FSSAM and MESAM on PASCAL- 5i and COCO- 20i under the 1-shot setting.