Few-Shot Medical Image Segmentation (FSMIS) aims to segment novel object classes in medical images using only minimal annotated examples, addressing the critical challenges of data scarcity and domain shifts prevalent in medical imaging. While Diffusion Models (DM) excel in visual tasks, their potential for FSMIS remains largely unexplored. We propose that the rich visual priors learned by large-scale DMs offer a powerful foundation for a more robust and data-efficient segmentation approach. In this paper, we introduce SD-FSMIS, a novel framework designed to effectively adapt the powerful pre-trained Stable Diffusion (SD) model for the FSMIS task. Our approach repurposes its conditional generative architecture by introducing two key components: a Support-Query Interaction (SQI) and a Visual-to-Textual Condition Translator (VTCT). Specifically, SQI provides a straightforward yet powerful means of adapting SD to the FSMIS paradigm. The VTCT module translates visual cues from the support set into an implicit textual embedding that guides the diffusion model, enabling precise conditioning of the generation process. Extensive experiments demonstrate that SD-FSMIS achieves competitive results compared to state-of-the-art methods in standard settings. Surprisingly, it also demonstrated excellent generalization ability in more challenging cross-domain scenarios. These findings highlight the immense potential of adapting large-scale generative models to advance data-efficient and robust medical image segmentation.
Figures & tables
Figure 1: Comparison between our proposed method and previous methods. (a) Previous fully supervised methods build task-specific networks from scratch and require pixel-level annotations. They generate class prototypes from the limited support set and perform segmentation via feature matching. Lacking strong priors and trained on constrained data, these models are often brittle and struggle with complex visual variations. (b) Instead of building a new network, we adapt a powerful pre-trained foundation model and do not rely on manual annotations. Our framework steers its vast, generalizable visual priors, achieving superior robustness and generalization, especially in challenging cross-domain scenarios.
Figure 2: SD-FSMIS Overview and Training Pipeline. Support and query sets are first encoded using the VAE encoder E . The query latent zqi is enhanced via the Query Enhancement module to obtain zq , while the support latent zsi and its mask latent zsm are concatenated along the channel dimension to form zs . These are then fed into the U-Net, where the query mask latent z^qm is generated under the condition of the text embedding E , produced by the Visual-to-Textual Condition Translator module.
Figure 4: Overview of the SD-FSMIS inference process.
Setting
Method
Ref.
Abd-MRI
Abd-CT
Spleen
Liver
LK
RK
Mean
Spleen
Liver
LK
RK
Mean
1
PANet [ 40 ]
ICCV’19
40.58
50.40
30.99
32.19
38.53
36.04
49.55
20.67
21.19
32.86
SENet [ 31 ]
MIA’20
47.30
29.02
45.78
47.96
42.51
43.66
35.42
24.42
12.51
29.00
SSL-ALPNet [ 28 ]
ECCV’20
72.18
76.10
81.92
85.18
78.84
70.96
78.29
72.36
71.81
73.35
ADNet [ 11 ]
MIA’22
72.29
82.11
73.86
85.80
78.51
63.48
77.24
72.13
79.06
72.97
AAS-DCL [ 44 ]
ECCV’22
76.24
72.33
80.37
86.11
78.76
72.30
75.41
74.69
74.18
73.66
Table 1: Quantitative comparison (in Dice score %) of different methods under setting 1 and 2 on the Abd-MRI and Abd-CT. The best value is shown in bold font, and the second best value is underlined. As ”DiffewS” was originally designed for natural images, we re-implemented and trained it on our medical datasets and protocols to ensure a direct and fair comparison.
Figure 5: Qualitative comparison between our method and DiffewS method on the Abd-MRI dataset and Abd-CT dataset.
Method
Ref.
Abd-CT → MRI
Abd-MRI → CT
Spleen
Liver
LK
RK
Mean
Spleen
Liver
LK
RK
Mean
PANet [ 40 ]
ICCV’19
41.83
40.21
35.36
40.32
39.43
36.58
54.63
21.95
29.19
35.58
SSL-ALPNet [ 28 ]
ECCV’20
54.01
50.62
47.30
53.07
51.25
39.23
60.80
33.01
38.24
42.82
RPT [ 52 ]
MICCAI’23
54.94
52.58
42.58
58.44
52.13
52.04
59.47
40.29
49.60
50.35
DR-Adapter [ 37 ]
CVPR’24
53.91
62.79
71.67
74.12
65.62
55.77
70.83
55.93
44.20
56.68
IFA T=3 [ 26 ]
CVPR’24
59.14
67.04
73.90
75.37
68.86
56.44
71.50
54.60
50.85
58.35
Table 2: Quantitative comparison (in Dice score %) of different cross-domain methods under setting 1. The best value is shown in bold font, and the second best value is underlined. As DiffewS was originally designed for natural images, we re-implemented and trained it on our medical datasets and protocols.
SII
QE
VTCT
Spleen
Liver
LK
RK
Mean
✓
85.54
81.86
71.72
81.31
80.11
✓
✓
83.00
82.83
81.39
85.45
83.17
✓
✓
83.74
81.97
80.23
83.36
82.27
✓
✓
✓
85.01
81.37
83.21
85.04
83.66
Table 3: Ablative results of various components of the proposed method on the Abd-CT dataset under setting 1.
Version
Spleen
Liver
LK
RK
Mean
SD-1.5
85.01
81.37
83.21
85.04
83.66
SD-2.1
83.11
83.02
80.01
85.21
82.84
Table 4: Comparison of different versions of Stable Diffusion on the Abd-CT dataset under setting 1.
Dataset
Type
MSE ↓
PSNR ↑
SSIM ↑
Abd-MRI
Image
0.0005
34.1592
0.9108
Maks
0.0007
32.2890
0.9597
Abd-CT
Image
0.0020
27.4889
0.8172
Maks
0.0009
32.0274
0.9461
Table 5: Quantification of VAE reconstruction quality on Abd-MRI and Abd-CT.
Method
Ref.
Abd-CT → MRI
Abd-MRI → CT
Spleen
Liver
LK
RK
Mean
Spleen
Liver
LK
RK
Mean
PANet [ 40 ]
ICCV’19
33.57
31.93
27.10
32.08
31.17
28.12
41.78
16.72
20.78
26.85
SSL-ALPNet [ 28 ]
ECCV’20
51.12
47.75
44.34
50.23
48.36
34.89
54.37
30.06
33.91
38.31
RPT [ 52 ]
MICCAI’23
52.70
50.29
40.36
56.21
49.89
48.25
53.76
38.64
45.78
46.61
DR-Adapter [ 37 ]
CVPR’24
53.66
60.06
67.01
70.28
62.75
54.43
62.52
54.15
40.81
52.98
IFA T=3 [ 26 ]
CVPR’24
56.14
63.36
71.58
73.75
66.21
55.31
68.11
51.23
46.04
55.17
Table 6: Quantitative comparison (in Dice score %) of different cross-domain methods under setting 2. The best value is shown in bold font, and the second best value is underlined.
Method
Spleen
Liver
LK
RK
Mean ↑
HD95 ↓
ASSD ↓
sample/s
UniverSeg
44.27
55.08
41.50
40.03
45.22
42.84
20.05
0.0080
MultiverSeg
61.42
70.03
64.33
70.72
66.62
54.59
20.76
0.0728
DiffewS
73.11
77.16
77.41
83.47
77.79
17.37
8.74
0.0768
Ours
77.25
78.58
85.03
88.27
82.28
13.15
7.38
0.0914
Table 7: Comparison on the Abd-MRI under setting 2.
Method
Spleen
Liver
LK
RK
Mean ↑
HD95 ↓
ASSD ↓
sample/s
UniverSeg
34.58
51.22
31.07
31.97
37.20
53.86
24.29
0.0077
MultiverSeg
62.39
76.76
54.19
53.93
61.82
57.63
24.36
0.0551
DiffewS
76.84
79.57
69.70
73.62
74.93
18.86
9.49
0.0732
Ours
83.08
82.59
82.22
85.10
83.25
13.20
7.62
0.0856
Table 8: Comparison on the Abd-CT under setting 2.
Figure 6: Qualitative comparison between our method and the universal models method on the Abd-MRI dataset and Abd-CT dataset.
Figure 7: Visualization of failure cases on the Abd-MRI dataset.
Segmentation models such as Segment Anything Model (SAM) and SAM2 achieve strong prompt-driven zero-shot performance. However, their training on natural images limits domain transfer to medical data. Consequently, accurate segmentation typically requires extensive fine-tuning and expert-designed prompts. We propose DiffuSAM, a diffusion-based adaptation of SAM2 for prompt-free medical image segmentation. Our framework synthesizes SAM2-compatible segmentation mask-like embeddings via a lightweight diffusion-prior from off-the-shelf frozen SAM2 image features. The generated embeddings are integrated into SAM2's mask decoder to produce accurate segmentations, thereby eliminating the need for user prompts. The diffusion prior is further conditioned on previously segmented slices, enforcing spatial consistency across volumes. Evaluated on the BTCV and CHAOS datasets for CT and MRI under Source-Free Unsupervised Domain Adaptation (SF-UDA) and Few-Shot settings, DiffuSAM achieves competitive performance with efficient training and inference. Code is available upon request from the corresponding author.
Tal Grossman, Noa Cahan, Lev Ayzenberg +1
School of Electrical Engineering, Tel Aviv University, Israel · School of Biomedical Engineering, Tel Aviv University, Israel
Few-shot medical image segmentation (FS-MIS) aims to segment novel regions of interest (ROIs) from a few annotated support examples. Despite rapid progress, existing FS-MIS solutions span diverse paradigms but are evaluated under inconsistent settings, leaving their relative effectiveness unclear. We introduce FAME, a unified benchmark for evaluating FS-MIS solutions, covering specialists, SAM-based methods, CLIP-based methods, and MLLM-based methods. FAME contains 14,958 test samples across 7 anatomical sites, 9 imaging modalities, and 14 ROI categories, and evaluates models under zero-shot and ten-shot settings with additional assessment of target-absence recognition and generalization under covariate and semantic shifts. Our evaluation reveals several findings. First, effective few-shot segmentation depends on how models exploit support examples: direct visual adaptation generally outperforms prompt-based strategies. Second, increasing support examples improves performance only when models can effectively utilize them. Third, semantic transfer remains substantially more challenging than imaging-domain adaptation, and strong localization ability does not necessarily imply reliable target-absence recognition. We hope FAME provides a comprehensive understanding of current FS-MIS solutions and facilitates the development of more effective and reliable few-shot medical segmentation methods.
Few-shot medical image segmentation (FSMIS) has achieved notable progress, yet most existing methods mainly rely on semantic correspondences from scarce annotations while under-utilizing a key property of medical imagery: anatomical targets exhibit repeatable high-frequency morphology (e.g., boundary geometry and spatial layout) across patients and acquisitions. We propose RAP, a training-free framework that retrieves, adapts, and prompts Segment Anything Model 2 (SAM2) for FSMIS. First, RAP retrieves morphologically compatible supports from an archive using DINOv3 features to reduce brittleness in single-support choice. Second, it adapts the retrieved support mask to the query by fitting boundary-aware structural cues, yielding an anatomy-consistent pre-mask under domain shifts. Third, RAP converts the pre-mask into prompts by sampling positive points via Voronoi partitioning and negative points via sector-based sampling, and feeds them into SAM2 for final refinement without any fine-tuning. Extensive experiments on multiple medical segmentation benchmarks show that RAP consistently surpasses prior FSMIS baselines and achieves state-of-the-art performance. Overall, RAP demonstrates that explicit structural fitting combined with retrieval-augmented prompting offers a simple and effective route to robust training-free few-shot medical segmentation.
Zhihao Mao, Bangpu Chen
School of Computer Science, China University of Geosciences, Wuhan, China