SD-FSMIS: Adapting Stable Diffusion for Few-Shot Medical Image Segmentation

Meihua Li; Yang Zhang; Weizhao He; Hu Qu; Yisong Li

cs.CVApr 3, 2026

SD-FSMIS: Adapting Stable Diffusion for Few-Shot Medical Image Segmentation

Authors: Meihua Li, Yang Zhang, Weizhao He, Hu Qu, Yisong Li

Organizations: Computer Vision Institute, College of Computer Science and Software Engineering, Shenzhen University

Abstract

Few-Shot Medical Image Segmentation (FSMIS) aims to segment novel object classes in medical images using only minimal annotated examples, addressing the critical challenges of data scarcity and domain shifts prevalent in medical imaging. While Diffusion Models (DM) excel in visual tasks, their potential for FSMIS remains largely unexplored. We propose that the rich visual priors learned by large-scale DMs offer a powerful foundation for a more robust and data-efficient segmentation approach. In this paper, we introduce SD-FSMIS, a novel framework designed to effectively adapt the powerful pre-trained Stable Diffusion (SD) model for the FSMIS task. Our approach repurposes its conditional generative architecture by introducing two key components: a Support-Query Interaction (SQI) and a Visual-to-Textual Condition Translator (VTCT). Specifically, SQI provides a straightforward yet powerful means of adapting SD to the FSMIS paradigm. The VTCT module translates visual cues from the support set into an implicit textual embedding that guides the diffusion model, enabling precise conditioning of the generation process. Extensive experiments demonstrate that SD-FSMIS achieves competitive results compared to state-of-the-art methods in standard settings. Surprisingly, it also demonstrated excellent generalization ability in more challenging cross-domain scenarios. These findings highlight the immense potential of adapting large-scale generative models to advance data-efficient and robust medical image segmentation.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison between our proposed method and previous methods. (a) Previous fully supervised methods build task-specific networks from scratch and require pixel-level annotations. They generate class prototypes from the limited support set and perform segmentation via feature matching. Lacking strong priors and trained on constrained data, these models are often brittle and struggle with complex visual variations. (b) Instead of building a new network, we adapt a powerful pre-trained foundation model and do not rely on manual annotations. Our framework steers its vast, generalizable visual priors, achieving superior robustness and generalization, especially in challenging cross-domain scenarios.

Figure 2: SD-FSMIS Overview and Training Pipeline. Support and query sets are first encoded using the VAE encoder $\mathcal{E}$ . The query latent $z^{qi}$ is enhanced via the Query Enhancement module to obtain $z^{q}$ , while the support latent $z^{si}$ and its mask latent $z^{sm}$ are concatenated along the channel dimension to form $z^{s}$ . These are then fed into the U-Net, where the query mask latent $\hat{z}^{qm}$ is generated under the condition of the text embedding $E$ , produced by the Visual-to-Textual Condition Translator module.

Figure 3: Modified BasicTransformerBlocks architecture.

Figure 4: Overview of the SD-FSMIS inference process.

Setting	Method	Ref.	Abd-MRI					Abd-CT
Setting	Method	Ref.	Spleen	Liver	LK	RK	Mean	Spleen	Liver	LK	RK	Mean
1	PANet [ 40 ]	ICCV’19	40.58	50.40	30.99	32.19	38.53	36.04	49.55	20.67	21.19	32.86
	SENet [ 31 ]	MIA’20	47.30	29.02	45.78	47.96	42.51	43.66	35.42	24.42	12.51	29.00
	SSL-ALPNet [ 28 ]	ECCV’20	72.18	76.10	81.92	85.18	78.84	70.96	78.29	72.36	71.81	73.35
	ADNet [ 11 ]	MIA’22	72.29	82.11	73.86	85.80	78.51	63.48	77.24	72.13	79.06	72.97
	AAS-DCL [ 44 ]	ECCV’22	76.24	72.33	80.37	86.11	78.76	72.30	75.41	74.69	74.18	73.66

Table 1: Quantitative comparison (in Dice score %) of different methods under setting 1 and 2 on the Abd-MRI and Abd-CT. The best value is shown in bold font, and the second best value is underlined. As ”DiffewS” was originally designed for natural images, we re-implemented and trained it on our medical datasets and protocols to ensure a direct and fair comparison.

Figure 5: Qualitative comparison between our method and DiffewS method on the Abd-MRI dataset and Abd-CT dataset.

Method	Ref.	Abd-CT $\rightarrow$ MRI					Abd-MRI $\rightarrow$ CT
Method	Ref.	Spleen	Liver	LK	RK	Mean	Spleen	Liver	LK	RK	Mean
PANet [ 40 ]	ICCV’19	41.83	40.21	35.36	40.32	39.43	36.58	54.63	21.95	29.19	35.58
SSL-ALPNet [ 28 ]	ECCV’20	54.01	50.62	47.30	53.07	51.25	39.23	60.80	33.01	38.24	42.82
RPT [ 52 ]	MICCAI’23	54.94	52.58	42.58	58.44	52.13	52.04	59.47	40.29	49.60	50.35
DR-Adapter [ 37 ]	CVPR’24	53.91	62.79	71.67	74.12	65.62	55.77	70.83	55.93	44.20	56.68
IFA T=3 [ 26 ]	CVPR’24	59.14	67.04	73.90	75.37	68.86	56.44	71.50	54.60	50.85	58.35

Table 2: Quantitative comparison (in Dice score %) of different cross-domain methods under setting 1. The best value is shown in bold font, and the second best value is underlined. As DiffewS was originally designed for natural images, we re-implemented and trained it on our medical datasets and protocols.

SII	QE	VTCT	Spleen	Liver	LK	RK	Mean
✓			85.54	81.86	71.72	81.31	80.11
✓		✓	83.00	82.83	81.39	85.45	83.17
✓	✓		83.74	81.97	80.23	83.36	82.27
✓	✓	✓	85.01	81.37	83.21	85.04	83.66

Table 3: Ablative results of various components of the proposed method on the Abd-CT dataset under setting 1.

Version	Spleen	Liver	LK	RK	Mean
SD-1.5	85.01	81.37	83.21	85.04	83.66
SD-2.1	83.11	83.02	80.01	85.21	82.84

Table 4: Comparison of different versions of Stable Diffusion on the Abd-CT dataset under setting 1.

Dataset	Type	MSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
Abd-MRI	Image	0.0005	34.1592	0.9108
Abd-MRI	Maks	0.0007	32.2890	0.9597
Abd-CT	Image	0.0020	27.4889	0.8172
Abd-CT	Maks	0.0009	32.0274	0.9461

Table 5: Quantification of VAE reconstruction quality on Abd-MRI and Abd-CT.

Method	Ref.	Abd-CT $\rightarrow$ MRI					Abd-MRI $\rightarrow$ CT
Method	Ref.	Spleen	Liver	LK	RK	Mean	Spleen	Liver	LK	RK	Mean
PANet [ 40 ]	ICCV’19	33.57	31.93	27.10	32.08	31.17	28.12	41.78	16.72	20.78	26.85
SSL-ALPNet [ 28 ]	ECCV’20	51.12	47.75	44.34	50.23	48.36	34.89	54.37	30.06	33.91	38.31
RPT [ 52 ]	MICCAI’23	52.70	50.29	40.36	56.21	49.89	48.25	53.76	38.64	45.78	46.61
DR-Adapter [ 37 ]	CVPR’24	53.66	60.06	67.01	70.28	62.75	54.43	62.52	54.15	40.81	52.98
IFA T=3 [ 26 ]	CVPR’24	56.14	63.36	71.58	73.75	66.21	55.31	68.11	51.23	46.04	55.17

Table 6: Quantitative comparison (in Dice score %) of different cross-domain methods under setting 2. The best value is shown in bold font, and the second best value is underlined.

Method	Spleen	Liver	LK	RK	Mean $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$	sample/s
UniverSeg	44.27	55.08	41.50	40.03	45.22	42.84	20.05	0.0080
MultiverSeg	61.42	70.03	64.33	70.72	66.62	54.59	20.76	0.0728
DiffewS	73.11	77.16	77.41	83.47	77.79	17.37	8.74	0.0768
Ours	77.25	78.58	85.03	88.27	82.28	13.15	7.38	0.0914

Table 7: Comparison on the Abd-MRI under setting 2.

Method	Spleen	Liver	LK	RK	Mean $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$	sample/s
UniverSeg	34.58	51.22	31.07	31.97	37.20	53.86	24.29	0.0077
MultiverSeg	62.39	76.76	54.19	53.93	61.82	57.63	24.36	0.0551
DiffewS	76.84	79.57	69.70	73.62	74.93	18.86	9.49	0.0732
Ours	83.08	82.59	82.22	85.10	83.25	13.20	7.62	0.0856

Table 8: Comparison on the Abd-CT under setting 2.

Figure 6: Qualitative comparison between our method and the universal models method on the Abd-MRI dataset and Abd-CT dataset.

Figure 7: Visualization of failure cases on the Abd-MRI dataset.

Figure 8: Visualization of the training process.

Explore similar work

DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation
Apr 27, 2026Tal Grossman, Noa Cahan, Lev Ayzenberg +1Source-Free Domain Adaptation Image Segmentation
School of Electrical Engineering, Tel Aviv University, Israel · School of Biomedical Engineering, Tel Aviv University, Israel
Benchmarking Foundation and Large Language Models for Few-Shot Medical Image Segmentation
Jul 30, 2026Jinghong Liu, Yuchuan Deng, Fanping Liu +2Medical Imaging Foundation Models Few-Shot Medical Image Segmentation
Renmin University of China
RAP: Retrieve, Adapt, and Prompt-Fit for Training-Free Few-Shot Medical Image Segmentation
Mar 29, 2026Zhihao Mao, Bangpu ChenMedical Image Segmentation Few-Shot Medical Image Segmentation
School of Computer Science, China University of Geosciences, Wuhan, China

Apr 27, 2026J/K move · Enter open · S save

DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation

Tal Grossman, Noa Cahan, Lev Ayzenberg, Hayit Greenspan

School of Electrical Engineering, Tel Aviv University, Israel · School of Biomedical Engineering, Tel Aviv University, Israel

Source-Free Domain Adaptation Image Segmentation Medical Imaging Foundation Models Medical Image Segmentation Few-Shot Medical Image Segmentation Diffusion Models

Open paper AskPDF arXiv

Segmentation models such as Segment Anything Model (SAM) and SAM2 achieve strong prompt-driven zero-shot performance. However, their training on natural images limits domain transfer to medical data. Consequently, accurate segmentation typically requires extensive fine-tuning and expert-designed prompts. We propose DiffuSAM, a diffusion-based adaptation of SAM2 for prompt-free medical image segmentation. Our framework synthesizes SAM2-compatible segmentation mask-like embeddings via a lightweight diffusion-prior from off-the-shelf frozen SAM2 image features. The generated embeddings are integrated into SAM2's mask decoder to produce accurate segmentations, thereby eliminating the need for user prompts. The diffusion prior is further conditioned on previously segmented slices, enforcing spatial consistency across volumes. Evaluated on the BTCV and CHAOS datasets for CT and MRI under Source-Free Unsupervised Domain Adaptation (SF-UDA) and Few-Shot settings, DiffuSAM achieves competitive performance with efficient training and inference. Code is available upon request from the corresponding author.

Abstract

Explore similar work

DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation

Benchmarking Foundation and Large Language Models for Few-Shot Medical Image Segmentation

RAP: Retrieve, Adapt, and Prompt-Fit for Training-Free Few-Shot Medical Image Segmentation