Focus on Background: Exploring SAM's Potential in Few-shot Medical Image Segmentation with Background-centric Prompting

Yuntian Bo; Yazhou Zhu; Piotr Koniusz; Haofeng Zhang

cs.CVMar 22, 2026

Focus on Background: Exploring SAM's Potential in Few-shot Medical Image Segmentation with Background-centric Prompting

Authors: Yuntian Bo, Yazhou Zhu, Piotr Koniusz, Haofeng Zhang

Organizations: Nanjing University of Science and Technology · University of New South Wales · Data61 CSIRO

Abstract

Conventional few-shot medical image segmentation (FSMIS) approaches face performance bottlenecks that hinder broader clinical applicability. Although the Segment Anything Model (SAM) exhibits strong category-agnostic segmentation capabilities, its direct application to medical images often leads to over-segmentation due to ambiguous anatomical boundaries. In this paper, we reformulate SAM-based FSMIS as a prompt localization task and propose FoB (Focus on Background), a background-centric prompt generator that provides accurate background prompts to constrain SAM's over-segmentation. Specifically, FoB bridges the gap between segmentation and prompt localization by category-agnostic generation of support background prompts and localizing them directly in the query image. To address the challenge of prompt localization for novel categories, FoB models rich contextual information to capture foreground-background spatial dependencies. Moreover, inspired by the inherent structural patterns of background prompts in medical images, FoB models this structure as a constraint to progressively refine background prompt predictions. Experiments on three diverse medical image datasets demonstrate that FoB outperforms other baselines by large margins, achieving state-of-the-art performance on FSMIS, and exhibiting strong cross-domain generalization. Our code is available at https://github.com/primebo1/FoB_SAM.

Figures & tables

Figure 1: (a) Methodological motivation. We observe that without accurate background prompts, SAM tends to over-segment ambiguous boundaries, highlighting the necessity of precise background guidance. (b) Technical motivation. Prior methods extract prompts from a foreground segment map, yielding only reliable foreground prompts. We reformulate SAM-based FSMIS as a prompt localization task, focusing specifically on precise background prompt identification to limit SAM’s over-segmentation.

Figure 2: The overall architecture of the proposed FoB model for FSMIS, which consists of three collaboratively designed modules. BPPC prepares prompt prototypes for subsequent matching and localization. BCM models the dependency between background prompts and the foreground to produce coarse localization. SPR (see Figure 3 ) imposes structural constraints in the feature space to refine background prompt predictions from BCM. FoB is trained independently of SAM and serves as a plug-and-play prompt generator during inference.

Figure 3: Schema of Structure-guided Prompt Refinement.

	Methods	Abd-MRI					Abd-CT					Skin-DS
	Methods	Liv	RK	LK	Spl	Avg.	Liv	RK	LK	Spl	Avg.	Mel	Nev	SK	Avg.
Setting I	ALPNet [ 6 ]	76.1	85.18	81.92	72.18	78.84	78.29	71.81	72.36	70.96	73.35	66.32	61.65	59.57	62.51
	RPT [ 60 ]	82.86	89.82	80.72	76.37	82.44	82.57	72.58	77.05	79.13	77.83	77.81	75.42	70.28	74.50
	GMRD [ 9 ]	81.42	90.12	83.96	76.09	82.90	79.6	74.46	81.7	78.31	78.52	79.23	72.78	71.32	74.44
	PGRNet [ 17 ]	83.27	87.44	81.44	81.72	83.47	82.48	79.88	74.23	72.09	77.17	71.39	70.21	65.87	69.16
	ProtoSAM [ 4 ]	83.14	82.36	82.75	77.98	81.56	84.79	75.67	71.31	70.24	75.50	73.61	76.26	68.37	72.75

Table 1: Comparison with SOTA methods (in Dice score %) on Abd-MRI, Abd-CT, and Skin-DS under Settings I and II. The best values are shown in bold font. Indicator “–” means that the original paper did not report results or release the code for a fair comparison.

	Methods	Liv	RK	LK	Spl	Avg.
CT $\to$ MRI	RobustEMD [ 59 ]	60.16	70.26	66.34	53.71	62.61
	FAMNet [ 5 ]	73.01	74.68	57.28	58.21	65.79
	FoB + SAM	75.05	79.57	70.38	68.21	73.30
MRI $\to$ CT	RobustEMD [ 59 ]	69.82	50.34	63.79	59.88	60.95
	FAMNet [ 5 ]	73.57	61.89	57.79	65.78	64.75
	FoB + SAM	81.36	58.81	57.18	70.71	67.02

Table 2: Comparison with SOTA methods (in Dice score %) under cross-domain setting using abdominal datasets. The best values are shown in bold font.

Figure 4: Qualitative results of our method across different datasets. See Appendix D for more visualizations.

BPPC	BCM	SPR	Liv	RK	LK	Spl	Avg
✓			81.58	83.21	79.88	79.38	81.01
✓		✓	84.2	86.27	82.13	81.35	83.49
✓	✓		84.87	87.61	84.01	83.99	85.12
✓	✓	✓	86.51	86.51	87.29	84.54	86.21

Table 3: Ablation studies of model components on Dice score (%). The best results are indicated in bold.

Figure 5: Analysis of the values of

N_{f}

and

N_{p}

$\mathcal{L}_{heat}$	$\mathcal{L}_{coor}$	$\mathcal{L}_{fore}$	$\mathcal{L}_{rac}$	Liv	RK	LK	Spl	Avg
	✓	✓	✓	59.24	76.92	77.63	68.01	70.45
✓	✓		✓	34.01	33.92	35.17	37.95	35.26
✓	✓	✓		83.67	83.71	83.02	85.33	83.93
✓	✓	✓	✓	86.51	86.51	87.29	84.54	86.21

Table 4: Ablation studies of the loss functions on Dice score (%). The best results are indicated in bold.

Figure 6: Visualization of the effectiveness of SPR.

Figure 7: Comparison of the previous method, ProtoSAM [ 3 ] (left), and our method (right). ProtoSAM connects an existing FSMIS model with SAM by extracting prompts from the coarse segmentation output of the FSMIS model, which often fails to provide accurate background prompts due to inherent flaws. In contrast, our method predicts both precise background and foreground prompts, improving SAM’s use in medical image segmentation.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Previous prompt extraction methods based on existing FSMIS model outputs. (a) Original image. (b) Prediction confidence-based method, which selects high-confidence foreground and background points as prompts. (c) Coarse binary mask-based method, which randomly samples prompts within the predicted foreground and background regions of the mask. The yellow line indicates the ground truth. The arrows in (b) and (c) indicate incorrect prompt locations.

Model	mDice	Param.	Latency
Model	mDice	(M)	(ms/img)
FoB-Transformer	86.21	49.31	57.49
FoB-Mamba	85.31	49.90	33.95

Appendix

Table 5: Comparison of Transformer- and Mamba-based implementations of the BCM module. “mDice” denotes the mean Dice score (%), and “Param.” indicates the number of parameters.

$r$	Liver	RK	LK	Spleen	Mean
19	82.12	83.41	82.28	84.01	82.96
15	86.51	86.51	87.29	84.54	86.21
11	86.37	87.33	86.12	84.93	86.19
7	79.21	82.66	84.76	80.47	81.78

Appendix

Table 6: Ablation study (Dice score % reported) on the impact of the dilation kernel size

r

. Smaller

r

moves background prompts closer to the foreground.

Figure 9: Visualization of support background prompts generated with different $r$ in BPPC. Best viewed under large zoom.

Setting	Abd-CT					Abd-MRI					Skin-DS
Setting	Liv	RK	LK	Spl	Avg.	Liv	RK	LK	Spl	Avg.	Mel	Nev	SK	Avg.
Setting I	69.51	65.79	73.01	57.93	66.56	71.54	78.57	80.46	68.81	74.85	71.59	75.34	68.75	71.89
Setting II	68.61	64.30	67.67	58.68	64.82	67.99	75.57	74.47	62.86	70.22	72.91	71.25	68.49	70.88

Appendix

Table 7: Segmentation results of MedSAM when prompted by FoB. Bounding box prompts are obtained by computing the minimum enclosing rectangle of the background prompts generated by FoB. These results further validate that medical SAMs are in fact unsuitable for FSMIS tasks, due to inherent architectural limitations and potential violations of the few-shot learning protocol.

$\sigma$	Liver	RK	LK	Spleen	Mean
8	80.02	84.94	85.78	80.21	82.74
4	86.51	86.51	87.29	84.54	86.21
2	79.76	75.56	75.97	79.91	77.80
1	76.33	61.93	64.46	62.57	66.32

Appendix

Table 8: Ablation study (Dice score % used) on the impact of the standard deviation

\sigma

for generating heatmaps.

$k$	Liver	RK	LK	Spleen	Mean
16	84.96	76.80	88.33	83.89	83.50
8	86.51	86.51	87.29	84.54	86.21
4	87.50	84.64	85.58	86.19	85.98
2	81.05	85.40	88.63	83.62	84.68

Appendix

Table 9: Ablation study (in Dice score %) on the impact of the deformable receptive field size

k

in SPR.

$\kappa$	Liver	RK	LK	Spleen	Mean
7	82.62	83.99	83.94	77.67	82.06
5	84.43	86.68	86.63	83.75	85.37
3	86.51	86.51	87.29	84.54	86.21
1	84.77	84.17	87.42	84.68	85.26

Appendix

Table 10: Ablation study (Dice score % used) on the impact of the number of deformable iterations

\kappa

in SPR.

$\mathcal{T}$	Liver	RK	LK	Spleen	Mean
0.95	88.75	86.03	83.99	85.27	86.01
0.90	86.51	86.51	87.29	84.54	86.21
0.85	87.94	83.77	85.62	82.47	84.95
0.80	87.38	84.55	84.01	81.79	84.43

Appendix

Table 11: Ablation study (Dice score % used) on the impact of the foreground prompt sampling threshold

\mathcal{T}

$\tau$	Liver	RK	LK	Spleen	Mean
0.7	81.74	86.31	85.72	80.09	83.47
0.5	84.96	86.79	84.94	82.01	84.68
0.3	87.24	86.37	86.97	83.17	85.94
0.1	86.51	86.51	87.29	84.54	86.21

Appendix

Table 12: Ablation study (Dice score % used) on the impact of the temperature

\tau

in the RAC loss.

Figure 10: Evolution of the foreground probability map over training epochs. As training proceeds, the foreground prediction becomes highly accurate even under ambiguous boundaries.

Figure 11: Visualization of matching robustness of the BCM module under significant background variations.

Graph	Liver	RK	LK	Spleen	Mean
No $\mathbf{A}$	81.32	87.81	82.43	78.55	82.53
$\mathbf{A}^{ring}$ only	84.99	88.26	84.09	78.95	84.07
$\mathbf{A}^{ada}$ only	83.87	87.00	84.88	77.39	83.29
$\mathbf{A}^{ada}+\mathbf{A}^{ring}$	85.61	88.18	84.76	79.31	84.46

Appendix

Table 13: Ablation study (Dice score % used) on different graph construction strategies.

Figure 12: Illustrative examples of superpixel-based pseudo labels on Skin-DS.

Figure 13: Performance curves of FoB & its variants on 3 datasets.

Figure 14: Qualitative segmentation results of our method on Skin-DS.

Figure 15: Qualitative effect of SPR on Abd-CT. Our proposed FoB with structure-aware refinement (w/ SPR) significantly outperforms the counterpart that solely uses BCM-predicted prompt sets (w/o SPR).

Figure 16: Visualization of prompts generated by the proposed FoB. Rows 1–3 correspond to Abd-CT, rows 4–6 to Abd-MRI, and rows 7–9 to Skin-DS. FoB produces highly reliable background prompts that play a crucial role in constraining SAM’s over-segmentation.

Figure 17: Qualitative comparison of segmentation results on Abd-MRI (upper) and Abd-CT (lower).

Explore similar work

RAP: Retrieve, Adapt, and Prompt-Fit for Training-Free Few-Shot Medical Image Segmentation
Mar 29, 2026Zhihao Mao, Bangpu ChenMedical Image Segmentation Few-Shot Medical Image Segmentation
School of Computer Science, China University of Geosciences, Wuhan, China
Benchmarking Foundation and Large Language Models for Few-Shot Medical Image Segmentation
Jul 30, 2026Jinghong Liu, Yuchuan Deng, Fanping Liu +2Medical Imaging Foundation Models Few-Shot Medical Image Segmentation
Renmin University of China
DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation
Apr 27, 2026Tal Grossman, Noa Cahan, Lev Ayzenberg +1Source-Free Domain Adaptation Image Segmentation
School of Electrical Engineering, Tel Aviv University, Israel · School of Biomedical Engineering, Tel Aviv University, Israel

Mar 29, 2026J/K move · Enter open · S save

RAP: Retrieve, Adapt, and Prompt-Fit for Training-Free Few-Shot Medical Image Segmentation

Zhihao Mao, Bangpu Chen

School of Computer Science, China University of Geosciences, Wuhan, China

Medical Image Segmentation Few-Shot Medical Image Segmentation Promptable Image Segmentation Medical Imaging Foundation Models

Open paper AskPDF arXiv

Few-shot medical image segmentation (FSMIS) has achieved notable progress, yet most existing methods mainly rely on semantic correspondences from scarce annotations while under-utilizing a key property of medical imagery: anatomical targets exhibit repeatable high-frequency morphology (e.g., boundary geometry and spatial layout) across patients and acquisitions. We propose RAP, a training-free framework that retrieves, adapts, and prompts Segment Anything Model 2 (SAM2) for FSMIS. First, RAP retrieves morphologically compatible supports from an archive using DINOv3 features to reduce brittleness in single-support choice. Second, it adapts the retrieved support mask to the query by fitting boundary-aware structural cues, yielding an anatomy-consistent pre-mask under domain shifts. Third, RAP converts the pre-mask into prompts by sampling positive points via Voronoi partitioning and negative points via sector-based sampling, and feeds them into SAM2 for final refinement without any fine-tuning. Extensive experiments on multiple medical segmentation benchmarks show that RAP consistently surpasses prior FSMIS baselines and achieves state-of-the-art performance. Overall, RAP demonstrates that explicit structural fitting combined with retrieval-augmented prompting offers a simple and effective route to robust training-free few-shot medical segmentation.

Abstract

Explore similar work

RAP: Retrieve, Adapt, and Prompt-Fit for Training-Free Few-Shot Medical Image Segmentation

Benchmarking Foundation and Large Language Models for Few-Shot Medical Image Segmentation

DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation