SAIF: A Stability-Aware Inference Framework for Medical Image Segmentation with Segment Anything Model

Ke Wu; Shiqi Chen; Yiheng Zhong; Hengxian Liu; Yingxue Su; Yifang Wang; Junhao Jin; Guangyu Ren

cs.CVMar 13, 2026

SAIF: A Stability-Aware Inference Framework for Medical Image Segmentation with Segment Anything Model

Authors: Ke Wu, Shiqi Chen, Yiheng Zhong, Hengxian Liu, Yingxue Su, Yifang Wang, Junhao Jin, Guangyu Ren

Organizations: Xi’an Jiaotong-Liverpool University, Suzhou, China · University College London, London, United Kingdom · Anhui Jianzhu University, Hefei, China

Abstract

Segment Anything Model (SAM) enable scalable medical image segmentation but suffer from inference-time instability when deployed as a frozen backbone. In practice, bounding-box prompts often contain localization errors, and fixed threshold binarization introduces additional decision uncertainty. These factors jointly cause high prediction variance, especially near object boundaries, degrading reliability. We propose the Stability-Aware Inference Framework (SAIF), a training-free and plug-and-play inference framework that improves robustness by explicitly modeling prompt and threshold uncertainty. SAIF constructs a joint uncertainty space via structured box perturbations and threshold variations, evaluates each hypothesis using decision stability and boundary consistency, and introduces a stability-consistency score to filter unstable candidates and perform stability-weighted fusion in probability space. Experiments on Synapse, CVC-ClinicDB, Kvasir-SEG, and CVC-300 demonstrate that SAIF consistently improves segmentation accuracy and robustness, achieving state-of-the-art performance without retraining or architectural modification. Our anonymous code is released at https://anonymous.4open.science/r/SAIF.

Figures & tables

Figure 1: Overview of medical adaptations of the Segment Anything Model. (a) Full retraining on large-scale medical data, (b) parameter-efficient fine-tuning, (c) structural modifications to the encoder and mask decoder, and (d) inference-time operating conditions affecting output reliability, addressed by approach.

Figure 2: This diagram shows the SAIF framework for stability-aware medical image segmentation. It starts with perturbing the original box prompt into multiple outer and inner candidates, which are processed through the model to generate probability maps. Stability-consistency scoring ranks candidates based on Soft IoU and percentiles. The top- $n$ candidates are selected and fused to produce the final segmentation output, using no additional network execution or training.

Category	Methods	CVC-ClinicDB		Kvasir-SEG		CVC-300
Category	Methods	mDice $\uparrow$	mIoU $\uparrow$	mDice $\uparrow$	mIoU $\uparrow$	mDice $\uparrow$	mIoU $\uparrow$
Train based	U-Net [ 15 ]	82.31	75.56	81.84	74.63	71.07	62.76
	U-Net++ [ 16 ]	79.45	72.92	82.19	74.35	70.72	62.46
	TransUNet [ 17 ]	89.46	85.02	86.91	80.58	-	-
	EU-Net [ 18 ]	90.24	84.67	90.81	85.42	83.76	76.55
	PraNet [ 19 ]	90.20	85.82	90.15	84.83	87.32	80.48

Table 1: Quantitative comparison on polyp segmentation benchmarks. All SAM-based methods use dataset-provided box prompts, while train-based methods are prompt-free. Light gray cells indicate methods equipped with the proposed strategy. Red values denote best-performing results, and purple values indicate improvements larger than 5% over the corresponding baselines. mDice: the mean Dice coefficient; mIoU: the mean Intersection-over-Union.

Group	Method	Spleen	Kidney (R)	Kidney (L)	Gallbladder	Liver	Stomach	Aorta	Pancreas	mDice[%] $\uparrow$	HD95[mm] $\downarrow$
Non SAM	TransUNet [ 17 ]	85.08	77.02	81.87	63.16	94.08	75.62	87.23	55.86	77.48	31.69
	SwinUNet [ 20 ]	90.66	79.61	83.28	66.53	94.29	76.60	85.47	56.58	79.13	21.55
	UNETR [ 23 ]	87.81	84.80	85.66	60.56	94.46	73.99	89.99	59.25	79.56	18.59
	nnU-Net [ 24 ]	90.31	91.46	86.07	71.71	95.84	79.01	92.39	82.92	86.21	10.63
	nnFormer [ 25 ]	90.51	86.25	86.57	70.17	96.84	86.83	92.40	83.35	86.57	10.78
SAM based	SAM3D [ 26 ]	84.29	85.64	86.31	49.81	95.42	76.11	89.57	69.32	79.56	23.10

Table 2: Quantitative comparison on the Synapse multi-organ CT dataset. Color coding follows the same convention as Table 1 . mDice: the mean Dice coefficient; HD95: the 95th percentile of the Hausdorff Distance.

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 788 papers

Learning from Noisy Prompts: Saliency-Guided Prompt Distillation for Robust Segmentation with SAM
Apr 25, 2026Jingxuan Kang, Ziqi Zhang, Shaoming Zheng +9Medical Imaging Foundation Models Promptable Image Segmentation
Imperial College London · Beihang University · National Health Service +1
Enhancing MedSAM with a Lightweight Box Predictor for Medical Image Segmentation
Jun 3, 2026Amirhossein Movahedisefat, Amirreza Fateh, Mohammad Reza MohammadiInteractive Medical Image Segmentation Medical Imaging Foundation Models
School of Computer Engineering, Iran University of Science and Technology (IUST), Tehran, Iran
Focus on Background: Exploring SAM's Potential in Few-shot Medical Image Segmentation with Background-centric Prompting
Mar 22, 2026Yuntian Bo, Yazhou Zhu, Piotr Koniusz +1Few-Shot Medical Image Segmentation Promptable Image Segmentation
Nanjing University of Science and Technology · University of New South Wales · Data61 CSIRO

Apr 25, 2026J/K move · Enter open · S save

Learning from Noisy Prompts: Saliency-Guided Prompt Distillation for Robust Segmentation with SAM

Jingxuan Kang, Ziqi Zhang, Shaoming Zheng, Shuang Li+8

Imperial College London · Beihang University · National Health Service+1

Medical Imaging Foundation Models Promptable Image Segmentation Medical Image Segmentation 3D Medical Image Segmentation

Open paper AskPDF arXiv

Segmentation is central to clinical diagnosis and monitoring, yet the reliability of modern foundation models in medical imaging still depends on the availability of precise prompts. The Segment Anything Model (SAM) offers powerful zero-shot capabilities, although it collapses under the weak, generic, and noisy prompts that dominate real clinical workflows. In practice, annotations such as centerline points are coarse and ambiguous, often drifting across neighboring anatomy and misguiding SAM toward inconsistent or incomplete masks. We introduce SPD, a Saliency-Guided Prompt Distillation framework that converts these unreliable cues into robust guidance. SPD first learns data-driven anatomical priors through a lightweight saliency head to obtain confident localization maps. These priors then drive Contextual Prompt Distillation, which validates and enriches noisy prompts using cues from anatomically adjacent slices, producing a consensus prompt set that matches the behavior of expert reasoning. A Pairwise Slice Consistency objective further enforces local anatomical coherence during segmentation. Experiments on four challenging MRI and CT benchmarks demonstrate that SPD consistently outperforms existing SAM adaptations and supervised baselines, delivering large gains in both region-based and boundary-based metrics. SPD provides a practical and principled path toward reliable foundation model deployment in clinical environments where only imperfect prompts are available.

Abstract

Medical Image Segmentation

Explore similar work

Learning from Noisy Prompts: Saliency-Guided Prompt Distillation for Robust Segmentation with SAM

Enhancing MedSAM with a Lightweight Box Predictor for Medical Image Segmentation

Focus on Background: Exploring SAM's Potential in Few-shot Medical Image Segmentation with Background-centric Prompting