RobustMedSAM: Degradation-Resilient Medical Image Segmentation via Robust Foundation Model Adaptation

Jieru Li; Matthew Chen; Micky C. Nnamdi; J. Ben Tamo; Benoit L. Marteau; May D. Wang

cs.CVApr 10, 2026

RobustMedSAM: Degradation-Resilient Medical Image Segmentation via Robust Foundation Model Adaptation

Authors: Jieru Li, Matthew Chen, Micky C. Nnamdi, J. Ben Tamo, Benoit L. Marteau, May D. Wang

Organizations: Georgia Institute of Technology

Abstract

Medical image segmentation models built on Segment Anything Model (SAM) achieve strong performance on clean benchmarks, yet their reliability often degrades under realistic image corruptions such as noise, blur, motion artifacts, and modality-specific distortions. Existing approaches address either medical-domain adaptation or corruption robustness, but not both jointly. In SAM, we find that these capabilities are concentrated in complementary modules: the image encoder preserves medical priors, while the mask decoder governs corruption robustness. Motivated by this observation, we propose RobustMedSAM, which adopts module-wise checkpoint fusion by initializing the image encoder from MedSAM and the mask decoder from RobustSAM under a shared ViT-B architecture. We then fine-tune only the mask decoder on 35 medical datasets from MedSegBench, spanning six imaging modalities and 12 corruption types, while freezing the remaining components to preserve pretrained medical representations. We additionally investigate an SVD-based parameter-efficient variant for limited encoder adaptation. Experiments on both in-distribution and out-of-distribution benchmarks show that RobustMedSAM improves degraded-image Dice from 0.613 to 0.719 (+0.106) over SAM, demonstrating that structured fusion of complementary pretrained models is an effective and practical approach for robust medical image segmentation.

Figures & tables

Figure 1: Overview of RobustMedSAM. During training, each medical image is paired with a degraded counterpart generated by medical degradation augmentation. Both the clean and degraded images are processed using the same frozen MedSAM image encoder and prompt encoder. The resulting features are decoded by a shared robust decoder initialized from RobustSAM. Finetuned modules are highlighted in color, while frozen modules are shown in gray. At inference time, RobustMedSAM requires only a single input image without any clean reference.

Modality	Applied Degradations
Ultrasound	Speckle noise, Salt-and-pepper noise
MRI	Rician noise, Rayleigh noise, Step-wise motion artifact
CT	Poisson noise, Step-wise motion artifact
X-Ray, Microscopy	Poisson noise
Endoscopy	Motion blur, Zoom blur, Compression artifacts
Fundus	ISO noise, Zoom blur, Compression artifacts

Table 1: Degradation types applied to each imaging modality

Figure 2: Tail robustness under degradations. Empirical CDF of degradation-level Dice across datasets. RobustMedSAM shows a consistent rightward shift under point prompts, indicating improved worst-case performance, while remaining comparable to MedSAM under box prompts.

Figure 3: Point Prompts: Clean vs. degraded comparison (overall Dice ↑, point prompts). RobustMedSAM improves robustness on degraded inputs while maintaining clean performance comparable to SAM; +SVD exhibits a robustness–clean trade-off.

Dice

\uparrow

IoU

\uparrow

NSD

\uparrow

(

\tau{=}2

px)

Dataset

SAM

MedSAM

Robust SAM

Robust MedSAM

Robust MedSAM +SVD

SAM

MedSAM

Robust SAM

Robust MedSAM

Robust MedSAM +SVD

SAM

MedSAM

Robust SAM

Robust MedSAM

Robust MedSAM +SVD

ISIC 2016

0.752

(

\pm 0.033

)

0.202

(

\pm 0.013

)

0.836

(

\pm 0.019

)

0.911

(

\pm 0.003

)

0.910

(

\pm 0.006

)

0.640

(

\pm 0.037

)

0.120

(

\pm 0.009

)

0.742

(

\pm 0.022

)

0.844

(

\pm 0.005

)

0.842

(

\pm 0.006

)

0.088

(

\pm 0.024

)

0.017

(

\pm 0.001

)

0.135

(

\pm 0.021

)

0.247

(

\pm 0.009

)

0.232

(

\pm 0.007

)

BUSI

0.718

(

\pm 0.036

)

0.249

(

\pm 0.016

)

0.686

(

\pm 0.061

)

0.760

(

\pm 0.017

)

0.678

(

\pm 0.032

)

0.609

(

\pm 0.039

)

0.152

(

\pm 0.010

)

0.578

(

\pm 0.061

)

0.642

(

\pm 0.018

)

0.561

(

\pm 0.028

)

0.226

(

\pm 0.020

)

0.044

(

\pm 0.001

)

0.223

(

\pm 0.030

)

0.228

(

\pm 0.004

)

0.206

(

\pm 0.012

)

PROMISE12

0.593

(

\pm 0.069

)

0.209

(

\pm 0.004

)

0.554

(

\pm 0.044

)

0.805

(

\pm 0.014

)

0.751

(

\pm 0.020

)

0.481

(

\pm 0.069

)

0.123

(

\pm 0.003

)

0.426

(

\pm 0.042

)

0.687

(

\pm 0.019

)

0.620

(

\pm 0.026

)

0.176

(

\pm 0.045

)

0.028

(

\pm 0.001

)

0.122

(

\pm 0.023

)

0.199

(

\pm 0.018

)

0.177

(

\pm 0.012

)

UWaterloo Skin Cancer

0.841

(

\pm 0.027

)

0.202

(

\pm 0.034

)

0.914

(

\pm 0.007

)

0.869

(

\pm 0.019

)

0.872

(

\pm 0.010

)

0.757

(

\pm 0.033

)

0.120

(

\pm 0.023

)

0.850

(

\pm 0.009

)

0.782

(

\pm 0.022

)

0.785

(

\pm 0.011

)

0.351

(

\pm 0.038

)

0.046

(

\pm 0.008

)

0.444

(

\pm 0.038

)

0.295

(

\pm 0.011

)

0.270

(

\pm 0.013

)

BriFiSeg

0.311

(

\pm 0.014

)

0.114

(

\pm 0.011

)

0.329

(

\pm 0.022

)

0.305

(

\pm 0.056

)

0.436

(

\pm 0.041

)

0.188

(

\pm 0.010

)

0.062

(

\pm 0.006

)

0.201

(

\pm 0.016

)

0.194

(

\pm 0.041

)

0.294

(

\pm 0.034

)

0.108

(

\pm 0.004

)

0.049

(

\pm 0.001

)

0.104

(

\pm 0.002

)

0.129

(

\pm 0.015

)

0.183

(

\pm 0.009

)

Table 2: Segmentation performance under various image degradations with point prompts . Values are reported as mean

\pm

std across degradation types per dataset. Metrics: Dice

\uparrow

, IoU

\uparrow

, and boundary accuracy NSD

\uparrow

(

\tau{=}2

px).

Figure 4: Per-degradation analysis . Dice comparison across degradation types under point and box prompts. RobustMedSAM shows the clearest gains under point prompting, while remaining competitive under box prompting.

Figure 5: Per-modality analysis . RobustMedSAM shows the clearest gains under point prompts, especially on MRI and ultrasound, while remaining competitive under box prompts.

Dataset	SAM	MedSAM	RobustSAM	Ours	+SVD
CHASE_DB1	0.153	0.078	0.152	0.121	0.059
DRIVE	0.209	0.093	0.216	0.224	0.141
ISIC 2018	0.788	0.183	0.843	0.871	0.870
Kvasir	0.770	0.195	0.775	0.774	0.754
PanDental	0.638	0.106	0.668	0.820	0.649
DCA1	0.272	0.107	0.313	0.297	0.217

Table 3: Out-of-distribution evaluation (Dice

\uparrow

, point prompts ).

Figure 6: Prompt sensitivity. Dice versus the number of point prompts

K

. RobustMedSAM improves rapidly and saturates around

K=3

, while box-prompted models are shown as reference baselines.

Panel of Figure 7 — Figure 7: Qualitative segmentation results under degradations. Each panel shows (left to right): degraded input with point prompts , ground truth, SAM, MedSAM, RobustMedSAM, and RobustMedSAM+SVD predictions. Green = true positive, red = false positive, blue = false negative.

Figure S1 : Detailed architecture of Anti-degradation Mask Feature Generation and Anti-degradation Output Token Generation modules.

	SAM			MedSAM			RobustMedSAM			+SVD
Dataset	Clean	Deg.	$\Delta$	Clean	Deg.	$\Delta$	Clean	Deg.	$\Delta$	Clean	Deg.	$\Delta$
ISIC 2016	0.734	0.752	+0.018	0.203	0.202	-0.001	0.809	0.911	+0.102	0.699	0.910	+0.210
BUSI	0.717	0.718	+0.001	0.252	0.249	-0.002	0.610	0.760	+0.151	0.434	0.678	+0.243
PROMISE12	0.639	0.593	-0.046	0.208	0.209	+0.001	0.714	0.805	+0.091	0.544	0.751	+0.207
UWaterloo	0.840	0.841	+0.001	0.201	0.202	+0.002	0.736	0.869	+0.132	0.759	0.872	+0.113
BriFiSeg	0.321	0.311	-0.010	0.112	0.114	+0.002	0.289	0.305	+0.016	0.125	0.436	+0.311

Table S1 : Clean vs. degraded with point prompts (Dice

\uparrow

\Delta

= degraded

-

clean; positive values indicate the model benefits from the clear-degraded pair protocol.

Degradation	SAM	MedSAM	RobustSAM	Ours - Robust MedSAM	Ours - Robust MedSAM+SVD
Brightness	$0.619$ $\pm 0.177$	$0.206$ $\pm 0.040$	$0.633$ $\pm 0.196$	$0.729$ $\pm 0.195$	$0.710$ $\pm 0.166$
Color Jitter	$0.801$ $\pm 0.059$	$0.201$ $\pm 0.017$	$0.870$ $\pm 0.043$	$0.897$ $\pm 0.012$	$0.880$ $\pm 0.024$
Compression	$0.813$ $\pm 0.042$	$0.198$ $\pm 0.002$	$0.879$ $\pm 0.042$	$0.887$ $\pm 0.024$	$0.905$ $\pm 0.004$
Contrast	$0.605$ $\pm 0.176$	$0.189$ $\pm 0.054$	$0.641$ $\pm 0.200$	$0.717$ $\pm 0.182$	$0.696$ $\pm 0.167$
Gaussian Blur	$0.609$ $\pm 0.163$	$0.203$ $\pm 0.050$	$0.632$ $\pm 0.206$	$0.711$ $\pm 0.213$	$0.705$ $\pm 0.169$
Gaussian Noise	$0.625$ $\pm 0.213$	$0.206$ $\pm 0.041$	$0.639$ $\pm 0.216$	$0.717$ $\pm 0.230$	$0.703$ $\pm 0.182$

Table S2 : Per-degradation with point prompts — Dice

\uparrow

. Values are mean

\pm

std across datasets. (Single-dataset degradations show mean only.)

Degradation	SAM	MedSAM	RobustSAM	Ours - Robust MedSAM	Ours - Robust MedSAM+SVD
Brightness	$0.506$ $\pm 0.190$	$0.123$ $\pm 0.027$	$0.523$ $\pm 0.219$	$0.621$ $\pm 0.212$	$0.595$ $\pm 0.192$
Color Jitter	$0.704$ $\pm 0.079$	$0.120$ $\pm 0.013$	$0.787$ $\pm 0.058$	$0.820$ $\pm 0.020$	$0.799$ $\pm 0.035$
Compression	$0.709$ $\pm 0.053$	$0.118$ $\pm 0.003$	$0.802$ $\pm 0.057$	$0.809$ $\pm 0.033$	$0.832$ $\pm 0.010$
Contrast	$0.494$ $\pm 0.187$	$0.113$ $\pm 0.036$	$0.532$ $\pm 0.224$	$0.607$ $\pm 0.199$	$0.582$ $\pm 0.193$
Gaussian Blur	$0.494$ $\pm 0.175$	$0.120$ $\pm 0.033$	$0.522$ $\pm 0.227$	$0.605$ $\pm 0.221$	$0.591$ $\pm 0.193$
Gaussian Noise	$0.519$ $\pm 0.226$	$0.123$ $\pm 0.027$	$0.532$ $\pm 0.235$	$0.615$ $\pm 0.232$	$0.589$ $\pm 0.201$

Table S3 : Per-degradation analysis with point prompts — IoU

\uparrow

. Values are mean

\pm

std across datasets. (Single-dataset degradations show mean only.)

Panel of Figure 15 — Figure S2 : Bounding Box: Qualitative segmentation results under degradation. Each panel shows (left to right): degraded input with ”bounding box prompts, ground truth, SAM, MedSAM, RobustMedSAM predictions. Green = true positive, red = false positive, blue = false negative.

Degradation	SAM	MedSAM	RobustSAM	Ours - Robust MedSAM	Ours - Robust MedSAM+SVD
Brightness	$0.191$ $\pm 0.088$	$0.036$ $\pm 0.011$	$0.203$ $\pm 0.112$	$0.199$ $\pm 0.066$	$0.182$ $\pm 0.053$
Color Jitter	$0.212$ $\pm 0.138$	$0.038$ $\pm 0.021$	$0.274$ $\pm 0.149$	$0.262$ $\pm 0.028$	$0.245$ $\pm 0.022$
Compression	$0.228$ $\pm 0.107$	$0.030$ $\pm 0.012$	$0.296$ $\pm 0.133$	$0.275$ $\pm 0.023$	$0.252$ $\pm 0.019$
Contrast	$0.184$ $\pm 0.092$	$0.036$ $\pm 0.011$	$0.209$ $\pm 0.127$	$0.205$ $\pm 0.069$	$0.195$ $\pm 0.062$
Gaussian Blur	$0.178$ $\pm 0.080$	$0.037$ $\pm 0.010$	$0.195$ $\pm 0.105$	$0.202$ $\pm 0.066$	$0.194$ $\pm 0.060$
Gaussian Noise	$0.190$ $\pm 0.116$	$0.038$ $\pm 0.012$	$0.207$ $\pm 0.141$	$0.199$ $\pm 0.069$	$0.193$ $\pm 0.058$

Table S4 : Per-degradation analysis with point prompts — NSD

\uparrow

(

\tau{=}2

px). Values are mean

\pm

std across datasets. (Single-dataset degradations show mean only.)

	Dice $\uparrow$			IoU $\uparrow$
Dataset	SAM	MedSAM	Robust MedSAM	SAM	MedSAM	Robust MedSAM
ISIC 2016	$0.851$ ( $\pm 0.012$ )	$0.929$ ( $\pm 0.003$ )	$0.910$ ( $\pm 0.006$ )	$0.747$ ( $\pm 0.018$ )	$0.869$ ( $\pm 0.004$ )	$0.844$ ( $\pm 0.009$ )
BUSI	$0.778$ ( $\pm 0.019$ )	$0.911$ ( $\pm 0.013$ )	$0.807$ ( $\pm 0.012$ )	$0.683$ ( $\pm 0.020$ )	$0.840$ ( $\pm 0.020$ )	$0.701$ ( $\pm 0.012$ )
PROMISE12	$0.876$ ( $\pm 0.038$ )	$0.886$ ( $\pm 0.018$ )	$0.888$ ( $\pm 0.009$ )	$0.793$ ( $\pm 0.048$ )	$0.803$ ( $\pm 0.028$ )	$0.804$ ( $\pm 0.012$ )
UWaterloo Skin Cancer	$0.881$ ( $\pm 0.011$ )	$0.956$ ( $\pm 0.007$ )	$0.869$ ( $\pm 0.017$ )	$0.797$ ( $\pm 0.017$ )	$0.917$ ( $\pm 0.012$ )	$0.791$ ( $\pm 0.019$ )
BriFiSeg	$0.250$ ( $\pm 0.021$ )	$0.244$ ( $\pm 0.007$ )	$0.505$ ( $\pm 0.035$ )	$0.146$ ( $\pm 0.013$ )	$0.142$ ( $\pm 0.005$ )	$0.348$ ( $\pm 0.031$ )

Table S5 : Bounding box prompt segmentation performance under various image degradations. Values are reported as mean

\pm

std across degradation types per dataset.

	Dice $\uparrow$			IoU $\uparrow$
Degradation	SAM	MedSAM	Robust MedSAM	SAM	MedSAM	Robust MedSAM
Brightness	$0.671$ $\pm 0.275$	$0.788$ $\pm 0.277$	$0.790$ $\pm 0.151$	$0.575$ $\pm 0.281$	$0.713$ $\pm 0.297$	$0.689$ $\pm 0.183$
Color Jitter	$0.862$ $\pm 0.016$	$0.944$ $\pm 0.020$	$0.868$ $\pm 0.046$	$0.766$ $\pm 0.029$	$0.895$ $\pm 0.034$	$0.795$ $\pm 0.052$
Compression	$0.885$ $\pm 0.013$	$0.943$ $\pm 0.022$	$0.898$ $\pm 0.026$	$0.799$ $\pm 0.023$	$0.894$ $\pm 0.039$	$0.829$ $\pm 0.035$
Contrast	$0.654$ $\pm 0.271$	$0.786$ $\pm 0.276$	$0.798$ $\pm 0.138$	$0.557$ $\pm 0.275$	$0.710$ $\pm 0.296$	$0.696$ $\pm 0.171$
Gaussian Blur	$0.647$ $\pm 0.292$	$0.785$ $\pm 0.266$	$0.800$ $\pm 0.150$	$0.552$ $\pm 0.290$	$0.705$ $\pm 0.284$	$0.701$ $\pm 0.183$

Table S6 : Per-degradation analysis (bounding box prompt) . Values are mean

\pm

std across datasets.

Explore similar work

When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning
Aug 21, 2026Marko Haralović, Sounic Akkaraju, Carlo Baretta +2Vision Foundation Model Adaptation Medical Imaging Foundation Models
University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, Croatia · University of Twente, Enschede, The Netherlands
Dual-Adaptive SAM3: Hierarchical Routing over Low-Rank Expert Layers for Parameter-Efficient Medical Image Segmentation
Jun 30, 2026Ying Chen, Jinyue Li, Kun Wang +2VLM Adaptation Text-Guided Image Segmentation
Shenzhen Research Institute, The Chinese University of Hong Kong · University of Science and Technology of China · Nanyang Technological University +1
PGE-SAM: Prompt-Guided Feature Enhancement for Interactive Segmentation under Degradation
Jun 29, 2026Tuan-Duc Nguyen, Anh-Tuan Mai, Duc-Trong LeInteractive Medical Image Segmentation Image Segmentation
FPT Software AI Center · VNU University of Engineering and Technology

Aug 21, 2026J/K move · Enter open · S save

When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning

Marko Haralović, Sounic Akkaraju, Carlo Baretta, Vasil Zapryanov+1

University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, Croatia · University of Twente, Enschede, The Netherlands

Vision Foundation Model Adaptation Medical Imaging Foundation Models Medical Image Segmentation OOD Generalization Parameter-Efficient Fine-Tuning Prompt Robustness

Open paper AskPDF arXiv

Foundation models for medical image segmentation, like prompt-based MedSAM, generalize well across domains and modalities, often in zero or few-shot setups. However, their performance depends on the quality of prompts and the adaptation of the models to custom datasets. This work systematically examines how MedSAM generalizes across diverse medical imaging benchmarks, with six adaptation strategies: full-model and encoder-only LoRA, shallow and deep visual prompt tuning (VPT), and decoder-only and full fine-tuning. Models are trained on the International Skin Imaging Collaboration Challenge (ISIC 2018) dataset and evaluated under clean and increasingly noisy prompts on IN and Out-of-Distribution (OOD) datasets: close-OOD PH2 (dermoscopy), far-OOD BUSI (Breast Ultrasound Images Dataset) and CBIS-DDSM (Curated Breast Imaging Subset of the Digital Database for Screening Mammography). We show that adaptation improves performance on IN and close-OOD data but often reduces performance on far-OOD data. Full fine-tuning provides the best tradeoff, while encoder-only LoRA is the strongest parameter-efficient alternative, outperforming standard LoRA and VPT under far-OOD shifts. Using Centered Kernel Alignment (CKA), we show that far-OOD degradation is strongly associated with drift in decoder representations, whereas encoder similarity alone does not explain robustness. This suggests encoder-only LoRA provides stronger robustness than standard LoRA by adapting the encoder to distribution shift in visual features, while preserving the decoder pathway. We further show that random 0-100 pixel jitter on prompts produces more robust and better performing models. We thus conclude that robust MedSAM adaptation requires the combined consideration of prompt noise exposure, domain shift, and representation preservation. We release our code: https://github.com/ImSounic/medsam-vpt

Abstract

Explore similar work

When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning

Dual-Adaptive SAM3: Hierarchical Routing over Low-Rank Expert Layers for Parameter-Efficient Medical Image Segmentation

PGE-SAM: Prompt-Guided Feature Enhancement for Interactive Segmentation under Degradation