When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning

Marko Haralović; Sounic Akkaraju; Carlo Baretta; Vasil Zapryanov; Alexia Briassouli

cs.CVAug 21, 2026

When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning

Authors: Marko Haralović, Sounic Akkaraju, Carlo Baretta, Vasil Zapryanov, Alexia Briassouli

Organizations: University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, Croatia · University of Twente, Enschede, The Netherlands

Abstract

Foundation models for medical image segmentation, like prompt-based MedSAM, generalize well across domains and modalities, often in zero or few-shot setups. However, their performance depends on the quality of prompts and the adaptation of the models to custom datasets. This work systematically examines how MedSAM generalizes across diverse medical imaging benchmarks, with six adaptation strategies: full-model and encoder-only LoRA, shallow and deep visual prompt tuning (VPT), and decoder-only and full fine-tuning. Models are trained on the International Skin Imaging Collaboration Challenge (ISIC 2018) dataset and evaluated under clean and increasingly noisy prompts on IN and Out-of-Distribution (OOD) datasets: close-OOD PH2 (dermoscopy), far-OOD BUSI (Breast Ultrasound Images Dataset) and CBIS-DDSM (Curated Breast Imaging Subset of the Digital Database for Screening Mammography). We show that adaptation improves performance on IN and close-OOD data but often reduces performance on far-OOD data. Full fine-tuning provides the best tradeoff, while encoder-only LoRA is the strongest parameter-efficient alternative, outperforming standard LoRA and VPT under far-OOD shifts. Using Centered Kernel Alignment (CKA), we show that far-OOD degradation is strongly associated with drift in decoder representations, whereas encoder similarity alone does not explain robustness. This suggests encoder-only LoRA provides stronger robustness than standard LoRA by adapting the encoder to distribution shift in visual features, while preserving the decoder pathway. We further show that random 0-100 pixel jitter on prompts produces more robust and better performing models. We thus conclude that robust MedSAM adaptation requires the combined consideration of prompt noise exposure, domain shift, and representation preservation. We release our code: https://github.com/ImSounic/medsam-vpt

Figures & tables

Figure 2: Robustness to bounding-box perturbation under variable training jitter. Dice is reported across evaluation jitter levels for all adaptation strategies on ID, near-OOD, and far-OOD datasets.

Regime	Method	0	20	r100	Mean	Best train	Best method
ID	Dec. only	-0.016	-0.007	- 0.014	$-0.003\pm 0.002$	r100
	VPT-S	-0.008	-0.001	- 0.009	$-0.000\pm 0.003$	r100
	VPT-D	-0.010	-0.005	- 0.013	$-0.001\pm 0.002$	r100
	LoRA	- 0.012	- 0.016	- 0.035	$\mathbf{0.021\pm 0.001}$	r100	LoRA
	Enc-LoRA	-0.027	-0.028	- 0.053	$-0.001\pm 0.002$	r100
	Full FT	-0.006	-0.005	- 0.034	$\underline{0.007\pm 0.002}$	r100

Table 1: Mean Dice gain over zero-shot by regime, method, and training jitter. Values are averaged over three runs; the Mean column reports mean

\pm

standard deviation. Bold and underline indicate the best and second-best methods.

CKA feature	ID (ISIC 2018)	Close-OOD (PH 2 )	Far-OOD (BUSI/CBIS)
IoU token layer	0.097 / 0.106	-0.018 / -0.021	0.760 ∗∗∗ / 0.692 ∗∗∗
Output layer	0.060 / 0.026	0.004 / -0.017	0.755 ∗∗∗ / 0.718 ∗∗∗
Decoder layers	-0.075 / -0.107	-0.022 / -0.079	0.726 ∗∗∗ / 0.687 ∗∗∗
Upscaled emb.	0.195 / 0.050	0.294 ∗ / 0.213	0.713 ∗∗∗ / 0.791 ∗∗∗
Encoder layers	-0.282 ∗ / -0.299 ∗	-0.281 ∗ / -0.352 ∗∗	0.091 / 0.059

Table 2: Spearman/Pearson correlations between CKA and Dice gain over zero-shot. Stars denote false discovery rate corrected significance across 30 tests: ∗

q<0.05

, ∗∗

q<0.01

, ∗∗∗

q<0.001

. Total sample size is

n=3000

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	pm=20	pm=50	pm=100	pm=200
ISIC2018-test	$11.70\%$	$30.51\%$	$66.40\%$	$152.26\%$
PH2	$6.57\%$	$16.84\%$	$35.11\%$	$72.06\%$
BUSI	$18.71\%$	$51.36\%$	$113.75\%$	$270.96\%$
CBIS-DDSM	$64.06\%$	$193.49\%$	$485.19\%$	$1327.90\%$
Overall	$\mathbf{17.61\%}$	$\mathbf{48.85\%}$	$\mathbf{112.67\%}$	$\mathbf{280.14\%}$

Appendix

Table 3: Relative increase in tight bounding-box area under expansion due to evaluation jitter. Values report the mean area increase with respect to the original tight bounding box for each dataset and perturbation level.

Method	ISIC 2018 (ID)	PH 2 (near-OOD)	BUSI (far-OOD)	CBIS-DDSM (far-OOD)
Zero-shot	6 [0, 15]	7 [1, 16]	14 [9, 25]	12 [7, 17]
Dec. only	0 [0, 3]	0 [0, 3]	4 [2, 17]	4 [3, 6]
VPT-S	0 [0, 3]	0 [0, 4]	21 [5, 387]	14 [8, 37]
VPT-D	0 [0, 3]	0 [0, 4]	25 [6, 394]	18 [10, 359]
LoRA	0 [0, 1]	0 [0, 1]	26 [6, 402]	47 [13, 438]
Enc-LoRA	0 [0, 0]	0 [0, 0]	3 [1, 11]	5 [3, 7]

Appendix

Table 4: HD95 at zero evaluation jitter. Values are median [interquartile range] in pixels at

1024\times 1024

; lower is better.

Figure 3: HD95 across evaluation jitter levels for each adaptation method and dataset. Values are shown on a logarithmic scale in pixels at $1024\times 1024$ ; lower is better.

Figure 4: CKA-performance relationship by method.

Figure 5: Overview of the relationship between CKA similarity and segmentation performance.

Figure 6: CKA similarity across datasets and jitters using perfect bounding boxes.

Figure 7: Dice degradation with increasing bounding-box jitter by dataset and method.

Figure 8: Heatmap of the degradation in Dice with increasing bounding-box jitter, by datasets across models

Dataset	Training	Method	Dice@0	mDice	Dice@200	CKA@0	mCKA	CKA@200
ISIC 2018 ID	0	Zero-shot	0.9072	0.9019	0.7673	1.0000	1.0000	1.0000
		Decoder-only	0.9487	0.8798	0.7075	0.9462	0.9432	0.9330
		VPT-shallow	0.9457	0.8858	0.7306	0.8845	0.8908	0.8913
		VPT-deep	0.9472	0.8863	0.7256	0.8767	0.8791	0.8718
		LoRA	0.9545	0.9015	0.7978	0.5685	0.5794	0.5771
		LoRA Encoder-only	0.9580	0.8435	0.6450	0.6484	0.6462	0.6379

Appendix

Table 5: Performance and CKA under bounding-box jitter. Dice performance and CKA are reported at zero jitter, averaged across evaluation jitters, and at 200 px jitter. For performance columns, underline marks the best method within each dataset and training-jitter block, while bold marks the best value for each dataset and evaluation column across all training-jitter settings. For CKA columns, bold and underline mark the best and second-best adapted methods, respectively, within each dataset and training-jitter block

Dataset	Train	Method	E0	E20	E50	E100	E200
ISIC 2018 trained on	pm=0	Zero-shot	0.9072	0.9206	0.9144	0.8708	0.7673
		Decoder-only	0.9487	0.9349	0.8900	0.8176	0.7102
		VPT-shallow	0.9449	0.9327	0.8952	0.8335	0.7352
		VPT-deep	0.9470	0.9343	0.8944	0.8281	0.7251
		LoRA	0.9556	0.9416	0.9040	0.8522	0.7867
		LoRA Encoder-only	0.9580	0.9364	0.8832	0.7951	0.6450

Appendix

Table 6: Dice under bounding-box jitter. Models are trained with 0 px, 20 px, or random 0–100 px jitter and evaluated at 0–200 px jitter. Values are means over 3 seeds; underline denotes the best method within each training block and bold the best per dataset/evaluation jitter.

Explore similar work

RobustMedSAM: Degradation-Resilient Medical Image Segmentation via Robust Foundation Model Adaptation
Apr 10, 2026Jieru Li, Matthew Chen, Micky C. Nnamdi +3Medical Image Segmentation Medical Imaging Foundation Models
Georgia Institute of Technology
FAN-LoRA: A Fourier-Adaptive Nonlinear Low-Rank Adaptor for Medical Foundation Model Domain Adaptation
Aug 27, 2026Ziquan Liu, Zhewei Zhu, Xuyang ShiVision Foundation Model Adaptation Low-Rank Adaptation
School of Information and Control Engineering, Southwest University of Science and Technology Mianyang, China
DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation
Apr 27, 2026Tal Grossman, Noa Cahan, Lev Ayzenberg +1Source-Free Domain Adaptation Image Segmentation
School of Electrical Engineering, Tel Aviv University, Israel · School of Biomedical Engineering, Tel Aviv University, Israel

Apr 10, 2026J/K move · Enter open · S save

RobustMedSAM: Degradation-Resilient Medical Image Segmentation via Robust Foundation Model Adaptation

Jieru Li, Matthew Chen, Micky C. Nnamdi, J. Ben Tamo+2

Georgia Institute of Technology

Medical Image Segmentation Medical Imaging Foundation Models Image Corruption Robustness

Open paper AskPDF arXiv

Medical image segmentation models built on Segment Anything Model (SAM) achieve strong performance on clean benchmarks, yet their reliability often degrades under realistic image corruptions such as noise, blur, motion artifacts, and modality-specific distortions. Existing approaches address either medical-domain adaptation or corruption robustness, but not both jointly. In SAM, we find that these capabilities are concentrated in complementary modules: the image encoder preserves medical priors, while the mask decoder governs corruption robustness. Motivated by this observation, we propose RobustMedSAM, which adopts module-wise checkpoint fusion by initializing the image encoder from MedSAM and the mask decoder from RobustSAM under a shared ViT-B architecture. We then fine-tune only the mask decoder on 35 medical datasets from MedSegBench, spanning six imaging modalities and 12 corruption types, while freezing the remaining components to preserve pretrained medical representations. We additionally investigate an SVD-based parameter-efficient variant for limited encoder adaptation. Experiments on both in-distribution and out-of-distribution benchmarks show that RobustMedSAM improves degraded-image Dice from 0.613 to 0.719 (+0.106) over SAM, demonstrating that structured fusion of complementary pretrained models is an effective and practical approach for robust medical image segmentation.

Abstract

Explore similar work

RobustMedSAM: Degradation-Resilient Medical Image Segmentation via Robust Foundation Model Adaptation

FAN-LoRA: A Fourier-Adaptive Nonlinear Low-Rank Adaptor for Medical Foundation Model Domain Adaptation

DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation