Medical Image Segmentation: Foundation Model Adaptation versus Task-Specific Training

Emre Kutlusan; Yongmin Li

cs.CVNov 10, 2026

Medical Image Segmentation: Foundation Model Adaptation versus Task-Specific Training

Authors: Emre Kutlusan, Yongmin Li

Organizations: Department of Computer Science, Brunel University of London, Uxbridge, UB8 3PH, United Kingdom

Abstract

Foundation models pretrained on large natural-image collections are increasingly adapted for medical image segmentation, in the expectation that they will replace models trained for each task. Most comparisons, however, use data from the training distribution and a single task-specific baseline whose configuration is rarely justified, and therefore provide limited evidence of performance at new clinical sites. It also remains unclear how much of a foundation model must be adapted, and what determines this. We adapted a foundation model, DINOv2, with strategies ranging from tuning only its bias terms to full fine-tuning, and compared it with U-Nets and ResU-Nets trained from scratch. Experiments covered colonoscopy, breast ultrasound and dermoscopy, with testing on held-out data from the training source and on independent datasets from other institutions. The adapted foundation model generalised better on every external dataset. Within the training distribution, the task-specific models remained competitive when training data were abundant, and delineated boundaries more precisely, while the foundation model more often missed small lesions. Only a small fraction of the foundation model required adaptation, and the capacity needed was determined by the imaging domain rather than by the adaptation method. A widely used polyp test set was also found to duplicate images from another benchmark. Foundation-model adaptation is therefore warranted where models must work across sites, with capacity selected on the target domain.

Figures & tables

Figure 1: Experimental design. The architecture axis compares an adapted DINOv2 against two models trained from random initialisation; the adaptation axis compares seven configurations of the same model by trainable parameter count. The two intersect at LoRA rank 32, selected on in-domain validation. All comparisons run across three domains and eight out-of-distribution test sets.

Table 2

Figure 2: Example cases from each domain. One image per dataset with its ground-truth contour in yellow; images are centre-cropped to a common square footprint and are otherwise unmodified. The training source of each domain is framed in colour, with one of polyp’s two training sources shown, and every other set is held out. Breast has one fewer test set than the other two domains, since only two independent breast ultrasound sources with segmentation masks were available. The asterisk marks skin test sets drawn from the same archive as the training source.

Figure 3: Model architectures. (a) The five-level encoder–decoder topology shared by both task-specific comparators. (b) One block in each variant. The two differ in the residual addition and its projection shortcut, applied in both encoder and decoder, while widths, normalisation, activation, downsampling and training recipe are identical. (c) The adapted model: a DINOv2-Small encoder producing 384-dimensional patch embeddings, reduced to a one-channel logit map by a single $1{\times}1$ convolution of 385 parameters and upsampled to the input resolution. The magnified block shows where each adaptation strategy intervenes. Blue marks parameters held frozen and red marks parameters updated.

Configuration	Trainable	% of full FT	Encoder parameters updated
Frozen	385	0.002	None
BitFit	51,841	0.24	All bias terms
LoRA $r{=}4$	74,113	0.34	Rank-4 factors on Q and V
LoRA $r{=}8$	147,841	0.67	Rank-8 factors on Q and V
LoRA $r{=}16$	295,297	1.34	Rank-16 factors on Q and V
LoRA $r{=}32$	590,209	2.68	Rank-32 factors on Q and V

Table 3: Adaptation strategies on the capacity axis. Every configuration also trains the 385-parameter linear decoder, which is included in the trainable count; the percentage is relative to full fine-tuning. Q and V: query and value projections of every attention block.

Configuration	Polyp	Breast	Skin
Adaptation, capacity axis	40	40	20
Adaptation, core multi-seed comparison	20	–	20
From-scratch	300	300	100
Data-size sweep	scaled inversely with subset size

Table 4: Epoch budgets, matched within every comparison but not across domains.

Test set	$n$	Model	DSC	IoU	Sens	Prec	HD95	ASSD	NSD	$\varnothing$
Polyp
Kvasir (in)	100	Plain U-Net	0.8804	0.813	0.875	0.917	0.071	0.017	0.682	0
		ResU-Net	0.8611	0.793	0.858	0.907	0.067	0.018	0.669	0
		LoRA $r{=}32$	0.8877	0.823	0.883	0.918	0.058	0.016	0.649	0
CVC-ClinicDB (in)	62	Plain U-Net	0.8803	0.831	0.892	0.874	0.042	0.016	0.782	0
		ResU-Net	0.8826	0.833	0.886	0.889	0.047	0.027	0.791	1

Table 5: Six metrics by model and test set. DSC, IoU, sensitivity (Sens), precision (Prec) and NSD: higher is better; HD95 and ASSD are divided by the image diagonal (Section 3.6 ): lower is better. NSD at a tolerance of 2/224 of the image width.

\varnothing

: images with an empty prediction. Resolution 224 px in polyp and breast; in skin 392 px for DINOv2 and 384 px for the U-Nets. Task-specific models trained for 300 epochs in polyp and breast and 100 in skin; DINOv2 for 40 epochs in polyp and breast and 20 in skin, single-seed except skin DINOv2, a three-seed mean (its

\varnothing

is the mean per seed). (in) marks in-domain sets; BUSI-val also served for model selection. Asterisks mark skin sets drawn from the same archive as the training data; all CVC-T images also occur in CVC-ColonDB (Section 3.2 ). DSC values are those of the original evaluation, which the re-evaluation that produced the other metrics reproduces to within 0.0007 (Section 3.6 ). Bold marks the best value for each test set.

Figure 4: Mean performance by domain, model and evaluation regime, averaged over each domain’s test sets in that regime. (a) In-domain DSC; (b)–(d) out-of-distribution DSC, HD95 divided by the image diagonal (lower is better) and NSD at 2/224 of the image width. Skin has no in-domain test set. The ordering between the two model families reverses between (a) and (b); (c) follows (b), while (d) does not.

Table 9

			All images			Non-empty predictions
Domain	Test set	Model	HD95 (px)	ASSD (px)	NSD 4	HD95	ASSD	NSD 2
Polyp	Kvasir	Plain U-Net	22.6	5.5	0.794	0.071	0.017	0.682
		ResU-Net	21.3	5.8	0.776	0.067	0.018	0.669
		LoRA $r{=}32$	18.4	5.2	0.821	0.058	0.016	0.649
	CVC-ClinicDB	Plain U-Net	13.2	5.1	0.868	0.042	0.016	0.782
		ResU-Net	14.9	8.5	0.881	0.031	0.011	0.804

Table 8: Boundary metrics for the models of Table 5 , in the units not shown there. HD95 and ASSD in pixels at the evaluated resolution (224 px in polyp and breast; skin 392 px for DINOv2, 384 px for the U-Nets), so pixel values are comparable only within a resolution; NSD at the secondary tolerance of 4/224 of the image width. The last three columns repeat HD95, ASSD (divided by the image diagonal) and NSD at 2/224 over the images on which the model’s prediction is not empty, so that the penalty assigned to empty predictions is visible as the difference from Table 5 . Skin DINOv2 values are three-seed means of per-image values; an image counts as empty for it when its seed-averaged empty indicator is at least 0.5.

Figure 5: Per-image DSC (top) and HD95 divided by the image diagonal (bottom, symmetric-log scale; lower is better) on the eight out-of-distribution sets. Boxes span the interquartile range with the median in white; whiskers extend to 1.5 times the interquartile range; diamonds mark the mean. An empty prediction scores DSC 0 and HD95 1. Models and budgets as in Table 5 ; skin DINOv2 values are per-image means over three seeds.

Model	CVC-ColonDB	ETIS	CVC-T
U-Net [ 52 ]	0.512	0.398	0.710
U-Net++ [ 68 ]	0.483	0.401	0.707
ACSNet [ 67 ]	0.716	0.578	0.863
MSEG [ 29 ]	0.735	0.700	0.874
Polyp-PVT [ 16 ]	0.808	0.787	0.900
SAM-Mamba [ 18 ]	0.853	0.848	0.920

Table 9: Comparison with published results (DSC) on the three out-of-distribution polyp sets under the PraNet protocol [ 21 ] . Rows from this work are single-seed. U-Net, U-Net++, ACSNet, MSEG and Polyp-PVT scores are as reported by Dong et al. [16] ; SAM-Mamba scores are from Dutta et al. [18] . Citations in the model column identify the original methods.

Figure 6: Out-of-distribution cases selected by rule: in each domain, the images at the 90th and 10th percentiles of the per-image difference DSC(DINOv2 + LoRA $r{=}32$ ) $-$ DSC(plain U-Net), pooled over that domain’s out-of-distribution sets. Yellow: reference contour; filled region and outline: model prediction, with its DSC. Images and masks are shown at $256{\times}256$ ; predictions are resampled from 224 px (polyp, breast), 384 px (skin U-Nets) and 392 px (skin DINOv2, seed 42).

Domain	Lesion	Area (%)	$n$	Plain	ResU	LoRA	$\Delta$
Polyp	Small	0.1–2.0	212	0.542	0.531	0.560	$+0.019$
	Medium	2.0–5.0	212	0.819	0.804	0.870	$+0.051$
	Large	5.0–63.1	212	0.785	0.759	0.840	$+0.055$
Breast	Small	0.2–2.8	171	0.547	0.507	0.612	$+0.065$
	Medium	2.8–7.2	171	0.676	0.636	0.740	$+0.064$
	Large	7.2–46.2	170	0.707	0.723	0.775	$+0.051$

Table 11: Out-of-distribution DSC by lesion size. Images are split into terciles of reference lesion area (per cent of the image) within each domain, pooled over its out-of-distribution sets. Plain: plain U-Net; ResU: ResU-Net; LoRA: DINOv2 + LoRA

r{=}32

\Delta

: LoRA minus the better task-specific model. Models and budgets as in Table 5 .

224 px for both models (data-size sweep, 100%)
	DSC		NSD (2/224)		NSD (4/224)		HD95
Test set	L	R	L	R	L	R	L	R
ISIC 2016*	0.920	0.910	0.432	0.459	0.676	0.663	0.031	0.038
ISIC 2017*	0.859	0.856	0.402	0.415	0.581	0.576	0.053	0.054
PH2	0.918	0.917	0.380	0.402	0.591	0.626	0.033	0.038
Native resolution, 392 px (L) and 384 px (R) (Table 5 )

Table 12: Boundary quality in skin at two resolutions: DINOv2 + LoRA

r{=}32

(L) against ResU-Net (R). The 224 px runs are the 100% fraction of the data-size sweep, in which budgets were scaled (LoRA 20 epochs, ResU-Net 80); the native runs are those of Table 5 (LoRA three-seed mean, 20 epochs; ResU-Net 100 epochs), so the comparison changes resolution together with these budgets rather than resolution alone. HD95 divided by the image diagonal. Bold marks the better model in each pair.

Table 16

Figure 7: Out-of-distribution gap between ResU-Net and DINOv2 + LoRA

r{=}32

against training-set size, by domain. Negative values indicate the task-specific model is behind; the shaded band marks the 0.013 measurement floor.

		In-domain gap			Out-of-distribution gap
Domain	Fraction	DSC	HD95	NSD	DSC	HD95	NSD
Polyp	10%	$-0.101$	$+0.036$	$-0.036$	$-0.191$	$+0.054$	$-0.175$
	25%	$-0.050$	$+0.021$	$+0.008$	$-0.135$	$+0.071$	$-0.126$
	50%	$-0.042$	$+0.014$	$+0.049$	$-0.132$	$+0.066$	$-0.096$
	100%	$-0.006$	$+0.005$	$+0.079$	$-0.066$	$+0.042$	$-0.011$
Breast	25%	$-0.004$	$+0.021$	$+0.125$	$-0.104$	$+0.094$	$-0.007$

Table 15: Data-size sweep in three metrics: ResU-Net minus DINOv2 + LoRA

r{=}32

on the identical subset, averaged over each domain’s in-domain and out-of-distribution sets (DSC as in Table 14 ). Positive DSC and NSD gaps and negative HD95 gaps favour training from scratch. Skin at 224 px at every fraction; the 100% rows of polyp and breast are the models of Table 5 .

Test set	LoRA $r{=}32$	Full FT	$\Delta$	$p$	Verdict
Kvasir (in-domain)	0.8825	0.8767	$+0.0058$	0.0003	LoRA
CVC-ClinicDB (in-domain)	0.8485	0.8384	$+0.0101$	0.0194	LoRA
CVC-ColonDB	0.7401	0.7380	$+0.0022$	0.0001	LoRA
ETIS	0.6986	0.6856	$+0.0130$	0.1191	No difference
CVC-T	0.8671	0.8665	$+0.0006$	0.3931	No difference

Table 16: LoRA

r{=}32

against full fine-tuning, polyp, 20 epochs, three seeds. Values are three-seed means (seeds 42, 123 and 2024); the seed-42 LoRA

r{=}32

run alone appears in Table 24 .

\Delta

is LoRA minus full fine-tuning; the verdict applies the criterion of Section 3.6 , not the measurement floor. “No difference” records a failure to reject, not equivalence.

Polyp, LoRA $r{=}32$ $-$ full FT
Test set	DSC	IoU	Sens	Prec	HD95	NSD
Kvasir	$+0.0058^{*}$	$+0.0103^{*}$	$+0.0041$	$+0.0146^{*}$	$-0.0059$	$+0.0451^{*}$
CVC-ClinicDB	$+0.0101^{*}$	$+0.0151^{*}$	$+0.0166^{*}$	$-0.0005$	$-0.0018$	$+0.0399^{*}$
CVC-ColonDB	$+0.0022^{*}$	$+0.0121^{*}$	$+0.0180^{*}$	$-0.0486^{*}$	$+0.0284$	$+0.0264^{*}$
ETIS	$+0.0130$	$+0.0133$	$+0.0416^{*}$	$-0.0147^{*}$	$-0.0028$	$+0.0130$
CVC-T	$+0.0007$	$+0.0040$	$+0.0391^{*}$	$-0.0319^{*}$	$+0.0022$	$+0.0049$

Table 17: LoRA against full fine-tuning in all metrics, three seeds (42, 123, 2024), 20 epochs; polyp as in Table 16 , skin as in Table 21 . Each entry is the mean over images of the seed-averaged per-image difference, LoRA minus full fine-tuning; for HD95 (divided by the image diagonal) a negative value favours LoRA, for all other metrics a positive one. An asterisk marks

p<0.05

in a Wilcoxon signed-rank test on the seed-averaged per-image vectors (Section 3.6 ).

Domain	Test set	Regime	20 epochs	40 epochs
Breast	BrEaST	OOD	$+0.030$	$+0.001$
Breast	BUS-UCLM	OOD	$+0.023$	$+0.014$
Breast	BUSI-val	in-domain	$+0.040$	$+0.038$
Polyp	CVC-ColonDB	OOD	$+0.046$	$+0.055$

Table 18: Effect of doubling the epoch budget on the measured rank effect. Breast rows use

r{=}16-r{=}4

and the polyp row

r{=}32-r{=}4

, in each case that domain’s span at 20 epochs. Single seed.

Figure 8: Capacity axis by domain: DSC against trainable parameter count (logarithmic scale), with the values of Table 19 . Polyp and breast at 40 epochs, skin at 20, all seed 42; from left to right the points are frozen (polyp and breast only), BitFit, LoRA at ranks 4, 8, 16 and 32, and full fine-tuning. Solid lines: in-domain DSC (Kvasir, BUSI-val); dashed lines: mean over the domain’s out-of-distribution sets. The three panels share the vertical axis.

Table 23

Learning rate	Val	Kvasir	ClinicDB	ColonDB	ETIS	CVC-T
1e-2	0.8866	0.8480	0.8359	0.7256	0.6509	0.8558
3e-3	0.8712	0.8503	0.7691	0.7293	0.5689	0.8273
1e-3	0.8958	0.8960	0.8306	0.7204	0.6867	0.8634
3e-4	0.8826	0.8731	0.8158	0.6892	0.6604	0.8587

Table 22: BitFit learning-rate search, polyp, 40 epochs. Validation selects 1e-3. Bold marks the best value in each column.

Domain	Directions at 90%	at 95%	at 99%	Effective rank
Skin	28.4	48.5	113.2	21.3
Breast (20 epochs)	39.0	65.1	139.3	26.1
Polyp	42.1	69.8	146.2	27.0
Breast (40 epochs)	44.3	71.7	145.5	30.7

Table 23: Spectral properties of the full fine-tuning update. The reported quantity is the number of singular directions capturing 90 per cent of the update’s energy. Polyp and skin at 20 epochs.

Test set	Full FT	Trunc $r{=}4$	LoRA $r{=}4$	Trunc $r{=}32$	LoRA $r{=}32$
Kvasir	0.8753	0.8638	0.8609	0.8743	0.8765
CVC-ClinicDB	0.8369	0.7838	0.8166	0.8295	0.8481
CVC-ColonDB	0.7482	0.7244	0.7237	0.7469	0.7700
ETIS	0.6983	0.6610	0.6582	0.6957	0.6961
CVC-T	0.8690	0.8561	0.8701	0.8682	0.8578

Table 24: Truncated full fine-tuning against LoRA, polyp, 20 epochs, seed 42 (single seed). Truncation columns report the update projected onto its leading

r

singular directions; LoRA columns an adapter trained at the same rank. These are 20-epoch checkpoints, distinct from the 40-epoch runs of Table 19 ; the LoRA

r{=}32

run is the seed-42 member of the three-seed mean in Table 16 .

Quantity	Configuration	With CVC-T	Without CVC-T
Mean DSC (Table 5 )	Plain U-Net	0.7370	0.6852
	ResU-Net	0.7120	0.6709
	LoRA $r{=}32$	0.7781	0.7339
Deficit (Table 7 )	Plain U-Net	$-0.041$	$-0.049$
	ResU-Net	$-0.066$	$-0.063$
OOD gap, data-size sweep (Table 14 )	10%	$-0.191$	$-0.178$

Table 25: Polyp out-of-distribution means with and without CVC-T, whose 60 images are pixel-identical to images in CVC-ColonDB (Section 3.2 ). Means are unweighted averages over test sets: CVC-ColonDB, ETIS and CVC-T (with) or CVC-ColonDB and ETIS (without). Deficits and gaps are task-specific model minus DINOv2 + LoRA

r{=}32

. Recomputed from the same checkpoints as the main text; BitFit learning rates from the per-set values of Table 22 .

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 834 papers

Adaptation of Foundation Models in Medical Image Analysis: Mechanisms, Evidence, and Deployment Considerations
Mar 15, 2026Karma Phuntsho, Abdullah, Kyungmi Lee +2Medical Imaging Foundation Models Vision Foundation Model Adaptation
James Cook University, 14-88 McGregor Rd, Smithfield, 4878, Queensland, Australia
MedAdaptAgent: An Automatic Harness for Adapting Medical Imaging Models to New Sites
Oct 26, 2026Hanxue Gu, Zhonghao Liu, Xingrui Gu +2Unsupervised Domain Adaptation Medical Image Segmentation
University of California, San Francisco · University of California, Berkeley
Spectral Rectification for Parameter-Efficient Adaptation of Foundation Models in Colonoscopy Depth Estimation
Mar 16, 2026Xiaoxian Zhang, Minghai Shi, Lei LiMonocular Depth Estimation Vision Foundation Model Adaptation
Department of Biomedical Engineering, National University of Singapore, Singapore · Department of Radiotherapy, The First Affiliated Hospital of Xi’an Jiaotong University, Xi’an, China

Mar 15, 2026J/K move · Enter open · S save

Adaptation of Foundation Models in Medical Image Analysis: Mechanisms, Evidence, and Deployment Considerations

Karma Phuntsho, Abdullah, Kyungmi Lee, Ickjai Lee+1

James Cook University, 14-88 McGregor Rd, Smithfield, 4878, Queensland, Australia

Medical Imaging Foundation Models Vision Foundation Model Adaptation

Open paper AskPDF arXiv

Foundation models, pretrained on diverse data, objectives, or modalities for reuse across downstream tasks, are increasingly applied to medical image analysis. Their performance depends not only on pretraining but also on how they are adapted to specific tasks, data regimes, and deployment settings. Whereas previous reviews have considered adaptation within broader syntheses, this review focuses specifically on post-pretraining interventions applied to existing FMs. Using a structured narrative review with a systematic, auditable workflow, we identified 719 eligible reports representing 705 independent studies. Of these, 473 provided controlled comparative evidence: 322 formally published studies formed the primary synthesis and 151 preprint-only studies a sensitivity layer. Adaptation was organized into five non-mutually exclusive families: parameter-level, representation-level, objective-level, data-centric, and architectural/sequence-level adaptation, with model use and update regime analyzed separately. Controlled comparisons showed no consistent ranking among frozen, parameter-efficient, partial, or full updating, or across broader adaptation families; outcomes varied by task, data regime, comparator, and evaluation setting. Clinical-readiness evidence remained limited: 173/705 studies (24.5%) had confirmed independent external validation, 10 (1.4%) reported calibration or uncertainty evidence, and 97/322 formally published comparative studies (30.1%) included a qualifying externally validated arm. Overall, FM adaptation is a context-dependent design space rather than a hierarchy of preferred strategies.

Abstract

Medical Image Segmentation

Explore similar work

Adaptation of Foundation Models in Medical Image Analysis: Mechanisms, Evidence, and Deployment Considerations

MedAdaptAgent: An Automatic Harness for Adapting Medical Imaging Models to New Sites

Spectral Rectification for Parameter-Efficient Adaptation of Foundation Models in Colonoscopy Depth Estimation