MedAdaptAgent: An Automatic Harness for Adapting Medical Imaging Models to New Sites

Hanxue Gu; Zhonghao Liu; Xingrui Gu; Kang Wang; Yang Yang

cs.CVOct 26, 2026

MedAdaptAgent: An Automatic Harness for Adapting Medical Imaging Models to New Sites

Authors: Hanxue Gu, Zhonghao Liu, Xingrui Gu, Kang Wang, Yang Yang

Organizations: University of California, San Francisco · University of California, Berkeley

Abstract

Reusing pretrained medical-imaging models offers an alternative to training from scratch. A model trained for the same task may perform poorly at a new hospital because imaging protocols and preprocessing differ. Effective adaptation depends on the model architecture, domain shift, and available labels and compute. Adaptation can improve scores on a small labeled set yet reduce performance on unseen data. We present MedAdaptAgent, an automatic harness that turns a supplied model, target data, and a free-text request into a new-site adaptation workflow. Its LLM-driven Adaptation Planner uses model-site evidence to choose compatible experiments and revise subsequent adjustments. The harness requires evidence of improvement over the pretrained model, then delivers the selected model as a complete, runnable package. Evaluation spans 26 segmentation and classification transfers across architectures and compute settings, with segmentation annotation budgets ranging from zero-shot transfer to few-shot adaptation. With 20 local annotations, mean segmentation Dice rises from 0.483 to 0.774 across 10 transfers, a 60.1% relative gain, with no harmful replacements. Mean classification AUROC rises from 0.810 to 0.837 across 13 transfers, a 3.4% relative gain. Comparing adapted models with the pretrained model reduces harmful classification replacements by 30.8% versus selecting the highest-scoring model on the available labels, using identical candidates. Seven public-model deployments demonstrate adaptation and reproducible package delivery across contrast and modality shifts. A CT-to-MRI study extends the evaluation to four LLM planners, including two open-weight models. We release the code on GitHub.

Figures & tables

Figure 1: Automatic new-site model adaptation with the MedAdaptAgent harness. A supplied model and target-hospital request enter the Onboarding Analyzer , which inspects the model-site pair and constructs the feasible action space. The LLM-driven Adaptation Planner iterates planning, tool execution, and evidence updates within deterministic compatibility and budget controls. The Evidence Gate authorizes replacement. The harness then packages the pretrained model or an evidence-supported adapted model, and returns a safe-stop report when no tested option satisfies the deployment objective.

family	examples	labels	GPU?	principal requirement
input matching	histogram, stain, frequency	0	no	compatible input semantics
normalization/TTA	BN reset, Tent, SAR, EATA	0	mixed	supported trainable statistics
frozen probe	linear or per-pixel head	$\geq 5$	yes	recognized output head
model tuning	head, partial, LoRA, surgical	$\geq 5$	yes	training compute

Table 1: Adjustment families exposed to the planner after capability and resource filtering. The GPU requirement is mixed for normalization/TTA because the family contains both forward-only BN reset and gradient-based methods.

Figure 2: Segmentation adaptation across annotation and compute budgets. (a,b) Mean Dice and harmful replacements across 10 U-Net transfers. Teal denotes no supervised target training, warm colors denote training availability, and gray denotes direct deployment. The offset $K{=}0$ markers show BN reset, Tent, SAR, and EATA, from left to right. (c) Target-test Dice across four models on the same RUNMC-to-HK prostate shift. Open markers show pretrained-model performance, including cases where the harness stops without an update; filled markers deploy an adapted model. (d) Selected actions. (e) Per-dataset changes from direct deployment; teal and warm row labels indicate supervised training unavailable and available, respectively. All test scores are measured after model selection is complete.

Policy	Prostate	RIGA	CHAOS	AMOS	BraTS	MMWHS	Fundus	Fundus-ROI	M&Ms-V	M&Ms-F	Mean $\uparrow$	Harm $\downarrow$
Direct deployment	0.792	0.744	0.089	0.723	0.449	0.039	0.000	0.383	0.773	0.842	0.483	0
Fixed BN reset, $K{=}0$	0.635	0.483	0.385	0.539	0.404	0.051	0.000	0.520	0.549	0.822	0.439	6
Fixed Tent, $K{=}0$	0.791	0.796	0.237	0.324	0.131	0.000	0.000	0.707	0.746	0.835	0.457	6
Fixed SAR, $K{=}0$	0.755	0.000	0.224	0.490	0.000	0.113	0.000	0.000	0.506	0.691	0.278	7
Fixed EATA, $K{=}0$	0.756	0.000	0.223	0.490	0.000	0.111	0.000	0.000	0.497	0.688	0.277	7
MedAdaptAgent , $K{=}0$	0.792	0.796	0.089	0.723	0.449	0.039	0.000	0.520	0.773	0.842	0.502	0

Table 2: Per-dataset segmentation Dice across target-label budgets using the same U-Net architecture. Fixed rows apply one adjustment recipe to every transfer; MedAdaptAgent analyzes each model-site pair and selects among compatible actions. Mean Dice uses the pretrained-model score when the harness stops without an update. Best values within each annotation-budget block are bold. M&Ms-V/F denote vendor and field-strength shifts.

replacement rule	mean AUROC $\uparrow$	harmed $\downarrow$	worst $\Delta$ $\uparrow$
highest selection AUROC	$0.827\pm 0.015$	13/52	$-0.343$
MedAdaptAgent	$\boldsymbol{0.839\pm 0.006}$	9/52	$\boldsymbol{-0.184}$

Table 3: Classification selection with 12 labels (18 for nine-class CRC-VAL): 13 transfers over four draws yield 52 decisions. AUROC is mean

\pm

population standard deviation of the four draw-level means. Both rules use identical candidates and labels; the score-based baseline selects the model with the highest AUROC on the labeled selection set ( Xiong et al., 2026 ) . Bold marks the best values.

model	source $\rightarrow$ target	model/site diagnosis	$K$	direct $\uparrow$	delivered $\uparrow$	outcome
U-Net	MSD CE $\rightarrow$ TotalSeg NC	mixed; BN	5	0.386	0.386	retain source
SwinUNETR	BTCV CE $\rightarrow$ TotalSeg NC	mixed; IN/LN	5	0.669	0.705	LoRA
SegResNet	TotalSeg CT $\rightarrow$ MSD CT	intensity; GN	5	0.951	0.951	retain source
U-Net	MSD CT $\rightarrow$ AMOS MRI	modality; BN	5	0.000	0.120	Tent
SwinUNETR	BTCV CT $\rightarrow$ AMOS MRI	modality; IN/LN	5	0.040	0.191	partial fine-tuning
U-Net	MSD CT $\rightarrow$ AMOS MRI	modality; BN	20	0.000	0.233	LoRA

Table 4: Verified deployments from seven public-model configurations. U-Net v0.6.1 ( MONAI Consortium, 2025a ) , SwinUNETR v0.5.8 ( MONAI Consortium, 2025b ) , and SegResNet v0.2.7 are evaluated by Dice on 10 held-out scans per target. Direct and delivered scores refer to the pretrained and packaged models. The diagnosis combines the inferred data shift and normalization layers: BN, BatchNorm; IN/LN, InstanceNorm/LayerNorm; GN, GroupNorm. CE/NC denote contrast-enhanced/non-contrast CT; TotalSeg denotes TotalSegmentator. Each package reproduces all 10 masks after reload.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

policy	mean AUROC $\uparrow$	harmed $\downarrow$	worst $\Delta$ $\uparrow$
source (no adaptation)	0.810	0/13	0.000
MedAdaptAgent ( $K{=}0$ , label-free estimate)	0.809	4/13	$-0.082$
BN reset	0.787	6/11	$-0.124$
Tent	0.779	7/11	$-0.154$
fixed head fine-tuning	0.846	6/13	$-0.067$
fixed full fine-tuning	0.712	10/13	$-0.510$

Appendix

Table 5: Classification with 12 labels (18 for nine-class CRC-VAL), plus a zero-label control. BN reset and Tent cover 11 transfers (pretrained-model mean 0.802 on that subset); all other rows cover 13. Harmed counts transfers with lower test AUROC after adaptation; worst

\Delta

is the largest decrease relative to the pretrained model.

selection rule	decision value	replacements	harmful / replaced
retain source	0.483	0/10	n/a
Minimum-entropy selector	0.460	9/10	5/9
scalar proxy $q$	0.460	8/10	4/8
VLM preference only	0.496	3/10	1/3
full proxy consensus	0.502	2/10	0/2

Appendix

Table 6: Zero-label selector ablation on the 10 primary U-Net transfers. The entropy baseline selects the lowest-entropy model from the same pretrained and TTA models available to the harness. Test labels are withheld from all selection rules. Decision value averages test Dice, using the pretrained-model score when no replacement is made.

configuration	normalization	BN reset $\Delta$	Tent $\Delta$	interpretation
ViT-small classifier	LayerNorm	0.000	0.000	no BN layers
Swin-Tiny classifier	LayerNorm	0.000	0.000	no BN layers
SegFormer 2-D	LayerNorm encoder + 1 BN head	$+0.033$	$-0.576$	method-dependent benefit
SwinUNETR 3-D ( Hatamizadeh et al., 2022 )	InstanceNorm + LayerNorm	0.000	0.000	no BN layers
SegResNet 3-D	GroupNorm	0.000	0.000	no BN layers
3-D U-Net, TotalSeg CE	17 BatchNorm3d layers	$-0.373$	$-0.114$	both reduce Dice

Appendix

Table 7: Architecture dependence of BN-based adaptation. Deltas are changes from the pretrained model in AUROC (classification) or Dice (segmentation), evaluated on the full target set. Exact zeros identify actions that leave the model unchanged.

model and evaluation set	AUROC	$\Delta$ vs. target source
source, in-domain RUMC ( $n{=}159$ )	0.708	n/a
source, target PCNN ( $n{=}350$ )	0.479	0.000
BN reset	0.467	$-0.012$
Tent	0.508	$+0.029$
head fine-tuning ( $K{=}12$ )	0.477	n/a
partial fine-tuning ( $K{=}12$ )	0.501	n/a

Appendix

Table 8: Recoverability stress test for cross-center biparametric-MRI classification of clinically significant prostate cancer (csPCa). Fine-tuned models are evaluated on the 338 target cases outside the twelve-label adaptation set; other target rows use all 350 PCNN cases. Each reported delta uses a matched evaluation set.

planner	selected adjustment	test Dice	reload agreement
GPT-5.2	LoRA	0.230	10/10
GPT-5.6 Luna	LoRA	0.196	10/10
Qwen3-VL-4B-Instruct	LoRA	0.215	10/10
Qwen3-VL-8B-Instruct	partial fine-tuning	0.205	10/10

Appendix

Table 9: Planner comparison on one public-model CT-to-MRI transfer with twenty annotations. Dice is measured against expert annotations on the same ten test scans. The pretrained model has zero Dice. Reload agreement counts identical prediction masks.

transfer	architecture	dominant shift	source	output	selected action
AMOS CT $\rightarrow$ MRI multiorgan	U-Net	modality	0.723	0.928	partial fine-tuning
BraTS T1 $\rightarrow$ FLAIR tumor	U-Net	sequence	0.449	0.871	partial fine-tuning
CHAOS CT $\rightarrow$ MRI liver	U-Net	modality	0.089	0.662	partial fine-tuning
REFUGE full-FOV $\rightarrow$ RIM-ONE	U-Net	scale/preproc.	0.000	0.657	final-layer probe
REFUGE ROI $\rightarrow$ RIM-ONE	U-Net	intensity/scale	0.383	0.715	partial fine-tuning
MMWHS CT whole heart	U-Net	contrast	0.039	0.633	partial fine-tuning

Appendix

Table 10: Per-transfer segmentation Dice at

K{=}20

. Fundus rows distinguish full-field-of-view (FOV) and region-of-interest (ROI) preprocessing; the RIM-ONE target is disc-centered in both cases.

transfer	architecture	diagnosed shift	source	output	action
APTOS $\rightarrow$ DDR	ResNet18	mixed	0.827	0.830	head fine-tuning
APTOS $\rightarrow$ EyePACS	ResNet18	mixed	0.716	0.704	head fine-tuning
APTOS $\rightarrow$ Messidor-2	ResNet18	mixed	0.730	0.623	partial fine-tuning
BreakHis 40 $\times\rightarrow$ 400 $\times$	ResNet18	resolution	0.801	0.866	head fine-tuning
CAMELYON17 cross-hospital	ResNet18	intensity	0.931	0.931	retain source model
CAMELYON17 cross-hospital	Swin-Tiny	intensity	0.981	0.997	source histogram match

Appendix

Table 11: Per-transfer classification AUROC for the annotation draw summarized in Table 5 , using 12 labels (18 for nine-class CRC-VAL).

Unsupervised Domain Adaptation

Unsupervised domain adaptation transfers models from labeled source data to unlabeled target domains with different data distributions.

See all 183 papers

Feb 22, 2026

SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation

Medical image segmentation is vital for clinical diagnosis and quantitative analysis, yet remains challenging due to the heterogeneity of imaging modalities and the high cost of pixel-level annotations. Although general interactive segmentation models like SAM have achieved remarkable progress, their transfer to medical imaging still faces two key bottlenecks: (i) the lack of adaptive mechanisms for modality- and anatomy-specific tasks, which limits generalization in out-of-distribution medical scenarios; and (ii) current medical adaptation methods fine-tune on large, heterogeneous datasets without selection, leading to noisy supervision, higher cost, and negative transfer. To address these issues, we propose SegMoTE, an efficient and adaptive framework for medical image segmentation. SegMoTE preserves SAM's original prompt interface, efficient inference, and zero-shot generalization while introducing only a small number of learnable parameters to dynamically adapt across modalities and tasks. In addition, we design a progressive prompt tokenization mechanism that enables fully automatic segmentation, significantly reducing annotation dependence. Trained on MedSeg-HQ, a curated dataset less than 1% of existing large-scale datasets, SegMoTE achieves SOTA performance across diverse imaging modalities and anatomical tasks. It represents the first efficient, robust, and scalable adaptation of general segmentation models to the medical domain under extremely low annotation cost, advancing the practical deployment of foundation vision models in clinical applications.

Yujie Lu, Jingwen Li, Sibo Ju +5

Sichuan University · Xinjiang University · Fuzhou University +1

Medical Image Segmentation Vision Foundation Model Adaptation Promptable Image Segmentation Mixture of Experts

Ask PDF

Jul 20, 2026

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation

Test-time adaptation (TTA) aims to mitigate distribution shifts by adapting models with unlabeled target data at inference time. While TTA with vision-language models (VLMs) has shown promising results in classification, extending it to medical image segmentation remains challenging. In this setting, the adaptation gains from optimizing on VLM-generated predictions are often outweighed by the degradation to the VLM's strong pretrained features caused by noisy, update-driven learning, resulting in limited and unstable improvements. We therefore propose Memory-Supported Synergistic Adaptation (MSSA), a novel training-free TTA framework for medical image segmentation. Without updating model parameters, MSSA dynamically selects reliable image-text predictions to construct an online memory, uses them as text-guided semantic priors, and couples them with cross-image structural alignment for robust adaptation. Specifically, MSSA consists of (i) a noise-aware memory construction module that filters and stabilizes cross-modal predictions, and (ii) a relevance-driven prototype alignment module that aligns the target sample with structurally consistent memory samples and their reliable predictions to improve adaptation. Extensive experiments on multiple medical segmentation benchmarks demonstrate that MSSA consistently improves VLM-based segmentation models and outperforms existing fine-tuning-based TTA methods by a clear margin, with gains of up to 12.2% DSC and 11.7% mIoU. Project page: https://lingrayy.github.io/MSSA/ .

Lingrui Li, Nan Pu, Dong Zhao +4

School of Computer Science, University of Nottingham, UK · School of Computer Science and Information Engineering, Hefei University of Technology, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore

Memory-Augmented VLMs VLM Adaptation Text-Guided Image Segmentation Medical Image Segmentation

Ask PDF

Mar 15, 2026

Adaptation of Foundation Models in Medical Image Analysis: Mechanisms, Evidence, and Deployment Considerations

Foundation models, pretrained on diverse data, objectives, or modalities for reuse across downstream tasks, are increasingly applied to medical image analysis. Their performance depends not only on pretraining but also on how they are adapted to specific tasks, data regimes, and deployment settings. Whereas previous reviews have considered adaptation within broader syntheses, this review focuses specifically on post-pretraining interventions applied to existing FMs. Using a structured narrative review with a systematic, auditable workflow, we identified 719 eligible reports representing 705 independent studies. Of these, 473 provided controlled comparative evidence: 322 formally published studies formed the primary synthesis and 151 preprint-only studies a sensitivity layer. Adaptation was organized into five non-mutually exclusive families: parameter-level, representation-level, objective-level, data-centric, and architectural/sequence-level adaptation, with model use and update regime analyzed separately. Controlled comparisons showed no consistent ranking among frozen, parameter-efficient, partial, or full updating, or across broader adaptation families; outcomes varied by task, data regime, comparator, and evaluation setting. Clinical-readiness evidence remained limited: 173/705 studies (24.5%) had confirmed independent external validation, 10 (1.4%) reported calibration or uncertainty evidence, and 97/322 formally published comparative studies (30.1%) included a qualifying externally validated arm. Overall, FM adaptation is a context-dependent design space rather than a hierarchy of preferred strategies.

Karma Phuntsho, Abdullah, Kyungmi Lee +2

James Cook University, 14-88 McGregor Rd, Smithfield, 4878, Queensland, Australia

Medical Imaging Foundation Models Vision Foundation Model Adaptation

Ask PDF

Abstract

Unsupervised Domain Adaptation

Explore similar work

SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation

Adaptation of Foundation Models in Medical Image Analysis: Mechanisms, Evidence, and Deployment Considerations