When Online Adaptation Hurts: Parameter-Frozen Test-Time Ensembling for Continual Medical Image Segmentation

Ruijie Huang

cs.CVSep 18, 2026

When Online Adaptation Hurts: Parameter-Frozen Test-Time Ensembling for Continual Medical Image Segmentation

Authors: Ruijie Huang

Organizations: Hong Kong Baptist University, Faculty of Science and Technology

Abstract

Medical image segmenters often get worse when sites, scanner vendors, or protocols change. Continual test-time adaptation (CTTA) addresses this problem without target labels, but it can be impossible to update a model on a non-stationary stream and can lead to a lot of errors. We examine a more reasonable and meaningful alternative: parameter-frozen inference enhancement(PIE). We use a source-trained segmenter that learns about anatomy-preserving scale and flip views, maps their predictions back to the native location, and averages the probabilities. We do not modify the weights of the model or the normalization statistics. On a cardiac MRI stream from M&Ms, which is trained on vendor A and evaluated sequentially on vendors B, C, and D, PIE has 0.7786 mean Dice, compared to 0.7680 for source-only inference and 0.7388--0.7416 for five other online-adaptation baselines. The controlled ablations show that performance saturates at 28 views, and confidence weighting, class-prior correction, connected-component filtering, morphological refinement, and inter-slice smoothing have no effect or cause negative transfer. Qualitative results on cardiac MRI and fundus images are also consistent with the frozen ensemble keeping thinner and nested anatomical structures. These results provide a strong, stable baseline for medical CTTA and expose an important failure mode: adaptation and handcrafted refinement can be less reliable than carefully designed inference.

Figures & tables

	Per-Domain Dice $\uparrow$			Per-Class Dice $\uparrow$			Avg $\uparrow$
Method	B (Philips)	C (GE)	D (Canon)	LV	MYO	RV
Ours (PIE)	0.8025	0.7584	0.7395	0.8439	0.7575	0.7343	0.7786
Source-Only	0.7914	0.7505	0.7274	0.8293	0.7438	0.7309	0.7680
SAR (ICLR’23)	0.7334	0.7423	0.7609	0.8046	0.7082	0.7119	0.7416
EATA (ICML’22)	0.7340	0.7423	0.7565	0.8042	0.7085	0.7101	0.7409
TENT (ICLR’21)	0.7318	0.7424	0.7607	0.8035	0.7084	0.7100	0.7406

Table 1: Continual Test-Time Adaptation on the M&Ms Cardiac Segmentation Benchmark. Model is trained on Vendor A (Siemens) and continually adapts to Vendors B

\rightarrow

\rightarrow

D. All numbers are Dice

\uparrow

. Bold indicates the best and underlined the second best. Our PIE method (highlighted) is a parameter-frozen inference-time enhancement that provably lower-bounds Source-Only.

	Per-Domain Dice $\uparrow$			Per-Class Dice $\uparrow$			Avg $\uparrow$	$\Delta$
Method	B (Philips)	C (GE)	D (Canon)	LV	MYO	RV
Source-Only (baseline)	0.7914	0.7505	0.7274	0.8293	0.7438	0.7309	0.7680	$-0.0136$
+ A1 (Multi-Scale TTA) *	0.8056	0.7607	0.7429	0.8471	0.7604	0.7374	0.7816	—
+ A2 (Confidence Weight)	0.8048	0.7589	0.7412	0.8455	0.7590	0.7367	0.7804	$-0.0012$
+ A3 (Class-Prior Reweight) $\dagger$	0.7805	0.7394	0.7699	0.8511	0.7164	0.7399	0.7691	$-0.0125$
+ A4 (Largest-CC)	0.8053	0.7600	0.7419	0.8470	0.7595	0.7367	0.7811	$-0.0005$

Table 2: Level 1: Component ablation on the M&Ms benchmark. Starting from the source-only baseline, each row cumulatively adds one inference-time component to the Multi-Scale TTA (A1) core.

\Delta

denotes the change of Avg Dice relative to the Full model (i.e., “+A1”) on target vendors B/C/D. Positive

\Delta

= the component helps; negative

\Delta

= the component actually hurts and is therefore excluded from our final design. All numbers are Dice

\uparrow

	Per-Domain Dice $\uparrow$			Per-Class Dice $\uparrow$			Avg $\uparrow$	$\Delta$
Method	B (Philips)	C (GE)	D (Canon)	LV	MYO	RV
TTA 1-view (identity)	0.7914	0.7505	0.7274	0.8293	0.7438	0.7309	0.7680	$-0.0136$
TTA 4-view (flips only)	0.8004	0.7552	0.7249	0.8363	0.7540	0.7301	0.7734	$-0.0082$
TTA 12-view (0.9–1.1)	0.8026	0.7590	0.7402	0.8439	0.7580	0.7350	0.7789	$-0.0027$
TTA 20-view (0.8–1.2)	0.8044	0.7610	0.7425	0.8460	0.7593	0.7375	0.7809	$-0.0007$
TTA 28-view (0.7–1.3) [Full]	0.8056	0.7607	0.7429	0.8471	0.7604	0.7374	0.7816	—

Table 3: Level 2: Sensitivity analysis on the number of TTA views. We vary the number of augmentation views by adjusting the range of scaling factors. All configurations include horizontal and vertical flips. Performance monotonically increases with the number of views and peaks at 28 views, beyond which further scaling (36 views) yields no improvement while incurring 28% additional inference cost. This justifies our choice of 28 views (7 scales

\times

4 flips) for the final model.

	Per-Domain Dice $\uparrow$			Per-Class Dice $\uparrow$			Avg $\uparrow$	$\Delta$
Method	B (Philips)	C (GE)	D (Canon)	LV	MYO	RV
No TTA (identity)	0.7914	0.7505	0.7274	0.8293	0.7438	0.7309	0.7680	$-0.0163$
Multi-Scale only	0.7983	0.7572	0.7459	0.8388	0.7517	0.7418	0.7774	$-0.0069$
Flips only	0.8004	0.7552	0.7249	0.8363	0.7540	0.7301	0.7734	$-0.0109$
Multi-Scale + Flips	0.8056	0.7607	0.7429	0.8471	0.7604	0.7374	0.7816	$-0.0027$
Multi-Scale + Flips + Rot90 [Full]	0.8063	0.7638	0.7501	0.8533	0.7681	0.7316	0.7843	—

Table 4: Level 3: Ablation of TTA transformation types. Starting from a plain (No TTA) baseline, we cumulatively enable multi-scale resizing (7 scales), horizontal/vertical flips, and

90^{\circ}

rotations. Each transformation family contributes complementary information; the full combination (Multi-Scale + Flips + Rot90) yields the best performance, supporting the design of our final model.

Figure 1: Qualitative M&Ms results on vendor B (Philips). Columns show input, ground truth, anatomy-prior visualization, PIE prediction, and error map. Red, yellow, and blue denote LV, MYO, and RV, respectively.

Figure 2: Qualitative REFUGE2 results. Predictions preserve the optic disc/cup nesting under changes in illumination and appearance. Yellow and red denote optic disc and optic cup.

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 788 papers

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation
Jul 20, 2026Lingrui Li, Nan Pu, Dong Zhao +4Memory-Augmented VLMs VLM Adaptation
School of Computer Science, University of Nottingham, UK · School of Computer Science and Information Engineering, Hefei University of Technology, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore
MedAdaptAgent: An Automatic Harness for Adapting Medical Imaging Models to New Sites
Oct 26, 2026Hanxue Gu, Zhonghao Liu, Xingrui Gu +2Unsupervised Domain Adaptation Medical Image Segmentation
University of California, San Francisco · University of California, Berkeley
The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging
Feb 24, 2026Sameer Ambekar, Reza Nasirigerdeh, Peter J. Schuffler +3Model Merging Domain Generalization
School of Computation, Information and Technology, Technical University of Munich, Germany · Institute of Machine Learning in Biomedical Imaging, Helmholtz Munich, Germany · relAI – Konrad Zuse School of Excellence in Reliable AI +3

Jul 20, 2026J/K move · Enter open · S save

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation

Lingrui Li, Nan Pu, Dong Zhao, Wenjing Li+3

School of Computer Science, University of Nottingham, UK · School of Computer Science and Information Engineering, Hefei University of Technology, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore

Memory-Augmented VLMs VLM Adaptation Text-Guided Image Segmentation Medical Image Segmentation

Open paper AskPDF arXiv

Test-time adaptation (TTA) aims to mitigate distribution shifts by adapting models with unlabeled target data at inference time. While TTA with vision-language models (VLMs) has shown promising results in classification, extending it to medical image segmentation remains challenging. In this setting, the adaptation gains from optimizing on VLM-generated predictions are often outweighed by the degradation to the VLM's strong pretrained features caused by noisy, update-driven learning, resulting in limited and unstable improvements. We therefore propose Memory-Supported Synergistic Adaptation (MSSA), a novel training-free TTA framework for medical image segmentation. Without updating model parameters, MSSA dynamically selects reliable image-text predictions to construct an online memory, uses them as text-guided semantic priors, and couples them with cross-image structural alignment for robust adaptation. Specifically, MSSA consists of (i) a noise-aware memory construction module that filters and stabilizes cross-modal predictions, and (ii) a relevance-driven prototype alignment module that aligns the target sample with structurally consistent memory samples and their reliable predictions to improve adaptation. Extensive experiments on multiple medical segmentation benchmarks demonstrate that MSSA consistently improves VLM-based segmentation models and outperforms existing fine-tuning-based TTA methods by a clear margin, with gains of up to 12.2% DSC and 11.7% mIoU. Project page: https://lingrayy.github.io/MSSA/ .

Abstract

Medical Image Segmentation

Explore similar work

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation

MedAdaptAgent: An Automatic Harness for Adapting Medical Imaging Models to New Sites

The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging