SPEGC: Continual Test-Time Adaptation via Semantic-Prompt-Enhanced Graph Clustering for Medical Image Segmentation

Xiaogang Du; Jiawei Zhang; Tongfei Liu; Tao Lei; Yingbo Wang

cs.CVMar 12, 2026

SPEGC: Continual Test-Time Adaptation via Semantic-Prompt-Enhanced Graph Clustering for Medical Image Segmentation

Authors: Xiaogang Du, Jiawei Zhang, Tongfei Liu, Tao Lei, Yingbo Wang

Organizations: Shaanxi Joint Laboratory of Artificial Intelligence, Shaanxi University of Science and Technology. Xi’an, China

Abstract

In medical image segmentation tasks, the domain gap caused by the difference in data collection between training and testing data seriously hinders the deployment of pre-trained models in clinical practice. Continual Test-Time Adaptation (CTTA) aims to enable pre-trained models to adapt to continuously changing unlabeled domains, providing an effective approach to solving this problem. However, existing CTTA methods often rely on unreliable supervisory signals, igniting a self-reinforcing cycle of error accumulation that culminates in catastrophic performance degradation. To overcome these challenges, we propose a CTTA via Semantic-Prompt-Enhanced Graph Clustering (SPEGC) for medical image segmentation. First, we design a semantic prompt feature enhancement mechanism that utilizes decoupled commonality and heterogeneity prompt pools to inject global contextual information into local features, alleviating their susceptibility to noise interference under domain shift. Second, based on these enhanced features, we design a differentiable graph clustering solver. This solver reframes global edge sparsification as an optimal transport problem, allowing it to distill a raw similarity matrix into a refined and high-order structural representation in an end-to-end manner. Finally, this robust structural representation is used to guide model adaptation, ensuring predictions are consistent at a cluster-level and dynamically adjusting decision boundaries. Extensive experiments demonstrate that SPEGC outperforms other state-of-the-art CTTA methods on two medical image segmentation benchmarks. The source code is available at https://github.com/Jwei-Z/SPEGC-for-MIS.

Figures & tables

Figure 1: Conceptual comparison of different paradigms in CTTA. Compared to (A) Prompt adaptation only and (B) Source Prototype Alignment, (C) our SPEGC innovatively leverages graph clustering to extract structural information, achieving ”Good” performance across Performance (Perf), Generalization (Gen), Error Accumulation (EA), and Catastrophic Forgetting (CF), while maintaining an acceptable Computational Complexity (CC).

Figure 2: Overview of the SPEGC. For the continual target domains stream, we first extract local node features ( $V$ ) and construct a Pseudo mini-batch via a feature queue (Enqueue/Dequeue). Subsequently, the Semantic Prompt Feature Enhancement (SPFE, Sec. 3.2 ) module utilizes Attention Pooling to generate a Query, retrieving information from the decoupled commonality ( $P_{CO}$ ) and heterogeneity ( $P_{HE}$ ) prompt pools to generate the enhanced batch features $V^{\star}$ , thereby mitigating noisy information from the domain shift. Based on these features, the Differentiable Graph Clustering Solver (DGCS, Sec. 3.3 ) computes the initial edge similarity matrix $S^{\prime}$ and refines it end-to-end into $S^{\star}$ . Finally, the segmentation network is fine-tuned via backpropagation to achieve efficient adaptation ( Sec. 3.4 ).

Methods	Domain A			Domain B			Domain C			Domain D			Domain E			Average
Methods	DSC	$E_{\phi}^{\max}$	$S_{\alpha}$	DSC	$E_{\phi}^{\max}$	$S_{\alpha}$	DSC	$E_{\phi}^{\max}$	$S_{\alpha}$	DSC	$E_{\phi}^{\max}$	$S_{\alpha}$	DSC	$E_{\phi}^{\max}$	$S_{\alpha}$	DSC $\uparrow$	$E_{\text{max}}^{b}\uparrow$	$S_{\alpha}\uparrow$
No Adapt (ResUNet-50) [ 10 ]	71.92	88.67	80.84	79.31	90.42	84.82	75.42	89.24	81.62	63.77	85.49	78.28	73.32	89.67	81.81	72.75	88.70	81.47
SAR(ICLR’23) [ 46 ]	74.23 $\pm$ 6.13	90.17 $\pm$ 0.34	83.59 $\pm$ 0.19	80.23 $\pm$ 2.54	93.06 $\pm$ 0.43	85.89 $\pm$ 0.17	72.22 $\pm$ 4.97	92.15 $\pm$ 0.16	83.41 $\pm$ 0.18	70.19 $\pm$ 1.22	86.31 $\pm$ 0.17	79.47 $\pm$ 0.78	70.35 $\pm$ 5.31	88.53 $\pm$ 0.87	86.70 $\pm$ 0.30	73.44	90.14	83.81
Domain Adaptor(CVPR’23) [ 69 ]	76.98 $\pm$ 3.87	90.75 $\pm$ 0.40	84.12 $\pm$ 0.08	76.57 $\pm$ 3.98	91.73 $\pm$ 0.27	86.01 $\pm$ 0.09	70.81 $\pm$ 7.50	91.21 $\pm$ 0.20	83.71 $\pm$ 0.12	66.98 $\pm$ 9.02	84.59 $\pm$ 0.27	79.01 $\pm$ 0.06	77.21 $\pm$ 4.02	89.11 $\pm$ 0.15	85.02 $\pm$ 0.03	73.71	89.48	83.57
NC-TTT(CVPR’24) [ 48 ]	77.21 $\pm$ 2.52	93.99 $\pm$ 0.23	85.69 $\pm$ 0.13	83.19 $\pm$ 3.21	93.57 $\pm$ 0.21	86.94 $\pm$ 0.07	78.73 $\pm$ 5.07	92.46 $\pm$ 0.16	84.22 $\pm$ 0.10	75.31 $\pm$ 3.28	87.94 $\pm$ 0.24	81.34 $\pm$ 0.19	81.73 $\pm$ 0.19	92.81 $\pm$ 0.11	85.71 $\pm$ 0.12	79.23	92.15	84.78
VPTTA(CVPR’24) [ 5 ]	75.57 $\pm$ 4.14	92.64 $\pm$ 0.04	85.71 $\pm$ 0.01	79.42 $\pm$ 3.87	92.12 $\pm$ 0.05	85.14 $\pm$ 0.02	71.69 $\pm$ 2.48	92.97 $\pm$ 0.06	83.02 $\pm$ 0.02	64.48 $\pm$ 5.61	86.14 $\pm$ 0.11	77.82 $\pm$ 0.09	75.83 $\pm$ 2.59	90.81 $\pm$ 0.10	85.15 $\pm$ 0.08	73.40	91.34	83.77

Table 1: Average performance (Mean ± Std.) of our SPEGC, ”No adapt”, and six SOTA methods on the OD/OC segmentation task, based on five experimental runs. ”Domain A” denotes training on Domain A and testing on Domain B-E, and so forth for the other domains. The best results are highlighted in bold red .

Methods	Domain A			Domain B			Domain C			Domain D			Average
Methods	DSC	$E_{\text{max}}^{b}$	$S_{\alpha}$	DSC	$E_{\text{max}}^{b}$	$S_{\alpha}$	DSC	$E_{\text{max}}^{b}$	$S_{\alpha}$	DSC	$E_{\text{max}}^{b}$	$S_{\alpha}$	DSC $\uparrow$	$E_{\text{max}}^{b}\uparrow$	$S_{\alpha}\uparrow$
No Adapt (ResUNet-50) [ 10 ]	70.32	86.82	82.14	68.33	85.33	81.32	70.48	87.38	81.92	76.81	87.19	84.14	71.49	86.68	82.38
SAR(ICLR’23) [ 46 ]	69.33 $\pm$ 0.87	86.08 $\pm$ 0.16	80.72 $\pm$ 0.10	68.34 $\pm$ 1.94	84.01 $\pm$ 0.14	81.74 $\pm$ 0.07	70.91 $\pm$ 1.02	88.41 $\pm$ 0.15	82.42 $\pm$ 0.07	68.27 $\pm$ 2.94	84.51 $\pm$ 0.10	79.82 $\pm$ 0.04	69.21	85.75	81.17
Domain Adaptor(CVPR’23) [ 69 ]	78.53 $\pm$ 0.93	92.07 $\pm$ 0.04	86.99 $\pm$ 0.09	71.13 $\pm$ 1.54	89.56 $\pm$ 0.03	82.59 $\pm$ 0.07	71.77 $\pm$ 1.98	92.13 $\pm$ 0.07	83.11 $\pm$ 0.02	69.35 $\pm$ 1.31	84.13 $\pm$ 0.07	79.70 $\pm$ 0.03	72.70	89.47	83.10
NC-TTT(CVPR’24) [ 48 ]	77.92 $\pm$ 1.05	91.74 $\pm$ 0.12	86.69 $\pm$ 0.08	73.77 $\pm$ 2.06	91.72 $\pm$ 0.02	83.48 $\pm$ 0.06	69.14 $\pm$ 0.91	90.09 $\pm$ 0.21	81.97 $\pm$ 0.07	80.94 $\pm$ 1.12	89.67 $\pm$ 0.09	84.51 $\pm$ 0.07	75.44	90.81	84.16
VPTTA(CVPR’24) [ 5 ]	77.68 $\pm$ 0.51	92.61 $\pm$ 0.05	87.01 $\pm$ 0.07	71.76 $\pm$ 0.62	88.82 $\pm$ 0.04	82.14 $\pm$ 0.09	71.68 $\pm$ 0.58	91.98 $\pm$ 0.04	82.17 $\pm$ 0.07	80.77 $\pm$ 0.24	90.17 $\pm$ 0.09	84.71 $\pm$ 0.10	73.40	90.90	84.01

Table 2: Average performance (Mean ± Std.) of our SPEGC, ”No adapt”, and six SOTA methods on the polyp segmentation task, based on five experimental runs. ”Domain A” denotes training on Domain A and testing on Domain B-D, and so forth for the other domains. The best results are highlighted in bold red .

Panel of Table 3 — Table 3: Performance of our SPEGC and five competing methods on the OD/OC segmentation task under Long-term Continual Test-Time Adaptation. Numbers in bold red indicate the performance gain relative to the ”No Adapt” baseline. Performance degradation is calculated as the difference between the overall average DSC and the first-round average DSC. ”Ave”: Abbreviation for ”Average”.

Methods				Domain A	Domain B	Domain C	Domain D	Domain E	Average
Clustering	MC Dropout	CO-Prompt+ $L_{C}$	HE-Prompt	DSC $\uparrow$	DSC $\uparrow$	DSC $\uparrow$	DSC $\uparrow$	DSC $\uparrow$	DSC $\uparrow$
				71.92	79.31	75.42	63.77	73.32	72.75
✓				73.49	80.94	78.09	64.47	76.19	74.64
✓	✓			74.02	82.27	77.58	68.91	79.82	76.52
✓	✓		✓	74.14	80.21	77.22	67.31	78.08	75.39
✓	✓	✓		77.09	84.24	82.50	78.32	83.18	81.07

Table 4: Ablation study results on the OD/OC segmentation task. The best in each column is marked in bold .

Figure 3: t-SNE [ 40 ] visualization of the embeddings learned by D-Prompt and SEC-Prompt on the OD/OC task. Different colors represent images from different domains.

Figure 4: Performance of SPEGC with various Z on the OD/OC segmentation task.

Figure 5: Performance of our SPEGC with various M on the OD/OC segmentation task.

Feature Pool	Avg. DSC $\uparrow$	FLOPs(G) $\downarrow$	Time(s/img) $\downarrow$	VRAM(MB) $\downarrow$
1	79.21	2.917	0.314	2798
3	84.37	5.817	0.521	3276
7	85.24	21.72	1.471	4817
15	80.14	120.3	4.774	8738

Table 5: Ablation study of feature pool size on the OD/OC task. ”Avg. DSC” denotes the average DSC across the five sites, ”time” indicates the inference time per image, and ”VRAM” represents the GPU memory usage.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	Domain A			Domain B			Domain C			Domain D			Domain E			Average
Methods	DSC	$E_{\phi}^{\max}$	$S_{\alpha}$	DSC	$E_{\phi}^{\max}$	$S_{\alpha}$	DSC	$E_{\phi}^{\max}$	$S_{\alpha}$	DSC	$E_{\phi}^{\max}$	$S_{\alpha}$	DSC	$E_{\phi}^{\max}$	$S_{\alpha}$	DSC $\uparrow$	$E_{\text{max}}^{b}\uparrow$	$S_{\alpha}\uparrow$
No Adapt (ResUNet-50) [ 10 ]	71.92	88.67	80.84	79.31	90.42	84.82	75.42	89.24	81.62	63.77	85.49	78.28	73.32	89.67	81.81	72.75	88.70	81.47
SAR(ICLR’23) [ 46 ]	74.03 $\pm$ 6.43	90.07 $\pm$ 0.31	83.52 $\pm$ 0.17	80.33 $\pm$ 2.42	92.86 $\pm$ 0.31	85.29 $\pm$ 0.19	71.42 $\pm$ 4.67	91.55 $\pm$ 0.12	83.01 $\pm$ 0.16	69.89 $\pm$ 1.32	86.01 $\pm$ 0.14	79.17 $\pm$ 0.98	69.75 $\pm$ 5.61	88.03 $\pm$ 0.97	86.12 $\pm$ 0.47	73.08	89.70	83.42
Domain Adaptor(CVPR’23) [ 69 ]	76.28 $\pm$ 4.47	90.75 $\pm$ 0.40	83.82 $\pm$ 0.14	76.27 $\pm$ 4.28	91.03 $\pm$ 0.37	85.71 $\pm$ 0.14	70.21 $\pm$ 8.12	91.01 $\pm$ 0.31	82.44 $\pm$ 0.24	66.18 $\pm$ 9.82	83.41 $\pm$ 0.42	78.34 $\pm$ 0.17	76.31 $\pm$ 4.42	88.71 $\pm$ 0.21	84.62 $\pm$ 0.08	73.05	88.98	82.99
NC-TTT(CVPR’24) [ 48 ]	76.81 $\pm$ 2.12	92.79 $\pm$ 0.23	85.69 $\pm$ 0.13	82.74 $\pm$ 3.48	93.47 $\pm$ 0.34	86.71 $\pm$ 0.10	77.93 $\pm$ 6.07	92.26 $\pm$ 0.21	84.02 $\pm$ 0.14	75.05 $\pm$ 4.12	87.74 $\pm$ 0.31	81.54 $\pm$ 0.39	81.23 $\pm$ 0.39	92.61 $\pm$ 0.14	85.01 $\pm$ 0.12	78.75	91.77	84.59
VPTTA(CVPR’24) [ 5 ]	75.17 $\pm$ 5.01	92.44 $\pm$ 0.10	85.71 $\pm$ 0.01	79.02 $\pm$ 3.94	92.01 $\pm$ 0.07	84.84 $\pm$ 0.06	71.41 $\pm$ 2.56	92.24 $\pm$ 0.10	82.79 $\pm$ 0.06	64.02 $\pm$ 6.02	85.89 $\pm$ 0.14	77.61 $\pm$ 0.11	75.73 $\pm$ 3.41	90.72 $\pm$ 0.14	84.41 $\pm$ 0.11	73.07	90.66	83.08

Appendix

Table A.1 : Quantitative comparison under Mixed Distribution Shifts on the OD/OC segmentation task. Models are trained on the indicated source domain and adapted to a composite target stream formed by shuffling samples from all remaining domains. We report the Mean

\pm

Std. over five independent runs. The best results are highlighted in bold red .

Methods	Domain A			Domain B			Domain C			Domain D			Average
Methods	DSC	$E_{\text{max}}^{b}$	$S_{\alpha}$	DSC	$E_{\text{max}}^{b}$	$S_{\alpha}$	DSC	$E_{\text{max}}^{b}$	$S_{\alpha}$	DSC	$E_{\text{max}}^{b}$	$S_{\alpha}$	DSC $\uparrow$	$E_{\text{max}}^{b}\uparrow$	$S_{\alpha}\uparrow$
No Adapt (ResUNet-50) [ 10 ]	70.32	86.82	82.14	68.33	85.33	81.32	70.48	87.38	81.92	76.81	87.19	84.14	71.49	86.68	82.38
SAR(ICLR’23) [ 46 ]	69.01 $\pm$ 0.98	85.79 $\pm$ 0.14	80.21 $\pm$ 0.14	67.81 $\pm$ 2.32	83.87 $\pm$ 0.07	81.51 $\pm$ 0.14	70.51 $\pm$ 1.47	88.31 $\pm$ 0.17	82.40 $\pm$ 0.09	68.02 $\pm$ 2.44	84.37 $\pm$ 0.14	79.68 $\pm$ 0.06	68.84	85.59	80.95
Domain Adaptor(CVPR’23) [ 69 ]	78.17 $\pm$ 1.14	91.84 $\pm$ 0.07	86.41 $\pm$ 0.12	70.87 $\pm$ 1.74	89.36 $\pm$ 0.05	82.13 $\pm$ 0.12	71.63 $\pm$ 2.04	91.98 $\pm$ 0.12	83.04 $\pm$ 0.04	68.76 $\pm$ 1.14	83.97 $\pm$ 0.07	79.52 $\pm$ 0.04	72.36	89.29	82.78
NC-TTT(CVPR’24) [ 48 ]	77.64 $\pm$ 1.31	91.54 $\pm$ 0.17	86.52 $\pm$ 0.12	73.51 $\pm$ 2.34	91.64 $\pm$ 0.04	83.17 $\pm$ 0.04	68.93 $\pm$ 1.02	89.81 $\pm$ 0.17	81.63 $\pm$ 0.12	80.67 $\pm$ 1.04	89.51 $\pm$ 0.14	84.28 $\pm$ 0.13	75.19	90.63	83.90
VPTTA(CVPR’24) [ 5 ]	77.38 $\pm$ 0.68	92.11 $\pm$ 0.09	86.76 $\pm$ 0.10	71.48 $\pm$ 0.87	88.42 $\pm$ 0.07	81.97 $\pm$ 0.13	71.01 $\pm$ 0.82	91.64 $\pm$ 0.08	81.93 $\pm$ 0.10	80.42 $\pm$ 0.29	89.93 $\pm$ 0.11	84.54 $\pm$ 0.07	75.07	90.53	83.80

Appendix

Table A.2 : Quantitative comparison under Mixed Distribution Shifts on the polyp segmentation task. Models are trained on the indicated source domain and adapted to a composite target stream formed by shuffling samples from all remaining domains. We report the Mean

\pm

Std. over five independent runs. The best results are highlighted in bold red .

Figure A.1 : Visualization comparison of segmentation results for the No Adapt baseline, VPTTA [ 5 ] , GraTa [ 6 ] , TTDG [ 39 ] , and SPEGC(Ours) in retinal fundus segmentation. Different colors represent the segmentation instances of different classes identified by the network.

Figure A.2 : Visualization comparison of segmentation results for the No Adapt baseline, VPTTA [ 5 ] , GraTa [ 6 ] , TTDG [ 39 ] , and SPEGC(Ours) in polyp segmentation.

Methods	Domain A	Domain B	Domain C	Domain D	Domain E
No Adapt	46.17	37.14	39.82	54.27	44.72
SAR	42.28±30.71	35.81±26.93	44.59±32.47	48.90±27.19	47.31±30.72
Domain Adaptor	40.54±27.54	38.17±29.92	46.15±31.72	51.23±33.02	36.28±20.39
NC-TTT	38.94±28.19	24.82±20.41	36.54±26.35	40.62±28.40	30.29±21.79
VPTTA	45.28±32.54	34.33±26.17	46.21±29.86	53.81±34.92	39.34±30.83
GraTA	34.62±24.47	29.74±23.22	35.77±27.58	38.04±28.49	37.81±26.04

Appendix

Table A.3 : Average Symmetric Surface Distance (ASSD, in pixels) for the OD/OC segmentation task. Red and blue indicate the best and second-best results, respectively.

Methods	Domain A	Domain B	Domain C	Domain D
No Adapt	30.49	34.36	32.19	27.02
SAR	27.70±12.71	35.17±16.97	32.04±21.40	34.83±15.60
Domain Adaptor	23.19±11.94	29.76±14.57	32.49±19.92	33.91±17.42
NC-TTT	24.72±14.62	25.57±14.33	33.12±24.63	27.43±13.81
VPTTA	23.91±12.02	27.03±19.72	31.99±25.72	29.07±16.03
GraTA	22.54±16.55	30.11±18.06	30.54±23.99	26.39±15.44

Appendix

Table A.4 : Average Symmetric Surface Distance (ASSD, in pixels) for the polyp segmentation task. Red and blue indicate the best and second-best results, respectively.

Methods	Domain A			Domain B			Domain C			Domain D
Methods	LV	MYO	RV	LV	MYO	RV	LV	MYO	RV	LV	MYO	RV
NoAdapt	83.72	70.19	72.34	76.29	69.92	68.57	77.91	68.67	65.07	78.39	68.37	64.72
SPEGC	87.64 $\pm$ 1.86	79.80 $\pm$ 2.72	75.28 $\pm$ 2.17	84.16 $\pm$ 1.54	78.49 $\pm$ 2.63	75.34 $\pm$ 2.09	86.49 $\pm$ 1.68	76.10 $\pm$ 1.82	70.62 $\pm$ 2.11	85.17 $\pm$ 1.20	74.39 $\pm$ 2.17	69.32 $\pm$ 2.45

Appendix

Table A.5 : DSC (Mean ± Std.) performance of SPEGC on the 3D M&MS dataset. Red indicates the best results.

Methods	Domain A			Domain B			Domain C			Domain D
Methods	LV	MYO	RV	LV	MYO	RV	LV	MYO	RV	LV	MYO	RV
NoAdapt	4.37	4.61	4.59	4.82	5.07	5.11	4.39	5.16	5.71	5.38	4.80	4.91
SPEGC	3.76 $\pm$ 3.37	4.04 $\pm$ 3.92	4.21 $\pm$ 3.47	4.17 $\pm$ 3.88	4.39 $\pm$ 4.01	4.82 $\pm$ 3.90	3.26 $\pm$ 3.07	4.37 $\pm$ 3.52	5.02 $\pm$ 4.31	4.14 $\pm$ 2.28	4.06 $\pm$ 3.16	4.72 $\pm$ 3.75

Appendix

Table A.6 : ASSD (pixels) (Mean ± Std.) performance of SPEGC on the 3D M&MS dataset. Red indicates the best results.

Metric	$\lambda=0$	$\lambda=0.1$	$\lambda=0.2$	$\lambda=0.4$	$\lambda=0.5$	$\lambda=0.6$	$\lambda=0.8$
DSC (%)	76.83	81.60	84.37	83.79	82.14	80.42	79.59

Appendix

Table A.7 : Ablation study of the hyperparameter

\lambda

on the OD/OC segmentation task (Average DSC). Red indicates the best result.

Continual Test-Time Adaptation

Continual test-time adaptation updates pretrained models on unlabeled, changing deployment data while retaining useful knowledge across shifts.

See all 72 papers

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation
Jul 20, 2026Lingrui Li, Nan Pu, Dong Zhao +4Memory-Augmented VLMs VLM Adaptation
School of Computer Science, University of Nottingham, UK · School of Computer Science and Information Engineering, Hefei University of Technology, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore
Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation of SAM3 in Medical Image Segmentation
Jun 22, 2026Yubo Zhou, Jianghao Wu, Ping Ye +2Test-Time Adaptation Promptable Image Segmentation
University of Electronic Science and Technology of China, Chengdu, China · Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia · Shanghai Artificial Intelligence Laboratory, Shanghai, China
EviATTA: Evidential Active Test-Time Adaptation for Medical Segment Anything Models
Mar 15, 2026Jiayi Chen, Yasmeen George, Winston Chong +1Test-Time Adaptation Vision Foundation Model Adaptation
Department of Data Science & AI, Faculty of Information Technology, Monash University, VIC 3800, Australia · Alfred Health, VIC 3004, Australia

Jul 20, 2026J/K move · Enter open · S save

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation

Lingrui Li, Nan Pu, Dong Zhao, Wenjing Li+3

School of Computer Science, University of Nottingham, UK · School of Computer Science and Information Engineering, Hefei University of Technology, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore

Memory-Augmented VLMs VLM Adaptation Text-Guided Image Segmentation Medical Image Segmentation

Open paper AskPDF arXiv

Test-time adaptation (TTA) aims to mitigate distribution shifts by adapting models with unlabeled target data at inference time. While TTA with vision-language models (VLMs) has shown promising results in classification, extending it to medical image segmentation remains challenging. In this setting, the adaptation gains from optimizing on VLM-generated predictions are often outweighed by the degradation to the VLM's strong pretrained features caused by noisy, update-driven learning, resulting in limited and unstable improvements. We therefore propose Memory-Supported Synergistic Adaptation (MSSA), a novel training-free TTA framework for medical image segmentation. Without updating model parameters, MSSA dynamically selects reliable image-text predictions to construct an online memory, uses them as text-guided semantic priors, and couples them with cross-image structural alignment for robust adaptation. Specifically, MSSA consists of (i) a noise-aware memory construction module that filters and stabilizes cross-modal predictions, and (ii) a relevance-driven prototype alignment module that aligns the target sample with structurally consistent memory samples and their reliable predictions to improve adaptation. Extensive experiments on multiple medical segmentation benchmarks demonstrate that MSSA consistently improves VLM-based segmentation models and outperforms existing fine-tuning-based TTA methods by a clear margin, with gains of up to 12.2% DSC and 11.7% mIoU. Project page: https://lingrayy.github.io/MSSA/ .

Abstract

Continual Test-Time Adaptation

Explore similar work

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation

Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation of SAM3 in Medical Image Segmentation

EviATTA: Evidential Active Test-Time Adaptation for Medical Segment Anything Models