EviATTA: Evidential Active Test-Time Adaptation for Medical Segment Anything Models

Jiayi Chen; Yasmeen George; Winston Chong; Jianfei Cai

cs.CVMar 15, 2026

EviATTA: Evidential Active Test-Time Adaptation for Medical Segment Anything Models

Authors: Jiayi Chen, Yasmeen George, Winston Chong, Jianfei Cai

Organizations: Department of Data Science & AI, Faculty of Information Technology, Monash University, VIC 3800, Australia · Alfred Health, VIC 3004, Australia

Abstract

Deploying foundational medical Segment Anything Models (SAMs) via test-time adaptation (TTA) is challenging under large distribution shifts, where test-time supervision is often unreliable. While active test-time adaptation (ATTA) introduces limited expert feedback to improve reliability, existing ATTA methods still suffer from unreliable uncertainty estimation and inefficient utilization of sparse annotations. To address these issues, we propose Evidential Active Test-Time Adaptation (EviATTA), which is, to our knowledge, the first ATTA framework tailored for medical SAMs. Specifically, we adopt the Dirichlet-based Evidential Modeling to decompose overall predictive uncertainty into distribution uncertainty and data uncertainty. Building on this decomposition, we design a Hierarchical Evidential Sampling strategy, where image-wise distribution uncertainty is used to select informative shifted samples, while distance-aware data uncertainty guides sparse pixel annotations to resolve data ambiguities. We further introduce Dual Consistency Regularization, which enforces progressive prompt consistency on sparsely labeled samples to better exploit sparse supervision and applies variational feature consistency on unlabeled samples to stabilize adaptation. Extensive experiments on six medical image segmentation datasets demonstrate that EviATTA consistently improves adaptation reliability with minimal expert feedback under both batch-wise and instance-wise test-time adaptation settings.

Figures & tables

Figure 1: Comparison of TTA, vanilla ATTA, and EviATTA. (a) TTA suffers from unreliable adaptation due to unreliable self-supervision. (b) Vanilla ATTA partially improves adaptation but still struggles with unreliable uncertainty estimation and inefficient utilization of sparse annotations. (c) EviATTA employs decoupled uncertainty for more reliable uncertainty estimation and leverages dual consistency regularization to exploit sparse annotations.

Figure 2: Framework of EviATTA. It comprises three components: (a) Dirichlet-based Evidential Modeling, which formulates categorical prediction as following a Dirichlet distribution; (b) Hierarchical Evidential Sampling, which decomposes overall uncertainty into distribution and data uncertainties, selects samples via image-wise distribution uncertainty, and annotates pixels using pixel-wise data uncertainty; (c) Dual Consistency Regularization, which enforces progressive prompt consistency to leverage sparse annotations and variational feature consistency to stabilize adaptation.

Type	Method	BraTS-SSA				BraTS-PED				FHPsAOP
Type	Method	Dice $\uparrow$	Jaccard $\uparrow$	ASD $\downarrow$	HD95 $\downarrow$	Dice $\uparrow$	Jaccard $\uparrow$	ASD $\downarrow$	HD95 $\downarrow$	Dice $\uparrow$	Jaccard $\uparrow$	ASD $\downarrow$	HD95 $\downarrow$
Zero-shot	SAM-Med2D [ 1 ]	82.34	71.64	2.73	13.10	84.84	75.32	2.12	7.77	73.87	60.84	11.16	29.58
TTA	TENT [ 13 ]	81.67	70.81	2.75	13.82	86.30	77.39	1.96	7.55	67.66	52.36	12.76	34.24
	CoTTA [ 16 ]	84.09	73.90	2.70	13.01	85.87	76.53	2.16	7.74	74.19	61.36	11.74	30.24
	SAR [ 20 ]	82.08	71.31	2.76	13.72	85.36	76.00	2.09	7.69	68.42	53.23	12.7	33.81
	GraTa [ 15 ]	82.42	71.55	2.90	12.90	84.02	74.04	2.29	7.94	75.92	62.53	11.03	29.52

Table I: Performance comparison with SOTA TTA and ATTA methods under batch-wise setting. Best results are highlighted as First , Second , and Third .

10\%

and

20\%

indicate selecting 10% and 20% samples per batch for sparse annotation.

Type	Method	BraTS-SSA				BraTS-PED				FHPsAOP
Type	Method	Dice $\uparrow$	Jaccard $\uparrow$	ASD $\downarrow$	HD95 $\downarrow$	Dice $\uparrow$	Jaccard $\uparrow$	ASD $\downarrow$	HD95 $\downarrow$	Dice $\uparrow$	Jaccard $\uparrow$	ASD $\downarrow$	HD95 $\downarrow$
Zero-shot	SAM-Med2D [ 1 ]	82.34	71.64	2.73	13.10	84.84	75.32	2.12	7.77	73.87	60.84	11.16	29.58
TTA	TENT [ 13 ]	80.59	69.42	2.84	14.19	86.03	77.04	1.98	7.64	65.48	49.98	13.07	35.33
	CoTTA [ 16 ]	84.61	74.61	2.68	12.91	86.01	76.7	2.16	7.68	72.06	58.78	12.48	31.65
	SAR [ 20 ]	82.46	71.75	2.76	13.21	85.44	76.07	2.11	7.73	65.80	50.12	13.43	35.46
	GraTa [ 15 ]	82.31	71.63	2.72	12.89	84.24	74.62	2.07	7.67	70.62	58.21	10.07	30.40

Table II: Performance comparison with SOTA TTA and ATTA methods under instance-wise setting.

Annotation	Consistency Regularization			BraTS-SSA
Annotation	$\mathcal{L}_{prompt}$	$\mathcal{L}_{replay}$	$\mathcal{L}_{var}$	Dice $\uparrow$	Jaccard $\uparrow$	ASD $\downarrow$	HD95 $\downarrow$
				82.34	71.64	2.73	13.10
✓				83.25	72.83	2.67	13.15
✓	✓			88.02	79.46	2.10	11.35
✓	✓	✓		88.29	79.95	2.00	11.13
✓	✓	✓	✓	89.26	81.42	1.80	10.61

Table III: Effect of each component.

Uncertainty Estimation		BraTS-SSA
Sample-wise	Pixel-wise	Dice $\uparrow$	Jaccard $\uparrow$	ASD $\downarrow$	HD95 $\downarrow$
-	-	82.34	71.64	2.73	13.10
Random	Random	83.25	72.98	2.64	12.89
$U$	$U$	85.94	76.66	2.30	12.36
$U_{data}$	$U_{data}$	87.34	78.36	2.35	11.71
$U_{dis}$	$U_{dis}$	87.38	78.39	2.33	11.66

Table IV: Effect of Sampling Strategy.

Figure 3: Visualization of uncertainty map. $U$ denotes predictive uncertainty. $U_{dis}$ and $U_{data}$ represent distribution and data uncertainties.

Figure 4: Effect of pixel annotation budgets on BraTS-SSA.

Figure 5: Effect of buffer size on BraTS-SSA.

Weight Matrix				BraTS-SSA
$\boldsymbol{W}_{Q}$	$\boldsymbol{W}_{K}$	$\boldsymbol{W}_{V}$	$\boldsymbol{W}_{O}$	Dice $\uparrow$	Jaccard $\uparrow$	ASD $\downarrow$	HD95 $\downarrow$
				83.25	72.83	2.67	13.15
✓				87.23	78.43	2.27	11.60
	✓			87.23	78.44	2.27	11.60
		✓		89.09	81.13	1.86	10.69
			✓	89.26	81.42	1.80	10.61

Table V: Effect of LoRA Placement

Figure 6: Qualitative segmentation results of different methods across six medical image segmentation datasets.

Figure 7: Effect of initial prompt on six datasets.

Figure 8: Segmentation results under different numbers of point prompts.

Explore similar work

Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation of SAM3 in Medical Image Segmentation
Jun 22, 2026Yubo Zhou, Jianghao Wu, Ping Ye +2Test-Time Adaptation Promptable Image Segmentation
University of Electronic Science and Technology of China, Chengdu, China · Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia · Shanghai Artificial Intelligence Laboratory, Shanghai, China
Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation
Jul 20, 2026Lingrui Li, Nan Pu, Dong Zhao +4Memory-Augmented VLMs VLM Adaptation
School of Computer Science, University of Nottingham, UK · School of Computer Science and Information Engineering, Hefei University of Technology, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore
CPATTA: Conformal Supervision Allocation For Active Test-Time Adaptation
Sep 30, 2025Tingyu Shi, Fan Lyu, Haihua Zhu +2Distribution Shift Robustness Test-Time Adaptation
Computer Science and Engineering, University of California San Diego · New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences · College of Computer Science and Electronic Engineering, Hunan University

Jun 22, 2026J/K move · Enter open · S save

Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation of SAM3 in Medical Image Segmentation

Yubo Zhou, Jianghao Wu, Ping Ye, Shaoting Zhang+1

University of Electronic Science and Technology of China, Chengdu, China · Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia · Shanghai Artificial Intelligence Laboratory, Shanghai, China

Test-Time Adaptation Promptable Image Segmentation Vision Foundation Model Adaptation Medical Image Segmentation

Open paper AskPDF arXiv

Concept segmentation models like Segment Anything Model 3 (SAM3) show strong generalization on natural images, yet their performance degrades in medical imaging due to the domain gap caused by different imaging principles and styles. Test-Time Adaptation (TTA) is essential for improving the testing performance by updating the model on the fly without annotations. However, existing vision-language TTA methods are mainly driven by image-level uncertainty minimization, which does not necessarily reflect region-level semantic correctness in medical segmentation. Moreover, they often lack mechanisms to maintain stability in continual one-pass adaptation, leading to limited performance when reliable dense supervision is missing for segmentation. To address these issues, we propose Concept Alignment Contrast and LongShort Prompt Memory for Test-Time Adaptation (CM-TTA) of SAM3 for medical images. First, for a test sample with multiple augmentations, we introduce a novel Concept Alignment Contrast (CAC) metric, which leverages textual-visual semantic consistency to robustly evaluate prediction quality to select the best augmented view as the supervision. Second, to balance rapid and stable adaptation, we design a Long-Short Prompt Memory (LSPM) module. The short memory dynamically fuses recent prompts based on CAC scores for agile local adaptation, while the long memory maintains a stable global prompt to generate enhanced pseudo-labels. Finally, a Densely Supervised Prompt Update (DSPU) strategy is proposed to optimize the prompt embeddings with enhanced pseudo labels as dense supervision. Extensive experiments on prostate and skin lesion segmentation demonstrate that our CM-TTA framework significantly outperforms existing methods for TTA of SAM3. The code is available at https://github.com/SherlockZYB/CM-TTA.

Abstract

Explore similar work

Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation of SAM3 in Medical Image Segmentation

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation

CPATTA: Conformal Supervision Allocation For Active Test-Time Adaptation