CPATTA: Conformal Supervision Allocation For Active Test-Time Adaptation

Tingyu Shi; Fan Lyu; Haihua Zhu; Dadi Wang; Shaoliang Peng

cs.LGSep 30, 2025

CPATTA: Conformal Supervision Allocation For Active Test-Time Adaptation

Authors: Tingyu Shi, Fan Lyu, Haihua Zhu, Dadi Wang, Shaoliang Peng

Organizations: Computer Science and Engineering, University of California San Diego · New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences · College of Computer Science and Electronic Engineering, Hunan University

Abstract

Active Test-Time Adaptation (ATTA) improves model robustness under domain shift by selectively querying human annotations at deployment, but existing methods use heuristic uncertainty measures and suffer from low data selection efficiency, wasting human annotation budget. We propose Conformal Prediction Active TTA (CPATTA), which first brings principled, conformal uncertainty with coverage-aware online calibration into ATTA. CPATTA employs smoothed conformal scores with a top- $K$ certainty measure, an online weight-update algorithm driven by pseudo coverage, a domain-shift detector that adapts human supervision, and a staged update scheme that balances human-labeled and model-labeled data. Extensive experiments demonstrate that CPATTA consistently outperforms the state-of-the-art ATTA methods by around 5% in accuracy.

Figures & tables

Figure 1 : Comparison between existing ATTA methods and CPATTA. Selection efficiency is defined as the fraction of useful samples: correct predictions for model-annotated (pseudo-label) data and incorrect predictions for human-annotated data. CPATTA achieves higher selection efficiency in both cases, enabling better real-time and post-adaptation performance under the same annotation budget.

Figure 2 : CPATTA Overview . The real-time model $f(\cdot;\theta)$ makes the predictions right away when an online batch of data arrives. DSS raises the human budget after detected shifts; real-time and pretrained CPs rank samples for human and model annotation, respectively. Pseudo coverage updates their weights, and the resulting buffers drive staged adaptation.

		PACS Dataset									VLCS Dataset
		Real-Time Accuracy				Post-Adaptation Accuracy					Real-Time Accuracy				Post-Adaptation Accuracy
Methods		A $\rightarrow$	$\rightarrow$ C $\rightarrow$	$\rightarrow$ S	Acc	P	A	C	S	Acc	L $\rightarrow$	$\rightarrow$ S $\rightarrow$	$\rightarrow$ V	Acc	C	L	S	V	Acc
	Tent [ 4 ]	67.53	69.16	66.94	67.71	93.53	68.46	71.25	73.12	75.14	46.26	41.22	55.72	47.91	90.11	52.80	45.70	57.08	56.90
	CoTTA [ 6 ]	65.72	65.91	65.28	65.57	83.71	60.60	63.48	71.24	69.32	45.84	39.49	54.92	46.89	47.32	52.88	39.85	50.24	47.33
	SAR [ 7 ]	65.53	66.13	63.43	64.71	94.19	65.63	67.02	65.11	70.53	42.69	37.51	50.74	43.80	90.47	44.90	38.76	50.71	50.86
TTA	EATA [ 5 ]	67.43	68.26	67.24	67.57	94.85	66.31	66.72	70.60	72.86	42.95	37.26	51.36	44.01	91.24	43.93	37.72	51.74	50.73

Table 1 : Real-time and post-adaptation accuracy on PACS, VLCS, and Tiny-ImageNet-C. Acc denotes the overall sample-level accuracy across all target domains or corruptions, rather than the unweighted arithmetic mean of the individual results.

	PACS		VLCS		Tiny-ImageNet-C
Method	$\text{Eff}_{H}$	$\text{Eff}_{M}$	$\text{Eff}_{H}$	$\text{Eff}_{M}$	$\text{Eff}_{H}$	$\text{Eff}_{M}$
SimATTA [ 8 ]	47.60	67.60	57.04	64.78	79.25	50.00
CEMA [ 9 ]	24.91	N/A	44.38	N/A	75.82	N/A
EATTA [ 10 ]	53.71	90.71	58.38	63.60	62.90	32.31
CPATTA( $\alpha=0.1/0.3$ )	66.30	93.41	62.89	81.91	90.45	48.14
CPATTA( $\alpha=0.2/0.4$ )	65.94	93.65	61.86	85.11	91.00	47.83

Table 2 : Data selection efficiencies with

\alpha\in\{0.1,0.2,0.3\}

for PACS/VLCS and

\alpha\in\{0.3,0.4,0.5\}

for Tiny-ImageNet-C

Variants	PACS	VLCS	Tiny	PACS	VLCS	Tiny
	Realtime Accuracy			Post-Adaptation Accuracy
Random Selection	68.16	55.58	31.98	78.37	70.77	40.96
w/o adaptive weighting (Gometric Decaying)	74.31	63.29	29.22	83.37	75.14	37.14
Fixed hard-set size	72.02	58.34	29.38	83.09	68.03	37.58
w/o DSS-triggered budget adjustment	68.50	64.64	30.69	84.83	74.91	38.45
Joint human–pseudo update	69.53	57.01	28.32	84.70	72.78	36.38

Table 3 : Ablation under the same human annotation budget

Panel of Figure 3 — Figure 3 : Compare coverage gap and adaptation acc with other CPs.

	Real-Time				Post-Adapt
Method	A $\rightarrow$	$\rightarrow$ C $\rightarrow$	$\rightarrow$ S	Acc	Acc
SimATTA with Replay	74.32	50.64	72.77	66.92	79.14
CEMA with Replay	65.58	68.64	58.28	63.00	75.38
EATTA with Replay	69.97	69.07	65.59	67.65	74.85
CPATTA( $\alpha=0.1$ )	71.39	68.05	76.18	72.71	85.25
CPATTA( $\alpha=0.2$ )	73.10	70.01	79.51	75.26	87.13

Table 4 : Replay Experiment on PACS

Metric	Real-Time CP		Pretrained CP
Metric	PACS	VLCS	PACS	VLCS
Corr.	0.9978	0.9997	0.9988	0.9853

Table 5 : Pseudo-coverage validation on PACS and VLCS.

Explore similar work

WISE-ATTA: When to Ask for Labels in Budgeted Active Test-Time Adaptation
Sep 29, 2026Muhammad Huzaifa, Lea Schönherr, Thorsten EisenhoferTest-Time Adaptation Pool-Based Active Learning
CISPA Helmholtz Center for Information Security
Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions
Jul 9, 2026Sarthak Kumar Maharana, Shambhavi Mishra, Yunbei Zhang +6Test-Time Adaptation Mitigating Domain Shift
The University of Texas at Dallas, USA · LIVIA ETS Montreal, ILLS International Laboratory on Learning Systems (ILLS), McGill - ÉTS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec · Tulane University, USA +2
Distilling Image Prototypes for Guided Test-Time Adaptation
Sep 9, 2026Liwen Wang, Xingbo Dong, Iman Yi Liao +4Stable Test-Time Adaptation Test-Time Adaptation
Anhui Provincial Key Laboratory of Secure Artificial Intelligence, Anhui Provincial International Joint Research Center for Advanced Technology in Medical Imaging, School of Artificial Intelligence, Anhui University, Hefei 230093, China · School of Computer Science, University of Nottingham (Malaysia Campus), Selangor Darul Ehsan, 43500, Malaysia · Department of Engineering, University of Sassari, Sassari, Italy +1

Sep 29, 2026, cs.AIJ/K move · Enter open · S save

WISE-ATTA: When to Ask for Labels in Budgeted Active Test-Time Adaptation

Muhammad Huzaifa, Lea Schönherr, Thorsten Eisenhofer

CISPA Helmholtz Center for Information Security

Test-Time Adaptation Pool-Based Active Learning Distribution Shifts Imagenet Token Budget Allocation

Open paper AskPDF arXiv

Active test-time adaptation (ATTA) improves robustness under distribution shift by updating a deployed model during inference while selectively querying supervision. However, most existing ATTA methods implicitly assume that supervision can be requested for every incoming test batch, which can incur substantial annotation cost over long test streams. In this work, we introduce \emph{budgeted ATTA} in which labels are available for only a fraction of test batches. This formulation shifts the central challenge from deciding \emph{what} to label within a batch to deciding \emph{when} supervision should be applied over time. To address this challenge, we propose a budget-aware approach \emph{WISE-ATTA} that allocates supervision over the test stream based on lightweight signals computed online, prioritizing periods where supervision is likely to be most useful. When a batch is selected for supervision, we further employ a drift-based sample selection criterion that targets samples exhibiting ongoing, unconverged adaptation dynamics, enabling effective updates from a single labeled example. We evaluate this approach on synthetic corruptions (ImageNet-C) and natural distribution shifts (ImageNet-R/K/A). Across settings, WISE-ATTA achieves competitive or improved performance compared to recent ATTA methods while requiring substantially fewer labels. Overall, we find that the timing of supervision is a key, yet underexplored, aspect of active test-time adaptation. Code: https://github.com/Muhammad-Huzaifaa/WISE-ATTA

Abstract

Explore similar work

WISE-ATTA: When to Ask for Labels in Budgeted Active Test-Time Adaptation

Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions

Distilling Image Prototypes for Guided Test-Time Adaptation