Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts

Seungmin Oh; Seunghun Kang; Jongbin Ryu

cs.CVOct 5, 2026

Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts

Authors: Seungmin Oh, Seunghun Kang, Jongbin Ryu

Organizations: Ajou University

Abstract

Vision-language models (VLMs) achieve strong zero-shot transferability but remain vulnerable to target-domain shifts at inference time. Test-time adaptation (TTA) offers a practical remedy, yet most existing VLM-TTA methods follow a prediction-side adaptation paradigm. They use test samples to adjust logits, prototypes, caches, priors, or feature statistics, often incurring additional computational overhead. In this paper, we take a different perspective and reframe VLM-TTA as candidate verification rather than prediction adjustment. We propose Test-Time Correction (TTC), a hypothesis-based correction framework guided by a simple principle: hypothesize, reconstruct, correct. Given a test feature and its top-k candidate labels, TTC treats each candidate label as a hypothesis, reconstructs the feature within the corresponding latent subspace stored in a memory bank, and measures the resulting divergence shift. This shift quantifies how much the candidate subspace and its relations to other candidates change after the hypothetical insertion of the test feature. A correct candidate hypothesis induces only a small shift, whereas an incorrect one perturbs the subspace more strongly. TTC therefore corrects the prediction by selecting the candidate with the minimum aggregated divergence shift. This training-free candidate-verification mechanism avoids iterative optimization and provides a favorable accuracy-efficiency trade-off. Across five TTA settings and 15 benchmark datasets, including zero-shot classification, domain generalization, few-shot classification, base-to-novel generalization, and cross-dataset evaluation, TTC consistently improves accuracy over state-of-the-art VLM-TTA methods while achieving up to 2x speedup, over 3x lower CPU memory usage, and up to 1.4x lower GPU memory usage than the lowest-memory training-free baseline.

Figures & tables

Figure 1: Overview of the Test-Time Correction (TTC) based on divergence shift estimation. TTC first selects the top-2 candidate labels, denoted by

y_{1}

and

y_{2}

. Each candidate label is then treated as a hypothesis, and TTC estimates the divergence shifts,

\delta_{y_{1}\to y_{2}}(\mathbf{x})

and

\delta_{y_{2}\to y_{1}}(\mathbf{x})

, induced by inserting the test feature

\mathbf{x}

into the corresponding candidate subspace. Although the original top-1 prediction is

y_{1}

, TTC corrects the prediction to

y_{2}

because inserting

\mathbf{x}

into

\mathcal{S}_{y_{2}}

induces a smaller divergence shift, yielding the lowest aggregated verification score.

✓ Model	Caltech	DTD	ESAT	Aircraft	Flower	Food	INet	Pets	Cars	SUN	UCF	Avg.
✓ ResNet50 (ICML’21)	83.68	41.68	27.69	17.40	63.14	74.95	59.90	80.79	58.33	59.53	55.74	56.62
✓ TPT (NeurIPS’22)	87.02	40.84	28.33	17.58	62.69	74.88	60.74	84.49	58.46	61.46	60.82	57.94
✓ DiffTPT (ICCV’23)	86.89	40.72	41.04	17.60	63.53	79.21	60.80	83.40	60.71	62.72	62.67	59.94
✓ C-TPT (ICLR’24)	86.90	42.20	27.80	17.00	65.20	74.70	60.20	84.10	56.50	61.00	59.70	57.75
✓ DPE (NeurIPS’24)	90.83	50.18	41.67	19.80	67.60	77.83	61.54	85.97	59.26	64.23	61.98	61.90
✓ TDA (CVPR’24)	89.70	43.74	42.11	17.61	68.74	77.75	61.35	86.18	57.78	62.53	64.18	61.06

Table 1: Experimental results of the zero-shot classification task with downstream datasets. The check mark (‘✓’) denotes the training-free methods.

✓ Model	# of Aug.	INet	A	R	Ske.	V2	Dist.	Avg.
✓ ResNet50 (ICML’21)	0	59.90	22.80	60.60	35.51	52.86	42.94	46.33
✓ OGA (CVPR’25)	0	59.65	22.71	59.73	36.54	51.02	42.50	45.93
✓ TTC (Ours)	0	62.58	23.44	61.40	38.85	54.85	44.64	48.23
✓ TPT (NeurIPS’22)	63	60.74	26.67	59.11	35.09	54.70	43.89	47.26
✓ DiffTPT (ICCV’23)	63	60.80	31.06	58.80	37.10	55.80	45.69	48.71
✓ C-TPT (ICLR’24)	63	60.20	27.00	59.70	35.70	54.80	44.30	47.48

Table 2: Experimental results of the domain generalization task. ‘Dist.’ denotes the average accuracy of distribution-shift datasets. The check mark (‘✓’) denotes the training-free methods.

Panel of Figure 2 — Figure 2: Experimental results of few-shot classification using CoOp trained with a specified shot.

Model	Caltech	DTD	ESAT	Aircraft	Flower	Food	INet	Pets	Cars	SUN	UCF	Avg.
CoOp	93.36	57.83	69.55	31.57	75.45	85.57	71.88	94.33	67.93	71.13	69.48	72.32
+ TPT	93.71	57.92	60.62	31.82	74.81	84.47	73.67	93.95	69.07	71.51	70.43	71.74
+ DPE	94.56	59.80	64.06	31.54	76.29	86.81	73.51	94.37	70.30	72.58	71.52	73.08
+ TDA	94.32	58.91	66.50	31.89	76.59	85.80	72.09	94.02	69.39	72.29	70.73	72.70
+ OGA	93.14	59.16	76.75	31.91	76.60	87.99	73.33	94.88	69.01	74.11	72.08	74.07
+ BCA	93.33	57.74	67.75	30.93	75.32	85.46	71.76	93.99	68.01	71.11	69.36	72.04

Table 3: Experimental results of the base-to-novel generalization task using a CoOp-tuned ViT-B/16. The results are the harmonic mean of base- and novel-class accuracies.

	ResNet50					ViT-B/16
Model	GPU (GB) $\downarrow$	CPU (MB) $\downarrow$	Thr. (img/s) $\uparrow$	Wall (s) $\downarrow$	Acc. (%) $\uparrow$	GPU (GB) $\downarrow$	CPU (MB) $\downarrow$	Thr. (img/s) $\uparrow$	Wall (s) $\downarrow$	Acc. (%) $\uparrow$
DPE (NeurIPS’24)	0.38	1039	16.2	3080.3	61.54	0.20	1009	17.0	2935.5	69.91
TDA (CVPR’24)	0.40	520	163.1	306.6	61.35	0.24	520	184.1	271.6	69.51
OGA (CVPR’25)	2.26	544	10.6	4712.4	59.65	2.01	545	20.8	2398.7	68.50
BCA (CVPR’25)	0.40	472	956.2	52.4	61.81	0.26	471	932.5	53.7	70.22
TTC (Ours)	0.31	142	1103.4	45.4	62.58	0.17	135	2049.0	24.5	71.57

Table 5: Comparison of computing resources such as GPU and CPU memory, throughput (Thr.), wall-clock time (Wall), and accuracy (Acc.) on ImageNet zero-shot classification. All performance measurements are conducted using a single NVIDIA RTX A5000.

	ZS		DG		FS	B2N		Cross
	ResNet50	ViT-B/16	ResNet50	ViT-B/16	ViT-B/16	Base	Novel	ViT-B/16
Hit@3	80.90	85.39	67.37	81.04	89.24	94.46	81.70	80.30
$A_{\mathrm{in}}=P(\mathrm{correct}\mid\mathrm{GT}\in\text{top-}3)$
Initial top- $1$	72.62	78.27	68.45	76.24	81.41	86.75	76.94	77.49
DPE (NeurIPS’24)	73.83	79.33	69.02	77.03	82.91	87.66	77.30	77.21
TDA (CVPR’24)	72.78	77.14	69.06	76.85	81.26	86.55	77.68	77.51

Table 6: Top-

3

hit rate (Hit@3) and accuracy inside (

A_{\mathrm{in}}

) and outside (

A_{\mathrm{out}}

) the top-

3

candidates. Hit@3 is shared by all methods in a column, and

A_{\mathrm{out}}=0

for the initial top-

1

and Ours (TTC). ZS: zero-shot, DG: domain generalization, FS: few-shot, B2N: base-to-novel, Cross: cross-dataset.

Case	% of hit set	Margin	Representative pairs
Success (correct)	82.07	0.434	Alaskan tundra wolf $\to$ grey wolf; standard $\to$ miniature Schnauzer
Failure (right $\to$ wrong)	3.33	0.067	projectile $\to$ missile; sunglass $\to$ sunglasses
Failure (wrong $\to$ wrong)	14.60	0.127	ear of corn $\to$ corn; gazelle $\to$ impala

Table 7: Success and failure cases of TTC within the hit set, macro-averaged over the five settings with ViT-B/16. The margin is the median gap between the verification scores of the chosen candidate and the runner-up, and the pairs are representative ImageNet confusions.

Panel of Figure 3 — Figure 3: Ablation studies on top- $k$ and memory under the zero-shot classification setting.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Train Samples	Test Samples	Classes
ImageNet	16000	50000	1000
ImageNet-A	–	7500	200
ImageNet-R	–	30000	200
ImageNet-Sketch	–	50889	1000
ImageNet-V2	–	10000	1000
Caltech101	1600	2465	100

Appendix

Table 8: Detailed dataset information used in experiments. The number of training samples matches that used when training CoOp for few-shot, base-to-novel, and cross-dataset evaluation.

	ZS/DG (ResNet50)			ZS/DG (ViT-B/16)			FS			B2N			Cross
Dataset	Hit@3	Init.	TTC	Hit@3	Init.	TTC	Hit@3	Init.	TTC	Hit@3	Init.	TTC	Hit@3	Init.	TTC
Caltech	98.17	90.70	92.77	99.23	94.44	94.97	99.29	94.63	95.18	98.92	94.55	95.40	99.11	93.00	94.72
DTD	73.82	64.21	72.78	74.53	70.74	74.46	78.12	75.52	79.58	79.92	76.90	80.92	58.29	71.07	74.38
ESAT	75.52	49.45	63.45	78.78	68.55	77.20	91.93	76.92	85.25	92.56	78.80	82.78	75.15	64.27	75.02
Aircraft	38.97	50.04	54.12	51.31	53.68	55.38	59.37	56.30	59.70	57.16	58.08	60.44	37.76	53.04	53.31
Flower	81.97	80.39	87.57	85.02	85.72	90.11	97.57	92.87	96.21	88.50	88.82	91.16	80.77	83.11	87.15

Appendix

Table 9: Per-dataset top-

3

hit rate (Hit@3) and

A_{\mathrm{in}}=P(\mathrm{correct}\mid\mathrm{GT}\in\text{top-}3)

of the initial top-

1

candidate (Init.) and of Ours (TTC). FS is averaged over 1 to 16 shots and B2N over the base and novel splits, with CoOp-tuned ViT-B/16 prompts, where Init. is the CoOp prediction.

Variant	Mean $k$	Acc.	Thr. (img/s)
Fixed $k=3$ (default)	3.00	71.10	2049.0 (100%)
Entropy, $k_{\max}=5$	3.47	71.28	1636 (80%)
Entropy, $k_{\max}=8$	4.30	71.34	1174 (57%)

Appendix

Table 10: Fixed and entropy-guided dynamic top-

k

with ViT-B/16, averaged over the 11 zero-shot classification datasets.

	ResNet50		ViT-B/16
Metric	Acc. (%) $\uparrow$	Thr. (img/s) $\uparrow$	Acc. (%) $\uparrow$	Thr. (img/s) $\uparrow$
Cos. Sim.	61.72	1890.1	70.37	3578.8
NSE (default)	62.58	1103.4	71.57	2049.0

Appendix

Table 11: Accuracy and throughput with regard to the divergence metric on ImageNet zero-shot task.

Panel of Figure 4 — Figure 4: Memory purity along the stream under contaminated memory initialization, macro-averaged over the 11 zero-shot datasets. The legend gives the final accuracy of each setting, and the dashed line marks the initial purity of the standard initialization.

Backbone	Stream	Group	CLIP	TDA	OGA	BCA	TTC
ResNet50	$\sim$ 21K (3:1)	Overall	61.20	61.82	60.23	61.27	61.94
	$\sim$ 21K (3:1)	Tail 20%	60.81	61.55	60.49	61.18	62.22
	$\sim$ 12K (6:1)	Overall	61.26	61.58	59.76	61.36	61.21
	$\sim$ 12K (6:1)	Tail 20%	60.32	60.29	58.19	60.80	62.47
ViT-B/16	$\sim$ 21K (3:1)	Overall	69.66	70.12	69.54	69.76	70.96
	$\sim$ 21K (3:1)	Tail 20%	69.66	69.97	70.12	69.87	72.11

Appendix

Table 12: Accuracy on long-tailed ImageNet streams with ResNet50 and ViT-B/16.

Figure 5: Additional experimental results of the few-shot classification based on CoOp.

Model	Caltech	DTD	ESAT	Aircraft	Flower	Food	INet	Pets	Cars	SUN	UCF	Avg.
ViT-B/32 (ICML’21)	92.54	49.47	51.23	20.61	66.99	80.23	64.65	86.56	61.65	65.82	64.31	64.01
DPE (NeurIPS’24)	93.39	52.36	53.05	21.09	69.59	80.73	65.44	87.27	62.60	67.59	66.38	65.41
TDA (CVPR’24)	92.82	51.30	54.32	20.64	69.35	80.45	65.27	86.84	61.71	66.93	65.13	64.98
OGA (CVPR’25)	91.56	50.95	53.75	21.06	68.90	79.98	64.60	87.65	62.77	66.78	65.58	64.87
BCA (CVPR’25)	92.33	50.12	50.86	21.09	67.11	80.05	64.76	86.89	61.72	66.00	64.66	64.15
TTC (Ours)	93.79	52.96	56.74	21.63	71.99	81.17	66.00	88.39	63.60	68.64	67.51	66.58

Appendix

Table 13: Experimental results of the zero-shot classification task on CLIP ViT-B/32 and ViT-L/14.

Model	# of Aug.	INet	A	R	Ske.	V2	Dist.	Avg.
ViT-B/32 (ICML’21)	0	64.65	31.68	69.57	43.04	57.06	50.34	53.20
OGA (CVPR’25)	0	64.60	32.12	69.73	44.19	56.53	50.64	53.43
TTC (Ours)	0	66.00	32.15	70.20	45.70	57.59	51.41	54.33
DPE (NeurIPS’24)	63	65.44	45.89	74.82	48.52	60.46	57.42	59.03
TDA (CVPR’24)	63	65.27	45.71	74.21	47.58	60.47	56.99	58.65
BCA (CVPR’25)	63	64.76	45.33	74.53	47.05	60.12	56.76	58.36

Appendix

Table 14: Experimental results of the domain generalization task on CLIP ViT-B/32 and ViT-L/14. ‘Dist.’ denotes the average accuracy of distribution-shift datasets.

Model	Sets	Caltech	DTD	ESAT	Aircraft	Flower	Food	INet	Pets	Cars	SUN	UCF	Avg.
	Base	97.80	79.83	91.70	41.30	97.83	88.53	76.40	93.57	78.13	80.73	84.30	82.74
	Novel	89.33	45.33	56.03	25.63	61.47	82.80	67.87	95.10	60.13	63.60	59.23	64.23
CoOp	H.M.	93.36	57.83	69.55	31.57	75.45	85.57	71.88	94.33	67.93	71.13	69.48	72.32
	Base	98.28	80.44	80.26	40.02	96.20	85.78	77.88	93.51	80.38	79.16	85.33	81.57
	Novel	89.56	45.25	48.70	26.41	61.21	83.20	69.88	94.39	60.56	65.21	59.96	64.03
+ TPT	H.M.	93.71	57.92	60.62	31.82	74.81	84.47	73.67	93.95	69.07	71.51	70.43	71.74

Appendix

Table 15: Experimental results of the base-to-novel generalization task using a CoOp-tuned ViT-B/16. ‘H.M.’ denotes the harmonic mean of base- and novel-class accuracies.

	Success (correct)		Failure (right $\to$ wrong)		Failure (wrong $\to$ wrong)
Setting	%	Margin	%	Margin	%	Margin
ZS	81.77	0.414	3.78	0.075	14.45	0.122
DG	78.14	0.399	3.57	0.062	18.29	0.139
FS	84.73	0.457	2.45	0.051	12.82	0.118
B2N	84.75	0.492	2.39	0.064	12.86	0.141
Cross	80.96	0.405	4.46	0.080	14.58	0.116

Appendix

Table 16: Success and failure cases of TTC within the hit set for each setting with ViT-B/16.

	Metric	Caltech	DTD	ESAT	Aircraft	Flower	Food	INet	Pets	Cars	SUN	UCF	Avg.
ResNet50	–	83.68	41.68	27.69	17.40	63.14	74.95	59.90	80.79	58.33	59.53	55.74	56.62
	$l_{2}$ dist.	72.70	38.24	27.60	14.58	51.93	72.23	52.68	81.00	48.90	50.16	40.10	50.01
	Cos. Sim.	90.63	51.65	35.63	20.40	68.66	77.17	61.72	85.58	57.94	64.15	63.86	61.58
	NSE	91.08	53.72	47.91	21.09	71.78	77.86	62.58	86.48	60.73	65.53	65.03	63.98
ViT-B/16	–	90.89	44.87	47.01	24.40	67.65	85.35	68.50	86.20	67.60	64.65	64.36	64.68
	$l_{2}$ dist.	85.11	42.08	32.75	22.32	55.26	83.79	62.12	87.11	58.09	55.11	49.83	57.60

Appendix

Table 17: Per-dataset results of the ablation study of different divergence metrics.

	$d_{\mathbf{x}}$	$\delta_{y\to r}$	$\delta_{r\to y}$	Caltech	DTD	ESAT	Aircraft	Flower	Food	INet	Pets	Cars	SUN	UCF	Avg.
ResNet50	✓			86.21	52.30	41.73	20.49	66.91	72.66	59.41	73.43	56.83	62.60	62.57	59.56
		✓		90.10	49.53	39.57	20.10	68.45	77.65	62.35	85.88	58.86	64.48	63.23	61.84
			✓	91.03	51.30	41.33	20.55	69.31	77.79	62.55	86.26	59.59	64.95	64.26	62.63
	✓	✓		90.99	52.42	47.59	21.09	70.81	77.90	62.81	86.05	60.12	65.47	64.31	63.60
	✓		✓	90.87	53.01	47.46	20.82	71.13	77.89	62.83	86.07	60.34	65.54	65.11	63.73
		✓	✓	91.12	52.48	42.79	21.27	70.20	77.98	62.80	86.64	59.97	65.51	65.00	63.25

Appendix

Table 18: Per-dataset results of the ablation study of the three components of the aggregated verification score.

	ZS/DG		FS					B2N			Cross
Dataset	ResNet50	ViT-B/16	1	2	4	8	16	Base	Novel	H.M.	ViT-B/16
Caltech	0.231	0.080	0.256	0.672	0.842	0.417	0.191	0.527	2.428	1.547	0.138
DTD	0.677	0.201	0.470	2.140	1.266	0.278	0.536	2.108	2.714	2.710	1.378
ESAT	0.500	3.164	2.724	3.546	1.102	3.924	1.167	0.880	7.051	5.303	1.910
Aircraft	0.347	0.278	1.842	0.641	2.076	0.706	0.711	0.745	2.652	2.298	1.588
Flower	0.306	0.096	1.255	0.351	1.044	0.253	0.046	0.598	5.018	3.721	2.421

Appendix

Table 19: 95% confidence intervals of our experimental results over three random seeds. ZS/DG, FS, B2N, and Cross denote zero-shot classification and domain generalization, few-shot classification, base-to-novel generalization, and cross-dataset evaluation, respectively.

		Per-seed difference
Setting	Baseline	Seed 1	Seed 2	Seed 3	Gap $\pm$ 95% CI	Baseline CI
ZS Avg. (ResNet50)	TDA	$+2.19$	$+1.87$	$+2.29$	$+2.12\pm 0.55$	$\pm 0.17$
	OGA	$+2.44$	$+1.92$	$+1.99$	$+2.12\pm 0.70$	$\pm 0.74$
	BCA	$+3.83$	$+3.60$	$+3.91$	$+3.78\pm 0.41$	$\pm 0.09$
ZS Avg. (ViT-B/16)	TDA	$+1.78$	$+1.37$	$+1.83$	$+1.66\pm 0.63$	$\pm 0.40$
	OGA	$+1.60$	$+1.57$	$+1.70$	$+1.62\pm 0.17$	$\pm 0.21$

Appendix

Table 20: Paired accuracy differences between TTC and each baseline on the same three stream permutations. We report per-seed differences, their mean with the 95% confidence interval (Gap), and each baseline’s own 95% accuracy confidence interval across seeds (Baseline CI).

Explore similar work

Prototype-Based Test-Time Adaptation of Vision-Language Models
Apr 23, 2026Zhaohong Huang, Yuxin Zhang, Wenjing Liu +2Vision-Language Model Adaptation Test-Time Adaptation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
Local Margin Restoration for Test-Time Adaptation of Vision-Language Models
Aug 3, 2026Yan Huang, Guowei Wang, Xu Wang +2Vision-Language Model Adaptation Test-Time Adaptation
Guangzhou University Guangzhou, China · The Second Affiliated Hospital of Guangzhou University of Chinese Medicine Guangzhou, China · Jinan University Zhuhai, China +1
To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models
Sep 8, 2026Siru Jiang, Yuwei Liang, Jian Liang +2Vision-Language Model Adaptation Test-Time Adaptation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China +1

Apr 23, 2026, cs.CVJ/K move · Enter open · S save

Prototype-Based Test-Time Adaptation of Vision-Language Models

Zhaohong Huang, Yuxin Zhang, Wenjing Liu, Fei Chao+1

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.

Vision-Language Model Adaptation Test-Time Adaptation Topology

Open paper AskPDF arXiv

Test-time adaptation (TTA) has emerged as a promising paradigm for vision-language models (VLMs) to bridge the distribution gap between pre-training and test data. Recent works have focused on backpropagation-free TTA methods that rely on cache-based designs, but these introduce two key limitations. First, inference latency increases as the cache grows with the number of classes, leading to inefficiencies in large-scale settings. Second, suboptimal performance occurs when the cache contains insufficient or incorrect samples. In this paper, we present Prototype-Based Test-Time Adaptation (PTA), an efficient and effective TTA paradigm that uses a set of class-specific knowledge prototypes to accumulate knowledge from test samples. Particularly, knowledge prototypes are adaptively weighted based on the zero-shot class confidence of each test sample, incorporating the sample's visual features into the corresponding class-specific prototype. It is worth highlighting that the knowledge from past test samples is integrated and utilized solely in the prototypes, eliminating the overhead of cache population and retrieval that hinders the efficiency of existing TTA methods. This endows PTA with extremely high efficiency while achieving state-of-the-art performance on 15 image recognition benchmarks and 4 robust point cloud analysis benchmarks. For example, PTA improves CLIP's accuracy from 65.64% to 69.38% on 10 cross-domain benchmarks, while retaining 92% of CLIP's inference speed on large-scale ImageNet-1K. In contrast, the cache-based TDA achieves a lower accuracy of 67.97% and operates at only 50% of CLIP's inference speed.

Abstract

Explore similar work

Prototype-Based Test-Time Adaptation of Vision-Language Models

Local Margin Restoration for Test-Time Adaptation of Vision-Language Models

To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models