Bridging CLIP with DINO: Cross-Modal Information Maximization for Online Test-Time Adaptation

Taehoon Kim; Wonjun Hwang; Hyunsouk Cho

cs.CVNov 3, 2026

Bridging CLIP with DINO: Cross-Modal Information Maximization for Online Test-Time Adaptation

Authors: Taehoon Kim, Wonjun Hwang, Hyunsouk Cho

Organizations: Korea University · Ajou University

Abstract

Vision-language models such as CLIP enable zero-shot classification from arbitrary class names, but under test-time corruptions, their visual encoder projects distorted features onto incorrect textual anchors with high confidence: a phenomenon we term modality bias. Standard entropy minimization reinforces these overconfident errors, which creates a confirmation-bias loop and collapses the predicted label distribution. We propose an online test-time adaptation (TTA) method that breaks this loop at three levels. At the representation level, a frozen self-supervised backbone (e.g., DINO v2) is combined with CLIP through a cross-covariance logit ensemble. CLIP text prototypes are projected into the feature space of this text-free backbone, so the self-supervised branch classifies native image features and is less affected by the text bias of CLIP. At the optimization level, a gated test-time information-maximization objective pairs per-sample sharpening with batch-level diversity, while an adaptive gate excludes unreliable samples from the gradient. At the prediction level, an online logit adjustment corrects the residual class-distribution skew. CDIM outperforms the compared CLIP-based online TTA baselines on ImageNet-C, CIFAR10-C, and CIFAR100-C, and remains competitive in continual and mixed-domain TTA. Extensive analyses and ablation studies verify the contribution of each of its components. Our code is available at https://github.com/B1ueC1oud/CDIM

Figures & tables

Figure 1: Confidence of incorrect predictions on ImageNet-C (all corruptions, severity 5). CLIP-based entropy minimization (red) assigns high confidence even to incorrect predictions under corruption. CDIM (blue) shifts the confidence of incorrect predictions toward moderate values, which yields a flatter distribution. The corresponding reliability diagram and ECE with 10 equal-width bins are shown in Figure 1(b) .

Panel of Figure 2 — Figure 2: (a) Left : standard TTA uses a vision-language model (e.g., CLIP) that adapts through entropy minimization on its own predictions. Middle : continual TTA (CTTA) uses a source-pretrained backbone and a moving-average teacher to enforce teacher-student consistency. Right : CDIM adds a frozen self-supervised backbone, in a role similar to that of the teacher in the middle, to support the target CLIP. (b) Reliability diagram with 10 equal-width bins. CLIP+TENT (blue) is severely over-confident (below the $y=x$ line, ECE = 0.400). Adding the DINO v2 ensemble (orange) or GT-IM (green) alone lowers the ECE to 0.210 and 0.101, respectively, and the full CDIM (red) reaches the lowest ECE (0.048).

Method		Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elast.	Pixel.	JPEG	Mean $\uparrow$
ImageNet-C	Zero Shot	11.1	12.5	12.1	23.3	15.3	24.4	22.7	32.4	29.9	35.9	54.1	17.3	12.7	31.0	33.3	24.5
	TDA [ 28 ]	12.4	14.8	14.8	23.7	16.7	26.1	24.0	33.5	32.1	38.0	55.3	19.3	14.2	33.5	35.0	26.2
	TENT [ 54 ]	5.2	5.7	7.4	25.3	19.3	26.9	24.2	33.5	30.5	37.8	54.3	22.5	13.9	35.1	36.0	25.2
	TPT [ 50 ]	8.5	9.4	10.2	24.1	15.1	25.0	24.0	34.0	32.2	37.1	55.5	16.5	13.7	33.9	33.6	24.9
	WATT [ 43 ]	11.2	12.8	13.0	25.1	18.1	26.9	25.2	33.4	30.2	37.4	54.0	21.8	15.2	33.5	34.9	26.2
	Mint [ 4 ]	19.7	20.4	19.4	26.9	21.5	29.9	26.0	32.7	29.4	38.8	55.1	24.3	19.1	36.4	38.2	29.2

Table 1: Online TTA accuracy (%) with ViT-B/16 at severity 5. Top-1 and Top-2 are shown in bold and underline , respectively.

Table 2: Accuracy (%) under (a) continual and (b) mixed-domain protocols. Continual: CLIP ViT-B/16. Mixed-domain: backbones follow [ 4 ] . CDIM uses DINO v2-base.

Table 3: Accuracy (%) under various settings. Default: CLIP ViT-B/16 with DINO v2-base. (a) Auxiliary self-supervised model and (b) DINO v2 scale for each CLIP backbone, on CIFAR10-C. (c) TinyCLIP (ViT-61M/32) on CIFAR10-C. (d) ImageNet variants with CLIP ViT-B/16.

ENS	GT-IM	OLA	ImageNet-C	CIFAR10-C	CIFAR100-C
			25.2	61.9	37.0
✓			28.7	63.1	40.0
	✓		33.8	76.9	45.2
	✓	✓	35.0	78.3	46.2
✓	✓		36.3	79.2	47.5
✓	✓	✓	36.9	79.8	48.2

Table 4: Ablation study on the contributions of CDIM. The first row is the CLIP+TENT.

Panel of Figure 3 — Figure 3: Three analyses during the CIFAR10-C TTA process, comparing CLIP+TENT and CDIM . (a) Image-text alignment of CLIP, measured as $\langle\mathbf{f}_{\text{img}},\mathbf{t}_{\text{true}}\rangle$ per batch during TTA. (b) Cumulative prediction frequency per class, with the ground-truth frequency for reference. (c) Marginal entropy $H(\bar{\mathbf{p}}_{t})$ per batch; the dashed line marks $\log K$ .

Figure 4: Comparison t-SNE visualization (CIFAR10-C).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

CLIP encoder	SSL model	Accuracy (%)	Peak mem (MB)	Latency (ms/batch)	Throughput (img/s)
ViT-B/16	None (TENT)	37.0	13146	555	360.6
ViT-B/16	None (BATCLIP)	42.2	14505	665	300.7
ViT-B/16	DINO v2-base	48.2	16643	1347	148.5
ViT-B/16	DINO v2-small	47.9	15394	1209	165.5
TinyCLIP ViT-61M/32	DINO v2-base	46.7	5837	1277	156.7
TinyCLIP ViT-61M/32	DINO v2-small	45.5	5335	1139	175.6

Appendix

Table 5: Efficiency comparison on CIFAR100-C measured on an NVIDIA V100 GPU.

ENS (w/ DINO)	GT-IM	OLA	ImageNet-C	CIFAR10-C	CIFAR100-C
			25.2	61.9	37.0
		✓	30.4	64.0	41.2
✓			28.7	63.1	40.0
✓		✓	33.1	66.5	44.5
	✓		33.8	76.9	45.2
	✓	✓	35.0	78.3	46.2

Appendix

Table 6: Ablation study of OLA. The first row is the CLIP+TENT baseline.

	CIFAR10-C (fixed seed: 79.8)
Batch Size	Noise			Blur				Weather				Digital				Avg.
	Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elastic	Pixel	JPEG	Avg.
20	67.3 (0.6)	70.2 (0.6)	75.5 (0.8)	82.5 (0.4)	68.2 (0.7)	82.7 (0.3)	84.6 (0.7)	85.0 (0.7)	85.1 (0.6)	85.5 (0.8)	91.5 (0.3)	88.1 (0.5)	74.4 (0.5)	79.8 (0.4)	72.1 (1.2)	79.5 (0.2)
50	69.5 (0.6)	71.9 (0.8)	78.1 (0.6)	84.7 (0.5)	70.0 (0.6)	84.2 (0.2)	86.3 (0.1)	87.1 (0.2)	87.3 (0.3)	86.7 (0.4)	93.3 (0.2)	90.0 (0.2)	76.9 (0.5)	81.0 (0.4)	74.2 (0.4)	81.4 (0.1)
100	69.3 (0.4)	72.4 (0.4)	77.9 (0.4)	84.5 (0.2)	69.1 (0.4)	84.2 (0.2)	86.5 (0.2)	87.7 (0.1)	87.5 (0.3)	86.4 (0.4)	93.3 (0.1)	89.2 (0.2)	76.6 (0.3)	80.4 (0.4)	74.7 (0.4)	81.3 (0.2)
200	67.0 (0.3)	70.9 (0.2)	76.4 (0.4)	83.1 (0.2)	66.9 (0.5)	83.0 (0.3)	85.1 (0.1)	86.5 (0.1)	86.7 (0.2)	85.2 (0.2)	92.5 (0.1)	87.0 (0.3)	74.8 (0.1)	78.0 (0.4)	73.3 (0.1)	79.8 (0.1)

Appendix

Table 7: Accuracy (mean (std) %) on OTTA corruption benchmarks with different batch sizes in ViT-B/16.

Variant	Cond.	Marg.	Gate	Ens.	CIFAR10-C	CIFAR100-C
(a) conditional only (= TENT)	✓				61.9	37.0
(b) marginal only		✓			67.9	35.6
(c) both, no gate	✓	✓			74.4	44.0
(d) + gate (full GT-IM)	✓	✓	✓		76.9	45.2
(e) + ensemble	✓	✓	✓	✓	79.2	47.5

Appendix

Table 8: Decomposition of GT-IM on CLIP+TENT. Cond.: conditional-entropy term, Marg.: marginal-entropy term, Gate: adaptive confidence gate, Ens.: DINO v2 logit ensemble. OLA is not used in any row.

Method	CIFAR100-C	ImageNet-C
TENT (CLIP only)	37.0	25.2
TENT + DINO v2 ensemble	40.0	28.7
BATCLIP (CLIP only)	42.2	30.1
BATCLIP + DINO v2 ensemble	44.6	32.6
CDIM (Ours)	48.2	36.9

Appendix

Table 9: Accuracy (%) of TENT and BATCLIP with and without the DINO v2 logit ensemble (default setting: CLIP ViT-B/16 and DINO v2-base).

Dataset	Severity	Noise			Blur				Weather				Digital				Mean $\uparrow$
Dataset	Severity	Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elast.	Pixel.	JPEG	Mean $\uparrow$
ImageNet-C	Level 1	60.5	60.0	56.9	58.3	58.3	62.2	54.4	57.7	58.7	60.8	66.0	63.4	60.6	62.1	61.3	60.1
	Level 2	56.2	55.1	51.8	54.1	53.9	58.0	48.5	49.1	51.0	58.9	65.4	62.5	44.6	60.5	59.1	55.2
	Level 3	48.2	47.7	46.9	46.7	41.7	51.2	44.4	50.3	44.8	56.3	63.8	58.9	57.1	56.9	57.1	51.5
	Level 4	38.3	35.7	36.9	39.0	35.2	41.9	40.2	44.9	42.8	52.9	61.8	49.9	50.8	49.7	51.5	44.8
	Level 5	26.0	27.9	26.7	31.1	30.1	35.9	34.4	42.2	38.3	45.6	59.6	34.1	33.1	45.1	43.8	36.9

Appendix

Table 10: CDIM accuracy (%) across corruption severity levels 1–5 on ImageNet-C, CIFAR10-C, and CIFAR100-C.

Table 11: Sensitivity of CDIM to (a) the EMA momentum

\alpha

(Eq. ( 9 )) and (b) the discard ratio

\rho

(Eq. ( 5 )), measured by accuracy (%) on CIFAR10-C, CIFAR100-C, and ImageNet-C. The default values are

\alpha=0.9

and

\rho=0.05

(rows 0.9 and 0.05). Bold marks the best value in each column.

Figure 5: Visualization of the mean class-wise representation similarity across different corruptions on CIFAR10-C. Darker colors indicate high similarity, meaning that the model extracts consistent features regardless of the corruption type. Lighter colors indicate low similarity, which reveals feature distortion and representation drift caused by the distribution shift.

Panel of Figure 6 — Figure 6: (a): Class separability $\bar{s}_{\text{within}}-\bar{s}_{\text{between}}$ of the per-batch feature matrix, measured on each of the fifteen CIFAR10-C corruption types; higher values indicate better-separated classes. (b): Reliability diagram for CLIP+TENT+OLA ( red ) and CDIM ( purple ). OLA lowers the ECE from 0.400 (CLIP+TENT) to 0.281, but residual overconfidence remains. CDIM, which adds the DINO v2 ensemble and GT-IM, removes most of it and reaches an ECE of 0.048.

Figure 7: t-SNE visualization of CLIP+TENT and CDIM features on CIFAR10-C for all 15 corruption types, together with an overall results (top right).

Dataset	Domain	ZS	TENT	WATT	BATCLIP	CDIM (Ours)
VisDA-C	3D (trainset)	84.4	84.9	85.4	85.3	86.0
VisDA-C	YT (valset)	84.5	84.7	84.7	84.9	85.5
OfficeHome	Art	78.3	79.4	80.4	79.9	80.7
	Clipart	64.0	63.2	66.9	66.4	68.0
	Product	87.1	88.4	87.5	88.5	87.7
	Real World	89.0	89.6	90.0	89.7	90.1

Appendix

Table 12: Domain generalization results across VisDA-C, OfficeHome, PACS, and VLCS. We denote Top-1/2 by bold and underline , respectively.

Method	ImageNet-A	ImageNet-V2	ImageNet-R	ImageNet-Sketch
CLIP [ 47 ]	49.2	60.4	72.7	44.9
CLIPArTT [ 1 ]	49.6	60.5	72.8	45.0
TDA [ 28 ]	51.0	61.2	73.9	46.4
TENT [ 54 ]	51.9	61.0	77.0	45.4
ETA [ 40 ]	52.0	61.0	77.4	46.8
Mint [ 4 ]	54.7	62.6	78.1	48.4

Appendix

Table 13: Comparison on ImageNet distribution-shift variants (ImageNet-A, -V2, -R, and -Sketch) with CLIP ViT-B/16.

Method		Noise			Blur				Weather				Digital				Mean $\uparrow$
Method		Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elast.	Pixel.	JPEG	Mean $\uparrow$
ImageNet-C	TENT [ 54 ]	5.1	0.3	0.2	19.5	16.1	25.9	25.5	32.0	30.5	37.6	52.8	25.6	14.6	35.0	32.2	23.5
	CoTTA [ 55 ]	17.6	18.3	18.8	17.9	19.2	16.3	28.7	19.0	16.9	17.5	41.5	24.2	26.4	20.8	25.4	21.9
	EATA [ 40 ]	22.8	26.9	27.0	25.7	27.0	30.7	31.6	37.1	37.2	43.3	54.5	33.4	34.6	39.9	39.6	34.1
	SAR [ 41 ]	19.5	21.6	18.2	25.3	21.3	32.9	22.8	32.5	28.8	42.4	48.6	28.6	12.5	39.8	37.8	28.8
	COME [ 61 ]	22.5	26.8	26.7	24.5	26.2	29.9	31.3	36.5	37.3	43.0	54.0	33.7	35.8	40.4	40.1	33.9

Appendix

Table 14: Continual TTA results with ViT-B/16, including per-corruption accuracy. Top-1 and Top-2 are shown in bold and underline , respectively.

Method		Noise			Blur				Weather				Digital				Mean $\uparrow$
Method		Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elast.	Pixel.	JPEG	Mean $\uparrow$
ImageNet-C	Zero Shot	11.1	12.5	12.1	23.3	15.3	24.4	22.7	32.4	29.9	35.9	54.1	17.3	12.7	31.0	33.3	24.5
	TDA [ 28 ]	12.4	14.8	14.8	23.7	16.7	26.1	24.0	33.5	32.1	38.0	55.3	19.3	14.2	33.5	35.0	26.2
	DMN [ 62 ]	11.5	14.1	14.0	22.3	15.8	24.6	22.5	31.5	30.4	35.3	54.2	16.3	12.5	32.5	24.8	24.2
	VTE [ 12 ]	9.2	10.8	10.8	24.7	14.4	24.3	25.2	35.4	32.4	38.2	55.5	16.2	14.3	38.7	34.0	25.6
	ZERO [ 13 ]	10.5	11.5	11.1	24.8	14.9	24.4	23.3	33.0	30.4	37.1	54.5	16.9	13.4	34.9	32.9	24.9

Appendix

Table 15: Online TTA results with ViT-B/16, including additional baselines. Top-1 and Top-2 are shown in bold and underline , respectively.

Test-Time Adaptation of VLMs

Test-time adaptation of vision-language models uses unlabeled deployment data to improve predictions under domain shifts without retraining.

See all 87 papers

What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective
Jun 12, 2026Jiazhen Huang, Xiao Chen, Zhiming Liu +3VLM Evaluation Test-Time Adaptation
Tsinghua University · Shenzhen Technology University
ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better
Mar 27, 2026Mriganka Nath, Anurag Das, Jiahao Xie +1VLM Hallucination Mitigation Test-Time Adaptation of VLMs
Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning
Jul 4, 2026Siru Jiang, Jian Liang, Ran He +1Ensemble Learning VLM Adaptation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +1

Jun 12, 2026J/K move · Enter open · S save

What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective

Jiazhen Huang, Xiao Chen, Zhiming Liu, Yaru Sun+2

Tsinghua University · Shenzhen Technology University

VLM Evaluation Test-Time Adaptation VLM Adaptation Test-Time Adaptation of VLMs

Open paper AskPDF arXiv

Vision-Language Models (VLMs) such as CLIP have become a standard backbone for open-vocabulary recognition, yet their zero-shot predictions remain vulnerable to distribution shifts encountered at deployment. Test-Time Adaptation (TTA) has recently been extended to CLIP as a lightweight solution, leading to a rapidly growing body of TTA4CLIP methods. However, empirical progress in this area has largely outpaced our understanding of what truly drives adaptation, where their gains originate, and under which shifts they remain reliable. In this paper, we take a step back from the pursuit of state-of-the-art accuracy and conduct a systematic controlled study of TTA4CLIP. We first organize existing methods into three unified paradigms according to what is updated at test time. We then introduce TTABC, an open-source TTA Benchmark for CLIP, which standardizes evaluation protocols and integrates more than 20 representative methods. Our controlled empirical analysis focuses on three key areas. First, we determine the driving factors in parameter-based methods, revealing that adaptation gains are primarily driven by test-time evidence and reliable proxies rather than heavy optimization. Second, we explore evidence utilization beyond heavy parameter tuning, showing that competitive and efficient performance can be achieved through cross- or current-sample evidence and lightweight prototype updates. Finally, we demonstrate that there is no silver bullet for TTA: no single adaptation paradigm is universally optimal, and the preferred paradigm depends on the nature of shift. We hope our benchmark and study provide a clearer understanding of the current TTA4CLIP landscape and establish a foundation for further research. Benchmark and codes are available at https://github.com/walawalagoose/TTABC.

Abstract

Test-Time Adaptation of VLMs

Explore similar work

What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective

ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better

USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning