RADC: Risk-Aware Dual Caching for Vision-Language Test-Time Adaptation

Siyu Huang; Yueyong Chen; Xuejiao Li; Jun Zhou

cs.CVOct 3, 2026

RADC: Risk-Aware Dual Caching for Vision-Language Test-Time Adaptation

Authors: Siyu Huang, Yueyong Chen, Xuejiao Li, Jun Zhou

Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen, China

Abstract

Cache-based test-time adaptation (TTA) for vision-language models is often hindered by background bias in global representations and unreliable entropy-based cache admission under representation variations. To address these limitations, we propose RADC, which enhances prototype learning through reliable dual caching. RADC introduces a Semantic Foreground Cache that aggregates category-consistent spatial evidence from CLIP representations, yielding foreground prototypes that complement the global cache while mitigating background interference. To reliably manage both caches, Gaussian Risk Admission models multi-view representations as diagonal Gaussian distributions and jointly considers class separation and feature uncertainty to prioritize reliable cache candidates. RADC integrates zero-shot logits with complementary global- and foreground-cache predictions for robust inference. Extensive experiments on cross-domain and out-of-distribution benchmarks demonstrate consistent state-of-the-art performance.

Figures & tables

Figure 1: Overview of RADC. (a) RADC complements the global cache with SFC and combines zero-shot, global, and foreground logits for prediction. (b) SFC localizes category-consistent regions from spatial CLIP features to construct foreground queries. (c) GRA evaluates class separation and feature uncertainty to manage both global and foreground caches.

Method	Aircraft	Caltech	Cars	DTD	EuroSAT	Flowers102	Food101	Pets	SUN397	UCF101	Average
CLIP-RN50 [ 18 ]	15.66	85.88	55.70	40.37	23.69	61.75	73.97	83.57	58.80	58.84	55.82
CoOp [ 30 ]	15.12	86.53	55.32	37.29	26.20	61.55	75.59	87.00	58.15	59.05	56.18
TPT [ 20 ]	17.58	87.02	58.46	40.84	28.33	62.69	74.88	84.49	61.46	60.82	57.66
DiffTPT [ 5 ]	17.60	86.89	60.71	40.72	41.04	63.53	79.21	83.40	62.72	62.67	59.85
DMN-ZS [ 29 ]	22.77	90.14	60.02	50.41	48.72	67.93	76.70	86.78	64.39	65.34	63.32
TDA [ 12 ]	17.61	89.70	57.78	43.74	42.11	68.74	77.75	86.18	62.53	64.18	61.03

Table 1: Top-1 accuracy (%) on cross-dataset generalization using ResNet-50 and ViT-B/16 CLIP visual backbones. The best result in each column is highlighted in bold , while the second-best result is underlined .

Method	ImageNet	ImageNet-A	ImageNet-V2	ImageNet-R	ImageNet-S	Average	OOD Average
CLIP-RN50 [ 18 ]	58.16	21.83	51.41	56.15	33.37	44.18	40.69
CoOp [ 30 ]	63.33	23.06	55.40	56.60	34.67	46.61	42.43
TPT [ 20 ]	60.74	26.67	54.70	59.11	35.09	47.26	43.89
DiffTPT [ 5 ]	60.80	31.06	55.80	58.80	37.10	48.71	45.69
DMN-ZS [ 29 ]	63.87	28.57	56.12	61.44	39.84	49.97	46.49
TDA [ 12 ]	61.35	30.29	55.54	62.58	38.12	49.58	46.63

Table 2: Top-1 accuracy (%) on robustness to natural distribution shifts using ResNet-50 and ViT-B/16 CLIP visual backbones. The best result in each column is highlighted in bold , while the second-best result is underlined .

Backbone	SFC	GRA	Flowers102	Pets	Cars	Average
RN50	–	–	67.60	85.97	59.26	70.94
	$\checkmark$	–	69.43	87.90	62.34	73.22
	–	$\checkmark$	69.75	87.54	61.97	73.09
	$\checkmark$	$\checkmark$	69.91	88.01	62.72	73.55
ViT-B/16	–	–	75.07	86.17	67.31	76.18
	$\checkmark$	–	78.12	92.40	70.41	80.31

Table 3: Ablation study of SFC and GRA on Flowers102, Pets, and Cars.

Panel of Figure 2 — Figure 2: Hyperparameter sensitivity of RADC using ViT-B/16. Left: foreground blending weight $\lambda_{b}$ . Right: global and foreground cache capacities.

Method	Time	Memory	Accuracy	Gain
CLIP-ViT-B/16 [ 18 ]	1.13 min	1122 MiB	47.87	0
DPE [ 25 ]	19.40 min	6466 MiB	59.63	+11.76
SCA [ 7 ]	7.14 min	988 MiB	60.33	+12.46
RADC (Ours)	12.26 min	2084 MiB	64.63	+16.76

Table 4: Comparison with state-of-the-art methods on inference efficiency on ImageNet-A.

Test-Time Adaptation of VLMs

Test-time adaptation of vision-language models uses unlabeled deployment data to improve predictions under domain shifts without retraining.

See all 79 papers

Prototype-Based Test-Time Adaptation of Vision-Language Models
Apr 23, 2026Zhaohong Huang, Yuxin Zhang, Wenjing Liu +2Test-Time Adaptation VLM Adaptation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models
Sep 8, 2026Siru Jiang, Yuwei Liang, Jian Liang +2Test-Time Adaptation VLM Adaptation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China +1
Local Margin Restoration for Test-Time Adaptation of Vision-Language Models
Aug 3, 2026Yan Huang, Guowei Wang, Xu Wang +2VLM Robustness Test-Time Adaptation
Guangzhou University Guangzhou, China · The Second Affiliated Hospital of Guangzhou University of Chinese Medicine Guangzhou, China · Jinan University Zhuhai, China +1

Apr 23, 2026J/K move · Enter open · S save

Prototype-Based Test-Time Adaptation of Vision-Language Models

Zhaohong Huang, Yuxin Zhang, Wenjing Liu, Fei Chao+1

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.

Test-Time Adaptation VLM Adaptation Test-Time Adaptation of VLMs Efficient VLM Inference

Open paper AskPDF arXiv

Test-time adaptation (TTA) has emerged as a promising paradigm for vision-language models (VLMs) to bridge the distribution gap between pre-training and test data. Recent works have focused on backpropagation-free TTA methods that rely on cache-based designs, but these introduce two key limitations. First, inference latency increases as the cache grows with the number of classes, leading to inefficiencies in large-scale settings. Second, suboptimal performance occurs when the cache contains insufficient or incorrect samples. In this paper, we present Prototype-Based Test-Time Adaptation (PTA), an efficient and effective TTA paradigm that uses a set of class-specific knowledge prototypes to accumulate knowledge from test samples. Particularly, knowledge prototypes are adaptively weighted based on the zero-shot class confidence of each test sample, incorporating the sample's visual features into the corresponding class-specific prototype. It is worth highlighting that the knowledge from past test samples is integrated and utilized solely in the prototypes, eliminating the overhead of cache population and retrieval that hinders the efficiency of existing TTA methods. This endows PTA with extremely high efficiency while achieving state-of-the-art performance on 15 image recognition benchmarks and 4 robust point cloud analysis benchmarks. For example, PTA improves CLIP's accuracy from 65.64% to 69.38% on 10 cross-domain benchmarks, while retaining 92% of CLIP's inference speed on large-scale ImageNet-1K. In contrast, the cache-based TDA achieves a lower accuracy of 67.97% and operates at only 50% of CLIP's inference speed.

Abstract

Test-Time Adaptation of VLMs

Explore similar work

Prototype-Based Test-Time Adaptation of Vision-Language Models

To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models

Local Margin Restoration for Test-Time Adaptation of Vision-Language Models