Test-time adaptation (TTA) has been widely explored in single-label recognition, effectively mitigating distribution shifts, especially when combined with vision-language models. However, real-world images often contain multiple objects, while the more practical multi-label test-time adaptation (MLTTA) has received little attention so far. Recent cache-based TTA methods have shown promising efficiency and effectiveness, yet directly extending them to multi-label scenarios suffers from a one-to-many mapping problem: a shared global representation entangling co-occurring objects is stored as class-wise cache prototypes, inducing dominant-label bias and compromised cache calibration. While introducing region-level cues helps isolate class-specific evidence, such regional evidence can also be unreliable under distribution shifts, making its identification and utilization non-trivial. To address these issues, we introduce PuRF, a novel PuRiFication-driven cache-based method for multi-label test-time adaptation of vision-language models. Specifically, PuRF first performs region purification to identify reliable regions, providing comprehensive regional cues for multi-label recognition and enabling fine-grained alignment. Based on these purified regions, PuRF conducts cache purification to enhance cache representation and adaptability, where episodic purification builds a discriminative region-based cache, and temporal refreshing further promotes long-term cache adaptability. Experiments demonstrate that PuRF consistently outperforms state-of-the-art methods, achieving a notable 4.05% mAP improvement on ViT-B/32 across five datasets.
Figures & tables
Figure 1: Comparison between existing cache-based methods and ours. (a) Existing cache-based TTA methods [ 23 , 65 , 29 ] suffer from one-to-many representation coupling in multi-label TTA, where shared global features are dominated by salient classes (e.g., “water”), misleading class-wise cache calibration. (b) PuRF performs region purification and cache purification, enabling fine-grained and accurate multi-label adaptation.
Figure 2: The overall framework of our proposed PuRF. We first adaptively identify purified regions for prediction aggregation and then derive reliable pseudo supervision by enforcing global-local prediction consistency. Guided by the pseudo labels, confident region-text pairs are stored as class-specific region prototypes for episodic cache purification, improving class-wise discriminability. Temporal refreshing further purifies the cache by incorporating newly reliable samples while maintaining long-term adaptivity of cache. H denotes the prediction entropy.
Figure 3: Empirical studies of cache dynamics on COCO2014. (a): The cache turnover rate measures the frequency of cache updates, where Nin(t) and Nout(t) denote the numbers of samples inserted and removed at adaptation step t , respectively. (b): The evolution of model performance during adaptation.
ResNet-50
ResNet-101
Methods
VOC07
VOC12
COCO14
COCO17
NUS
Avg.
VOC07
VOC12
COCO14
COCO17
NUS
Avg.
CLIP [ 41 ] ICML’21
75.91
74.25
47.53
47.32
41.53
57.31
76.72
74.21
48.83
48.15
41.93
57.97
TPT [ 45 ] NeurIPS’22
75.54
73.92
48.52
48.51
41.97
57.69
74.82
73.39
49.71
48.89
43.10
57.98
DiffTPT [ 13 ] CVPR’23
75.89
74.13
48.56
48.67
41.33
57.72
74.98
74.31
49.45
49.19
42.93
58.17
RLCF [ 71 ] ICLR’24
65.75
64.73
36.87
36.73
29.83
46.78
71.21
69.63
40.53
39.79
31.77
50.59
ML-TTA [ 57 ] ICLR’25
78.62
76.63
51.58
51.39
42.53
60.15
78.72
78.13
52.92
52.24
43.62
61.13
Table 1: Comparison with state-of-the-art methods under different visual encoder architectures w.r.t. mAP (%). Bold indicates best results.
Methods
VOC2007
VOC2012
COCO2014
COCO2017
NUS-WIDE
Avg.
CoOp
CoOp [ 72 ] IJCV’22
79.14
77.85
56.12
56.35
46.74
63.24
TPT [ 45 ] NeurIPS’22
79.72
77.85
55.35
55.23
47.27
63.08
DiffTPT [ 13 ] CVPR’23
79.86
77.61
55.30
55.47
47.13
63.07
RLCF [ 71 ] ICLR’24
80.15
78.24
56.72
56.18
47.62
63.78
ML-TTA [ 57 ] ICLR’25
83.17
81.36
59.68
59.33
48.12
66.33
Table 2: Comparison with state-of-the-art methods under different prompt initializations on ViT-B/16 w.r.t. mAP (%). Bold indicates best results.
Method
Inference Speed (FPS) ↑
Memory (GB) ↓
mAP (%) ↑
Gain (%) ↑
CLIP [ 41 ]
106.25
0.35
62.61
-
TPT [ 45 ]
3.59
5.77
61.72
-0.89
ML-TTA [ 57 ]
3.05
5.77
64.80
2.19
DPE [ 65 ]
4.78
7.49
67.02
4.41
ReTA [ 29 ]
6.35
0.74
67.89
5.28
PuRF (Ours)
5.74
1.04
71.48
8.87
Table 3: Efficiency and performance over five multi-label datasets on a single NVIDIA 3090 GPU (ViT-B/16).
Method
VOC2007
VOC2012
COCO2017
Avg.
EVA-02 [ 12 ] (ViT-B/16)
No Adapt.
84.05
83.88
57.88
75.27
DPE [ 65 ]
87.19
86.49
61.22
78.30
ReTA [ 29 ]
86.76
86.23
61.15
78.05
PuRF (Ours)
89.65
88.43
66.59
81.56
SigLIP2 [ 48 ] (SO400M/14)
No Adapt.
88.35
88.09
67.32
81.25
DPE [ 65 ]
89.40
88.98
68.65
82.34
Table 4: Comparison with state-of-the-art cache-based methods across diverse VLM backbones. Bold indicates best results.
RP (region)
EP (cache)
TR (cache)
VOC2007
COCO2017
1
84.83
60.71
2
✓
86.67
63.35
3
✓
85.79
62.59
4
✓
✓
87.50
64.56
5
✓
✓
87.36
64.12
Table 5: Ablation of key components on ViT-B/16.
Table 9
Figure 4: Impact of hyper-parameters. (a) Variation of κg in Eq. ( 5 ). (b) Variation of δ in Eq. ( 15 ). (c) Joint impact of λ1 and λ2 in Eq. ( 18 ) on COCO2017.
Figure 5: t-SNE visualization of cached visual features on COCO2017. The capacity of all caches is set to 7, and the visualization is taken at the 2000-th adaptation step.
Figure 6: Visualization of class activation maps for region and cache purification.
Cache-based test-time adaptation (TTA) for vision-language models is often hindered by background bias in global representations and unreliable entropy-based cache admission under representation variations. To address these limitations, we propose RADC, which enhances prototype learning through reliable dual caching. RADC introduces a Semantic Foreground Cache that aggregates category-consistent spatial evidence from CLIP representations, yielding foreground prototypes that complement the global cache while mitigating background interference. To reliably manage both caches, Gaussian Risk Admission models multi-view representations as diagonal Gaussian distributions and jointly considers class separation and feature uncertainty to prioritize reliable cache candidates. RADC integrates zero-shot logits with complementary global- and foreground-cache predictions for robust inference. Extensive experiments on cross-domain and out-of-distribution benchmarks demonstrate consistent state-of-the-art performance.
Siyu Huang, Yueyong Chen, Xuejiao Li +1
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen, China
Test-time adaptation (TTA) has emerged as a promising paradigm for vision-language models (VLMs) to bridge the distribution gap between pre-training and test data. Recent works have focused on backpropagation-free TTA methods that rely on cache-based designs, but these introduce two key limitations. First, inference latency increases as the cache grows with the number of classes, leading to inefficiencies in large-scale settings. Second, suboptimal performance occurs when the cache contains insufficient or incorrect samples. In this paper, we present Prototype-Based Test-Time Adaptation (PTA), an efficient and effective TTA paradigm that uses a set of class-specific knowledge prototypes to accumulate knowledge from test samples. Particularly, knowledge prototypes are adaptively weighted based on the zero-shot class confidence of each test sample, incorporating the sample's visual features into the corresponding class-specific prototype. It is worth highlighting that the knowledge from past test samples is integrated and utilized solely in the prototypes, eliminating the overhead of cache population and retrieval that hinders the efficiency of existing TTA methods. This endows PTA with extremely high efficiency while achieving state-of-the-art performance on 15 image recognition benchmarks and 4 robust point cloud analysis benchmarks. For example, PTA improves CLIP's accuracy from 65.64% to 69.38% on 10 cross-domain benchmarks, while retaining 92% of CLIP's inference speed on large-scale ImageNet-1K. In contrast, the cache-based TDA achieves a lower accuracy of 67.97% and operates at only 50% of CLIP's inference speed.
Zhaohong Huang, Yuxin Zhang, Wenjing Liu +2
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
Vision-language models (VLMs) such as CLIP exhibit remarkable zero-shot capabilities, yet their performance frequently degrades sharply under unexpected test-time distribution shifts. While Test-Time Adaptation (TTA) offers a promising solution, continuously adapting VLMs over an unlabeled test stream presents fundamental challenges. Conventional top-1-centric updates often reinforce errors by corrupting the local semantic geometry among related classes, while iterative adaptation exacerbates progressive bias accumulation, ultimately driving the model toward mode collapse. To overcome these coupled vulnerabilities, we propose Local Margin Restoration (LMR), a lightweight, one-step TTA framework. At the sample level, our Protected Margin Restoration (PMR) objective recovers local semantic geometry by shielding plausible near-top candidates from external hard negatives. Concurrently, to combat stream-level degradation, we introduce a dual-stage stabilization mechanism, featuring an Adaptive Margin (AM) controller and Bias Correction (BC), to dynamically disrupt progressive bias accumulation and prevent mode collapse. Extensive experiments on CIFAR-C, ImageNet-C, and ImageNet variants demonstrate that LMR consistently outperforms state-of-the-art TTA baselines, proving exceptionally robust and efficient even in challenging low-batch test-time regimes. Our code is available at https://github.com/DennisHuangYan/LMR.
Yan Huang, Guowei Wang, Xu Wang +2
Guangzhou University Guangzhou, China · The Second Affiliated Hospital of Guangzhou University of Chinese Medicine Guangzhou, China · Jinan University Zhuhai, China +1