Test-time adaptation (TTA) has been widely explored in single-label recognition, effectively mitigating distribution shifts, especially when combined with vision-language models. However, real-world images often contain multiple objects, while the more practical multi-label test-time adaptation (MLTTA) has received little attention so far. Recent cache-based TTA methods have shown promising efficiency and effectiveness, yet directly extending them to multi-label scenarios suffers from a one-to-many mapping problem: a shared global representation entangling co-occurring objects is stored as class-wise cache prototypes, inducing dominant-label bias and compromised cache calibration. While introducing region-level cues helps isolate class-specific evidence, such regional evidence can also be unreliable under distribution shifts, making its identification and utilization non-trivial. To address these issues, we introduce PuRF, a novel PuRiFication-driven cache-based method for multi-label test-time adaptation of vision-language models. Specifically, PuRF first performs region purification to identify reliable regions, providing comprehensive regional cues for multi-label recognition and enabling fine-grained alignment. Based on these purified regions, PuRF conducts cache purification to enhance cache representation and adaptability, where episodic purification builds a discriminative region-based cache, and temporal refreshing further promotes long-term cache adaptability. Experiments demonstrate that PuRF consistently outperforms state-of-the-art methods, achieving a notable 4.05% mAP improvement on ViT-B/32 across five datasets.
Figures & tables
Figure 1: Comparison between existing cache-based methods and ours. (a) Existing cache-based TTA methods [ 23 , 65 , 29 ] suffer from one-to-many representation coupling in multi-label TTA, where shared global features are dominated by salient classes (e.g., “water”), misleading class-wise cache calibration. (b) PuRF performs region purification and cache purification, enabling fine-grained and accurate multi-label adaptation.
Figure 2: The overall framework of our proposed PuRF. We first adaptively identify purified regions for prediction aggregation and then derive reliable pseudo supervision by enforcing global-local prediction consistency. Guided by the pseudo labels, confident region-text pairs are stored as class-specific region prototypes for episodic cache purification, improving class-wise discriminability. Temporal refreshing further purifies the cache by incorporating newly reliable samples while maintaining long-term adaptivity of cache. H denotes the prediction entropy.
Figure 3: Empirical studies of cache dynamics on COCO2014. (a): The cache turnover rate measures the frequency of cache updates, where Nin(t) and Nout(t) denote the numbers of samples inserted and removed at adaptation step t , respectively. (b): The evolution of model performance during adaptation.
ResNet-50
ResNet-101
Methods
VOC07
VOC12
COCO14
COCO17
NUS
Avg.
VOC07
VOC12
COCO14
COCO17
NUS
Avg.
CLIP [ 41 ] ICML’21
75.91
74.25
47.53
47.32
41.53
57.31
76.72
74.21
48.83
48.15
41.93
57.97
TPT [ 45 ] NeurIPS’22
75.54
73.92
48.52
48.51
41.97
57.69
74.82
73.39
49.71
48.89
43.10
57.98
DiffTPT [ 13 ] CVPR’23
75.89
74.13
48.56
48.67
41.33
57.72
74.98
74.31
49.45
49.19
42.93
58.17
RLCF [ 71 ] ICLR’24
65.75
64.73
36.87
36.73
29.83
46.78
71.21
69.63
40.53
39.79
31.77
50.59
ML-TTA [ 57 ] ICLR’25
78.62
76.63
51.58
51.39
42.53
60.15
78.72
78.13
52.92
52.24
43.62
61.13
Table 1: Comparison with state-of-the-art methods under different visual encoder architectures w.r.t. mAP (%). Bold indicates best results.
Methods
VOC2007
VOC2012
COCO2014
COCO2017
NUS-WIDE
Avg.
CoOp
CoOp [ 72 ] IJCV’22
79.14
77.85
56.12
56.35
46.74
63.24
TPT [ 45 ] NeurIPS’22
79.72
77.85
55.35
55.23
47.27
63.08
DiffTPT [ 13 ] CVPR’23
79.86
77.61
55.30
55.47
47.13
63.07
RLCF [ 71 ] ICLR’24
80.15
78.24
56.72
56.18
47.62
63.78
ML-TTA [ 57 ] ICLR’25
83.17
81.36
59.68
59.33
48.12
66.33
Table 2: Comparison with state-of-the-art methods under different prompt initializations on ViT-B/16 w.r.t. mAP (%). Bold indicates best results.
Method
Inference Speed (FPS) ↑
Memory (GB) ↓
mAP (%) ↑
Gain (%) ↑
CLIP [ 41 ]
106.25
0.35
62.61
-
TPT [ 45 ]
3.59
5.77
61.72
-0.89
ML-TTA [ 57 ]
3.05
5.77
64.80
2.19
DPE [ 65 ]
4.78
7.49
67.02
4.41
ReTA [ 29 ]
6.35
0.74
67.89
5.28
PuRF (Ours)
5.74
1.04
71.48
8.87
Table 3: Efficiency and performance over five multi-label datasets on a single NVIDIA 3090 GPU (ViT-B/16).
Method
VOC2007
VOC2012
COCO2017
Avg.
EVA-02 [ 12 ] (ViT-B/16)
No Adapt.
84.05
83.88
57.88
75.27
DPE [ 65 ]
87.19
86.49
61.22
78.30
ReTA [ 29 ]
86.76
86.23
61.15
78.05
PuRF (Ours)
89.65
88.43
66.59
81.56
SigLIP2 [ 48 ] (SO400M/14)
No Adapt.
88.35
88.09
67.32
81.25
DPE [ 65 ]
89.40
88.98
68.65
82.34
Table 4: Comparison with state-of-the-art cache-based methods across diverse VLM backbones. Bold indicates best results.
RP (region)
EP (cache)
TR (cache)
VOC2007
COCO2017
1
84.83
60.71
2
✓
86.67
63.35
3
✓
85.79
62.59
4
✓
✓
87.50
64.56
5
✓
✓
87.36
64.12
Table 5: Ablation of key components on ViT-B/16.
Table 9
Figure 4: Impact of hyper-parameters. (a) Variation of κg in Eq. ( 5 ). (b) Variation of δ in Eq. ( 15 ). (c) Joint impact of λ1 and λ2 in Eq. ( 18 ) on COCO2017.
Figure 5: t-SNE visualization of cached visual features on COCO2017. The capacity of all caches is set to 7, and the visualization is taken at the 2000-th adaptation step.
Figure 6: Visualization of class activation maps for region and cache purification.
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen, China
Guangzhou University Guangzhou, China · The Second Affiliated Hospital of Guangzhou University of Chinese Medicine Guangzhou, China · Jinan University Zhuhai, China +1