Organizations: VCIP, College of Computer Science, Nankai University, China · Australian Institute for Machine Learning · School of Intelligence Science and Technology, Nanjing University, China · NKIARI, Shenzhen Futian, Nankai University, China · AAIS, Nankai University, China
When a vision-language model encounters a new class or domain, how much should it trust its learned prompts? Because source adaptation can correct pretrained errors on some target images while disturbing valid predictions on others, we introduce Contextual Adaptive Fusion (CAF), which retains four prediction branches by pairing frozen and prompted visual and textual representations. To arbitrate when branches disagree on an image, CAF combines each branch's predictive confidence with how frequently the branches favor its proposed class across the unlabeled target collection, placing greater trust in predictions that are locally confident yet less dominated by collection-wide frequency bias. The fusion coefficients are learned once on source data and remain fixed at inference. Trained on ImageNet, CAF achieves 69.96% average accuracy across ten cross-dataset benchmarks (1.92 percentage points above ManiPT) and remains highly competitive across four ImageNet variants, setting a new best of 52.21% on ImageNet-S. In base-to-novel transfer across eleven datasets, CAF further reaches 79.77% novel-class accuracy and an 82.81% harmonic mean. Code is available at https://github.com/zhenyuanchenai/CAF.
Figures & tables
Figure 1: Comparison of CLIP prompting strategies: (a) text prompting in CoOp [ 50 ] ; (b) frozen-reference regularization in PromptSRC [ 19 ] ; (c) fixed feature fusion in ManiPT [ 42 ] ; and (d) contextual fusion in CAF, where instance-level confidence and collection-level context determine image-dependent weights for four prediction branches. All backbone encoders remain frozen.
Figure 2: Overview of CAF. (a) Frozen and prompted visual and textual representations form four prediction branches. (b) Normalized entropy measures instance-level uncertainty, while class probabilities averaged across images and branches provide collection-level context. (c) Two source-learned coefficients combine these signals into image-dependent weights that fuse the original logits over all classes. Prompt learning and fusion-coefficient learning use separate gradients; all learned parameters remain fixed at inference.
Source
Cross-dataset
Cross-domain
Method
ImageNet (IN)
Caltech 101
Oxford Pets
Stanford Cars
Flowers 102
Food 101
FGVC Aircraft
SUN 397
DTD
EuroSAT
UCF 101
Avg-10
IN-V2
IN-S
IN-A
IN-R
Avg-4
CoOp [ 50 ]
71.51
93.70
89.10
64.50
68.70
85.30
18.50
64.20
41.90
46.40
66.60
63.90
64.20
47.99
49.71
75.21
59.28
CoCoOp [ 49 ]
71.02
94.40
90.10
65.30
71.90
86.10
22.90
67.40
45.70
45.40
68.20
65.70
64.07
48.75
50.63
76.18
59.91
TPT [ 35 ]
68.98
94.10
87.70
66.80
68.90
84.60
24.70
65.50
47.70
42.40
68.00
65.10
63.45
47.94
54.77
77.06
60.81
VPT [ 8 ]
70.70
93.70
90.60
65.00
70.90
86.30
24.90
67.50
46.10
45.90
68.70
66.00
64.23
49.20
51.33
77.00
60.44
PromptSRC [ 19 ]
71.27
93.60
90.30
65.70
70.30
86.20
24.00
67.10
46.90
45.50
68.80
65.80
64.35
49.55
50.90
77.80
60.65
Table 1: Cross-dataset and cross-domain evaluation. The backbone is ViT-B/16; accuracies are percentages. CAF results are mean ± sample standard deviation over three seeds, with standard deviations placed below the means in smaller type. The ImageNet column reports source accuracy. The two averages cover ten cross-dataset targets and four ImageNet variants, respectively. The best mean accuracy in each column is shown in bold.
Method
(a) Average
(b) ImageNet
(c) Caltech101
(d) OxfordPets
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
CLIP [ 28 ]
69.34
74.22
71.70
72.43
68.14
70.22
96.84
94.00
95.40
91.17
97.26
94.12
CoOp [ 50 ]
82.69
63.22
71.66
76.47
67.88
71.92
98.00
89.81
93.73
93.67
95.29
94.47
CoCoOp [ 49 ]
80.47
71.69
75.83
75.98
70.43
73.10
97.96
93.81
95.84
95.20
97.69
96.43
PromptSRC [ 19 ]
84.26
76.10
79.97
77.60
70.73
74.01
98.10
94.03
96.02
95.33
97.30
96.30
CoPrompt [ 33 ]
84.00
77.23
80.48
77.67
71.27
74.33
98.27
94.90
96.55
95.67
98.10
96.87
Table 2: Base-to-novel generalization on 11 datasets. Subscripts give sample standard deviations over three seeds, using seed-level HMs and dataset averages. ∗ denotes dataset-specific loss weights. Bold and underline mark the best and second-best results.
Table 3: CAF ablations. Accuracies (%) on ImageNet (IN), ten cross-dataset targets (Avg-10), and four cross-domain targets (Avg-4). Time denotes median end-to-end inference time in seconds; Rel. is its ratio to the confidence-only baseline. ↑ / ↓ indicate that higher/lower is better.
Figure 3: Adaptive fusion across transfer settings. (a,b) Accuracy by branch disagreement across ten datasets. Braces give differences in percentage points. (c) Mean branch weights on OxfordPets and EuroSAT. Outlines mark the largest weight in each row.
Dataset
Task / domain
Classes
Train
Test
Standard benchmarks
ImageNet [ 7 ]
Objects
1,000
1,281,167
50,000
Caltech101 [ 11 ]
Objects
100
4,128
2,465
OxfordPets [ 27 ]
Fine-grained
37
2,944
3,669
StanfordCars [ 20 ]
Fine-grained
196
6,509
8,041
Flowers102 [ 26 ]
Fine-grained
102
4,093
2,463
Table A1 : Dataset statistics. Training counts exclude validation images and precede few-shot sampling and base/novel partitioning. A dash indicates a target-only dataset.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset(s)
λmargin
λtext
ImageNet, EuroSAT
1
2.5
Caltech101
0.5
0
OxfordPets
2
20
StanfordCars, UCF101
0
2.5
Flowers102
4
2.5
Food101
4
40
Appendix
Table A2 : Dataset-specific loss weights used by CAF ∗ for base-to-novel generalization in Tab. 2 .
Prediction
ImageNet ↑
Avg-10 ↑
Avg-4 ↑
00
69.642
66.873
60.024
V0
70.882
66.272
59.871
0T
71.220
66.766
60.762
VT
72.184
65.884
60.487
Uniform fusion
71.616
67.517
60.962
CAF (ours)
72.263
69.955
61.361
Appendix
Table B1 : Individual branches and prediction fusion. Branch and uniform-fusion accuracies (%) share one historical checkpoint; CAF repeats the learned-coefficient, three-seed means from Tab. 1 .
Statistics
Available images
Avg-10 ↑
Avg-4 ↑
Full collection
All N
69.524
61.649
Leave-one-out
N−1
69.530
61.650
Two-fold cross-fit
Other fold ( ≈N/2 )
69.532
61.567
Budget: 256
≤min(256,N)
68.917
60.597
Budget: 1,024
≤min(1024,N)
69.210
61.056
Budget: 4,096
≤min(4096,N)
69.462
61.462
Appendix
Table B2 : Unlabeled statistics: scope and budget. Single-seed accuracies (%) with a fixed prompt checkpoint and fusion coefficients. N denotes the collection size; budgeted references exclude the query image.
Figure B1 : Regularization-weight results. Accuracy (%) for (a) cross-dataset and (b) cross-domain transfer, with separate color scales. Results combine historical fixed-coefficient and learned-coefficient runs. Outlined cells at (1,40) report the three-seed means from Tab. 1 ; other cells report individual runs.
Figure B2 : Regularization weights for base-to-novel generalization. Accuracy (%) averaged over ten datasets and three seeds: (a) base classes using the VT branch and (b) novel classes using fusion, with separate color scales. Outlined cells mark the shared (4,2.5) setting used for CAF in Tab. 2 .
Weight assignment
Avg-10 ↑
Avg-4 ↑
Original (Branch-soft)
69.405
61.641
Dataset-mean weights
67.670
61.097
Predicted-class mean weights
68.058
61.137
Shuffle across all images
67.058
60.827
Shuffle within disagreement samples
67.071
60.666
Shared-soft CAF (reference)
69.524
61.649
Appendix
Table C1 : Dynamic weight assignment interventions. Interventions use Branch-soft weights from one checkpoint; shuffled rows average 20 permutations. Shared-soft CAF is a separate reference. Accuracies are percentages.
Test-time adaptation (TTA) has emerged as a prominent strategy for adapting vision-language models to distribution shifts during inference. We conduct a per-sample analysis of model predictions before and after adaptation, and observe two failure modes in existing TTA methods that echo previous work. Adaptations are frequently negligible, yielding no change in the model's predictions, and more severely, they can be detrimental by flipping previously correct predictions to incorrect ones. This naturally raises a question: Can we identify and skip such negligible or harmful adaptations? In this work, we introduce a new problem of selective adaptation, which aims to determine whether a given test sample should undergo adaptation or be skipped. To this end, we propose Cross-Augmentation Similarity (CAS), a simple baseline that performs adaptation only when predictions across augmented views exhibit low similarity. Notably, CAS not only preserves but in some cases improves overall accuracy, even when skipping nearly 85% of the adaptation process. We hope other researchers will explore this new direction and surpass the performance of our baseline. Our code is available at https://github.com/sirujiang/selective-adaptation.
Siru Jiang, Yuwei Liang, Jian Liang +2
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China +1
Efficient transfer learning of vision-language models (VLMs) commonly suffers from a Base-New Trade-off (BNT): improving performance on unseen (new) classes often degrades accuracy on known (base) classes. Addressing how to boost recognition of unseen classes without sacrificing known-class performance remains a central challenge. Existing work often simplistically attributes the BNT to overfitting on known classes. We observe an interesting phenomenon: VLMs frequently exhibit asymmetric confusion on certain downstream data, i.e., samples of class A are systematically mispredicted as class B, while the reverse confusion (B to A) rarely occurs. For known classes, this kind of bias can be mitigated by tuning using a cross-entropy loss, but for unseen classes, such pretraining-induced bias persists and harms generalization. Motivated by this, we propose NeRP, a plug-and-play prompting correction strategy that improves discrimination on unseen classes without modifying model parameters. NeRP leverages neutral text prompts and reference images to measure class-wise prior preferences along the pre-trained inter-class geometry, and combines them with the sample likelihood to obtain the model's surrogate score. If, for a given sample, the prior strongly favors the current prediction while the observed evidence is clearly insufficient, we perform a local flip between easily confusable class pairs, thereby correcting prior-dominated mispredictions. Extensive experiments across multiple backbones and 15 few-shot and cross-domain benchmarks show that NeRP substantially improves accuracy on unseen classes while preserving known-class prediction performance.
Prompt tuning adapts vision--language models with few trainable parameters, but existing approaches trade off efficiency and adaptation: static textual prompts can overfit source classes, image-conditioned prompts add per-instance computation, and multimodal tuning modifies the visual branch. We propose VDA (Visual Distribution Anchoring), a training-free target adaptation framework that augments a frozen semantic classifier with class-level visual prototypes estimated offline from an unlabeled target pool. We first ask whether prototypes can be synthesized from class names. A text-to-centroid mapper reconstructs held-out source prototypes but fails under dataset shift because class names specify semantic identity, not target-domain appearance. An oracle analysis confirms that true target prototypes are highly discriminative. VDA therefore uses frozen semantic and domain-template classifiers to partition unlabeled target images into class-correlated groups. Confidence-ranked image features form normalized prototypes, fused with the semantic classifier using one global weight. Adaptation requires no target labels, target-side optimization, uniform class-prior assumption, iterative refinement, or test-query access, and yields a fixed, cacheable classifier. Controlled experiments show that class-specific partitioning drives gains and that visually local pseudo-label errors can remain useful despite being class-incorrect. Across ten ImageNet-to-target transfers, the same frozen design improves zero-shot CLIP, TCP, and MaPLe by 3.22, 3.39, and 3.35 points, respectively, improving nine of ten targets in every setting. Its visual correction further improves leakage-free PromptKD by 2.79 points, complementing zero-shot, source-prompted, multimodal-prompted, and target-distilled classifiers.