Organizations: VCIP, College of Computer Science, Nankai University, China · Australian Institute for Machine Learning · School of Intelligence Science and Technology, Nanjing University, China · NKIARI, Shenzhen Futian, Nankai University, China · AAIS, Nankai University, China
When a vision-language model encounters a new class or domain, how much should it trust its learned prompts? Because source adaptation can correct pretrained errors on some target images while disturbing valid predictions on others, we introduce Contextual Adaptive Fusion (CAF), which retains four prediction branches by pairing frozen and prompted visual and textual representations. To arbitrate when branches disagree on an image, CAF combines each branch's predictive confidence with how frequently the branches favor its proposed class across the unlabeled target collection, placing greater trust in predictions that are locally confident yet less dominated by collection-wide frequency bias. The fusion coefficients are learned once on source data and remain fixed at inference. Trained on ImageNet, CAF achieves 69.96% average accuracy across ten cross-dataset benchmarks (1.92 percentage points above ManiPT) and remains highly competitive across four ImageNet variants, setting a new best of 52.21% on ImageNet-S. In base-to-novel transfer across eleven datasets, CAF further reaches 79.77% novel-class accuracy and an 82.81% harmonic mean. Code is available at https://github.com/zhenyuanchenai/CAF.
Figures & tables
Figure 1: Comparison of CLIP prompting strategies: (a) text prompting in CoOp [ 50 ] ; (b) frozen-reference regularization in PromptSRC [ 19 ] ; (c) fixed feature fusion in ManiPT [ 42 ] ; and (d) contextual fusion in CAF, where instance-level confidence and collection-level context determine image-dependent weights for four prediction branches. All backbone encoders remain frozen.
Figure 2: Overview of CAF. (a) Frozen and prompted visual and textual representations form four prediction branches. (b) Normalized entropy measures instance-level uncertainty, while class probabilities averaged across images and branches provide collection-level context. (c) Two source-learned coefficients combine these signals into image-dependent weights that fuse the original logits over all classes. Prompt learning and fusion-coefficient learning use separate gradients; all learned parameters remain fixed at inference.
Source
Cross-dataset
Cross-domain
Method
ImageNet (IN)
Caltech 101
Oxford Pets
Stanford Cars
Flowers 102
Food 101
FGVC Aircraft
SUN 397
DTD
EuroSAT
UCF 101
Avg-10
IN-V2
IN-S
IN-A
IN-R
Avg-4
CoOp [ 50 ]
71.51
93.70
89.10
64.50
68.70
85.30
18.50
64.20
41.90
46.40
66.60
63.90
64.20
47.99
49.71
75.21
59.28
CoCoOp [ 49 ]
71.02
94.40
90.10
65.30
71.90
86.10
22.90
67.40
45.70
45.40
68.20
65.70
64.07
48.75
50.63
76.18
59.91
TPT [ 35 ]
68.98
94.10
87.70
66.80
68.90
84.60
24.70
65.50
47.70
42.40
68.00
65.10
63.45
47.94
54.77
77.06
60.81
VPT [ 8 ]
70.70
93.70
90.60
65.00
70.90
86.30
24.90
67.50
46.10
45.90
68.70
66.00
64.23
49.20
51.33
77.00
60.44
PromptSRC [ 19 ]
71.27
93.60
90.30
65.70
70.30
86.20
24.00
67.10
46.90
45.50
68.80
65.80
64.35
49.55
50.90
77.80
60.65
Table 1: Cross-dataset and cross-domain evaluation. The backbone is ViT-B/16; accuracies are percentages. CAF results are mean ± sample standard deviation over three seeds, with standard deviations placed below the means in smaller type. The ImageNet column reports source accuracy. The two averages cover ten cross-dataset targets and four ImageNet variants, respectively. The best mean accuracy in each column is shown in bold.
Method
(a) Average
(b) ImageNet
(c) Caltech101
(d) OxfordPets
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
CLIP [ 28 ]
69.34
74.22
71.70
72.43
68.14
70.22
96.84
94.00
95.40
91.17
97.26
94.12
CoOp [ 50 ]
82.69
63.22
71.66
76.47
67.88
71.92
98.00
89.81
93.73
93.67
95.29
94.47
CoCoOp [ 49 ]
80.47
71.69
75.83
75.98
70.43
73.10
97.96
93.81
95.84
95.20
97.69
96.43
PromptSRC [ 19 ]
84.26
76.10
79.97
77.60
70.73
74.01
98.10
94.03
96.02
95.33
97.30
96.30
CoPrompt [ 33 ]
84.00
77.23
80.48
77.67
71.27
74.33
98.27
94.90
96.55
95.67
98.10
96.87
Table 2: Base-to-novel generalization on 11 datasets. Subscripts give sample standard deviations over three seeds, using seed-level HMs and dataset averages. ∗ denotes dataset-specific loss weights. Bold and underline mark the best and second-best results.
Table 3: CAF ablations. Accuracies (%) on ImageNet (IN), ten cross-dataset targets (Avg-10), and four cross-domain targets (Avg-4). Time denotes median end-to-end inference time in seconds; Rel. is its ratio to the confidence-only baseline. ↑ / ↓ indicate that higher/lower is better.
Figure 3: Adaptive fusion across transfer settings. (a,b) Accuracy by branch disagreement across ten datasets. Braces give differences in percentage points. (c) Mean branch weights on OxfordPets and EuroSAT. Outlines mark the largest weight in each row.
Dataset
Task / domain
Classes
Train
Test
Standard benchmarks
ImageNet [ 7 ]
Objects
1,000
1,281,167
50,000
Caltech101 [ 11 ]
Objects
100
4,128
2,465
OxfordPets [ 27 ]
Fine-grained
37
2,944
3,669
StanfordCars [ 20 ]
Fine-grained
196
6,509
8,041
Flowers102 [ 26 ]
Fine-grained
102
4,093
2,463
Table A1 : Dataset statistics. Training counts exclude validation images and precede few-shot sampling and base/novel partitioning. A dash indicates a target-only dataset.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset(s)
λmargin
λtext
ImageNet, EuroSAT
1
2.5
Caltech101
0.5
0
OxfordPets
2
20
StanfordCars, UCF101
0
2.5
Flowers102
4
2.5
Food101
4
40
Appendix
Table A2 : Dataset-specific loss weights used by CAF ∗ for base-to-novel generalization in Tab. 2 .
Prediction
ImageNet ↑
Avg-10 ↑
Avg-4 ↑
00
69.642
66.873
60.024
V0
70.882
66.272
59.871
0T
71.220
66.766
60.762
VT
72.184
65.884
60.487
Uniform fusion
71.616
67.517
60.962
CAF (ours)
72.263
69.955
61.361
Appendix
Table B1 : Individual branches and prediction fusion. Branch and uniform-fusion accuracies (%) share one historical checkpoint; CAF repeats the learned-coefficient, three-seed means from Tab. 1 .
Statistics
Available images
Avg-10 ↑
Avg-4 ↑
Full collection
All N
69.524
61.649
Leave-one-out
N−1
69.530
61.650
Two-fold cross-fit
Other fold ( ≈N/2 )
69.532
61.567
Budget: 256
≤min(256,N)
68.917
60.597
Budget: 1,024
≤min(1024,N)
69.210
61.056
Budget: 4,096
≤min(4096,N)
69.462
61.462
Appendix
Table B2 : Unlabeled statistics: scope and budget. Single-seed accuracies (%) with a fixed prompt checkpoint and fusion coefficients. N denotes the collection size; budgeted references exclude the query image.
Figure B1 : Regularization-weight results. Accuracy (%) for (a) cross-dataset and (b) cross-domain transfer, with separate color scales. Results combine historical fixed-coefficient and learned-coefficient runs. Outlined cells at (1,40) report the three-seed means from Tab. 1 ; other cells report individual runs.
Figure B2 : Regularization weights for base-to-novel generalization. Accuracy (%) averaged over ten datasets and three seeds: (a) base classes using the VT branch and (b) novel classes using fusion, with separate color scales. Outlined cells mark the shared (4,2.5) setting used for CAF in Tab. 2 .
Weight assignment
Avg-10 ↑
Avg-4 ↑
Original (Branch-soft)
69.405
61.641
Dataset-mean weights
67.670
61.097
Predicted-class mean weights
68.058
61.137
Shuffle across all images
67.058
60.827
Shuffle within disagreement samples
67.071
60.666
Shared-soft CAF (reference)
69.524
61.649
Appendix
Table C1 : Dynamic weight assignment interventions. Interventions use Branch-soft weights from one checkpoint; shuffled rows average 20 permutations. Shared-soft CAF is a separate reference. Accuracies are percentages.
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China +1