Organizations: Department of Computer Science, Qufu Normal University, China, and with Climind · Institute of Scientific and Industrial Research (SANKEN), The University of Osaka, Japan · Institute of High Performance Computing (IHPC), A*STAR, Singapore · Standard Chartered Bank, Singapore · School of Aerospace Intelligence, Hainan University, China · School of Computer Science, Qufu Normal University, China
Many few-shot adaptation methods for vision-language models classify with a convex combination of the zero-shot text prototype and the mean of the K labelled image features, with a single blending ratio routinely tuned on held-out labels, often on the test set itself. We ask what the family's own bias-variance justification invites: what is the right ratio, can it be estimated without validation data, and is finding it where the performance is? First, the ratio minimising prototype mean-squared error has a closed form whose support-set plug-in is exactly a positive-part James-Stein coefficient shrinking towards the text prototype. Across 4,800 cells (ten datasets, five backbones including SigLIP, five shot counts, five seeds, four prompt tiers) this theoretically optimal ratio is a reliable estimate of the wrong quantity: on the 950 primary-tier cells where it is defined it trails a test-set-oracle ratio by 8.5 points. It saturates near 1, discarding the text prior for a nearest-class-mean classifier, because 78% of the text-image prototype distance it treats as bias is a class-independent offset that the arg max largely cancels. We prove the mechanism and bound its share of the damage at 26% by a counterfactual. Second, leave-one-out on the support set alone sets a ratio landing within 0.9 points of the oracle blend, so it is estimable without validation data. Third, validation-free linear probes beat even the oracle-tuned blend: CLAP by +1.9 points and LP++ by +1.5 on average, and at K >= 4 all four validation-free baselines sit above the oracle, the linear probes by margins excluding zero. These results locate the ceiling in the model class, not the hyperparameter: the ratio can be set near-optimally for free, and it is still not where the performance is. Code, cached features, per-cell records: https://huggingface.co/datasets/Liangzhi-Li/clipbench-blending
Figures & tables
Fig. 1: A: PCA of the prototype geometry (Pets, ViT-B/16): the common offset Δ (the modality gap) is 75% of the squared text–image prototype distance here, yet costs far less at decision time (Thm. 1 ; Sec. VI-G ). Blending slides each prototype along its dotted path; marks are the two objectives’ choices. B: measured accuracy and prototype MSE vs. λ (Food-101, K=4 ): the MSE minimum ( λ≈0.96 , where Eq. ( 7 ) sits) costs 14.2 accuracy points against the accuracy peak ( λ≈0.09 , which leave-one-out finds). C: mean accuracy, K≥4 , balanced cells: every reproduced validation-free baseline exceeds the test-set-oracle blend (dashed): the family is capacity-limited, not mis-tuned.
Dataset
C
ntest
Pool split
ZS ens. (%)
Published
Aircraft
100
3,333
official
24.3
20–32
Caltech101
102
6,085
official
82.2
80–96
Cars
196
8,041
official
64.7
60–70
DTD
47
1,880
official
44.9
40–50
EuroSAT
10
5,400
official
55.9
36–62
Flowers102
102
6,149
official
71.2
65–78
TABLE I: Benchmark composition (clip-benchmark splits) and the zero-shot sanity gate: ViT-B/16 prompt-ensemble accuracy must land inside a coarse band of published CLIP results before any feature cache is trusted. All ten datasets pass. ImageNet and SUN397 ship only evaluation shards, so a deterministic 68/32 per-class pool/test split is used (Sec. V-A ); test sizes are post-split.
Fig. 2: Measured risk profiles: test accuracy of the blend vs. λ (ViT-B/16, K=4 , seed-mean) for a strong (Pets), medium (DTD), and weak (Aircraft) prior. The oracle ratio (star) moves right as the prior weakens; the LOO estimate (diamond) follows from the support set alone, its occasional misses (Aircraft) costing little on the broad plateau; the James–Stein estimate (triangle) sits at the cliff edge on every dataset.
Fig. 3: Estimated vs. accuracy-optimal blending ratio, one point per cell (dataset-specific prompt; per-class estimates averaged for display). The James–Stein estimate saturates near 1 irrespective of the oracle: a good estimate of the wrong quantity. The LOO estimate follows the diagonal ( ρ=0.79 , MAE 0.12 ).
Prior upgrade
Δ zero-shot acc. (pp)
Δd^2
Δλoracle
Δλ^LOO
→ dataset-specific
+1.42 [-0.10, +2.94]
-0.0121 [-0.0331, +0.0088]
-0.031 [-0.051, -0.011]
-0.030 [-0.053, -0.008]
→ prompt ensemble
+2.11 [+0.46, +3.76]
-0.0156 [-0.0345, +0.0032]
-0.047 [-0.067, -0.026]
-0.041 [-0.059, -0.022]
→ LLM descriptions (CuPL)
+2.28 [+0.05, +4.51]
-0.0177 [-0.0233, -0.0122]
-0.026 [-0.058, +0.005]
-0.028 [-0.050, -0.006]
TABLE II: Effect of upgrading the text prior from the generic “a photo of a {}.” template, paired within (dataset, backbone, K , seed) cells ( n=1200 per row; 95% CI, cluster bootstrap over datasets). Better priors raise zero-shot accuracy, shrink the prototype distance d^2 , and lower the accuracy-optimal blending ratio—and the support-set LOO estimate λ^LOO tracks that shift without being told.
Zero-shot
NCM
JS blend
LOO blend
GDA α=1
GDA α-LOO
HOSO
CLAP
LP++
Oracle λ
K=16 shots
Aircraft
24.8
40.7
40.8
41.3
46.8
49.0
42.0
45.7
45.4
41.7
Caltech101
82.4
88.0
88.1
88.4
91.5
91.5
91.5
91.5
93.1
88.8
Cars
63.7
72.3
72.5
76.4
81.8
81.6
78.6
80.3
81.1
76.5
DTD
43.0
62.4
62.6
64.3
72.0
70.8
70.4
71.0
71.6
64.9
EuroSAT
55.8
76.2
76.2
76.9
86.7
86.7
84.6
84.0
86.5
77.5
TABLE III: Per-dataset test accuracy (%) with CLIP ViT-B/16 and the dataset-specific prompt, averaged over five support-set seeds. All methods left of the rule are validation-free; “Oracle λ ” grid-searches the global blending ratio on the test set and is an unreachable upper bound for the blending family, not a method. Bold marks the best validation-free entry per row. Flowers102 supports only K≤4 (its pool holds ten images per class) and is therefore absent here; the mean covers the nine datasets with K=16 cells.
Method
K=1
K=2
K=4
K=8
K=16
Mean
Zero-shot
64.8
64.8
64.8
64.8
64.8
64.8
NCM (image prototypes)
40.9
52.5
61.5
67.6
71.3
63.2
James–Stein blend, Eq. ( 7 )
–
54.1
62.4
68.1
71.4
64.0
Centred JS blend
–
42.4
58.5
67.7
71.8
60.1
LOO blend
–
68.4
70.9
73.4
75.1
71.9
LOO blend + K -corr.
–
68.7
71.1
73.4
75.1
72.1
TABLE IV: Mean accuracy (%) of validation-free methods by shot count, aggregated over the 45 dataset × backbone combinations present at every K (the combination guard of Sec. V-D ; Flowers102 is excluded here). The Mean column averages the K≥2 column means , since several estimators need two shots to exist. Tip-Adapter’s grid is skipped on the 50 costliest K=16 cells; its K=16 entry covers the remaining 175. The lower block lists reference points that consume held-out or test labels; they are bounds, not methods.
All K
K≥4
Method vs. oracle λ
Δ [95% CI]
n
Δ [95% CI]
n
CLAP
+1.92 [+0.87, +2.98]
1200
+2.48 [+1.28, +3.69]
700
LP++
+1.46 [+0.34, +2.58]
1200
+2.78 [+1.23, +4.34]
700
HOSO
+0.36 [-0.63, +1.36]
950
+1.09 [+0.01, +2.17]
700
GDA ( α=1 )
-2.64 [-4.84, -0.44]
1200
+0.86 [-1.68, +3.41]
700
LOO blend
-0.82 [-1.31, -0.32]
950
-0.63 [-0.99, -0.27]
700
TABLE V: Paired accuracy difference (percentage points, cluster-bootstrap 95% CI, resampling the ten datasets) between each validation-free method and the test-set-oracle blending ratio—the best any global λ can do, obtained by grid search on the test labels. Positive means the method beats a bound the blending family itself cannot reach; bold marks CIs entirely above zero.
ZS acc.
class-indep.
vs. oracle blending ratio (pp)
Backbone
(%)
share (%)
JS blend
LOO blend
CLAP [95% CI]
RN50
57.3
73.9 [68.3, 79.4]
-11.24
-0.72
+2.64 [+1.38, +3.90]
ViT-B-32
60.9
77.6 [73.2, 82.0]
-9.65
-0.78
+2.19 [+1.06, +3.33]
ViT-B-16
64.5
78.0 [73.8, 82.2]
-8.84
-0.67
+2.11 [+1.05, +3.17]
ViT-L-14
70.6
75.7 [72.0, 79.3]
-7.06
-0.97
+1.40 [+0.41, +2.38]
SigLIP-B-16
73.7
86.6 [84.6, 88.6]
-5.77
-0.94
+1.28 [+0.11, +2.46]
TABLE VI: Backbone consistency, with the mechanism’s own predictor beside the damage it was supposed to predict. Columns: mean zero-shot accuracy; the class-independent share of the squared prototype distance (the quantity Eq. ( 4 ) mis-charges, over all four prompt tiers); and the paired deficit/surplus of three representative methods against the test-set-oracle blending ratio (pp, cluster-bootstrap 95% CI over datasets). The three qualitative findings hold on all five backbones. The ordering does not: SigLIP has the largest share and the smallest James–Stein deficit, and the deficit tracks zero-shot headroom instead (Sec. VI-D ).
Fig. 4: Accuracy vs. shots for all ten datasets (ViT-B/16, dataset-specific prompt, mean over five seeds): the probes lead almost everywhere; the James–Stein blend hugs or undercuts zero-shot where the prior is strong and forfeits it where weak. Flowers102 ends at K=4 (Sec. V-A ).
Fig. 5: Offset and per-class probes. A: accuracy as every prototype is translated by aΔ before renormalisation, at the oracle ratio; the shaded band is the range over which λ itself moves prototypes along that direction. Inside it the common offset is nearly free: what Theorem 1 (i) claims for inner-product scoring, and what the MSE objective charges in full. B: the per-class accuracy oracle, fitted and scored on the same test labels and then out of sample.
Headroom
Surplus
SD of λ^ , K=2
SD of λ^ , K=16
Dataset
Backbone
Ratio
(pp)
(pp)
deff
measured
predicted
measured
predicted
Aircraft
RN50
0.47
1.79
+2.69
32
2.1×10−2
1.9×10−2
6.1×10−4
6.8×10−4
Aircraft
ViT-B/32
0.54
1.15
+2.76
31
2.4×10−2
2.0×10−2
6.4×10−4
6.9×10−4
Aircraft
ViT-B/16
0.56
0.53
+2.61
32
2.0×10−2
1.8×10−2
5.6×10−4
6.4×10−4
Aircraft
ViT-L/14
0.55
0.14
+2.14
38
2.0×10−2
1.8×10−2
5.9×10−4
6.2×10−4
Aircraft
SigLIP-B/16
0.57
0.09
+0.53
44
1.3×10−2
1.3×10−2
4.2×10−4
4.4×10−4
TABLE S1: Per-stratum capacity and sampling measurements ( 50 strata; generated by code/probe_capacity.py and code/capacity_macros.py ). Ratio: mean path-to-full margin ratio. Headroom: mean error gap (pp) between the best blend-path direction and the unrestricted Mahalanobis direction. Surplus: measured CLAP − oracle-blend accuracy (pp). deff : median per-class participation ratio. SD columns: measured vs. predicted (Eq. (11)) seed-to-seed SD of λ^ .
James–Stein blend
LOO blend
CLAP
LP++
K=1
–
–
+0.30 [-0.49, +1.08]
−4.12 [-6.06, -2.17]
K=2
−11.57 [-16.30, -6.84]
−1.32 [-2.43, -0.21]
+1.22 [-0.08, +2.52]
−0.84 [-2.35, +0.68]
K=4
−5.98 [-8.74, -3.21]
−1.02 [-1.99, -0.04]
+1.48 [+0.17, +2.79]
+0.67 [-0.67, +2.01]
K=8
−3.87 [-5.73, -2.02]
−0.56 [-0.90, -0.22]
+1.77 [+0.42, +3.12]
+1.73 [-0.16, +3.62]
K=16
−2.51 [-3.88, -1.13]
−0.48 [-0.79, -0.18]
+1.80 [+0.44, +3.17]
+2.20 [+0.10, +4.29]
K≥4 pooled
−4.19 [-6.08, -2.29]
−0.70 [-1.21, -0.18]
+1.68 [+0.37, +2.98]
+1.50 [-0.22, +3.22]
TABLE S2: SigLIP2-B/16: the paper’s central contrasts, per shot count (pp versus the test-set-oracle blending ratio, paired within cells; cluster bootstrap over datasets, Student- t ). Generated by code/modern_macros.py from the released records.
DINOv3-B/16 (oracle reg.)
DINOv3-B/16 (LOO reg.)
DINOv2-B/14 (oracle reg.)
CLIP ViT-B/16 (oracle reg.)
K=1
−9.03 [-18.12, +0.07]
–
−14.30 [-25.09, -3.51]
−23.78 [-33.64, -13.92]
K=2
−0.66 [-7.60, +6.28]
−2.39 [-9.11, +4.33]
−7.00 [-15.20, +1.21]
−13.91 [-21.29, -6.54]
K=4
+4.55 [-1.56, +10.66]
+3.56 [-2.56, +9.68]
−2.01 [-7.60, +3.58]
−6.51 [-11.58, -1.43]
K=8
+7.62 [+0.38, +14.85]
+6.54 [-0.82, +13.90]
+1.57 [-2.88, +6.01]
−1.74 [-5.63, +2.16]
K=16
+9.76 [+2.06, +17.46]
+8.87 [+1.10, +16.64]
+5.33 [+0.23, +10.42]
+1.99 [-1.31, +5.29]
K≥4 pooled
+7.21 [+0.41, +14.01]
+6.22 [-0.62, +13.07]
+1.50 [-2.98, +5.97]
−2.24 [-6.25, +1.76]
TABLE S3: Text-free reference points: a plain logistic probe on self-supervised features versus the blending family’s test-set-oracle ratio (pp, paired within cells). “Oracle reg.” scores the probe at its best test-set regularisation—supremum against supremum; “LOO reg.” selects it validation-free on the support set. The same probe on CLIP’s own features is the control: the family’s ceiling is cleared by the text-free encoder, not by the probe construction.
Training-free few-shot adaptation methods have gained significant attention recently in the context of Vision-language Models (VLMs). Yet, current benchmarks rely on strong assumptions about the statistics of the adaptation data, e.g., class balance. We question these simplifying assumptions and introduce a more realistic benchmark that varies both the levels of class balance and the effective number of classes in few-shot tasks via Dirichlet sampling. Surprisingly, under our setting, we observe substantial drops in the performances of state-of-the-art methods, more so when the number of labeled samples increases. To mitigate this, we introduce PRiSM, a class-prototype regularization that can be deployed as a plug and play module on top of any existing baseline method, significantly improving performances. Our method optimizes a novel multi-term loss, which includes a regularizer maximizing inter-class pairwise distances, along with additional terms promoting support-feature alignment and fidelity to the baseline prototypes. Furthermore, we introduce an effective and computationally efficient block Majorize-Minimize optimizer for our objective. More specifically, we derive a valid blockwise Lipschitz constant (i.e., a bound on the Hessian's spectral norm), which can be computed efficiently via the Gershgorin circle theorem. Extensive experiments show that PRiSM improves several training-free baselines, with large gains when dealing with severe class imbalance and high numbers of classes.
Ghassen Baklouti, Omprakash Chakraborty, Jose Dolz +1
In many CLIP adaptation methods, a blending ratio hyperparameter controls the trade-off between general pretrained CLIP knowledge and the limited, dataset-specific supervision from the few-shot cases. Most few-shot CLIP adaptation techniques report results by ablation of the blending ratio on the test set or require additional validation sets to select the blending ratio per dataset, and thus are not strictly few-shot. We present a simple, validation-free method for learning the blending ratio in CLIP adaptation. Hold-One-Shot-Out (HOSO) presents a novel approach for CLIP-Adapter-style methods to compete in the newly established validation-free setting. CLIP-Adapter with HOSO (HOSO-Adapter) learns the blending ratio using a one-shot, hold-out set, while the adapter trains on the remaining few-shot support examples. Under the validation-free few-shot protocol, HOSO-Adapter outperforms the CLIP-Adapter baseline by more than 4 percentage points on average across 11 standard few-shot datasets. Interestingly, in the 8- and 16-shot settings, HOSO-Adapter outperforms CLIP-Adapter even with the optimal blending ratio selected on the test set. Ablation studies validate the use of a one-shot hold-out mechanism, decoupled training, and improvements over the naively learnt blending ratio baseline. Code is released here: https://github.com/chris-vorster/HOSO-Adapter
Chris Vorster, Mayug Maniparambil, Noel E. O'Connor +2
The adaptation of large-scale Vision-Language Models (VLMs) like CLIP to downstream tasks with extremely limited data -- specifically in the one-shot regime -- is often hindered by a significant "Stability-Plasticity" dilemma. While efficient caching mechanisms have been introduced by training-free methods such as Tip-Adapter, these approaches often function as local Nadaraya-Watson estimators. Such estimators are characterized by inherent boundary bias and a lack of global structural regularization. In this paper, ReHARK (Refined Hybrid Adaptive RBF Kernels) is proposed as a synergistic training-free framework that reinterprets few-shot adaptation through global proximal regularization in a Reproducing Kernel Hilbert Space (RKHS). A multistage refinement pipeline is introduced, consisting of: (1) Hybrid Prior Construction, where zero-shot textual knowledge from CLIP and GPT-3 is fused with visual class prototypes to form a robust semantic-visual anchor; (2) Support Set Augmentation (Bridging), where intermediate samples are generated to smooth the transition between visual and textual modalities; (3) Adaptive Distribution Rectification, where test feature statistics are aligned with the augmented support set to mitigate domain shifts; and (4) Multi-Scale RBF Kernels, where an ensemble of kernels is employed to capture complex feature geometries across diverse scales. Superior stability and accuracy are demonstrated through extensive experiments on 11 diverse benchmarks. A new state-of-the-art for one-shot adaptation is established by ReHARK, which achieves an average accuracy of 65.83%, significantly outperforming existing baselines. Code is available at https://github.com/Jahid12012021/ReHARK.
Md Jahidul Islam
Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh