Organizations: Department of Computer Science, Qufu Normal University, China, and with Climind · Institute of Scientific and Industrial Research (SANKEN), The University of Osaka, Japan · Institute of High Performance Computing (IHPC), A*STAR, Singapore · Standard Chartered Bank, Singapore · School of Aerospace Intelligence, Hainan University, China · School of Computer Science, Qufu Normal University, China
Many few-shot adaptation methods for vision-language models classify with a convex combination of the zero-shot text prototype and the mean of the K labelled image features, with a single blending ratio routinely tuned on held-out labels, often on the test set itself. We ask what the family's own bias-variance justification invites: what is the right ratio, can it be estimated without validation data, and is finding it where the performance is? First, the ratio minimising prototype mean-squared error has a closed form whose support-set plug-in is exactly a positive-part James-Stein coefficient shrinking towards the text prototype. Across 4,800 cells (ten datasets, five backbones including SigLIP, five shot counts, five seeds, four prompt tiers) this theoretically optimal ratio is a reliable estimate of the wrong quantity: on the 950 primary-tier cells where it is defined it trails a test-set-oracle ratio by 8.5 points. It saturates near 1, discarding the text prior for a nearest-class-mean classifier, because 78% of the text-image prototype distance it treats as bias is a class-independent offset that the arg max largely cancels. We prove the mechanism and bound its share of the damage at 26% by a counterfactual. Second, leave-one-out on the support set alone sets a ratio landing within 0.9 points of the oracle blend, so it is estimable without validation data. Third, validation-free linear probes beat even the oracle-tuned blend: CLAP by +1.9 points and LP++ by +1.5 on average, and at K >= 4 all four validation-free baselines sit above the oracle, the linear probes by margins excluding zero. These results locate the ceiling in the model class, not the hyperparameter: the ratio can be set near-optimally for free, and it is still not where the performance is. Code, cached features, per-cell records: https://huggingface.co/datasets/Liangzhi-Li/clipbench-blending
Figures & tables
Fig. 1: A: PCA of the prototype geometry (Pets, ViT-B/16): the common offset Δ (the modality gap) is 75% of the squared text–image prototype distance here, yet costs far less at decision time (Thm. 1 ; Sec. VI-G ). Blending slides each prototype along its dotted path; marks are the two objectives’ choices. B: measured accuracy and prototype MSE vs. λ (Food-101, K=4 ): the MSE minimum ( λ≈0.96 , where Eq. ( 7 ) sits) costs 14.2 accuracy points against the accuracy peak ( λ≈0.09 , which leave-one-out finds). C: mean accuracy, K≥4 , balanced cells: every reproduced validation-free baseline exceeds the test-set-oracle blend (dashed): the family is capacity-limited, not mis-tuned.
Dataset
C
ntest
Pool split
ZS ens. (%)
Published
Aircraft
100
3,333
official
24.3
20–32
Caltech101
102
6,085
official
82.2
80–96
Cars
196
8,041
official
64.7
60–70
DTD
47
1,880
official
44.9
40–50
EuroSAT
10
5,400
official
55.9
36–62
Flowers102
102
6,149
official
71.2
65–78
TABLE I: Benchmark composition (clip-benchmark splits) and the zero-shot sanity gate: ViT-B/16 prompt-ensemble accuracy must land inside a coarse band of published CLIP results before any feature cache is trusted. All ten datasets pass. ImageNet and SUN397 ship only evaluation shards, so a deterministic 68/32 per-class pool/test split is used (Sec. V-A ); test sizes are post-split.
Fig. 2: Measured risk profiles: test accuracy of the blend vs. λ (ViT-B/16, K=4 , seed-mean) for a strong (Pets), medium (DTD), and weak (Aircraft) prior. The oracle ratio (star) moves right as the prior weakens; the LOO estimate (diamond) follows from the support set alone, its occasional misses (Aircraft) costing little on the broad plateau; the James–Stein estimate (triangle) sits at the cliff edge on every dataset.
Fig. 3: Estimated vs. accuracy-optimal blending ratio, one point per cell (dataset-specific prompt; per-class estimates averaged for display). The James–Stein estimate saturates near 1 irrespective of the oracle: a good estimate of the wrong quantity. The LOO estimate follows the diagonal ( ρ=0.79 , MAE 0.12 ).
Prior upgrade
Δ zero-shot acc. (pp)
Δd^2
Δλoracle
Δλ^LOO
→ dataset-specific
+1.42 [-0.10, +2.94]
-0.0121 [-0.0331, +0.0088]
-0.031 [-0.051, -0.011]
-0.030 [-0.053, -0.008]
→ prompt ensemble
+2.11 [+0.46, +3.76]
-0.0156 [-0.0345, +0.0032]
-0.047 [-0.067, -0.026]
-0.041 [-0.059, -0.022]
→ LLM descriptions (CuPL)
+2.28 [+0.05, +4.51]
-0.0177 [-0.0233, -0.0122]
-0.026 [-0.058, +0.005]
-0.028 [-0.050, -0.006]
TABLE II: Effect of upgrading the text prior from the generic “a photo of a {}.” template, paired within (dataset, backbone, K , seed) cells ( n=1200 per row; 95% CI, cluster bootstrap over datasets). Better priors raise zero-shot accuracy, shrink the prototype distance d^2 , and lower the accuracy-optimal blending ratio—and the support-set LOO estimate λ^LOO tracks that shift without being told.
Zero-shot
NCM
JS blend
LOO blend
GDA α=1
GDA α-LOO
HOSO
CLAP
LP++
Oracle λ
K=16 shots
Aircraft
24.8
40.7
40.8
41.3
46.8
49.0
42.0
45.7
45.4
41.7
Caltech101
82.4
88.0
88.1
88.4
91.5
91.5
91.5
91.5
93.1
88.8
Cars
63.7
72.3
72.5
76.4
81.8
81.6
78.6
80.3
81.1
76.5
DTD
43.0
62.4
62.6
64.3
72.0
70.8
70.4
71.0
71.6
64.9
EuroSAT
55.8
76.2
76.2
76.9
86.7
86.7
84.6
84.0
86.5
77.5
TABLE III: Per-dataset test accuracy (%) with CLIP ViT-B/16 and the dataset-specific prompt, averaged over five support-set seeds. All methods left of the rule are validation-free; “Oracle λ ” grid-searches the global blending ratio on the test set and is an unreachable upper bound for the blending family, not a method. Bold marks the best validation-free entry per row. Flowers102 supports only K≤4 (its pool holds ten images per class) and is therefore absent here; the mean covers the nine datasets with K=16 cells.
Method
K=1
K=2
K=4
K=8
K=16
Mean
Zero-shot
64.8
64.8
64.8
64.8
64.8
64.8
NCM (image prototypes)
40.9
52.5
61.5
67.6
71.3
63.2
James–Stein blend, Eq. ( 7 )
–
54.1
62.4
68.1
71.4
64.0
Centred JS blend
–
42.4
58.5
67.7
71.8
60.1
LOO blend
–
68.4
70.9
73.4
75.1
71.9
LOO blend + K -corr.
–
68.7
71.1
73.4
75.1
72.1
TABLE IV: Mean accuracy (%) of validation-free methods by shot count, aggregated over the 45 dataset × backbone combinations present at every K (the combination guard of Sec. V-D ; Flowers102 is excluded here). The Mean column averages the K≥2 column means , since several estimators need two shots to exist. Tip-Adapter’s grid is skipped on the 50 costliest K=16 cells; its K=16 entry covers the remaining 175. The lower block lists reference points that consume held-out or test labels; they are bounds, not methods.
All K
K≥4
Method vs. oracle λ
Δ [95% CI]
n
Δ [95% CI]
n
CLAP
+1.92 [+0.87, +2.98]
1200
+2.48 [+1.28, +3.69]
700
LP++
+1.46 [+0.34, +2.58]
1200
+2.78 [+1.23, +4.34]
700
HOSO
+0.36 [-0.63, +1.36]
950
+1.09 [+0.01, +2.17]
700
GDA ( α=1 )
-2.64 [-4.84, -0.44]
1200
+0.86 [-1.68, +3.41]
700
LOO blend
-0.82 [-1.31, -0.32]
950
-0.63 [-0.99, -0.27]
700
TABLE V: Paired accuracy difference (percentage points, cluster-bootstrap 95% CI, resampling the ten datasets) between each validation-free method and the test-set-oracle blending ratio—the best any global λ can do, obtained by grid search on the test labels. Positive means the method beats a bound the blending family itself cannot reach; bold marks CIs entirely above zero.
ZS acc.
class-indep.
vs. oracle blending ratio (pp)
Backbone
(%)
share (%)
JS blend
LOO blend
CLAP [95% CI]
RN50
57.3
73.9 [68.3, 79.4]
-11.24
-0.72
+2.64 [+1.38, +3.90]
ViT-B-32
60.9
77.6 [73.2, 82.0]
-9.65
-0.78
+2.19 [+1.06, +3.33]
ViT-B-16
64.5
78.0 [73.8, 82.2]
-8.84
-0.67
+2.11 [+1.05, +3.17]
ViT-L-14
70.6
75.7 [72.0, 79.3]
-7.06
-0.97
+1.40 [+0.41, +2.38]
SigLIP-B-16
73.7
86.6 [84.6, 88.6]
-5.77
-0.94
+1.28 [+0.11, +2.46]
TABLE VI: Backbone consistency, with the mechanism’s own predictor beside the damage it was supposed to predict. Columns: mean zero-shot accuracy; the class-independent share of the squared prototype distance (the quantity Eq. ( 4 ) mis-charges, over all four prompt tiers); and the paired deficit/surplus of three representative methods against the test-set-oracle blending ratio (pp, cluster-bootstrap 95% CI over datasets). The three qualitative findings hold on all five backbones. The ordering does not: SigLIP has the largest share and the smallest James–Stein deficit, and the deficit tracks zero-shot headroom instead (Sec. VI-D ).
Fig. 4: Accuracy vs. shots for all ten datasets (ViT-B/16, dataset-specific prompt, mean over five seeds): the probes lead almost everywhere; the James–Stein blend hugs or undercuts zero-shot where the prior is strong and forfeits it where weak. Flowers102 ends at K=4 (Sec. V-A ).
Fig. 5: Offset and per-class probes. A: accuracy as every prototype is translated by aΔ before renormalisation, at the oracle ratio; the shaded band is the range over which λ itself moves prototypes along that direction. Inside it the common offset is nearly free: what Theorem 1 (i) claims for inner-product scoring, and what the MSE objective charges in full. B: the per-class accuracy oracle, fitted and scored on the same test labels and then out of sample.
Headroom
Surplus
SD of λ^ , K=2
SD of λ^ , K=16
Dataset
Backbone
Ratio
(pp)
(pp)
deff
measured
predicted
measured
predicted
Aircraft
RN50
0.47
1.79
+2.69
32
2.1×10−2
1.9×10−2
6.1×10−4
6.8×10−4
Aircraft
ViT-B/32
0.54
1.15
+2.76
31
2.4×10−2
2.0×10−2
6.4×10−4
6.9×10−4
Aircraft
ViT-B/16
0.56
0.53
+2.61
32
2.0×10−2
1.8×10−2
5.6×10−4
6.4×10−4
Aircraft
ViT-L/14
0.55
0.14
+2.14
38
2.0×10−2
1.8×10−2
5.9×10−4
6.2×10−4
Aircraft
SigLIP-B/16
0.57
0.09
+0.53
44
1.3×10−2
1.3×10−2
4.2×10−4
4.4×10−4
TABLE S1: Per-stratum capacity and sampling measurements ( 50 strata; generated by code/probe_capacity.py and code/capacity_macros.py ). Ratio: mean path-to-full margin ratio. Headroom: mean error gap (pp) between the best blend-path direction and the unrestricted Mahalanobis direction. Surplus: measured CLAP − oracle-blend accuracy (pp). deff : median per-class participation ratio. SD columns: measured vs. predicted (Eq. (11)) seed-to-seed SD of λ^ .
James–Stein blend
LOO blend
CLAP
LP++
K=1
–
–
+0.30 [-0.49, +1.08]
−4.12 [-6.06, -2.17]
K=2
−11.57 [-16.30, -6.84]
−1.32 [-2.43, -0.21]
+1.22 [-0.08, +2.52]
−0.84 [-2.35, +0.68]
K=4
−5.98 [-8.74, -3.21]
−1.02 [-1.99, -0.04]
+1.48 [+0.17, +2.79]
+0.67 [-0.67, +2.01]
K=8
−3.87 [-5.73, -2.02]
−0.56 [-0.90, -0.22]
+1.77 [+0.42, +3.12]
+1.73 [-0.16, +3.62]
K=16
−2.51 [-3.88, -1.13]
−0.48 [-0.79, -0.18]
+1.80 [+0.44, +3.17]
+2.20 [+0.10, +4.29]
K≥4 pooled
−4.19 [-6.08, -2.29]
−0.70 [-1.21, -0.18]
+1.68 [+0.37, +2.98]
+1.50 [-0.22, +3.22]
TABLE S2: SigLIP2-B/16: the paper’s central contrasts, per shot count (pp versus the test-set-oracle blending ratio, paired within cells; cluster bootstrap over datasets, Student- t ). Generated by code/modern_macros.py from the released records.
DINOv3-B/16 (oracle reg.)
DINOv3-B/16 (LOO reg.)
DINOv2-B/14 (oracle reg.)
CLIP ViT-B/16 (oracle reg.)
K=1
−9.03 [-18.12, +0.07]
–
−14.30 [-25.09, -3.51]
−23.78 [-33.64, -13.92]
K=2
−0.66 [-7.60, +6.28]
−2.39 [-9.11, +4.33]
−7.00 [-15.20, +1.21]
−13.91 [-21.29, -6.54]
K=4
+4.55 [-1.56, +10.66]
+3.56 [-2.56, +9.68]
−2.01 [-7.60, +3.58]
−6.51 [-11.58, -1.43]
K=8
+7.62 [+0.38, +14.85]
+6.54 [-0.82, +13.90]
+1.57 [-2.88, +6.01]
−1.74 [-5.63, +2.16]
K=16
+9.76 [+2.06, +17.46]
+8.87 [+1.10, +16.64]
+5.33 [+0.23, +10.42]
+1.99 [-1.31, +5.29]
K≥4 pooled
+7.21 [+0.41, +14.01]
+6.22 [-0.62, +13.07]
+1.50 [-2.98, +5.97]
−2.24 [-6.25, +1.76]
TABLE S3: Text-free reference points: a plain logistic probe on self-supervised features versus the blending family’s test-set-oracle ratio (pp, paired within cells). “Oracle reg.” scores the probe at its best test-set regularisation—supremum against supremum; “LOO reg.” selects it validation-free on the support set. The same probe on CLIP’s own features is the control: the family’s ceiling is cleared by the text-free encoder, not by the probe construction.