Unlabeled test data can improve the zero-shot classifier of a vision-language model, yet transductive and test-time adaptation methods often fall below the zero-shot baseline when only a few vocabulary classes are present in a batch or when streams are correlated. We trace this failure to a shared design: every class owns independent parameters estimated from its own samples, so absent or rare classes are left at biased text anchors or fitted to foreign samples. We propose Graph-Regularized Anchor-displacement Field Transduction (GRAFT), which models the adapted prototypes as text anchors plus a displacement field that is partially pooled over a semantic class graph through a Gaussian Markov random field prior, together with an explicit class-presence model. Inference is closed-form and gradient-free on cached features. Anchor-only shrinkage and a single global shift are the two extreme settings of the prior; absent classes are provably harmonic extensions of the present ones, and the pooling strength can be chosen by empirical Bayes. Under the realistic batch and online protocols on eleven benchmarks and four ImageNet shifts, GRAFT improves over the zero-shot model in every setting and over anchor-based transduction where few classes are present.
Figures & tables
Figure 1: Per-class adaptation on a realistic batch (conceptual illustration). Classes A and B are present, C is absent, and all images are shifted from their anchors. (a) Per-class fitting lets C capture samples of B. (b) Anchor-only shrinkage leaves C at its biased anchor. (c) GRAFT lets C inherit B’s displacement and removes it through the presence model.
Figure 2: Overview of GRAFT. Frozen encoders give features X and anchors T , which define the zero-shot prior y^ , the sample graph Ws (from X , images to images) and the semantic class graph LT (from T , classes to classes). Closed-form updates alternate the E-step over z , using y^ and Ws , with M-steps for the presence π , the displacement field D pooled over LT and the shared covariance; μ=T+D are the adapted prototypes and z the prediction.
Figure 3: Graph-pooled displacement field on a toy class graph (node size: nk ). (a) Empirical displacements. (b) Independent shrinkage ( λ=0 ) leaves absent classes at their anchors. (c) Eq. 7 transfers displacements along the graph.
Method
Very low
Low
Medium
Medium
High
Very high
All
B=64
B=64
B=64
B=1000
B=1000
B=1000
full
CLIP zero-shot [ 31 ]
65.2 ± 0.6
65.2 ± 0.5
65.2 ± 0.4
65.2 ± 0.2
65.2 ± 0.2
65.2 ± 0.1
65.2
MTA [ 45 ]
66.6 ± 0.6
66.6 ± 0.5
66.6 ± 0.4
66.6 ± 0.2
66.6 ± 0.2
66.6 ± 0.1
66.6
EM-Dirichlet [ 27 ]
68.5 ± 0.7
70.3 ± 0.6
67.5 ± 0.5
64.4 ± 0.4
45.3 ± 0.6
33.6 ± 0.7
29.5
ZLaP [ 40 ]
27.5 ± 1.1
35.2 ± 0.9
44.7 ± 0.8
41.5 ± 0.5
52.2 ± 0.4
58.4 ± 0.3
66.4
TransCLIP [ 47 ]
38.9 ± 1.0
40.4 ± 0.9
42.7 ± 0.8
56.5 ± 0.4
62.0 ± 0.3
64.4 ± 0.2
70.3
Table 1: Batch adaptation with a variable number of effective classes: top-1 accuracy (%) averaged over 1000 tasks and eleven datasets, CLIP ViT-B/16.
Figure 4: Top-1 accuracy versus the number of effective classes per batch (eleven-dataset mean, error bars: standard deviation over tasks); the dotted line is the zero-shot model.
Method
Low
Medium
High
Separate
CLIP zero-shot [ 31 ]
65.2 ± 0.3
65.2 ± 0.3
65.2 ± 0.3
65.2 ± 0.3
TENT [ 42 ]
65.8 ± 0.4
65.5 ± 0.5
65.3 ± 0.6
64.5 ± 0.8
TDA [ 21 ]
67.7 ± 0.4
67.1 ± 0.5
66.8 ± 0.6
66.6 ± 0.7
DMN [ 51 ]
67.2 ± 0.4
66.5 ± 0.5
66.3 ± 0.6
65.8 ± 0.7
OGA [ 10 ]
67.5 ± 0.5
69.1 ± 0.5
69.6 ± 0.6
69.1 ± 0.7
StatA [ 46 ]
67.0 ± 0.5
68.9 ± 0.5
69.5 ± 0.6
69.1 ± 0.7
Table 2: Online adaptation: top-1 accuracy (%) over 100 runs and eleven datasets, batches of 128, CLIP ViT-B/16, for stream correlation γ∈{0.1,0.01,0.001} and the separate-class stream.
Method
IN
SUN
Airc.
Euro.
Cars
Food
Pets
Flow.
Calt.
DTD
UCF
Avg.
CLIP zero-shot [ 31 ]
66.7
62.6
24.8
47.8
65.3
86.1
89.1
71.3
93.3
44.4
66.7
65.3
MTA [ 45 ]
68.0
64.2
27.0
49.5
66.4
86.8
89.6
72.6
93.7
47.0
67.9
66.6
EM-Dirichlet [ 27 ]
70.2
69.3
35.1
57.3
70.3
88.0
91.0
76.1
94.1
50.6
71.4
70.3
ZLaP [ 40 ]
35.4
30.7
13.3
28.4
40.3
48.4
47.7
38.2
44.9
26.6
33.3
35.2
TransCLIP [ 47 ]
43.2
35.8
16.8
32.6
44.4
50.8
48.8
45.9
62.5
24.2
39.5
40.4
StatA [ 46 ]
71.0
66.0
35.0
52.3
69.6
87.4
90.7
74.2
94.0
50.9
71.1
69.3
Table 3: Per-dataset top-1 accuracy (%) in the low setting ( B=64 , Keff∈[2,10] ), CLIP ViT-B/16.
Figure 5: Per-dataset accuracy in the low setting for the zero-shot model, StatA and GRAFT.
Figure 6: Posterior mass assigned to classes absent from the batch versus the number of effective classes (ImageNet, B=64 ).
Method
IN
V2
S
A
R
CLIP zero-shot
66.7
60.8
46.2
47.8
74.0
TransCLIP
43.2
35.5
29.1
28.9
42.8
StatA
71.0
66.2
51.1
51.9
79.4
MOON
71.7
66.8
51.7
52.6
79.7
GRAFT
72.3
68.3
53.1
54.3
81.9
Table 4: ImageNet (IN) and its shifts V2, Sketch (S), A and R in the low setting, top-1 accuracy (%), CLIP ViT-B/16.
Figure 7: Qualitative predictions in the low setting: ground truth (GT) and the predictions of the zero-shot model (ZS), StatA and GRAFT. Correct predictions in green, wrong ones in red.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
An image that is hard to classify on its own may have a helpful neighbor in the same unlabeled batch. Zero-shot vision-language models (VLMs) score images independently and leave this evidence unused. We study small unlabeled query batches with no labeled examples. Visual similarity can reflect pose or background, so it may connect images from different classes. Each image's scores over the class vocabulary offer another way to choose neighbors. We introduce Behavioral Affinity Transduction (BAT), which builds a sparse graph from these class-response profiles. The graph supplies neighbor evidence during prediction. For EM-Dirichlet, it also screens candidate classes and initializes assignments. BAT plugs into direct and iterative readouts of a frozen VLM. Across diverse datasets, BAT improves the mean accuracy of every tested readout. In the main RN50 comparison, soft and hard EM-Dirichlet gain 4.22 and 2.72 percentage points.
Junxi Huang, Wenshuo Chen, Xiucheng Wang +5
School of Telecommunications Engineering, Xidian University, Xi’an, Shaanxi 710071, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
Transductive adaptation improves vision-language models at test time by refining predictions on an unlabeled batch, but current methods face a key trade-off. Aggressive approaches exploit batch structure but can drift when batches are sparse, have prior shift, or cover little of the label space. Conservative methods better preserve CLIP text semantics, but fixed semantic references can under-adapt even when the batch provides strong corrective evidence. We propose JPTA (Joint Prior and Transport Adaptation), a prior-aware framework that casts transductive VLM adaptation as batch-conditioned semantic reference estimation. Instead of treating CLIP text prototypes as fixed priors or freely replaceable, JPTA views them as semantic references that shift toward image-side batch structure only under calibrated evidence. It estimates batch priors and soft image-side prototypes to form transported semantic references whose impact is scaled by batch reliability. Across standard benchmarks, low effective-class regimes, all-class evaluation, and online streams, JPTA consistently outperforms TransCLIP and StatA, with the largest gains on sparse and prior-shifted batches. Anchor-drift, active-support, and absent-class-mass diagnostics show that JPTA fixes effective-class mismatch without unconstrained transductive drift, indicating that robust transductive VLM adaptation should recalibrate semantic references using reliable batch evidence rather than choose between unrestricted transduction and fixed prototypes. Code is available at https://github.com/MR-Sherif/JPTA.git.
Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan +2
A range of methods aim to enhance the performance of vision-language models (VLMs) at test time. Among them, transduction has emerged as a promising paradigm due to its strong compatibility and efficiency. However, realistic evaluations often involve highly imbalanced class distributions, which cause performance degradation or even collapse. In this work, we systematically revisit transduction from the perspective of penalized likelihood estimation (PLE), showing that PLE with a KL-divergence anchor term naturally yields an adaptive shrinkage behavior between prior anchors and empirical estimates. From this viewpoint, the brittleness of transductive methods can be attributed to the absence of anchoring mechanism and static modeling of the shrinkage strength. Therefore, we propose Mixture of Von Mises-Fisher Models with Dynamic Shrinkage (MOON). MOON is built upon a mixture of von Mises-Fisher distributions to model feature representations on the unit hypersphere. To handle imbalance, MOON dynamically adjusts the shrinkage strength using zero-shot priors at both instance and class levels. Thus, it suppresses unreliable assignments and prevents harmful updates from outlier classes, thereby mitigating negative transfer. MOON is model-agnostic, training-free, and requires no task-specific hyperparameter tuning. Extensive experiments further validate the advantage of MOON in both performance and efficiency. Our code is available at https://github.com/walawalagoose/MOON
Jiazhen Huang, Zhiming Liu, Changhu Wang +3
Tsinghua University, China · Fred Hutchinson Cancer Center, USA · Peking University, China +2