Organizations: School of Telecommunications Engineering, Xidian University, Xi’an, Shaanxi 710071, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
An image that is hard to classify on its own may have a helpful neighbor in the same unlabeled batch. Zero-shot vision-language models (VLMs) score images independently and leave this evidence unused. We study small unlabeled query batches with no labeled examples. Visual similarity can reflect pose or background, so it may connect images from different classes. Each image's scores over the class vocabulary offer another way to choose neighbors. We introduce Behavioral Affinity Transduction (BAT), which builds a sparse graph from these class-response profiles. The graph supplies neighbor evidence during prediction. For EM-Dirichlet, it also screens candidate classes and initializes assignments. BAT plugs into direct and iterative readouts of a frozen VLM. Across diverse datasets, BAT improves the mean accuracy of every tested readout. In the main RN50 comparison, soft and hard EM-Dirichlet gain 4.22 and 2.72 percentage points.
Figures & tables
Figure 1: OxfordPets visual and response t-SNE projections.
Figure 2: BAT graph construction and readout integration.
Figure 3: Visual and response class separation by Cohen’s d .
Figure 4: Accuracy across query sizes for EM-Dirichlet, Gaussian, and K-means readouts.
Soft
Hard
Dataset
Base
BAT
Base
BAT
Caltech101
88.42
91.28
89.54
91.29
DTD
47.17
49.76
48.66
50.27
Aircraft
19.25
22.16
20.38
22.09
Flowers102
71.62
75.54
73.17
75.51
Food101
88.21
93.01
90.39
93.10
Table 1: EM-Dirichlet [ 11 ] accuracy on RN50 (%).
Readout
Base
+BAT
Gain
Direct prediction [ 14 ]
60.75
68.22
+7.47
Soft K-means [ 1 ]
25.26
71.82
+46.56
Hard K-means [ 9 ]
41.19
71.42
+30.23
KL K-means [ 4 ]
57.94
71.53
+13.60
EM-Gaussian [ 2 ]
19.69
68.46
+48.77
Soft EM-Dir. [ 11 ]
68.99
73.06
+4.07
Table 2: Readout accuracy (%) across ten datasets. Gains use unrounded means.
Backbone
Readout
Base
BAT
Gain
RN50
Soft
68.99
73.06
+4.07
Hard
70.69
73.28
+2.59
ViT-L/14
Soft
77.89
81.71
+3.83
Hard
79.16
81.83
+2.67
Table 3: EM-Dirichlet [ 11 ] by backbone (%).
Metric
Visual
Response
Same@5 (%)
11.29
14.64
Weighted Same@5 (%)
17.62
29.70
Oracle ratio (%)
67.26
83.81
Table 4: Graph quality across ten datasets.
Graph space
Soft
Hard
Visual
71.62
71.89
Response
72.13
72.37
Table 5: EM-Dirichlet accuracy by graph space (%).
Efficient transfer learning of vision-language models (VLMs) commonly suffers from a Base-New Trade-off (BNT): improving performance on unseen (new) classes often degrades accuracy on known (base) classes. Addressing how to boost recognition of unseen classes without sacrificing known-class performance remains a central challenge. Existing work often simplistically attributes the BNT to overfitting on known classes. We observe an interesting phenomenon: VLMs frequently exhibit asymmetric confusion on certain downstream data, i.e., samples of class A are systematically mispredicted as class B, while the reverse confusion (B to A) rarely occurs. For known classes, this kind of bias can be mitigated by tuning using a cross-entropy loss, but for unseen classes, such pretraining-induced bias persists and harms generalization. Motivated by this, we propose NeRP, a plug-and-play prompting correction strategy that improves discrimination on unseen classes without modifying model parameters. NeRP leverages neutral text prompts and reference images to measure class-wise prior preferences along the pre-trained inter-class geometry, and combines them with the sample likelihood to obtain the model's surrogate score. If, for a given sample, the prior strongly favors the current prediction while the observed evidence is clearly insufficient, we perform a local flip between easily confusable class pairs, thereby correcting prior-dominated mispredictions. Extensive experiments across multiple backbones and 15 few-shot and cross-domain benchmarks show that NeRP substantially improves accuracy on unseen classes while preserving known-class prediction performance.
Transductive adaptation improves vision-language models at test time by refining predictions on an unlabeled batch, but current methods face a key trade-off. Aggressive approaches exploit batch structure but can drift when batches are sparse, have prior shift, or cover little of the label space. Conservative methods better preserve CLIP text semantics, but fixed semantic references can under-adapt even when the batch provides strong corrective evidence. We propose JPTA (Joint Prior and Transport Adaptation), a prior-aware framework that casts transductive VLM adaptation as batch-conditioned semantic reference estimation. Instead of treating CLIP text prototypes as fixed priors or freely replaceable, JPTA views them as semantic references that shift toward image-side batch structure only under calibrated evidence. It estimates batch priors and soft image-side prototypes to form transported semantic references whose impact is scaled by batch reliability. Across standard benchmarks, low effective-class regimes, all-class evaluation, and online streams, JPTA consistently outperforms TransCLIP and StatA, with the largest gains on sparse and prior-shifted batches. Anchor-drift, active-support, and absent-class-mass diagnostics show that JPTA fixes effective-class mismatch without unconstrained transductive drift, indicating that robust transductive VLM adaptation should recalibrate semantic references using reliable batch evidence rather than choose between unrestricted transduction and fixed prototypes. Code is available at https://github.com/MR-Sherif/JPTA.git.
Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan +2
A range of methods aim to enhance the performance of vision-language models (VLMs) at test time. Among them, transduction has emerged as a promising paradigm due to its strong compatibility and efficiency. However, realistic evaluations often involve highly imbalanced class distributions, which cause performance degradation or even collapse. In this work, we systematically revisit transduction from the perspective of penalized likelihood estimation (PLE), showing that PLE with a KL-divergence anchor term naturally yields an adaptive shrinkage behavior between prior anchors and empirical estimates. From this viewpoint, the brittleness of transductive methods can be attributed to the absence of anchoring mechanism and static modeling of the shrinkage strength. Therefore, we propose Mixture of Von Mises-Fisher Models with Dynamic Shrinkage (MOON). MOON is built upon a mixture of von Mises-Fisher distributions to model feature representations on the unit hypersphere. To handle imbalance, MOON dynamically adjusts the shrinkage strength using zero-shot priors at both instance and class levels. Thus, it suppresses unreliable assignments and prevents harmful updates from outlier classes, thereby mitigating negative transfer. MOON is model-agnostic, training-free, and requires no task-specific hyperparameter tuning. Extensive experiments further validate the advantage of MOON in both performance and efficiency. Our code is available at https://github.com/walawalagoose/MOON
Jiazhen Huang, Zhiming Liu, Changhu Wang +3
Tsinghua University, China · Fred Hutchinson Cancer Center, USA · Peking University, China +2