Organizations: School of Telecommunications Engineering, Xidian University, Xi’an, Shaanxi 710071, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
An image that is hard to classify on its own may have a helpful neighbor in the same unlabeled batch. Zero-shot vision-language models (VLMs) score images independently and leave this evidence unused. We study small unlabeled query batches with no labeled examples. Visual similarity can reflect pose or background, so it may connect images from different classes. Each image's scores over the class vocabulary offer another way to choose neighbors. We introduce Behavioral Affinity Transduction (BAT), which builds a sparse graph from these class-response profiles. The graph supplies neighbor evidence during prediction. For EM-Dirichlet, it also screens candidate classes and initializes assignments. BAT plugs into direct and iterative readouts of a frozen VLM. Across diverse datasets, BAT improves the mean accuracy of every tested readout. In the main RN50 comparison, soft and hard EM-Dirichlet gain 4.22 and 2.72 percentage points.
Figures & tables
Figure 1: OxfordPets visual and response t-SNE projections.
Figure 2: BAT graph construction and readout integration.
Figure 3: Visual and response class separation by Cohen’s d .
Figure 4: Accuracy across query sizes for EM-Dirichlet, Gaussian, and K-means readouts.
Soft
Hard
Dataset
Base
BAT
Base
BAT
Caltech101
88.42
91.28
89.54
91.29
DTD
47.17
49.76
48.66
50.27
Aircraft
19.25
22.16
20.38
22.09
Flowers102
71.62
75.54
73.17
75.51
Food101
88.21
93.01
90.39
93.10
Table 1: EM-Dirichlet [ 11 ] accuracy on RN50 (%).
Readout
Base
+BAT
Gain
Direct prediction [ 14 ]
60.75
68.22
+7.47
Soft K-means [ 1 ]
25.26
71.82
+46.56
Hard K-means [ 9 ]
41.19
71.42
+30.23
KL K-means [ 4 ]
57.94
71.53
+13.60
EM-Gaussian [ 2 ]
19.69
68.46
+48.77
Soft EM-Dir. [ 11 ]
68.99
73.06
+4.07
Table 2: Readout accuracy (%) across ten datasets. Gains use unrounded means.
Backbone
Readout
Base
BAT
Gain
RN50
Soft
68.99
73.06
+4.07
Hard
70.69
73.28
+2.59
ViT-L/14
Soft
77.89
81.71
+3.83
Hard
79.16
81.83
+2.67
Table 3: EM-Dirichlet [ 11 ] by backbone (%).
Metric
Visual
Response
Same@5 (%)
11.29
14.64
Weighted Same@5 (%)
17.62
29.70
Oracle ratio (%)
67.26
83.81
Table 4: Graph quality across ten datasets.
Graph space
Soft
Hard
Visual
71.62
71.89
Response
72.13
72.37
Table 5: EM-Dirichlet accuracy by graph space (%).