Organizations: University of California, Santa Barbara · University of California, Los Angeles · George Mason University · The Chinese University of Hong Kong
Large vision-language models (VLMs) can judge visual similarity, but their judgments are not directly available as compact image embeddings for efficient comparison. We study how to transfer these preferences into CLIP while retaining its image--text capabilities. We introduce ASK, a kernel-based steering method that learns from elicited pairwise judgments without accessing teacher embeddings or collecting new human similarity annotations. ASK constructs positive semidefinite target kernels within small image groups and combines visual kernel matching with an image--text distributional anchor. Low-rank adapters jointly update the visual and text encoders while regularizing predictions toward frozen CLIP. After adaptation, retrieval uses CLIP image embeddings and cosine similarity, with no VLM calls. Experiments across five image domains, four CLIP backbones, and six judges evaluate teacher agreement, retrieval, and recognition retention. For ViT-B/16, mean retrieval mAP on classes excluded from adaptation increases from 53.8 to 75.0, compared with 71.7 for DINOv2 targets with KL anchoring. Mean zero-shot accuracy with jointly adapted encoders increases from 61.8% to 62.4%, averaged over 12 benchmarks and the five adaptation domains. Prompting provides an additional capability: selecting which visual distinctions the student learns. Human-annotated evaluations across four datasets support this criterion-specific control.
Figures & tables
Figure 1: Transferring VLM similarity preferences into CLIP while retaining recognition. (a) Criterion-specific retrieval. For one selected car query, columns show the exact top-two cosine matches from CLIP and ASK , ranked top to bottom. Separate ASK encoders emphasize car model; colour and lighting; pose, viewpoint, and framing; or background. Retrieval uses image embeddings alone, with no VLM calls. Additional examples and selection details are in Appendix F.2 . (b) Retrieval and recognition together. Eight CLIP ViT-B/16 methods on Dogs (Table 2 ) compare unseen-breed retrieval mAP with mean zero-shot accuracy over 12 benchmarks. The green region shows retrieval gains while matching or exceeding CLIP’s recognition accuracy. Feature labels identify teachers, including SigLIP ( Zhai et al., 2023 ) ; CLIP and KUEA nearly coincide.
Figure 2: ASK transfers a VLM’s similarity preferences into CLIP for retrieval and recognition. (1) Elicit supervision. A prompted VLM scores image pairs within small groups. Digit probabilities, when available, yield expected scores; symmetrization, positive-semidefinite (PSD) projection, and diagonal normalization produce a target kernel for each group. Neither teacher embeddings nor new human similarity annotations are required. (2) Adapt both encoders. Visual kernel matching teaches the image encoder the requested similarity structure. A KL anchor keeps image–text predictions over shared text prompts close to frozen CLIP’s predictions, updating LoRA adapters in both towers while pretrained weights remain fixed. (3) Deploy CLIP. Both adapted encoders are retained: image–image cosine similarity supports retrieval, and image–text similarity supports zero-shot classification. Deployment requires no VLM calls; a new criterion requires new judgments and adaptation. Heatmaps are measured; probability bars are illustrative.
Untrained CLIP
SupCon
KUEA
Qwen2.5-VL tower
DINOv2 + KL
ASK (ours)
Domain
Fid.
mAP
ZS
Fid.
mAP
ZS
Fid.
mAP
ZS
Fid.
mAP
ZS
Fid.
mAP
ZS
Fid.
mAP
ZS
Dogs
0.281
22.1
61.8
0.466
47.7
59.2
0.286
22.3
61.9
0.397
30.3
62.7
0.539
52.0
62.1
0.794
60.2
62.3
Textures
0.371
44.1
61.8
0.366
42.6
58.6
0.373
44.5
61.9
0.447
48.7
62.2
0.323
49.1
62.1
0.706
54.5
62.9
Flowers
0.679
77.4
61.8
0.440
83.3
60.7
0.681
77.8
61.8
0.686
83.5
61.7
0.486
86.8
61.6
0.798
85.8
62.1
Cars
0.616
67.2
61.8
0.434
81.1
61.3
0.618
67.4
61.9
0.651
68.3
61.7
0.581
83.9
61.9
0.859
85.1
62.1
Pets
0.582
58.2
61.8
0.471
87.0
60.7
0.589
58.7
61.9
0.556
67.0
62.2
0.548
86.9
62.2
0.917
89.3
62.6
Table 1: Supervision across five adaptation domains. CLIP ViT-B/16 with label, feature, or judgment supervision. KUEA uses an L2 feature anchor; other adapted methods use KL. Mean averages the five domains. Blue/red indicates improvement/decline relative to untrained CLIP; intensity is scaled separately per metric. Bold marks the best score.
ViT-B/32
ViT-B/16
ViT-L/14
ViT-L/14-336
Method
Fid.
mAP
ZS
Fid.
mAP
ZS
Fid.
mAP
ZS
Fid.
mAP
ZS
Untrained CLIP
0.288
18.7
60.6
0.281
22.1
61.8
0.283
30.5
66.1
0.264
30.0
66.9
Shuffled teacher
0.221
16.5
58.2
0.203
18.0
60.3
0.253
24.3
65.2
0.206
22.9
65.5
SupCon ( Khosla et al., 2020 )
0.513
39.3
56.2
0.466
47.7
59.2
0.503
65.6
66.4
0.435
63.4
65.5
KUEA ( Gong et al., 2025 )
0.291
18.8
60.6
0.286
22.3
61.9
0.286
30.9
66.1
0.269
30.4
67.0
DINOv2 + KL
0.533
40.7
59.3
0.539
52.0
62.1
0.521
64.1
67.0
0.518
65.8
67.8
Table 2: Supervision across four CLIP backbones. Models are adapted on Dogs. Blue/red indicates improvement/decline relative to each backbone’s untrained CLIP; intensity is scaled separately per metric. Bold marks the best score per column.
Judge
Verdicts
Student fidelity to judge
Student
Logp.
Bits
Q3.5-397B
GPT
Kimi
Sonnet
Q3.8-27B
G4-12B
mAP
ZS
Qwen3.5-397B
✓
2.22
0.79
0.76
0.80
0.74
0.78
0.68
59.66
62.00
GPT-5.6 Sol
✓
2.20
0.75
0.79
0.79
0.73
0.77
0.68
58.94
61.95
Kimi-K3
✓
2.34
0.77
0.77
0.84
0.74
0.78
0.67
58.96
61.85
Claude Sonnet 5
×
2.32
0.77
0.76
0.79
0.78
0.75
0.65
58.24
61.91
Qwen3.8-27B
✓
1.68
0.74
0.75
0.78
0.69
0.76
0.67
58.94
62.00
Table 3: Effect of the VLM judge. Rows identify the training judge; fidelity columns identify the evaluation judge. All use 31 training groups. Blue/red indicates improvement/decline relative to untrained CLIP, scaled per metric. Bold marks column bests.
Figure 3: Visual geometry and criterion-specific retrieval. (a) Original CLIP, the VLM breed teacher, and breed-aligned CLIP on 250 held-out dog images. Normalized kernels share breed ordering and a [0,1] colour scale; the teacher kernel is PSD-projected. (b) Corresponding similarity distributions for same-breed and different-breed pairs, each normalized independently; Δμ is the same-breed mean minus the different-breed mean. (c) Cosine retrieval on CUB-200-2011, with human annotations defining relevance for wing colour, bill shape, and breast pattern. Queries retrieve images of other species; bars report nDCG@10 ×100 (higher is better). Each ASK bar uses the corresponding criterion-trained encoder. Human attributes do not supervise ASK ; Appendix D.5 details evaluation and baseline settings.
(a) Cross-domain retrieval ( Δ mAP)
(b) Criterion specificity ( Δ Spearman)
Elicited on
Dogs
Textures
Flowers
Cars
Pets
Asked about
Breed
Colour/ lighting
Pose/view
Coat tex.
Backgr.
Untrained
22.1
44.1
77.4
67.2
58.2
Untrained
0.28
0.34
0.49
0.19
0.27
Dogs
+38.1
+6.2
+8.9
+14.7
+29.0
Breed
+0.51
+0.07
− 0.25
+0.53
− 0.12
Textures
+2.4
+10.4
+4.0
+4.1
+3.8
Colour/lighting
+0.20
+0.41
− 0.10
+0.16
+0.18
Flowers
+7.3
+4.6
+8.4
+7.4
+9.9
Pose/view
+0.02
+0.02
+0.26
+0.00
+0.05
Cars
+10.9
+3.2
+7.0
+17.9
+12.8
Coat tex.
+0.41
− 0.04
− 0.28
+0.68
− 0.17
Table 4: Cross-domain retrieval and criterion specificity on fixed images. (a) Training domain versus retrieval domain, using unseen-class labels. (b) Training criterion versus held-out teacher, on fixed dog image splits. DINOv2+KL and Shuffled are dog-trained; Shuffled is a negative control with permuted breed targets. Entries are gains over CLIP (absolute scores first). Blue/red denotes improvement/decline; bold marks column bests.
Judge
Neg.%
Δ Fid
Δ mAP
Δ ZS
Gemma 4 12B
6.4
− 0.009
− 1.43
− 0.41
GPT-5.6
3.8
− 0.005
− 0.53
− 0.29
Qwen3.5
3.5
+ 0.002
− 1.45
− 0.26
Sonnet 5
2.6
− 0.001
− 0.17
− 0.23
Table 5: Ablations of the alignment recipe. (a) Raw minus PSD for the four cases with highest negative spectral mass (Neg.%). (b) Supervision targets under matched MSE, KL, and LoRA; mAP on unseen breeds. (c,d) KL-anchored heads versus LoRA. (e) Anchor regularisation. (f) Training objective. Blue/red indicates improvement/decline relative to PSD in (a), LoRA in (c,d), and the default in (b,e,f); bold marks metric bests.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Method
IN-1k
C-10
C-100
Caltech
FER
Pets
DTD
RESISC
EuroSAT
PCam
IN-Sk.
IN-O
Mean
ViT-B/32
Untrained CLIP
63.5
89.7
63.3
81.6
41.4
87.3
44.3
53.6
50.4
62.3
42.3
47.8
60.6
Shuffled teacher
62.6
87.6
59.3
80.6
36.7
86.0
42.6
51.0
42.0
62.1
41.4
47.0
58.2
SupCon
58.7
83.5
57.5
79.1
45.8
79.9
40.9
50.7
38.9
57.4
38.9
43.5
56.2
KUEA
63.5
89.6
63.4
81.6
41.5
87.4
44.0
53.6
51.4
62.1
42.4
47.3
60.6
DINOv2 + KL
63.5
87.6
62.0
81.0
37.4
88.0
42.2
53.4
46.7
60.7
42.0
47.2
59.3
SigLIP tower
63.6
89.0
63.0
81.3
39.3
88.2
43.6
54.3
52.3
61.4
42.3
48.2
60.5
Appendix
Table 6: Per-benchmark zero-shot accuracy across CLIP backbones. Top-1 accuracy (%); Mean is the unweighted average over 12 benchmarks. Blue/red cells indicate gains/losses relative to untrained CLIP within each backbone, scaled separately per column across backbones; bold marks the best result per column.
In domain: 25 unseen breeds
Out of domain: 50 non-dog classes
Unseen-breed fidelity ( ρ )
Method
R@1
R@5
P@5
mAP
Δ
R@1
R@5
P@5
mAP
Δ
All
Within
Between
Target
Untrained CLIP
47.0
82.2
38.8
22.1
—
77.0
93.5
44.1
70.0
—
0.268
0.231
0.245
Shuffled teacher
36.6
76.0
28.8
18.0
−4.1
73.5
93.0
43.1
67.7
−2.3
0.239
0.227
0.211
SupCon
69.4
87.2
62.0
47.7
+25.6
78.5
92.0
43.9
71.3
+1.3
0.546
0.471
0.508
KUEA
47.4
82.2
38.9
22.3
+0.2
78.0
93.5
44.0
70.1
+0.1
0.271
0.232
0.247
DINOv2 + KL
75.2
92.0
67.5
52.0
+29.9
81.0
95.0
47.1
74.8
+4.8
0.560
0.536
0.523
Appendix
Table 7: Retrieval and fidelity on unseen classes. CLIP ViT-B/16. Retrieval uses 500 unseen-breed and 200 non-dog images; scores are percentages and Δ is the mAP gain over CLIP. Fidelity is Spearman ρ against Qwen3.8-27B on 250 unseen-breed images. All uses every pair; Within uses pairs of the same breed; Between uses pairs of different breeds. Blue/red indicates improvement/decline from CLIP, scaled per metric column; bold marks the best score within each block.
Training criterion / model
Colour/lighting
Pose/view
Background
Untrained CLIP
0.343
0.464
0.535
Breed
0.302
0.390
0.472
Colour/lighting
0.428
0.451
0.576
Pose/view
0.332
0.492
0.552
Coat texture
0.320
0.412
0.498
Background
0.387
0.413
0.795
Appendix
Table 8: Criterion transfer beyond dogs. Spearman fidelity to raw Qwen3.5-397B judgments on 200 non-dog images (50 classes; 19,900 pairs). Dog-trained encoders receive no non-dog adaptation, and each column reuses its dog scoring prompt unchanged. Each model keeps its training-fitted cubic link; CLIP uses the breed link. Blue/red shading denotes improvement/decline relative to CLIP, scaled within the table; bold marks column bests.
Training criterion / model
Car model
Colour/lighting
Pose/view
Background
Untrained CLIP
0.616
0.232
0.317
0.081
Car model
0.859
0.227
0.384
0.051
Colour/lighting
0.409
0.704
0.257
0.231
Pose/view
0.497
0.196
0.756
0.096
Background
0.209
0.303
0.154
0.813
DINOv2+KL
0.597
0.190
0.310
0.057
Appendix
Table 9: Criterion specificity on cars. Spearman fidelity to raw Qwen3.5-397B judgments on the same 250 held-out car images (25 adaptation classes; 31,125 pairs). Rows identify car-trained encoders and columns the evaluation criteria. Each model keeps its training-fitted cubic link; CLIP uses the car-model link. Blue/red shading denotes improvement/decline relative to CLIP, scaled within the table; bold marks column bests.
(a) Attribute retrieval
Model
Wing colour
Bill shape
Breast pattern
Untrained CLIP
34.55
49.98
52.31
DINOv2+KL
35.62
50.28
52.72
CLIP text subspace
35.42
38.94
45.97
CLIP text profile
34.44
38.25
45.91
VLM2Vec
34.87
50.09
52.10
Appendix
Table 10: Criterion retrieval and baseline checks on CUB-200-2011. Cosine nDCG@10 ( ×100 ) with cross-species galleries. (a) Model comparison; each ASK row is one fixed criterion-trained student. (b) Instruction settings and AligNet output spaces. Symmetric conditioning uses the same instruction for query and gallery; * marks validation-selected settings used in (a), where AligNet uses its image encoder. Blue/red denotes improvement/decline from CLIP in (a) or each block's first row in (b), scaled per column; bold marks column bests in (a) and block bests in (b).
Figure 4: Attribute retrieval on species excluded from adaptation. ASK and DINOv2+KL adapt on 100 CUB species and evaluate on the other 100, with cross-species galleries. Each bar reports cosine nDCG@10 for one fixed encoder; ASK uses the matching criterion. VLM2Vec and MagicLens retain the earlier CUB settings, selected on validation images spanning all 200 species; their prompt selection is not species-disjoint.
Figure 5: Criterion transfer evaluated with independent annotations. (a) Car viewpoint from PASCAL3D+ azimuth annotations. (b) Background composition from COCO-Stuff masks. (c) Non-dog body pose from AP-10K keypoints. Each bar reports cosine nDCG@10 for one fixed encoder, with no weight updates on these datasets. VLM2Vec and MagicLens use separate validation images to select instructions. ASK uses the car Pose/view, dog background, and dog Pose/view encoders, respectively; DINOv2+KL uses the corresponding adaptation domain.
Method
IN-1k
C-10
C-100
Caltech
FER
Pets
DTD
RESISC
EuroSAT
PCam
IN-Sk.
IN-O
Mean
Untrained CLIP
68.3
90.0
65.6
82.2
46.3
89.0
44.9
58.2
55.9
50.7
48.3
42.3
61.8
Dogs
SupCon
63.7
87.5
62.2
79.7
41.9
84.8
43.3
57.8
53.2
53.0
44.6
39.0
59.2
KUEA
68.3
90.0
65.8
82.0
46.6
88.9
44.8
58.1
56.2
51.2
48.3
42.6
61.9
Qwen2.5-VL tower
68.3
90.4
67.1
82.5
47.8
90.4
45.1
57.5
58.1
53.7
48.1
43.2
62.7
DINOv2 + KL
68.5
89.5
66.3
82.2
45.9
91.0
43.8
58.1
57.1
50.9
48.4
43.1
62.1
ASK
67.7
89.2
66.6
82.7
44.9
90.3
43.9
60.4
59.0
51.6
47.8
43.1
62.3
Appendix
Table 11: Per-benchmark accuracy after domain adaptation. CLIP ViT-B/16; Mean averages the 12 benchmarks. Blue/red indicates improvement/decline from untrained CLIP, scaled per column across domains. Bold marks the highest adapted score within each domain when it exceeds CLIP. Same-domain evaluations are discussed in the text.
Judge
Agreement with judge
Self, swapped order
Own student
Q3.5-397B
GPT
Kimi
Sonnet
Q3.8-27B
G4-12B
ρS
ρP
∣Δ∣
Fid
Ratio
Qwen3.5-397B
—
0.753
0.799
0.706
0.819
0.702
0.837
0.921
0.35
0.791
94%
GPT-5.6 Sol
0.753
—
0.811
0.725
0.770
0.679
0.901
0.926
0.32
0.794
88%
Kimi-K3
0.799
0.811
—
0.750
0.829
0.719
0.948
0.975
0.20
0.838
88%
Claude Sonnet 5
0.706
0.725
0.750
—
0.690
0.605
0.820
0.912
0.33
0.778
95%
Qwen3.8-27B
0.819
0.770
0.829
0.690
—
0.738
0.829
0.858
0.87
0.760
92%
Appendix
Table 12: Agreement and order consistency of VLM judges. Pairwise agreement is Spearman correlation. Blue intensity encodes absolute agreement on a shared scale; bold marks the highest cross-judge value per column. Self-agreement uses swapped image order; Ratio is student fidelity divided by self-Spearman agreement.
Fidelity
Retrieval
Zero-shot
Judge
ρS
ρP
Base ρS
mAP
12 sets
IN-1k
ViT-B/32
Untrained CLIP
—
—
—
18.7
60.6
63.5
Qwen3.5-397B
0.761
0.769
0.230
48.0
58.1
61.9
GPT-5.6 Sol
0.745
0.733
0.274
47.0
57.2
61.6
Kimi-K3
0.815
0.813
0.182
49.1
57.7
61.7
Claude Sonnet 5
0.719
0.752
0.251
48.0
58.0
61.7
Appendix
Table 13: VLM judges across CLIP backbones. All students use 31 scoring groups. Spearman shading compares ρS with Base ρS in the same judge row. These judge-specific CLIP baselines replace a single value in the Untrained CLIP row. Retrieval and zero-shot shading compare with that backbone’s Untrained CLIP row. Blue/red denotes improvement/decline, scaled per metric; Pearson is unshaded. Bold marks column bests within each backbone.
Judge
IN-1k
C-10
C-100
Caltech
FER
Pets
DTD
RESISC
EuroSAT
PCam
IN-Sk.
IN-O
Mean
ViT-B/32
Untrained CLIP
63.5
89.7
63.3
81.6
41.4
87.3
44.3
53.6
50.4
62.3
42.3
47.8
60.6
Qwen3.5-397B
61.9
86.5
60.8
80.1
36.8
86.2
41.1
52.4
42.1
61.2
41.1
46.5
58.1
GPT-5.6 Sol
61.6
85.6
58.6
79.9
32.6
85.6
41.7
52.6
40.9
59.4
41.0
46.7
57.2
Kimi-K3
61.7
86.1
59.8
79.9
36.6
85.2
41.1
51.6
43.3
59.9
40.6
46.6
57.7
Claude Sonnet 5
61.7
86.2
60.7
80.1
38.2
85.7
41.8
52.7
41.6
60.1
41.0
46.8
58.0
ViT-B/16
Untrained CLIP
68.3
90.0
65.6
82.2
46.3
89.0
44.9
58.2
55.9
50.7
48.3
42.3
61.8
Appendix
Table 14: Per-benchmark zero-shot accuracy across judges and backbones. Top-1 accuracy (%) for Table 13 ; Mean averages the 12 benchmarks. Blue/red indicates improvement/decline from each backbone's untrained CLIP, scaled per column across backbones. Bold marks the highest adapted-model score within each backbone when it exceeds CLIP.
Judge
Provider
$/M tokens input / output
P50 latency (s)
Output tokens/s
Est. cost ($)
Est. time (h)
Qwen3.5-397B
Alibaba Cloud Int.
0.39 / 2.34
1.11
61
5.4–12.3
1.4
GPT-5.6 Sol
OpenAI
2.00 / 10.00
3.58
49
27.6–63.1
4.5
Kimi-K3
Parasail
3.00 / 15.00
1.11
68
41.3–94.7
1.4
Claude Sonnet 5
Anthropic
2.00 / 10.00
2.30
63
27.6–63.1
2.9
Qwen3.8-27B
Parasail
0.24 / 2.20
1.35
58
3.4–7.7
1.7
Gemma 4 12B †
DeepInfra
0.05 / 0.15
0.75
14
0.7–1.6
1.1
Appendix
Table 15: OpenRouter teacher-scoring budget estimates. For 44,450 judgments: 300–700 input tokens per pair (including both images), two output tokens without reasoning, and ten concurrent requests. Names follow the experiments; † marks Gemma 3 12B pricing used as a proxy for Gemma 4 12B.
Figure 6: Strong performance with fewer teacher API calls. (a) Distinct judgments, including kernel calibration. (b) Cosine retrieval on unseen dog breeds. (c) Zero-shot recognition across 12 benchmarks. Full budget is the default 93-group baseline; Sparse 15 uses 15 pairs per block.
Setting
Blocks
Pairs/block
Distinct judgments
Fidelity
Retrieval
Zero-shot
Update pairs
Total pairs
ρS
mAP
12 sets
Full budget (dense)
93
496
44,450
44,450
0.794
60.16
62.28
Dense, 62 groups
62
496
30,174
30,174
0.790
60.07
62.28
Dense, 31 groups
31
496
15,376
15,376
0.781
58.88
62.01
Sparse 15 (random pairs)
400
15
5,624
14,320
0.780
59.41
62.56
All cached pairs per block
400
≈ 45
14,703
21,311
0.793
60.06
62.47
Appendix
Table 16: Query budgets and performance. Dog-trained ViT-B/16, 3,000 updates. Update pairs count distinct training judgments; total pairs also include the 9,920 calibration pairs, without double counting. Blue/red indicates improvement/decline from the full-budget baseline, scaled per metric; bold marks the best score per column.
Supervision
Fidelity
Retrieval mAP
ZS
Spearman
Pearson
Unseen breeds
Non-dog
Untrained CLIP
0.281
0.280
22.06
70.01
61.81
Binary class labels
0.453
0.707
43.11
64.40
59.83
Grouped class labels
0.451
0.706
43.71
63.81
60.49
WordNet path similarity
0.444
0.707
52.01
69.41
60.70
Class-pair mean (VLM)
0.730
0.801
57.17
72.66
61.77
Appendix
Table 17: Class labels versus VLM judgments. CLIP ViT-B/16 on Dogs, with shared kernel MSE and KL anchoring. Fidelity reports Spearman/Pearson agreement with raw held-out judgments. Retrieval mAP and the 12-benchmark zero-shot mean (ZS) are percentages. Blue/red indicates improvement/decline relative to untrained CLIP, scaled per column; bold marks the best score.
Judge
Criterion
Neg.%
Fidelity
mAP
ZS
PSD
raw
PSD
raw
PSD
raw
Gemma 4 12B
Dogs / breed
6.4
0.771
0.761
55.93
54.49
61.71
61.30
GPT-5.6 Sol
Dogs / breed
3.8
0.790
0.785
58.99
58.45
62.10
61.81
Qwen3.5-397B
Cars / pose-view
3.5
0.758
0.760
45.75
44.30
60.88
60.63
Claude Sonnet 5
Dogs / breed
2.6
0.777
0.776
58.57
58.40
62.07
61.84
Qwen3.5-397B
Dogs / breed
1.8
0.792
0.791
59.79
59.57
62.26
62.29
Appendix
Table 18: PSD projection across judges and criteria. Neg.% denotes negative spectral mass. Retrieval uses unseen breed labels for Dogs and unseen car-model labels for Cars. Raw-target cells show gains (blue) or losses (red) relative to PSD targets, scaled per metric; bold marks the higher score within each pair. Displayed ties are uncoloured.
Figure 7: Verdict extraction with Qwen3.5-397B. Expected-digit scores (green), rounded expectations (blue), and argmax digits (coral). (a) Fidelity uses the normalized kernel against expected-digit judgments. (b) Cosine-retrieval mAP uses unseen-breed labels. (c) Zero-shot accuracy averages 12 benchmarks. Labels give exact reported values; dashed lines mark expected-digit results. Each panel uses its own zoomed vertical scale; higher is better.
Table 19: Regulariser families and weights. Dog-trained CLIP ViT-B/16. Here, P0 and P denote the frozen and adapted image–text prediction distributions, respectively. Blue/red indicates improvement/decline from the default KL anchor, scaled per metric across all rows. Bold marks the best score within each block.
(a) Fidelity to the criterion’s own teacher
(b) Δ mAP over untrained CLIP
Criterion
Linear
MLP
LoRA
Retrieval on
Linear
MLP
LoRA
Breed
0.622
0.734
0.794
Unseen breeds
+ 13.9
+ 22.8
+ 38.1
Colour/lighting
0.491
0.673
0.747
Non-dog
+ 1.3
+ 1.9
+ 4.9
Pose/view
0.564
0.674
0.748
Textures
+ 1.3
+ 2.2
+ 6.2
Coat texture
0.699
0.826
0.874
Flowers
+ 2.6
+ 3.2
+ 8.9
Background
0.730
0.887
0.919
Cars
+ 3.7
+ 5.5
+ 14.7
Appendix
Table 20: Frozen-feature heads versus LoRA. All models use KL anchoring. (a) Criterion-specific Spearman fidelity. (b) Retrieval gains of breed-trained models. Blue/red indicates improvement/decline relative to LoRA, scaled per panel; bold marks row bests.
Training similarity
Formula
Loss
Fidelity
mAP
ZS
Listed similarity
Cosine
ρS
ρP
ρS
ρP
Untrained CLIP
—
–
0.281
0.280
0.281
0.256
22.06
61.81
Dot product
γxi⊤xj+c
MSE
0.696
0.677
0.688
0.672
40.50
60.63
Cosine
x^i⊤x^j
MSE
0.756
0.811
0.756
0.811
58.35
62.57
Normalized cubic ( ASK )
k~γ,c(xi,xj)
MSE
0.794
0.840
0.794
0.792
60.16
62.28
Appendix
Table 21: Training similarity and objective under matched LoRA adaptation. All adapted rows share PSD targets and the KL anchor. Fidelity compares the listed similarity and deployed cosine similarity with the same raw held-out judgments, without test-time fitting. Untrained CLIP uses the default cubic link in the listed-similarity columns. mAP is cosine retrieval on unseen breeds; ZS is the 12-benchmark mean (both %). Blue/red indicates improvement/decline relative to cubic MSE; bold marks column bests.
Factor
Variant
Fidelity
Retrieval
Zero-shot
ρS
ρP
mAP
12 sets
IN-1k
Default recipe ( ASK )
0.794
0.840
60.16
62.28
67.70
Kernel loss default: Frobenius
Spectral norm
0.803
0.856
60.75
61.60
66.47
Top-4 eigenvalues
0.801
0.854
60.79
61.85
66.21
Pair budget default: 93 groups
31 groups (15,376 unique pairs)
0.781
0.822
58.88
62.01
67.71
62 groups (30,174 unique pairs)
0.790
0.836
60.07
62.28
67.68
Appendix
Table 22: Sensitivity to training choices. One factor changes from the default at a time. Blue/red indicates improvement/decline from the default, scaled per metric; bold marks the best score per column.
Figure 8: Kernel alignment across VLM judges. Original CLIP, teacher target, aligned CLIP, and signed residual on 100 held-out dog images. Each row uses its judge’s saved link; residuals are aligned minus target.
Figure 9: Kernel geometry under alternative supervision. Top: target and encoder kernels on 100 held-out dog images. Bottom: signed residuals relative to the Qwen breed target. The VLM-aligned model uses the 93-group protocol.
Figure 10: Target and learned geometry for five criteria. Rows show the VLM target, criterion-aligned CLIP, and signed residual on the same 250 held-out dog images. Columns retain the same image ordering across criteria.
Figure 11: Dog retrieval under different criteria. Each row shows the query (teal border) and its top three neighbours. Green/red borders indicate matching/different breeds only in the class-retrieval rows.
Figure 12: Car retrieval under different criteria. Each row shows the query (teal border) and its top three neighbours. Green/red borders indicate matching/different car classes only in the class-retrieval rows.
Figure 13: Different models, different nearest neighbours. Each row shows the query (teal) and the model’s top two cosine retrievals. ASK rows identify the training judge, not direct retrieval by the VLM. Green/red borders indicate matching/different dataset class labels.
Figure 14: Same candidates, different notions of similarity. Scores are query–candidate cosine similarities. Outlined cells mark the highest score among the four displayed candidates in each row; colours identify criteria.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
CLIP's visual encoder produces only global image representations, limiting its use in region-level tasks. Existing adaptations rely on visual prompting, input masking, or encoder fine-tuning, each compromising pre-trained representations. We propose LAS-CLIP, a Lightweight Adapter Steering approach that keeps every CLIP parameter frozen. A compact MaskAdapter generates per-head, per-layer attention biases from an input mask and injects them into the frozen self-attention layers, steering attention toward the target region. Crucially, because the backbone remains strictly untouched, LAS-CLIP seamlessly reverts to vanilla CLIP when no mask is provided, preserving its foundational zero-shot capabilities. With approximately 116K to 145K trainable parameters and 100K training samples on two T4 GPUs, LAS-CLIP achieves competitive or superior results compared to Alpha-CLIP on ImageNet-S zero-shot classification and RefCOCO referring expression comprehension, despite the latter fine-tuning its entire encoder on millions of samples. Qualitative analysis further confirms stronger representational fidelity under incorrect masks and in downstream generation.
Anh-Khoa Dinh-Duc, Duc-Tai Dinh, Tam V. Nguyen +1
University of Science, Ho Chi Minh City, Vietnam · Viet Nam National University, Ho Chi Minh City, Vietnam · University of Dayton, USA
Contrastive Language-Image Pre-training (CLIP) relies on Vision Transformers whose attention mechanism is susceptible to spurious correlations, and scales quadratically with resolution. To address these limitations, We present CLIMP, the first fully Mamba-based contrastive vision-language model that replaces both the vision and text encoders with Mamba. The new architecture encodes sequential structure in both vision and language, with VMamba capturing visual spatial inductive biases, reducing reliance on spurious correlations and producing an embedding space favorable for cross-modal retrieval and out-of-distribution robustness-surpassing OpenAI's CLIP-ViT-B by 7.5% on ImageNet-O. CLIMP naturally supports variable input resolutions without positional encoding interpolation or specialized training, achieving up to 6.6% higher retrieval accuracy at 16x training resolution while using 5x less memory and 1.8x fewer FLOPs. The autoregressive text encoder further overcomes CLIP's fixed context limitation, enabling dense captioning retrieval. Our findings suggest that Mamba exhibits advantageous properties for vision-language learning, making it a compelling alternative to Transformer-based CLIP.The code and models are publicly available at https://github.com/NimrodShabtay/CLIMP}
Pretrained vision-language models such as CLIP excel at zero-shot recognition but often fail at compositionality, particularly attribute-object and relational structures. Recent studies mitigate this issue by augmenting training with synthetic hard negatives generated by a cascade of large language models and text-to-image models, which incurs substantial pipeline overhead. We instead propose MLLMCLIP, a heterogeneous distillation framework that transfers multimodal knowledge directly from a generative Multimodal Large Language Model (MLLM) teacher into a discriminative CLIP student, bypassing synthetic data entirely. To bridge the architectural mismatch between the two paradigms, we introduce an attention-based per-layer token selection and a CKA-based distillation loss. Compared to prior CLIP-enhancement methods, MLLMCLIP achieves state-of-the-art compositional accuracy while delivering consistent gains on standard zero-shot classification and image-text retrieval, showing that feature-level distillation strengthens both compositional and general vision-language representation capability.