Kernel-Based Steering of CLIP with Vision-Language Model Preferences

Sajjad Ghiasvand; Haniyeh Ehsani Oskouie; Sina Mansouri; Mahnoosh Alizadeh; Farzan Farnia; Ramtin Pedarsani

cs.CVSep 26, 2026

Kernel-Based Steering of CLIP with Vision-Language Model Preferences

Authors: Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Sina Mansouri, Mahnoosh Alizadeh, Farzan Farnia, Ramtin Pedarsani

Organizations: University of California, Santa Barbara · University of California, Los Angeles · George Mason University · The Chinese University of Hong Kong

Abstract

Large vision-language models (VLMs) can judge visual similarity, but their judgments are not directly available as compact image embeddings for efficient comparison. We study how to transfer these preferences into CLIP while retaining its image--text capabilities. We introduce ASK, a kernel-based steering method that learns from elicited pairwise judgments without accessing teacher embeddings or collecting new human similarity annotations. ASK constructs positive semidefinite target kernels within small image groups and combines visual kernel matching with an image--text distributional anchor. Low-rank adapters jointly update the visual and text encoders while regularizing predictions toward frozen CLIP. After adaptation, retrieval uses CLIP image embeddings and cosine similarity, with no VLM calls. Experiments across five image domains, four CLIP backbones, and six judges evaluate teacher agreement, retrieval, and recognition retention. For ViT-B/16, mean retrieval mAP on classes excluded from adaptation increases from 53.8 to 75.0, compared with 71.7 for DINOv2 targets with KL anchoring. Mean zero-shot accuracy with jointly adapted encoders increases from 61.8% to 62.4%, averaged over 12 benchmarks and the five adaptation domains. Prompting provides an additional capability: selecting which visual distinctions the student learns. Human-annotated evaluations across four datasets support this criterion-specific control.

Figures & tables

Figure 1: Transferring VLM similarity preferences into CLIP while retaining recognition. (a) Criterion-specific retrieval. For one selected car query, columns show the exact top-two cosine matches from CLIP and ASK , ranked top to bottom. Separate ASK encoders emphasize car model; colour and lighting; pose, viewpoint, and framing; or background. Retrieval uses image embeddings alone, with no VLM calls. Additional examples and selection details are in Appendix F.2 . (b) Retrieval and recognition together. Eight CLIP ViT-B/16 methods on Dogs (Table 2 ) compare unseen-breed retrieval mAP with mean zero-shot accuracy over 12 benchmarks. The green region shows retrieval gains while matching or exceeding CLIP’s recognition accuracy. Feature labels identify teachers, including SigLIP ( Zhai et al., 2023 ) ; CLIP and KUEA nearly coincide.

Figure 2: ASK transfers a VLM’s similarity preferences into CLIP for retrieval and recognition. (1) Elicit supervision. A prompted VLM scores image pairs within small groups. Digit probabilities, when available, yield expected scores; symmetrization, positive-semidefinite (PSD) projection, and diagonal normalization produce a target kernel for each group. Neither teacher embeddings nor new human similarity annotations are required. (2) Adapt both encoders. Visual kernel matching teaches the image encoder the requested similarity structure. A KL anchor keeps image–text predictions over shared text prompts close to frozen CLIP’s predictions, updating LoRA adapters in both towers while pretrained weights remain fixed. (3) Deploy CLIP. Both adapted encoders are retained: image–image cosine similarity supports retrieval, and image–text similarity supports zero-shot classification. Deployment requires no VLM calls; a new criterion requires new judgments and adaptation. Heatmaps are measured; probability bars are illustrative.

	Untrained CLIP			SupCon			KUEA			Qwen2.5-VL tower			DINOv2 $+$ KL			ASK (ours)
Domain	Fid.	mAP	ZS	Fid.	mAP	ZS	Fid.	mAP	ZS	Fid.	mAP	ZS	Fid.	mAP	ZS	Fid.	mAP	ZS
Dogs	0.281	22.1	61.8	0.466	47.7	59.2	0.286	22.3	61.9	0.397	30.3	62.7	0.539	52.0	62.1	0.794	60.2	62.3
Textures	0.371	44.1	61.8	0.366	42.6	58.6	0.373	44.5	61.9	0.447	48.7	62.2	0.323	49.1	62.1	0.706	54.5	62.9
Flowers	0.679	77.4	61.8	0.440	83.3	60.7	0.681	77.8	61.8	0.686	83.5	61.7	0.486	86.8	61.6	0.798	85.8	62.1
Cars	0.616	67.2	61.8	0.434	81.1	61.3	0.618	67.4	61.9	0.651	68.3	61.7	0.581	83.9	61.9	0.859	85.1	62.1
Pets	0.582	58.2	61.8	0.471	87.0	60.7	0.589	58.7	61.9	0.556	67.0	62.2	0.548	86.9	62.2	0.917	89.3	62.6

Table 1: Supervision across five adaptation domains. CLIP ViT-B/16 with label, feature, or judgment supervision. KUEA uses an L2 feature anchor; other adapted methods use KL. Mean averages the five domains. Blue/red indicates improvement/decline relative to untrained CLIP; intensity is scaled separately per metric. Bold marks the best score.

	ViT-B/32			ViT-B/16			ViT-L/14			ViT-L/14-336
Method	Fid.	mAP	ZS	Fid.	mAP	ZS	Fid.	mAP	ZS	Fid.	mAP	ZS
Untrained CLIP	0.288	18.7	60.6	0.281	22.1	61.8	0.283	30.5	66.1	0.264	30.0	66.9
Shuffled teacher	0.221	16.5	58.2	0.203	18.0	60.3	0.253	24.3	65.2	0.206	22.9	65.5
SupCon ( Khosla et al., 2020 )	0.513	39.3	56.2	0.466	47.7	59.2	0.503	65.6	66.4	0.435	63.4	65.5
KUEA ( Gong et al., 2025 )	0.291	18.8	60.6	0.286	22.3	61.9	0.286	30.9	66.1	0.269	30.4	67.0
DINOv2 $+$ KL	0.533	40.7	59.3	0.539	52.0	62.1	0.521	64.1	67.0	0.518	65.8	67.8

Table 2: Supervision across four CLIP backbones. Models are adapted on Dogs. Blue/red indicates improvement/decline relative to each backbone’s untrained CLIP; intensity is scaled separately per metric. Bold marks the best score per column.

Judge	Verdicts		Student fidelity to judge						Student
Judge	Logp.	Bits	Q3.5-397B	GPT	Kimi	Sonnet	Q3.8-27B	G4-12B	mAP	ZS
Qwen3.5-397B	✓	2.22	0.79	0.76	0.80	0.74	0.78	0.68	59.66	62.00
GPT-5.6 Sol	✓	2.20	0.75	0.79	0.79	0.73	0.77	0.68	58.94	61.95
Kimi-K3	✓	2.34	0.77	0.77	0.84	0.74	0.78	0.67	58.96	61.85
Claude Sonnet 5	×	2.32	0.77	0.76	0.79	0.78	0.75	0.65	58.24	61.91
Qwen3.8-27B	✓	1.68	0.74	0.75	0.78	0.69	0.76	0.67	58.94	62.00

Table 3: Effect of the VLM judge. Rows identify the training judge; fidelity columns identify the evaluation judge. All use 31 training groups. Blue/red indicates improvement/decline relative to untrained CLIP, scaled per metric. Bold marks column bests.

Figure 3: Visual geometry and criterion-specific retrieval. (a) Original CLIP, the VLM breed teacher, and breed-aligned CLIP on 250 held-out dog images. Normalized kernels share breed ordering and a $[0,1]$ colour scale; the teacher kernel is PSD-projected. (b) Corresponding similarity distributions for same-breed and different-breed pairs, each normalized independently; $\Delta\mu$ is the same-breed mean minus the different-breed mean. (c) Cosine retrieval on CUB-200-2011, with human annotations defining relevance for wing colour, bill shape, and breast pattern. Queries retrieve images of other species; bars report nDCG@10 $\times 100$ (higher is better). Each ASK bar uses the corresponding criterion-trained encoder. Human attributes do not supervise ASK ; Appendix D.5 details evaluation and baseline settings.

(a) Cross-domain retrieval ( $\Delta$ mAP)						(b) Criterion specificity ( $\Delta$ Spearman)
Elicited on	Dogs	Textures	Flowers	Cars	Pets	Asked about	Breed	Colour/ lighting	Pose/view	Coat tex.	Backgr.
Untrained	22.1	44.1	77.4	67.2	58.2	Untrained	0.28	0.34	0.49	0.19	0.27
Dogs	+38.1	+6.2	+8.9	+14.7	+29.0	Breed	+0.51	+0.07	$-$ 0.25	+0.53	$-$ 0.12
Textures	+2.4	+10.4	+4.0	+4.1	+3.8	Colour/lighting	+0.20	+0.41	$-$ 0.10	+0.16	+0.18
Flowers	+7.3	+4.6	+8.4	+7.4	+9.9	Pose/view	+0.02	+0.02	+0.26	+0.00	+0.05
Cars	+10.9	+3.2	+7.0	+17.9	+12.8	Coat tex.	+0.41	$-$ 0.04	$-$ 0.28	+0.68	$-$ 0.17

Table 4: Cross-domain retrieval and criterion specificity on fixed images. (a) Training domain versus retrieval domain, using unseen-class labels. (b) Training criterion versus held-out teacher, on fixed dog image splits. DINOv2+KL and Shuffled are dog-trained; Shuffled is a negative control with permuted breed targets. Entries are gains over CLIP (absolute scores first). Blue/red denotes improvement/decline; bold marks column bests.

Judge	Neg.%	$\Delta$ Fid	$\Delta$ mAP	$\Delta$ ZS
Gemma 4 12B	6.4	$-$ 0.009	$-$ 1.43	$-$ 0.41
GPT-5.6	3.8	$-$ 0.005	$-$ 0.53	$-$ 0.29
Qwen3.5	3.5	$+$ 0.002	$-$ 1.45	$-$ 0.26
Sonnet 5	2.6	$-$ 0.001	$-$ 0.17	$-$ 0.23

Table 5: Ablations of the alignment recipe. (a) Raw minus PSD for the four cases with highest negative spectral mass (Neg.%). (b) Supervision targets under matched MSE, KL, and LoRA; mAP on unseen breeds. (c,d) KL-anchored heads versus LoRA. (e) Anchor regularisation. (f) Training objective. Blue/red indicates improvement/decline relative to PSD in (a), LoRA in (c,d), and the default in (b,e,f); bold marks metric bests.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

	Method	IN-1k	C-10	C-100	Caltech	FER	Pets	DTD	RESISC	EuroSAT	PCam	IN-Sk.	IN-O	Mean
ViT-B/32	Untrained CLIP	63.5	89.7	63.3	81.6	41.4	87.3	44.3	53.6	50.4	62.3	42.3	47.8	60.6
	Shuffled teacher	62.6	87.6	59.3	80.6	36.7	86.0	42.6	51.0	42.0	62.1	41.4	47.0	58.2
	SupCon	58.7	83.5	57.5	79.1	45.8	79.9	40.9	50.7	38.9	57.4	38.9	43.5	56.2
	KUEA	63.5	89.6	63.4	81.6	41.5	87.4	44.0	53.6	51.4	62.1	42.4	47.3	60.6
	DINOv2 $+$ KL	63.5	87.6	62.0	81.0	37.4	88.0	42.2	53.4	46.7	60.7	42.0	47.2	59.3
	SigLIP tower	63.6	89.0	63.0	81.3	39.3	88.2	43.6	54.3	52.3	61.4	42.3	48.2	60.5

Appendix

Table 6: Per-benchmark zero-shot accuracy across CLIP backbones. Top-1 accuracy (%); Mean is the unweighted average over 12 benchmarks. Blue/red cells indicate gains/losses relative to untrained CLIP within each backbone, scaled separately per column across backbones; bold marks the best result per column.

		In domain: 25 unseen breeds					Out of domain: 50 non-dog classes					Unseen-breed fidelity ( $\rho$ )
	Method	R@1	R@5	P@5	mAP	$\Delta$	R@1	R@5	P@5	mAP	$\Delta$	All	Within	Between
Target	Untrained CLIP	47.0	82.2	38.8	22.1	—	77.0	93.5	44.1	70.0	—	0.268	0.231	0.245
	Shuffled teacher	36.6	76.0	28.8	18.0	$-4.1$	73.5	93.0	43.1	67.7	$-2.3$	0.239	0.227	0.211
	SupCon	69.4	87.2	62.0	47.7	$+25.6$	78.5	92.0	43.9	71.3	$+1.3$	0.546	0.471	0.508
	KUEA	47.4	82.2	38.9	22.3	$+0.2$	78.0	93.5	44.0	70.1	$+0.1$	0.271	0.232	0.247
	DINOv2 $+$ KL	75.2	92.0	67.5	52.0	$+29.9$	81.0	95.0	47.1	74.8	$+4.8$	0.560	0.536	0.523

Appendix

Table 7: Retrieval and fidelity on unseen classes. CLIP ViT-B/16. Retrieval uses 500 unseen-breed and 200 non-dog images; scores are percentages and

\Delta

is the mAP gain over CLIP. Fidelity is Spearman

\rho

against Qwen3.8-27B on 250 unseen-breed images. All uses every pair; Within uses pairs of the same breed; Between uses pairs of different breeds. Blue/red indicates improvement/decline from CLIP, scaled per metric column; bold marks the best score within each block.

Training criterion / model	Colour/lighting	Pose/view	Background
Untrained CLIP	0.343	0.464	0.535
Breed	0.302	0.390	0.472
Colour/lighting	0.428	0.451	0.576
Pose/view	0.332	0.492	0.552
Coat texture	0.320	0.412	0.498
Background	0.387	0.413	0.795

Appendix

Table 8: Criterion transfer beyond dogs. Spearman fidelity to raw Qwen3.5-397B judgments on 200 non-dog images (50 classes; 19,900 pairs). Dog-trained encoders receive no non-dog adaptation, and each column reuses its dog scoring prompt unchanged. Each model keeps its training-fitted cubic link; CLIP uses the breed link. Blue/red shading denotes improvement/decline relative to CLIP, scaled within the table; bold marks column bests.

Training criterion / model	Car model	Colour/lighting	Pose/view	Background
Untrained CLIP	0.616	0.232	0.317	0.081
Car model	0.859	0.227	0.384	0.051
Colour/lighting	0.409	0.704	0.257	0.231
Pose/view	0.497	0.196	0.756	0.096
Background	0.209	0.303	0.154	0.813
DINOv2+KL	0.597	0.190	0.310	0.057

Appendix

Table 9: Criterion specificity on cars. Spearman fidelity to raw Qwen3.5-397B judgments on the same 250 held-out car images (25 adaptation classes; 31,125 pairs). Rows identify car-trained encoders and columns the evaluation criteria. Each model keeps its training-fitted cubic link; CLIP uses the car-model link. Blue/red shading denotes improvement/decline relative to CLIP, scaled within the table; bold marks column bests.

(a) Attribute retrieval
Model	Wing colour	Bill shape	Breast pattern
Untrained CLIP	34.55	49.98	52.31
DINOv2+KL	35.62	50.28	52.72
CLIP text subspace	35.42	38.94	45.97
CLIP text profile	34.44	38.25	45.91
VLM2Vec	34.87	50.09	52.10

Appendix

Table 10: Criterion retrieval and baseline checks on CUB-200-2011. Cosine nDCG@10 (

\times 100

) with cross-species galleries. (a) Model comparison; each ASK row is one fixed criterion-trained student. (b) Instruction settings and AligNet output spaces. Symmetric conditioning uses the same instruction for query and gallery; * marks validation-selected settings used in (a), where AligNet uses its image encoder. Blue/red denotes improvement/decline from CLIP in (a) or each block's first row in (b), scaled per column; bold marks column bests in (a) and block bests in (b).

Figure 4: Attribute retrieval on species excluded from adaptation. ASK and DINOv2+KL adapt on 100 CUB species and evaluate on the other 100, with cross-species galleries. Each bar reports cosine nDCG@10 for one fixed encoder; ASK uses the matching criterion. VLM2Vec and MagicLens retain the earlier CUB settings, selected on validation images spanning all 200 species; their prompt selection is not species-disjoint.

Figure 5: Criterion transfer evaluated with independent annotations. (a) Car viewpoint from PASCAL3D+ azimuth annotations. (b) Background composition from COCO-Stuff masks. (c) Non-dog body pose from AP-10K keypoints. Each bar reports cosine nDCG@10 for one fixed encoder, with no weight updates on these datasets. VLM2Vec and MagicLens use separate validation images to select instructions. ASK uses the car Pose/view, dog background, and dog Pose/view encoders, respectively; DINOv2+KL uses the corresponding adaptation domain.

	Method	IN-1k	C-10	C-100	Caltech	FER	Pets	DTD	RESISC	EuroSAT	PCam	IN-Sk.	IN-O	Mean
	Untrained CLIP	68.3	90.0	65.6	82.2	46.3	89.0	44.9	58.2	55.9	50.7	48.3	42.3	61.8
Dogs	SupCon	63.7	87.5	62.2	79.7	41.9	84.8	43.3	57.8	53.2	53.0	44.6	39.0	59.2
	KUEA	68.3	90.0	65.8	82.0	46.6	88.9	44.8	58.1	56.2	51.2	48.3	42.6	61.9
	Qwen2.5-VL tower	68.3	90.4	67.1	82.5	47.8	90.4	45.1	57.5	58.1	53.7	48.1	43.2	62.7
	DINOv2 $+$ KL	68.5	89.5	66.3	82.2	45.9	91.0	43.8	58.1	57.1	50.9	48.4	43.1	62.1
	ASK	67.7	89.2	66.6	82.7	44.9	90.3	43.9	60.4	59.0	51.6	47.8	43.1	62.3

Appendix

Table 11: Per-benchmark accuracy after domain adaptation. CLIP ViT-B/16; Mean averages the 12 benchmarks. Blue/red indicates improvement/decline from untrained CLIP, scaled per column across domains. Bold marks the highest adapted score within each domain when it exceeds CLIP. Same-domain evaluations are discussed in the text.

Judge	Agreement with judge						Self, swapped order			Own student
Judge	Q3.5-397B	GPT	Kimi	Sonnet	Q3.8-27B	G4-12B	$\rho_{\mathrm{S}}$	$\rho_{\mathrm{P}}$	$\|\Delta\|$	Fid	Ratio
Qwen3.5-397B	—	0.753	0.799	0.706	0.819	0.702	0.837	0.921	0.35	0.791	94%
GPT-5.6 Sol	0.753	—	0.811	0.725	0.770	0.679	0.901	0.926	0.32	0.794	88%
Kimi-K3	0.799	0.811	—	0.750	0.829	0.719	0.948	0.975	0.20	0.838	88%
Claude Sonnet 5	0.706	0.725	0.750	—	0.690	0.605	0.820	0.912	0.33	0.778	95%
Qwen3.8-27B	0.819	0.770	0.829	0.690	—	0.738	0.829	0.858	0.87	0.760	92%

Appendix

Table 12: Agreement and order consistency of VLM judges. Pairwise agreement is Spearman correlation. Blue intensity encodes absolute agreement on a shared scale; bold marks the highest cross-judge value per column. Self-agreement uses swapped image order; Ratio is student fidelity divided by self-Spearman agreement.

		Fidelity			Retrieval	Zero-shot
	Judge	$\rho_{\mathrm{S}}$	$\rho_{\mathrm{P}}$	Base $\rho_{\mathrm{S}}$	mAP	12 sets	IN-1k
ViT-B/32	Untrained CLIP	—	—	—	18.7	60.6	63.5
	Qwen3.5-397B	0.761	0.769	0.230	48.0	58.1	61.9
	GPT-5.6 Sol	0.745	0.733	0.274	47.0	57.2	61.6
	Kimi-K3	0.815	0.813	0.182	49.1	57.7	61.7
	Claude Sonnet 5	0.719	0.752	0.251	48.0	58.0	61.7

Appendix

Table 13: VLM judges across CLIP backbones. All students use 31 scoring groups. Spearman shading compares

\rho_{\mathrm{S}}

with Base

\rho_{\mathrm{S}}

in the same judge row. These judge-specific CLIP baselines replace a single value in the Untrained CLIP row. Retrieval and zero-shot shading compare with that backbone’s Untrained CLIP row. Blue/red denotes improvement/decline, scaled per metric; Pearson is unshaded. Bold marks column bests within each backbone.

	Judge	IN-1k	C-10	C-100	Caltech	FER	Pets	DTD	RESISC	EuroSAT	PCam	IN-Sk.	IN-O	Mean
ViT-B/32	Untrained CLIP	63.5	89.7	63.3	81.6	41.4	87.3	44.3	53.6	50.4	62.3	42.3	47.8	60.6
	Qwen3.5-397B	61.9	86.5	60.8	80.1	36.8	86.2	41.1	52.4	42.1	61.2	41.1	46.5	58.1
	GPT-5.6 Sol	61.6	85.6	58.6	79.9	32.6	85.6	41.7	52.6	40.9	59.4	41.0	46.7	57.2
	Kimi-K3	61.7	86.1	59.8	79.9	36.6	85.2	41.1	51.6	43.3	59.9	40.6	46.6	57.7
	Claude Sonnet 5	61.7	86.2	60.7	80.1	38.2	85.7	41.8	52.7	41.6	60.1	41.0	46.8	58.0
ViT-B/16	Untrained CLIP	68.3	90.0	65.6	82.2	46.3	89.0	44.9	58.2	55.9	50.7	48.3	42.3	61.8

Appendix

Table 14: Per-benchmark zero-shot accuracy across judges and backbones. Top-1 accuracy (%) for Table 13 ; Mean averages the 12 benchmarks. Blue/red indicates improvement/decline from each backbone's untrained CLIP, scaled per column across backbones. Bold marks the highest adapted-model score within each backbone when it exceeds CLIP.

Judge	Provider	$/M tokens input / output	P50 latency (s)	Output tokens/s	Est. cost ($)	Est. time (h)
Qwen3.5-397B	Alibaba Cloud Int.	0.39 / 2.34	1.11	61	5.4–12.3	1.4
GPT-5.6 Sol	OpenAI	2.00 / 10.00	3.58	49	27.6–63.1	4.5
Kimi-K3	Parasail	3.00 / 15.00	1.11	68	41.3–94.7	1.4
Claude Sonnet 5	Anthropic	2.00 / 10.00	2.30	63	27.6–63.1	2.9
Qwen3.8-27B	Parasail	0.24 / 2.20	1.35	58	3.4–7.7	1.7
Gemma 4 12B †	DeepInfra	0.05 / 0.15	0.75	14	0.7–1.6	1.1

Appendix

Table 15: OpenRouter teacher-scoring budget estimates. For 44,450 judgments: 300–700 input tokens per pair (including both images), two output tokens without reasoning, and ten concurrent requests. Names follow the experiments;

\dagger

marks Gemma 3 12B pricing used as a proxy for Gemma 4 12B.

Figure 6: Strong performance with fewer teacher API calls. (a) Distinct judgments, including kernel calibration. (b) Cosine retrieval on unseen dog breeds. (c) Zero-shot recognition across 12 benchmarks. Full budget is the default 93-group baseline; Sparse 15 uses 15 pairs per block.

Setting	Blocks	Pairs/block	Distinct judgments		Fidelity	Retrieval	Zero-shot
			Update pairs	Total pairs	$\rho_{\mathrm{S}}$	mAP	12 sets
Full budget (dense)	93	496	44,450	44,450	0.794	60.16	62.28
Dense, 62 groups	62	496	30,174	30,174	0.790	60.07	62.28
Dense, 31 groups	31	496	15,376	15,376	0.781	58.88	62.01
Sparse 15 (random pairs)	400	15	5,624	14,320	0.780	59.41	62.56
All cached pairs per block	400	$\approx$ 45	14,703	21,311	0.793	60.06	62.47

Appendix

Table 16: Query budgets and performance. Dog-trained ViT-B/16, 3,000 updates. Update pairs count distinct training judgments; total pairs also include the 9,920 calibration pairs, without double counting. Blue/red indicates improvement/decline from the full-budget baseline, scaled per metric; bold marks the best score per column.

Supervision	Fidelity		Retrieval mAP		ZS
Supervision	Spearman	Pearson	Unseen breeds	Non-dog	ZS
Untrained CLIP	0.281	0.280	22.06	70.01	61.81
Binary class labels	0.453	0.707	43.11	64.40	59.83
Grouped class labels	0.451	0.706	43.71	63.81	60.49
WordNet path similarity	0.444	0.707	52.01	69.41	60.70
Class-pair mean (VLM)	0.730	0.801	57.17	72.66	61.77

Appendix

Table 17: Class labels versus VLM judgments. CLIP ViT-B/16 on Dogs, with shared kernel MSE and KL anchoring. Fidelity reports Spearman/Pearson agreement with raw held-out judgments. Retrieval mAP and the 12-benchmark zero-shot mean (ZS) are percentages. Blue/red indicates improvement/decline relative to untrained CLIP, scaled per column; bold marks the best score.

Judge	Criterion	Neg.%	Fidelity		mAP		ZS
Judge	Criterion	Neg.%	PSD	raw	PSD	raw	PSD	raw
Gemma 4 12B	Dogs / breed	6.4	0.771	0.761	55.93	54.49	61.71	61.30
GPT-5.6 Sol	Dogs / breed	3.8	0.790	0.785	58.99	58.45	62.10	61.81
Qwen3.5-397B	Cars / pose-view	3.5	0.758	0.760	45.75	44.30	60.88	60.63
Claude Sonnet 5	Dogs / breed	2.6	0.777	0.776	58.57	58.40	62.07	61.84
Qwen3.5-397B	Dogs / breed	1.8	0.792	0.791	59.79	59.57	62.26	62.29

Appendix

Table 18: PSD projection across judges and criteria. Neg.% denotes negative spectral mass. Retrieval uses unseen breed labels for Dogs and unseen car-model labels for Cars. Raw-target cells show gains (blue) or losses (red) relative to PSD targets, scaled per metric; bold marks the higher score within each pair. Displayed ties are uncoloured.

Figure 7: Verdict extraction with Qwen3.5-397B. Expected-digit scores (green), rounded expectations (blue), and argmax digits (coral). (a) Fidelity uses the normalized kernel against expected-digit judgments. (b) Cosine-retrieval mAP uses unseen-breed labels. (c) Zero-shot accuracy averages 12 benchmarks. Labels give exact reported values; dashed lines mark expected-digit results. Each panel uses its own zoomed vertical scale; higher is better.

Regulariser	Weight	Fidelity		Retrieval	Zero-shot
		$\rho_{\mathrm{S}}$	$\rho_{\mathrm{P}}$	mAP	12 sets	IN-1k
Default: $\mathrm{KL}(P^{0}\\|P)$ , weight $0.2$ , text trainable ( ASK )		0.794	0.840	60.16	62.28	67.70
Untrained CLIP		0.281	0.280	22.06	61.81	68.29
L2 on features (KUEA’s anchor)	0.001	0.709	0.733	46.83	61.33	67.61
	0.01	0.391	0.398	26.72	61.95	68.60
	0.1	0.298	0.298	22.76	61.89	68.41

Appendix

Table 19: Regulariser families and weights. Dog-trained CLIP ViT-B/16. Here,

P^{0}

and

P

denote the frozen and adapted image–text prediction distributions, respectively. Blue/red indicates improvement/decline from the default KL anchor, scaled per metric across all rows. Bold marks the best score within each block.

(a) Fidelity to the criterion’s own teacher				(b) $\Delta$ mAP over untrained CLIP
Criterion	Linear	MLP	LoRA	Retrieval on	Linear	MLP	LoRA
Breed	0.622	0.734	0.794	Unseen breeds	$+$ 13.9	$+$ 22.8	$+$ 38.1
Colour/lighting	0.491	0.673	0.747	Non-dog	$+$ 1.3	$+$ 1.9	$+$ 4.9
Pose/view	0.564	0.674	0.748	Textures	$+$ 1.3	$+$ 2.2	$+$ 6.2
Coat texture	0.699	0.826	0.874	Flowers	$+$ 2.6	$+$ 3.2	$+$ 8.9
Background	0.730	0.887	0.919	Cars	$+$ 3.7	$+$ 5.5	$+$ 14.7

Appendix

Table 20: Frozen-feature heads versus LoRA. All models use KL anchoring. (a) Criterion-specific Spearman fidelity. (b) Retrieval gains of breed-trained models. Blue/red indicates improvement/decline relative to LoRA, scaled per panel; bold marks row bests.

Training similarity	Formula	Loss	Fidelity				mAP	ZS
			Listed similarity		Cosine
			$\rho_{\mathrm{S}}$	$\rho_{\mathrm{P}}$	$\rho_{\mathrm{S}}$	$\rho_{\mathrm{P}}$
Untrained CLIP	—	–	0.281	0.280	0.281	0.256	22.06	61.81
Dot product	$\gamma x_{i}^{\top}x_{j}+c$	MSE	0.696	0.677	0.688	0.672	40.50	60.63
Cosine	$\hat{x}_{i}^{\top}\hat{x}_{j}$	MSE	0.756	0.811	0.756	0.811	58.35	62.57
Normalized cubic ( ASK )	$\tilde{k}_{\gamma,c}(x_{i},x_{j})$	MSE	0.794	0.840	0.794	0.792	60.16	62.28

Appendix

Table 21: Training similarity and objective under matched LoRA adaptation. All adapted rows share PSD targets and the KL anchor. Fidelity compares the listed similarity and deployed cosine similarity with the same raw held-out judgments, without test-time fitting. Untrained CLIP uses the default cubic link in the listed-similarity columns. mAP is cosine retrieval on unseen breeds; ZS is the 12-benchmark mean (both %). Blue/red indicates improvement/decline relative to cubic MSE; bold marks column bests.

Factor	Variant	Fidelity		Retrieval	Zero-shot
		$\rho_{\mathrm{S}}$	$\rho_{\mathrm{P}}$	mAP	12 sets	IN-1k
Default recipe ( ASK )		0.794	0.840	60.16	62.28	67.70
Kernel loss default: Frobenius	Spectral norm	0.803	0.856	60.75	61.60	66.47
Kernel loss default: Frobenius	Top-4 eigenvalues	0.801	0.854	60.79	61.85	66.21
Pair budget default: 93 groups	31 groups (15,376 unique pairs)	0.781	0.822	58.88	62.01	67.71
Pair budget default: 93 groups	62 groups (30,174 unique pairs)	0.790	0.836	60.07	62.28	67.68

Appendix

Table 22: Sensitivity to training choices. One factor changes from the default at a time. Blue/red indicates improvement/decline from the default, scaled per metric; bold marks the best score per column.

Figure 8: Kernel alignment across VLM judges. Original CLIP, teacher target, aligned CLIP, and signed residual on 100 held-out dog images. Each row uses its judge’s saved link; residuals are aligned minus target.

Figure 9: Kernel geometry under alternative supervision. Top: target and encoder kernels on 100 held-out dog images. Bottom: signed residuals relative to the Qwen breed target. The VLM-aligned model uses the 93-group protocol.

Figure 10: Target and learned geometry for five criteria. Rows show the VLM target, criterion-aligned CLIP, and signed residual on the same 250 held-out dog images. Columns retain the same image ordering across criteria.

Figure 11: Dog retrieval under different criteria. Each row shows the query (teal border) and its top three neighbours. Green/red borders indicate matching/different breeds only in the class-retrieval rows.

Figure 12: Car retrieval under different criteria. Each row shows the query (teal border) and its top three neighbours. Green/red borders indicate matching/different car classes only in the class-retrieval rows.

Figure 13: Different models, different nearest neighbours. Each row shows the query (teal) and the model’s top two cosine retrievals. ASK rows identify the training judge, not direct retrieval by the VLM. Green/red borders indicate matching/different dataset class labels.

Figure 14: Same candidates, different notions of similarity. Scores are query–candidate cosine similarities. Outlined cells mark the highest score among the four displayed candidates in each row; colours identify criteria.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

LAS-CLIP: A Lightweight Adapter Steering Approach for CLIP's Visual Encoder
Oct 2, 2026Anh-Khoa Dinh-Duc, Duc-Tai Dinh, Tam V. Nguyen +1Referring Expression Comprehension VLM Adaptation
University of Science, Ho Chi Minh City, Vietnam · Viet Nam National University, Ho Chi Minh City, Vietnam · University of Dayton, USA
CLIMP: Contrastive Language-Image Mamba Pretraining
Jan 11, 2026Nimrod Shabtay, Itamar Zimerman, Eli Schwartz +1Vision-Language Pretraining VLM Robustness
Tel-Aviv University · IBM Research
MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations
Aug 26, 2026Jongsuk Kim, Qiyu Wu, Zhuoyuan Mao +3Contrastive Learning for VLMs Vision-Language Models
KAIST · Sony Group Corporation

Oct 2, 2026J/K move · Enter open · S save

LAS-CLIP: A Lightweight Adapter Steering Approach for CLIP's Visual Encoder

Anh-Khoa Dinh-Duc, Duc-Tai Dinh, Tam V. Nguyen, Minh-Triet Tran

University of Science, Ho Chi Minh City, Vietnam · Viet Nam National University, Ho Chi Minh City, Vietnam · University of Dayton, USA

Referring Expression Comprehension VLM Adaptation Zero-Shot Image Classification Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

CLIP's visual encoder produces only global image representations, limiting its use in region-level tasks. Existing adaptations rely on visual prompting, input masking, or encoder fine-tuning, each compromising pre-trained representations. We propose LAS-CLIP, a Lightweight Adapter Steering approach that keeps every CLIP parameter frozen. A compact MaskAdapter generates per-head, per-layer attention biases from an input mask and injects them into the frozen self-attention layers, steering attention toward the target region. Crucially, because the backbone remains strictly untouched, LAS-CLIP seamlessly reverts to vanilla CLIP when no mask is provided, preserving its foundational zero-shot capabilities. With approximately 116K to 145K trainable parameters and 100K training samples on two T4 GPUs, LAS-CLIP achieves competitive or superior results compared to Alpha-CLIP on ImageNet-S zero-shot classification and RefCOCO referring expression comprehension, despite the latter fine-tuning its entire encoder on millions of samples. Qualitative analysis further confirms stronger representational fidelity under incorrect masks and in downstream generation.

Abstract

VLM Adaptation

Explore similar work

LAS-CLIP: A Lightweight Adapter Steering Approach for CLIP's Visual Encoder

CLIMP: Contrastive Language-Image Mamba Pretraining

MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations