MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations

Jongsuk Kim; Qiyu Wu; Zhuoyuan Mao; Hiromi Wakaki; Junmo Kim; Yuki Mitsufuji

cs.CVAug 26, 2026

MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations

Authors: Jongsuk Kim, Qiyu Wu, Zhuoyuan Mao, Hiromi Wakaki, Junmo Kim, Yuki Mitsufuji

Organizations: KAIST · Sony Group Corporation

Abstract

Pretrained vision-language models such as CLIP excel at zero-shot recognition but often fail at compositionality, particularly attribute-object and relational structures. Recent studies mitigate this issue by augmenting training with synthetic hard negatives generated by a cascade of large language models and text-to-image models, which incurs substantial pipeline overhead. We instead propose MLLMCLIP, a heterogeneous distillation framework that transfers multimodal knowledge directly from a generative Multimodal Large Language Model (MLLM) teacher into a discriminative CLIP student, bypassing synthetic data entirely. To bridge the architectural mismatch between the two paradigms, we introduce an attention-based per-layer token selection and a CKA-based distillation loss. Compared to prior CLIP-enhancement methods, MLLMCLIP achieves state-of-the-art compositional accuracy while delivering consistent gains on standard zero-shot classification and image-text retrieval, showing that feature-level distillation strengthens both compositional and general vision-language representation capability.

Figures & tables

Panel of Figure 1 — Figure 1: Two distillation paradigms. ( 1(a) ) Prior methods synthesize hard-negative samples via cascaded LLM and image generators. ( 1(b) ) We distill hidden states from a frozen MLLM into CLIP directly.

Type	Model (Backbone)	First	Second	Avg.
MLLM	LLAVA-1.6-mistral-7B	95.6	74.3	85.0
	LLaMA-3.2-Vision-11B	92.1	92.5	92.3
	Qwen2-VL-2B	88.9	95.4	92.2
	Qwen3-VL-2B	94.0	95.0	94.5
	Qwen3.5-2B	88.5	96.0	92.3
	InternVL3.5-2B	94.2	89.4	91.8

Table 1: SugarCrepe accuracy of representative MLLM-as-judge and embedding-based teachers. Embedding models are position-invariant; full results are in Table F .

Figure 2: Overview of the MLLMCLIP framework. Student features are passed through auxiliary layers before alignment, and attention-based token selection obtains salient teacher tokens.

Method	External Source	ARO	CREPE	EQBEN	ImageCODE	SugarCrepe	SvoProbes	VALSE	VLChecklist	WhatsUp	Winoground	SPEC	Average
CLIP	–	35.6	11.6	14.8	15.0	59.7	75.8	55.1	64.5	41.3	7.00	28.5	37.1
LaCLIP	LLAMA-7B	35.2	9.98	14.4	14.8	64.0	76.9	56.6	64.8	41.5	4.75	28.3	37.5
NegCLIP	Qwen3-4B	36.2	11.7	14.8	15.8	62.4	76.3	56.1	64.0	40.9	7.00	28.2	37.6
FSC-CLIP	Qwen3-4B	35.1	9.86	16.3	14.9	63.8	78.8	57.4	64.4	41.8	5.50	29.2	38.0
TripletCLIP	Qwen3-4B, SDXL-Turbo	34.6	10.5	16.4	16.9	65.7	78.8	59.4	64.8	41.1	4.25	29.0	38.3
MLLMCLIP (Ours)	LLaVA-1.6-Mistral-7B	32.1	10.7	15.5	15.6	69.3	79.4	55.9	64.6	43.6	7.25	26.5	38.2

Table 2: Performance on 11 compositionality benchmarks.

Method	External Source	Caltech 101	CIFAR-10	CIFAR-100	DTD	EuroSAT	FER 2013	Flower 102	Food 101	ImageNet	KITTI	Pet	RESISC45	VOC 2007	Average
CLIP	–	37.8	53.3	21.7	7.93	12.1	12.4	8.88	8.91	12.4	20.0	9.95	15.9	54.0	21.2
LaCLIP	LLAMA-7B	41.7	49.7	21.2	11.2	14.3	19.5	11.2	9.34	13.7	30.7	10.1	17.1	60.9	23.9
NegCLIP	Qwen3-4B	41.4	55.3	24.1	10.6	18.3	18.9	9.16	9.49	13.7	30.4	8.81	17.9	56.8	24.2
FSC-CLIP	Qwen3-4B	41.5	54.2	25.1	9.84	18.0	16.8	9.03	9.40	13.5	20.8	10.1	20.3	63.1	24.0
TripletCLIP	Qwen3-4B, SDXL-Turbo	42.4	46.5	22.3	13.5	24.6	17.2	10.4	9.97	14.2	23.9	10.8	22.8	56.3	24.2
MLLMCLIP (Ours)	LLaVA-1.6-Mistral-7B	48.2	65.0	32.1	13.8	19.0	14.8	8.4	10.8	15.4	33.2	10.2	24.4	66.6	27.8

Table 3: Zero-shot classification performance on 13 classification datasets.

		Image–to–text retrieval						Text–to–image retrieval
		MSCOCO			Flickr-30K			MSCOCO			Flickr-30K
Method	External Source	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
CLIP	–	9.02	24.3	33.7	18.3	39.9	50.4	6.85	18.7	27.1	12.9	31.6	41.5
LaCLIP	LLAMA-7B	9.80	24.8	33.7	20.8	40.4	51.5	6.72	19.1	27.8	15.9	36.8	47.8
NegCLIP	Qwen3-4B	9.38	24.4	34.4	20.1	42.8	53.6	7.66	20.0	28.5	13.7	32.2	42.7
FSC-CLIP	Qwen3-4B	11.4	27.8	37.7	22.0	46.7	58.0	8.91	23.5	32.8	17.4	38.4	49.0

Table 4: Zero-shot retrieval performance on MSCOCO and Flickr-30K datasets.

Image Teacher	Text Teacher	Comp.	Zero-shot Cls.	Ret.
None	None	37.1	21.2	11.8
[Image End]	None	37.5	25.2	16.2
[Text End]	None	38.0	26.6	17.8
[Text End]	[Text End]	38.2	28.1	17.8
Attention-weighted pooling		40.2	31.1	18.9
Top- $k$ multi-token pooling ( $k=8$ )		39.7	30.8	18.2

Table 5: Effect of teacher token selection and per-layer aggregation. [Image End] and [Text End] denote the token positions at the end of the image segment and of the full multimodal sequence. The lower block varies how many attention-selected tokens are kept per layer.

Type	Loss	Comp.	Zero-shot Cls.	Ret.
Direct	MSE	30.2	7.00	0.05
	Cosine	38.6	27.5	17.2
	KL Div.	38.2	28.1	18.4
	JS Div.	37.4	25.8	16.0
Relational	CKA	40.0	31.3	19.1

Table 6: Effect of different distillation loss functions on downstream performance.

	Data	Feature
Text gen. (s)	$0.278\pm 0.001$	–
Image gen. (s)	$0.557\pm 0.012$	–
Feature ext. (s)	–	$0.232\pm 0.005$
Total (s)	$0.835\pm 0.012$	$\mathbf{0.232\pm 0.005}$

Table 7: Per-batch preprocessing time at batch size 32, comparing data synthesis against feature extraction. Mean

\pm

std over 5 batches on a single GPU.

Model	#Params (B)	Inf. time (ms)	SugarCrepe
VLM2Vec-v1	2.21	$114.60\pm 0.33$	64.6
VLM2Vec-v2	2.22	$115.92\pm 0.23$	72.7
Qwen3-VL-Emb.-2B	2.13	$52.36\pm 0.21$	83.0
MLLMCLIP	0.15	$\mathbf{4.57\pm 0.04}$	71.3

Table 8: Single-sample inference cost and SugarCrepe accuracy. Mean

\pm

std over 10 runs on a single GPU.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Supervision	Comp.	Zero-shot Cls.	Ret.
NegCLIP	Data	37.9	25.0	12.9
TripletCLIP	Data	38.5	25.6	15.7
MLLMCLIP	Feature	40.0	31.3	19.1

Appendix

Table A: Data-level versus feature-level distillation with the same MLLM (Qwen3.5-2B) supplying both forms of supervision. The baseline rows are controlled reruns, not the main-table rows; TripletCLIP still renders its negative images with SDXL-Turbo.

Teacher Prompt	Comp.	Zero-shot Cls.	Ret.
Plain caption ( {caption} only)	39.6	30.9	18.2
Compositional prompt (ours)	40.0	31.3	19.1

Appendix

Table B: Effect of the prompt used for teacher feature extraction. Both rows share the same Qwen3.5-2B teacher and training settings; only the extraction prompt differs.

Layer Selection	Comp.	Zero-shot Cls.	Ret.
Lower Block (1–12)	38.9	30.1	17.2
Middle Block (13–24)	38.2	28.6	16.3
Upper Block (25–36)	38.7	29.9	16.1
Strided (1,4,7…)	40.0	31.3	19.1

Appendix

Table C: Downstream performance with varying teacher layer selection strategies for distillation.

$\lambda$	Comp.	Zero-shot Cls.	Ret.
0.1	38.5	25.3	13.5
1	40.0	31.3	19.1
10	40.2	30.9	18.5
100	37.8	26.3	17.9

Appendix

Table D: Downstream performance with varying weight parameter during pre-training.

Model	Source
Qwen3-4B (hard-negative captions)	https://huggingface.co/Qwen/Qwen3-4B
SDXL-Turbo (hard-negative images)	https://huggingface.co/stabilityai/sdxl-turbo
LLaVA-1.6-Mistral-7B	https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf
LLaVA-1.6-Vicuna-7B	https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf
LLaMA-3.2-11B-Vision-Instruct	https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct
Qwen2-VL-2B-Instruct	https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct

Appendix

Table E: Public sources of the models used in our experiments.

MLLM-as-Judge models
Model	GT-position / Backbone	Replace			Swap		Add		Average
		Object	Attribute	Relation	Object	Attribute	Object	Attribute	Average
LLAVA-1.6-mistral-7B	First	99.3	98.0	95.8	89.4	96.2	97.2	93.1	95.6
	Second	93.8	82.6	67.6	69.4	77.6	79.3	50.0	74.3
	Average	96.5	90.3	81.7	79.4	86.9	88.2	71.5	85.0
LLAVA-1.6-vicuna-7B	First	98.1	96.1	89.6	80.8	89.9	93.9	85.5	90.6

Appendix

Table F: Comparison of MLLM-as-judge models and MLLM-based embedding models on the SugarCrepe benchmark. Judge models are queried twice per example with the ground-truth caption at position A ( First ) and position B ( Second ); embedding models score each caption by image–text similarity and report a single per-task accuracy.

Contrastive Learning for VLMs

See all 18 papers

CLIP-RD: Relational Distillation for Efficient CLIP Knowledge Distillation
Mar 26, 2026Jeannie Chung, Hanna Jang, Ingyeong Yang +2Cross-Modal Learning Cross-Modal Knowledge Distillation
Ewha Womans University, Seoul 03760, South Korea
CLIMP: Contrastive Language-Image Mamba Pretraining
Jan 11, 2026Nimrod Shabtay, Itamar Zimerman, Eli Schwartz +1Vision-Language Pretraining VLM Robustness
Tel-Aviv University · IBM Research
CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning
Sep 8, 2026SeongJun Jeong, Minjoon Jung, Woo Suk Choi +2VLM Robustness Contrastive Learning
Seoul National University

Mar 26, 2026J/K move · Enter open · S save

CLIP-RD: Relational Distillation for Efficient CLIP Knowledge Distillation

Jeannie Chung, Hanna Jang, Ingyeong Yang, Uiwon Hwang+1

Ewha Womans University, Seoul 03760, South Korea

Cross-Modal Learning Cross-Modal Knowledge Distillation Vision-Language Models VLM Distillation Relational Knowledge Distillation Knowledge Distillation

Open paper AskPDF arXiv

Contrastive Language-Image Pre-training (CLIP) demonstrates strong zero-shot generalization, but due to substantial computational and memory costs, distillation into lightweight models is required. Existing relational objectives do not explicitly model multidirectional relationships between teacher and student embeddings, potentially leaving the geometric relationships insufficiently constrained. This may disrupt the modality-gap structure important for zero-shot transfer. To address these limitations, we propose a relational distillation framework, CLIP-RD, which introduces two relational methods, Vertical Relational Distillation (VRD) and Cross Relational Distillation (XRD). VRD aligns teacher-student intra-modal similarity distributions to enforce consistent distillation strength across image and text embeddings. Meanwhile, XRD aligns the teacher-image-student-text and teacher-text-student-image similarity distributions to impose bidirectional cross-modal symmetry. By jointly modeling these multidirectional relational structures, CLIP-RD aligns the student's embedding geometry more faithfully to the teacher's, outperforming CLIP-KD by 1.8%p. This performance improvement is maintained across diverse architectures, teacher scales, retrieval tasks, downstream tasks, and corruption settings, with negligible additional training-time overhead.

Abstract

Contrastive Learning for VLMs

Explore similar work

CLIP-RD: Relational Distillation for Efficient CLIP Knowledge Distillation

CLIMP: Contrastive Language-Image Mamba Pretraining

CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning