Multimodal Dataset Distillation via Phased Teacher Models

Shengbin Guo; Hang Zhao; Senqiao Yang; Chenyang Jiang; Yuhang Cheng; Xiangru Peng; Rui Shao; Zhuotao Tian

cs.CVMar 26, 2026

Multimodal Dataset Distillation via Phased Teacher Models

Authors: Shengbin Guo, Hang Zhao, Senqiao Yang, Chenyang Jiang, Yuhang Cheng, Xiangru Peng, Rui Shao, Zhuotao Tian

Organizations: Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong

Abstract

Multimodal dataset distillation aims to construct compact synthetic datasets that enable efficient compression and knowledge transfer from large-scale image-text data. However, existing approaches often fail to capture the complex, dynamically evolving knowledge embedded in the later training stages of teacher models. This limitation leads to degraded student performance and compromises the quality of the distilled data. To address critical challenges such as pronounced cross-stage performance gaps and unstable teacher trajectories, we propose Phased Teacher Model with Shortcut Trajectory (PTM-ST) -- a novel phased distillation framework. PTM-ST leverages stage-aware teacher modeling and a shortcut-based trajectory construction strategy to accurately fit the teacher's learning dynamics across distinct training phases. This enhances both the stability and expressiveness of the distillation process. Through theoretical analysis and comprehensive experiments, we show that PTM-ST significantly mitigates optimization oscillations and inter-phase knowledge gaps, while also reducing storage overhead. Our method consistently surpasses state-of-the-art baselines on Flickr30k and COCO, achieving up to 13.5% absolute improvement and an average gain of 9.53% on Flickr30k. Code: https://github.com/Previsior/PTM-ST.

Figures & tables

Figure 1: PTM-ST has achieved far superior performance than other selection or distillation methods in different metrics on Flickr30k.

Figure 2: (a) shows that using a late-stage, high-performing teacher during distillation leads to a decline in student performance. (b) shows the directions of different matching range gradients (after PCA dimensionality reduction).

Figure 3: (a) shows the conventional single-stage training with a fixed teacher model and uniform data use. In contrast, (b) depicts our Phased Teacher Model (PTM), which employs different teacher models across multiple training stages to distill knowledge to specific data subsets. (c) illustrates the aggregation of all distilled subsets for final student evaluation. Additionally, (d) presents our Shortcut Trajectory (ST) strategy that dynamically generates stage-adaptive teacher models, improving distillation effectiveness and robustness.

Figure 4: Gradient cosine similarity on the synthetic dataset across different epochs for the original and shortcut trajectories.

Pairs	Ratio	Metric	Core-set Selection				Dataset Distillation
Pairs	Ratio	Metric	RAND	HERD	K-CENT	FORGET	MTT-VL 2 2 2 Intercepted from the original paper.	LoRS 3 3 3 Reproduced by ourselves.	EDGE 2 2 2 Intercepted from the original paper.	PTM-ST	$\triangle$
100	0.3%	IR@1	1.0	0.7	0.7	0.7	4.7 $\pm$ 0.2	7.8 $\pm$ 0.2	-	9.6 $\pm$ 0.3	+ 1.8
		IR@5	4.0	2.8	3.1	2.4	15.7 $\pm$ 0.5	24.4 $\pm$ 0.3	-	28.4 $\pm$ 0.5	+ 4.0
		IR@10	6.5	5.3	6.1	5.6	24.6 $\pm$ 1.0	35.5 $\pm$ 0.4	-	41.5 $\pm$ 0.5	+ 6.0
		TR@1	1.3	1.1	0.6	1.2	9.9 $\pm$ 0.3	10.2 $\pm$ 0.3	-	14.4 $\pm$ 0.4	+ 4.2
		TR@5	5.9	4.7	4.2	4.2	28.3 $\pm$ 0.5	30.9 $\pm$ 0.8	-	38.6 $\pm$ 0.8	+ 7.7

Table 1: Results on Flickr30k. The metrics for training model on the full dataset are IR@1=27.3, IR@5=57.1, IR@10=69.7; TR@1=33.9, TR@5=65.1, TR@10=75.2.

Pairs	Ratio	Metric	Core-set Selection				Dataset Distillation
Pairs	Ratio	Metric	RAND	HERD	K-CENT	FORGET	MTT-VL 2 2 2 Intercepted from the original paper.	LoRS 3 3 3 Reproduced by ourselves.	EDGE 2 2 2 Intercepted from the original paper.	PTM-ST	$\triangle$
100	0.8‰	IR@1	0.3	0.5	0.4	0.3	1.3 $\pm$ 0.1	1.7 $\pm$ 0.1	-	2.3 $\pm$ 0.1	+ 0.6
		IR@5	1.3	1.4	1.4	1.5	5.4 $\pm$ 0.3	6.9 $\pm$ 0.2	-	9.0 $\pm$ 0.1	+ 2.1
		IR@10	2.7	3.5	2.5	2.5	9.5 $\pm$ 0.5	11.9 $\pm$ 0.3	-	15.6 $\pm$ 0.3	+ 3.7
		TR@1	0.8	0.8	1.4	0.7	2.5 $\pm$ 0.3	3.0 $\pm$ 0.3	-	4.1 $\pm$ 0.2	+ 1.1
		TR@5	3.0	2.1	3.7	2.6	10.0 $\pm$ 0.5	11.0 $\pm$ 0.2	-	13.4 $\pm$ 0.2	+ 2.4

Table 2: Results on the COCO dataset. The metrics for training model on the full dataset are IR@1=16.9, IR@5=41.9, IR@10=55.9; TR@1=19.6, TR@5=45.6, TR@10=59.5.

Pairs	LoRS						Ours
Pairs	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
100	1.2	4.6	7.7	1.7	6.9	11.4	2.3	8.2	13.2	2.9	10.0	15.9
200	1.4	5.3	8.7	2.4	8.5	13.6	2.7	9.7	15.8	3.7	11.9	18.4
500	1.7	6.2	10.1	2.5	8.7	13.8	3.3	11.4	17.9	4.1	13.2	19.9

Table 3: Results on LLaVA-cc3m dataset. The metrics for training model on the full dataset are IR@1=9.3, IR@5=25.9, IR@10=36.5; TR@1=9.8, TR@5=26.4, TR@10=37.3.

No.	Model	Flickr IR@K			Flickr TR@K			COCO IR@K			COCO TR@K
No.	Model	1	5	10	1	5	10	1	5	10	1	5	10
(1)	BASE (N/A)	12.2	33.0	45.7	16.2	39.4	54.0	3.4	11.7	19.1	4.2	13.8	20.8
(2)	EMA	12.9	33.7	46.3	16.2	40.6	54.3	3.5	12.3	19.8	4.0	13.6	20.8
(3)	PTM	13.4	35.2	48.1	19.6	43.2	55.5	4.2	14.2	22.6	4.8	15.1	23.4
(4)	ST	14.2	37.8	50.8	19.5	45.1	59.3	4.9	15.9	24.7	4.8	15.1	23.8
(5)	PTM + EMA	14.3	37.7	50.7	18.8	45.0	58.5	4.5	14.8	23.1	4.8	15.7	24.3

Table 4: Various ablation studies with 500 pairs on Flickr30k and COCO.

Methods	VQA on COCO-QA			ImageNet-50 Classification
Methods	ACC@1	ACC@5	ACC@10	ACC@1	ACC@5	ACC@10
LoRS	10.8	33.0	39.9	18.6	42.1	55.9
Ours	16.3	38.9	47.4	22.1	52.7	67.9

Table 5: Performance on additional downstream tasks.

Dataset	Method	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
Flickr30k	LoRS	9.8	29.0	41.6	14.9	39.8	53.5
	EDGE	9.9	28.2	40.5	14.5	38.3	51.7
	Ours	17.5	42.8	56.3	23.8	53.6	67.2
COCO	LoRS	2.5	9.9	16.6	4.2	15.2	24.1
	EDGE	2.8	9.8	16.2	3.9	13.0	21.0
	Ours	7.0	21.8	32.3	6.8	20.9	30.8

Table 6: Performance comparison on Flickr30k and COCO with 1000 pairs.

Figure 5: Examples of initial (left) and synthetic (right) image-text pairs.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Model	2	4	6	8	10
COCO	Teacher	35.4	39.0	40.4	41.9	42.8
COCO	Student	11.4	7.9	6.9	5.7	5.1
Flickr30K	Teacher	45.6	48.9	51.2	53.6	55.3
Flickr30K	Student	33.8	35.1	33.5	31.4	30.1

Appendix

Table 7: Performance of teacher and student models at different epochs on COCO and Flickr30K.

Pairs	Ratio	Metric	Dataset Distillation
Pairs	Ratio	Metric	MTT-VL 1 1 1 Intercepted from the original paper.	LoRS 2 2 2 Reproduced by ourselves.	DATM 2 2 2 Reproduced by ourselves.	PTM-ST
100	0.8‰	IR@1	1.3 $\pm$ 0.1	1.7 $\pm$ 0.1	1.2 $\pm$ 0.1	2.3 $\pm$ 0.1
		IR@5	5.4 $\pm$ 0.3	6.9 $\pm$ 0.2	5.1 $\pm$ 0.2	9.0 $\pm$ 0.1
		IR@10	9.5 $\pm$ 0.5	11.9 $\pm$ 0.3	8.9 $\pm$ 0.4	15.6 $\pm$ 0.3
		TR@1	2.5 $\pm$ 0.3	3.0 $\pm$ 0.3	2.5 $\pm$ 0.3	4.1 $\pm$ 0.2
		TR@5	10.0 $\pm$ 0.5	11.0 $\pm$ 0.2	9.1 $\pm$ 0.3	13.4 $\pm$ 0.2

Appendix

Table 8: Results on the COCO dataset. The metrics for training the model on the full dataset are IR@1=16.9, IR@5=41.9, IR@10=55.9; TR@1=19.6, TR@5=45.6, TR@10=59.5.

Figure 6: (a) shows the gradients of the teacher model on the synthetic dataset across different epochs. (b) illustrates the parameter differences between various training trajectories of the teacher model. (c) present a visualization of the new teacher model constructed using our proposed method.

Parameter	Flickr30k	COCO
syn_steps	8	8
expert_epochs	1	1
lr_img	1000	1000
lr_txt	1000	1000
lr_lr	1e-2	1e-2
lr_teacher_img	0.1	0.1

Appendix

Table 9: Hyper-parameters for the distillation run.

Table 16

	Flickr-30K				MS-COCO
	100	200	500	1000	100	200	500	1000
subset_num / $P$	1	2	2	3	1	2	2	3
syn_steps / $t$	8	8	8	8	8	8	8	8
expert_epochs / $\Delta T$	1	1	1	1	1	1	1	1
min_start_epoch / $T^{-}_{p}$	0	0, 1	0, 1	0, 1, 2	0	0, 1	2	0, 1, 2
max_start_epoch / $T^{+}_{p}$	2	2, 3	2, 3	3, 4, 5	2	2, 3	2	2, 3, 4

Appendix

Table 12: Hyper-parameter settings for distillation.

Pairs	Setting	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10	Mean
200	PTM w/o shift	11.8	33.2	46.3	18.1	44.3	59.0	35.5
	ST w/o shift	11.6	33.3	47.0	18.5	43.5	57.8	35.3
	full	12.5	34.7	48.5	19.3	45.9	58.9	36.6
500	PTM w/o shift	15.2	39.6	53.1	21.6	48.6	62.9	40.2
	ST w/o shift	15.3	39.9	53.5	21.1	49.6	63.1	40.4
	full	16.0	40.5	54.0	22.2	51.1	64.6	41.4

Appendix

Table 13: Ablation on whether to shift under different data sizes.

Pairs	Interpolation endpoint	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10	Mean
100	6	9.6	28.4	41.5	14.4	38.6	52.7	30.9
	8	8.4	26.6	39.7	15.4	39.9	54.6	30.8
	10	8.8	26.8	39.7	14.9	39.6	53.6	30.6
200	4, 6	11.7	33.2	46.3	18.5	43.5	58.6	35.3
	6, 8	12.5	34.7	48.5	19.3	45.9	58.9	36.6
	8, 10	11.2	32.3	45.4	18.1	46.4	60.2	35.6

Appendix

Table 14: Comparison of distillation results at different interpolation endpoints.

Method	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10	Mean
$\Delta d=0$	15.2	39.6	53.1	21.6	48.6	62.9	40.2
$\Delta d=1$	16.0	40.5	54.0	22.2	51.1	64.6	41.4
$\Delta d=2$	15.9	40.1	53.2	21.0	50.1	63.7	40.7
LoRS	12.7	32.9	44.9	14.7	37.6	51.1	32.3

Appendix

Table 15: Effect of

\Delta d

on retrieval performance.

Pairs	LoRS						Ours
Pairs	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
100	5.8	20.6	30.7	10.2	29.4	39.8	8.1	23.5	34.9	13.5	33.8	46.4
200	6.1	20.9	32.1	12.0	31.4	43.1	8.4	24.1	35.4	14.1	35.6	48.4
500	7.7	22.9	33.3	12.7	33.5	46.1	9.1	25.9	37.2	15.0	36.5	50.1

Appendix

Table 16: Results on DiNo-v2 + BGE-1.5 on MS-COCO. The model trained on the full dataset performs: IR@1=22.5, IR@5=50.8, IR@10=65.0, TR@1=31.7, TR@5=61.4, TR@10=74.0.

Method	Model	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10	Mean
LoRS	NFNet + BERT	12.1	33.2	45.9	16.6	42.2	55.7	34.3
	NFNet + DistilBERT	7.6	24.5	34.7	12.7	34.5	46.1	26.7
	RegNet + BERT	4.5	15.0	23.7	5.4	21.0	32.4	17.0
Ours	NFNet + BERT	14.6	39.1	52.5	21.5	50.6	64.0	40.4
	NFNet + DistilBERT	11.3	29.8	41.3	19.0	41.3	56.5	33.2
	RegNet + BERT	4.2	14.4	22.3	8.7	23.8	36.0	18.2

Appendix

Table 17: Cross architecture generalization.

Method	time (s/iter)			peak memory (GB)			avg memory (GB)
Method	100	200	500	100	200	500	100	200	500
LoRS	4.522	4.913	7.392	13.713	16.333	20.942	5.339	5.586	7.985
Ours	4.570	4.554	7.250	13.768	13.818	20.363	5.390	5.404	7.592

Appendix

Table 18: Runtime and memory comparison across different numbers of pairs.

Clusters	ipc	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10	Mean
100	1	0.7	3.2	5.7	2.1	7.1	10.5	4.9
100	2	1.4	5.5	9.6	2.9	9.3	14.5	7.2
100	5	3.0	10.7	17.9	4.8	15.8	25.8	13.0
200	1	1.4	5.5	9.6	2.9	9.3	14.5	7.2
200	2	2.5	9.6	15.6	5.6	15.9	23.8	12.2
500	1	3.2	11.7	18.9	6.1	18.1	26.0	14.0

Appendix

Table 19: Results of the NCFM method on Flickr30K.

Figure 7: Flickr-30K before and after distillation. (Left) The original image-text pairs before the distillation. (Right) The image-text pairs after distillation.

Figure 8: COCO before and after distillation. (Left) The original image-text pairs before the distillation. (Right) The image-text pairs after distillation.

Dataset Distillation

Dataset distillation compresses training data into a small synthetic or selected set that retains the information needed to train useful models.

See all 66 papers

Multimodal Distribution Matching for Vision-Language Dataset Distillation
May 22, 2026Jongoh Jeong, Hoyong Kwon, Minseok Kim +1Cross-Modal Learning Cross-Modal Alignment
Visual Intelligence Lab., KAIST
Multimodal Dataset Distillation Made Simple by Prototype-Guided Data Synthesis
Feb 23, 2026Junhyeok Choi, Sangwoo Mo, Minwoo ChaeDataset Distillation Multimodal Dataset
Department of Industrial and Management Engineering, Pohang University of Science and Technology
Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm
Jun 9, 2026Trong Khiem Tran, Anh Duc Chu, Quang Hung Pham +2Cross-Modal Knowledge Distillation Cross-Modal Alignment
School of Electrical Engineering and Computer Science, Washington State University, Pullman, US · School of Information and Communications Technology, Hanoi University of Science and Technology, Hanoi, Vietnam

May 22, 2026J/K move · Enter open · S save

Multimodal Distribution Matching for Vision-Language Dataset Distillation

Jongoh Jeong, Hoyong Kwon, Minseok Kim, Kuk-Jin Yoon

Visual Intelligence Lab., KAIST

Cross-Modal Learning Cross-Modal Alignment VLM Distillation Distribution Matching Dataset Distillation

Open paper AskPDF arXiv

Dataset distillation compresses large training sets into compact synthetic datasets while preserving downstream performance. As modern systems increasingly operate on paired vision-language inputs, multimodal distillation must preserve representation quality and cross-modal alignment under tight compute and memory budgets, yet prior methods often require heavy computes and overlook their correlations. To address this, we present Multimodal Distribution Matching (MDM), a geometry-aware framework for efficient and generalizable multimodal distillation. Specifically, MDM integrates complementary components at the data, model, and loss levels. At the data level, it initializes synthetic image-text pairs by sampling from clusters in the joint embedding space. At the model level, it forms a mixed teacher by interpolating independently fine-tuned models in weight space according to their angular deviation from the pretrained anchor. At the loss level, it matches joint distributions on the unit hypersphere using a geometry-aware matching objective that exploits the joint features in the cross-modal agreement and discrepancy directions along with symmetric contrastive learning. Across image-text retrieval benchmarks with cross-architecture evaluation, MDM yields compact synthetic sets that preserve multimodal semantics, substantially reduce distillation cost, and remain robust across architectures.

Abstract

Dataset Distillation

Explore similar work

Multimodal Distribution Matching for Vision-Language Dataset Distillation

Multimodal Dataset Distillation Made Simple by Prototype-Guided Data Synthesis

Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm