Multimodal Dataset Distillation Made Simple by Prototype-Guided Data Synthesis

Junhyeok Choi; Sangwoo Mo; Minwoo Chae

cs.CVFeb 23, 2026

Multimodal Dataset Distillation Made Simple by Prototype-Guided Data Synthesis

Authors: Junhyeok Choi, Sangwoo Mo, Minwoo Chae

Organizations: Department of Industrial and Management Engineering, Pohang University of Science and Technology

Abstract

Recent advances in multimodal learning have achieved remarkable success across diverse vision-language tasks. However, such progress heavily relies on large-scale image-text datasets, making training costly and inefficient. Prior efforts in dataset filtering and pruning attempt to mitigate this issue, but still require relatively large subsets to maintain performance and fail under very small subsets. Dataset distillation offers a promising alternative, yet existing multimodal dataset distillation methods require full-dataset training and joint optimization of image pixels and text features, making them architecture-dependent and limiting cross-architecture generalization. To overcome this, we propose a learning-free dataset distillation framework that eliminates the need for large-scale training and optimization while enhancing generalization across architectures. Our method uses CLIP to extract aligned image-text embeddings, obtains prototypes, and employs an unCLIP decoder to synthesize images, enabling efficient and scalable multimodal dataset distillation. Extensive experiments demonstrate that our approach consistently outperforms optimization-based dataset distillation and subset selection methods, achieving state-of-the-art cross-architecture generalization.

Figures & tables

Figure 1: Concept. Optimization-based methods are computationally expensive and architecture-dependent. Our learning-free framework is simple, efficient, and generalizable across architectures.

Figure 2: Our PDS framework distills multimodal datasets by synthesizing samples from image-text prototypes through three stages: (i) modality-specific clustering of CLIP embeddings, (ii) cross-modal cluster matching via a linear assignment to obtain prototypes, and (iii) image synthesis using an unCLIP decoder guided by image prototype and caption embeddings.

Evaluation Model	Pairs	Methods	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	100	TESLA-VL	4.1 ± 0.3	14.7 ± 0.9	22.9 ± 1.2	6.5 ± 0.4	17.8 ± 1.4	27.3 ± 1.4
		LoRS	6.3 ± 0.1	18.6 ± 0.1	28.0 ± 0.2	9.1 ± 0.2	24.3 ± 0.4	34.5 ± 0.8
		PDS	7.9 ± 0.3	25.8 ± 0.4	37.3 ± 0.3	10.2 ± 0.3	28.2 ± 0.9	39.0 ± 0.3
	300	TESLA-VL	10.3 ± 0.3	28.8 ± 0.6	40.6 ± 0.8	14.9 ± 0.9	36.2 ± 1.1	48.8 ± 1.7
		LoRS	8.6 ± 0.9	23.5 ± 1.5	33.5 ± 1.8	14.7 ± 1.2	33.5 ± 2.8	44.1 ± 3.2
		PDS	14.4 ± 0.4	38.1 ± 0.2	51.4 ± 0.4	18.7 ± 0.5	45.0 ± 0.4	57.8 ± 0.6

Table 1: PDS distills datasets with superior cross-architecture generalization compared to multimodal dataset distillation baselines. We evaluate on unseen vision backbones, comparing against TESLA-VL and LoRS on Flickr30K (top) and MS-COCO (bottom). Across all distilled dataset sizes and evaluation backbones, PDS consistently outperforms both baselines, demonstrating that its distilled datasets generalize effectively to unseen backbones.

Evaluation Model	Metric	Subset selection					Distillation
Evaluation Model	Metric	K-center	Herding	CLIP score	LAION filtering	Image- based	PDS
ResNet	IR@1	2.9 ± 0.1	3.6 ± 0.2	2.5 ± 0.1	2.4 ± 0.1	2.2 ± 0.2	7.9 ± 0.3
	IR@5	10.4 ± 0.3	12.6 ± 0.5	8.7 ± 0.3	8.5 ± 0.2	8.6 ± 0.3	25.8 ± 0.4
	IR@10	16.8 ± 0.3	20.1 ± 0.6	14.5 ± 0.3	14.5 ± 0.3	13.6 ± 0.4	37.3 ± 0.3
	TR@1	5.3 ± 0.3	6.7 ± 0.3	4.7 ± 0.4	4.7 ± 0.3	4.0 ± 0.3	10.2 ± 0.3
	TR@5	15.6 ± 0.5	19.4 ± 0.7	12.6 ± 0.7	12.4 ± 0.6	10.5 ± 0.4	28.2 ± 0.9

Table 2: The representative samples obtained by interpolating diverse semantic features are more suitable than subsets of the original dataset when the reduced dataset is extremely small. We compare PDS with subset selection methods in the 100-pair setting on Flickr30K. Baselines restricted to real samples are limited in preserving broad semantic diversity. In contrast, PDS synthesizes representative samples that reflect diverse semantics and consistently achieves superior results.

Evaluation Model	Methods	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	D 4 M	1.3 ± 0.1	5.8 ± 0.2	9.8 ± 0.3	1.4 ± 0.2	5.6 ± 0.5	9.5 ± 0.3
	MGD 3	2.6 ± 0.1	10.0 ± 0.2	17.2 ± 0.5	3.4 ± 0.2	11.6 ± 0.3	17.5 ± 0.4
	PDS	7.9 ± 0.3	25.8 ± 0.4	37.3 ± 0.3	10.2 ± 0.3	28.2 ± 0.9	39.0 ± 0.3

Table 3: Cross-modal alignment is crucial for effective multimodal dataset distillation. We compare PDS with learning-free methods for image classification in the 100-pair setting on Flickr30K. Both

\text{D}^{4}\text{M}

and

\text{MGD}^{3}

rely on the VAE encoder, which produces image features that are not aligned with text features, leading to low retrieval accuracy. In contrast, PDS uses CLIP encoders to enforce cross-modal alignment, outperforming the baselines and highlighting the importance of alignment-aware distillation.

Figure 3: The synthesized images. Left (col. 1-3): Images from multimodal dataset distillation baselines, which are nearly identical to the initialization image. Middle: Image generated via CLIP inversion from image prototypes, which is not realistic. Right (col. 5-7): Given a text prototype, we first retrieve the most similar caption and present three images: the paired real image, the UnCLIP-generated image from this caption, and the PDS-generated image. Unlike UnCLIP, which strictly follows captions, PDS produces realistic and semantically enriched images by conditioning on image prototypes.

Evaluation Model	Methods	IR@1	TR@1	Memory (GB)	Time (s)
ResNet	Text alignment	1.4 ± 0.1	2.0 ± 0.5	6.13	1477.71
	Image alignment	4.4 ± 0.2	4.2 ± 0.3	6.13	1477.71
	PDS	7.9 ± 0.3	10.2 ± 0.3	4.34	9.71

Table 4: Using a generative model for image synthesis improves both efficiency and performance. We evaluate two variants of CLIP inversion: (i) Image prototype alignment and (ii) Text prototype alignment, both of which directly optimize pixel values, in the 100-pair setting on Flickr30K. By employing a generative model, PDS accelerates image generation, reduces memory usage, and produces realistic images, achieving superior performance over CLIP inversion.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

\displaystyle\min_{P\in\{0,1\}^{M\times M}}\sum_{i=1}^{M}\sum_{j=1}^{M}K_{ij}P_{ij}\,\,\text{subject to }\,\,\sum_{j=1}^{M}P_{ij}=1,\;\sum_{i=1}^{M}P_{ij}=1,\;P_{ij}\in\{0,1\},

Appendix

Algorithm 1 PDS framework

Evaluation Model	Methods	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	GMM	8.3±0.1	26.2±0.3	37.4±0.2	10.9±0.4	30.1±0.4	41.1±0.6
	Agglomerative	9.3±0.2	28.9±0.3	41.4±0.1	11.6±0.6	30.2±0.3	41.6±0.6
	Mini-batch k-means	7.9±0.3	25.8±0.4	37.3±0.3	10.2±0.3	28.2±0.9	39.0±0.3
ViT	GMM	6.0±0.4	19.2±0.3	28.3±0.1	6.1±0.4	18.2±0.9	26.9±0.7
	Agglomerative	6.3±0.2	19.9±0.2	29.5±0.4	6.4±0.5	18.5±0.3	27.4±0.4
	Mini-batch k-means	6.8±0.3	19.2±0.3	28.5±0.4	6.6±0.5	17.5±0.5	26.9±0.5

Appendix

Table 6: Comparison of clustering methods. We evaluate PDS in the 100-pair setting on Flickr30K using different clustering algorithms. PDS achieves robust performance, indicating low sensitivity to the choice of clustering algorithm.

	GMM	Agglomerative	Mini-batch k-means
Clustering Time (s)	1,469.7	4,387.9	9.0

Appendix

Table 7: Clustering time comparison. Clustering time (in seconds) across different methods.

Figure 4: Histograms of cosine similarity between paired image and text prototypes in the 100-pair setting on Flickr30K, shown with and without filtering. Filtering leads to higher similarity.

Evaluation Model	Methods	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	Non-filtering	5.3 ± 0.1	18.4 ± 0.1	28.1 ± 0.3	7.4 ± 0.5	21.2 ± 0.7	30.6 ± 0.5
ResNet	Filtering	7.9 ± 0.3	25.8 ± 0.4	37.3 ± 0.3	10.2 ± 0.3	28.2 ± 0.9	39.0 ± 0.3
ViT	Non-filtering	3.9 ± 0.0	14.0 ± 0.0	21.3 ± 0.0	4.5 ± 0.0	12.4 ± 0.0	19.6 ± 0.0
ViT	Filtering	6.8 ± 0.3	19.2 ± 0.3	28.5 ± 0.4	6.6 ± 0.5	17.5 ± 0.5	26.9 ± 0.5

Appendix

Table 8: Filtering enhances cross-modal alignment. We compare our filtering strategy against a non-filtering baseline in the 100-pair setting on Flickr30K. Prototypes obtained from embeddings of the shared image–text pairs consistently improve performance.

	Pairs
	100	300	500	1000	1500
Number of pairless clusters	1	8	24	87	166

Appendix

Table 9: Larger distilled sizes yield more pairless clusters. We distill Flickr30K into various numbers of pairs. As the distilled size increases, the number of pairless clusters grows substantially.

Figure 5: Histograms of cosine similarity between paired image and text prototypes in the 1000-pair setting on Flickr30K, shown for pairless clusters and clusters with shared pairs. Pairless clusters exhibit lower similarity.

Evaluation Model	Pairs	Methods	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	100	Keeping	7.9 ± 0.3	25.8 ± 0.4	37.3 ± 0.3	10.2 ± 0.3	28.2 ± 0.9	39.0 ± 0.3
	100	Discarding	7.8 ± 0.2	25.9 ± 0.4	37.5 ± 0.2	10.0 ± 0.3	27.5 ± 1.2	38.8 ± 0.9
	300	Keeping	14.4 ± 0.4	38.1 ± 0.2	51.4 ± 0.4	18.7 ± 0.5	45.0 ± 0.4	57.8 ± 0.6
	300	Discarding	14.9 ± 0.2	38.6 ± 0.5	51.9 ± 0.3	19.3 ± 0.4	45.0 ± 0.5	57.2 ± 0.6
	500	Keeping	16.3 ± 0.2	41.3 ± 0.2	54.9 ± 0.4	22.5 ± 0.3	49.3 ± 0.8	61.4 ± 0.4
	500	Discarding	17.1 ± 0.2	43.0 ± 0.2	56.4 ± 0.1	23.9 ± 0.9	50.5 ± 0.6	62.0 ± 0.3

Appendix

Table 10: Discarding pairless clusters becomes increasingly beneficial at larger distilled sizes. We distill Flickr30K to a wide range of distilled dataset sizes and compare two strategies, keeping all clusters or discarding the pairless clusters. While both strategies perform similarly in small-scale settings, discarding pairless clusters becomes preferable when the distilled size becomes large.

Evaluation Model	Methods	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	Image prototype	6.0 ± 0.3	19.2 ± 0.3	28.3 ± 0.1	6.1 ± 0.4	18.2 ± 0.9	26.9 ± 0.7
ResNet	Both prototypes	7.9 ± 0.3	25.8 ± 0.4	37.3 ± 0.3	10.2 ± 0.3	28.2 ± 0.9	39.0 ± 0.3
ViT	Image prototype	5.4 ± 0.1	17.5 ± 0.2	26.5 ± 0.5	5.4 ± 0.3	17.8 ± 0.5	25.7 ± 1.1
ViT	Both prototypes	6.8 ± 0.3	19.2 ± 0.3	28.5 ± 0.4	6.6 ± 0.5	17.5 ± 0.5	26.9 ± 0.5

Appendix

Table 11: Joint conditioning improves performance. We compare our method with a baseline that generates images conditioned solely on the image prototype in the 100-pair setting on Flickr30K. Conditioning on both the image prototype and a caption retrieved from the text prototype improves performance.

Method	Evaluation Model	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
TESLA-VL	ResNet+BERT	0.3 ± 0.1	1.9 ± 0.2	3.4 ± 0.2	2.0 ± 0.4	7.9 ± 0.5	12.2 ± 0.5
	ResNet+CLIP	4.1 ± 0.3	14.7 ± 0.9	22.9 ± 1.2	6.5 ± 0.4	17.8 ± 1.4	27.3 ± 1.4
	ViT+BERT	0.2 ± 0.1	1.0 ± 0.2	1.9 ± 0.5	0.5 ± 0.2	2.0 ± 0.6	3.9 ± 1.4
	ViT+CLIP	2.1 ± 0.3	7.8 ± 0.7	13.1 ± 1.2	2.6 ± 0.6	8.7 ± 0.9	13.7 ± 1.4
LoRS	ResNet+BERT	2.0 ± 0.1	7.3 ± 0.3	11.8 ± 0.2	3.7 ± 0.4	13.4 ± 0.5	19.8 ± 0.8
	ResNet+CLIP	6.3 ± 0.1	18.6 ± 0.1	28.0 ± 0.2	9.1 ± 0.2	24.3 ± 0.4	34.5 ± 0.8

Appendix

Table 12: Comparison of BERT and CLIP as text encoders in the 100-pair setting on Flickr30K. Within each method, we compare BERT and CLIP as text encoders under the same vision backbone (ResNet or ViT). In all cases, replacing BERT with CLIP consistently improves performance.

Figure 6: Ablation study on hyperparameters. The first three panels show the results obtained by varying the guidance scale, the number of sampling steps, and the pruning threshold ratio, respectively. Each panel reports the aggregated retrieval accuracy, computed as the average of IR@1, IR@5, IR@10, TR@1, TR@5, and TR@10, using distilled sets of 100 and 300 pairs on Flickr30K. The final panel presents a boxplot of retrieval accuracy across 20 different clustering seeds. All results in the figure were evaluated using a ResNet backbone.

Evaluation Model	Pairs	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	100	7.9 ± 0.3	25.8 ± 0.4	37.3 ± 0.3	10.2 ± 0.3	28.2 ± 0.9	39.0 ± 0.3
	300	14.4 ± 0.4	38.1 ± 0.2	51.4 ± 0.4	18.7 ± 0.5	45.0 ± 0.4	57.8 ± 0.6
	500	17.1 ± 0.2	43.0 ± 0.2	56.4 ± 0.1	23.9 ± 0.9	50.5 ± 0.6	62.0 ± 0.3
	1000	19.5 ± 0.3	46.4 ± 0.4	58.9 ± 0.2	27.4 ± 0.4	55.3 ± 0.6	67.5 ± 0.6
	1500	20.5 ± 0.2	48.7 ± 0.3	62.5 ± 0.2	28.8 ± 0.5	59.2 ± 0.8	71.2 ± 0.4
	Full Dataset	28.5 ± 0.2	59.6 ± 0.1	71.4 ± 0.1	46.0 ± 0.6	76.2 ± 0.3	84.4 ± 0.2

Appendix

Table 13: Increasing the distilled dataset size consistently improves performance. We distill Flickr30K using a wide range of distilled dataset sizes and evaluate the resulting distilled datasets. In all cases, larger distilled sets lead to better performance.

Evaluation Model	CLIP variant	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	ViT-L/14	14.4 ± 0.4	38.1 ± 0.2	51.4 ± 0.4	18.7 ± 0.5	45.0 ± 0.4	57.8 ± 0.6
ResNet	ViT-H/14	17.2 ± 0.5	42.6 ± 0.3	55.6 ± 0.4	23.7 ± 0.9	47.3 ± 0.8	60.0 ± 0.8
ViT	ViT-L/14	9.1 ± 0.1	27.3 ± 0.4	38.4 ± 0.4	9.6 ± 0.3	26.1 ± 0.5	37.5 ± 1.2
ViT	ViT-H/14	12.0 ± 0.1	32.8 ± 0.3	43.2 ± 0.6	12.7 ± 0.6	32.4 ± 0.2	44.3 ± 0.2

Appendix

Table 14: Results with an alternative CLIP variant in the 300-pair setting on Flickr30K. We evaluate PDS using CLIP ViT-H/14 to examine whether the improvements generalize across a different variant. CLIP ViT-H/14 achieves higher performance, likely due to its stronger image-text alignment.

Evaluation Model	Methods	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	K-center	8.5 ± 1.4	26.7 ± 0.9	40.0 ± 0.8	15.4 ± 0.9	40.3 ± 0.5	54.0 ± 1.6
	Herding	9.4 ± 0.5	28.7 ± 0.6	43.7 ± 1.7	15.6 ± 1.2	38.8 ± 2.0	51.3 ± 1.4
	CLIP score	7.2 ± 0.4	22.3 ± 0.4	33.5 ± 0.5	12.8 ± 1.6	30.6 ± 1.2	42.9 ± 2.1
	LAION filtering	6.6 ± 0.3	20.2 ± 0.5	31.8 ± 0.3	8.8 ± 0.5	24.3 ± 1.0	36.3 ± 1.8
	Image-based	6.7 ± 0.5	22.4 ± 0.3	33.3 ± 0.3	12.3 ± 1.8	30.2 ± 1.4	42.1 ± 1.5
	TESLA-VL	11.0 ± 1.7	35.0 ± 1.5	50.9 ± 2.1	15.0 ± 2.9	39.6 ± 3.9	52.5 ± 3.0

Appendix

Table 15: PDS is more robust to rare samples than existing methods. We evaluate each method on 200 rare test samples from Flickr30K, defined as those farthest from the mean embedding of the training dataset. In the 100-pair setting on Flickr30K, PDS consistently outperforms both subset selection and other distillation baselines, demonstrating stronger robustness to rare samples.

Evaluation Model	Methods	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	K-center	0.8 ± 0.8	3.4 ± 3.1	6.1 ± 5.2	1.2 ± 1.3	3.9 ± 3.5	6.6 ± 5.1
	Herding	0.8 ± 0.9	3.3 ± 3.4	5.6 ± 5.5	1.0 ± 1.4	3.8 ± 4.4	6.0 ± 6.5
	CLIP score	3.5 ± 0.2	12.5 ± 0.2	19.4 ± 0.3	5.5 ± 0.4	14.4 ± 0.6	22.1 ± 0.5
	LAION filtering	3.5 ± 0.1	13.0 ± 0.3	20.4 ± 0.3	6.4 ± 0.5	15.2 ± 0.9	22.6 ± 0.7
	Image-based	3.4 ± 0.2	12.3 ± 0.2	18.8 ± 0.2	5.6 ± 0.3	14.1 ± 0.6	21.9 ± 0.4
	PDS	8.5 ± 0.2	27.0 ± 0.3	39.8 ± 0.3	10.3 ± 0.2	27.1 ± 0.7	38.3 ± 0.4

Appendix

Table 16: Comparison of PDS with subset selection methods on Flickr8K. We evaluate PDS and subset selection baselines in the 100-pair setting on Flickr8K, and PDS consistently outperforms all subset selection methods.

Evaluation Model	Methods	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	Text prototype	8.1 ± 0.2	24.9 ± 0.2	36.6 ± 0.4	10.8 ± 0.4	28.2 ± 0.7	38.3 ± 0.7
	Average prototype	7.9 ± 0.2	26.2 ± 0.5	37.9 ± 0.5	11.1 ± 0.3	28.2 ± 0.5	39.4 ± 0.3
	Image prototype	7.9 ± 0.3	25.8 ± 0.4	37.3 ± 0.3	10.2 ± 0.3	28.2 ± 0.9	39.0 ± 0.3
ViT	Text prototype	5.6 ± 0.2	17.3 ± 0.2	26.4 ± 0.1	4.6 ± 0.4	16.3 ± 0.7	23.9 ± 0.6
	Average prototype	5.9 ± 0.1	18.5 ± 0.1	27.7 ± 0.3	6.5 ± 0.6	18.4 ± 0.3	27.3 ± 0.3
	Image prototype	6.8 ± 0.3	19.2 ± 0.3	28.5 ± 0.4	6.6 ± 0.5	17.5 ± 0.5	26.9 ± 0.5

Appendix

Table 17: Comparison of conditioning on text, image, and average prototypes. We evaluate different conditioning strategies in the 100-pair setting on Flickr30K. Conditioning on the text prototype degrades performance, while using the average of the text and image prototypes achieves results comparable to conditioning on the image prototype, providing no additional gains.

Dataset	Pairs	Methods	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
Flickr30K	100	Joint	8.3 ± 0.1	25.3 ± 0.2	36.8 ± 0.4	11.0 ± 0.4	29.5 ± 0.4	40.7 ± 0.4
	100	Separate	7.9 ± 0.3	25.8 ± 0.4	37.3 ± 0.3	10.2 ± 0.3	28.2 ± 0.9	39.0 ± 0.3
	300	Joint	14.6 ± 0.5	38.8 ± 0.3	52.3 ± 0.4	17.5 ± 0.4	43.5 ± 0.4	56.4 ± 0.8
	300	Separate	14.4 ± 0.4	38.1 ± 0.2	51.4 ± 0.4	18.7 ± 0. 5	45.0 ± 0.4	57.8 ± 0.6
MS-COCO	100	Joint	2.5 ± 0.1	10.2 ± 0.2	17.5 ± 0.3	4.0 ± 0.1	12.9 ± 0.2	20.6 ± 0.2
	100	Separate	2.8 ± 0.1	10.0 ± 0.2	17.3 ± 0.3	4.5 ± 0.2	14.0 ± 0.3	21.4 ± 0.4

Appendix

Table 18: Comparison of joint-embedding clustering (Joint) and separate clustering (Separate). We evaluate different clustering strategies by distilling the Flickr30K and MS-COCO into 100 and 300 pairs, and evaluating the resulting distilled sets using a ResNet backbone. Overall, both clustering strategies perform comparably, with the separate clustering approach showing slight improvements in a few cases.

Evaluation Model	Methods	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	K-center	3.8	14.4	24.2	4.0	14.8	22.7
	Herding	6.0	21.5	32.5	6.9	19.1	28.1
	CLIP score	2.1	9.5	16.9	3.5	11.5	18.7
	LAION filtering	3.1	11.2	17.9	3.4	12.5	17.4
	Image-based	2.3	9.6	17.1	4.1	12.6	18.3
	TESLA-VL	5.3	19.4	29.8	5.0	16.8	25.9

Appendix

Table 19: Application of the distilled dataset as anchors in ASIF. To evaluate how well distilled or selected datasets can serve as compact anchors for ASIF, we construct 300 anchors from Flickr30K for each method, including subset selection, other distillation approaches, and PDS. Using a ResNet image encoder and a CLIP text encoder to compute relative representations, we measure retrieval accuracy on Flickr30K. Anchors produced by PDS consistently outperform those obtained from all other baselines, demonstrating that PDS yields more informative and effective anchors for ASIF.

Evaluation Model	Method	IR@1	IR@5	IR@10	TR@1	TR@5	TR@10
ResNet	TESLA-VL	4.1 ± 0.3	14.7 ± 0.9	22.9 ± 1.2	6.5 ± 0.4	17.8 ± 1.4	27.3 ± 1.4
	TESLA-VL +PDS	5.2 ± 0.4	17.7 ± 0.6	27.5 ± 0.4	7.5 ± 1.0	22.4 ± 1.3	33.1 ± 1.3
	LoRS	6.3 ± 0.1	18.6 ± 0.1	28.0 ± 0.2	9.1 ± 0.2	24.3 ± 0.4	34.5 ± 0.8
	LoRS+PDS	5.7 ± 0.2	19.5 ± 0.6	29.2 ± 0.9	8.3 ± 1.2	24.1 ± 1.3	34.3 ± 1.4
	PDS	7.9 ± 0.3	25.8 ± 0.4	37.3 ± 0.3	10.2 ± 0.3	28.2 ± 0.9	39.0 ± 0.3

Appendix

Table 20: Integrating PDS into optimization-based distillation baselines provides little advantage. We evaluate optimization-based multimodal distillation methods initialized with the PDS-distilled dataset in the 100-pair setting on Flickr30K. Although this initialization provides a more informative starting point, the subsequent optimization makes the distilled dataset architecture-dependent, limiting the cross-architecture generalization achieved by PDS. Consequently, while PDS initialization tends to improve these baselines, this integrated approach still underperforms compared to using PDS alone.

Figure 7: Examples of synthetic images and retrieved captions in the 100-pair setting on Flickr30K.

Figure 8: Examples of synthetic images and retrieved captions in the 300-pair setting on Flickr30K.

Figure 9: Examples of synthetic images and retrieved captions in the 100-pair setting on MS-COCO.

Figure 10: Examples of synthetic images and retrieved captions in the 300-pair setting on MS-COCO.

Dataset Distillation

Dataset distillation compresses training data into a small synthetic or selected set that retains the information needed to train useful models.

See all 67 papers

Multimodal Distribution Matching for Vision-Language Dataset Distillation
May 22, 2026Jongoh Jeong, Hoyong Kwon, Minseok Kim +1Cross-Modal Learning Cross-Modal Alignment
Visual Intelligence Lab., KAIST
Multimodal Dataset Distillation via Phased Teacher Models
Mar 26, 2026Shengbin Guo, Hang Zhao, Senqiao Yang +5Dataset Distillation Knowledge Distillation
Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong
Omnimodal Dataset Distillation via High-order Proxy Alignment
Apr 12, 2026Yuxuan Gao, Xiaohao Liu, Xiaobo Xia +1Dataset Distillation Multimodal Dataset
University of Science and Technology of China · Xidian University · National University of Singapore +1

May 22, 2026J/K move · Enter open · S save

Multimodal Distribution Matching for Vision-Language Dataset Distillation

Jongoh Jeong, Hoyong Kwon, Minseok Kim, Kuk-Jin Yoon

Visual Intelligence Lab., KAIST

Cross-Modal Learning Cross-Modal Alignment VLM Distillation Distribution Matching Dataset Distillation

Open paper AskPDF arXiv

Dataset distillation compresses large training sets into compact synthetic datasets while preserving downstream performance. As modern systems increasingly operate on paired vision-language inputs, multimodal distillation must preserve representation quality and cross-modal alignment under tight compute and memory budgets, yet prior methods often require heavy computes and overlook their correlations. To address this, we present Multimodal Distribution Matching (MDM), a geometry-aware framework for efficient and generalizable multimodal distillation. Specifically, MDM integrates complementary components at the data, model, and loss levels. At the data level, it initializes synthetic image-text pairs by sampling from clusters in the joint embedding space. At the model level, it forms a mixed teacher by interpolating independently fine-tuned models in weight space according to their angular deviation from the pretrained anchor. At the loss level, it matches joint distributions on the unit hypersphere using a geometry-aware matching objective that exploits the joint features in the cross-modal agreement and discrepancy directions along with symmetric contrastive learning. Across image-text retrieval benchmarks with cross-architecture evaluation, MDM yields compact synthetic sets that preserve multimodal semantics, substantially reduce distillation cost, and remain robust across architectures.

Abstract

Dataset Distillation

Explore similar work

Multimodal Distribution Matching for Vision-Language Dataset Distillation

Multimodal Dataset Distillation via Phased Teacher Models

Omnimodal Dataset Distillation via High-order Proxy Alignment