Recent advances in multimodal learning have achieved remarkable success across diverse vision-language tasks. However, such progress heavily relies on large-scale image-text datasets, making training costly and inefficient. Prior efforts in dataset filtering and pruning attempt to mitigate this issue, but still require relatively large subsets to maintain performance and fail under very small subsets. Dataset distillation offers a promising alternative, yet existing multimodal dataset distillation methods require full-dataset training and joint optimization of image pixels and text features, making them architecture-dependent and limiting cross-architecture generalization. To overcome this, we propose a learning-free dataset distillation framework that eliminates the need for large-scale training and optimization while enhancing generalization across architectures. Our method uses CLIP to extract aligned image-text embeddings, obtains prototypes, and employs an unCLIP decoder to synthesize images, enabling efficient and scalable multimodal dataset distillation. Extensive experiments demonstrate that our approach consistently outperforms optimization-based dataset distillation and subset selection methods, achieving state-of-the-art cross-architecture generalization.
Figures & tables
Figure 1: Concept. Optimization-based methods are computationally expensive and architecture-dependent. Our learning-free framework is simple, efficient, and generalizable across architectures.
Figure 2: Our PDS framework distills multimodal datasets by synthesizing samples from image-text prototypes through three stages: (i) modality-specific clustering of CLIP embeddings, (ii) cross-modal cluster matching via a linear assignment to obtain prototypes, and (iii) image synthesis using an unCLIP decoder guided by image prototype and caption embeddings.
Evaluation Model
Pairs
Methods
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
100
TESLA-VL
4.1 ± 0.3
14.7 ± 0.9
22.9 ± 1.2
6.5 ± 0.4
17.8 ± 1.4
27.3 ± 1.4
LoRS
6.3 ± 0.1
18.6 ± 0.1
28.0 ± 0.2
9.1 ± 0.2
24.3 ± 0.4
34.5 ± 0.8
PDS
7.9 ± 0.3
25.8 ± 0.4
37.3 ± 0.3
10.2 ± 0.3
28.2 ± 0.9
39.0 ± 0.3
300
TESLA-VL
10.3 ± 0.3
28.8 ± 0.6
40.6 ± 0.8
14.9 ± 0.9
36.2 ± 1.1
48.8 ± 1.7
LoRS
8.6 ± 0.9
23.5 ± 1.5
33.5 ± 1.8
14.7 ± 1.2
33.5 ± 2.8
44.1 ± 3.2
PDS
14.4 ± 0.4
38.1 ± 0.2
51.4 ± 0.4
18.7 ± 0.5
45.0 ± 0.4
57.8 ± 0.6
Table 1: PDS distills datasets with superior cross-architecture generalization compared to multimodal dataset distillation baselines. We evaluate on unseen vision backbones, comparing against TESLA-VL and LoRS on Flickr30K (top) and MS-COCO (bottom). Across all distilled dataset sizes and evaluation backbones, PDS consistently outperforms both baselines, demonstrating that its distilled datasets generalize effectively to unseen backbones.
Evaluation Model
Metric
Subset selection
Distillation
K-center
Herding
CLIP score
LAION filtering
Image- based
PDS
ResNet
IR@1
2.9 ± 0.1
3.6 ± 0.2
2.5 ± 0.1
2.4 ± 0.1
2.2 ± 0.2
7.9 ± 0.3
IR@5
10.4 ± 0.3
12.6 ± 0.5
8.7 ± 0.3
8.5 ± 0.2
8.6 ± 0.3
25.8 ± 0.4
IR@10
16.8 ± 0.3
20.1 ± 0.6
14.5 ± 0.3
14.5 ± 0.3
13.6 ± 0.4
37.3 ± 0.3
TR@1
5.3 ± 0.3
6.7 ± 0.3
4.7 ± 0.4
4.7 ± 0.3
4.0 ± 0.3
10.2 ± 0.3
TR@5
15.6 ± 0.5
19.4 ± 0.7
12.6 ± 0.7
12.4 ± 0.6
10.5 ± 0.4
28.2 ± 0.9
Table 2: The representative samples obtained by interpolating diverse semantic features are more suitable than subsets of the original dataset when the reduced dataset is extremely small. We compare PDS with subset selection methods in the 100-pair setting on Flickr30K. Baselines restricted to real samples are limited in preserving broad semantic diversity. In contrast, PDS synthesizes representative samples that reflect diverse semantics and consistently achieves superior results.
Evaluation Model
Methods
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
D 4 M
1.3 ± 0.1
5.8 ± 0.2
9.8 ± 0.3
1.4 ± 0.2
5.6 ± 0.5
9.5 ± 0.3
MGD 3
2.6 ± 0.1
10.0 ± 0.2
17.2 ± 0.5
3.4 ± 0.2
11.6 ± 0.3
17.5 ± 0.4
PDS
7.9 ± 0.3
25.8 ± 0.4
37.3 ± 0.3
10.2 ± 0.3
28.2 ± 0.9
39.0 ± 0.3
Table 3: Cross-modal alignment is crucial for effective multimodal dataset distillation. We compare PDS with learning-free methods for image classification in the 100-pair setting on Flickr30K. Both D4M and MGD3 rely on the VAE encoder, which produces image features that are not aligned with text features, leading to low retrieval accuracy. In contrast, PDS uses CLIP encoders to enforce cross-modal alignment, outperforming the baselines and highlighting the importance of alignment-aware distillation.
Figure 3: The synthesized images. Left (col. 1-3): Images from multimodal dataset distillation baselines, which are nearly identical to the initialization image. Middle: Image generated via CLIP inversion from image prototypes, which is not realistic. Right (col. 5-7): Given a text prototype, we first retrieve the most similar caption and present three images: the paired real image, the UnCLIP-generated image from this caption, and the PDS-generated image. Unlike UnCLIP, which strictly follows captions, PDS produces realistic and semantically enriched images by conditioning on image prototypes.
Evaluation Model
Methods
IR@1
TR@1
Memory (GB)
Time (s)
ResNet
Text alignment
1.4 ± 0.1
2.0 ± 0.5
6.13
1477.71
Image alignment
4.4 ± 0.2
4.2 ± 0.3
PDS
7.9 ± 0.3
10.2 ± 0.3
4.34
9.71
Table 4: Using a generative model for image synthesis improves both efficiency and performance. We evaluate two variants of CLIP inversion: (i) Image prototype alignment and (ii) Text prototype alignment, both of which directly optimize pixel values, in the 100-pair setting on Flickr30K. By employing a generative model, PDS accelerates image generation, reduces memory usage, and produces realistic images, achieving superior performance over CLIP inversion.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
P∈{0,1}M×Mmini=1∑Mj=1∑MKijPijsubject to j=1∑MPij=1,i=1∑MPij=1,Pij∈{0,1},
Appendix
Algorithm 1 PDS framework
Evaluation Model
Methods
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
GMM
8.3±0.1
26.2±0.3
37.4±0.2
10.9±0.4
30.1±0.4
41.1±0.6
Agglomerative
9.3±0.2
28.9±0.3
41.4±0.1
11.6±0.6
30.2±0.3
41.6±0.6
Mini-batch k-means
7.9±0.3
25.8±0.4
37.3±0.3
10.2±0.3
28.2±0.9
39.0±0.3
ViT
GMM
6.0±0.4
19.2±0.3
28.3±0.1
6.1±0.4
18.2±0.9
26.9±0.7
Agglomerative
6.3±0.2
19.9±0.2
29.5±0.4
6.4±0.5
18.5±0.3
27.4±0.4
Mini-batch k-means
6.8±0.3
19.2±0.3
28.5±0.4
6.6±0.5
17.5±0.5
26.9±0.5
Appendix
Table 6: Comparison of clustering methods. We evaluate PDS in the 100-pair setting on Flickr30K using different clustering algorithms. PDS achieves robust performance, indicating low sensitivity to the choice of clustering algorithm.
GMM
Agglomerative
Mini-batch k-means
Clustering Time (s)
1,469.7
4,387.9
9.0
Appendix
Table 7: Clustering time comparison. Clustering time (in seconds) across different methods.
Figure 4: Histograms of cosine similarity between paired image and text prototypes in the 100-pair setting on Flickr30K, shown with and without filtering. Filtering leads to higher similarity.
Evaluation Model
Methods
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
Non-filtering
5.3 ± 0.1
18.4 ± 0.1
28.1 ± 0.3
7.4 ± 0.5
21.2 ± 0.7
30.6 ± 0.5
Filtering
7.9 ± 0.3
25.8 ± 0.4
37.3 ± 0.3
10.2 ± 0.3
28.2 ± 0.9
39.0 ± 0.3
ViT
Non-filtering
3.9 ± 0.0
14.0 ± 0.0
21.3 ± 0.0
4.5 ± 0.0
12.4 ± 0.0
19.6 ± 0.0
Filtering
6.8 ± 0.3
19.2 ± 0.3
28.5 ± 0.4
6.6 ± 0.5
17.5 ± 0.5
26.9 ± 0.5
Appendix
Table 8: Filtering enhances cross-modal alignment. We compare our filtering strategy against a non-filtering baseline in the 100-pair setting on Flickr30K. Prototypes obtained from embeddings of the shared image–text pairs consistently improve performance.
Pairs
100
300
500
1000
1500
Number of pairless clusters
1
8
24
87
166
Appendix
Table 9: Larger distilled sizes yield more pairless clusters. We distill Flickr30K into various numbers of pairs. As the distilled size increases, the number of pairless clusters grows substantially.
Figure 5: Histograms of cosine similarity between paired image and text prototypes in the 1000-pair setting on Flickr30K, shown for pairless clusters and clusters with shared pairs. Pairless clusters exhibit lower similarity.
Evaluation Model
Pairs
Methods
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
100
Keeping
7.9 ± 0.3
25.8 ± 0.4
37.3 ± 0.3
10.2 ± 0.3
28.2 ± 0.9
39.0 ± 0.3
Discarding
7.8 ± 0.2
25.9 ± 0.4
37.5 ± 0.2
10.0 ± 0.3
27.5 ± 1.2
38.8 ± 0.9
300
Keeping
14.4 ± 0.4
38.1 ± 0.2
51.4 ± 0.4
18.7 ± 0.5
45.0 ± 0.4
57.8 ± 0.6
Discarding
14.9 ± 0.2
38.6 ± 0.5
51.9 ± 0.3
19.3 ± 0.4
45.0 ± 0.5
57.2 ± 0.6
500
Keeping
16.3 ± 0.2
41.3 ± 0.2
54.9 ± 0.4
22.5 ± 0.3
49.3 ± 0.8
61.4 ± 0.4
Discarding
17.1 ± 0.2
43.0 ± 0.2
56.4 ± 0.1
23.9 ± 0.9
50.5 ± 0.6
62.0 ± 0.3
Appendix
Table 10: Discarding pairless clusters becomes increasingly beneficial at larger distilled sizes. We distill Flickr30K to a wide range of distilled dataset sizes and compare two strategies, keeping all clusters or discarding the pairless clusters. While both strategies perform similarly in small-scale settings, discarding pairless clusters becomes preferable when the distilled size becomes large.
Evaluation Model
Methods
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
Image prototype
6.0 ± 0.3
19.2 ± 0.3
28.3 ± 0.1
6.1 ± 0.4
18.2 ± 0.9
26.9 ± 0.7
Both prototypes
7.9 ± 0.3
25.8 ± 0.4
37.3 ± 0.3
10.2 ± 0.3
28.2 ± 0.9
39.0 ± 0.3
ViT
Image prototype
5.4 ± 0.1
17.5 ± 0.2
26.5 ± 0.5
5.4 ± 0.3
17.8 ± 0.5
25.7 ± 1.1
Both prototypes
6.8 ± 0.3
19.2 ± 0.3
28.5 ± 0.4
6.6 ± 0.5
17.5 ± 0.5
26.9 ± 0.5
Appendix
Table 11: Joint conditioning improves performance. We compare our method with a baseline that generates images conditioned solely on the image prototype in the 100-pair setting on Flickr30K. Conditioning on both the image prototype and a caption retrieved from the text prototype improves performance.
Method
Evaluation Model
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
TESLA-VL
ResNet+BERT
0.3 ± 0.1
1.9 ± 0.2
3.4 ± 0.2
2.0 ± 0.4
7.9 ± 0.5
12.2 ± 0.5
ResNet+CLIP
4.1 ± 0.3
14.7 ± 0.9
22.9 ± 1.2
6.5 ± 0.4
17.8 ± 1.4
27.3 ± 1.4
ViT+BERT
0.2 ± 0.1
1.0 ± 0.2
1.9 ± 0.5
0.5 ± 0.2
2.0 ± 0.6
3.9 ± 1.4
ViT+CLIP
2.1 ± 0.3
7.8 ± 0.7
13.1 ± 1.2
2.6 ± 0.6
8.7 ± 0.9
13.7 ± 1.4
LoRS
ResNet+BERT
2.0 ± 0.1
7.3 ± 0.3
11.8 ± 0.2
3.7 ± 0.4
13.4 ± 0.5
19.8 ± 0.8
ResNet+CLIP
6.3 ± 0.1
18.6 ± 0.1
28.0 ± 0.2
9.1 ± 0.2
24.3 ± 0.4
34.5 ± 0.8
Appendix
Table 12: Comparison of BERT and CLIP as text encoders in the 100-pair setting on Flickr30K. Within each method, we compare BERT and CLIP as text encoders under the same vision backbone (ResNet or ViT). In all cases, replacing BERT with CLIP consistently improves performance.
Figure 6: Ablation study on hyperparameters. The first three panels show the results obtained by varying the guidance scale, the number of sampling steps, and the pruning threshold ratio, respectively. Each panel reports the aggregated retrieval accuracy, computed as the average of IR@1, IR@5, IR@10, TR@1, TR@5, and TR@10, using distilled sets of 100 and 300 pairs on Flickr30K. The final panel presents a boxplot of retrieval accuracy across 20 different clustering seeds. All results in the figure were evaluated using a ResNet backbone.
Evaluation Model
Pairs
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
100
7.9 ± 0.3
25.8 ± 0.4
37.3 ± 0.3
10.2 ± 0.3
28.2 ± 0.9
39.0 ± 0.3
300
14.4 ± 0.4
38.1 ± 0.2
51.4 ± 0.4
18.7 ± 0.5
45.0 ± 0.4
57.8 ± 0.6
500
17.1 ± 0.2
43.0 ± 0.2
56.4 ± 0.1
23.9 ± 0.9
50.5 ± 0.6
62.0 ± 0.3
1000
19.5 ± 0.3
46.4 ± 0.4
58.9 ± 0.2
27.4 ± 0.4
55.3 ± 0.6
67.5 ± 0.6
1500
20.5 ± 0.2
48.7 ± 0.3
62.5 ± 0.2
28.8 ± 0.5
59.2 ± 0.8
71.2 ± 0.4
Full Dataset
28.5 ± 0.2
59.6 ± 0.1
71.4 ± 0.1
46.0 ± 0.6
76.2 ± 0.3
84.4 ± 0.2
Appendix
Table 13: Increasing the distilled dataset size consistently improves performance. We distill Flickr30K using a wide range of distilled dataset sizes and evaluate the resulting distilled datasets. In all cases, larger distilled sets lead to better performance.
Evaluation Model
CLIP variant
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
ViT-L/14
14.4 ± 0.4
38.1 ± 0.2
51.4 ± 0.4
18.7 ± 0.5
45.0 ± 0.4
57.8 ± 0.6
ViT-H/14
17.2 ± 0.5
42.6 ± 0.3
55.6 ± 0.4
23.7 ± 0.9
47.3 ± 0.8
60.0 ± 0.8
ViT
ViT-L/14
9.1 ± 0.1
27.3 ± 0.4
38.4 ± 0.4
9.6 ± 0.3
26.1 ± 0.5
37.5 ± 1.2
ViT-H/14
12.0 ± 0.1
32.8 ± 0.3
43.2 ± 0.6
12.7 ± 0.6
32.4 ± 0.2
44.3 ± 0.2
Appendix
Table 14: Results with an alternative CLIP variant in the 300-pair setting on Flickr30K. We evaluate PDS using CLIP ViT-H/14 to examine whether the improvements generalize across a different variant. CLIP ViT-H/14 achieves higher performance, likely due to its stronger image-text alignment.
Evaluation Model
Methods
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
K-center
8.5 ± 1.4
26.7 ± 0.9
40.0 ± 0.8
15.4 ± 0.9
40.3 ± 0.5
54.0 ± 1.6
Herding
9.4 ± 0.5
28.7 ± 0.6
43.7 ± 1.7
15.6 ± 1.2
38.8 ± 2.0
51.3 ± 1.4
CLIP score
7.2 ± 0.4
22.3 ± 0.4
33.5 ± 0.5
12.8 ± 1.6
30.6 ± 1.2
42.9 ± 2.1
LAION filtering
6.6 ± 0.3
20.2 ± 0.5
31.8 ± 0.3
8.8 ± 0.5
24.3 ± 1.0
36.3 ± 1.8
Image-based
6.7 ± 0.5
22.4 ± 0.3
33.3 ± 0.3
12.3 ± 1.8
30.2 ± 1.4
42.1 ± 1.5
TESLA-VL
11.0 ± 1.7
35.0 ± 1.5
50.9 ± 2.1
15.0 ± 2.9
39.6 ± 3.9
52.5 ± 3.0
Appendix
Table 15: PDS is more robust to rare samples than existing methods. We evaluate each method on 200 rare test samples from Flickr30K, defined as those farthest from the mean embedding of the training dataset. In the 100-pair setting on Flickr30K, PDS consistently outperforms both subset selection and other distillation baselines, demonstrating stronger robustness to rare samples.
Evaluation Model
Methods
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
K-center
0.8 ± 0.8
3.4 ± 3.1
6.1 ± 5.2
1.2 ± 1.3
3.9 ± 3.5
6.6 ± 5.1
Herding
0.8 ± 0.9
3.3 ± 3.4
5.6 ± 5.5
1.0 ± 1.4
3.8 ± 4.4
6.0 ± 6.5
CLIP score
3.5 ± 0.2
12.5 ± 0.2
19.4 ± 0.3
5.5 ± 0.4
14.4 ± 0.6
22.1 ± 0.5
LAION filtering
3.5 ± 0.1
13.0 ± 0.3
20.4 ± 0.3
6.4 ± 0.5
15.2 ± 0.9
22.6 ± 0.7
Image-based
3.4 ± 0.2
12.3 ± 0.2
18.8 ± 0.2
5.6 ± 0.3
14.1 ± 0.6
21.9 ± 0.4
PDS
8.5 ± 0.2
27.0 ± 0.3
39.8 ± 0.3
10.3 ± 0.2
27.1 ± 0.7
38.3 ± 0.4
Appendix
Table 16: Comparison of PDS with subset selection methods on Flickr8K. We evaluate PDS and subset selection baselines in the 100-pair setting on Flickr8K, and PDS consistently outperforms all subset selection methods.
Evaluation Model
Methods
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
Text prototype
8.1 ± 0.2
24.9 ± 0.2
36.6 ± 0.4
10.8 ± 0.4
28.2 ± 0.7
38.3 ± 0.7
Average prototype
7.9 ± 0.2
26.2 ± 0.5
37.9 ± 0.5
11.1 ± 0.3
28.2 ± 0.5
39.4 ± 0.3
Image prototype
7.9 ± 0.3
25.8 ± 0.4
37.3 ± 0.3
10.2 ± 0.3
28.2 ± 0.9
39.0 ± 0.3
ViT
Text prototype
5.6 ± 0.2
17.3 ± 0.2
26.4 ± 0.1
4.6 ± 0.4
16.3 ± 0.7
23.9 ± 0.6
Average prototype
5.9 ± 0.1
18.5 ± 0.1
27.7 ± 0.3
6.5 ± 0.6
18.4 ± 0.3
27.3 ± 0.3
Image prototype
6.8 ± 0.3
19.2 ± 0.3
28.5 ± 0.4
6.6 ± 0.5
17.5 ± 0.5
26.9 ± 0.5
Appendix
Table 17: Comparison of conditioning on text, image, and average prototypes. We evaluate different conditioning strategies in the 100-pair setting on Flickr30K. Conditioning on the text prototype degrades performance, while using the average of the text and image prototypes achieves results comparable to conditioning on the image prototype, providing no additional gains.
Dataset
Pairs
Methods
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
Flickr30K
100
Joint
8.3 ± 0.1
25.3 ± 0.2
36.8 ± 0.4
11.0 ± 0.4
29.5 ± 0.4
40.7 ± 0.4
Separate
7.9 ± 0.3
25.8 ± 0.4
37.3 ± 0.3
10.2 ± 0.3
28.2 ± 0.9
39.0 ± 0.3
300
Joint
14.6 ± 0.5
38.8 ± 0.3
52.3 ± 0.4
17.5 ± 0.4
43.5 ± 0.4
56.4 ± 0.8
Separate
14.4 ± 0.4
38.1 ± 0.2
51.4 ± 0.4
18.7 ± 0. 5
45.0 ± 0.4
57.8 ± 0.6
MS-COCO
100
Joint
2.5 ± 0.1
10.2 ± 0.2
17.5 ± 0.3
4.0 ± 0.1
12.9 ± 0.2
20.6 ± 0.2
Separate
2.8 ± 0.1
10.0 ± 0.2
17.3 ± 0.3
4.5 ± 0.2
14.0 ± 0.3
21.4 ± 0.4
Appendix
Table 18: Comparison of joint-embedding clustering (Joint) and separate clustering (Separate). We evaluate different clustering strategies by distilling the Flickr30K and MS-COCO into 100 and 300 pairs, and evaluating the resulting distilled sets using a ResNet backbone. Overall, both clustering strategies perform comparably, with the separate clustering approach showing slight improvements in a few cases.
Evaluation Model
Methods
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
K-center
3.8
14.4
24.2
4.0
14.8
22.7
Herding
6.0
21.5
32.5
6.9
19.1
28.1
CLIP score
2.1
9.5
16.9
3.5
11.5
18.7
LAION filtering
3.1
11.2
17.9
3.4
12.5
17.4
Image-based
2.3
9.6
17.1
4.1
12.6
18.3
TESLA-VL
5.3
19.4
29.8
5.0
16.8
25.9
Appendix
Table 19: Application of the distilled dataset as anchors in ASIF. To evaluate how well distilled or selected datasets can serve as compact anchors for ASIF, we construct 300 anchors from Flickr30K for each method, including subset selection, other distillation approaches, and PDS. Using a ResNet image encoder and a CLIP text encoder to compute relative representations, we measure retrieval accuracy on Flickr30K. Anchors produced by PDS consistently outperform those obtained from all other baselines, demonstrating that PDS yields more informative and effective anchors for ASIF.
Evaluation Model
Method
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet
TESLA-VL
4.1 ± 0.3
14.7 ± 0.9
22.9 ± 1.2
6.5 ± 0.4
17.8 ± 1.4
27.3 ± 1.4
TESLA-VL +PDS
5.2 ± 0.4
17.7 ± 0.6
27.5 ± 0.4
7.5 ± 1.0
22.4 ± 1.3
33.1 ± 1.3
LoRS
6.3 ± 0.1
18.6 ± 0.1
28.0 ± 0.2
9.1 ± 0.2
24.3 ± 0.4
34.5 ± 0.8
LoRS+PDS
5.7 ± 0.2
19.5 ± 0.6
29.2 ± 0.9
8.3 ± 1.2
24.1 ± 1.3
34.3 ± 1.4
PDS
7.9 ± 0.3
25.8 ± 0.4
37.3 ± 0.3
10.2 ± 0.3
28.2 ± 0.9
39.0 ± 0.3
Appendix
Table 20: Integrating PDS into optimization-based distillation baselines provides little advantage. We evaluate optimization-based multimodal distillation methods initialized with the PDS-distilled dataset in the 100-pair setting on Flickr30K. Although this initialization provides a more informative starting point, the subsequent optimization makes the distilled dataset architecture-dependent, limiting the cross-architecture generalization achieved by PDS. Consequently, while PDS initialization tends to improve these baselines, this integrated approach still underperforms compared to using PDS alone.
Figure 7: Examples of synthetic images and retrieved captions in the 100-pair setting on Flickr30K.
Figure 8: Examples of synthetic images and retrieved captions in the 300-pair setting on Flickr30K.
Figure 9: Examples of synthetic images and retrieved captions in the 100-pair setting on MS-COCO.
Figure 10: Examples of synthetic images and retrieved captions in the 300-pair setting on MS-COCO.
Dataset distillation compresses large training sets into compact synthetic datasets while preserving downstream performance. As modern systems increasingly operate on paired vision-language inputs, multimodal distillation must preserve representation quality and cross-modal alignment under tight compute and memory budgets, yet prior methods often require heavy computes and overlook their correlations. To address this, we present Multimodal Distribution Matching (MDM), a geometry-aware framework for efficient and generalizable multimodal distillation. Specifically, MDM integrates complementary components at the data, model, and loss levels. At the data level, it initializes synthetic image-text pairs by sampling from clusters in the joint embedding space. At the model level, it forms a mixed teacher by interpolating independently fine-tuned models in weight space according to their angular deviation from the pretrained anchor. At the loss level, it matches joint distributions on the unit hypersphere using a geometry-aware matching objective that exploits the joint features in the cross-modal agreement and discrepancy directions along with symmetric contrastive learning. Across image-text retrieval benchmarks with cross-architecture evaluation, MDM yields compact synthetic sets that preserve multimodal semantics, substantially reduce distillation cost, and remain robust across architectures.
Multimodal dataset distillation aims to construct compact synthetic datasets that enable efficient compression and knowledge transfer from large-scale image-text data. However, existing approaches often fail to capture the complex, dynamically evolving knowledge embedded in the later training stages of teacher models. This limitation leads to degraded student performance and compromises the quality of the distilled data. To address critical challenges such as pronounced cross-stage performance gaps and unstable teacher trajectories, we propose Phased Teacher Model with Shortcut Trajectory (PTM-ST) -- a novel phased distillation framework. PTM-ST leverages stage-aware teacher modeling and a shortcut-based trajectory construction strategy to accurately fit the teacher's learning dynamics across distinct training phases. This enhances both the stability and expressiveness of the distillation process. Through theoretical analysis and comprehensive experiments, we show that PTM-ST significantly mitigates optimization oscillations and inter-phase knowledge gaps, while also reducing storage overhead. Our method consistently surpasses state-of-the-art baselines on Flickr30k and COCO, achieving up to 13.5% absolute improvement and an average gain of 9.53% on Flickr30k. Code: https://github.com/Previsior/PTM-ST.
Shengbin Guo, Hang Zhao, Senqiao Yang +5
Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong
Dataset distillation compresses large-scale datasets into compact synthetic sets while preserving training performance, but existing methods are largely restricted to single-modal or bimodal settings. Extending dataset distillation to scenarios involving more than two modalities, i.e., Omnimodal Dataset Distillation, remains underexplored and challenging due to increased heterogeneity and complex cross-modal interactions. In this work, we identify the key determinant that bounds the endpoint discrepancy in the omnimodal setting, which is exacerbated with an increasing number of modalities. To this end, we propose HoPA, a unified method that captures high-order cross-modal alignments via a compact proxy, which is compatible with trajectory matching as well. By abstracting omnimodal alignment with a shared similarity structure, our method avoids the combinatorial complexity of pairwise modality modeling and enables scalable joint distillation across heterogeneous modalities. Theoretical analysis from the spectral perspective reveals the rationality of our proposed method against bimodal dataset distillation techniques. Extensive experiments on various benchmarks demonstrate that the proposed method achieves superior compression-performance trade-offs compared to existing competitors. The source code will be publicly released.
Yuxuan Gao, Xiaohao Liu, Xiaobo Xia +1
University of Science and Technology of China · Xidian University · National University of Singapore +1