Multimodal dataset distillation (MDD) aims to reduce the cost of vision-language pre-training by synthesizing compact image-text datasets. Although recent feature-matching methods have improved distillation efficiency, synthetic data may become specialized to the model representations encountered during optimization, limiting generalization to unseen model states. We propose Model Distribution-Aware Multimodal Dataset Distillation (MDA-DD), which leverages supervision from a dynamic staggered queue of models at different training stages. We further introduce asymmetric dual projection for complementary alignment in visual and linguistic spaces and an uncentered second-order moment objective that retains mean-related cross-modal information. Experiments on Flickr30K and MS-COCO demonstrate consistent retrieval improvements over existing methods across synthetic-data budgets. With only 100 synthetic pairs, MDA-DD improves IR@1 and TR@1 on Flickr30K by 12.9 and 18.2 percentage points, respectively, over the strongest baseline at the same budget, outperforming all compared methods using 500 pairs on both datasets across all six retrieval metrics.
Figures & tables
Figure 1: Radar charts comparing MDA-DD against all baselines on Flickr30K across retrieval metrics.
Figure 2: (a) Joint t-SNE visualization of multimodal embeddings under a different model initialization. Circles denote image embeddings and triangles denote text embeddings; colors indicate the source ( yellow:real data, cyan:CovMatch, orange:MDA-DD ). The red dashed circle highlights a subset of CovMatch text embeddings that collapses to an isolated region. (b) Retrieval performance of CovMatch variants on Flickr30K with 100 pairs. Removing the image projection (yellow) has minimal impact relative to the baseline (grey) , while random model initialization (blue) yields consistent gains across all metrics.
Figure 3: Overview of MDA-DD. A dynamic staggered model queue maintains expert models at different training stages. (a) Update Data: Synthetic pairs are optimized by matching encoder-level second-order cross-modal moments (SOCM) and projection-level feature means with real pairs. (b) Update Model: Each expert is trained on real pairs using asymmetric dual projections for bidirectional contrastive learning.
Pairs
Ratio
Metric
Coreset selection
Dataset distillation
Rand
Herd
K-Cent
Forget
LoRS
CovMatch
RepBlend
PTM-ST
Ours
100
0.3%
IR@1
1.0
0.7
0.7
0.7
8.3 ± 0.2
10.1 ± 0.2
11.5 ± 0.4
9.6 ± 0.3
24.4 ± 0.6
IR@5
4.0
2.8
3.1
2.4
24.1 ± 0.2
28.6 ± 0.4
32.0 ± 0.7
28.4 ± 0.5
53.4 ± 0.7
IR@10
6.5
5.3
6.1
5.6
35.1 ± 0.3
40.9 ± 0.6
44.5 ± 0.6
41.5 ± 0.5
67.3 ± 0.7
TR@1
1.3
1.1
0.6
1.2
11.8 ± 0.2
14.8 ± 0.9
16.2 ± 0.8
14.4 ± 0.4
34.4 ± 0.7
TR@5
5.9
4.7
5.0
4.2
35.8 ± 0.6
38.0 ± 0.4
41.7 ± 0.9
38.6 ± 0.8
65.2 ± 0.9
Table 1: Cross-modal retrieval on Flickr30K (NFNet+BERT). The model trained on full dataset achieves: IR@1 = 47.8, IR@5 = 79.0, IR@10 = 87.4, TR@1 = 61.5, TR@5 = 85.1, TR@10 = 92.1.
Pairs
Ratio
Metric
Coreset selection
Dataset distillation
Rand
Herd
K-Cent
Forget
LoRS
CovMatch
RepBlend
PTM-ST
Ours
100
0.8‰
IR@1
0.3
0.5
0.4
0.3
1.8 ± 0.1
2.8 ± 0.1
4.1 ± 0.3
2.3 ± 0.1
7.3 ± 0.2
IR@5
1.3
1.4
1.4
1.5
7.1 ± 0.2
10.5 ± 0.2
13.9 ± 0.8
9.0 ± 0.1
22.6 ± 0.4
IR@10
2.7
3.5
2.5
2.5
12.2 ± 0.2
17.7 ± 0.3
22.3 ± 0.5
15.6 ± 0.3
34.4 ± 0.6
TR@1
0.8
0.8
1.4
0.7
3.3 ± 0.2
3.8 ± 0.1
5.2 ± 0.5
4.1 ± 0.2
10.5 ± 0.2
TR@5
3.0
2.1
3.7
2.6
12.2 ± 0.3
13.1 ± 0.3
17.9 ± 0.9
13.4 ± 0.2
28.6 ± 0.6
Table 2: Cross-modal retrieval on MS-COCO (NFNet+BERT). The model trained on full dataset achieves: IR@1 = 24.9, IR@5 = 53.7, IR@10 = 67.5, TR@1 = 32.5, TR@5 = 61.9, TR@10 = 74.0.
Dataset
Pairs
Method
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
mR
Flickr30K
5K
Real Data
26.6
57.4
69.6
34.3
64.3
76.5
54.8
CovMatch
24.0
53.9
64.9
32.1
61.7
73.3
51.7
MDA-DD
31.7
61.7
73.0
43.2
73.5
82.1
60.9
MS-COCO
5K
Real Data
11.9
32.2
45.4
15.3
38.3
51.0
32.4
CovMatch
10.8
29.7
42.8
15.1
36.8
50.4
30.9
MDA-DD
12.9
34.6
47.6
19.3
44.7
58.5
36.3
Table 3: Scaling performance on Flickr30K and MS-COCO with larger synthetic-data budgets (NFNet+BERT). The real-data baseline uses equally sized subsets sampled from the original training data. mR is the mean of six retrieval metrics (%).
Figure 4: Component-wise ablation on Flickr30K with NFNet+BERT. We compare centered covariance (COV) and SOCM under different configurations of asymmetric dual projection (ADP) and dynamic model queue (DQ).
Distillation
Downstream
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
mR
Original
Original
12.1
33.5
46.3
17.7
43.8
57.0
35.1
Original
ADP
15.3
39.1
52.1
18.3
49.3
62.5
39.4
ADP
Original
15.4
37.7
49.8
20.8
45.9
58.2
38.0
ADP
ADP
15.6
40.3
54.2
21.1
50.6
64.2
41.0
Table 4: Effects of asymmetric dual projection (ADP) during distillation and downstream training on Flickr30K with 500 synthetic pairs. Original denotes the CovMatch projection design. Results use the controlled ablation setting, and mR averages the six retrieval metrics.
Method
Expert
Distill
storage
time
time
RepBlend
16.6 GB
78.6 h
1.5 h
CovMatch
0
2.0 h
2.8 h
MDA-DD
0
2.0 h
1.3 h
Table 5: Computational cost on MS-COCO with 100 synthetic pairs (NVIDIA RTX 5090, 32 GB).
Backbone
Method
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet+BERT
LoRS
3.3 ± 0.2
12.7 ± 0.3
20.4 ± 0.2
6.8 ± 0.2
19.6 ± 1.3
31.1 ± 0.3
RepBlend
4.2 ± 0.2
14.1 ± 0.2
23.6 ± 0.6
8.4 ± 0.2
23.1 ± 0.8
35.0 ± 1.3
CovMatch
10.6 ± 0.8
29.0 ± 1.3
41.0 ± 1.2
13.4 ± 1.3
34.2 ± 0.9
46.2 ± 1.4
Ours
14.5 ± 2.2
36.4 ± 4.6
49.1 ± 5.2
17.8 ± 2.2
42.2 ± 4.5
54.1 ± 5.3
RegNet+BERT
LoRS
3.5 ± 0.1
12.6 ± 0.3
21.1 ± 0.4
6.8 ± 0.3
20.8 ± 0.3
30.2 ± 0.3
RepBlend
3.9 ± 0.2
13.9 ± 0.3
24.0 ± 0.6
7.9 ± 0.3
24.2 ± 0.3
36.2 ± 1.1
Table 6: Cross-modal retrieval performance under alternative image encoder architectures on Flickr30K with 500 synthetic pairs. Results are reported as the mean and standard deviation over 5 independent runs.
Pairs
Method
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
200
LoRS
1.56
5.33
8.77
1.01
4.04
6.81
RepBlend
3.53
12.38
19.33
4.39
13.45
20.18
CovMatch
2.04
6.95
12.31
2.67
9.46
14.93
MDA-DD
7.92
22.29
32.13
10.21
28.54
39.62
500
LoRS
1.96
6.72
10.55
1.41
5.11
8.51
RepBlend
4.45
15.09
22.31
5.14
14.89
22.97
Table 7: Cross-modal retrieval on LLaVA-CC3M with NFNet+BERT (%). The full-data reference achieves IR@1/5/10 = 23.77/50.95/62.84 and TR@1/5/10 = 24.66/50.82/63.58.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Flickr30K
MS-COCO
100
200
500
100
200
500
Scaling factor ρ
2
2
1
2
2
1
Feature matching weight λ
0.1
0.1
0.5
0.1
0.1
0.5
Batch size (real)
128
128
128
128
128
128
Batch size (synthetic)
–
–
200
–
–
200
Learning rate (image)
1
1
1
1
1
1
Appendix
Table 8: Hyperparameter configurations of MDA-DD on Flickr30K and MS-COCO under varying synthetic budget sizes.
Architecture
Model Variant
Parameter Count
NFNet [ 5 ]
nfnet_l0
26.4M
NF-ResNet [ 4 ]
nf_resnet50
25.6M
NF-RegNet [ 45 ]
nf_regnet_b1
21.0M
ViT [ 12 ]
vit_base_patch16_224
85.8M
BERT [ 10 ]
bert-base-uncased
110.0M
Appendix
Table 9: Network architectures and parameter counts.
Backbone
Method
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet+BERT
Random
2.2
8.0
13.4
3.0
10.1
16.4
MTT-VL
1.8
7.0
11.9
2.9
10.5
16.4
LoRS
1.6
7.0
12.1
2.9
11.1
18.0
RepBlend
1.2
5.3
9.7
4.1
11.2
18.9
CovMatch
4.7
15.4
23.4
5.5
17.3
26.6
Ours
13.3
35.4
47.3
13.6
33.8
47.0
Appendix
Table 10: Cross-architecture evaluation on Flickr30K with N=100 synthetic pairs. All methods are distilled with NFNet+BERT and evaluated on unseen image encoders.
Figure 5: Effect of queue size K on cross-modal retrieval performance (Flickr30K, 100 synthetic pairs). Performance improves steadily up to K=8 and saturates beyond K=10 .
Figure 6: Effect of model lifetime Tmax on cross-modal retrieval performance (Flickr30K, 100 synthetic pairs). The optimum lies around Tmax=200 – 250 .
Dataset
Pairs
Ratio
Method
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
Flickr30K
50
0.2%
RepBlend
7.5 ± 0.5
24.8 ± 1.1
36.0 ± 1.7
12.5 ± 0.5
34.7 ± 0.7
48.4 ± 0.8
CovMatch
7.8 ± 0.2
23.8 ± 0.5
35.0 ± 0.3
11.5 ± 0.3
31.8 ± 1.3
44.2 ± 1.1
Ours
20.5 ± 1.0
49.4 ± 0.8
63.8 ± 0.6
30.6 ± 0.5
60.0 ± 0.7
71.8 ± 1.0
MS-COCO
50
0.4‰
RepBlend
3.0 ± 0.2
10.9 ± 0.4
18.6 ± 0.6
4.4 ± 0.2
15.6 ± 0.3
24.7 ± 0.4
CovMatch
2.3 ± 0.1
8.6 ± 0.2
14.8 ± 0.4
2.8 ± 0.2
10.3 ± 0.4
16.8 ± 0.6
Ours
5.0 ± 0.4
16.5 ± 0.8
26.0 ± 1.1
7.7 ± 0.3
22.7 ± 0.9
33.8 ± 1.0
Appendix
Table 11: Cross-modal retrieval performance with 50 synthetic pairs (0.2% of Flickr30K, 0.4‰ of MS-COCO). Results are reported as mean ± standard deviation over 5 independent runs.
Dataset distillation compresses large training sets into compact synthetic datasets while preserving downstream performance. As modern systems increasingly operate on paired vision-language inputs, multimodal distillation must preserve representation quality and cross-modal alignment under tight compute and memory budgets, yet prior methods often require heavy computes and overlook their correlations. To address this, we present Multimodal Distribution Matching (MDM), a geometry-aware framework for efficient and generalizable multimodal distillation. Specifically, MDM integrates complementary components at the data, model, and loss levels. At the data level, it initializes synthetic image-text pairs by sampling from clusters in the joint embedding space. At the model level, it forms a mixed teacher by interpolating independently fine-tuned models in weight space according to their angular deviation from the pretrained anchor. At the loss level, it matches joint distributions on the unit hypersphere using a geometry-aware matching objective that exploits the joint features in the cross-modal agreement and discrepancy directions along with symmetric contrastive learning. Across image-text retrieval benchmarks with cross-architecture evaluation, MDM yields compact synthetic sets that preserve multimodal semantics, substantially reduce distillation cost, and remain robust across architectures.
Recent advances in multimodal learning have achieved remarkable success across diverse vision-language tasks. However, such progress heavily relies on large-scale image-text datasets, making training costly and inefficient. Prior efforts in dataset filtering and pruning attempt to mitigate this issue, but still require relatively large subsets to maintain performance and fail under very small subsets. Dataset distillation offers a promising alternative, yet existing multimodal dataset distillation methods require full-dataset training and joint optimization of image pixels and text features, making them architecture-dependent and limiting cross-architecture generalization. To overcome this, we propose a learning-free dataset distillation framework that eliminates the need for large-scale training and optimization while enhancing generalization across architectures. Our method uses CLIP to extract aligned image-text embeddings, obtains prototypes, and employs an unCLIP decoder to synthesize images, enabling efficient and scalable multimodal dataset distillation. Extensive experiments demonstrate that our approach consistently outperforms optimization-based dataset distillation and subset selection methods, achieving state-of-the-art cross-architecture generalization.
Junhyeok Choi, Sangwoo Mo, Minwoo Chae
Department of Industrial and Management Engineering, Pohang University of Science and Technology
Multimodal dataset distillation aims to construct compact synthetic datasets that enable efficient compression and knowledge transfer from large-scale image-text data. However, existing approaches often fail to capture the complex, dynamically evolving knowledge embedded in the later training stages of teacher models. This limitation leads to degraded student performance and compromises the quality of the distilled data. To address critical challenges such as pronounced cross-stage performance gaps and unstable teacher trajectories, we propose Phased Teacher Model with Shortcut Trajectory (PTM-ST) -- a novel phased distillation framework. PTM-ST leverages stage-aware teacher modeling and a shortcut-based trajectory construction strategy to accurately fit the teacher's learning dynamics across distinct training phases. This enhances both the stability and expressiveness of the distillation process. Through theoretical analysis and comprehensive experiments, we show that PTM-ST significantly mitigates optimization oscillations and inter-phase knowledge gaps, while also reducing storage overhead. Our method consistently surpasses state-of-the-art baselines on Flickr30k and COCO, achieving up to 13.5% absolute improvement and an average gain of 9.53% on Flickr30k. Code: https://github.com/Previsior/PTM-ST.
Shengbin Guo, Hang Zhao, Senqiao Yang +5
Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong