Multimodal dataset distillation (MDD) aims to reduce the cost of vision-language pre-training by synthesizing compact image-text datasets. Although recent feature-matching methods have improved distillation efficiency, synthetic data may become specialized to the model representations encountered during optimization, limiting generalization to unseen model states. We propose Model Distribution-Aware Multimodal Dataset Distillation (MDA-DD), which leverages supervision from a dynamic staggered queue of models at different training stages. We further introduce asymmetric dual projection for complementary alignment in visual and linguistic spaces and an uncentered second-order moment objective that retains mean-related cross-modal information. Experiments on Flickr30K and MS-COCO demonstrate consistent retrieval improvements over existing methods across synthetic-data budgets. With only 100 synthetic pairs, MDA-DD improves IR@1 and TR@1 on Flickr30K by 12.9 and 18.2 percentage points, respectively, over the strongest baseline at the same budget, outperforming all compared methods using 500 pairs on both datasets across all six retrieval metrics.
Figures & tables
Figure 1: Radar charts comparing MDA-DD against all baselines on Flickr30K across retrieval metrics.
Figure 2: (a) Joint t-SNE visualization of multimodal embeddings under a different model initialization. Circles denote image embeddings and triangles denote text embeddings; colors indicate the source ( yellow:real data, cyan:CovMatch, orange:MDA-DD ). The red dashed circle highlights a subset of CovMatch text embeddings that collapses to an isolated region. (b) Retrieval performance of CovMatch variants on Flickr30K with 100 pairs. Removing the image projection (yellow) has minimal impact relative to the baseline (grey) , while random model initialization (blue) yields consistent gains across all metrics.
Figure 3: Overview of MDA-DD. A dynamic staggered model queue maintains expert models at different training stages. (a) Update Data: Synthetic pairs are optimized by matching encoder-level second-order cross-modal moments (SOCM) and projection-level feature means with real pairs. (b) Update Model: Each expert is trained on real pairs using asymmetric dual projections for bidirectional contrastive learning.
Pairs
Ratio
Metric
Coreset selection
Dataset distillation
Rand
Herd
K-Cent
Forget
LoRS
CovMatch
RepBlend
PTM-ST
Ours
100
0.3%
IR@1
1.0
0.7
0.7
0.7
8.3 ± 0.2
10.1 ± 0.2
11.5 ± 0.4
9.6 ± 0.3
24.4 ± 0.6
IR@5
4.0
2.8
3.1
2.4
24.1 ± 0.2
28.6 ± 0.4
32.0 ± 0.7
28.4 ± 0.5
53.4 ± 0.7
IR@10
6.5
5.3
6.1
5.6
35.1 ± 0.3
40.9 ± 0.6
44.5 ± 0.6
41.5 ± 0.5
67.3 ± 0.7
TR@1
1.3
1.1
0.6
1.2
11.8 ± 0.2
14.8 ± 0.9
16.2 ± 0.8
14.4 ± 0.4
34.4 ± 0.7
TR@5
5.9
4.7
5.0
4.2
35.8 ± 0.6
38.0 ± 0.4
41.7 ± 0.9
38.6 ± 0.8
65.2 ± 0.9
Table 1: Cross-modal retrieval on Flickr30K (NFNet+BERT). The model trained on full dataset achieves: IR@1 = 47.8, IR@5 = 79.0, IR@10 = 87.4, TR@1 = 61.5, TR@5 = 85.1, TR@10 = 92.1.
Pairs
Ratio
Metric
Coreset selection
Dataset distillation
Rand
Herd
K-Cent
Forget
LoRS
CovMatch
RepBlend
PTM-ST
Ours
100
0.8‰
IR@1
0.3
0.5
0.4
0.3
1.8 ± 0.1
2.8 ± 0.1
4.1 ± 0.3
2.3 ± 0.1
7.3 ± 0.2
IR@5
1.3
1.4
1.4
1.5
7.1 ± 0.2
10.5 ± 0.2
13.9 ± 0.8
9.0 ± 0.1
22.6 ± 0.4
IR@10
2.7
3.5
2.5
2.5
12.2 ± 0.2
17.7 ± 0.3
22.3 ± 0.5
15.6 ± 0.3
34.4 ± 0.6
TR@1
0.8
0.8
1.4
0.7
3.3 ± 0.2
3.8 ± 0.1
5.2 ± 0.5
4.1 ± 0.2
10.5 ± 0.2
TR@5
3.0
2.1
3.7
2.6
12.2 ± 0.3
13.1 ± 0.3
17.9 ± 0.9
13.4 ± 0.2
28.6 ± 0.6
Table 2: Cross-modal retrieval on MS-COCO (NFNet+BERT). The model trained on full dataset achieves: IR@1 = 24.9, IR@5 = 53.7, IR@10 = 67.5, TR@1 = 32.5, TR@5 = 61.9, TR@10 = 74.0.
Dataset
Pairs
Method
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
mR
Flickr30K
5K
Real Data
26.6
57.4
69.6
34.3
64.3
76.5
54.8
CovMatch
24.0
53.9
64.9
32.1
61.7
73.3
51.7
MDA-DD
31.7
61.7
73.0
43.2
73.5
82.1
60.9
MS-COCO
5K
Real Data
11.9
32.2
45.4
15.3
38.3
51.0
32.4
CovMatch
10.8
29.7
42.8
15.1
36.8
50.4
30.9
MDA-DD
12.9
34.6
47.6
19.3
44.7
58.5
36.3
Table 3: Scaling performance on Flickr30K and MS-COCO with larger synthetic-data budgets (NFNet+BERT). The real-data baseline uses equally sized subsets sampled from the original training data. mR is the mean of six retrieval metrics (%).
Figure 4: Component-wise ablation on Flickr30K with NFNet+BERT. We compare centered covariance (COV) and SOCM under different configurations of asymmetric dual projection (ADP) and dynamic model queue (DQ).
Distillation
Downstream
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
mR
Original
Original
12.1
33.5
46.3
17.7
43.8
57.0
35.1
Original
ADP
15.3
39.1
52.1
18.3
49.3
62.5
39.4
ADP
Original
15.4
37.7
49.8
20.8
45.9
58.2
38.0
ADP
ADP
15.6
40.3
54.2
21.1
50.6
64.2
41.0
Table 4: Effects of asymmetric dual projection (ADP) during distillation and downstream training on Flickr30K with 500 synthetic pairs. Original denotes the CovMatch projection design. Results use the controlled ablation setting, and mR averages the six retrieval metrics.
Method
Expert
Distill
storage
time
time
RepBlend
16.6 GB
78.6 h
1.5 h
CovMatch
0
2.0 h
2.8 h
MDA-DD
0
2.0 h
1.3 h
Table 5: Computational cost on MS-COCO with 100 synthetic pairs (NVIDIA RTX 5090, 32 GB).
Backbone
Method
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet+BERT
LoRS
3.3 ± 0.2
12.7 ± 0.3
20.4 ± 0.2
6.8 ± 0.2
19.6 ± 1.3
31.1 ± 0.3
RepBlend
4.2 ± 0.2
14.1 ± 0.2
23.6 ± 0.6
8.4 ± 0.2
23.1 ± 0.8
35.0 ± 1.3
CovMatch
10.6 ± 0.8
29.0 ± 1.3
41.0 ± 1.2
13.4 ± 1.3
34.2 ± 0.9
46.2 ± 1.4
Ours
14.5 ± 2.2
36.4 ± 4.6
49.1 ± 5.2
17.8 ± 2.2
42.2 ± 4.5
54.1 ± 5.3
RegNet+BERT
LoRS
3.5 ± 0.1
12.6 ± 0.3
21.1 ± 0.4
6.8 ± 0.3
20.8 ± 0.3
30.2 ± 0.3
RepBlend
3.9 ± 0.2
13.9 ± 0.3
24.0 ± 0.6
7.9 ± 0.3
24.2 ± 0.3
36.2 ± 1.1
Table 6: Cross-modal retrieval performance under alternative image encoder architectures on Flickr30K with 500 synthetic pairs. Results are reported as the mean and standard deviation over 5 independent runs.
Pairs
Method
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
200
LoRS
1.56
5.33
8.77
1.01
4.04
6.81
RepBlend
3.53
12.38
19.33
4.39
13.45
20.18
CovMatch
2.04
6.95
12.31
2.67
9.46
14.93
MDA-DD
7.92
22.29
32.13
10.21
28.54
39.62
500
LoRS
1.96
6.72
10.55
1.41
5.11
8.51
RepBlend
4.45
15.09
22.31
5.14
14.89
22.97
Table 7: Cross-modal retrieval on LLaVA-CC3M with NFNet+BERT (%). The full-data reference achieves IR@1/5/10 = 23.77/50.95/62.84 and TR@1/5/10 = 24.66/50.82/63.58.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Flickr30K
MS-COCO
100
200
500
100
200
500
Scaling factor ρ
2
2
1
2
2
1
Feature matching weight λ
0.1
0.1
0.5
0.1
0.1
0.5
Batch size (real)
128
128
128
128
128
128
Batch size (synthetic)
–
–
200
–
–
200
Learning rate (image)
1
1
1
1
1
1
Appendix
Table 8: Hyperparameter configurations of MDA-DD on Flickr30K and MS-COCO under varying synthetic budget sizes.
Architecture
Model Variant
Parameter Count
NFNet [ 5 ]
nfnet_l0
26.4M
NF-ResNet [ 4 ]
nf_resnet50
25.6M
NF-RegNet [ 45 ]
nf_regnet_b1
21.0M
ViT [ 12 ]
vit_base_patch16_224
85.8M
BERT [ 10 ]
bert-base-uncased
110.0M
Appendix
Table 9: Network architectures and parameter counts.
Backbone
Method
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
ResNet+BERT
Random
2.2
8.0
13.4
3.0
10.1
16.4
MTT-VL
1.8
7.0
11.9
2.9
10.5
16.4
LoRS
1.6
7.0
12.1
2.9
11.1
18.0
RepBlend
1.2
5.3
9.7
4.1
11.2
18.9
CovMatch
4.7
15.4
23.4
5.5
17.3
26.6
Ours
13.3
35.4
47.3
13.6
33.8
47.0
Appendix
Table 10: Cross-architecture evaluation on Flickr30K with N=100 synthetic pairs. All methods are distilled with NFNet+BERT and evaluated on unseen image encoders.
Figure 5: Effect of queue size K on cross-modal retrieval performance (Flickr30K, 100 synthetic pairs). Performance improves steadily up to K=8 and saturates beyond K=10 .
Figure 6: Effect of model lifetime Tmax on cross-modal retrieval performance (Flickr30K, 100 synthetic pairs). The optimum lies around Tmax=200 – 250 .
Dataset
Pairs
Ratio
Method
IR@1
IR@5
IR@10
TR@1
TR@5
TR@10
Flickr30K
50
0.2%
RepBlend
7.5 ± 0.5
24.8 ± 1.1
36.0 ± 1.7
12.5 ± 0.5
34.7 ± 0.7
48.4 ± 0.8
CovMatch
7.8 ± 0.2
23.8 ± 0.5
35.0 ± 0.3
11.5 ± 0.3
31.8 ± 1.3
44.2 ± 1.1
Ours
20.5 ± 1.0
49.4 ± 0.8
63.8 ± 0.6
30.6 ± 0.5
60.0 ± 0.7
71.8 ± 1.0
MS-COCO
50
0.4‰
RepBlend
3.0 ± 0.2
10.9 ± 0.4
18.6 ± 0.6
4.4 ± 0.2
15.6 ± 0.3
24.7 ± 0.4
CovMatch
2.3 ± 0.1
8.6 ± 0.2
14.8 ± 0.4
2.8 ± 0.2
10.3 ± 0.4
16.8 ± 0.6
Ours
5.0 ± 0.4
16.5 ± 0.8
26.0 ± 1.1
7.7 ± 0.3
22.7 ± 0.9
33.8 ± 1.0
Appendix
Table 11: Cross-modal retrieval performance with 50 synthetic pairs (0.2% of Flickr30K, 0.4‰ of MS-COCO). Results are reported as mean ± standard deviation over 5 independent runs.