Dataset distillation compresses large-scale datasets into compact synthetic sets while preserving training performance, but existing methods are largely restricted to single-modal or bimodal settings. Extending dataset distillation to scenarios involving more than two modalities, i.e., Omnimodal Dataset Distillation, remains underexplored and challenging due to increased heterogeneity and complex cross-modal interactions. In this work, we identify the key determinant that bounds the endpoint discrepancy in the omnimodal setting, which is exacerbated with an increasing number of modalities. To this end, we propose HoPA, a unified method that captures high-order cross-modal alignments via a compact proxy, which is compatible with trajectory matching as well. By abstracting omnimodal alignment with a shared similarity structure, our method avoids the combinatorial complexity of pairwise modality modeling and enables scalable joint distillation across heterogeneous modalities. Theoretical analysis from the spectral perspective reveals the rationality of our proposed method against bimodal dataset distillation techniques. Extensive experiments on various benchmarks demonstrate that the proposed method achieves superior compression-performance trade-offs compared to existing competitors. The source code will be publicly released.
Figures & tables
Figure 1: The overall framework of omnimodal dataset distillation beyond bimodal data. Different modal contents are encoded into multimodal representations ( left ). We decompose these representations via SVD, rather than combining pairwise modeling, to handle the increased heterogeneity among modalities. We maximize the leading singular value σ1 and utilize the principal right singular vector v1 as a compact low-rank proxy to guide the alignment ( middle ). We distill the dataset into the synthetic set by matching the training trajectories. A shared similarity matrix Se is jointly learned to preserve the multimodal alignment within the distilled data ( right ).
Triplets (Ratio)
Methods
R@1 ↑
R@5 ↑
Time (mins) ↓
V → T
T → V
V → A
A → V
T → A
A → T
Avg.
V → T
T → V
V → A
A → V
T → A
A → T
Avg.
100 (1.11%)
Random
29.1 ± 0.5
37.2 ± 0.0
22.5 ± 0.1
19.9 ± 0.0
6.9 ± 0.1
6.0 ± 0.1
20.3
54.1 ± 0.2
60.7 ± 0.2
43.8 ± 0.1
40.0 ± 0.1
17.8 ± 0.1
17.3 ± 0.0
39.0
-
MTT-VL
34.3 ± 0.6
36.7 ± 0.8
22.4 ± 0.1
19.1 ± 0.1
6.7 ± 0.5
5.8 ± 0.4
20.8
59.0 ± 0.2
58.9 ± 1.2
43.3 ± 0.1
39.3 ± 0.2
18.6 ± 0.6
18.4 ± 0.7
39.6
75.7
TESLA
34.0 ± 0.5
36.5 ± 0.8
22.4 ± 0.1
19.2 ± 0.1
6.7 ± 0.3
5.8 ± 0.1
20.8
58.8 ± 0.6
58.8 ± 1.0
43.1 ± 0.2
39.2 ± 0.1
18.4 ± 0.2
17.6 ± 0.4
39.3
75.4
LoRS
36.1 ± 0.8
36.6 ± 0.3
21.8 ± 0.1
20.5 ± 0.6
7.8 ± 0.1
6.8 ± 0.5
21.6
60.0 ± 0.4
60.4 ± 1.1
43.3 ± 0.5
41.7 ± 1.1
19.8 ± 0.6
19.6 ± 0.4
40.8
77.1
RepBlend
36.2 ± 0.7
36.7 ± 0.7
21.7 ± 0.4
19.8 ± 0.9
7.4 ± 0.3
6.6 ± 0.4
21.4
60.7 ± 0.6
60.0 ± 1.2
43.3 ± 0.1
40.6 ± 0.8
19.7 ± 0.5
18.6 ± 0.3
40.5
82.0
Table 1: Experimental results on MSR-VTT. This table shows the testing retrieval recall of models trained from scratch on the small coreset or synthetic set. “V”: Video, “T”: Text, and “A”: Audio. We report the mean and standard deviation of the results. In each case, the best performance is shown in bold.
Triplets (Ratio)
Methods
R@1 ↑
R@5 ↑
Time (mins) ↓
V → T
T → V
V → A
A → V
T → A
A → T
Avg.
V → T
T → V
V → A
A → V
T → A
A → T
Avg.
100 (1.25%)
Random
5.7 ± 0.5
7.7 ± 0.0
26.3 ± 0.0
25.7 ± 0.0
5.6 ± 0.0
3.7 ± 0.3
12.4
25.9 ± 0.3
28.3 ± 0.1
54.3 ± 0.1
52.1 ± 0.1
20.5 ± 0.1
18.6 ± 0.2
33.3
-
MTT-VL
6.2 ± 0.0
7.6 ± 0.0
26.1 ± 0.2
25.8 ± 0.0
5.9 ± 0.2
4.3 ± 0.4
12.7
26.1 ± 0.0
27.5 ± 0.0
53.7 ± 0.4
52.5 ± 0.1
20.4 ± 0.3
18.8 ± 0.2
33.2
80.9
TESLA
5.9 ± 0.3
7.6 ± 0.2
26.1 ± 0.2
25.8 ± 0.1
6.2 ± 0.3
3.8 ± 0.1
12.6
25.6 ± 0.6
27.4 ± 0.4
53.8 ± 0.4
52.6 ± 0.1
22.0 ± 0.2
20.5 ± 0.7
33.7
81.5
LoRS
5.4 ± 0.5
7.8 ± 0.2
27.8 ± 0.3
26.2 ± 0.1
6.3 ± 0.2
4.5 ± 0.2
13.0
26.5 ± 0.6
28.2 ± 0.1
53.1 ± 0.4
52.2 ± 0.3
22.3 ± 0.4
21.6 ± 0.4
34.0
84.9
RepBlend
5.6 ± 0.5
7.6 ± 0.3
27.2 ± 0.1
25.0 ± 0.1
6.1 ± 0.3
4.5 ± 0.1
12.7
27.1 ± 0.4
28.6 ± 0.3
54.1 ± 0.1
51.4 ± 0.3
20.9 ± 0.1
19.9 ± 0.6
33.7
81.9
Table 2: Experimental results on VGGSound-S . This table shows the testing retrieval recall of models trained from scratch on the small coreset or synthetic set. “V”: Video, “T”: Text, and “A”: Audio. We report the mean and standard deviation of the results. In each case, the best performance is shown in bold.
Triplets (Ratio)
Methods
R@1 ↑
R@5 ↑
Time (mins) ↓
V → T
T → V
V → A
A → V
T → A
A → T
Avg.
V → T
T → V
V → A
A → V
T → A
A → T
Avg.
100 (1.19%)
Random
27.4 ± 0.1
27.7 ± 0.1
17.4 ± 0.1
15.5 ± 0.1
4.4 ± 0.1
3.8 ± 0.1
16.0
50.3 ± 0.1
54.1 ± 0.1
36.6 ± 0.1
36.2 ± 0.1
13.8 ± 0.1
13.7 ± 0.0
34.1
-
MTT-VL
26.6 ± 0.1
26.0 ± 0.4
15.8 ± 0.6
18.8 ± 0.5
4.4 ± 0.3
4.4 ± 0.4
16.0
51.5 ± 0.4
50.7 ± 1.5
37.8 ± 0.3
41.6 ± 0.5
13.4 ± 0.8
14.0 ± 1.0
34.8
77.3
TESLA
26.6 ± 0.1
26.0 ± 0.4
16.2 ± 0.1
18.9 ± 0.6
4.4 ± 0.4
4.2 ± 0.3
16.1
51.5 ± 0.4
50.7 ± 1.5
37.9 ± 0.3
41.8 ± 0.5
13.5 ± 1.1
14.6 ± 0.1
35.0
78.1
LoRS
28.0 ± 0.5
28.4 ± 0.5
17.9 ± 0.2
18.0 ± 0.8
4.3 ± 0.1
4.7 ± 0.4
16.9
52.7 ± 0.3
54.1 ± 0.7
40.2 ± 1.2
42.2 ± 0.6
13.3 ± 0.5
13.4 ± 1.0
36.0
79.4
RepBlend
27.7 ± 0.5
28.5 ± 0.7
18.0 ± 0.2
18.2 ± 0.9
4.3 ± 0.1
4.7 ± 0.5
16.9
52.9 ± 0.0
54.3 ± 0.8
40.8 ± 0.7
41.9 ± 0.1
13.4 ± 0.6
13.5 ± 1.3
36.1
78.2
Table 3: Experimental results on DiDeMo. This table shows the testing retrieval recall of models trained from scratch on the small coreset or synthetic set. “V”: Video, “T”: Text, and “A”: Audio. We report the mean and standard deviation of the results. In each case, the best performance is shown in bold.
Model
Setting
R@1
R@5
R@10
ImageBind
Zero-shot
12.25
32.87
44.32
Distilled
13.15
35.33
47.34
LanguageBind
Zero-shot
4.91
19.94
29.56
Distilled
5.93
22.11
33.32
OmniBind
Zero-shot
12.92
34.88
48.03
Distilled
13.13
35.63
48.87
Table 4: Cross architecture generalization results under ‘Zero-shot’ and ‘Distilled’ settings.
Datasets
Queries
Avg. R@1
Avg. R@5
HoPA
3-pair
T-bind
V-bind
HoPA
3-pair
T-bind
V-bind
MSR-VTT
100
21.78
20.21
20.42
21.05
42.43
42.08
41.45
41.97
200
21.93
19.94
20.49
19.24
42.73
42.40
41.35
42.28
500
22.25
18.11
20.19
19.85
42.96
41.60
38.65
41.86
VGGSound-S
100
13.19
12.78
12.10
12.92
34.89
35.20
34.01
35.00
200
13.15
12.31
11.30
12.74
35.51
34.90
33.76
35.51
Table 5: Empirical comparisons Pairwise distillation and omniwise distillation. We compare our omniwise distillation strategy with three pair-wise variants on MSR-VTT, VGGSound-S, and DiDeMo. In each case, the best performance is shown in bold.
Figure 2: Ablation study of each learning objective across all datasets and different numbers of queries.
Dataset
Triplets
R@1
R@5
R@10
Rank-1
Rank-2
Rank-1
Rank-2
Rank-1
Rank-2
MSR-VTT
100
21.78
20.80
42.43
39.99
52.28
49.77
200
21.93
21.09
42.73
39.81
52.61
49.37
500
22.25
20.48
42.96
39.78
52.96
49.57
VGGSound-S
100
13.19
12.18
34.89
32.85
46.72
44.13
200
13.15
12.25
35.33
32.58
47.34
43.72
Table 6: Effect of the proxy rank on distillation performance. We compare Rank-1 (Ours) against Rank-2 across different metrics. In each case, the best result is shown in bold. As can be seen, Rank-1 consistently outperforms Rank-2 .
Stage
w/o SVD
with SVD
SVD Cost
Buffer Training (min/traj)
10.85
11.25
+3.5%
Distillation (sec/100 iters)
53.28
87.48
+39.1%
Table 7: Computational efficiency analysis by comparing the runtime with and without the SVD on MSR-VTT. The distillation time is measured per 100 iterations.
Figure 3: The visualization of distilled omnimodal data on ImageBind with the VGGSound-S dataset. Comparison between samples before and after distillation, shown with one representative video frame, one audio map, and the corresponding text ( left ). Illustrative cases, each visualized with three video frames, three audio maps, and the corresponding text description ( right ).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
MSR-VTT
VGGSound-S
DiDeMo
epochs
10
10
10
num_experts
20
20
20
batch_size
128
128
128
lr_teacher
0.01
0.01
0.01
τ
0.1
0.1
0.1
τ′
0.2
0.2
0.2
Appendix
Table 8: Hyperparameter settings for buffer.
Dataset
MSR-VTT
VGGSound-S
DiDeMo
#instances
100
200
500
100
200
500
100
200
500
lr_data
100
100
100
100
100
100
100
100
100
lr_lr
1e-4
1e-4
1e-4
1e-4
1e-4
1e-4
1e-4
1e-4
1e-4
lr_sim
10.0
10.0
100.0
10.0
10.0
100.0
10.0
10.0
100.0
lr_teacher
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
sim_rank
10
20
20
10
20
40
10
20
20
Appendix
Table 9: Hyperparameter settings for dataset distillation.
Dataset
Triplets
Method Variant
R@1
R@5
R@10
MSR-VTT
100
Ours (Full Model)
21.78
42.43
52.28
w/o Similarity Mining
20.67
40.30
51.02
w/o Instance Loss ( LwBCE )
20.16
38.75
49.42
w/o Modality Loss ( LM )
20.08
37.96
48.03
200
Ours (Full Model)
21.93
42.73
52.61
w/o Similarity Mining
21.02
41.18
51.92
Appendix
Table 10: Complete ablation study results. We report Recall@1, Recall@5, and Recall@10 (%) on MSR-VTT, VGGSound-S, and DiDeMo across varying synthetic sizes. The best results in each setting are marked in bold .
Figure 4: Examples of distilled instances from the VGGSound-S dataset with N=100 triplets.
Recent advances in multimodal learning have achieved remarkable success across diverse vision-language tasks. However, such progress heavily relies on large-scale image-text datasets, making training costly and inefficient. Prior efforts in dataset filtering and pruning attempt to mitigate this issue, but still require relatively large subsets to maintain performance and fail under very small subsets. Dataset distillation offers a promising alternative, yet existing multimodal dataset distillation methods require full-dataset training and joint optimization of image pixels and text features, making them architecture-dependent and limiting cross-architecture generalization. To overcome this, we propose a learning-free dataset distillation framework that eliminates the need for large-scale training and optimization while enhancing generalization across architectures. Our method uses CLIP to extract aligned image-text embeddings, obtains prototypes, and employs an unCLIP decoder to synthesize images, enabling efficient and scalable multimodal dataset distillation. Extensive experiments demonstrate that our approach consistently outperforms optimization-based dataset distillation and subset selection methods, achieving state-of-the-art cross-architecture generalization.
Junhyeok Choi, Sangwoo Mo, Minwoo Chae
Department of Industrial and Management Engineering, Pohang University of Science and Technology
Dataset distillation compresses large training sets into compact synthetic datasets while preserving downstream performance. As modern systems increasingly operate on paired vision-language inputs, multimodal distillation must preserve representation quality and cross-modal alignment under tight compute and memory budgets, yet prior methods often require heavy computes and overlook their correlations. To address this, we present Multimodal Distribution Matching (MDM), a geometry-aware framework for efficient and generalizable multimodal distillation. Specifically, MDM integrates complementary components at the data, model, and loss levels. At the data level, it initializes synthetic image-text pairs by sampling from clusters in the joint embedding space. At the model level, it forms a mixed teacher by interpolating independently fine-tuned models in weight space according to their angular deviation from the pretrained anchor. At the loss level, it matches joint distributions on the unit hypersphere using a geometry-aware matching objective that exploits the joint features in the cross-modal agreement and discrepancy directions along with symmetric contrastive learning. Across image-text retrieval benchmarks with cross-architecture evaluation, MDM yields compact synthetic sets that preserve multimodal semantics, substantially reduce distillation cost, and remain robust across architectures.
Multimodal dataset distillation aims to construct compact synthetic datasets that enable efficient compression and knowledge transfer from large-scale image-text data. However, existing approaches often fail to capture the complex, dynamically evolving knowledge embedded in the later training stages of teacher models. This limitation leads to degraded student performance and compromises the quality of the distilled data. To address critical challenges such as pronounced cross-stage performance gaps and unstable teacher trajectories, we propose Phased Teacher Model with Shortcut Trajectory (PTM-ST) -- a novel phased distillation framework. PTM-ST leverages stage-aware teacher modeling and a shortcut-based trajectory construction strategy to accurately fit the teacher's learning dynamics across distinct training phases. This enhances both the stability and expressiveness of the distillation process. Through theoretical analysis and comprehensive experiments, we show that PTM-ST significantly mitigates optimization oscillations and inter-phase knowledge gaps, while also reducing storage overhead. Our method consistently surpasses state-of-the-art baselines on Flickr30k and COCO, achieving up to 13.5% absolute improvement and an average gain of 9.53% on Flickr30k. Code: https://github.com/Previsior/PTM-ST.
Shengbin Guo, Hang Zhao, Senqiao Yang +5
Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong