cs.CVApr 12, 2026
SaveOmnimodal Dataset Distillation via High-order Proxy Alignment
Organizations: University of Science and Technology of China · Xidian University · National University of Singapore · The University of Sydney
Abstract
Dataset distillation compresses large-scale datasets into compact synthetic sets while preserving training performance, but existing methods are largely restricted to single-modal or bimodal settings. Extending dataset distillation to scenarios involving more than two modalities, i.e., Omnimodal Dataset Distillation, remains underexplored and challenging due to increased heterogeneity and complex cross-modal interactions. In this work, we identify the key determinant that bounds the endpoint discrepancy in the omnimodal setting, which is exacerbated with an increasing number of modalities. To this end, we propose HoPA, a unified method that captures high-order cross-modal alignments via a compact proxy, which is compatible with trajectory matching as well. By abstracting omnimodal alignment with a shared similarity structure, our method avoids the combinatorial complexity of pairwise modality modeling and enables scalable joint distillation across heterogeneous modalities. Theoretical analysis from the spectral perspective reveals the rationality of our proposed method against bimodal dataset distillation techniques. Extensive experiments on various benchmarks demonstrate that the proposed method achieves superior compression-performance trade-offs compared to existing competitors. The source code will be publicly released.
Figures & tables
Figure 1: The overall framework of omnimodal dataset distillation beyond bimodal data. Different modal contents are encoded into multimodal representations ( left ). We decompose these representations via SVD, rather than combining pairwise modeling, to handle the increased heterogeneity among modalities. We maximize the leading singular value and utilize the principal right singular vector as a compact low-rank proxy to guide the alignment ( middle ). We distill the dataset into the synthetic set by matching the training trajectories. A shared similarity matrix is jointly learned to preserve the multimodal alignment within the distilled data ( right ).
| Triplets (Ratio) | Methods | R@1 | R@5 | Time (mins) | ||||||||||||
| V T | T V | V A | A V | T A | A T | Avg. | V T | T V | V A | A V | T A | A T | Avg. | |||
| 100 (1.11%) | Random | 29.1 0.5 | 37.2 0.0 | 22.5 0.1 | 19.9 0.0 | 6.9 0.1 | 6.0 0.1 | 20.3 | 54.1 0.2 | 60.7 0.2 | 43.8 0.1 | 40.0 0.1 | 17.8 0.1 | 17.3 0.0 | 39.0 | - |
| MTT-VL | 34.3 0.6 | 36.7 0.8 | 22.4 0.1 | 19.1 0.1 | 6.7 0.5 | 5.8 0.4 | 20.8 | 59.0 0.2 | 58.9 1.2 | 43.3 0.1 | 39.3 0.2 | 18.6 0.6 | 18.4 0.7 | 39.6 | 75.7 | |
| TESLA | 34.0 0.5 | 36.5 0.8 | 22.4 0.1 | 19.2 0.1 | 6.7 0.3 | 5.8 0.1 | 20.8 | 58.8 0.6 | 58.8 1.0 | 43.1 0.2 | 39.2 0.1 | 18.4 0.2 | 17.6 0.4 | 39.3 | 75.4 | |
| LoRS | 36.1 0.8 | 36.6 0.3 | 21.8 0.1 | 20.5 0.6 | 7.8 0.1 | 6.8 0.5 | 21.6 | 60.0 0.4 | 60.4 1.1 | 43.3 0.5 | 41.7 1.1 | 19.8 0.6 | 19.6 0.4 | 40.8 | 77.1 | |
| RepBlend | 36.2 0.7 | 36.7 0.7 | 21.7 0.4 | 19.8 0.9 | 7.4 0.3 | 6.6 0.4 | 21.4 | 60.7 0.6 | 60.0 1.2 | 43.3 0.1 | 40.6 0.8 | 19.7 0.5 | 18.6 0.3 | 40.5 | 82.0 | |
Table 1: Experimental results on MSR-VTT. This table shows the testing retrieval recall of models trained from scratch on the small coreset or synthetic set. “V”: Video, “T”: Text, and “A”: Audio. We report the mean and standard deviation of the results. In each case, the best performance is shown in bold.
| Triplets (Ratio) | Methods | R@1 | R@5 | Time (mins) | ||||||||||||
| V T | T V | V A | A V | T A | A T | Avg. | V T | T V | V A | A V | T A | A T | Avg. | |||
| 100 (1.25%) | Random | 5.7 0.5 | 7.7 0.0 | 26.3 0.0 | 25.7 0.0 | 5.6 0.0 | 3.7 0.3 | 12.4 | 25.9 0.3 | 28.3 0.1 | 54.3 0.1 | 52.1 0.1 | 20.5 0.1 | 18.6 0.2 | 33.3 | - |
| MTT-VL | 6.2 0.0 | 7.6 0.0 | 26.1 0.2 | 25.8 0.0 | 5.9 0.2 | 4.3 0.4 | 12.7 | 26.1 0.0 | 27.5 0.0 | 53.7 0.4 | 52.5 0.1 | 20.4 0.3 | 18.8 0.2 | 33.2 | 80.9 | |
| TESLA | 5.9 0.3 | 7.6 0.2 | 26.1 0.2 | 25.8 0.1 | 6.2 0.3 | 3.8 0.1 | 12.6 | 25.6 0.6 | 27.4 0.4 | 53.8 0.4 | 52.6 0.1 | 22.0 0.2 | 20.5 0.7 | 33.7 | 81.5 | |
| LoRS | 5.4 0.5 | 7.8 0.2 | 27.8 0.3 | 26.2 0.1 | 6.3 0.2 | 4.5 0.2 | 13.0 | 26.5 0.6 | 28.2 0.1 | 53.1 0.4 | 52.2 0.3 | 22.3 0.4 | 21.6 0.4 | 34.0 | 84.9 | |
| RepBlend | 5.6 0.5 | 7.6 0.3 | 27.2 0.1 | 25.0 0.1 | 6.1 0.3 | 4.5 0.1 | 12.7 | 27.1 0.4 | 28.6 0.3 | 54.1 0.1 | 51.4 0.3 | 20.9 0.1 | 19.9 0.6 | 33.7 | 81.9 | |
Table 2: Experimental results on VGGSound-S . This table shows the testing retrieval recall of models trained from scratch on the small coreset or synthetic set. “V”: Video, “T”: Text, and “A”: Audio. We report the mean and standard deviation of the results. In each case, the best performance is shown in bold.
| Triplets (Ratio) | Methods | R@1 | R@5 | Time (mins) | ||||||||||||
| V T | T V | V A | A V | T A | A T | Avg. | V T | T V | V A | A V | T A | A T | Avg. | |||
| 100 (1.19%) | Random | 27.4 0.1 | 27.7 0.1 | 17.4 0.1 | 15.5 0.1 | 4.4 0.1 | 3.8 0.1 | 16.0 | 50.3 0.1 | 54.1 0.1 | 36.6 0.1 | 36.2 0.1 | 13.8 0.1 | 13.7 0.0 | 34.1 | - |
| MTT-VL | 26.6 0.1 | 26.0 0.4 | 15.8 0.6 | 18.8 0.5 | 4.4 0.3 | 4.4 0.4 | 16.0 | 51.5 0.4 | 50.7 1.5 | 37.8 0.3 | 41.6 0.5 | 13.4 0.8 | 14.0 1.0 | 34.8 | 77.3 | |
| TESLA | 26.6 0.1 | 26.0 0.4 | 16.2 0.1 | 18.9 0.6 | 4.4 0.4 | 4.2 0.3 | 16.1 | 51.5 0.4 | 50.7 1.5 | 37.9 0.3 | 41.8 0.5 | 13.5 1.1 | 14.6 0.1 | 35.0 | 78.1 | |
| LoRS | 28.0 0.5 | 28.4 0.5 | 17.9 0.2 | 18.0 0.8 | 4.3 0.1 | 4.7 0.4 | 16.9 | 52.7 0.3 | 54.1 0.7 | 40.2 1.2 | 42.2 0.6 | 13.3 0.5 | 13.4 1.0 | 36.0 | 79.4 | |
| RepBlend | 27.7 0.5 | 28.5 0.7 | 18.0 0.2 | 18.2 0.9 | 4.3 0.1 | 4.7 0.5 | 16.9 | 52.9 0.0 | 54.3 0.8 | 40.8 0.7 | 41.9 0.1 | 13.4 0.6 | 13.5 1.3 | 36.1 | 78.2 | |
Table 3: Experimental results on DiDeMo. This table shows the testing retrieval recall of models trained from scratch on the small coreset or synthetic set. “V”: Video, “T”: Text, and “A”: Audio. We report the mean and standard deviation of the results. In each case, the best performance is shown in bold.
| Model | Setting | R@1 | R@5 | R@10 |
| ImageBind | Zero-shot | 12.25 | 32.87 | 44.32 |
| Distilled | 13.15 | 35.33 | 47.34 | |
| LanguageBind | Zero-shot | 4.91 | 19.94 | 29.56 |
| Distilled | 5.93 | 22.11 | 33.32 | |
| OmniBind | Zero-shot | 12.92 | 34.88 | 48.03 |
| Distilled | 13.13 | 35.63 | 48.87 |
Table 4: Cross architecture generalization results under ‘Zero-shot’ and ‘Distilled’ settings.
| Datasets | Queries | Avg. R@1 | Avg. R@5 | ||||||
| HoPA | 3-pair | T-bind | V-bind | HoPA | 3-pair | T-bind | V-bind | ||
| MSR-VTT | 100 | 21.78 | 20.21 | 20.42 | 21.05 | 42.43 | 42.08 | 41.45 | 41.97 |
| 200 | 21.93 | 19.94 | 20.49 | 19.24 | 42.73 | 42.40 | 41.35 | 42.28 | |
| 500 | 22.25 | 18.11 | 20.19 | 19.85 | 42.96 | 41.60 | 38.65 | 41.86 | |
| VGGSound-S | 100 | 13.19 | 12.78 | 12.10 | 12.92 | 34.89 | 35.20 | 34.01 | 35.00 |
| 200 | 13.15 | 12.31 | 11.30 | 12.74 | 35.51 | 34.90 | 33.76 | 35.51 | |
Table 5: Empirical comparisons Pairwise distillation and omniwise distillation. We compare our omniwise distillation strategy with three pair-wise variants on MSR-VTT, VGGSound-S, and DiDeMo. In each case, the best performance is shown in bold.
Figure 2: Ablation study of each learning objective across all datasets and different numbers of queries.
| Dataset | Triplets | R@1 | R@5 | R@10 | |||
| Rank-1 | Rank-2 | Rank-1 | Rank-2 | Rank-1 | Rank-2 | ||
| MSR-VTT | 100 | 21.78 | 20.80 | 42.43 | 39.99 | 52.28 | 49.77 |
| 200 | 21.93 | 21.09 | 42.73 | 39.81 | 52.61 | 49.37 | |
| 500 | 22.25 | 20.48 | 42.96 | 39.78 | 52.96 | 49.57 | |
| VGGSound-S | 100 | 13.19 | 12.18 | 34.89 | 32.85 | 46.72 | 44.13 |
| 200 | 13.15 | 12.25 | 35.33 | 32.58 | 47.34 | 43.72 | |
Table 6: Effect of the proxy rank on distillation performance. We compare Rank-1 (Ours) against Rank-2 across different metrics. In each case, the best result is shown in bold. As can be seen, Rank-1 consistently outperforms Rank-2 .
| Stage | w/o SVD | with SVD | SVD Cost |
| Buffer Training (min/traj) | 10.85 | 11.25 | +3.5% |
| Distillation (sec/100 iters) | 53.28 | 87.48 | +39.1% |
Table 7: Computational efficiency analysis by comparing the runtime with and without the SVD on MSR-VTT. The distillation time is measured per 100 iterations.
Figure 3: The visualization of distilled omnimodal data on ImageBind with the VGGSound-S dataset. Comparison between samples before and after distillation, shown with one representative video frame, one audio map, and the corresponding text ( left ). Illustrative cases, each visualized with three video frames, three audio maps, and the corresponding text description ( right ).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | MSR-VTT | VGGSound-S | DiDeMo |
| epochs | 10 | 10 | 10 |
| num_experts | 20 | 20 | 20 |
| batch_size | 128 | 128 | 128 |
| lr_teacher | 0.01 | 0.01 | 0.01 |
| 0.1 | 0.1 | 0.1 | |
| 0.2 | 0.2 | 0.2 |
Table 8: Hyperparameter settings for buffer.
| Dataset | MSR-VTT | VGGSound-S | DiDeMo | ||||||
| #instances | 100 | 200 | 500 | 100 | 200 | 500 | 100 | 200 | 500 |
| lr_data | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |
| lr_lr | 1e-4 | 1e-4 | 1e-4 | 1e-4 | 1e-4 | 1e-4 | 1e-4 | 1e-4 | 1e-4 |
| lr_sim | 10.0 | 10.0 | 100.0 | 10.0 | 10.0 | 100.0 | 10.0 | 10.0 | 100.0 |
| lr_teacher | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 |
| sim_rank | 10 | 20 | 20 | 10 | 20 | 40 | 10 | 20 | 20 |
Table 9: Hyperparameter settings for dataset distillation.
| Dataset | Triplets | Method Variant | R@1 | R@5 | R@10 |
| MSR-VTT | 100 | Ours (Full Model) | 21.78 | 42.43 | 52.28 |
| w/o Similarity Mining | 20.67 | 40.30 | 51.02 | ||
| w/o Instance Loss ( ) | 20.16 | 38.75 | 49.42 | ||
| w/o Modality Loss ( ) | 20.08 | 37.96 | 48.03 | ||
| 200 | Ours (Full Model) | 21.93 | 42.73 | 52.61 | |
| w/o Similarity Mining | 21.02 | 41.18 | 51.92 |
Table 10: Complete ablation study results. We report Recall@1, Recall@5, and Recall@10 (%) on MSR-VTT, VGGSound-S, and DiDeMo across varying synthetic sizes. The best results in each setting are marked in bold .
Figure 4: Examples of distilled instances from the VGGSound-S dataset with triplets.
More in this topic
Dataset Distillation
Dataset distillation compresses training data into a small synthetic or selected set that retains the information needed to train useful models.
See all 66 papers