cs.CVMar 14, 2026
SaveIMS3: Breaking Distributional Aggregation in Diffusion-Based Dataset Distillation
Organizations: AGI Lab, Westlake University · Eindhoven University of Technology · Jinan University · IHPC, Agency for Science, Technology and Research, Singapore ·
Abstract
Dataset Distillation aims to synthesize compact datasets that can approximate the training efficacy of large-scale real datasets, offering an efficient solution to the increasing computational demands of modern deep learning. Recently, diffusion-based dataset distillation methods have shown great promise by leveraging the strong generative capacity of diffusion models to produce diverse and structurally consistent samples. However, a fundamental goal misalignment persists: diffusion models are optimized for generative likelihood rather than discriminative utility, resulting in over-concentration in high-density regions and inadequate coverage of boundary samples crucial for classification. To address this issue, we propose two complementary strategies. Inversion-Matching (IM) introduces an inversion-guided fine-tuning process that aligns denoising trajectories with their inversion counterparts, broadening distributional coverage and enhancing diversity. Selective Subgroup Sampling(S^3) is a training-free sampling mechanism that improves inter-class separability by selecting synthetic subsets that are both representative and distinctive. Extensive experiments demonstrate that our approach significantly enhances the discriminative quality and generalization of distilled datasets, achieving state-of-the-art performance among diffusion-based methods.
Figures & tables
Figure 1: t-SNE visualization of feature distributions for different distilled datasets. The blue density map represents the feature distribution of the original dataset. The purple squares correspond to samples generated by the DiT [ 23 ] , the orange dots correspond to Minimax [ 8 ] , and the red stars indicate samples produced by our ImS 3 . Our method covers a broader area of the feature space, indicating improved distribu- tional coverage.
Figure 2: Overview of the proposed ImS 3 framework. (a) IM fine-tuning expands distribution coverage by aligning denoised and inverted latents through the IM loss . (b) S 3 enhances discriminative diversity during sampling by selecting representative subgroups whose centroids are close to real data centroids while maintaining inter-class separation, optimized via . denotes the centroid of real samples from the -th class, while represents the centroid of the -th candidate subgroup synthesized by the diffusion model for the same class. The selected optimal subgroup centroid strikes the best balance between representativeness to the real class distribution and distinctiveness from other classes.
| IPC (Ratio) | Model | Random | K-Center [ 28 ] | Herding [ 41 ] | DiT [ 23 ] | Minimax [ 8 ] | DM [ 47 ] | IDC-1 [ 14 ] | D 4 M [ 32 ] | D3HR [ 50 ] | DDVLCP [ 53 ] | ImS 3 (Ours) | Full |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 10 (0.8%) | ConvNet-6 | 24.3 1.1 | 19.4 0.9 | 26.7 0.5 | 34.2 1.1 | 34.1 0.4 | 26.9 1.2 | 33.3 1.1 | 29.4 0.9 | – | 34.8 2.4 | 35.6 1.8 | 86.4 0.2 |
| ResNetAP-10 | 29.4 0.8 | 22.1 0.1 | 32.0 0.3 | 34.7 0.5 | 35.7 0.3 | 30.3 1.2 | 39.1 0.5 | 33.2 2.1 | 40.7 1.0 | 39.5 1.5 | 41.8 0.3 | 87.5 0.5 | |
| ResNet-18 | 27.7 0.9 | 21.1 0.4 | 30.2 1.2 | 34.7 0.4 | 37.6 0.9 | 33.4 0.7 | 37.3 0.2 | 32.3 1.2 | 39.6 1.0 | 39.9 2.6 | 41.3 1.1 | 89.3 1.2 | |
| 20 (1.6%) | ConvNet-6 | 29.1 0.7 | 21.5 0.8 | 29.5 0.3 | 36.1 0.8 | 37.9 0.2 | 29.9 1.0 | 35.5 0.8 | 34.0 2.3 | – | 37.9 1.9 | 39.4 0.5 | 86.4 0.2 |
| ResNetAP-10 | 32.7 0.4 | 25.1 0.7 | 34.9 0.1 | 41.1 0.8 | 43.3 0.3 | 35.2 0.6 | 43.4 0.3 | 40.1 1.6 | – | 44.5 2.2 | 45.8 1.2 | 87.5 0.5 | |
| ResNet-18 | 29.7 0.5 | 23.6 0.3 | 32.2 0.6 | 40.5 0.5 | 40.9 0.6 | 29.8 1.7 | 38.6 0.2 | 38.4 1.1 | – | 44.5 2.0 | 44.9 1.6 | 89.3 1.2 |
Table 1: Comparison of SOTA dataset distillation methods on ImageWoof under various IPC and backbone settings. The best results are marked as bold, and the second are underlined.
| IPC | Random | DiT [ 23 ] | Minimax [ 8 ] | D 4 M [ 32 ] | ImS 3 (Ours) |
|---|---|---|---|---|---|
| 10 | 54.2 1.6 | 59.1 0.7 | 58.6 0.4 | 60.9 1.7 | 62.9 1.2 |
| 20 | 63.5 0.5 | 64.8 1.2 | 70.6 0.3 | 66.3 1.3 | 71.6 0.7 |
| 50 | 76.1 1.1 | 73.3 0.9 | 83.7 0.4 | 77.7 1.1 | 84.2 1.0 |
Table 2: Comparison on ImageNette under different IPC settings. All the results are obtained on ResNetAP-10. The best results are marked as bold and the second are underlined.
| IPC | Random | SRe 2 L [ 45 ] | DiT [ 23 ] | RDED [ 33 ] | Minimax [ 14 ] | ImS 3 (Ours) |
|---|---|---|---|---|---|---|
| 1 | 12.7 0.5 | 17.2 1.2 | 26.7 1.4 | 15.9 0.9 | 22.4 0.6 | 28.5 2.0 |
| 10 | 48.1 0.8 | 23.6 2.0 | 54.1 0.4 | 33.8 2.0 | 51.9 1.4 | 56.2 0.6 |
| 50 | 68.1 0.7 | 45.7 1.3 | 76.4 0.3 | 78.3 0.9 | 78.3 0.2 | 78.5 1.5 |
Table 3: Comparison on ImageIDC under different IPC settings. All the results are obtained on ResNet-18. The best results are marked as bold, and the second are underlined.
Figure 3: Heatmap of classification accuracy under different combinations of and . The heatmap shows that balanced values of and yield the best performance.
| Method | ImageWoof | ImageNette | ||
|---|---|---|---|---|
| IPC = 10 | IPC = 50 | IPC = 10 | IPC = 50 | |
| DiT [ 23 ] | 34.7 0.5 | 49.3 0.2 | 58.9 1.2 | 82.9 0.1 |
| + IM | 37.3 0.1 | 53.5 0.5 | 60.0 0.9 | 81.5 0.5 |
| + S 3 | 40.9 0.2 | 54.9 0.5 | 62.2 1.9 | 81.3 1.2 |
| + ImS 3 | 41.8 0.3 | 61.0 0.7 | 62.9 1.2 | 84.2 1.0 |
Table 4: Ablation study of IM and S 3 under IPC = 10 and 50. Results on ImageWoof are obtained using ResNetAP-10, while results on ImageNette are obtained using ResNet-18. The best results are marked as bold.
Figure 4: Analysis of real-images usage from original dataset used to compute reference centroids in Eq. 8 . Moderate sample counts provide stable and reliable centroids for subgroup selection, while too few samples introduce noise and degrade selection quality.
| IPC | Model | Random | Herding [ 41 ] | IDC-1 [ 14 ] | Minimax [ 14 ] | MGD 3 [ 3 ] | ImS 3 (Ours) |
|---|---|---|---|---|---|---|---|
| 10 | ConvNet-6 | 17.0 0.3 | 17.2 0.3 | 24.3 0.5 | 22.3 0.5 | 23.4 0.9 | 24.3 1.1 |
| ResNetAP-10 | 19.1 0.4 | 19.8 0.3 | 25.7 0.1 | 24.8 0.2 | 25.8 0.5 | 28.4 0.2 | |
| ResNet-18 | 17.5 0.5 | 16.1 0.2 | 25.1 0.2 | 22.5 0.3 | 23.6 0.4 | 28.3 0.3 | |
| 20 | ConvNet-6 | 24.8 0.2 | 24.3 0.4 | 28.8 0.3 | 29.3 0.4 | 30.6 0.4 | 30.7 0.4 |
| ResNetAP-10 | 26.7 0.5 | 27.6 0.1 | 29.9 0.2 | 32.3 0.1 | 33.9 1.1 | 34.8 0.2 | |
| ResNet-18 | 25.5 0.3 | 24.7 0.1 | 30.2 0.2 | 31.2 0.1 | 32.6 0.4 | 36.3 0.2 |
Table 5: Performance comparison on ImageNet-100 under different IPC and backbone settings. Results are Top-1 accuracy.
| Backbone | IPC | SRe 2 L [ 45 ] | Minimax [ 8 ] | RDED [ 33 ] | CaO 2 [ 37 ] | ImS 3 (Ours) |
|---|---|---|---|---|---|---|
| ResNet-18 | 10 | 20.2 0.2 | 40.1 1.0 | 38.5 2.1 | 45.6 1.4 | 45.9 1.3 |
| 50 | 23.3 0.3 | 67.0 1.8 | 68.5 0.7 | 68.9 1.1 | 71.2 1.3 | |
| ResNet-50 | 10 | 17.3 1.7 | 37.3 1.1 | 29.9 2.2 | 40.1 0.1 | 42.7 1.8 |
| 50 | 24.8 0.7 | 64.3 0.9 | 67.8 0.3 | 68.2 1.1 | 68.3 0.2 | |
| ResNet-101 | 10 | 17.7 0.9 | 34.2 1.7 | 31.3 1.3 | 36.5 1.4 | 38.2 1.6 |
| 50 | 21.2 0.2 | 62.7 1.6 | 59.1 0.7 | 63.1 1.3 | 66.1 1.8 |
Table 6: Performance comparison under different IPC and backbone settings on ImageWoof. Best results in each setting are marked as bold.
| Loss Type | ImageWoof | ImageNette | ||
|---|---|---|---|---|
| IPC = 10 | IPC = 50 | IPC = 10 | IPC = 50 | |
| 38.7 0.2 | 58.1 0.2 | 62.6 0.9 | 81.7 0.7 | |
| 39.0 0.2 | 58.1 0.1 | 62.7 1.0 | 82.8 0.7 | |
| 41.8 0.3 | 60.1 0.7 | 62.9 1.2 | 84.2 1.0 | |
Table 7: Ablation study of different similarity losses used in the IM on ImageWoof and ImageNette under IPC = 10 and 50. Results are reported as top-1 accuracy.
Figure 5: Validation accuracy under different matching strengths. Performance of our method across a range of matching coefficients for IPC = 10, 20, and 50. Each curve corresponds to a fixed IPC, and shaded regions denote performance variability across runs.
Figure 6: Effect of subgroup pool size under different IPC settings on ImageWoof. A moderate increase in provides richer intra-class variation and improves selection quality, especially in low-IPC regimes. However, excessively large pools introduce redundant or noisy candidates, which destabilizes selection and leads to degraded performance.
| IPC | Res18 | Res50 | Res101 | Efficient | CLIP |
|---|---|---|---|---|---|
| 10 | 41.8 0.3 | 39.6 0.3 | 38.8 0.2 | 39.5 0.2 | 38.1 0.5 |
| 20 | 45.8 1.2 | 45.7 0.3 | 45.0 0.2 | 45.8 0.3 | 44.3 0.2 |
Table 8: Ablation study on feature extractors for centroid selection on ImageWoof. Top-1 accuracy (%).
| DiT | Minimax | MGD3 | IMS 3 | |
|---|---|---|---|---|
| Fine-tuning | - | 32min | - | 19.5min |
| Sampling | 1.5min | 1.5min | 21min | 7min |
| Total | 1.5min | 33.5min | 21min | 26.5min |
Table 9: Runtime comparison on ImageWoof (IPC=10).
| DiT | Minimax | CaO 2 | IMS 3 | |
|---|---|---|---|---|
| IPC=10 | 36.6 0.2 | 33.3 0.2 | 37.5 0.3 | 37.9 0.2 |
| IPC=50 | 52.5 0.2 | 51.4 0.5 | 53.8 0.6 | 59.6 0.3 |
Table 10: 512 512 results on ImageWoof.
| Model | Minimax | MGD3 | IMS 3 | |
|---|---|---|---|---|
| IPC=10 | EfficientNet | 31.3 0.7 | 32.3 0.3 | 34.2 0.2 |
| ViT | 18.9 0.0 | 17.7 0.2 | 20.0 0.0 | |
| IPC=50 | EfficientNet | 46.3 1.0 | 49.3 0.5 | 51.7 0.4 |
| ViT | 18.8 0.3 | 17.6 0.1 | 19.9 0.5 |
Table 11: Cross-architecture evaluation on ImageWoof.
| SRe 2 L | RDED | Minimax | MGD3 | IMS 3 | |
|---|---|---|---|---|---|
| Top-1 Acc. | 21.3 0.6 | 42.0 0.1 | 44.3 0.5 | 45.5 0.1 | 45.6 0.3 |
Table 12: ImageNet-1K results (IPC=10).
Figure 7: Feature-space visualization of distilled samples.
| DiT | Minimax | IMS | IMS | |
|---|---|---|---|---|
| IPC=10 | 34.7 0.5 | 35.7 0.3 | 39.6 0.4 | 41.8 0.3 |
| IPC=50 | 49.3 0.2 | 54.4 0.6 | 55.5 0.3 | 57.3 0.5 |
Table 13: Gen-data centroid results on ImageWoof
Figure 8: t-SNE visualization of distilled samples on ImageNet-100 and ImageWoof. ImS 3 produces tighter intra-class clusters and clearer inter-class separation, whereas MinimaxDiffusion exhibits more dispersed and overlapping structures.
Figure 9: Comparison between samples selected by Minimax [ 8 ] (left) and generated by the proposed ImS 3 (right) for ImageWoof. The class names are marked at the left of each row.
Figure 10: Comparison between samples selected by Minimax [ 8 ] (left) and generated by the proposed ImS 3 (right) for ImageNette. The class names are marked at the left of each row.
Figure 11: Comparison between samples selected by Minimax [ 8 ] (left) and generated by the proposed ImS 3 (right) for ImageNet-100 classes 0-9. The class names are marked at the left of each row.
Figure 12: Comparison between samples selected by Minimax [ 8 ] (left) and generated by the proposed ImS 3 (right) for ImageNet-100 classes 10-19. The class names are marked at the left of each row.
Figure 13: Comparison between samples selected by Minimax [ 8 ] (left) and generated by the proposed ImS 3 (right) for ImageNet-100 classes 20-29. The class names are marked at the left of each row.
Figure 14: Comparison between samples selected by Minimax [ 8 ] (left) and generated by the proposed ImS 3 (right) for ImageNet-100 classes 30-39. The class names are marked at the left of each row.
Figure 15: Comparison between samples selected by Minimax [ 8 ] (left) and generated by the proposed ImS 3 (right) for ImageNet-100 classes 40-49. The class names are marked at the left of each row.
Figure 16: Comparison between samples selected by Minimax [ 8 ] (left) and generated by the proposed ImS 3 (right) for ImageNet-100 classes 50-59. The class names are marked at the left of each row.
Figure 17: Comparison between samples selected by Minimax [ 8 ] (left) and generated by the proposed ImS 3 (right) for ImageNet-100 classes 60-69. The class names are marked at the left of each row.
Figure 18: Comparison between samples selected by Minimax [ 8 ] (left) and generated by the proposed ImS 3 (right) for ImageNet-100 classes 70-79. The class names are marked at the left of each row.
Figure 19: Comparison between samples selected by Minimax [ 8 ] (left) and generated by the proposed ImS 3 (right) for ImageNet-100 classes 80-89. The class names are marked at the left of each row.
Figure 20: Comparison between samples selected by Minimax [ 8 ] (left) and generated by the proposed ImS 3 (right) for ImageNet-100 classes 90-99. The class names are marked at the left of each row.