In recent times, large datasets hinder efficient model training while also containing redundant concepts. Dataset distillation aims to synthesize compact datasets that preserve the knowledge of large-scale training sets while drastically reducing storage and computation. Recent advances in diffusion models have enabled training-free distillation by leveraging pre-trained generative priors; however, existing guidance strategies remain limited. Current score-based methods either perform unguided denoising or rely on simple mode-based guidance toward instance prototype centroids (IPC centroids), which often are rudimentary and suboptimal. We propose Manifold-Guided Distillation (ManifoldGD), a training-free diffusion-based framework that integrates manifold consistent guidance at every denoising timestep. Our method employs IPCs computed via a hierarchical, divisive clustering of VAE latent features, yielding a multi-scale coreset of IPCs that captures both coarse semantic modes and fine intra-class variability. Using a local neighborhood of the extracted IPC centroids, we create the latent manifold for each diffusion denoising timestep. At each denoising step, we project the mode-alignment vector onto the local tangent space of the estimated latent manifold, thus constraining the generation trajectory to remain manifold-faithful while preserving semantic consistency. This formulation improves representativeness, diversity, and image fidelity without requiring any model retraining. Empirical results demonstrate consistent gains over existing training-free and training-based baselines in terms of FID, l2 distance among real and synthetic dataset embeddings, and classification accuracy, establishing ManifoldGD as the first geometry-aware training-free data distillation framework.
Figures & tables
Figure 1: Manifold Guidance. Overall denoising trajectory correction by manifold guidance ( gmanifoldt ) to recitfy the off manifold component of mode guidance ( gmodet ). Mdata is mebedded inside Mt and removing noise ϵt from Mt transforms it to Mt−1 . As t→0 , Mt→Mdata .
Figure 2: Qualitative samples generated by DiT [ 31 ] , MGD [ 37 ] , and ManifoldGD. The samples generated by ManifoldGD have better image structure and quality (eg. dog image of MGD is having legs in unusual position, dog image generated by DiT is blurry. Similarly, the buildings have uncommon structure for MGD. The ball image generated by DiT is of poor quality).
Figure 3: Qualitative evolution of generated samples across denoising timesteps ( t ). Comparison between MGD [ 37 ] and our ManifoldGD shows that early timesteps ( t>25 ) capture coarse semantic structure via mode guidance, while later stages ( t≤20 ) refine geometry and texture under manifold constraints. ManifoldGD consistently yields sharper, more coherent generations with enhanced semantic fidelity and contrast. The zoomed insets (red boxes) highlight that MGD [ 37 ] often blurs key regions (e.g., missing eyes and unclear nose in the dog, coarse floor textures in the church), whereas ManifoldGD preserves fine-grained (e.g., different reflections from chairs and colored windows) lighting variations, reflections, and high-frequency details, demonstrating geometrically consistent and visually realistic synthesis.
Figure 4: ℓ2 and MMD comparison. DiT [ 31 ] achieves inferior results ( ↑ℓ2 MMD) and while ManifoldGD achieves the best result ( ↓ L2 MMD).
Dataset
IPC
Full
Training-Based
Training-Free
Herding
IDC-1
D 4 M
DM
Li et al.
MinMaxDiff
Zou et al.
Random
LDM
DiT*
MGD*
ManifoldGD
[ 49 ]
[ 22 ]
[ 43 ]
[ 55 ]
[ 25 ]
[ 15 ]
[ 57 ]
[ 32 ]
[ 34 ]
[ 31 ]
[ 37 ]
Nette
10
95.1 ± 1.2
-
-
60.4 ± 3.4
60.8 ± 0.6
61.5 ± 0.9
62.0 ± 0.2
64.8 ± 3.6
54.2 ± 1.6
60.3 ± 3.6
59.1 ± 0.7
61.9 ± 0.6
64.1 ± 1.2 (+2.2)
20
95.1 ± 1.2
-
-
65.5 ± 1.2
66.5 ± 1.1
66.9 ± 0.5
66.8 ± 0.4
71.0 ± 0.5
63.5 ± 0.5
62.0 ± 2.6
64.8 ± 1.2
66.5 ± 0.5
69.7 ± 0.4 (+3.2)
50
95.1 ± 1.2
-
-
73.8 ± 1.7
76.2 ± 0.4
76.8 ± 0.7
76.6 ± 0.2
81.2 ± 0.8
76.1 ± 1.1
71.0 ± 1.4
73.3 ± 0.9
77.5 ± 1.1
78.4 ± 1.0 (+1.4)
ImageNet100
10
80.3 ± 0.2
19.8 ± 0.3
25.7 ± 0.1
-
-
-
24.8 ± 0.2
-
19.1 ± 0.4
-
23.2 ± 0.3
26.1 ± 0.6
27.6 ± 0.5 (+1.5)
Table 1: Performance comparison with state-of-the-art methods on ImageNet subsets, evaluated using the hard-label protocol. Results are based on ResNetAP-10 with average pooling, with the best performance highlighted in bold . Underlined results are the one where ManifoldGD outperforms training-based methods. AccS→D is used as the evaluation metric. * are the methods we re-implemented. Full indicates training the classifiers on the entire real training data and reporting classification accuracy on the real test data ( AccDtrain→Dtest )
IPC (Ratio)
Test Model
Full
Training-Based
Training-Free
-
Herding
DM
K-Center
Zou et al.
IDC-1
GLaD
MinMaxDiff
Random
DiT*
MGD*
ManifoldGD
[ 49 ]
[ 55 ]
[ 38 ]
[ 57 ]
[ 22 ]
[ 4 ]
[ 15 ]
[ 32 ]
[ 31 ]
[ 37 ]
10 (0.8%)
ConvNet-6
86.4 ± 0.2
26.7 ± 0.5
26.9 ± 1.2
19.4 ± 0.9
34.8 ± 2.4
33.3 ± 1.1
33.8 ± 0.9
37.0 ± 1.0
24.3 ± 1.1
34.2 ± 1.1
35.1 ± 1.0
36.9 ± 0.6 (+1.8)
ResNetAP-10
87.5 ± 0.5
32.0 ± 0.3
30.3 ± 1.2
22.1 ± 0.1
39.5 ± 1.5
39.1 ± 0.5
32.9 ± 0.9
39.2 ± 1.3
29.4 ± 0.8
34.7 ± 0.5
37.5 ± 2.2
38.3 ± 0.4 (+1.3)
ResNet-18
89.3 ± 1.2
30.2 ± 1.2
33.4 ± 0.7
21.1 ± 0.4
39.9 ± 2.6
37.3 ± 0.2
31.7 ± 0.8
37.6 ± 0.9
27.7 ± 0.9
34.7 ± 0.4
37.9 ± 0.7
39.2 ± 0.7 (+1.3)
20 (1.6%)
ConvNet-6
86.4 ± 0.2
29.5 ± 0.3
29.9 ± 1.0
21.5 ± 0.8
37.9 ± 1.9
35.5 ± 0.8
–
37.6 ± 0.2
29.1 ± 0.7
36.1 ± 0.8
36.5 ± 1.0
37.7 ± 0.6 (+1.2)
Table 2: Performance comparison with state-of-the-art methods on ImageWoof, evaluated using the hard-label protocol. Results are based on ResNetAP-10, ConvNet-6 and ResNet-18 with average pooling, with the best performance highlighted in bold . Underlined results are the one where ManifoldGD outperforms training-based methods. AccS→D is used as the evaluation metric. * are the methods we re-implemented. Full indicates training the classifiers on the entire real training data and reporting classification accuracy on the real test data ( AccDtrain→Dtest )
Figure 5: FID, Representativeness and Diversity comparison of DiT [ 31 ] , MGD [ 37 ] , and ManifoldGD. IPC 10,20, and 50 are used for ImageNet-100 and ImageNette. ManifoldGD achieves lower FID (% drop over MGD [ 37 ] marked above the bars), higher representativeness and diversity (R=representativeness, D=diversity, % increase over MGD [ 37 ] shown in the plot) across all settings.
Figure 6: VAE feature space and IPC illustration for Nette IPC=10. KMeans do not capture the entire feature cloud whereas our divisive-levelwise occupy regions near the mean as well as the outlines of the feature cloud. The convex hull area ratio (in percentage) A_{CH}=\frac{\text{Area}(\text{IPC entroid (c_{s}) hull})}{\text{Area}(\text{entire VAE feature space hull})} is higher for our divisive method, indicating better spatial coverage.
Method
C
L
A
ConvNet-6
ResNet-AP-10
ResNet-18
KMeans
✓
✗
✗
56.3
61.0
59.7
Agglomerative
✓
✗
✗
37.7
45.9
42.6
Divisive
✓
✗
✗
57.4
62.5
58.5
Divisive-levelwise
✓
✓
✗
59.2
63.3
61.1
Ours
✓
✓
✗
60.5
64.0
62.3
Ours (annealed)
✓
✓
✓
60.8
64.5
62.7
Table 3: Ablation study on the components of the proposed method. All evaluations are done for IPC = 10 on ImageNette. C → Clustering, L → Level-wise, A → Annealing
Exponential
Cosine
Linear
t
ConvNet-6
ResNet-18
ConvNet-6
ResNet-18
ConvNet-6
ResNet-18
5
60.8
62.7
59.4
62.7
59.6
62.5
10
60.1
61.9
58.3
62.1
58.1
61.2
15
58.5
62.1
57.9
61.9
57.8
62.3
Table 4: Comparison of learning rate schedules across warm-up steps. Results are reported for ConvNet-6 and ResNet-18. Faster decay of the radius of Ns achieves the best result.
Figure 7: Ablation of diffusion step t and corresponding TSTOP . We vary t∈{10,20,30,40,50} and set TSTOP=t/2 . The results show that mode guidance dominates the early denoising phase, while manifold guidance progressively takes over as t increases. This transition indicates that early denoising benefits from strong class-alignment forces ( gmodet ), whereas later stages require geometric correction ( gmanifoldt ) to maintain fidelity and prevent off-manifold drift.
Figure 8: Ablation experiments varying TSTOP for ImageNette (first row) and ImageNet-100 (last row). Increasing TSTOP till 20 (steady growth region) improves image quality ( ↓ FID) and classification performance ( ↑AccS→D ). From TSTOP=20→30 we see a stable FID and AccS→D (saturation region). Further increase in TSTOP beyond 30 leads to drop in both image quality ( ↑ FID) and performance ( ↓AccS→D ).
Kernel
Without manifold guidance ( gmodet )
With manifold guidance ( gmodet + gmant )
ConvNet-6
ResNet-18
ResNetAP-10
ConvNet-6
ResNet-18
ResNetAP-10
RBF
56.7
61.5
62.5
58.5
62.5
64.4
Laplace
56.3
56.1
58.8
57.0
57.2
60.2
IMQ
54.4
56.1
57.1
55.6
56.7
58.5
Table 5: Ablation on kernel functions for mode guidance on ImageNette (IPC = 10). We compare RBF (MGD [ 37 ] ), Laplace, and IMQ kernels with and without manifold guidance across three classifiers. Manifold guidance consistently improves performance regardless of kernel type, demonstrating its general applicability.
Setting
Method
Aust. Terr.
Border Terr.
Samoyed
Beagle
Shih-Tzu
Ridgeback
Dingo
Golden Ret.
Old Eng. Sheep.
Germ. Shep.
IPC = 10
ConvNet
MGD [ 37 ]
24.9
40.7
16.7
27.7
22.2
31.2
28.9
31.8
40.3
32.2
ManifoldGD
24.7
36.8
21.5
40.6
31.2
36.1
28.7
34.6
34.5
35.6
ResNet-AP
MGD [ 37 ]
35.0
44.1
21.5
46.0
24.7
32.4
29.4
37.4
38.9
45.9
ManifoldGD
36.4
39.3
22.6
49.1
35.2
39.6
38.4
48.6
38.2
50.0
ResNet-18
MGD [ 37 ]
28.9
43.4
18.2
37.5
37.2
31.7
36.2
34.4
45.2
35.6
Table 6: Class-wise Top-1 Accuracy (%) on ImageWoof for IPC=10 and IPC=50. MGD [ 37 ] vs ManifoldGD across three classifiers. Columns list the 10 ImageWoof classes. Class abbreviations: Aust. Terrier (Australian Terrier), Border Terrier, Samoyed, Beagle, Shih-Tzu, Rhodesian Ridgeback, Dingo, Golden Retriever, Old English Sheepdog, German Shepherd.
Figure 9: VAE feature space and IPC illustration for Nette IPC=10. Agglomerative clustering outputs IPC cluster centroids that near the edge of the feature cloud whereas Divisive clustering outputs IPC centroids that are near the mean of the feature cloud.
Figure 10: Ablation experiments to find out the level of divisive-layerwise clustering. It can be seen that for datasets with less class (ImageNette) the start level ( sstart ) is more whereas for datasets with more class (ImageNet-100) the level is less.
ImageNette (IPC=10)
ImageNet-100 (IPC=10)
Classifier
r=0.05
r=0.1
r=0.2
r=0.05
r=0.1
r=0.2
ConvNet-6
60.5
59.7
59.5
23.7
24.5
24.0
ResNetAP-10
63.7
63.2
63.0
26.5
27.4
27.2
ResNet-18
62.3
62.0
61.2
23.1
23.8
23.4
Table 7: Effect of radius r on downstream classification accuracy (%). Best result is highlighted with bold .
Method
ResNet-18
ResNet-10
ConvNet-6
MGD [ 37 ]
64.6
62.7
60.8
ManifoldGD
66.5 ( +1.9 )
63.6 ( +0.9 )
62.1 ( +1.3 )
Table 8: Performance comparison using DDIM. Using ImageNette IPC=10 setting for comparison.
d
1
2
3
4
5
Nette
58.6
58.5
59.5
58.2
57.9
ImageNet-100
24.0
24.1
24.5
24.4
23.8
Table 9: Effect of d on ConvNet-6 performance (IPC=10), transposed.
Figure 11: Inference time vs Accuracy of MGD [ 37 ] , DiT [ 31 ] and ManifoldGD for ImageNette IPC 10. We see that ManifoldGD has a higher inference compared to MGD [ 37 ] and DiT [ 31 ] while also producing superior performance.
Figure 12: Trajectory comparison of MGD [ 37 ] , DiT [ 31 ] and ManifoldGD for ImageNette IPC 10. We see that ManifoldGD has a higher final trajectory variance of the norm of the latent vector ∣∣x∣∣ compared to MGD [ 37 ] and DiT [ 31 ] .
Figure 13: Qualitative comparison of MGD [ 37 ] , DiT [ 31 ] and ManifoldGD. We provide the Query to instruct GPT 4.o to evaluate the images (a) → DiT, b) → MGD, c) → ManifoldGD). We see from the answers of GPT 4.o that ManifoldGD achieves better images that discriminate the concept (fish) from other information (background) better. Green highlighted text indicates positive attributes whereas Red highlighted text indicates negative attributes.
Figure 14: Qualitative comparison of MGD [ 37 ] , DiT [ 31 ] and ManifoldGD. We provide the Query to instruct GPT 4.o to evaluate the images (a) → DiT, b) → MGD, c) → ManifoldGD). We see from the answers of GPT 4.o that ManifoldGD achieves better images that discriminate the concept (monkey) from other information (background) better. Green highlighted text indicates positive attributes whereas Red highlighted text indicates negative attributes.
Figure 15: Qualitative comparison of MGD [ 37 ] , DiT [ 31 ] and ManifoldGD for ImageNette. ManifoldGD produces sharper, more semantically aligned and structurally coherent samples compared to MGD, while also avoiding the occasional blurring or texture flattening observed in DiT. Differences are especially prominent in edges, fine textures (fur, feathers), and object–background boundaries.
Figure 16: Qualitative comparison of MGD [ 37 ] , DiT [ 31 ] and ManifoldGD for ImageNet-100. All three training-free methods generate visually plausible samples for this larger and more diverse dataset, though ManifoldGD still achieves cleaner geometry, fewer artifacts, and improved object–background separation. Differences are more subtle than in ImageNette due to the higher visual entropy and class variability of ImageNet-100.
IPC
DiT* [ 31 ]
MGD* [ 37 ]
ManifoldGD
1
2.6±0.3
2.8±0.9
3.1 ±0.9 ( +0.3% )
50
18.5±1.3
20.3±1.1
21.4 ±1.5 ( +1.1% )
Table 10: Comparison on ImageNet-1k using ConvNet-6. Best results are in bold , second-best are underlined . * are results of methods achieved after reimplementation
Dataset distillation enables efficient training by distilling the information of large-scale datasets into significantly smaller synthetic datasets. Diffusion based paradigms have emerged in recent years, offering novel perspectives for dataset distillation. However, they typically necessitate additional fine-tuning stages, and effective guidance mechanisms remain underexplored. To address these limitations, we rethink diffusion based dataset distillation and propose a Dual Matching Guided Diffusion (DMGD) framework, centered on efficient training-free guidance. We first establish Semantic Matching via conditional likelihood optimization, eliminating the need for auxiliary classifiers. Furthermore, we propose a dynamic guidance mechanism that enhances the diversity of synthetic data while maintaining semantic alignment. Simultaneously, we introduce an optimal transport (OT) based Distribution Matching approach to further align with the target distribution structure. To ensure efficiency, we develop two enhanced strategies for diffusion based framework: Distribution Approximate Matching and Greedy Progressive Matching. These strategies enable effective distribution matching guidance with minimal computational overhead. Experimental results on ImageNet-Woof, ImageNet-Nette, and ImageNet-1K demonstrate that our training-free approach achieves significant improvements, outperforming state-of-the-art (SOTA) methods requiring additional fine-tuning by average accuracy gains of 2.1%, 5.4%, and 2.4%, respectively.
Qichao Wang, Yunhong Lu, Hengyuan Cao +2
Zhejiang University · Shanghai Institute for Advanced Study-Zhejiang University · Shanghai Institute for Mathematics and Interdisciplinary Sciences
Training machine learning models on massive datasets is expensive and time-consuming. Dataset distillation addresses this by creating a small synthetic dataset that achieves the same performance as the full dataset. Recent methods use diffusion models to generate distilled data, either by promoting diversity or matching training gradients. However, existing approaches produce redundant training signals, where samples convey overlapping information. Empirically, disjoint subsets of distilled datasets capture 80-90% overlapping signals. This redundancy stems from optimizing visual diversity or average training dynamics without accounting for similarity across samples, leading to datasets where multiple samples share similar information rather than complementary knowledge. We propose learnability-driven dataset distillation, which constructs synthetic datasets incrementally through successive stages. Starting from a small set, we train a model and generate new samples guided by learnability scores that identify what the current model can learn from, creating an adaptive curriculum. We introduce Learnability-Guided Diffusion (LGD), which balances training utility for the current model with validity under a reference model to generate curriculum-aligned samples. Our approach reduces redundancy by 39.1%, promotes specialization across training stages, and achieves state-of-the-art results on ImageNet-1K (60.1%), ImageNette (87.2%), and ImageWoof (72.9%). Our code is available on our project page https://jachansantiago.github.io/learnability-guided-distillation/.
Jeffrey A. Chan-Santiago, Mubarak Shah
Institute of Artificial Intelligence, University of Central Florida
Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamics, enabling one-step or few-step generation. However, existing methods often depend on curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks, which complicate model training and scaling. In this work, we propose Consistent Distribution Matching, a simulation-free and data-free distillation method for accelerating diffusion and flow models while preserving strong generative capacity. Our key insight is to unify sample generation and score estimation with one student network. Thus, our framework uses only two models, a frozen teacher and a trainable student, and optimizes one objective. We prove that minimizing our objective indicates Wasserstein convergence of the student flow-map pushforwards to the teacher marginals. On ImageNet 256×256, our method attains an FID of 2.04 with a single function evaluation (1-NFE) and a 4-NFE FID of 1.37 within 40 epochs of training, surpassing the state-of-the-art distillation baselines without data. Our code code and model are available at https://consistentdmd.github.io/.
Yuxiang Fu, Qi Yan, Zike Wu +4
University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair