Training machine learning models on massive datasets is expensive and time-consuming. Dataset distillation addresses this by creating a small synthetic dataset that achieves the same performance as the full dataset. Recent methods use diffusion models to generate distilled data, either by promoting diversity or matching training gradients. However, existing approaches produce redundant training signals, where samples convey overlapping information. Empirically, disjoint subsets of distilled datasets capture 80-90% overlapping signals. This redundancy stems from optimizing visual diversity or average training dynamics without accounting for similarity across samples, leading to datasets where multiple samples share similar information rather than complementary knowledge. We propose learnability-driven dataset distillation, which constructs synthetic datasets incrementally through successive stages. Starting from a small set, we train a model and generate new samples guided by learnability scores that identify what the current model can learn from, creating an adaptive curriculum. We introduce Learnability-Guided Diffusion (LGD), which balances training utility for the current model with validity under a reference model to generate curriculum-aligned samples. Our approach reduces redundancy by 39.1%, promotes specialization across training stages, and achieves state-of-the-art results on ImageNet-1K (60.1%), ImageNette (87.2%), and ImageWoof (72.9%). Our code is available on our project page https://jachansantiago.github.io/learnability-guided-distillation/.
Figures & tables
Figure 1: Learnability-Guided Dataset Distillation. We partition the distilled dataset DS into increments {I0,I1,…,Ik} ( Top ). Bottom Left (DiT): Standard distillation generates increments independently, producing redundant samples—a model trained on I0 achieves 98.0% accuracy on I1 , indicating no new information. Bottom Right (LGD): We condition the next increment on the model parameters θI0 to guide synthesis toward samples that complement I0 . The resulting increment I1 achieves only 17.0% accuracy when evaluated by the prior model, indicating it introduces substantial new learning signal.
Figure 2: Cross-validation across distilled data increments ( I1−I5 ) for IPC 50 on ImageNette. Each heatmap shows accuracy when training on one increment (rows) and evaluating on another (columns). DiT [ 23 ] and IGD [ 5 ] exhibit high cross-increment accuracy due to overlapping information, while LGD yields lower off-diagonal scores, indicating more complementary and diverse increments.
Figure 3: Overview of our learnability-guided iterative generation framework. ( Top ) Incremental distillation loop: we iteratively train model θt on cumulative dataset Dt , generate samples using our learnability guidance, select high-quality samples via learnability ranking, and augment the dataset. ( Bottom ) Effect on sample space: The current model θt (green) expands over iterations, while the reference model θ∗ (purple, fixed) defines the learnable region. Generated samples (red × ) land in the learnable gap between boundaries, automatically synthesizing samples that complement the current model’s learned distribution.
Dataset
Model
IPC
Random
DM [ 37 ]
IDC-1 [ 17 ]
DiT [ 23 ]
Minimax [ 10 ]
IGD [ 5 ]
MGD 3 [ 4 ]
LGD (Ours)
Full
Nette
ConvNet-6
50
71.8 ± 1.2
70.3 ± 0.8
72.4 ± 0.7
74.1 ± 0.6
76.9 ± 0.9
80.9 ± 0.9
80.9 ± 2.3
82.6 ± 0.7
94.3 ± 0.5
100
79.9 ± 0.8
78.5 ± 0.8
80.6 ± 1.1
78.2 ± 0.3
81.1 ± 0.3
84.5 ± 0.7
86.5 ± 0.9
87.2 ± 0.7
ResNetAP-10
50
77.3 ± 1.0
76.7 ± 1.0
77.4 ± 0.7
76.9 ± 0.5
78.2 ± 0.7
81.0 ± 1.2
81.2 ± 1.0
84.3 ± 0.5
94.6 ± 0.5
100
81.1 ± 0.6
80.9 ± 0.7
81.5 ± 1.2
80.1 ± 1.1
81.3 ± 0.9
85.2 ± 0.5
85.5 ± 1.0
87.2 ± 0.9
ResNet-18
50
75.8 ± 1.1
75.0 ± 1.0
77.8 ± 0.7
75.2 ± 0.9
78.1 ± 0.6
81.0 ± 0.7
81.5 ± 3.4
85.0 ± 0.9
95.3 ± 0.6
100
82.0 ± 0.4
81.5 ± 0.4
81.7 ± 0.8
77.8 ± 0.6
81.3 ± 0.7
84.4 ± 0.8
85.6 ± 0.2
86.9 ± 0.6
Table 1: Comparison across distilled IPC budgets on Nette and Woof evaluated on different network architectures . Mean ± std accuracy; best per row in bold . IGD, MGD 3 , and LGD used a pretrained DiT as the diffusion backbone.
Figure 4: Incremental training dynamics of DiT and our method. (a-b) show the training loss across successive data increments ( D1→D5 ), where each increment adds new samples followed by a 300-epoch learning-rate decay (light beige). Our method yields stronger loss spikes ( Δ ) after each increment, suggesting the added data is harder and complementary. (c) compares normalized validation accuracy per increment between DiT and our method, highlighting consistent accuracy gains and faster convergence for ours.
SRe 2 L [ 32 ]
G-VBSM [ 19 ]
RDED [ 29 ]
DiT [ 23 ]
Minimax [ 10 ]
DiT-IGD [ 5 ]
MGD 3 [ 4 ]
DiT+LGD (Ours)
46.8 ± 0.2
51.8 ± 0.4
56.5 ± 0.1
52.9 ± 0.6
58.6 ± 0.3
59.8 ± 0.3
60.2 ± 0.1
60.1 ± 0.1
Table 2: ImageNet-1K: Performance comparison over ResNet-18 with state-of-the-art dataset distillation methods.
Evaluated on
IPC
Surrogate
ConvNet-6
ResNetAP-10
ResNet-18
50
ConvNet-6
83.5 ± 0.4
83.5 ± 0.8
83.3 ± 1.0
ResNetAP-10
82.6 ± 0.7
84.7 ± 1.2
85.0 ± 0.9
ResNet-18
81.1 ± 1.1
82.3 ± 0.3
83.6 ± 1.0
100
ConvNet-6
86.0 ± 0.6
86.7 ± 0.9
86.6 ± 0.6
ResNetAP-10
87.2 ± 0.7
87.7 ± 0.6
86.9 ± 0.6
Table 3: Cross-Architecture Evaluation on ImageNette. Accuracy (%) when models trained on surrogate architectures are evaluated on different target architectures. Bold indicates the best within each IPC.
Method
IPC10
IPC20
IPC30
IPC40
IPC50
IPC100
DiT
61.0 ± 1.5
68.2 ± 1.1
73.0 ± 1.7
75.2 ± 1.8
75.6 ± 1.1
78.2 ± 0.7
IGD
64.5 ± 0.9
71.6 ± 1.3
74.8 ± 1.7
78.1 ± 1.2
79.9 ± 0.3
84.3 ± 0.1
LGD (Ours)
64.1 ± 1.3
75.9 ± 0.3
79.3 ± 1.6
83.1 ± 0.8
85.2 ± 0.8
89.1 ± 0.7
Table 4: Progression across data increments on ImageNette. Accuracy at each IPC level as additional increments are sequentially added (IPC10 → IPC100).
Figure 5: Learning-dynamics visualization of original and distilled samples. Each point shows a sample’s mean and standard deviation of ground-truth class probability across training (50 epochs). Top-left points are easy (high μ , low σ2 ), bottom-left are hard, and mid-right indicate informative samples. Our method yields distilled samples that form a more informative (16.2%) and harder dataset (2.6%), roughly 3 × and 2 × over IGD, respectively, aligning more closely with the original training dynamics distribution, as shown by the lowest Jensen–Shannon divergence (JS ↓ ).
Figure 6: In-distribution and learning-dynamics analysis of distilled datasets. Each point represents a sample described by its ground-truth (GT) class probability from the reference model ( x -axis, measuring in-distribution likelihood p(y∣x) ) and its mean GT probability across training epochs ( y -axis, reflecting sample difficulty). Original samples (gray) delineate the in-distribution region, where higher y indicates easier examples. DiT concentrates on easy, high-confidence samples; DiT+IGD extends slightly toward mid and hard areas; DiT+Loss covers a broader difficulty range but introduces several non-representative (low- x ) samples; DiT+Ours achieves a balanced spread—capturing informative mid and hard examples while remaining closely aligned with the in-distribution region.
Figure 7: Visual diversity in incrementally distilled datasets. Samples from increments I1−I5 (50 IPC total) on the Church class. DiT generates repetitive samples with similar architectural styles and lighting. IGD improves slightly but exhibits clustering around Gothic exteriors. LGD (Ours) produces diverse samples spanning multiple architectural styles (traditional, modern, ornate), perspectives (exterior and interior views), and lighting conditions (day, night, golden hour). This diversity reflects our curriculum-based synthesis: early increments capture simpler structures while later increments progressively introduce architectural complexity and challenging interior scenes, reducing redundancy and maximizing sample utility across the distilled dataset.
Dataset
Model
IPC
Minimax [ 10 ]
Minimax-IGD [ 5 ]
Minimax+Ours
Full
Nette
ConvNet-6
50
76.9 ± 0.9
80.6 ± 0.8
77.8 ± 1.0
94.3 ± 0.5
100
81.1 ± 0.3
85.1 ± 0.5
85.1 ± 0.7
ResNetAP-10
50
78.2 ± 0.7
81.5 ± 0.3
82.2 ± 0.4
94.6 ± 0.5
100
81.3 ± 0.9
85.6 ± 0.3
86.4 ± 0.4
ResNet-18
50
78.1 ± 0.6
82.2 ± 0.4
82.5 ± 0.4
95.3 ± 0.6
100
81.3 ± 0.7
85.3 ± 1.0
87.2 ± 0.8
Table 5: Comparison across distilled IPC budgets with Minimax Diffusion [ 10 ] on Nette and Woof Mean ± std accuracy; best per row is shown in bold . Full represents training with the entire dataset.
Figure 8: Cross-increment redundancy analysis. Errors on increment It+1 using a model trained only on It . DiT and IGD show <15 errors (high redundancy); our method shows 22-88 errors, confirming reduced redundancy.
Figure 9: Incremental accuracy across different seed initializations on our method. Top-1 accuracy as the distilled dataset grows from 10 to 50 IPC. Three seed sources are compared: IGD [ 5 ] (green), DiT [ 23 ] (red), and random real images (blue). IGD initialization achieves the best final performance (84.4%), but all seeds substantially outperform previous SOTA (81.2%, dashed line), demonstrating the robustness of our learnability-guided framework to initialization choice.
Figure 10: Hyperparameter sensitivity on ImageNette. Top-1 accuracy (%) as we vary (a) reference weight ω , (b) guidance strength λ , and (c) deviation strength γ across different IPC settings.
κ
IPC 20
IPC 30
IPC 40
IPC 50
IPC 100
1
77.5±0.5
79.9±0.3
83.2±0.8
84.4±1.6
88.5±1.1
3
77.3±1.0
80.9±0.7
83.3±0.6
85.1±0.8
88.4±1.4
10
77.8±0.4
81.1±1.5
83.9±0.8
85.6±0.5
89.1±0.6
Table 6: Experiment with Batch Size for Learnability Rank on ImageNette. Top-1 accuracy (%) for different batch sizes and IPC settings.
Method
IPC 20
IPC 30
IPC 40
IPC 50
DiT (Baseline)
68.2 ± 1.1
73.0 ± 1.7
75.2 ± 1.8
75.6 ± 1.1
+ Learnability Guidance
75.2 ± 0.7
77.8 ± 0.2
79.9 ± 0.7
82.5 ± 0.3
+ Deviation Guidance
77.5 ± 0.5
79.9 ± 0.3
83.2 ± 0.8
84.4 ± 1.6
+ Learnability Ranking (Ours)
77.8 ± 0.4
81.1 ± 1.5
83.9 ± 0.8
85.6 ± 0.5
Improvements over baseline:
Learnability Guidance
+7.0
+4.8
+4.7
+6.9
Table 7: Component ablation study on ImageNette. We progressively add components starting from DiT baseline. Mean ± std accuracy (%) across 3 runs with ResNet-AP-10.
Config
Value
Explanation
(a) Diffusion Synthesis
Sampler
DDPM
–
Sampling Steps
50
–
CFG Scale
4.0
Classifier-free guidance
(b) Learnability Guidance
Learner Arch.
ResNet-10-AP
–
Table 8: Hyperparameter settings for data synthesis and learnability scoring.
Figure 11: Visual diversity in incrementally distilled datasets. Samples from increments I1−I5 (50 IPC total) of the Parachute class.
Figure 12: Visual diversity in incrementally distilled datasets. Samples from increments I1−I5 (50 IPC total) of the Tench class.
Figure 13: Visual diversity in incrementally distilled datasets. Samples from increments I1−I5 (50 IPC total) of the Australian terrier class.
Figure 14: Visual diversity in incrementally distilled datasets. Samples from increments I1−I5 (50 IPC total) of the Dingo class.
Dataset distillation enables efficient training by distilling the information of large-scale datasets into significantly smaller synthetic datasets. Diffusion based paradigms have emerged in recent years, offering novel perspectives for dataset distillation. However, they typically necessitate additional fine-tuning stages, and effective guidance mechanisms remain underexplored. To address these limitations, we rethink diffusion based dataset distillation and propose a Dual Matching Guided Diffusion (DMGD) framework, centered on efficient training-free guidance. We first establish Semantic Matching via conditional likelihood optimization, eliminating the need for auxiliary classifiers. Furthermore, we propose a dynamic guidance mechanism that enhances the diversity of synthetic data while maintaining semantic alignment. Simultaneously, we introduce an optimal transport (OT) based Distribution Matching approach to further align with the target distribution structure. To ensure efficiency, we develop two enhanced strategies for diffusion based framework: Distribution Approximate Matching and Greedy Progressive Matching. These strategies enable effective distribution matching guidance with minimal computational overhead. Experimental results on ImageNet-Woof, ImageNet-Nette, and ImageNet-1K demonstrate that our training-free approach achieves significant improvements, outperforming state-of-the-art (SOTA) methods requiring additional fine-tuning by average accuracy gains of 2.1%, 5.4%, and 2.4%, respectively.
Qichao Wang, Yunhong Lu, Hengyuan Cao +2
Zhejiang University · Shanghai Institute for Advanced Study-Zhejiang University · Shanghai Institute for Mathematics and Interdisciplinary Sciences
Dataset Distillation aims to synthesize compact datasets that can approximate the training efficacy of large-scale real datasets, offering an efficient solution to the increasing computational demands of modern deep learning. Recently, diffusion-based dataset distillation methods have shown great promise by leveraging the strong generative capacity of diffusion models to produce diverse and structurally consistent samples. However, a fundamental goal misalignment persists: diffusion models are optimized for generative likelihood rather than discriminative utility, resulting in over-concentration in high-density regions and inadequate coverage of boundary samples crucial for classification. To address this issue, we propose two complementary strategies. Inversion-Matching (IM) introduces an inversion-guided fine-tuning process that aligns denoising trajectories with their inversion counterparts, broadening distributional coverage and enhancing diversity. Selective Subgroup Sampling(S^3) is a training-free sampling mechanism that improves inter-class separability by selecting synthetic subsets that are both representative and distinctive. Extensive experiments demonstrate that our approach significantly enhances the discriminative quality and generalization of distilled datasets, achieving state-of-the-art performance among diffusion-based methods.
Chenru Wang, Yunyi Chen, Zijun Yang +2
AGI Lab, Westlake University · Eindhoven University of Technology · Jinan University +2
Dataset distillation provides an effective approach to reduce memory and computational costs by optimizing a compact dataset that achieves performance comparable to the full original. However, for large-scale datasets and complex deep networks (e.g., ImageNet-1K with ResNet-101), the vast optimization space hinders distillation effectiveness, limiting practical applications. Recent methods leverage pre-trained diffusion models to directly generate informative images, thereby bypassing pixel-level optimization and achieving promising results. Nonetheless, these approaches often suffer from distribution shifts between the pre-trained diffusion prior and target datasets, as well as the need for multiple distillation steps under varying settings. To overcome these challenges, we propose a novel framework that is orthogonal to existing diffusion-based distillation techniques by utilizing the diffusion prior for patch selection rather than generation. Our method predicts noise from the diffusion model conditioned on input images and optional text prompts (with or without label information), and computes the associated loss for each image-patch pair. Based on the loss differences, we identify distinctive regions within the original images. Furthermore, we apply intra-class clustering and ranking on the selected patches to enforce diversity constraints. This streamlined pipeline enables a one-step distillation process. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art methods across various metrics and settings.
Xinhao Zhong, Shuoyang Sun, Zhaoyang Xu +4
Harbin Institute of Technology, Shenzhen · Shanghai Jiao Tong University