Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment

Tongfei Liu; Yufan Liu; Bing Li; Weiming Hu

cs.CVMar 30, 2026

Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment

Authors: Tongfei Liu, Yufan Liu, Bing Li, Weiming Hu

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chines Academy of Sciences, 95 Zhongguancun East Road, Beijing, 100190, Beijing, China. · School of Artificial Intelligence, University of Chinese Academy of Sciences, 1 Yanqihu East Road, Beijing, 101408, Beijing, China. · PeopleAI, Inc., 68 Zhichun Road, Beijing, 100098, Beijing, China. · School of Information Science and Technology, ShanghaiTech University, 393 Huaxia Middle Road, Shanghai, 201210, Shanghai, China.

Abstract

The high cost and accessibility problem associated with large datasets hinder the development of large-scale visual recognition systems. Dataset Distillation addresses these problems by synthesizing compact surrogate datasets for efficient training, storage, transfer, and privacy preservation. The existing state-of-the-art diffusion-based dataset distillation methods face three issues: lack of theoretical justification, poor efficiency in scaling to high data volumes, and failure in data-free scenarios. To address these issues, we establish a theoretical framework that justifies the use of diffusion models by proving the equivalence between dataset distillation and distribution matching, and reveals an inherent efficiency limit in the dataset distillation paradigm. We then propose a Dataset Concentration (DsCo) framework that uses a diffusion-based Noise-Optimization (NOpt) method to synthesize a small yet representative set of samples, and optionally augments the synthetic data via "Doping", which mixes selected samples from the original dataset with the synthetic samples to overcome the efficiency limit of dataset distillation. DsCo is applicable in both data-accessible and data-free scenarios, achieving SOTA performances for low data volumes, and it extends well to high data volumes, where it nearly reduces the dataset size by half with no performance degradation.

Figures & tables

Figure 1: The graphical illustration of the target features and surrogate features plotted in the RKHS.

Figure 2: The Dataset Concentration framework. Enclosed in the gray region is the iterative denoising process of NOpt; the dashed red arrows indicate the gradient flow during each noise-optimization.

ImageNette
Model	ConvNet6			ResNetAP-10			ResNet-18
Full	94.3 $\pm$ 0.5			94.6 $\pm$ 0.5			95.3 $\pm$ 0.6
IPC	10	50	100	10	50	100	10	50	100
Data-Accessible
Random	46.0 $\pm$ 0.5	71.8 $\pm$ 1.2	79.9 $\pm$ 0.8	54.2 $\pm$ 1.2	77.3 $\pm$ 1.0	81.1 $\pm$ 0.6	55.8 $\pm$ 1.0	75.8 $\pm$ 1.1	82.0 $\pm$ 0.4
DM	49.8 $\pm$ 1.1	70.3 $\pm$ 0.8	78.5 $\pm$ 0.8	60.2 $\pm$ 0.7	76.7 $\pm$ 1.1	80.9 $\pm$ 0.7	60.9 $\pm$ 0.7	75.0 $\pm$ 1.0	81.5 $\pm$ 0.4

Table 1: Performance comparison on ImageNette under low-IPC setting. Best performances are denoted in bold .

ImageWoof
Model	ConvNet6			ResNetAP-10			ResNet-18
Full	85.9 $\pm$ 0.4			87.2 $\pm$ 0.6			89.0 $\pm$ 0.6
IPC	10	50	100	10	50	100	10	50	100
Data-Accessible
Random	25.2 $\pm$ 1.1	41.9 $\pm$ 1.4	52.3 $\pm$ 1.5	31.6 $\pm$ 0.8	50.1 $\pm$ 1.6	59.2 $\pm$ 0.9	30.9 $\pm$ 1.3	54.0 $\pm$ 0.8	63.6 $\pm$ 0.5
DM	27.6 $\pm$ 1.2	43.8 $\pm$ 0.8	50.1 $\pm$ 0.9	29.8 $\pm$ 1.0	47.8 $\pm$ 1.2	59.8 $\pm$ 1.3	30.2 $\pm$ 0.6	53.9 $\pm$ 0.7	64.9 $\pm$ 0.7

Table 2: Performance comparison on ImageWoof under low-IPC setting.Best performances are denoted in bold .

Dataset	ImageNette
IPC	200	300	400	700	1000	1300	Full
Ratio%	15.5	23.3	31.0	54.3	77.6	100.8	Full
Model	ConvNet6
Random	92.6 $\pm$ 0.8	92.6 $\pm$ 1.1	93.1 $\pm$ 0.7	94.7 $\pm$ 0.4	95.9 $\pm$ 0.2	96.5 $\pm$ 0.2	96.5 $\pm$ 0.2
DsCo	94.0 $\pm$ 0.7	94.2 $\pm$ 0.5	94.5 $\pm$ 0.6	95.7 $\pm$ 0.5	96.5 $\pm$ 0.2	96.6 $\pm$ 0.2	96.5 $\pm$ 0.2
Model	ResNetAP-10

Table 3: Validation Accuracy vs IPC for Dataset Concentration on ImageNette.

Dataset	ImageWoof
IPC	200	300	400	700	1000	1300	Full
Ratio%	16.0	24.1	32.1	56.2	80.3	104.4	Full
Model	ConvNet6
Random	80.2 $\pm$ 1.0	80.1 $\pm$ 1.0	83.1 $\pm$ 0.5	87.4 $\pm$ 0.6	87.4 $\pm$ 0.1	88.3 $\pm$ 0.3	88.3 $\pm$ 0.3
DsCo	81.0 $\pm$ 0.5	82.7 $\pm$ 0.5	84.0 $\pm$ 0.7	88.4 $\pm$ 0.4	88.3 $\pm$ 0.4	88.3 $\pm$ 0.2	88.3 $\pm$ 0.3
Model	ResNetAP-10

Table 4: Validation Accuracy vs IPC for Dataset Concentration on ImageWoof.

Method	ResNet-18		MobileNet-V2		EfficientNet-B0
Method	IPC10	IPC50	IPC10	IPC50	IPC10	IPC50
Data-Accessible
RDED	42.0 $\pm$ 0.1	56.5 $\pm$ 0.1	40.4 $\pm$ 0.1	53.3 $\pm$ 0.2	31.0 $\pm$ 0.1	58.5 $\pm$ 0.4
DiT-IGD	45.5 $\pm$ 0.5	59.8 $\pm$ 0.3	39.2 $\pm$ 0.2	57.8 $\pm$ 0.2	47.7 $\pm$ 0.1	62.0 $\pm$ 0.1
Minimax-IGD	46.2 $\pm$ 0.6	60.3 $\pm$ 0.4	39.7 $\pm$ 0.4	58.5 $\pm$ 0.3	48.5 $\pm$ 0.1	62.7 $\pm$ 0.2
DsCo(ours)	47.0 $\pm$ 0.2	60.4 $\pm$ 0.1	42.3 $\pm$ 0.2	59.0 $\pm$ 0.2	50.9 $\pm$ 0.1	63.0 $\pm$ 0.1

Table 5: Low-IPC Performance comparison on ImageNet-1k. The best performances are denoted in bold .

Figure 3: ResNet-18 Accuracy vs IPC for ImageNet-1k. The sub-figure enclosed in the major figure depicts the 10IPC and 50IPC performances, which are commonly reported in the dataset distillation literature.

Model	DsCo Accuracy	Full Accuracy
ResNet-18	69.0 $\pm$ 0.1	69.0 $\pm$ 0.1
ResNet-34	69.1 $\pm$ 0.1	69.1 $\pm$ 0.2
ResNet-101	70.1 $\pm$ 0.1	70.1 $\pm$ 0.1
EfficientNet-B0	66.4 $\pm$ 0.2	66.8 $\pm$ 0.3
DenseNet-121	67.2 $\pm$ 0.2	67.6 $\pm$ 0.2

Table 6: ImageNet-1k cross-architecture performance of the 700IPC concentrated dataset on ResNet-18, ResNet-34, ResNet-101, EfficientNet-B0, and DenseNet-121.

Components		ImageNette, ResNetAP-10
$\mathcal{L}_{\mathrm{align}}$	$\mathcal{L}_{\mathrm{real}}$	Acc@10	Acc@50	Acc@100
$\times$	$\times$	64.8	75.5	80.5
$\times$	$\checkmark$	64.6	75.6	81.1
$\checkmark$	$\times$	68.9	81.4	85.5
$\checkmark$	$\checkmark$	69.8	84.0	86.9

Table 7: Component ablation analysis for low-IPC Dataset Concentration

Components		ImageNette, ResNet-18
NOpt	Doping	200	300	400	700
$\times$	$\times$	95.1	95.6	96.1	97.0
$\times$	$\checkmark$	95.5	95.4	96.4	97.3
$\checkmark$	$\times$	95.4	95.5	96.4	97.2
$\checkmark$	$\checkmark$	96.0	96.1	96.9	97.9

Table 8: Stage ablation analysis for high-IPC Dataset Concentration

Components			ImageNette, ResNetAP-10
$\mathcal{L}_{\mathrm{stats}}$	$\mathcal{L}_{\mathrm{maxoc}}$	$\mathcal{L}_{\mathrm{real}}$	Acc@10IPC	Acc@50IPC
$\checkmark$	$\times$	$\times$	60.3	76.2
$\times$	$\checkmark$	$\times$	61.7	74.1
$\times$	$\times$	$\checkmark$	58.5	74.5
$\times$	$\checkmark$	$\checkmark$	61.5	74.9
$\checkmark$	$\times$	$\checkmark$	60.9	74.6

Table 9: Component ablation analysis of Data-Free Dataset Concentration. In this experiment, the evaluation model is trained with the SGD Bottou et al. (2018) optimizer.

$\lambda_{\mathrm{align}}$	ImageNette, ResNetAP-10
$\lambda_{\mathrm{align}}$	Acc@10	Acc@50	Acc@100
5e-2	58.6	80.8	80.6
5e-3	66.6	81.6	86.2
5e-4	69.8	84.0	86.9
5e-5	69.0	83.0	85.8
5e-6	67.8	78.4	82.2

Table 10: Hyperparameter sensitivity analysis for low-IPC Dataset Concentration

$\lambda_{\mathrm{stats}}$	ImageNette, ResNetAP-10
$\lambda_{\mathrm{stats}}$	Acc@10	Acc@50
1.0	58.4	72.0
0.1	61.0	73.9
0.01	64.6	78.2
1e-3	65.0	78.7
1e-4	63.8	76.1

Table 11: Hyperparameter sensitivity analysis for low-IPC Dataset Concentration in data-free scenario (DFDsCo). In this experiment, the models are trained with the SGD Bottou et al. (2018) optimizer.

Method	Preparation	Generation			Total			Peak Memory
Method	Preparation	IPC10	IPC50	IPC100	IPC10	IPC50	IPC100	Peak Memory
Minimax	253h	7h	35h	70h	260h	288h	323h	14.8G
DiT-IGD	124h	69h	347h	694h	193h	471h	818h	13.2G
DsCo	14h	28h	62h	119h	42h	76h	133h	4.3G
DFDsCo	142h	15h	53h	106h	157h	195h	247h	4.7G

Table 12: Low-IPC synthesis cost analysis of ImageNet-1k on NVIDIA GeForce RTX2080 Ti GPUs.

Figure 4: Synthesis costs, measured in running time on NVIDIA GeForce RTX 2080 Ti GPUs, plotted against IPC for contemporary open-source diffusion-based dataset distillation or concentration methods. The dashed lines are estimated costs obtained through linear extrapolation.

Figure 5: Visualization of the real samples (left), DiT synthetic samples (middle left), DsCo synthetic samples (middle right), and the DFDsCo synthetic samples (right) of the Church class of the ImageNette dataset.

Figure 6: The t-SNE distribution visualization for the ’church’ (left) and ’parachute’ (right) classes of ImageNette. The latent codes of the synthetic samples are compressed to 2-D and plotted on top of the compressed latent codes of real samples. The synthetic samples are synthesized with DiT (top), DFDsCo (middle), and DsCo (bottom).

Figure 7: The real samples with increasing confusion scores for the ’church’ and ’parachute’ classes of the ImageNette dataset.

Figure 8: The average intra-class latent-space nearest-neighbor mutual L-2 distance for five groups of target samples of descending confusion scores computed within each group (diagonal) and between different groups (off-diagonal).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

$N$	10	50	100
mean	6.5	31.7	63.4
std	1.0	2.2	3.2

Appendix

Table 13: The means and standard deviations of the number of occupied regions.

Dataset Distillation

Dataset distillation compresses training data into a small synthetic or selected set that retains the information needed to train useful models.

See all 65 papers

Learnability-Guided Diffusion for Dataset Distillation
Apr 1, 2026Jeffrey A. Chan-Santiago, Mubarak ShahDataset Distillation Diffusion-Based Dataset Distillation
Institute of Artificial Intelligence, University of Central Florida
Accelerating Large-Scale Dataset Distillation via Exploration-Exploitation Optimization
Feb 17, 2026Muhammad J. Alahmadi, Peng Gao, Feiyi Wang +1Dataset Distillation
North Carolina State University · King Abdulaziz University · Oak Ridge National Laboratory
DMGD: Train-Free Dataset Distillation with Semantic-Distribution Matching in Diffusion Models
May 5, 2026Qichao Wang, Yunhong Lu, Hengyuan Cao +2Diffusion-Based Dataset Distillation Diffusion Model Guidance
Zhejiang University · Shanghai Institute for Advanced Study-Zhejiang University · Shanghai Institute for Mathematics and Interdisciplinary Sciences

Apr 1, 2026J/K move · Enter open · S save

Learnability-Guided Diffusion for Dataset Distillation

Jeffrey A. Chan-Santiago, Mubarak Shah

Institute of Artificial Intelligence, University of Central Florida

Dataset Distillation Diffusion-Based Dataset Distillation

Open paper AskPDF arXiv

Training machine learning models on massive datasets is expensive and time-consuming. Dataset distillation addresses this by creating a small synthetic dataset that achieves the same performance as the full dataset. Recent methods use diffusion models to generate distilled data, either by promoting diversity or matching training gradients. However, existing approaches produce redundant training signals, where samples convey overlapping information. Empirically, disjoint subsets of distilled datasets capture 80-90% overlapping signals. This redundancy stems from optimizing visual diversity or average training dynamics without accounting for similarity across samples, leading to datasets where multiple samples share similar information rather than complementary knowledge. We propose learnability-driven dataset distillation, which constructs synthetic datasets incrementally through successive stages. Starting from a small set, we train a model and generate new samples guided by learnability scores that identify what the current model can learn from, creating an adaptive curriculum. We introduce Learnability-Guided Diffusion (LGD), which balances training utility for the current model with validity under a reference model to generate curriculum-aligned samples. Our approach reduces redundancy by 39.1%, promotes specialization across training stages, and achieves state-of-the-art results on ImageNet-1K (60.1%), ImageNette (87.2%), and ImageWoof (72.9%). Our code is available on our project page https://jachansantiago.github.io/learnability-guided-distillation/.

Abstract

Dataset Distillation

Explore similar work

Learnability-Guided Diffusion for Dataset Distillation

Accelerating Large-Scale Dataset Distillation via Exploration-Exploitation Optimization

DMGD: Train-Free Dataset Distillation with Semantic-Distribution Matching in Diffusion Models