Accelerating Large-Scale Dataset Distillation via Exploration-Exploitation Optimization

Muhammad J. Alahmadi; Peng Gao; Feiyi Wang; Dongkuan Xu

cs.CVFeb 17, 2026

Accelerating Large-Scale Dataset Distillation via Exploration-Exploitation Optimization

Authors: Muhammad J. Alahmadi, Peng Gao, Feiyi Wang, Dongkuan Xu

Organizations: North Carolina State University · King Abdulaziz University · Oak Ridge National Laboratory

Abstract

Dataset distillation compresses the original data into compact synthetic datasets, reducing training time and storage while retaining model performance, enabling deployment under limited resources. Although recent decoupling-based distillation methods enable dataset distillation at large scale, they continue to face an efficiency gap: optimization-based decoupling methods achieve higher accuracy but demand intensive computation, whereas optimization-free decoupling methods are efficient but sacrifice accuracy. To overcome this trade-off, we propose Exploration--Exploitation Distillation (E $^2$ D), a simple, practical method that minimizes redundant computation through an efficient pipeline that begins with full-image initialization to preserve semantic integrity and feature diversity. It then uses a two-phase optimization strategy: an exploration phase that performs uniform updates and identifies high-loss regions, and an exploitation phase that focuses updates on these regions to accelerate convergence. We evaluate E $^2$ D on large-scale benchmarks, surpassing the state-of-the-art on ImageNet-1K while being $18\times$ faster, and on ImageNet-21K, our method substantially improves accuracy while remaining $4.3\times$ faster. These results demonstrate that targeted, redundancy-reducing updates, rather than brute-force optimization, bridge the gap between accuracy and efficiency in large-scale dataset distillation. Code is available at https://github.com/ncsu-dk-lab/E2D.

Figures & tables

Figure 1: Comparison of Top-1 accuracy and synthesis time on ImageNet-1K using ResNet-18 for various dataset distillation methods at IPC 10 and IPC 50. Synthesis time is measured on a single RTX A6000 GPU. Our method converges substantially faster and achieves the highest accuracy, leading to the best accuracy–efficiency trade‑off.

Figure 2: Semantic cosine similarity across ImageNet‑1K classes at IPC 50 using a ResNet‑18 teacher. Lower values indicate greater diversity and reduced redundancy; our method consistently achieves the lowest similarity.

Panel of Figure 3 — Figure 3: Visual comparison of synthetic data generated by SRe 2 L, RDED, DELLT, EDC, and our method E 2 D, which produces more diverse, less redundant samples and preserves semantic integrity with full‑size feature representations.

Figure 4: Overview of our proposed method E 2 D. The pipeline consists of four components: (1) Full-size Image Initialization , which preserves the semantic and structural information of the original data, preventing distortion or redundancy; (2) Exploration Phase , which identifies challenging high-loss regions and ensure balanced optimization; (3) Exploitation Phase , which iteratively refines these challenging regions for efficient convergence; and (4) Accelerated Learning Schedule , applied during student training to further speed up convergence. Together, these components enable fast and effective dataset distillation with minimal redundancy.

Figure 5: Cosine similarity trends across optimization steps.

Panel of Figure 6 — Figure 6: Top‑1 accuracy vs. speedup for our method and baseline EDC under two IPC settings. Our method converges faster and starts from a stronger no‑optimization starting point. In our method, extra optimization past convergence degrades performance, counter to the common assumption that more optimization is always better.

Table 5: Synthesis time and acceleration factor comparison. Each subtable reports synthesis time (hours) and speedup over baseline on a single NVIDIA RTX A6000 GPU with ResNet‑18, evaluated on different datasets.

Synthesis		Relabeling & Evaluation
Iteration	200	Epochs	300
Optimizer	AdamW $\beta_{1}=0.5$ , $\beta_{2}=0.9$	Optimizer	AdamW
		Loss Type	MSE-GT + 0.025·CE
Learning rate	0.05	Learning rate	0.001
Batch Size	80	Batch Size	100
$K$ , $\epsilon$	140, 0.5	LR Schedule	SSRS, $\zeta=2$ SmoothingLR at IPC = 1

Table 9: Hyperparameter settings for ImageNet-1K.

Synthesis		Relabeling & Evaluation
Iteration	400	Epochs	300
Optimizer	AdamW $\beta_{1}=0.5$ , $\beta_{2}=0.9$	Optimizer	AdamW
		Loss Type	KL-Div
Learning rate	0.005	Learning rate	0.005
Batch Size	80	Batch Size	100
$K$ , $\epsilon$	280, 0.3	LR Schedule	SSRS, $\zeta=2$

Table 10: Hyperparameter settings for ImageNet-21K.

	Run 1	Run 2	Run 3
Ours 10×	10.89	12.29	12.90

Table 13: Performance across three independent runs of our method with different RDED initialization seeds on ImageNet-1K at IPC = 1 using a ResNet-18 architecture.

Method	IPC
Method	10	50
EDC	44.5	57.1
Ours	47.9	58.4

Table 14: Effect of initialization strategy. Accuracy on ImageNet‑1K when comparing the patch‑based initialization of EDC with the full‑image initialization adopted in our method (E 2 D) across two IPC settings.

Figure 7: t-SNE visualization of logit embeddings for synthetic data generated by our method and state-of-the-art baselines on ImageNet‑1K. Our method exhibits broader dispersion and clearer clusters, indicating better generalization and stronger class separability. Outlier points outside class clusters are highlighted by yellow circles.

Method	IPC 10	IPC 50
EDC (reported)	$63.7\pm 0.3$	$68.6\pm 0.2$
EDC (reproduced)
No optimization	$65.1\pm 0.2$	$68.5\pm 0.2$
With optimization	$65.2\pm 0.1$	$68.8\pm 0.2$

Table 15: Effect of optimization iterations in EDC on CIFAR-100. The top row reports the value from the original paper; the last two rows are our reproduced results. Final accuracy remains stable even without optimization, indicating that the relabeling stage alone suffices for low-resolution datasets.

Figure 8: Visualization of synthetic images produced by our method on ImageNet-1K.

Figure 9: Visualization of synthetic images produced by our method on ImageNet-21K.

Dataset Distillation

Dataset distillation compresses training data into a small synthetic or selected set that retains the information needed to train useful models.

See all 65 papers

Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment
Mar 30, 2026Tongfei Liu, Yufan Liu, Bing Li +1Dataset Distillation Dataset Condensation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chines Academy of Sciences, 95 Zhongguancun East Road, Beijing, 100190, Beijing, China. · School of Artificial Intelligence, University of Chinese Academy of Sciences, 1 Yanqihu East Road, Beijing, 101408, Beijing, China. · PeopleAI, Inc., 68 Zhichun Road, Beijing, 100098, Beijing, China. +1
FD $^2$ : A Dedicated Framework for Fine-Grained Dataset Distillation
Mar 26, 2026Hongxu Ma, Guang Li, Shijie Wang +5Fine-Grained Image Classification Dataset Distillation
Zhejiang University · Hokkaido University · The University of Queensland +2
Learnability-Guided Diffusion for Dataset Distillation
Apr 1, 2026Jeffrey A. Chan-Santiago, Mubarak ShahDataset Distillation Diffusion-Based Dataset Distillation
Institute of Artificial Intelligence, University of Central Florida

Mar 30, 2026J/K move · Enter open · S save

Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment

Tongfei Liu, Yufan Liu, Bing Li, Weiming Hu

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chines Academy of Sciences, 95 Zhongguancun East Road, Beijing, 100190, Beijing, China. · School of Artificial Intelligence, University of Chinese Academy of Sciences, 1 Yanqihu East Road, Beijing, 101408, Beijing, China. · PeopleAI, Inc., 68 Zhichun Road, Beijing, 100098, Beijing, China.+1

Dataset Distillation Dataset Condensation Diffusion-Based Dataset Distillation Distribution Matching Diffusion Models Synthetic Data Augmentation

Open paper AskPDF arXiv

The high cost and accessibility problem associated with large datasets hinder the development of large-scale visual recognition systems. Dataset Distillation addresses these problems by synthesizing compact surrogate datasets for efficient training, storage, transfer, and privacy preservation. The existing state-of-the-art diffusion-based dataset distillation methods face three issues: lack of theoretical justification, poor efficiency in scaling to high data volumes, and failure in data-free scenarios. To address these issues, we establish a theoretical framework that justifies the use of diffusion models by proving the equivalence between dataset distillation and distribution matching, and reveals an inherent efficiency limit in the dataset distillation paradigm. We then propose a Dataset Concentration (DsCo) framework that uses a diffusion-based Noise-Optimization (NOpt) method to synthesize a small yet representative set of samples, and optionally augments the synthetic data via "Doping", which mixes selected samples from the original dataset with the synthetic samples to overcome the efficiency limit of dataset distillation. DsCo is applicable in both data-accessible and data-free scenarios, achieving SOTA performances for low data volumes, and it extends well to high data volumes, where it nearly reduces the dataset size by half with no performance degradation.

Abstract

Dataset Distillation

Explore similar work

Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment

FD2^22: A Dedicated Framework for Fine-Grained Dataset Distillation

Learnability-Guided Diffusion for Dataset Distillation

FD $^2$ : A Dedicated Framework for Fine-Grained Dataset Distillation