Disentangled Embedding Optimization for Diffusion-Based Dataset Distillation
Authors: Hongcheng Li, Yucan Zhou, Yibin Wang, Xiaoyan Gu, Bo Li, Weiping Wang
Organizations: Institute of Information Engineering, Chinese Academy of Sciences & School of Cyber Security, University of Chinese Academy of Sciences & State Key Laboratory of Cyberspace Security Defense, Beijing, China · College of Intelligence and Computing, Tianjin University, Tianjin, China · Institute of Information Engineering, Chinese Academy of Sciences State Key Laboratory of Cyberspace Security Defense, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
Dataset distillation aims to compress large-scale datasets into compact synthetic subsets that enable efficient model training with comparable performance to the original data. Recent advances show that diffusion models are highly effective for this task due to their strong generative capability. Existing diffusion-based approaches typically perform class-wise clustering and use the obtained cluster prototypes as diffusion inputs to generate distilled samples. However, the uniformly distributed original samples in a category inevitably lead to entanglement near cluster boundaries, pulling the cluster prototypes closer and reducing the inter-cluster separation. Consequently, the diffusion model receives overly similar latent inputs, resulting in synthetic data with limited diversity. To address this issue, we propose Disentangled Embedding Optimization (DEO), a simple yet effective framework that replaces static clustering with trainable embeddings. These embeddings are optimized to match the Gaussian statistics (i.e., mean and covariance) of the original data distribution, while a diversity regularizer is designed to encourage separation among these embeddings to improve the intra-class variability. Extensive experiments demonstrate that our proposed DEO can generate more diversified and representative distilled data than existing cluster-based diffusion methods, achieving state-of-the-art performance and superior generalization across multiple datasets.
Figures & tables
Figure 1. Illustration of the entanglement issue in distilled data generation. (a) t-SNE visualization shows that entangled sub-cluster prototypes in the original data reduce separability. (b) These less-separable latent inputs for the diffusion model result in indistinguishable synthesized images.
Figure 2. Overview of our Disentangled Embedding Optimization (DEO) for diffusion-based distillation. ( a ) Learnable latent embeddings Zc are optimized to match the mean and covariance of the original data, with a diversity loss to encourage separation. ( b ) The optimized Zc and text prompts are fed into the diffusion model to generate distilled samples S .
Method
ImageWoof
ImageNette
ImageIDC
IPC=10
IPC=20
IPC=50
IPC=10
IPC=20
IPC=50
IPC=10
IPC=20
IPC=50
Random
29.4 ± 0.8
32.7 ± 0.4
47.2 ± 1.3
54.2 ± 1.6
63.5 ± 0.5
76.1 ± 1.1
48.1 ± 0.8
52.5 ± 0.9
68.1 ± 0.7
K-Center
22.1 ± 0.1
25.1 ± 0.7
40.6 ± 0.4
–
–
–
–
–
–
Herding
32.0 ± 0.3
34.9 ± 0.1
49.1 ± 0.7
–
–
–
–
–
–
DiT
34.7 ± 0.5
41.1 ± 0.8
49.3 ± 0.2
59.1 ± 0.7
64.8 ± 1.2
73.3 ± 0.9
54.1 ± 0.4
58.9 ± 0.2
64.3 ± 0.6
DM
30.3 ± 1.2
35.2 ± 0.6
47.1 ± 1.1
60.8 ± 0.6
66.5 ± 1.1
76.2 ± 0.4
52.8 ± 0.5
58.5 ± 0.4
69.1 ± 0.8
Table 1. Performance comparison on ImageWoof, ImageNette, and ImageIDC with ResNetAP-10 under different IPC settings. The best and second-best results are highlighted in bold and underline , respectively; “–” denotes unavailable results.
IPC
SRe2L
RDED
DiT
Minimax
VLCP
DEO (Ours)
10
21.3 ± 0.6
42.0 ± 0.1
39.6 ± 0.4
44.3 ± 0.5
46.7 ± 0.4
49.2 ± 0.8
50
46.8 ± 0.2
56.5 ± 0.1
52.9 ± 0.6
58.6 ± 0.3
60.5 ± 0.2
61.4 ± 0.3
Table 2. Performance comparison on ImageNet-1K with IPC=10/50.
Dataset
IPC
SRe2L
RDED
VLCP
DEO (Ours)
CIFAR-10
10
29.3 ± 0.5
37.1 ± 0.3
39.0 ± 0.7
41.8 ± 0.6
50
45.0 ± 0.7
62.1 ± 0.1
63.2 ± 0.3
64.1 ± 0.7
CIFAR-100
10
27.0 ± 0.4
42.6 ± 0.2
50.6 ± 0.7
52.4 ± 0.3
50
50.2 ± 0.4
62.6 ± 0.1
66.1 ± 0.3
66.8 ± 0.5
Table 3. Performance comparison on CIFAR-10 and CIFAR-100 with IPC=10/50.
Backbone
IPC
Random
K-Center
Herding
DiT
DM
IDC-1
Minimax
D 4 M
VLCP
DEO (Ours)
ConvNet
10
24.3 ± 1.1
19.4 ± 0.9
26.7 ± 0.5
34.2 ± 1.1
26.9 ± 1.2
33.3 ± 1.1
33.3 ± 1.7
29.4 ± 0.9
34.8 ± 2.4
36.2 ± 0.8
20
29.1 ± 0.7
21.5 ± 0.8
29.5 ± 0.3
36.1 ± 0.8
29.9 ± 1.0
35.5 ± 0.8
37.3 ± 0.1
34.0 ± 2.3
37.9 ± 1.9
39.4 ± 0.3
50
41.3 ± 0.6
36.5 ± 1.0
40.3 ± 0.7
46.5 ± 0.8
44.4 ± 1.0
43.9 ± 1.2
50.9 ± 0.8
47.4 ± 0.9
54.5 ± 0.6
55.6 ± 0.3
ResNet-18
10
27.7 ± 0.9
21.1 ± 0.4
30.2 ± 1.2
34.7 ± 0.4
33.4 ± 0.7
37.3 ± 0.2
35.7 ± 1.6
32.3 ± 1.2
39.9 ± 2.6
41.5 ± 1.1
20
29.7 ± 0.5
23.6 ± 0.3
32.2 ± 0.6
40.5 ± 0.5
29.8 ± 1.7
38.6 ± 0.2
41.8 ± 1.9
38.4 ± 1.1
44.5 ± 2.0
45.0 ± 0.8
50
47.9 ± 1.8
39.6 ± 1.0
48.3 ± 1.2
50.1 ± 0.5
46.2 ± 0.6
48.3 ± 0.8
53.7 ± 0.6
53.7 ± 2.2
58.9 ± 1.5
60.1 ± 0.9
Table 4. Performance comparison of dataset distillation methods on ImageWoof under varying IPC values. Evaluations are conducted using ConvNet and ResNet-18. The best results are shown in bold .
Method
Lμ
LCov
Ldiv
IPC=10
IPC=20
IPC=50
D 4 M
-
-
-
33.2
40.1
51.7
Ours
✓
-
-
34.3
39.6
52.2
✓
✓
-
39.8
44.3
57.8
✓
-
✓
36.2
42.1
54.1
✓
✓
✓
40.9
46.7
59.9
Table 5. Ablation study on ImageWoof.
Figure 3. Performance of D 4 M and DEO under different high IPC settings on ImageWoof.
Figure 4. Effect of λ1 and λ2 on ImageWoof with IPC∈{10,20,50} . Red, blue, and green lines represent IPC=10,20, and 50 , respectively, while dashed lines denote the corresponding results from D 4 M.
Figure 5. Comparison of intra-class diversity. Average cosine distance between distilled samples and class prototypes on CIFAR-100 and ImageNet-1K.
Stage (min)
CIFAR-100
ImageNet-1K
VLCP
DEO (Ours)
VLCP
DEO (Ours)
Clustering
1.3
–
3.5
–
LLaVA Inf.
8.2
–
21.7
–
Emb. Stats
–
3.6
–
9.6
Optim.
–
3.8
–
3.8
Total Time
9.5
7.4
25.2
13.4
Table 6. Per-class computational cost (minutes) at IPC=10 on CIFAR-100 and ImageNet-1K.
Figure 6. t-SNE visualization of latent distributions for original and distilled images. Results are shown for ImageWoof under IPC 10 and 50. Blue dots indicate original data; red stars denote distilled data from D 4 M and DEO.
Figure 7. Visualization of distilled data. We compare original samples with distilled data generated by D 4 M and DEO on (a) CIFAR-100 (baby, bear) and (b) ImageWoof (Shih-Tzu, Beagle).
Dataset Distillation aims to synthesize compact datasets that can approximate the training efficacy of large-scale real datasets, offering an efficient solution to the increasing computational demands of modern deep learning. Recently, diffusion-based dataset distillation methods have shown great promise by leveraging the strong generative capacity of diffusion models to produce diverse and structurally consistent samples. However, a fundamental goal misalignment persists: diffusion models are optimized for generative likelihood rather than discriminative utility, resulting in over-concentration in high-density regions and inadequate coverage of boundary samples crucial for classification. To address this issue, we propose two complementary strategies. Inversion-Matching (IM) introduces an inversion-guided fine-tuning process that aligns denoising trajectories with their inversion counterparts, broadening distributional coverage and enhancing diversity. Selective Subgroup Sampling(S^3) is a training-free sampling mechanism that improves inter-class separability by selecting synthetic subsets that are both representative and distinctive. Extensive experiments demonstrate that our approach significantly enhances the discriminative quality and generalization of distilled datasets, achieving state-of-the-art performance among diffusion-based methods.
Chenru Wang, Yunyi Chen, Zijun Yang +2
AGI Lab, Westlake University · Eindhoven University of Technology · Jinan University +2
Dataset distillation provides an effective approach to reduce memory and computational costs by optimizing a compact dataset that achieves performance comparable to the full original. However, for large-scale datasets and complex deep networks (e.g., ImageNet-1K with ResNet-101), the vast optimization space hinders distillation effectiveness, limiting practical applications. Recent methods leverage pre-trained diffusion models to directly generate informative images, thereby bypassing pixel-level optimization and achieving promising results. Nonetheless, these approaches often suffer from distribution shifts between the pre-trained diffusion prior and target datasets, as well as the need for multiple distillation steps under varying settings. To overcome these challenges, we propose a novel framework that is orthogonal to existing diffusion-based distillation techniques by utilizing the diffusion prior for patch selection rather than generation. Our method predicts noise from the diffusion model conditioned on input images and optional text prompts (with or without label information), and computes the associated loss for each image-patch pair. Based on the loss differences, we identify distinctive regions within the original images. Furthermore, we apply intra-class clustering and ranking on the selected patches to enforce diversity constraints. This streamlined pipeline enables a one-step distillation process. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art methods across various metrics and settings.
Xinhao Zhong, Shuoyang Sun, Zhaoyang Xu +4
Harbin Institute of Technology, Shenzhen · Shanghai Jiao Tong University
Dataset distillation aims to synthesize compact datasets that can approximate the performance of full-data training while significantly reducing computational and storage costs. However, diffusion-based distillation methods often struggle to preserve structural coherence and generalization, especially in visually complex domains. This issue often stems from latent prototypes that are weakly aligned with class-discriminative regions and contaminated by irrelevant background, thereby degrading generation quality and generalization. To address this limitation, we propose a saliency-driven distillation framework that constructs class-discriminative latent prototypes to enhance representativeness and generalization. The framework proceeds in two stages: (1) ensemble Grad-CAM++ saliency is used to construct prototypes emphasizing class-discriminative regions, and (2) hard-prototype refinement is then applied to construct challenging yet class-consistent prototypes, thereby enhancing discriminability and diversity. Importantly, the diffusion backbones (e.g., LDM and DiT) remain frozen; only lightweight classifiers used for saliency extraction are trained. Extensive experiments across multiple benchmarks demonstrate consistent performance improvements over strong baselines. Code will be released.
Yawen Zou, Wenqi Cai, Guang Li +3
University of Toyama · Hokkaido University · University of Fukui