Disentangled Embedding Optimization for Diffusion-Based Dataset Distillation
Authors: Hongcheng Li, Yucan Zhou, Yibin Wang, Xiaoyan Gu, Bo Li, Weiping Wang
Organizations: Institute of Information Engineering, Chinese Academy of Sciences & School of Cyber Security, University of Chinese Academy of Sciences & State Key Laboratory of Cyberspace Security Defense, Beijing, China · College of Intelligence and Computing, Tianjin University, Tianjin, China · Institute of Information Engineering, Chinese Academy of Sciences State Key Laboratory of Cyberspace Security Defense, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
Dataset distillation aims to compress large-scale datasets into compact synthetic subsets that enable efficient model training with comparable performance to the original data. Recent advances show that diffusion models are highly effective for this task due to their strong generative capability. Existing diffusion-based approaches typically perform class-wise clustering and use the obtained cluster prototypes as diffusion inputs to generate distilled samples. However, the uniformly distributed original samples in a category inevitably lead to entanglement near cluster boundaries, pulling the cluster prototypes closer and reducing the inter-cluster separation. Consequently, the diffusion model receives overly similar latent inputs, resulting in synthetic data with limited diversity. To address this issue, we propose Disentangled Embedding Optimization (DEO), a simple yet effective framework that replaces static clustering with trainable embeddings. These embeddings are optimized to match the Gaussian statistics (i.e., mean and covariance) of the original data distribution, while a diversity regularizer is designed to encourage separation among these embeddings to improve the intra-class variability. Extensive experiments demonstrate that our proposed DEO can generate more diversified and representative distilled data than existing cluster-based diffusion methods, achieving state-of-the-art performance and superior generalization across multiple datasets.
Figures & tables
Figure 1. Illustration of the entanglement issue in distilled data generation. (a) t-SNE visualization shows that entangled sub-cluster prototypes in the original data reduce separability. (b) These less-separable latent inputs for the diffusion model result in indistinguishable synthesized images.
Figure 2. Overview of our Disentangled Embedding Optimization (DEO) for diffusion-based distillation. ( a ) Learnable latent embeddings Zc are optimized to match the mean and covariance of the original data, with a diversity loss to encourage separation. ( b ) The optimized Zc and text prompts are fed into the diffusion model to generate distilled samples S .
Method
ImageWoof
ImageNette
ImageIDC
IPC=10
IPC=20
IPC=50
IPC=10
IPC=20
IPC=50
IPC=10
IPC=20
IPC=50
Random
29.4 ± 0.8
32.7 ± 0.4
47.2 ± 1.3
54.2 ± 1.6
63.5 ± 0.5
76.1 ± 1.1
48.1 ± 0.8
52.5 ± 0.9
68.1 ± 0.7
K-Center
22.1 ± 0.1
25.1 ± 0.7
40.6 ± 0.4
–
–
–
–
–
–
Herding
32.0 ± 0.3
34.9 ± 0.1
49.1 ± 0.7
–
–
–
–
–
–
DiT
34.7 ± 0.5
41.1 ± 0.8
49.3 ± 0.2
59.1 ± 0.7
64.8 ± 1.2
73.3 ± 0.9
54.1 ± 0.4
58.9 ± 0.2
64.3 ± 0.6
DM
30.3 ± 1.2
35.2 ± 0.6
47.1 ± 1.1
60.8 ± 0.6
66.5 ± 1.1
76.2 ± 0.4
52.8 ± 0.5
58.5 ± 0.4
69.1 ± 0.8
Table 1. Performance comparison on ImageWoof, ImageNette, and ImageIDC with ResNetAP-10 under different IPC settings. The best and second-best results are highlighted in bold and underline , respectively; “–” denotes unavailable results.
IPC
SRe2L
RDED
DiT
Minimax
VLCP
DEO (Ours)
10
21.3 ± 0.6
42.0 ± 0.1
39.6 ± 0.4
44.3 ± 0.5
46.7 ± 0.4
49.2 ± 0.8
50
46.8 ± 0.2
56.5 ± 0.1
52.9 ± 0.6
58.6 ± 0.3
60.5 ± 0.2
61.4 ± 0.3
Table 2. Performance comparison on ImageNet-1K with IPC=10/50.
Dataset
IPC
SRe2L
RDED
VLCP
DEO (Ours)
CIFAR-10
10
29.3 ± 0.5
37.1 ± 0.3
39.0 ± 0.7
41.8 ± 0.6
50
45.0 ± 0.7
62.1 ± 0.1
63.2 ± 0.3
64.1 ± 0.7
CIFAR-100
10
27.0 ± 0.4
42.6 ± 0.2
50.6 ± 0.7
52.4 ± 0.3
50
50.2 ± 0.4
62.6 ± 0.1
66.1 ± 0.3
66.8 ± 0.5
Table 3. Performance comparison on CIFAR-10 and CIFAR-100 with IPC=10/50.
Backbone
IPC
Random
K-Center
Herding
DiT
DM
IDC-1
Minimax
D 4 M
VLCP
DEO (Ours)
ConvNet
10
24.3 ± 1.1
19.4 ± 0.9
26.7 ± 0.5
34.2 ± 1.1
26.9 ± 1.2
33.3 ± 1.1
33.3 ± 1.7
29.4 ± 0.9
34.8 ± 2.4
36.2 ± 0.8
20
29.1 ± 0.7
21.5 ± 0.8
29.5 ± 0.3
36.1 ± 0.8
29.9 ± 1.0
35.5 ± 0.8
37.3 ± 0.1
34.0 ± 2.3
37.9 ± 1.9
39.4 ± 0.3
50
41.3 ± 0.6
36.5 ± 1.0
40.3 ± 0.7
46.5 ± 0.8
44.4 ± 1.0
43.9 ± 1.2
50.9 ± 0.8
47.4 ± 0.9
54.5 ± 0.6
55.6 ± 0.3
ResNet-18
10
27.7 ± 0.9
21.1 ± 0.4
30.2 ± 1.2
34.7 ± 0.4
33.4 ± 0.7
37.3 ± 0.2
35.7 ± 1.6
32.3 ± 1.2
39.9 ± 2.6
41.5 ± 1.1
20
29.7 ± 0.5
23.6 ± 0.3
32.2 ± 0.6
40.5 ± 0.5
29.8 ± 1.7
38.6 ± 0.2
41.8 ± 1.9
38.4 ± 1.1
44.5 ± 2.0
45.0 ± 0.8
50
47.9 ± 1.8
39.6 ± 1.0
48.3 ± 1.2
50.1 ± 0.5
46.2 ± 0.6
48.3 ± 0.8
53.7 ± 0.6
53.7 ± 2.2
58.9 ± 1.5
60.1 ± 0.9
Table 4. Performance comparison of dataset distillation methods on ImageWoof under varying IPC values. Evaluations are conducted using ConvNet and ResNet-18. The best results are shown in bold .
Method
Lμ
LCov
Ldiv
IPC=10
IPC=20
IPC=50
D 4 M
-
-
-
33.2
40.1
51.7
Ours
✓
-
-
34.3
39.6
52.2
✓
✓
-
39.8
44.3
57.8
✓
-
✓
36.2
42.1
54.1
✓
✓
✓
40.9
46.7
59.9
Table 5. Ablation study on ImageWoof.
Figure 3. Performance of D 4 M and DEO under different high IPC settings on ImageWoof.
Figure 4. Effect of λ1 and λ2 on ImageWoof with IPC∈{10,20,50} . Red, blue, and green lines represent IPC=10,20, and 50 , respectively, while dashed lines denote the corresponding results from D 4 M.
Figure 5. Comparison of intra-class diversity. Average cosine distance between distilled samples and class prototypes on CIFAR-100 and ImageNet-1K.
Stage (min)
CIFAR-100
ImageNet-1K
VLCP
DEO (Ours)
VLCP
DEO (Ours)
Clustering
1.3
–
3.5
–
LLaVA Inf.
8.2
–
21.7
–
Emb. Stats
–
3.6
–
9.6
Optim.
–
3.8
–
3.8
Total Time
9.5
7.4
25.2
13.4
Table 6. Per-class computational cost (minutes) at IPC=10 on CIFAR-100 and ImageNet-1K.
Figure 6. t-SNE visualization of latent distributions for original and distilled images. Results are shown for ImageWoof under IPC 10 and 50. Blue dots indicate original data; red stars denote distilled data from D 4 M and DEO.
Figure 7. Visualization of distilled data. We compare original samples with distilled data generated by D 4 M and DEO on (a) CIFAR-100 (baby, bear) and (b) ImageWoof (Shih-Tzu, Beagle).