The task of dataset distillation concerns the learning of a small set of synthetic samples that can serve as an effective replacement for a large real dataset when used for downstream tasks. While many solutions have been proposed since the problem's inception, most of them primarily seek to optimize the distilled data with respect to a classifier. In this work, we propose a new distillation method that learns a set of samples such that they directly approximate the per-class data distribution. Specifically, we "drift" our distilled data in the latent space of a feature extractor such that they move both toward the real data and away from each other without using a classification objective. We find that our drifted data cover the real distribution significantly better than existing methods while still serving as powerful training samples for classification. Furthermore, we showcase our method's use as a visualization tool for better interpreting models and extend it beyond images to other modalities. Our work advances the frontier of dataset distillation, offering a novel synthesis method and proposing distribution-based metrics as a complementary evaluation. More broadly, our findings help address the growing need to understand both data and models by uncovering more about how they interact.
Figures & tables
Figure 2: Augmentations of Distilled Images Cover the Real Distribution. At each distillation step, we take multiple augmentations of each synthetic sample (left) . The synthetic data, together with their augmentations, approximately cover the distribution of real data (right) . In this plot, the stars represent the distilled images, the triangles 10 augmentations of each distilled image, and the circles real data. Drifting multiple augmentations of each synthetic sample at each step leads to high intra -image diversity, allowing some distilled images (such as the orange star) to cover many parts of the real distribution at the same time.
Figure 3: We represent distilled data as a Laplacian pyramid of cascading resolutions. To reconstruct a synthetic sample, each level ( top ) is bicubically interpolated to the full resolution ( middle ) before being added together (bottom) . This acts as a hyperparameter-free implicit regularizer on the distilled data, preventing overfitting to adversarial patterns often seen in model inversion tasks.
Hard-Label
Soft-Label
Knowledge Distillation
Method
IPC 10
IPC 50
IPC 10
IPC 50
IPC 10
IPC 50
Noise init
SRe 2 L ( 72 )
0 8.1 ± 0.2
10.7 ± 0.3
17.5 ± 0.2
22.9 ± 0.3
40.6 ± 0.3
50.1 ± 0.1
CDA ( 71 )
0 9.5 ± 0.1
16.1 ± 0.3
19.2 ± 0.2
28.8 ± 0.1
41.1 ± 0.2
50.8 ± 0.1
G-VBSM † ( 49 )
11.7 ± 0.3
24.9 ± 0.9
29.8 ± 0.2
44.9 ± 0.3
43.8 ± 0.3
53.0 ± 0.3
WMDD ( 35 )
10.2 ± 0.3
11.6 ± 0.6
29.4 ± 0.5
40.6 ± 0.1
43.7 ± 0.1
53.2 ± 0.2
Table 1: From-Scratch Training. This table shows ImageNet-1k ( 12 ) classification performance when training a ResNet-18 ( 26 ) from scratch. We include results using one-hot hard labels ( left ), one-time soft labels ( middle ), and full knowledge distillation ( right ). Methods marked with a dagger † distill using an ensemble of encoders. Across all three training paradigms, our distilled data lead to the best downstream performance.
Figure 4: Comparing Noise-Initialized Distillation Methods. Even when using ResNet-18 as the distillation backbone, our method yields higher-quality and more diverse samples. When using a stronger backbone (DINOv3 ( 53 ) ) , the visual quality and diversity of our method increase further.
Coverage
k-NN
Linear
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
Method
IPC 10
IPC 50
IPC 10
IPC 50
IPC 10
IPC 50
IPC 10
IPC 50
IPC 10
IPC 50
IPC 10
IPC 50
Noise init
SRe 2 L ( 72 )
0 7.2
11.6
0 4.9
0 8.5
50.1
54.3
46.0
49.1
51.2
55.6
41.1
45.4
CDA ( 71 )
0 6.5
11.1
0 4.6
0 8.7
44.5
51.7
44.8
49.5
51.0
57.7
43.2
50.9
G-VBSM † ( 49 )
0 5.7
0 8.6
0 5.3
0 8.4
41.3
45.9
39.7
42.5
62.3
70.5
47.1
61.8
Table 2: Feature-Space Metrics. Coverage ( left ), k -NN accuracy ( middle ), and linear probe accuracy ( right ) of distilled ImageNet-1k data in the SigLIP2 ( 60 ) and MoCo-v3 ( 7 ) latent spaces (see Appendix D ). Parenthesized entries are self-evaluations and are not ranked. Methods marked with a dagger † distill using an ensemble of encoders. Our method outperforms all baselines in every feature-space setting except for RDED ( 56 ) on IPC 10 k -NN in both latent spaces. The performance gap is especially large on the coverage metric.
Figure 5: Distilled images elucidate model representations. Each of the models shown here induces a different style of distilled images to best cover its own distribution. CLIP ( 45 ) is a notable outlier, possibly due to its strong text-entanglement and the contrastive loss not used in more recent vision-text models, such as SigLIP2 ( 60 ) .
Figure 7
Figure 7: Distilling Canvases. We can distill a collection of images into a single large canvas via drifting by using a large number of random crops at each step. The above 2048×2048 pixel images were synthesized by drifting SigLIP2 ( 60 ) features to match the distribution of Claude Monet’s ( left ) and Qi Baishi’s ( right ) paintings.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Effects of augmentations. Here, we visualize the effect the number of augmentations per step has on the distilled data. As we increase the number of augmentations, the images appear to (qualitatively) become richer in semantic information. Images shown here were initialized with noise and distilled with DINOv3 to match ImageNet-1k’s “orange” class. For quantitative results, please see Table A.1 .
ResNet-18 from scratch
Coverage
k-NN
Linear
Hard
Soft
KD
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
None
0 0.2 ± 0.0
0 0.2 ± 0.0
0 1.9 ± 0.1
0 1.0
0 0.6
0 2.7
0 5.8
0 6.5
0 4.9
1
0 7.5 ± 0.3
27.4 ± 0.3
39.3 ± 0.3
35.8
28.6
56.0
50.2
69.5
59.2
2
15.5 ± 0.4
34.4 ± 0.2
42.4 ± 0.6
65.5
66.0
66.5
57.4
76.6
64.9
5
19.1 ± 0.4
35.4 ± 0.3
44.2 ± 0.4
75.1
79.9
69.4
60.3
77.3
66.1
10 †
21.3 ± 0.5
35.9 ± 0.1
45.0 ± 0.2
77.0
83.4
69.9
61.1
77.5
66.3
Appendix
Table A.1: Augmented copies per step. The number of augmented views of each distilled image used at every optimization step. None is a single deterministic resized view. Performance monotonically increases with the number of augmentations, with diminishing returns as we approach 10.
Figure A.2 : Effect of backbone size. All 10 distilled images of the ImageNet-1k “lorikeet” class using DINOv3 ViT-S, ViT-B (default), and ViT-L. For quantitative results, see Table A.2 .
ResNet-18 from scratch
Coverage
k-NN
Linear
Hard
Soft
KD
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
Small
18.5 ± 0.5
31.9 ± 0.5
41.9 ± 0.5
61.8
77.4
67.6
60.9
77.2
65.3
Base †
21.3 ± 0.5
35.9 ± 0.1
45.0 ± 0.2
77.0
83.4
69.9
61.1
77.5
66.3
Large
18.3 ± 0.3
34.0 ± 0.4
44.6 ± 0.4
75.6
78.1
69.8
60.2
76.9
65.6
Appendix
Table A.2: Backbone model size. Base size performs the best with our settings, but the larger model might improve with some tuning.
Figure A.3 : Effect of kernel temperature. All 10 distilled images of the ImageNet-1k “terrapin” class for three drifting kernel temperatures τ (default τ=0.5 ). For quantitative results, see Table A.3 .
ResNet-18 from scratch
Coverage
k-NN
Linear
Hard
Soft
KD
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
τ=0.05
17.9 ± 0.2
26.4 ± 0.1
39.6 ± 0.2
71.6
66.6
72.7
62.7
76.9
64.4
τ=0.5†
21.3 ± 0.5
35.9 ± 0.1
45.0 ± 0.2
77.0
83.4
69.9
61.1
77.5
66.3
τ=5
17.7 ± 0.1
35.9 ± 0.4
45.1 ± 0.5
64.7
70.7
66.2
57.8
76.1
65.0
Appendix
Table A.3: Drifting distance kernel temperature. The method remains decently robust to a higher temperature, but a lower one significantly hurts performance.
Figure A.4 : Effect of the drifted features. All 10 distilled images of the ImageNet-1k “golden retriever” class when the drifting loss is applied to the CLS token only, the CLS token plus the mean patch token, or the CLS token plus every patch token (default). For quantitative results, see Table A.4 .
ResNet-18 from scratch
Coverage
k-NN
Linear
Hard
Soft
KD
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
CLS
11.8 ± 0.2
26.4 ± 0.0
41.3 ± 0.3
70.2
72.5
69.4
61.8
76.7
66.0
+ mean
15.0 ± 0.3
31.0 ± 0.3
43.6 ± 0.6
74.2
80.0
69.3
61.3
77.4
66.6
+ grid †
21.3 ± 0.5
35.9 ± 0.1
45.0 ± 0.2
77.0
83.4
69.9
61.1
77.5
66.3
Appendix
Table A.4: Drifted features. The DINOv3 tokens to which the drifting loss is applied: the CLS token only, the CLS token plus the mean patch token ( + mean), or the CLS token plus every patch token ( + grid). Drifting the CLS token with the full grid performs best.
Figure A.5 : Effect of the parameterization. All 10 distilled images of the ImageNet-1k “flamingo” class with our full parameterization, without color decorrelation ( − color), and with raw pixels instead of a Laplacian pyramid ( − pyramid). For quantitative results, see Table A.5 .
ResNet-18 from scratch
Coverage
k-NN
Linear
Hard
Soft
KD
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
Full †
21.3 ± 0.5
35.9 ± 0.1
45.0 ± 0.2
77.0
83.4
69.9
61.1
77.5
66.3
− color
16.5 ± 0.5
31.4 ± 0.8
38.1 ± 0.8
62.0
76.5
69.7
60.6
77.4
65.7
− pyramid
0 3.3 ± 0.3
11.0 ± 1.0
24.7 ± 1.0
41.9
46.4
69.2
59.7
77.2
62.1
Appendix
Table A.5: Parameterization. Our full parameterization, without color decorrelation ( − color), and with raw pixels in place of the Laplacian pyramid ( − pyramid). The full parameterization performs best.
Hard-Label
Soft-Label
Knowledge Distillation
Method
IPC 10
IPC 50
IPC 10
IPC 50
IPC 10
IPC 50
Noise init
ResNet-18
17.6 ± 0.2
18.4 ± 0.5
30.5 ± 0.8
43.1 ± 0.3
44.8 ± 0.7
55.0 ± 0.3
ViT
20.7 ± 0.4
28.4 ± 0.6
34.0 ± 0.6
48.7 ± 0.1
44.2 ± 0.2
54.9 ± 0.2
CLIP
0 6.1 ± 0.2
0 8.4 ± 0.3
18.5 ± 0.2
29.2 ± 0.5
32.9 ± 0.3
44.3 ± 0.4
SigLIP2
14.9 ± 0.2
21.0 ± 0.0
30.7 ± 0.5
44.5 ± 0.7
40.8 ± 0.3
52.2 ± 0.6
Appendix
Table B.1: From-scratch training across distillation backbones. Most backbones yield distilled data that can effectively train a model from scratch, with MoCo-v3 ( 7 ) and DINOv3 ( 53 ) eking out a slight edge over the others.
Hard-Label
Soft-Label
Knowledge Distillation
Method
IPC 10
IPC 50
IPC 10
IPC 50
IPC 10
IPC 50
Diffusion
Minimax ( 23 )
22.2 ± 0.2
36.1 ± 0.3
35.9 ± 0.2
48.8 ± 0.2
49.7 ± 0.1
58.1 ± 0.1
D4M ( 55 )
17.0 ± 0.1
30.5 ± 0.2
35.8 ± 0.2
50.3 ± 0.1
49.6 ± 0.3
58.5 ± 0.1
D 3 HR ( 77 )
23.7 ± 0.2
38.0 ± 0.4
35.7 ± 0.1
50.2 ± 0.2
48.1 ± 0.3
57.7 ± 0.1
CaO 2 ( 61 )
28.8 ± 0.2
44.3 ± 0.1
40.7 ± 0.2
54.1 ± 0.1
51.4 ± 0.3
59.5 ± 0.1
Appendix
Table B.2: Comparison with Diffusion-Based Methods. Our method matches the performance of the best diffusion-based methods in the soft-label setting while falling slightly behind on hard-label and knowledge distillation. Despite not leveraging a pre-trained generative model, our distilled data remain competitive.
Coverage
k -NN
Linear
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
SigLIP2
MoCo-v3
Method
IPC 10
IPC 50
IPC 10
IPC 50
IPC 10
IPC 50
IPC 10
IPC 50
IPC 10
IPC 50
IPC 10
IPC 50
Diffusion
Minimax ( 23 )
60.6
82.3
67.6
88.2
67.2
70.3
57.9
61.4
71.9
74.4
61.2
64.4
D4M ( 55 )
37.1
60.3
47.1
76.3
57.8
62.3
48.1
53.5
65.6
67.7
55.2
62.2
D 3 HR ( 77 )
55.0
78.9
65.0
88.5
67.9
71.0
59.4
62.7
72.7
75.1
62.4
66.2
Appendix
Table B.3: Feature-Space Comparison with Diffusion-Based Methods. Unlike in the from-scratch regime, our method (DINOv3 ( 53 ) variant) consistently outperforms the diffusion-based methods across all feature-space metrics. This is quite remarkable given that our method does not use a generative model to constrain the distilled data to the real image manifold.
Hard-Label
Knowledge Distillation
Method
IPC 10
IPC 50
IPC 10
IPC 50
Noise init
SRe 2 L ( 72 )
10.6 ± 0.3
15.9 ± 0.7
77.6 ± 1.0
91.8 ± 0.3
CDA ( 71 )
17.3 ± 0.8
32.7 ± 0.3
75.7 ± 2.2
91.0 ± 0.3
G-VBSM ( 49 )
23.4 ± 1.2
21.5 ± 0.5
82.8 ± 0.4
93.6 ± 0.3
WMDD ( 35 )
42.4 ± 1.2
55.7 ± 0.1
81.2 ± 0.7
92.4 ± 0.1
Appendix
Table B.4: ImageNette From-Scratch Training. On the “easy” 10-class ImageNette ( 19 ) dataset, we outperform the baselines in the hard-label setting and match or beat them in the knowledge distillation regime.
Hard-Label
Knowledge Distillation
Method
IPC 10
IPC 50
IPC 10
IPC 50
Noise init
SRe 2 L ( 72 )
15.3 ± 0.3
25.6 ± 0.5
56.7 ± 0.6
80.6 ± 0.3
CDA ( 71 )
15.8 ± 0.1
24.2 ± 0.6
57.1 ± 0.6
80.8 ± 0.2
G-VBSM ( 49 )
18.7 ± 0.8
26.8 ± 0.5
62.3 ± 0.2
84.1 ± 0.3
WMDD ( 35 )
16.7 ± 0.2
20.7 ± 0.9
33.3 ± 0.9
65.9 ± 1.4
Appendix
Table B.5: ImageWoof From-Scratch Training. On the “hard” 10-class ImageWoof ( 19 ) dataset, we also outperform the baselines in the hard-label setting and match or beat them in the knowledge distillation regime.
Feature space
Distilled set
ViT
CLIP
SigLIP2
EVA-02
MoCo-v3
EuPE
DINOv3
ViT ( 16 )
(94.6)
59.9
50.6
57.2
69.9
41.7
39.6
CLIP ( 45 )
29.0
(77.2)
50.4
59.1
20.4
41.3
28.6
SigLIP2 ( 60 )
73.5
84.7
(78.0)
84.8
56.9
71.5
56.1
EVA-02 ( 18 )
76.2
86.2
70.5
(91.8)
58.7
76.5
64.6
MoCo-v3 ( 7 )
89.5
72.2
61.2
75.7
(91.9)
52.3
48.2
Appendix
Table B.6: Inter-Model Coverage. Coverage of our noise-initialized ImageNet-1k distilled sets (IPC 10), distilled with the backbone in each row and evaluated in the feature space of the backbone in each column. Some encoders yield data that generalize very well (e.g., DINOv3) while data yielded by other encoders seem very overfit to the source model (e.g., CLIP). Same-model performance is indicated by (parentheses).