PSM: Dataset Distillation Based on Precise Statistical Matching by Difficulty

Hongxu Ma; Guang Li; Shijie Wang; Dongzhan Zhou; Suorong Yang; Baoli Sun; Takahiro Ogawa; Miki Haseyama

cs.CVSep 28, 2026

PSM: Dataset Distillation Based on Precise Statistical Matching by Difficulty

Authors: Hongxu Ma, Guang Li, Shijie Wang, Dongzhan Zhou, Suorong Yang, Baoli Sun, Takahiro Ogawa, Miki Haseyama, +1 more

Organizations: Zhejiang University · Hokkaido University · The University of Queensland · Shanghai Artificial Intelligence Laboratory · National University of Singapore · Dalian University of Technology

Abstract

Dataset distillation (DD) condenses a large original dataset into a small distilled dataset with high training utility. Decoupled statistical matching methods substantially reduce distillation time and memory overhead while achieving strong performance. However, they typically supervise all distilled samples using running statistics estimated from the entire original dataset. These statistics mainly capture the average feature distribution while overlooking differences in sample difficulty, limiting their ability to characterize the difficulty structure of the original data. To address this issue, we propose Precise Statistical Matching (PSM) by difficulty. After pretraining, PSM uses the Global Precision Score (GPS) to estimate image difficulty, ranks the samples within each class, and partitions each class into IPC (images per class) difficulty groups. During distillation, Statistics Updated Again (SUA) updates the teacher's batch normalization (BN) running statistics through forward passes on original samples from each group, providing difficulty-specific supervision for the corresponding distilled batch. Meanwhile, Initial Sample Screening (ISS) initializes distilled samples using original images from the corresponding difficulty group, providing an effective starting point for precise matching. Experiments across multiple datasets and model architectures demonstrate that PSM broadens the difficulty range of distilled samples and improves downstream performance in most evaluated settings. Code will be released.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Visualizations for the English Springer class in the original ImageNette dataset and the distilled datasets. The original data exhibit a progression in difficulty from easy to hard. Ours effectively preserves this difficulty structure, whereas the data distilled by FADRM show limited variation in difficulty. (b) Difficulty distributions of original and distilled data on Imagenette/Woof. FADRM exhibits nearly constant difficulty across $id_{\mathrm{IPC}}$ , indicating limited variation. In contrast, ours increases with $id_{\mathrm{IPC}}$ in line with the original data and covers a broader difficulty range.

Figure 2: Overview of the PSM. First, GPS evaluates the difficulty of the original data and ranks them from easy to hard within each class. The ordered data are then divided into $\mathrm{IPC}$ difficulty groups. When distilling the $g$ -th batch, SUA feeds the original data in difficulty group $g$ into the teacher for forward propagation, allowing the statistics of each BN layer to characterize the data features. Meanwhile, ISS selects original data from the same difficulty group to initialize the corresponding distilled data, thereby providing a favorable starting point for subsequent optimization.

Student

ResNet18

ResNet50 ( He et al., 2016 )

Dataset

IPC

SRe 2 L

RDED

FADRM

PSM

FADRM+

PSM+

SRe 2 L

RDED

FADRM

PSM

FADRM+

PSM+

CIFAR-10

16.6_{\pm 0.9}

\underline{22.9}_{\pm 0.4}

19.3_{\pm 0.6}

20.9_{\pm 0.1}

(\uparrow 1.6)

\textbf{23.7}_{\pm 0.8}

22.3_{\pm 0.6}

(\downarrow 1.4)

15.2_{\pm 1.3}

19.7_{\pm 1.7}

23.2_{\pm 0.7}

23.2_{\pm 0.2}

(=0.0)

\textbf{23.5}_{\pm 1.3}

\underline{23.3}_{\pm 0.4}

(\downarrow 0.2)

29.3_{\pm 0.5}

37.1_{\pm 0.3}

48.2_{\pm 0.4}

54.5_{\pm 1.3}

(\uparrow 6.3)

\underline{55.9}_{\pm 1.0}

\textbf{61.1}_{\pm 0.8}

(\uparrow 5.2)

30.3_{\pm 1.7}

32.5_{\pm 0.9}

43.8_{\pm 1.1}

52.7_{\pm 0.1}

(\uparrow 8.9)

\underline{55.1}_{\pm 0.8}

\textbf{56.8}_{\pm 1.1}

(\uparrow 1.7)

45.0_{\pm 0.7}

62.1_{\pm 0.1}

80.6_{\pm 0.7}

81.1_{\pm 0.1}

(\uparrow 0.5)

\underline{85.8}_{\pm 0.1}

\textbf{86.7}_{\pm 0.2}

(\uparrow 0.9)

52.9_{\pm 1.3}

52.5_{\pm 2.0}

79.7_{\pm 1.2}

82.7_{\pm 0.7}

(\uparrow 3.0)

\underline{83.7}_{\pm 0.6}

\textbf{86.0}_{\pm 0.9}

(\uparrow 2.3)

Whole dataset

92.9

93.1

CIFAR-100

6.6_{\pm 0.2}

11.1_{\pm 0.3}

31.3_{\pm 0.2}

32.9_{\pm 0.9}

(\uparrow 1.6)

\underline{37.9}_{\pm 0.8}

\textbf{39.1}_{\pm 0.1}

(\uparrow 1.2)

6.0_{\pm 0.1}

11.6_{\pm 0.4}

24.4_{\pm 1.6}

26.2_{\pm 1.4}

(\uparrow 1.8)

\underline{33.6}_{\pm 0.8}

\textbf{34.9}_{\pm 1.3}

(\uparrow 1.3)

Table 1: Top-1 accuracy comparison of PSM, extended variant PSM+, and other SOTA methods. For each dataset, IPC, and student setting, the mean values of the best and second best results are marked in bold and underlined , respectively. Values in parentheses after the PSM and PSM+ results indicate their performance changes relative to FADRM and FADRM+, respectively.

Panel of Figure 3 — Figure 3: (a) With ResNet18 as the teacher and $\mathrm{IPC}=50$ , the difficulty distributions of the original data and the data distilled by FADRM and PSM on CIFAR-10/100. (b) Runtime and peak GPU memory usage of GPS, as well as SUA and distillation for a single $id_{\mathrm{IPC}}$ , with $\mathrm{IPC}=10$ . “//” indicates that the corresponding value exceeds the plotting range.

Datasets		CIFAR-10		CIFAR-100		ImageNette		ImageWoof
Student	Parameters	FADRM	PSM	FADRM	PSM	FADRM	PSM	FADRM	PSM
ShuffleNetV2	2.3M	$39.3_{\pm 1.0}$	$41.5_{\pm 0.8}$ $(\uparrow 2.2)$	$57.2_{\pm 0.3}$	$58.6_{\pm 0.3}$ $(\uparrow 1.4)$	$48.4_{\pm 1.1}$	$52.1_{\pm 0.2}$ $(\uparrow 3.7)$	$30.5_{\pm 2.2}$	$28.9_{\pm 1.6}$ $(\downarrow 1.6)$
MobileNetV2	3.4M	$43.0_{\pm 0.4}$	$41.3_{\pm 0.5}$ $(\downarrow 1.7)$	$56.4_{\pm 0.2}$	$57.8_{\pm 0.3}$ $(\uparrow 1.4)$	$54.2_{\pm 0.2}$	$56.1_{\pm 0.6}$ $(\uparrow 1.9)$	$30.1_{\pm 1.0}$	$32.0_{\pm 1.4}$ $(\uparrow 1.9)$
DenseNet121	8.0M	$48.0_{\pm 0.6}$	$46.0_{\pm 1.0}$ $(\downarrow 2.0)$	$62.1_{\pm 0.3}$	$63.9_{\pm 0.2}$ $(\uparrow 1.8)$	$66.3_{\pm 0.6}$	$67.8_{\pm 1.0}$ $(\uparrow 1.5)$	$43.0_{\pm 1.0}$	$43.6_{\pm 0.5}$ $(\uparrow 0.6)$
ResNet18	11.7M	$48.2_{\pm 0.4}$	$54.5_{\pm 1.3}$ $(\uparrow 6.3)$	$64.7_{\pm 0.4}$	$66.7_{\pm 0.4}$ $(\uparrow 2.0)$	$64.0_{\pm 0.1}$	$67.1_{\pm 0.2}$ $(\uparrow 3.1)$	$43.8_{\pm 0.2}$	$44.5_{\pm 0.5}$ $(\uparrow 0.7)$
ResNet50	25.6M	$36.5_{\pm 1.0}$	$40.2_{\pm 1.3}$ $(\uparrow 3.7)$	$60.2_{\pm 0.2}$	$62.6_{\pm 0.7}$ $(\uparrow 2.4)$	$62.4_{\pm 0.7}$	$65.0_{\pm 1.0}$ $(\uparrow 2.6)$	$40.5_{\pm 0.4}$	$40.5_{\pm 1.1}$ $(=0.0)$

Table 2: Cross-architecture Top-1 accuracy on small-scale datasets with

\mathrm{IPC}=10

. FADRM and PSM use ResNet18 as the teacher. Results evaluated with ResNet18 as the student are also reported.

Datasets		Tiny-ImageNet		ImageNet-1K
Student	Parameters	FADRM	PSM	FADRM	PSM
DeiT-Ti	5.7M	$18.6_{\pm 0.3}$	$19.9_{\pm 0.2}$ $(\uparrow 1.3)$	$36.5_{\pm 0.5}$	$37.3_{\pm 0.1}$ $(\uparrow 0.8)$
ResNet18	11.7M	$51.4_{\pm 0.1}$	$51.7_{\pm 0.1}$ $(\uparrow 0.3)$	$60.9_{\pm 0.1}$	$61.3_{\pm 0.1}$ $(\uparrow 0.4)$
ResNet101	44.5M	$51.1_{\pm 1.8}$	$53.1_{\pm 0.9}$ $(\uparrow 2.0)$	$65.5_{\pm 0.1}$	$65.2_{\pm 0.1}$ $(\downarrow 0.3)$

Table 3: Cross-architecture Top-1 accuracy on large-scale datasets with

\mathrm{IPC}=50

. To further evaluate the generalization of the distilled data across different models, we also use DeiT ( Touvron et al., 2021 ) , which adopts a Transformer ( Vaswani et al., 2017 ) architecture, as a student.

Table 7

Figure 4: Hyperparameter study of SUA and ISS. On CIFAR-10/100 and ImageNette/Woof, we set

\mathrm{IPC}=10

and use ResNet18 as both the teacher and student. We investigate the forward epoch ratio

r_{\mathrm{fwd}}\in\{0.1,0.2,\ldots,1.0\}

in SUA and the screening method (Random, Front, Middle, Back) in ISS.

Figure 5: Distilled data on ImageNette with $\mathrm{IPC}=10$ and ResNet18 as the teacher, using neither mechanism (FADRM), ISS, SUA, or both. Spearman’s $\rho$ measures the agreement between the difficulty trends of the distilled and original data, with values closer to 1 indicating better agreement.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Relationship between GPS and classification difficulty on different datasets.

Dataset	IPC	Coreset		Gradient	Distribution	Trajectory			Generative	Decoupled
Dataset	IPC	Random	Herding	DSA	DM	MTT	DATM	TESLA	Minimax	SRe 2 L	FADRM	PSM
CIFAR-10	1	–	–	–	–	–	–	–	–	$16.6_{\pm 0.9}$	$\underline{19.3}_{\pm 0.6}$	$\textbf{20.9}_{\pm 0.1}$
	10	$25.1_{\pm 0.5}$	$28.4_{\pm 0.1}$	$42.1_{\pm 0.6}$	$38.2_{\pm 1.1}$	$46.1_{\pm 1.4}$	48.66	$\underline{48.9}_{\pm 2.2}$	–	$29.3_{\pm 0.5}$	$48.2_{\pm 0.4}$	$\textbf{54.5}_{\pm 1.3}$
	50	54.96	–	$47.8_{\pm 0.9}$	$52.9_{\pm 0.4}$	$58.7_{\pm 0.2}$	66.27	–	–	$45.0_{\pm 0.7}$	$\underline{80.6}_{\pm 0.7}$	$\textbf{81.1}_{\pm 0.1}$
CIFAR-100	1	–	–	–	–	–	–	–	–	$6.6_{\pm 0.2}$	$\underline{31.3}_{\pm 0.2}$	$\textbf{32.9}_{\pm 0.9}$
	10	$10.9_{\pm 0.1}$	$13.3_{\pm 0.3}$	$21.9_{\pm 0.4}$	$18.7_{\pm 0.5}$	$26.8_{\pm 0.6}$	–	$27.1_{\pm 0.7}$	–	$27.0_{\pm 0.4}$	$\underline{64.7}_{\pm 0.4}$	$\textbf{66.7}_{\pm 0.4}$

Appendix

Table 6: Top-1 accuracy (%) comparison of representative dataset distillation methods with ResNet18 as the student. Standard deviations are reported when available.

Hyperparameter	CIFAR-10	CIFAR-100	Tiny-ImageNet	ImageNette	ImageWoof	ImageNet-1K
learning rate	$1.00\mathrm{E}{-03}$	$1.00\mathrm{E}{-03}$	$1.00\mathrm{E}{-02}$	$1.00\mathrm{E}{-02}$	$1.00\mathrm{E}{-02}$	PyTorch
optimizer	Adam	Adam	SGD	SGD	SGD
scheduler	cos	cos	cos	cos	cos
epoch	150	100	50	150	200	90
batch	512	512	128	64	64	64
$\beta$	1	1	1	1	1	1

Appendix

Table 9: Hyperparameter settings during pretraining and GPS.

Hyperparameter	CIFAR-10	CIFAR-100	Tiny-ImageNet	ImageNette	ImageWoof	ImageNet-1K
Epoch ratio	0.9	0.9	0.9	1.0	0.4	0.9
Screening method	Back	Back	Back	Front	Back	Back
PSM+ teachers	ShuffleNetV2, MobileNetV2, DenseNet121, ResNet18

Appendix

Table 10: Hyperparameter settings during distillation.

Figure 9: Visualization of the distilled CIFAR-10 samples generated by FADRM with $\mathrm{IPC}=10$ and teacher=ResNet18.

Figure 10: Visualization of the distilled CIFAR-10 samples generated by PSM with $\mathrm{IPC}=10$ and teacher=ResNet18.

Figure 11: Visualization of the distilled ImageNette samples generated by FADRM with $\mathrm{IPC}=10$ and teacher=ResNet18.

Figure 12: Visualization of the distilled ImageNette samples generated by PSM with $\mathrm{IPC}=10$ and teacher=ResNet18.

Figure 13: Visualization of the distilled ImageWoof samples generated by FADRM with $\mathrm{IPC}=10$ and teacher=ResNet18.

Figure 14: Visualization of the distilled ImageWoof samples generated by PSM with $\mathrm{IPC}=10$ and teacher=ResNet18.

Explore similar work

Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching
Feb 5, 2026Qianxin Xia, Jiawei Du, Yuhan Zhang +6Diffusion-Based Dataset Distillation Self-Supervised Learning
University of Electronic Science and Technology of China, Chengdu, China · CFAR, Singapore · SWJTU, Chengdu, China
FD $^2$ : A Dedicated Framework for Fine-Grained Dataset Distillation
Mar 26, 2026Hongxu Ma, Guang Li, Shijie Wang +5Diffusion-Based Dataset Distillation Dataset Distillation
Zhejiang University · Hokkaido University · The University of Queensland +2
Dataset Distillation by Influence Matching
Jul 18, 2026Haoru Tan, Wang Wang, Sitong Wu +5Diffusion-Based Dataset Distillation Dataset Distillation
1HKU · 2CUHK · 3Stanford

Feb 5, 2026, cs.CVJ/K move · Enter open · S save

Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching

Qianxin Xia, Jiawei Du, Yuhan Zhang, Xin Zhang+5

University of Electronic Science and Technology of China, Chengdu, China · CFAR, Singapore · SWJTU, Chengdu, China

Diffusion-Based Dataset Distillation Self-Supervised Learning Contrastive Flow Matching Conditional Flow Matching Synthetic Data Bilevel Optimization

Open paper AskPDF arXiv

Dataset distillation seeks to synthesize a compact surrogate dataset that enables performance comparable to training on the original dataset for downstream tasks. For the scenario where pre-trained self-supervised models serve as priors, traditional Linear Gradient Matching optimizes synthetic images by encouraging them to mimic the gradient updates induced by real images on the linear probe. However, this batch-level formulation requires loading thousands of real images and applying multiple differentiable augmentations to synthetic images at each distillation step, leading to substantial computational and memory overheads. In this paper, we revisit the linear gradient and theoretically derive that it is essentially a local relative distribution directed from target class centers toward non-target class centers, which we term flow. This property causes suboptimality and instability, often necessitating expensive multiple augmentations to compensate. To address this, we introduce Statistical Flow Matching, an optimal, stable, and efficient supervised learning framework that optimizes synthetic images by aligning global statistical flows in the original data. Our approach loads raw statistics only once and performs a single augmentation pass on the synthetic data, achieving performance comparable to or better than the state-of-the-art method with 10x less GPU memory usage and 4x faster distillation time. Moreover, increasing the number of augmentations for our method yields further performance gains while incurring lower additional cost. Our code is publicly available at https://github.com/einsteinxia/SFM.

Abstract

Explore similar work

Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching

FD2^22: A Dedicated Framework for Fine-Grained Dataset Distillation

Dataset Distillation by Influence Matching

FD $^2$ : A Dedicated Framework for Fine-Grained Dataset Distillation