Towards Principled Dataset Distillation: A Spectral Distribution Perspective

Ruixi Wu; Shaobo Wang; Jiahuan Chen; Zhiyuan Liu; Yicun Yang; Zhaorun Chen; Zekai Li; Kaixin Li

cs.CVMar 2, 2026

Towards Principled Dataset Distillation: A Spectral Distribution Perspective

Authors: Ruixi Wu, Shaobo Wang, Jiahuan Chen, Zhiyuan Liu, Yicun Yang, Zhaorun Chen, Zekai Li, Kaixin Li,

Organizations: EPIC Lab, SJTU · UChicago · NUS · CASIA ·

Abstract

Dataset distillation (DD) aims to compress large-scale datasets into compact synthetic counterparts for efficient model training. However, existing DD methods exhibit substantial performance degradation on long-tailed datasets. We identify two fundamental challenges: heuristic design choices for distribution discrepancy measure and uniform treatment of imbalanced classes. To address these limitations, we propose Class-Aware Spectral Distribution Matching (CSDM), which reformulates distribution alignment via the spectrum of a well-behaved kernel function. This technique maps the original samples into frequency space, resulting in the Spectral Distribution Distance (SDD). To mitigate class imbalance, we exploit the unified form of SDD to perform amplitude-phase decomposition, which adaptively prioritizes the realism in tail classes. On CIFAR-10-LT, with 10 images per class, CSDM achieves a 14.0% improvement over state-of-the-art DD methods, with only a 5.7% performance drop when the number of images in tail classes decreases from 500 to 25, demonstrating strong stability on long-tailed data.

Figures & tables

Figure 1: Dataset Distillation with Class-Aware Spectral Distribution Matching (CSDM). (a) Kernel Embedding. A feature network extracts features from real and synthetic data. Unlike previous methods using a linear kernel that maps to a suboptimal RKHS, we design a universal kernel that enables accurate measurement of distribution discrepancies. (b) Class-Aware Spectral Distribution Matching. We quantify distribution discrepancies using the Spectral Distribution Distance (SDD). By decomposing SDD into amplitude and phase and applying class-specific weights, CSDM dynamically adapts to class imbalance.

Kernel	Universal	Shift-Invariant	$K(\mathbf{x},\mathbf{y})$	$p(\mathbf{t})$
RBF	$\checkmark$	$\checkmark$	$\exp(-\bm{\gamma}\\|\mathbf{x}-\mathbf{y}\\|^{2})$	$\propto\exp\left(-\\|\mathbf{t}\\|^{2}/(4\bm{\gamma})\right)$
Laplace	$\checkmark$	$\checkmark$	$\exp(-\bm{\gamma}\\|\mathbf{x}-\mathbf{y}\\|)$	$\propto\prod_{i=1}^{d}\dfrac{\bm{\gamma}}{\bm{\gamma}^{2}+t_{i}^{2}}$
Linear	$\times$	$\times$	$\mathbf{x}^{\top}\mathbf{y}$	–
Polynomial	$\times$	$\times$	$(\mathbf{x}^{\top}\mathbf{y}+c)^{d}$	–
Sigmoid	$\times$	$\times$	$\tanh(\alpha\,\mathbf{x}^{\top}\mathbf{y}+c)$	–
Cosine	$\times$	$\checkmark$	$\cos(\omega\\|\mathbf{x}-\mathbf{y}\\|)$	$\propto\delta(\\|\mathbf{t}\\|-\omega)$

Table 1: Comparison of commonly used kernels.

	CIFAR-10-LT						CIFAR-100-LT
Imbalance Factor	10			50			10
IPC	10	20	50	10	20	50	10	20	50
Random	32.5 $\pm$ 2.2	39.6 $\pm$ 0.9	51.9 $\pm$ 1.5	33.2 $\pm$ 0.4	42.0 $\pm$ 1.3	51.6 $\pm$ 1.3	14.2 $\pm$ 0.6	21.7 $\pm$ 0.6	32.1 $\pm$ 0.6
K-Center	21.9 $\pm$ 0.8	24.2 $\pm$ 0.8	31.7 $\pm$ 0.9	17.8 $\pm$ 0.2	20.8 $\pm$ 0.5	26.1 $\pm$ 0.2	10.7 $\pm$ 0.9	15.9 $\pm$ 1.0	24.8 $\pm$ 0.2
Graph-Cut	28.7 $\pm$ 0.9	34.2 $\pm$ 1.0	40.6 $\pm$ 1.0	24.2 $\pm$ 0.7	28.6 $\pm$ 0.8	33.9 $\pm$ 0.4	16.9 $\pm$ 0.3	22.2 $\pm$ 0.4	29.9 $\pm$ 0.4
DC	37.9 $\pm$ 0.9	38.5 $\pm$ 0.9	37.4 $\pm$ 1.4	37.3 $\pm$ 0.9	37.3 $\pm$ 0.9	35.8 $\pm$ 1.2	24.0 $\pm$ 0.3	27.4 $\pm$ 0.3	27.4 $\pm$ 0.3

Table 2: Results on CIFAR-10-LT and CIFAR-100-LT.

Dataset	ImageMeow				ImageSquawk
Imbalance Factor	5		10		5		10
IPC	1	10	1	10	1	10	1	10
Random	12.8 $\pm$ 0.6	14.5 $\pm$ 1.0	10.2 $\pm$ 0.7	10.3 $\pm$ 0.6	13.1 $\pm$ 0.9	13.2 $\pm$ 0.8	9.6 $\pm$ 0.6	11.7 $\pm$ 0.8
DM	9.5 $\pm$ 0.8	12.5 $\pm$ 1.3	8.4 $\pm$ 1.1	12.8 $\pm$ 1.3	10.0 $\pm$ 2.3	10.8 $\pm$ 0.8	8.3 $\pm$ 0.8	10.8 $\pm$ 0.3
MTT	12.3 $\pm$ 0.9	13.2 $\pm$ 1.0	11.1 $\pm$ 1.0	14.1 $\pm$ 0.5	11.1 $\pm$ 1.4	12.5 $\pm$ 0.8	9.2 $\pm$ 0.7	12.2 $\pm$ 1.1
RDED	9.9 $\pm$ 0.6	13.7 $\pm$ 0.7	9.9 $\pm$ 1.0	12.7 $\pm$ 0.7	10.5 $\pm$ 1.1	11.6 $\pm$ 0.5	7.8 $\pm$ 0.8	10.8 $\pm$ 0.4

Table 3: Results on long-tailed ImageNet Subsets.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	CIFAR-10			CIFAR-100
IPC	1	10	50	10	50
Ratio (%)	0.02	0.2	2	2	10
Random	$14.4\pm 0.2$	$26.0\pm 1.0$	$43.4\pm 1.0$	$15.1\pm 0.5$	$30.7\pm 0.4$
Herding	$21.5\pm 1.2$	$31.0\pm 1.0$	$46.3\pm 1.1$	$18.5\pm 1.0$	$34.5\pm 0.6$
Forgetting	$13.5\pm 1.2$	$23.3\pm 1.0$	$23.3\pm 1.1$	$15.1\pm 1.0$	$30.5\pm 0.7$
DC	$28.3\pm 0.5$	$44.9\pm 0.5$	$49.5\pm 0.5$	$25.5\pm 0.5$	$34.8\pm 0.5$

Appendix

Table 5: Experimental Results on CIFAR-10 and CIFAR-100 Datasets

Figure 4: Performance of Different Kernel Methods on CIFAR-10LT with Varying Imbalance Factors.

Figure 5: Effect of the $\bm{\alpha}$ parameter: Images synthesized by CSDM on CIFAR-10 (IPC=10) for values of 0 and 1. As shown in the images, amplitude-only distillation ( $\alpha=1$ ) yields diverse but less realistic images, whereas phase-only distillation ( $\alpha=0$ ) produces realistic images but suffers from limited diversity.

Dataset Distillation

Dataset distillation compresses training data into a small synthetic or selected set that retains the information needed to train useful models.

See all 67 papers

Long-Tailed 3D Point Cloud Dataset Distillation
Jul 29, 2026Jiahao You, Xu Han, Jinfeng Xu +1Long-Tailed Classification Dataset Distillation
Huazhong University of Science and Technology · Hunan University
Spectral Gradient Surgery for Domain-Generalizable Dataset Distillation
May 13, 2026Minyoung Oh, Najeong Chae, Jae-Young SimOOD Generalization Distribution Matching
Graduate School of Artificial Intelligence Ulsan National Institute of Science and Technology (UNIST) Ulsan, Republic of Korea
IMS3: Breaking Distributional Aggregation in Diffusion-Based Dataset Distillation
Mar 14, 2026Chenru Wang, Yunyi Chen, Zijun Yang +2Diffusion-Based Dataset Distillation Dataset Distillation
AGI Lab, Westlake University · Eindhoven University of Technology · Jinan University +2

Jul 29, 2026J/K move · Enter open · S save

Long-Tailed 3D Point Cloud Dataset Distillation

Jiahao You, Xu Han, Jinfeng Xu, Xianzhi Li

Huazhong University of Science and Technology · Hunan University

Long-Tailed Classification Dataset Distillation

Open paper AskPDF arXiv

Dataset distillation compresses large-scale datasets into compact synthetic sets while preserving their training utility, enabling efficient 3D point cloud training. Current point cloud dataset distillation methods only tackle geometric and representation challenges while ignoring the distributional imbalance prevalent in point cloud datasets where both training and test splits follow long-tailed class distributions. To our knowledge, we present the first study on long-tailed point cloud dataset distillation. Rather than focusing primarily on geometric and representation properties or simply constructing a class-balanced synthetic set, our framework explicitly accounts for long-tailed class distributions via two core modules. First, we design Adaptive Synthetic Budgeting to allocate class-wise synthetic budgets according to class quantity and the expected benefit of additional synthetic samples. Given the allocated budgets, we further design 3D Long-Tailed Distribution Matching to optimize synthetic point clouds through Global-Local Feature Alignment and Prior-Aware Supervision. The former preserves both global class distributions and diverse intra-class structures, while the latter provides class-dependent expert supervision to keep tail-class samples recognizable while maintaining diverse head-class patterns. Extensive experiments demonstrate the effectiveness of our method, lifting classification accuracy by 7.0 points on ShapeNet55 against state-of-the-art methods.

Abstract

Dataset Distillation

Explore similar work

Long-Tailed 3D Point Cloud Dataset Distillation

Spectral Gradient Surgery for Domain-Generalizable Dataset Distillation

IMS3: Breaking Distributional Aggregation in Diffusion-Based Dataset Distillation