Vector-Quantized Soft Label Compression for Dataset Distillation

Ali Abbasi; Ashkan Shahbazi; Hamed Pirsiavash; Soheil Kolouri

cs.CVMar 4, 2026

Vector-Quantized Soft Label Compression for Dataset Distillation

Authors: Ali Abbasi, Ashkan Shahbazi, Hamed Pirsiavash, Soheil Kolouri

Organizations: Vanderbilt University · University of California, Davis

Abstract

Dataset distillation is an emerging technique for reducing the computational and storage costs of training machine learning models by synthesizing a small, informative subset of data that captures the essential characteristics of a much larger dataset. Recent methods pair synthetic samples and their augmentations with soft labels from a teacher model, enabling student models to generalize effectively despite the small size of the distilled dataset. While soft labels are critical for effective distillation, the storage and communication overhead they incur, especially when accounting for augmentations, is often overlooked. In practice, each distilled sample is associated with multiple soft labels, making them the dominant contributor to storage costs, particularly in large-class settings such as ImageNet-1K. In this paper, we present a rigorous analysis of bit requirements across dataset distillation frameworks, quantifying the storage demands of both distilled samples and their soft labels. To address the overhead, we introduce a vector-quantized autoencoder (VQAE) for compressing soft labels, achieving substantial compression while preserving the effectiveness of the distilled data. We validate our method on both vision and language distillation benchmarks. On ImageNet-1K, our proposed VQAE achieves 30--40x additional compression over RDED, LPLD, SRE2L, and CDA baselines while retaining over $90\%$ of their original performance.

Figures & tables

Figure 1: (a) We begin with a standard dataset distillation pipeline, where a small number of distilled inputs per class (IPC) are passed through a teacher model to produce soft labels across multiple augmentations. These soft labels, although highly informative, impose a significant memory and communication burden, especially for large numbers of classes and training epochs. (b) To mitigate this overhead, we introduce a vector-quantized autoencoder (VQ-AE) that encodes soft labels into compact latent representations segmented and quantized using a learned dictionary. Only the code indices, dictionary, and decoder need to be stored and transmitted. At distillation time, soft labels are reconstructed from these compact codes and used to train student models via KL divergence. This approach dramatically reduces storage while preserving the fidelity of the teacher’s knowledge.

	Method	IPC 10	IPC 20	IPC 50	IPC 100
Comp. 1×	RDED	$39.3{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.5}$	$48.3{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.2}$	$55.0{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.1}$	$58.6{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.3}$
	LPLD	$36.6{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.8}$	$47.0{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.1}$	$54.8{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.4}$	$58.5{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.2}$
	SRE2L	$34.2{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.5}$	$42.8{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.4}$	$50.4{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.1}$	$54.9{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.2}$
	CDA	$33.7{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.2}$	$43.8{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.3}$	$52.1{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.8}$	$56.6{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.2}$
Compression 10×	RDED + LPLD	$39.1{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\phantom{0.000}}$	$48.1{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\phantom{0.000}}$	$54.3{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\phantom{0.000}}$	$57.8{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\phantom{0.000}}$
	RDED + Ours	$39.7{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.4}$	$48.5{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.3}$	$55.0{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.2}$	$58.6{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.2}$

Table 1: Test accuracy percent on ImageNet-1K across soft label compression ratios and images per class (IPC). Results combine a data synthesis method (SRe2L, CDA, RDED or LPLD) with label compression by LPLD or our VQAE using ResNet18 as both teacher and student and average three runs. Standard deviations appear for our results and no compression numbers only; LPLD stds are blank when no std was reported in their work.

Figure 2: Performance ratio of training with compressed over uncompressed soft labels at different compression levels, averaged across image distillation techniques presented in Table 1 at each IPC level. Our method consistently outperforms LPLD across all compression factors.

Model	#Params	Method	Dolly	SelfInst	Vicuna
GPT‑2	1.5B	Teacher	27.6	14.3	16.3
	120M	SFT w/o KD	23.3	10.0	14.7
		KD	22.8	10.8	13.4
		SeqKD	22.7	10.1	14.3
		Ours	23.5	10.4	15.2
	760M	SFT w/o KD	25.4	12.4	16.1

Table 2: Comparison of distillation methods on GPT‑2 and LLaMA models.

Comp.	Method	Value
1×	No compression	39.3
$<$ 10×	Quant. (7.98×, 2 bits)	34.1
$<$ 10×	Quant. (5.36×, 3 bits)	36.1
40×	PCA (24 PCs)	8.9
	RPCA (24 PCs)	8.1
	Topk (k=15)	20.1

Table 3: Baselines on ImageNet‑1K at several compressions.

Figure 3: Accuracy vs. storage across codebook size

k

and block size

d_{c}

Method	IPC	Image Size (GB)	Label Size (GB)	Total Size (GB)
RDED	10	0.127	5.588	5.715
	20	0.250	11.176	11.426
	50	0.610	27.940	28.550
	100	1.230	55.879	57.109
SRe2L	10	0.150	5.588	5.738
	20	0.300	11.176	11.476

Table S6: Storage breakdown of soft labels and data across methods without any compression

Rate	IPC	Ours (GB)	LPLD (GB)
10x	10	0.558	0.559
	20	1.114	1.118
	50	2.779	2.794
	100	5.556	5.588
20x	10	0.257	0.279
	20	0.511	0.559

Table S7: Comparison of storage size (GB) between our method and LPLD across IPC levels and compression rates

Comp. Rate	$d_{h}$	$d_{c}$	$k$
10x	795	5	1024
20x	990	15	2048
30x	1000	20	1024
40x	1000	25	512
100x	1000	50	128
200x	1000	100	64

Table S8: Hyperparameter settings for our method under various compression rates

Method	100×		200×
	IPC 10	IPC 20	IPC 10	IPC 20
RDED + LPLD	$14.09$	$24.80$	$9.34$	$15.26$
RDED + Ours	$19.07$	$26.16$	14.85	24.73
LPLD + LPLD	$9.14$	$17.56$	$5.11$	$8.74$
LPLD + Ours	25.43	33.58	$14.00$	$22.17$
SRE2L + LPLD	$6.90$	$15.37$	$3.51$	$7.45$

Table S9: Results for compression ratio of 100× and 200×

Dataset Distillation

Dataset distillation compresses training data into a small synthetic or selected set that retains the information needed to train useful models.

See all 65 papers

Soft Label Pruning and Quantization for Large-Scale Dataset Distillation
Apr 20, 2026Xiao Lingao, Yang HeDataset Distillation Image Classification
CFAR, Agency for Science, Technology and Research, Singapore 138632 · IHPC, Agency for Science, Technology and Research, Singapore 138632 · National University of Singapore, Singapore 119077
From Fewer Samples to Fewer Bits: Reframing Dataset Distillation as Joint Optimization of Precision and Compactness
Mar 2, 2026My H. Dinh, Aditya Sant, Akshay Malhotra +2Quantization-Aware Distillation Dataset Distillation
InterDigital Communications, Inc.
Condensing Large-Scale Datasets Directly with Minimal Information Loss
Jul 1, 2026Xinyi Shang, Peng Sun, Bei Shi +2Distribution Shift Distribution Matching
University College London, United Kingdom · Zhejiang University, China · Westlake University, China +1

Apr 20, 2026J/K move · Enter open · S save

Soft Label Pruning and Quantization for Large-Scale Dataset Distillation

Xiao Lingao, Yang He

CFAR, Agency for Science, Technology and Research, Singapore 138632 · IHPC, Agency for Science, Technology and Research, Singapore 138632 · National University of Singapore, Singapore 119077

Dataset Distillation Image Classification

Open paper AskPDF arXiv

Large-scale dataset distillation requires storing auxiliary soft labels that can be 30-40x larger on ImageNet-1K and 200x larger on ImageNet-21K than the condensed images, undermining the goal of dataset compression. We identify two fundamental issues necessitating such extensive labels: (1) insufficient image diversity, where high within-class similarity in synthetic images requires extensive augmentation, and (2) insufficient supervision diversity, where limited variety in supervisory signals during training leads to performance degradation at high compression rates. To address these challenges, we propose Label Pruning and Quantization for Large-scale Distillation (LPQLD). We enhance image diversity via class-wise batching and batch-normalization supervision during synthesis. For supervision diversity, we introduce Label Pruning with Dynamic Knowledge Reuse to improve label-per-augmentation diversity, and Label Quantization with Calibrated Student-Teacher Alignment to improve augmentation-per-image diversity. Our approach reduces soft label storage by 78x on ImageNet-1K and 500x on ImageNet-21K while improving accuracy by up to 7.2% and 2.8%, respectively. Extensive experiments validate the superiority of LPQLD across different network architectures and dataset distillation methods. Code is available at https://github.com/he-y/soft-label-pruning-quantization-for-dataset-distillation.

Abstract

Dataset Distillation

Explore similar work

Soft Label Pruning and Quantization for Large-Scale Dataset Distillation

From Fewer Samples to Fewer Bits: Reframing Dataset Distillation as Joint Optimization of Precision and Compactness

Condensing Large-Scale Datasets Directly with Minimal Information Loss