From Fewer Samples to Fewer Bits: Reframing Dataset Distillation as Joint Optimization of Precision and Compactness

My H. Dinh; Aditya Sant; Akshay Malhotra; Keya Patani; Shahab Hamidi-Rad

cs.CVMar 2, 2026

From Fewer Samples to Fewer Bits: Reframing Dataset Distillation as Joint Optimization of Precision and Compactness

Authors: My H. Dinh, Aditya Sant, Akshay Malhotra, Keya Patani, Shahab Hamidi-Rad

Organizations: InterDigital Communications, Inc.

Abstract

Dataset Distillation (DD) compresses large datasets into compact synthetic ones that maintain training performance. However, current methods mainly target sample reduction, with limited consideration of data precision and its impact on efficiency. We propose Quantization-aware Dataset Distillation (QuADD), a unified framework that jointly optimizes dataset compactness and precision under fixed bit budgets. QuADD integrates a differentiable quantization module within the distillation loop, enabling end-to-end co-optimization of synthetic samples and quantization parameters. Guided by the rate-distortion perspective, we empirically analyze how bit allocation between sample count and precision influences learning performance. Our framework supports both uniform and adaptive non-uniform quantization, where the latter learns quantization levels from data to represent information-dense regions better. Experiments on image classification and 3GPP beam management tasks show that QuADD surpasses existing DD and post-quantized baselines in accuracy per bit, establishing a new standard for information-efficient dataset distillation.

Figures & tables

Figure 1: Illustration of the quantized dataset distillation (QuADD) framework, applied to a general DD algorithm.

Figure 2: Rate–distortion curve. Each point represents the best achievable accuracy under a fixed transmission budget (in bits).

Figure 3: QuADD performance on CIFAR-10 under varying $(M,b)$ . Lighter & smaller markers indicate lower-bit precision. The storage budget scales as $b\times 3\times D\times M$ (constants omitted).

		CIFAR-10			CIFAR-100			Imagenette
		IPC 1	IPC 10	IPC 50	IPC 1	IPC 10	IPC 50	IPC 10
Total storage		983K	9.8M	49.1M	9.8M	98.3M	491M	157.2M
Coreset	Random	14.4 $\pm$ 0.2	26.0 $\pm$ 1.2	43.4 $\pm$ 1.0	4.2 $\pm$ 0.3	14.6 $\pm$ 0.5	30.0 $\pm$ 0.4	–
	Herding [ 28 ]	21.5 $\pm$ 1.3	31.6 $\pm$ 0.7	40.4 $\pm$ 0.6	8.4 $\pm$ 0.3	17.3 $\pm$ 0.3	33.7 $\pm$ 0.5	–
	K-Center [ 20 ]	23.3 $\pm$ 0.9	36.4 $\pm$ 0.6	48.7 $\pm$ 0.3	8.6 $\pm$ 0.3	20.7 $\pm$ 0.2	33.6 $\pm$ 0.4	–
Unquantized DD	DD [ 26 ]	–	36.8 $\pm$ 1.2	–	–	–	–	–

Table 1: Comparison of dataset distillation on CIFAR-10 , CIFAR-100 , and Imagenette under different IPC budgets. Best result for Parameterized DD in each column is bolded . For AutoPalette, we quantize images to 7-bit ; for QuADD, to 9-bit (3 bits/channel) ; for FreD, we use their best reported hyperparameters. Higher value indicates better performance.

Figure 4: Accuracy and overhead reduction on CIFAR-10/100. Lines show test accuracy (left $y$ -axis); bars show compression ratio (right $y$ -axis) relative to the unquantized baseline.

Figure 5: Training time on CIFAR-10 for DD baselines (DATM, FreD, AutoPalette, QuADD) at IPC = 1, 10, 50. Lower is better.

Method	AlexNet			VGG			ResNet
Method	IPC1	IPC10	IPC50	IPC1	IPC10	IPC50	IPC1	IPC10	IPC50
Unquantized Baseline (DATM)	15.6 $\pm$ 1.0	25.5 $\pm$ 1.5	54.1 $\pm$ 1.3	29.8 $\pm$ 0.9	39.5 $\pm$ 1.0	63.1 $\pm$ 0.3	30.1 $\pm$ 1.0	49.0 $\pm$ 1.5	65.2 $\pm$ 1.0
FreD	26.8 $\pm$ 1.2	35.3 $\pm$ 1.5	56.9 $\pm$ 1.3	34.7 $\pm$ 0.8	43.6 $\pm$ 0.8	55.8 $\pm$ 0.6	34.9 $\pm$ 0.7	43.1 $\pm$ 0.6	62.0 $\pm$ 0.7
AutoPalette	19.4 $\pm$ 2.5	32.0 $\pm$ 1.1	56.8 $\pm$ 0.7	28.3 $\pm$ 1.1	39.2 $\pm$ 1.1	61.7 $\pm$ 0.5	29.7 $\pm$ 2.1	50.1 $\pm$ 1.3	61.3 $\pm$ 0.5
QuADD (Ours)	21.4 $\pm$ 0.7	29.7 $\pm$ 0.7	50.3 $\pm$ 0.8	29.8 $\pm$ 1.0	39.6 $\pm$ 0.7	63.5 $\pm$ 0.5	30.7 $\pm$ 1.2	51.8 $\pm$ 0.5	66.6 $\pm$ 0.4

Table 2: Accuracy comparison of different methods across AlexNet, VGG, and ResNet under varying IPC values on CIFAR-10. Best results in each column are highlighted in bold.

Method	DM			TM
Method	IPC1	IPC10	IPC50	IPC1	IPC10	IPC50
Vanilla (unquantized)	26.0 $\pm$ 0.8	48.3 $\pm$ 0.6	63.0 $\pm$ 0.4	45.7 $\pm$ 0.8	65.2 $\pm$ 0.7	71.4 $\pm$ 0.2
FreD	33.6 $\pm$ 0.4	48.1 $\pm$ 0.8	62.0 $\pm$ 0.6	43.1 $\pm$ 0.3	57.3 $\pm$ 0.5	69.0 $\pm$ 0.2
AutoPalette	12.0 $\pm$ 0.4	15.3 $\pm$ 0.2	48.5 $\pm$ 0.4	42.0 $\pm$ 1.1	61.2 $\pm$ 0.2	69.5 $\pm$ 0.2
QuADD (Ours)	17.5 $\pm$ 0.5	45.1 $\pm$ 0.3	62.4 $\pm$ 0.3	44.7 $\pm$ 0.5	63.2 $\pm$ 0.2	70.5 $\pm$ 0.2

Table 3: Performance comparison across distillation methods on DM and TM datasets under varying IPC values on CIFAR-10. The unquantized baseline serves as a reference for all methods. Best results in each column are highlighted in bold.

Figure 6: Overhead reduction and prediction accuracy on beam management problem. Higher is better.

Figure 7: Accuracy–rate trade-off on the 3GPP beam management dataset for different sample counts (SPC = 10, 50, Full).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Benchmark comparison between uniform quantization variants and non-uniform (APoT) quantization on CIFAR-10 for IPC 1 and IPC 10. Each curve evaluates synthetic data quantized from 1 to 6 bits per channel (resulting in $b\times 3$ bits per RGB pixel). The unquantized baseline (DATM) is stored at standard 32-bit precision.

Dataset	IPC	Synth. BS	Synth. steps	Expert ep.	Max start ep.	Quant. LR	Img. LR	Step LR	Teacher LR	ZCA
CIFAR-10	1	200	80	2	15	0.1	500	$10^{-7}$	$10^{-2}$	True
	10	400	35	2	40	0.1	1000	$10^{-5}$	$10^{-2}$	True
	50	200	60	2	40	0.1	500	$10^{-5}$	$10^{-2}$	True
CIFAR-100	1	200	60	2	25	0.1	1000	$10^{-5}$	$10^{-2}$	True
	10	500	50	2	70	0.1	1000	$10^{-5}$	$10^{-2}$	True
	50	800	80	2	70	0.1	1000	$10^{-5}$	$10^{-2}$	True

Appendix

Table 4: Hyperparameters for all methods based on Trajectory Matching (TM) framework.

Dataset	IPC	Synthetic batch size	LR (Quantizer)	LR (Synthetic Image)	ZCA
CIFAR-10	1	–	0.1	1	True
	10	–	0.1	1	True
	50	–	0.1	10	True
CIFAR-100	1	–	0.1	1	True
	10	–	0.1	1	True
	50	50	0.1	10	True

Appendix

Table 5: Hyperparameters for our method based on the Distribution Matching (DM) framework.

Figure 9: Illustration of the 3GPP beam management problem under one-sided beam scanning. The base station (BS) transmits an omnidirectional or fixed reference beam, while the user equipment (UE) performs directional scanning across azimuth (

\theta

) and elevation (

\phi

) angles. The resulting

N_{\theta}\times N_{\phi}

grid of received beam powers represents the UE’s spatial response pattern. Only a subset of these directions—here, 50%—is measured, simulating limited probing due to time or energy constraints. The masked and observed beam powers form a tabular dataset used as input to the AI/ML model, with missing entries representing unmeasured spatial directions.

Figure 10: AI/ML-based inference for optimal beam prediction in the 3GPP beam management problem. The masked beam-power matrix from Fig. 9 is processed by an AI/ML model trained to nonlinearly interpolate missing measurements and predict the beam direction corresponding to the maximum reconstructed power. The model outputs the estimated index

(i_{\text{max}},j_{\text{max}})

of the optimal beam, trained using a cross-entropy loss over all possible beam indices.

Figure 11: Effect of sampling strategy on the imbalanced 3GPP wireless dataset at IPC 50. For classes with fewer than 50 real samples, we initialize $\mathcal{S}$ either by (i) using all available samples (sampling without replacement), or (ii) duplicating samples to reach 50 via sampling with replacement.

Quantization-Aware Distillation

See all 14 papers

Vector-Quantized Soft Label Compression for Dataset Distillation
Mar 4, 2026Ali Abbasi, Ashkan Shahbazi, Hamed Pirsiavash +1Dataset Distillation Vector Quantization
Vanderbilt University · University of California, Davis
Post Training Quantization for Efficient Dataset Condensation
Mar 7, 2026Linh-Tam Tran, Sung-Ho BaeDataset Condensation Post-Training Quantization
Kyung Hee University, Republic of Korea
Accelerating Large-Scale Dataset Distillation via Exploration-Exploitation Optimization
Feb 17, 2026Muhammad J. Alahmadi, Peng Gao, Feiyi Wang +1Dataset Distillation
North Carolina State University · King Abdulaziz University · Oak Ridge National Laboratory

Mar 4, 2026J/K move · Enter open · S save

Vector-Quantized Soft Label Compression for Dataset Distillation

Ali Abbasi, Ashkan Shahbazi, Hamed Pirsiavash, Soheil Kolouri

Vanderbilt University · University of California, Davis

Dataset Distillation Vector Quantization Knowledge Distillation

Open paper AskPDF arXiv

Dataset distillation is an emerging technique for reducing the computational and storage costs of training machine learning models by synthesizing a small, informative subset of data that captures the essential characteristics of a much larger dataset. Recent methods pair synthetic samples and their augmentations with soft labels from a teacher model, enabling student models to generalize effectively despite the small size of the distilled dataset. While soft labels are critical for effective distillation, the storage and communication overhead they incur, especially when accounting for augmentations, is often overlooked. In practice, each distilled sample is associated with multiple soft labels, making them the dominant contributor to storage costs, particularly in large-class settings such as ImageNet-1K. In this paper, we present a rigorous analysis of bit requirements across dataset distillation frameworks, quantifying the storage demands of both distilled samples and their soft labels. To address the overhead, we introduce a vector-quantized autoencoder (VQAE) for compressing soft labels, achieving substantial compression while preserving the effectiveness of the distilled data. We validate our method on both vision and language distillation benchmarks. On ImageNet-1K, our proposed VQAE achieves 30--40x additional compression over RDED, LPLD, SRE2L, and CDA baselines while retaining over

90\%

of their original performance.

Abstract

Quantization-Aware Distillation

Explore similar work

Vector-Quantized Soft Label Compression for Dataset Distillation

Post Training Quantization for Efficient Dataset Condensation

Accelerating Large-Scale Dataset Distillation via Exploration-Exploitation Optimization