A Dataset is Worth 1 MB

Elad Kimchi Shoshani; Leeyam Gabay; Yedid Hoshen

cs.LGFeb 26, 2026

A Dataset is Worth 1 MB

Authors: Elad Kimchi Shoshani, Leeyam Gabay, Yedid Hoshen

Organizations: School of Computer Science and Engineering, The Hebrew University of Jerusalem, Israel.

Abstract

A dataset server must often distribute the same large payload to many clients, incurring massive communication costs. Since clients frequently operate on diverse hardware and software frameworks, transmitting a pre-trained model is often infeasible; instead, agents require raw data to train their own task-specific models locally. While dataset distillation attempts to compress training signals, current methods struggle to scale to high-resolution data and rarely achieve sufficiently small files. In this paper, we propose Pseudo-Labels as Data (PLADA), a method that completely eliminates pixel transmission. We assume agents are preloaded with a large, generic, unlabeled reference dataset (e.g., ImageNet-1K, ImageNet-21K) and communicate a new task by transmitting only the class labels for specific images. To address the distribution mismatch between the reference and target datasets, we introduce a pruning mechanism that filters the reference dataset to retain only the labels of the most semantically relevant images for the target task. This selection process simultaneously maximizes training efficiency and minimizes transmission payload. Experiments on 10 diverse datasets demonstrate that our approach can transfer task knowledge with a payload of less than 1 MB while retaining high classification accuracy, offering a promising solution for efficient dataset serving.

Figures & tables

Figure 1: Motivation. A dataset server transmits the same large dataset many times at massive cost. Our method allows the server to send a compressed payload of less than 1 MB, enabling clients with heterogeneous hardware, even if they have ultra-narrow bandwidth, to train their own models locally.

Figure 2: The PLADA Pipeline. The server (left) trains a teacher classifier on the task dataset and distills this task knowledge into hard labels on the reference data. It then filters to the lowest-uncertainty $p\%$ of pseudo-labels and transmits a compressed payload ( $<1$ MB). The client (right) reconstructs a virtual dataset using its preloaded reference dataset and the payload to train the student model.

Figure 3: Reference set images vs. energy percentile. High-confidence (low-energy) samples retrieved from ImageNet-21K demonstrate semantic and structural alignment with the target domains. For additional visualizations see Appendix C .

Figure 4: Class distribution of the RESISC45 pseudo hard-labels , before and after filtering using safety-net. The yellow bars show the original global distribution, which is heavily imbalanced - RESISC45 has images extracted using Google Earth, out of which class 0 is airplane . Standard global filtering would eliminate some of the tail classes entirely. The blue bars demonstrate our Safety-Net Filtering (keeping 5%,

\alpha=-0.2

), which effectively preserves a representation of under-represented classes even under extreme compression. Note that the Y-axis uses a cube-root scale to visually accommodate the large magnitude differences between the ‘strong‘ and ‘weak‘ classes.

	ImageNet-21K (14.2M images)						ImageNet-1K (1.2M images)
Dataset	1%	5%	10%	25%	50%	100%*	1%	5%	10%	25%	50%	100%*
Caltech-101	79.84%	88.94%	90.21%	90.73%	92.45%	92.74%	66.59%	77.13%	82.49%	86.35%	86.52%	87.50%
CIFAR-10	63.31%	85.31%	88.12%	91.68%	92.62%	86.13%	53.75%	72.02%	74.83%	79.90%	84.96%	87.66%
CUB-200	82.49%	82.36%	82.44%	81.34%	81.09%	74.94%	22.94%	45.89%	51.40%	56.19%	55.60%	52.97%
DTD	66.65%	70.16%	70.69%	70.80%	68.83%	68.14%	52.45%	58.99%	60.48%	60.90%	61.97%	62.29%
FGVC-Aircraft	53.62%	45.51%	46.41%	45.87%	43.53%	32.16%	20.04%	23.91%	26.58%	29.16%	30.18%	29.40%

Table 1: Keep rate evaluations. Student accuracy when trained on the top-

p

images of the reference set, according to logit Energy. We achieve accurate classification on target tasks using only a fraction of the reference set.

Dataset	Ours (p=1%)	Random	K-Centers	Random	K-Centers	DD †
		Using 100 Images		Using 500 Images
Caltech-101	86.69%	32.78%	34.16%	47.98%	51.04%	–
CIFAR-10	76.75%	28.66%	19.33%	31.29%	27.20%	73.2%
CUB-200	82.49%	4.58%	3.69%	9.67%	7.55%	16.2%
DTD	68.09%	19.04%	14.73%	36.49%	28.35%	–
FGVC-Aircraft	53.62%	2.76%	2.10%	4.62%	4.59%	–

Table 2: Baseline comparison. We compare student accuracy across 10 benchmarks. Our method, PLADA (using 1% keep ratio with possible Safety-Net filtering on ImageNet-21K, see Table 3 ), outperforms data-transmission baselines—including random sampling, K-Center coresets, and Dataset Distillation (DD)—in both accuracy and payload size. Notably, PLADA achieves superior task recovery while requiring a payload significantly smaller than even the aggressive 100-image JPEG-compressed subsets.

Dataset	1% Vanil.	1% Oppos.	1%+Safe ( $\alpha=0.5$ )	1%+Safe ( $\alpha=-0.2$ )
Caltech-101	79.84%	74.42%	86.69%	86.29%
CIFAR-10	63.31%	54.80%	76.75%	74.62%
CUB-200	82.49%	6.19%	81.21%	80.53%
DTD	66.65%	26.12%	66.70%	68.09%
FGVC-Aircraft	53.62%	3.45%	43.23%	44.58%
Food-101	75.50%	6.28%	70.91%	71.66%

Table 3: Impact of safety-net filtering on Student Accuracy. All 1% subsets are sampled from ImageNet-21K. We compare the default lowest-energy filtering and its counterpart (highest-energy filtering) against Safety-Net variants. Low-energy samples (Vanilla) outperform high-energy ones. Safety-Net often further improves accuracy by preventing class collapse, with the same payload budget. The difference is the highest for RESISC45 (cf. Figure 4 ). Additional results are reported in Table 10 .

$p$	Raw Size	Huffman	Zstd
0.5%	0.41–1.83 MB	77–305 KB	45–109 KB
1%	0.81–1.96 MB	151–396 KB	85–206 KB
5%	3.05 MB	0.57–1.10 MB	0.40–0.88 MB
10%	4.40–8.12 MB	0.88–1.95 MB	0.67–1.58 MB
25%	8.46 MB	1.65–4.34 MB	1.21–3.47 MB
50%	15.23–40.62 MB	2.49–7.88 MB	1.87–6.42 MB

Table 4: Payload size analysis. We compare across keep ratios (

p

) and compression schemes. Ranges represent the minimum and maximum sizes observed across all 10 different datasets, and across different filtering options (with and without safety-net), with ImageNet-21K as the reference dataset. Raw indicates uncompressed fixed-width binary storage. Zstd represents the final compressed payload size using differential encoding and Zstandard compression (level 19). Full compression experiments results are provided in Appendix E .

Dataset	1% Vanil.	1% Oppos.	1%+Safe ( $\alpha=0.5$ )	1%+Safe ( $\alpha=-0.2$ )
BloodMNIST	18.24%	59.28%	47.00%	41.45%
DermaMNIST	53.32%	67.68%	47.58%	38.05%
RetinaMNIST	56.50%	56.75%	55.25%	55.00%
NCT-CRC-HE	18.69%	43.51%	32.57%	32.37%

Table 5: Results on medical datasets † . ImageNet-21K ref. set.

Figure 5: Bandwidth-Accuracy Baselines (CUB-200). Comparison of PLADA against weight and data transmission baselines. PLADA (red star) dominates the top-left corner, achieving higher accuracy than weight-based methods while requiring a smaller payload (¡35 KB). Data-centric baselines (Random Subset/K-Center) fail to provide a viable signal at this extreme budget. All payloads are Zstd-compressed (level 19).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Target Dataset	Total Intersection	Test Intersection
Oxford Flowers 102	0 / 8,189	0 / 6,149
Food-101	0 / 101,000	0 / 25,250
DTD	0 / 3,760	0 / 1,880
CIFAR-10	0 / 60,000	0 / 10,000
RESISC45	0 / 31,500	0 / 6,300
Places365	0 / 1,839,960	0 / 36,500

Appendix

Table 6: Intersection analysis between Target Datasets and ImageNet-21K (14.2M). Columns show the number of duplicates found in the full target dataset (

x/n_{all}

) and the specific test set (

y/n_{test}

	1% Filter		5% Filter		10% Filter		25% Filter
Dataset	Entropy	Energy	Entropy	Energy	Entropy	Energy	Entropy	Energy
Caltech-101	75.06%	79.84%	84.22%	88.94%	87.56%	90.21%	91.71%	90.73%
CIFAR-10	72.57%	63.31%	86.00%	85.31%	89.14%	88.12%	91.14%	91.68%
CUB-200	77.44%	82.49%	81.42%	82.36%	81.76%	82.44%	81.59%	81.34%
DTD	67.02%	66.65%	69.79%	70.16%	71.22%	70.69%	71.06%	70.80%
FGVC-Aircraft	37.32%	53.62%	44.13%	45.51%	45.69%	46.41%	45.90%	45.87%

Appendix

Table 7: Entropy vs Energy Pruning. This table reports student accuracy † under different filtering budgets, comparing entropy-based pruning with energy-based pruning (ImageNet-21K).

	ImageNet-21K (14.2M images)			ImageNet-1K (1.2M images)
Dataset	1%	5%	10%	1%	5%	10%	25%
Caltech-101	74.42%	85.66%	87.27%	61.29%	79.32%	82.89%	85.54%
CIFAR-10	54.80%	64.45%	70.12%	34.90%	45.84%	58.46%	67.09%
CUB-200	6.19%	11.79%	15.48%	3.22%	6.66%	8.78%	14.16%
DTD	26.12%	37.34%	42.77%	19.41%	27.45%	35.90%	40.00%
FGVC-Aircraft	3.45%	5.37%	8.16%	2.46%	3.57%	4.74%	8.70%

Appendix

Table 8: Opposite Energy-Based Pruning. This table shows the student accuracy obtained by training on the highest -

p\%

energy-score images (which are usually the ”worst” images), using ImageNet-1K and ImageNet-21K as reference datasets.

	ImageNet-21K (14.2M images)		ImageNet-1K (1.2M images)
Dataset	1%	5%	1%	5%
BloodMNIST	59.28%	57.88%	32.65%	38.03%
DermaMNIST	67.68%	66.58%	66.43%	67.33%
RetinaMNIST	56.75%	58.00%	50.25%	55.50%
NCT-CRC-HE-100K	43.51%	52.48%	26.46%	35.33%

Appendix

Table 9: Medical Datasets Opposite Energy-Based Pruning. This table presents student accuracy obtained by training on the highest

p\%

energy-score images of the reference sets. As explained in Section 5.3 , we unexpectedly observed higher student accuracy when applying the opposite filtering strategy on the medical datasets.

	ImageNet-21K				ImageNet-1K
	1%		5%		1%		5%		10%		25%
Dataset	$\alpha\!=\!-0.2$	$\alpha\!=\!0.5$	$\alpha\!=\!-0.2$	$\alpha\!=\!0.5$	$\alpha\!=\!-0.2$	$\alpha\!=\!0.5$	$\alpha\!=\!-0.2$	$\alpha\!=\!0.5$	$\alpha\!=\!-0.2$	$\alpha\!=\!0.5$	$\alpha\!=\!-0.2$	$\alpha\!=\!0.5$
Caltech-101	86.29%	86.69%	90.67%	91.07%	77.25%	77.25%	83.41%	83.81%	86.81%	85.20%	86.06%	85.83%
CIFAR-10	74.62%	76.75%	83.83%	85.44%	58.07%	58.59%	69.97%	68.27%	73.51%	72.56%	77.00%	78.26%
CUB-200	80.53%	81.21%	82.06%	82.06%	15.48%	15.78%	41.26%	39.61%	52.54%	52.50%	48.39%	49.15%
DTD	68.09%	66.70%	70.59%	70.48%	53.62%	51.65%	60.16%	60.11%	61.81%	62.29%	61.28%	62.77%

Appendix

Table 10: Safety-net Filtering. This table shows student accuracy when using Safety-Net filtering with

\alpha=-0.2,0.5

across different filtering keep ratios, with ImageNet-21K and ImageNet-1K as reference sets.

	ImageNet-21K				ImageNet-1K
	1%		5%		1%		5%
Dataset	$\alpha\!=\!-0.2$	$\alpha\!=\!0.5$	$\alpha\!=\!-0.2$	$\alpha\!=\!0.5$	$\alpha\!=\!-0.2$	$\alpha\!=\!0.5$	$\alpha\!=\!-0.2$	$\alpha\!=\!0.5$
BloodMNIST	41.45%	47.00%	56.71%	45.19%	38.91%	42.03%	45.31%	48.44%
DermaMNIST	38.05%	47.58%	68.33%	68.33%	51.67%	53.42%	57.06%	57.91%
NCT-CRC-HE-100K	32.37%	32.57%	41.89%	37.54%	23.79%	28.23%	33.88%	39.47%
RetinaMNIST	55.00%	55.25%	63.25%	61.25%	53.75%	55.75%	58.50%	58.25%

Appendix

Table 11: Medical Datasets Safety-net Filtering. This table shows student accuracy when using Safety-Net filtering with

\alpha=-0.2,0.5

across different filtering keep ratios, with ImageNet-21K and ImageNet-1K as reference sets.

Figure 6: Energy percentiles on ImageNet-1K. For three target tasks (rows: Oxford-Flowers102, DTD, FGVC-Aircraft), we score every ImageNet-1K image using the corresponding target teacher and sort the reference set by logit energy ( Equation 5 ; lower is better). We then show exemplar reference images at fixed energy percentiles (columns). The horizontal bar under each row visualizes the full energy range over the entire reference set (dashed ticks mark the sampled percentiles). As the percentile increases (left $\rightarrow$ right), samples transition from target-aligned content (flowers/texture patterns/aircraft) to increasingly irrelevant images.

Figure 7: Energy percentiles on ImageNet-21K. Same visualization as Figure 6 but using ImageNet-21K (14.2M images) as the reference set. The larger and more diverse dataset typically provides closer semantic neighbors in the low-energy tail (e.g., more flower varieties and texture-like patterns).

Figure 8: Low-energy reference images (ImageNet-1K). For each target teacher (rows; top-to-bottom: Caltech-101, CIFAR-10, CUB-200, DTD, FGVC-Aircraft, Food-101, Oxford-Flowers, Oxford-Pets, Places365, RESISC45), we show ImageNet-1K reference images drawn from increasingly larger low-energy percentiles (columns: 0.0001%–1.5%). The extreme low-energy tail tends to contain canonical instances of the target concepts (e.g., birds for CUB-200, food dishes for Food-101, flowers for Oxford-Flowers).

Figure 9: Low-energy reference images (ImageNet-21K). Same as Figure 8 but using ImageNet-21K as the reference set. The larger reference set yields a richer and often more semantically aligned set of low-energy exemplars across targets, consistent with the higher student accuracy obtained with ImageNet-21K as the reference set in Table 1 .

	ImageNet-21K (14.2M images)				ImageNet-1K (1.2M images)
$p$	Raw	Compact	Huffman	Zstd	Raw	Compact	Huffman	Zstd
0.1% (Idx)	83.19 KB	69.32 KB	21.66 KB	20.95 KB	7.51 KB	6.25 KB	1.57 KB	1.73 KB
0.1% (Bmp)	1.72 MB	1.71 MB	230.84 KB	21.61 KB	158.89 KB	157.64 KB	20.71 KB	1.62 KB
0.5% (Idx)	415.93 KB	346.61 KB	106.16 KB	95.39 KB	37.53 KB	18.76 KB	8.63 KB	6.77 KB
0.5% (Bmp)	1.83 MB	1.76 MB	289.91 KB	84.14 KB	168.90 KB	162.65 KB	26.05 KB	6.52 KB
1% (Idx)	831.86 KB	415.93 KB	200.09 KB	151.80 KB	75.06 KB	37.53 KB	16.86 KB	12.59 KB

Appendix

Table 12: Compression Summary for Caltech-101, CIFAR-10, CUB-200-2011. In bold we highlight the best compression for every 2 rows (as we also compare between bitmap and delta indices encodings for the pruning bit).

	ImageNet-21K (14.2M images)				ImageNet-1K (1.2M images)
$p$	Raw	Compact	Huffman	Zstd	Raw	Compact	Huffman	Zstd
0.1% (Idx)	83.19 KB	69.32 KB	21.10 KB	22.48 KB	7.51 KB	3.75 KB	1.46 KB	1.61 KB
0.1% (Bmp)	1.72 MB	1.71 MB	230.01 KB	23.76 KB	158.89 KB	157.64 KB	20.55 KB	1.91 KB
0.5% (Idx)	415.93 KB	207.96 KB	102.81 KB	91.67 KB	37.53 KB	18.76 KB	7.94 KB	7.50 KB
0.5% (Bmp)	1.83 MB	1.76 MB	286.34 KB	95.17 KB	168.90 KB	162.65 KB	25.03 KB	7.47 KB
1% (Idx)	831.86 KB	415.93 KB	197.48 KB	175.57 KB	75.06 KB	37.53 KB	15.83 KB	14.57 KB

Appendix

Table 13: Compression Summary for DTD, FGVC-AIRCRAFT, FOOD-101

	ImageNet-21K (14.2M images)				ImageNet-1K (1.2M images)
$p$	Raw	Compact	Huffman	Zstd	Raw	Compact	Huffman	Zstd
0.1% (Idx)	83.19 KB	69.32 KB	18.69 KB	15.95 KB	7.51 KB	6.25 KB	0.96 KB	1.32 KB
0.1% (Bmp)	1.72 MB	1.71 MB	231.20 KB	14.31 KB	158.89 KB	157.64 KB	20.37 KB	1.13 KB
0.5% (Idx)	415.93 KB	346.61 KB	87.85 KB	60.86 KB	37.53 KB	18.76 KB	7.23 KB	6.80 KB
0.5% (Bmp)	1.83 MB	1.76 MB	292.83 KB	51.47 KB	168.90 KB	162.65 KB	24.94 KB	7.07 KB
1% (Idx)	831.86 KB	693.21 KB	168.27 KB	107.58 KB	75.06 KB	37.53 KB	16.24 KB	14.81 KB

Appendix

Table 14: Compression Summary for Oxford-Flowers102, Oxford-IIIT-Pet, Places365. Note that compression is less effective on Places365 due to its larger number of classes and reduced redundancy.

	ImageNet-21K (14.2M images)				ImageNet-1K (1.2M images)
$p$	Raw	Compact	Huffman	Zstd	Raw	Compact	Huffman	Zstd
0.1% (Idx)	83.19 KB	41.59 KB	19.83 KB	22.82 KB	7.51 KB	3.75 KB	1.79 KB	2.35 KB
0.1% (Bmp)	1.72 MB	1.71 MB	226.58 KB	25.10 KB	158.89 KB	157.64 KB	20.44 KB	3.03 KB
0.5% (Idx)	415.93 KB	207.96 KB	87.16 KB	94.19 KB	37.53 KB	18.76 KB	8.58 KB	9.92 KB
0.5% (Bmp)	1.83 MB	1.76 MB	270.34 KB	91.43 KB	168.90 KB	162.65 KB	24.27 KB	10.31 KB
1% (Idx)	831.86 KB	415.93 KB	162.79 KB	172.20 KB	75.06 KB	37.53 KB	16.18 KB	18.20 KB

Appendix

Table 15: Compression Summary for RESISC45

$p$	Raw	Compact	Huffman	Zstd
0.1%	7.51–158.89 KB	3.75–157.64 KB	1.46–20.71 KB	1.08–2.53 KB
0.5%	37.53–168.90 KB	18.76–168.90 KB	7.23–26.28 KB	3.15–9.92 KB
1%	181.41 KB	168.90–181.41 KB	28.34–33.70 KB	7.86–17.95 KB
5%	281.51 KB	218.95–281.51 KB	68.02–98.31 KB	35.13–77.31 KB
10%	406.62 KB	281.51–406.62 KB	110.48–179.47 KB	67.80–142.45 KB
25%	0.76–1.83 MB	469.18–781.96 KB	223.66–403.47 KB	149.08–314.62 KB

Appendix

Table 16: Compression Summary Over All Datasets: ImageNet-1K (1.2M images). This table summarizes the compression results over all of the datasets for all

p

values. The values in the table are min-max sizes.

$p$	Raw	Compact	Huffman	Zstd
0.1%	0.08–1.72 MB	0.04–1.71 MB	17.43–233.01 KB	14.07–27.13 KB
0.5%	0.41–1.83 MB	0.20–1.83 MB	77.75–305.21 KB	45.48–108.84 KB
1%	0.81–1.96 MB	0.41–1.96 MB	151.00–396.44 KB	84.83–206.08 KB
5%	3.05 MB	2.37–3.05 MB	0.57–1.10 MB	401.63–877.05 KB
10%	4.40–8.12 MB	2.71–4.40 MB	0.88–1.95 MB	0.67–1.58 MB
25%	8.46 MB	5.08–8.46 MB	1.65–4.34 MB	1.21–3.47 MB

Appendix

Table 17: Compression Summary Over All Datasets: ImageNet-21K (14.2M images). In comparison to Table 16 , the payloads are roughly 10-12x larger.

Data-Efficient Learning

Data-efficient learning develops methods that achieve accurate, robust models with fewer labeled examples, training samples, or interactions.

See all 243 papers

Soft Label Pruning and Quantization for Large-Scale Dataset Distillation
Apr 20, 2026Xiao Lingao, Yang HeDataset Distillation Image Classification
CFAR, Agency for Science, Technology and Research, Singapore 138632 · IHPC, Agency for Science, Technology and Research, Singapore 138632 · National University of Singapore, Singapore 119077
Condensing Large-Scale Datasets Directly with Minimal Information Loss
Jul 1, 2026Xinyi Shang, Peng Sun, Bei Shi +2Distribution Shift Distribution Matching
University College London, United Kingdom · Zhejiang University, China · Westlake University, China +1
Accelerating Large-Scale Dataset Distillation via Exploration-Exploitation Optimization
Feb 17, 2026Muhammad J. Alahmadi, Peng Gao, Feiyi Wang +1Dataset Distillation
North Carolina State University · King Abdulaziz University · Oak Ridge National Laboratory

Apr 20, 2026J/K move · Enter open · S save

Soft Label Pruning and Quantization for Large-Scale Dataset Distillation

Xiao Lingao, Yang He

CFAR, Agency for Science, Technology and Research, Singapore 138632 · IHPC, Agency for Science, Technology and Research, Singapore 138632 · National University of Singapore, Singapore 119077

Dataset Distillation Image Classification

Open paper AskPDF arXiv

Large-scale dataset distillation requires storing auxiliary soft labels that can be 30-40x larger on ImageNet-1K and 200x larger on ImageNet-21K than the condensed images, undermining the goal of dataset compression. We identify two fundamental issues necessitating such extensive labels: (1) insufficient image diversity, where high within-class similarity in synthetic images requires extensive augmentation, and (2) insufficient supervision diversity, where limited variety in supervisory signals during training leads to performance degradation at high compression rates. To address these challenges, we propose Label Pruning and Quantization for Large-scale Distillation (LPQLD). We enhance image diversity via class-wise batching and batch-normalization supervision during synthesis. For supervision diversity, we introduce Label Pruning with Dynamic Knowledge Reuse to improve label-per-augmentation diversity, and Label Quantization with Calibrated Student-Teacher Alignment to improve augmentation-per-image diversity. Our approach reduces soft label storage by 78x on ImageNet-1K and 500x on ImageNet-21K while improving accuracy by up to 7.2% and 2.8%, respectively. Extensive experiments validate the superiority of LPQLD across different network architectures and dataset distillation methods. Code is available at https://github.com/he-y/soft-label-pruning-quantization-for-dataset-distillation.

Abstract

Data-Efficient Learning

Explore similar work

Soft Label Pruning and Quantization for Large-Scale Dataset Distillation

Condensing Large-Scale Datasets Directly with Minimal Information Loss

Accelerating Large-Scale Dataset Distillation via Exploration-Exploitation Optimization