Data Density Scaling Laws for Image Self-Distillation

Alvard Barseghyan; Ani Vanyan; Hakob Tamazyan; Hrant Khachatrian

cs.LGOct 13, 2026

Data Density Scaling Laws for Image Self-Distillation

Authors: Alvard Barseghyan, Ani Vanyan, Hakob Tamazyan, Hrant Khachatrian

Organizations: YerevaNN and Yerevan State University, Yerevan, Armenia

Abstract

Current scaling law formulations suggest that increasing the number of unique samples in a dataset improves model performance, while repeated exposure to the same samples provides diminishing value. In this work, we examine this assumption for image self-distillation models. Through systematic experiments across diverse datasets, namely Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M, we demonstrate that the effects of sample repetition on downstream performance are non-trivial and dataset-dependent. We identify settings where repeatedly training on a smaller random subset can outperform training on a larger set of unique samples. This finding highlights a limitation of existing scaling laws. To address this gap, we propose a modified version of a Chinchilla-style scaling law that explicitly accounts for two factors: the number of unique samples and repetition per sample. This formulation provides a framework for understanding data-compute trade-offs and improves prediction in some regimes where repetition plays a central role, although its benefits are not consistent across datasets. Finally, we examine how subset selection and augmentation affect downstream performance.

Figures & tables

Figure 1: Samples from pretraining datasets

Figure 2: Pretraining grid across datasets. Each point corresponds to a run with unique data size

U

and compute budget

C

(both in millions). Colors denote Maxar, Waymo, and ImageNet. Markers indicate sampling strategy: full data, random subsets, and hierarchical subsets. Diagonal lines show constant repetition

R

, highlighting the trade-off between unique data and reuse. The fire icons indicate non-stable pre-trainings.

Figure 3: Behaviour of different random subsets under the kNN evaluation protocol. The first column fixes compute

C

and plots downstream performance as a function of unique images

U

. The middle column fixes

U

and plots performance as a function of compute

C

. The third column fixes

U

and plots performance as a function of repetition

R

. The figure shows the trade-off between unique support and repetition for full-data and random-subset runs.

Figure 4: Scaling-law fits and extrapolations under the kNN evaluation protocol. Columns compare three existing-law baselines—compute-only, data-constrained, and prescriptive—with our density-interaction law. These four forms test whether compute alone, diminishing returns from repetition, a repetition penalty, or a support–repetition interaction captures the observed behavior. Rows show Maxar, nuPlan, ImageNet, and YFCC15M. For compactness, SATLAS and Ego4D are omitted because they show similar qualitative patterns to Maxar, while Waymo shows similar observed trends to nuPlan. Curves are fitted only on the shaded low-compute region and evaluated on higher-compute runs. Fitting curves for all datasets and the logistic-regression comparison can be found in Appendices D and F .

Law	Maxar		SATLAS		Waymo		nuPlan		Ego4D		ImageNet		YFCC15M
	fit	extrap.	extrap.	fit	extrap.	fit	extrap.	fit	extrap.	fit	extrap.	fit	extrap.
Compute-only	0.0050	0.0116	0.0022	0.0060	0.0019	0.0021	0.0010	0.0028	0.0012	0.0033	0.0234	0.0507	0.0033	0.0129
Data-constrained	0.0050	0.0116	0.0022	0.0058	0.0017	0.0024	0.0009	0.0033	0.0012	0.0033	0.0125	0.0575	0.0034	0.0132
Prescriptive	0.0050	0.0116	0.0020	0.0057	0.0015	0.0023	0.0009	0.0028	0.0011	0.0034	0.0110	0.0644	0.0033	0.0129
Density interaction	0.0032	0.0093	0.0016	0.0043	0.0015	0.0022	0.0008	0.0200	0.0012	0.0027	0.0080	0.0948	0.0019	0.0224

Table 1: kNN scaling-law MAE in accuracy units, fitted on

C\leq 50

M and evaluated on

C>50

M. Only eligible random subsets and Full are included. Each task uses a fixed

k

chosen from Full-50M reference seeds. Bold and underlining indicate the lowest and second-lowest distinct unrounded held-out errors in each dataset.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Sampling	100prc	75prc	50prc	37.5prc	25prc	12.5prc	6.25prc
Maxar	Random	47.99	47.99	47.99	47.90	47.90	47.96	47.89
	Hierarchical	–	60.51	60.51	66.38	66.38	70.61	74.67
Waymo	Random	73.38	73.40	73.39	73.42	73.40	73.48	73.42
	Hierarchical	–	87.81	107.26	119.67	136.68	–	172.22
ImageNet	Random	589.65	589.75	589.59	589.61	589.73	589.77	589.64
	Hierarchical	–	584.31	587.46	–	–	–	–

Appendix

Table 2: Vendi scores by dataset, sampling strategy, and target subset percentage.

Task	Train	Eval	Classes	Maxar $k$	SATLAS $k$
Brick Kiln [ 27 ]	58,890	7,361	2	10	5
EuroSAT [ 28 ]	16,200	5,400	10	5	5
RESISC45 [ 29 ]	18,900	6,300	45	1	5
UCMerced [ 30 ]	1,260	420	21	1	1
PatternNet [ 31 ]	18,240	6,080	38	1	5
AID [ 32 ]	6,000	2,000	30	1	1

Appendix

Table 3: Remote-sensing downstream tasks and fixed reference

k

values.

Task	Train	Eval	Classes	Waymo $k$	nuPlan $k$
BDD100K scene [ 33 ]	69,863	10,000	7	30	30
BDD100K weather [ 33 ]	69,863	10,000	7	30	30
ACDC condition [ 34 ]	1,600	2,000	4	50	30
BDD100K MOT instance	–	–	–	–	–
BDD100K MOT category	–	–	–	–	–

Appendix

Table 4: Driving downstream tasks and fixed reference

k

values. Tracking scores enter only the kNN aggregate; their sample counts are unspecified.

Task	Train	Eval	Classes	Ego $k$	IN $k$	YFCC $k$
ImageNet-1K [ 12 ]	1,281,167	50,000	1,000	30	10	30
Stanford Cars [ 35 ]	8,144	8,041	196	1	1	1
CIFAR-100 [ 36 ]	50,000	10,000	100	10	10	30
CIFAR-10 [ 36 ]	50,000	10,000	10	10	10	10
CUB-200 [ 37 ]	5,994	5,794	200	30	30	30
DTD [ 38 ]	1,880	1,880	47	10	1	10

Appendix

Table 5: General and egocentric downstream tasks. Fixed reference

k

values are shown for Ego4D (Ego), ImageNet (IN), and YFCC15M (YFCC). A dash indicates that the task is not evaluated for that group.

Figure 5: Unique-support and repetition trade-offs under logistic-regression evaluation. Rows correspond to Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M. Columns show mean downstream accuracy as unique images

U

vary at fixed compute

C

, compute varies at fixed

U

, and repetition

R=C/\min(C,U)

varies at fixed

U

, respectively. Points represent random-subset and Full-data runs; lines connect observations. Colors indicate compute in the left column and unique support in the middle and right columns, which share a color key. Horizontal axes are logarithmic; image counts are in millions.

Figure 6: Scaling-law fits and extrapolations under the logistic-regression evaluation protocol for all seven datasets. Columns compare the compute-only Chinchilla-style law, the data-constrained effective-data law, the prescriptive scaling law, and our density-interaction law. Rows correspond to Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M. Curves are fitted only in the shaded low-compute region (

C\leq 50

M) and evaluated on higher-compute runs (

C>50

M). Each panel shows the fitted equation and held-out MAE in accuracy units. Parameters are fitted separately from the kNN evaluation.

Figure 7: Unique-support and repetition trade-offs under kNN evaluation. Rows correspond to Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M. The left column fixes compute

C

and varies unique images

U

; the middle column fixes

U

and varies

C

; the right column fixes

U

and varies repetition

R=C/\min(C,U)

. Points show mean downstream accuracy for random-subset and Full-data runs, with lines connecting observations. Colors indicate compute in the left column and unique support in the middle and right columns, which share a color key. For each pretraining group and downstream task,

k

is selected from the Full-data 50M reference runs and held fixed across evaluated models. Horizontal axes are logarithmic; image counts are in millions.

Figure 8: Scaling-law fits and extrapolations under the kNN evaluation protocol for all seven datasets. Columns compare the compute-only Chinchilla-style law, the data-constrained effective-data law, the prescriptive scaling law, and our density-interaction law. Rows correspond to Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M. Curves are fitted only in the shaded low-compute region (

C\leq 50

M) and evaluated on higher-compute runs (

C>50

M). Each panel shows the fitted equation and held-out MAE in accuracy units.

Law	Formula	Maxar fit	Maxar extrap.	Waymo fit	Waymo extrap.	ImageNet fit	ImageNet extrap.
Compute-only	$L-A/C^{\alpha}$	0.0068	0.0089	0.0034	0.0018	0.0166	0.0249
Data-constrained	$L-A/U_{\mathrm{eff}}^{\beta}$	0.0119	0.0089	0.0034	0.0067	0.0140	0.0283
Density interaction	$L-A/C^{\alpha}-D\ell_{U}^{\beta}\ell_{R}^{\gamma}$	0.0053	0.0070	0.0029	0.0037	0.0114	0.0112
Unique-only density	$L-A/C^{\alpha}-D_{U}\ell_{U}^{\beta}$	0.0053	0.0070	0.0030	0.0018	0.0142	0.0190
Repeat-only density	$L-A/C^{\alpha}-D_{R}\ell_{R}^{\gamma}$	0.0068	0.0089	0.0029	0.0036	0.0122	0.0145
Separate density	$L-A/C^{\alpha}-D_{U}\ell_{U}^{\beta}-D_{R}\ell_{R}^{\gamma}$	0.0053	0.0070	0.0028	0.0062	0.0119	0.0134

Appendix

Table 6: Logistic-regression scaling-law comparison. Each law is fitted on runs with

C\leq 50

M and evaluated on runs with

C>50

M. Lower MAE is better. We use

\ell_{U}=\log(1+U)

and

\ell_{R}=\log(1+R)

Figure 9: kNN ablation plots for the density-interaction law. Rows compare the proposed interaction law against unique-only, repeat-only, separate-additive, raw-

U

, and raw-

R

variants.

Figure 10: Logistic-regression ablation plots for the density-interaction law.

Eval.	Dataset	$L$	$A$	$\alpha$	$D$	$\beta$	$\gamma$
kNN	Maxar	0.9169	5.2508	2.0030	0.002364	1.5387	0.3352
kNN	Waymo	0.7760	0.2263	0.8500	0.000001	-0.3792	5.0000
kNN	ImageNet	1.0052	1.0393	0.3219	0.000046	0.5322	5.0000
Logres	Maxar	0.9550	0.5118	0.6884	0.000984	2.2653	0.0000
Logres	Waymo	0.7829	0.7126	1.3312	0.000004	0.0899	5.0000
Logres	ImageNet	1.0500	0.9501	0.3084	0.000034	0.5158	5.0000

Appendix

Table 7: Fitted coefficients for the density-interaction law,

\mathrm{Perf}=L-A/C^{\alpha}-D[\log(1+U)]^{\beta}[\log(1+R)]^{\gamma}

. Coefficients are fitted independently per dataset and evaluation protocol.

Figure 11: Maxar matched-pair comparison between hierarchical-clustering selection and random selection. Each line fixes the subset size

U

and plots the downstream-performance difference, hierarchical minus random, as compute increases.

AI Scaling Laws

AI scaling laws describe how model performance changes with training compute, data, model size, architecture, and inference-time resources.

See all 27 papers

Learnability-Guided Diffusion for Dataset Distillation
Apr 1, 2026Jeffrey A. Chan-Santiago, Mubarak ShahDataset Distillation Diffusion-Based Dataset Distillation
Institute of Artificial Intelligence, University of Central Florida
Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases
May 19, 2026Jingwen Liu, Ezra Edelman, Surbhi Goel +1Deep Learning Optimization Statistical Learning Theory
Columbia University · University of Pennsylvania · Kempner Institute, Harvard University
Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching
Feb 5, 2026Qianxin Xia, Jiawei Du, Yuhan Zhang +6Flow Matching Dataset Distillation
University of Electronic Science and Technology of China, Chengdu, China · CFAR, Singapore · SWJTU, Chengdu, China

Apr 1, 2026J/K move · Enter open · S save

Learnability-Guided Diffusion for Dataset Distillation

Jeffrey A. Chan-Santiago, Mubarak Shah

Institute of Artificial Intelligence, University of Central Florida

Dataset Distillation Diffusion-Based Dataset Distillation

Open paper AskPDF arXiv

Training machine learning models on massive datasets is expensive and time-consuming. Dataset distillation addresses this by creating a small synthetic dataset that achieves the same performance as the full dataset. Recent methods use diffusion models to generate distilled data, either by promoting diversity or matching training gradients. However, existing approaches produce redundant training signals, where samples convey overlapping information. Empirically, disjoint subsets of distilled datasets capture 80-90% overlapping signals. This redundancy stems from optimizing visual diversity or average training dynamics without accounting for similarity across samples, leading to datasets where multiple samples share similar information rather than complementary knowledge. We propose learnability-driven dataset distillation, which constructs synthetic datasets incrementally through successive stages. Starting from a small set, we train a model and generate new samples guided by learnability scores that identify what the current model can learn from, creating an adaptive curriculum. We introduce Learnability-Guided Diffusion (LGD), which balances training utility for the current model with validity under a reference model to generate curriculum-aligned samples. Our approach reduces redundancy by 39.1%, promotes specialization across training stages, and achieves state-of-the-art results on ImageNet-1K (60.1%), ImageNette (87.2%), and ImageWoof (72.9%). Our code is available on our project page https://jachansantiago.github.io/learnability-guided-distillation/.

Abstract

AI Scaling Laws

Explore similar work

Learnability-Guided Diffusion for Dataset Distillation

Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases

Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching