The Composition Gap in Dataset Distillation

Guang Li; Takahiro Ogawa; Miki Haseyama

cs.LGSep 28, 2026

The Composition Gap in Dataset Distillation

Authors: Guang Li, Takahiro Ogawa, Miki Haseyama

Organizations: Hokkaido University

Abstract

Dataset distillation compresses a training set into a small synthetic set, usually evaluated one at a time. In federated and data-governance settings, several parties distill their own data and a user trains on their union. We ask whether the union of separately distilled sets reproduces training on the union of the real data composability and show that it can fail even when every source is distilled exactly and the total budget admits an exact joint distillate. Compressing a training trajectory into fewer steps transforms the source statistics nonlinearly, so averaging compressed sources differs from compressing their average. For quadratic objectives we derive the exact composition error for two-to-one step compression in terms of the source-Hessian variance and the linear terms of the losses, and on a smooth network at small step sizes this prediction captures the local endpoint discrepancy in magnitude and direction. For learned synthetic sets, however, the composed error decomposes exactly into this local discrepancy and an aggregate source residual. Under endpoint matching the residual exceeds the structural term by more than an order of magnitude, and under distribution matching the two terms partly cancel. Joint distillation also retains an accuracy advantage when both sets are distilled from the same dataset, where the local discrepancy is exactly zero. Training fidelity and downstream accuracy are therefore distinct requirements, neither established by evaluating each set on its own.

Figures & tables

Panel of Figure 1 — Figure 1: Compression and mixing do not commute. Distilling each source and then mixing the distillates (orange) applies the finite-step compression map $f_{p}$ to each source Hessian before averaging. Distilling the union (blue) averages first and compresses once. The two paths can yield different training maps, and their difference is the composition discrepancy. The diagram is a schematic of exact quadratic compression at a general ratio $p$ for sources with a shared optimum; the local endpoint discrepancy $\Delta\Phi$ measured in Section 6 is its two-step instance on real data.

Figure 2: Exact constituents compose with an error set by curvature disagreement (Theorem 4.2 , quadratic models with a shared optimum at zero). (a) Two sources with different curvature ellipses. The real two-step and joint one-step endpoints coincide (blue), and the composed endpoint is displaced by

\eta^{2}V_{H}\theta_{0}

(orange). (b) Endpoint discrepancies of gradient-descent runs on random source pairs against the closed form. (c) Identical spectra with rotated eigenbases still yield a nonzero discrepancy, since

V_{H}

includes eigenvector alignment.

Figure 3: The observed error of a union of distillates is the net of the local discrepancy and the source residual. Exact decomposition

d_{\rm ind}=\Delta\Phi+r_{\rm src}

on the distribution-matching sets of Section 6 , with the step calibration described there (CIFAR-10, SVHN and CIFAR-100; level means over partitions, Appendix Table 8 ). (a) RMS norms of the local discrepancy

\Delta\Phi

, the aggregate source residual

r_{\rm src}

and the observed error

d_{\rm ind}

, with the quadratic prediction

\eta^{2}r_{\rm comp}

marked; the local discrepancy is exactly zero in the no-split control and

0.013

–

0.018\times 10^{-3}

at IID, below the resolution of the axis. The residual is the larger term at every level, and

\|d_{\rm ind}\|<\|r_{\rm src}\|

under skew. (b) Cosines at the skewed levels: the two terms are anti-aligned, the prediction is partly aligned with

\Delta\Phi

at this step size (

\eta L\approx 1

) and nearly orthogonal to

d_{\rm ind}

Figure 4: The excess endpoint error grows with label skew, whereas the accuracy gain of joint distillation persists in the no-split control. Independent versus joint distillation on CIFAR-10 with distribution matching at ten images per class per source. Each skew level has five partitions (circles), and squares with bars give their mean and SD. Within a partition, endpoint errors (after the per-set step calibration of Section 6 ) are RMS over three initializations and accuracies average three evaluation seeds. (a) RMS endpoint error of the separately distilled union and of the joint set against the real-union two-step map. (b) Their difference, the excess endpoint error. (c) Accuracy gain of joint over separate-then-union distillation. The no-split control (red) has three replicates of the full dataset, and its bar is the SD over replicates.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Maps
Symbol	Meaning
$\Phi_{D}^{K,\eta}$	$K$ -step full-batch GD map at step size $\eta$ on dataset $D$
$\Phi_{\rm ind},\ \Phi_{R}$	composed synthetic one-step map; real-union two-step map
$f_{p}(H)$	time-compression transform $(I-(I-\eta H)^{p})/\eta$ , $p=T/K$
Errors
$\mathcal{E},\ \mathcal{C}=\sqrt{\mathcal{E}}$	squared endpoint error over $\theta_{0}\sim P_{0}$ ; its RMS form

Appendix

Table 1: Notation.

Figure 5: Concavity of the finite-step clock.

g_{p}

is concave in the source curvature, so the average of two compressed sources (compress, then mix) lies below the compression of their average (mix, then compress); under gradient flow (dashed) the two coincide.

Figure 6: Scalar reweighting repairs one curvature mode, not two. Composition gap (normalized) as a function of the synthetic weight

\beta

: for a single mode there is a

\beta^{*}\neq\alpha

at which the gap is zero; for two modes with different

\beta^{*}_{j}

the minimum over

\beta

is a residual floor.

$n_{\rm syn}$	rank of Gram	rel. Frobenius error	endpoint loss	measured/closed-form defect	log-log corr.
4	4	48.0%	$4.8\times 10^{-1}$	11.7 $\pm$ 1.4	0.12
6	6	31.7%	$2.0\times 10^{-1}$	7.6 $\pm$ 0.9	0.16
8	8	19.9%	$8.9\times 10^{-2}$	4.8 $\pm$ 0.6	0.16
12	12	$<10^{-6}$	$1.3\times 10^{-31}$	1.000 $\pm$ 0.000	1.00
24	12	$<10^{-6}$	$1.6\times 10^{-31}$	1.000 $\pm$ 0.000	1.00

Appendix

Table 2: E5: endpoint matching recovers the transformed statistic once the budget reaches the dimension.

Figure 7: Where the arbitrary-ratio lower bound is certified. The zero-crossing of Theorem E.2 ’s sufficient-separation bracket as a function of effective defect rank, for several compression ratios; the bound is certified below each curve, and larger

p

, step size, or rank can make it uninformative even though the exact composition gap remains nonnegative. The dotted line marks

\eta L\approx 0.12

, the value of the ReLU 2,000-example subset diagnostic at

\eta_{\rm ref}

(the calibration cells of Figure 9 lie at

\eta L=0.04

–

0.11

); the independent-versus-joint comparison operates at

\eta L\approx 1

, outside the range, where only the exact

p=2

identity is invoked.

Figure 8: The detectability window (schematic). The structural term grows as

\eta^{2}

, constituent error as

\eta

, nonlinear error as

\eta^{3}

; the structural term is certifiably visible only where it exceeds their sum, the shaded interval of Corollary G.2 .

Component	Signature	Diagnostic	Minimum repair required
Global clock	common ratio	endpoint speed	scalar retiming
Source clock	coefficients $c_{i}$	source HVP scale	per-source reweighting
Curvature variance	$V_{H}\neq 0$	$P_{\rm comp}$	mode-dependent if the required retimings differ ( H.1 )
Affine conflict	$V_{H}\mu_{0}-w\neq 0$	mean shift	scalar retiming only if it matches the linear part and the affine offset simultaneously (the scalar example of Section 4 is repaired by the factor $3.6/3.5$ ); otherwise mode-dependent
Missing subspace	rank deficit	persistent residual	not repairable by retiming

Appendix

Table 3: Theory-derived repairability taxonomy.

Predictor	Spearman $\rho$	partial $\rho$ (adj. label div.)	partial $\rho$ (adj. grad. dis.)
$P_{\rm comp}$	0.991 [0.960, 0.999]	0.869 [0.568, 0.989]	0.701 [0.210, 0.971]
$P_{\rm iso}$	0.955 [0.857, 0.984]	0.316 [ $-$ 0.316, 0.785]	$-$ 0.001 [ $-$ 0.262, 0.496]
Label divergence	0.964 [0.907, 0.985]	N/A	N/A
Gradient disagreement	0.983 [0.937, 0.995]	N/A	N/A

Appendix

Table 4: E1: which statistic predicts the observed discrepancy.

Figure 9: Local-law validation with all activations and both levels. (a) Squared discrepancy

\|\Delta\Phi(\theta_{0})\|^{2}

against

\eta L

at Dir-0.1, with exponents fitted inside the shaded window

\eta L<1

and dotted lines at

\eta_{\rm ref}

and

\eta L=1

. (b) Calibration ratio

\|\Delta\Phi(\theta_{0})\|/(\eta^{2}\|r_{\rm comp}(\theta_{0})\|)

against

\eta L

(solid: Dir-0.1, dashed: IID, bands over partitions), with the small-step regime enlarged in the inset. At IID the ReLU structural term is at numerical zero, so its ratio is not meaningful.

Figure 10: Dense E1 sweep on the ReLU network. (a) Partition-level mean RMS discrepancy against

\eta L

at nine step sizes (five partitions per level with three initializations each, error bars are SDs over partitions); filled markers are the eight rates inside the strict window and lines are the fitted power laws. (b) Fitted exponents of

\mathcal{C}_{\rm loc}^{2}

\eta

with 95% intervals from a bootstrap over the five partitions; both skewed levels lie 3–6% below four, and the IID discrepancy is not governed by the structural term.

Bank	Iter.	Bank loss	Held-out	$\mathcal{U}_{\rm pw}$	$\mathcal{C}_{\rm ind}$	$\mathcal{C}_{\rm joint}$	$\widehat{\Gamma}_{\rm RMS}$	$\mathcal{U}_{\rm pw}/\mathcal{C}_{\rm struct}$	ineq.
64	1,100	0.116	$0.380\pm 0.008$	3.81	1.25	0.99	0.26	38.5	0/96
64	3,000	0.114	$0.379\pm 0.008$	3.79	1.25	0.99	0.26	38.4	0/96
512	1,100	0.132	$0.372\pm 0.008$	3.73	1.20	0.95	0.24	37.7	0/96
512	3,000	0.130	$0.370\pm 0.008$	3.71	1.20	0.95	0.25	37.5	0/96

Appendix

Table 5: E10: attribution of the source-error plateau. Dir-0.1, ten images per class per source,

(2,1)

\eta_{\rm ref}

, SiLU, three partitions, all conditions scored on the same 32 held-out initializations. Bank loss is the relative matching loss of the source sets on their training bank, held-out is the relative source error on the evaluation initializations (mean and SD over partitions), errors are in units of

10^{-3}

, and

\widehat{\Gamma}_{\rm RMS}=\mathcal{C}_{\rm ind}-\mathcal{C}_{\rm joint}

is computed from the unrounded RMS values. These runs were archived with

\mathcal{U}_{\rm pw}

, the RMS over initializations of

\sum_{i}\alpha_{i}\|e_{i}(\theta_{0})\|

, in place of

\mathcal{U}_{\rm src}

of Table 6 ; on the cell common to both tables the two differ by

0.2\%

(

3.808

against

3.814

). Ineq. counts initializations as in Table 6 .

Figure 11: Endpoint-matching distillates on the SiLU network (E10; Dir-0.1 partition 0 of the E1 subset, source 0, which holds the seven classes shown). Images are initialized from real images of the class and optimized for $1{,}100$ Adam iterations on the one-step-versus-two-step endpoint discrepancy; the columns show the initialization and the learned sets at one, five and ten images per class (up to five per class displayed). The learned images move far from their initialization (median relative displacement $1.1$ ), since the objective constrains the induced training step and not the appearance.

Figure 12: At small step sizes the local prediction is accurate, but the endpoint error of learned synthetic sets lies far above the local discrepancy. Endpoint-matching distillates on the SiLU network at Dir-0.1 (three partitions, RMS over

32

held-out initializations, mean and SD over partitions). (a) Weighted source error, error of the union of the learned sets and error of the jointly learned set against the image budget at

\eta_{\rm ref}

; the local reference

\mathcal{C}_{\rm loc}

and the structural composition error

\mathcal{C}_{\rm struct}

closely agree and lie far below. (b) The same quantities against the step size at five images per class, with the cosine between

\Delta\Phi

and

\eta^{2}r_{\rm comp}

at each step size.

Level	Setting	IPC	$\mathcal{U}_{\rm src}$	rel.	$\mathcal{C}_{\rm ind}$	$\mathcal{C}_{\rm joint}$	$\mathcal{C}_{\rm loc}$	$\mathcal{C}_{\rm struct}$	$\cos(\Delta\Phi,\hat{s})$	$\cos(d_{\rm ind},\hat{s})$	ineq.
IID	$(2,1)$ , $\eta_{\rm ref}$	1	2.03	0.51	1.84	1.51	0.00009	0.00009	1.000	0.05	0/96
IID	$(2,1)$ , $\eta_{\rm ref}$	5	1.10	0.27	1.04	1.01	0.00009	0.00009	1.000	0.04	0/96
IID	$(2,1)$ , $\eta_{\rm ref}$	10	1.02	0.25	0.99	0.99	0.00009	0.00009	1.000	0.04	0/96
IID	$(2,1)$ , $4\eta_{\rm ref}$	5	4.11	0.26	3.86	3.71	0.0014	0.0014	0.997	0.03	0/96
IID	$(2,1)$ , $16\eta_{\rm ref}$	5	12.9	0.23	11.6	10.6	0.021	0.022	0.955	$-0.04$	0/96
IID	$(4,2)$ , $\eta_{\rm ref}$	5	2.16	0.27	2.04	1.97	0.0005	0.0002	0.999	0.04	0/96

Appendix

Table 6: E10: endpoint-matching distillates on the network (SiLU unless stated). Per cell, RMS over 32 held-out initializations; mean over three partitions. Errors in units of

10^{-3}

\mathcal{U}_{\rm src}=\sum_{i}\alpha_{i}\|\Phi^{K,\eta}_{S_{i}}-\Phi^{T,\eta}_{D_{i}}\|_{L^{2}(P_{0})}

is the weighted source error of Section 6 at the row’s

(T,K)

, rel. is the weighted mean over sources of each set’s endpoint error divided by its own real

T

-step displacement,

\hat{s}=K\binom{p}{2}\eta^{2}r_{\rm comp}

is the structural prediction (coefficient

1

for

(2,1)

and

2

for

(4,2)

, Theorem E.2 ), and

d_{\rm ind}

is the per-initialization error of the union of the learned sets against the real-union endpoint. The last column (ineq.) is evaluated before aggregation: it counts initializations

\theta_{0}

at which

\|\hat{s}(\theta_{0})\|>\sum_{i}\alpha_{i}\|e_{i}(\theta_{0})\|

, with

e_{i}(\theta_{0})=\Phi^{K,\eta}_{S_{i}}(\theta_{0})-\Phi^{T,\eta}_{D_{i}}(\theta_{0})

Heterogeneity sweep (five partitions per level)
	Endpoint errors				Test accuracy
Source split	reference $\mathcal{C}_{\rm loc}$	separate $\mathcal{C}_{\rm ind}$	joint $\mathcal{C}_{\rm joint}$	excess $\widehat{\Gamma}_{\rm RMS}$	separate	joint	gain
IID	$0.016\pm 0$ .001)	$7.41\pm 0$ .04)	$7.34\pm 0$ .08)	$0.06\pm 0$ .10)	$46.3\pm 0$ .9)	$51.6\pm 0$ .6)	$+5.3\pm 0$ .9)
Dir-1.0	$4.85\pm 0$ .99)	$8.72\pm 0$ .29)	$7.34\pm 0$ .08)	$1.38\pm 0$ .25)	$46.2\pm 0$ .6)	$51.9\pm 0$ .5)	$+5.7\pm 0$ .6)
Dir-0.5	$5.58\pm 0$ .76)	$8.71\pm 0$ .31)	$7.33\pm 0$ .06)	$1.38\pm 0$ .27)	$45.8\pm 1$ .1)	$51.7\pm 0$ .5)	$+5.9\pm 1$ .2)
Dir-0.1	$8.09\pm 1$ .17)	$9.85\pm 1$ .33)	$7.50\pm 0$ .26)	$2.35\pm 1$ .44)	$45.8\pm 0$ .7)	$51.1\pm 1$ .1)	$+5.3\pm 0$ .7)

Appendix

Table 7: Separate-then-union versus joint distillation on CIFAR-10 (distribution matching, official schedule, 10 images per class per source). Operator quantities are RMS endpoint distances to the real-union two-step map, in units of

10^{-3}

. Accuracies in percent.

Figure 13: Independent versus joint distillation: excess endpoint error versus the local reference

\mathcal{C}_{\rm loc}

on CIFAR-10. Each point represents one partition (five per level, three seeds each); the line is a descriptive pooled no-intercept fit across the skewed levels (slope 0.28). Level means appear in Table 7 .

Figure 14: Synthetic images on one Dir-0.1 partition of CIFAR-10 (partition 0, DM at the official schedule; up to five images per class per set are shown). Left: the union of the two independently distilled sets (E2), with the frame colour marking the source of each image (blue, source 1, which holds five classes; orange, source 2, which holds all ten); each source stores ten images per class it holds, so the union has twenty images in the shared classes and ten in the others. Middle: the jointly distilled set at the same total budget of $150$ images, $15$ per class. Right: the class-allocation-matched joint set of E11 (Appendix J.8 ), which places $20$ images in the five shared classes and $10$ in the five classes held by one source, the per-class counts of the union.

Dataset	Level	$\\|d_{\rm ind}\\|$	$\\|d_{\rm joint}\\|$	$\\|\Delta\Phi\\|$	$\\|r_{\rm src}\\|$	$\\|\hat{s}\\|$	$\cos(\Delta\Phi,r_{\rm src})$	$\cos(\Delta\Phi,\hat{s})$	$\cos(d_{\rm ind},\hat{s})$
CIFAR-10	IID	7.41	7.34	0.016	7.41	0.002	$-0.01$	0.10	$-0.04$
CIFAR-10	Dir-1.0	8.72	7.34	4.85	9.15	4.84	$-0.35$	0.41	0.06
CIFAR-10	Dir-0.5	8.71	7.33	5.58	9.84	7.26	$-0.48$	0.51	0.01
CIFAR-10	Dir-0.1	9.85	7.50	8.09	10.4	12.3	$-0.46$	0.60	0.04
CIFAR-10	no split	7.43	7.34	0	7.43	0	–	–	–
SVHN	IID	6.08	6.13	0.013	6.08	0.002	$-0.02$	0.08	0.09

Appendix

Table 8: E9: exact decomposition of the observed discrepancy on the E2 sets. Norms in units of

10^{-3}

, RMS over seeds within partition and mean over partitions; cosines are means over seeds and partitions.

\hat{s}=\eta^{2}r_{\rm comp}

Dataset	Level	$\\|d_{\rm ind}\\|$	$\\|d_{\rm joint}\\|$	$\\|r_{\rm src}\\|$	$\cos(\Delta\Phi,r_{\rm src})$	excess
CIFAR-10	IID	7.41 / 13.4	7.34 / 13.4	7.41 / 13.4	$-0.01$ / $+0.02$	$+0.07\pm 0.09$ / $-0.00\pm 0.09$
CIFAR-10	Dir-1.0	8.72 / 13.4	7.34 / 13.4	9.15 / 13.6	$-0.35$ / $-0.20$	$+1.42\pm 0.25$ / $+0.08\pm 0.08$
CIFAR-10	Dir-0.5	8.71 / 13.5	7.33 / 13.3	9.84 / 13.9	$-0.48$ / $-0.27$	$+1.42\pm 0.27$ / $+0.17\pm 0.11$
CIFAR-10	Dir-0.1	9.85 / 13.7	7.50 / 13.4	10.4 / 14.4	$-0.46$ / $-0.36$	$+2.38\pm 1.45$ / $+0.28\pm 0.08$
SVHN	Dir-0.1	7.65 / 11.0	6.18 / 10.7	12.1 / 12.8	$-0.78$ / $-0.56$	$+1.47\pm 0.95$ / $+0.37\pm 0.16$
CIFAR-100	Dir-0.1	2.51 / 7.83	1.96 / 7.82	2.87 / 7.85	$-0.48$ / $-0.11$	$+0.56\pm 0.03$ / $+0.01\pm 0.03$

Appendix

Table 9: E9 without the per-set scalar calibration. Each entry gives the calibrated value of Table 8 followed by the uncalibrated value (class-mass weights,

c_{i}=1

). Norms in units of

10^{-3}

; the excess is

\widehat{\Gamma}_{\rm mean}

, the signed per-initialization difference

\|d_{\rm ind}\|-\|d_{\rm joint}\|

averaged within partition, mean

\pm

SD over partitions, which differs from

\widehat{\Gamma}_{\rm RMS}=\mathcal{C}_{\rm ind}-\mathcal{C}_{\rm joint}

of Table 7 by a few percent.

Dataset	Level	Weighting	$\mathcal{C}_{\rm ind}$	$\mathcal{C}_{\rm joint}$	removed (ind / joint)	excess
CIFAR-10	IID	paper	7.41 $\pm$ 0.04	7.34 $\pm$ 0.08	–	$+$ 0.06 $\pm$ 0.10
		bank	7.82 $\pm$ 0.05	7.83 $\pm$ 0.08	$-$ 0.11 / $-$ 0.14	$-$ 0.01 $\pm$ 0.12
		oracle	5.83 $\pm$ 0.05	5.76 $\pm$ 0.04	0.38 / 0.38	$+$ 0.07 $\pm$ 0.04
CIFAR-10	Dir-1.0	paper	8.72 $\pm$ 0.29	7.34 $\pm$ 0.08	–	$+$ 1.38 $\pm$ 0.25
		bank	7.93 $\pm$ 0.08	7.83 $\pm$ 0.08	0.17 / $-$ 0.14	$+$ 0.10 $\pm$ 0.08
		oracle	6.00 $\pm$ 0.10	5.76 $\pm$ 0.04	0.53 / 0.38	$+$ 0.24 $\pm$ 0.10

Appendix

Table 10: E12: endpoint error of the saved DM sets under three class weightings. RMS over the three evaluation initializations within partition, mean

\pm

SD over five partitions, in units of

10^{-3}

. Removed is the fraction of squared error removed relative to the paper’s weighting; excess is

\mathcal{C}_{\rm ind}-\mathcal{C}_{\rm joint}

under the same weighting.

Set	Iterations	Accuracy	vs. union	effective rank
union of the independent sets	0	46.44 $\pm$ 0.50	–	8.91
warm start from the union	500	46.16 $\pm$ 0.51	$-$ 0.28 $\pm$ 0.43	9.00
warm start from the union	2,000	46.88 $\pm$ 0.67	$+$ 0.44 $\pm$ 0.47	9.15
cold start from real images	2,000	48.27 $\pm$ 0.54	$+$ 1.83 $\pm$ 0.65	9.66
joint set (official schedule)	20,000	52.59 $\pm$ 0.36	$+$ 6.15 $\pm$ 0.66	10.29

Appendix

Table 11: E13: joint distillation warm-started from the union of the no-split sets (CIFAR-10, DM, 200 images). Accuracy in percent, mean

\pm

SD over three replicates; paired differences are within replicate; effective rank is the per-class participation ratio of the frozen embedding.

Partition (images)	$\mathcal{C}_{\rm ind}$	$\mathcal{C}_{\rm joint}$	$\widehat{\Gamma}_{\rm RMS}$	$\mathcal{C}_{\rm joint}$	$\widehat{\Gamma}_{\rm RMS}$	Acc. indep.	Acc. joint	Gain	Acc. joint	Gain
		Matched joint		Canonical joint			Matched		Canonical
0 (150)	9.48	7.92	1.55	7.95	1.53	$45.4\pm 0.7$	$46.4\pm 0.6$	$+1.0$	$49.9\pm 0.5$	$+4.5$
1 (170)	8.88	7.82	1.06	7.35	1.53	$46.1\pm 1.0$	$49.5\pm 0.5$	$+3.4$	$51.4\pm 0.4$	$+5.4$
2 (200)	12.18	7.29	4.90	7.29	4.90	$46.5\pm 0.8$	$52.5\pm 0.8$	$+6.0$	$52.8\pm 0.7$	$+6.3$

Appendix

Table 12: E11: class-allocation-matched joint set at Dir-0.1 on CIFAR-10. The image count is the budget of both joint sets on that partition. Endpoint quantities are RMS over three initialization seeds in units of

10^{-3}

, accuracy is the mean and SD over five evaluation seeds in percent, and the gain is joint minus independent.

Figure 15: Composed error follows constituent error (DM, Dirichlet-0.1, partition p0). (a) Relative constituent error at the snapshot checkpoints {0,1000,2500,5000,10000,15000,20000} of the official schedule; it plateaus after 5,000 iterations. (b) The composed error

\mathcal{C}_{\rm ind}

tracks it with unit elasticity and settles above the local reference

\mathcal{C}_{\rm loc}

(dashed), at

1.15\times

for this partition and about

1.2\times

over the five partitions of the official sweep.

Figure 16: Image budget at Dir-0.1 (the same three partitions at every budget). (a) Excess endpoint error

\widehat{\Gamma}_{\rm mean}

of the separately distilled union, aggregated as the mean over seeds of the per-seed endpoint-distance difference as in Table 13 , with the local reference

\mathcal{C}_{\rm loc}

, which the partitions fix. (b) Accuracy gain of joint over separate-then-union distillation.

IPC	$\epsilon_{\rm rel}$	$\mathcal{C}_{\rm ind}$	$\mathcal{C}_{\rm ind}/\mathcal{C}_{\rm loc}$	$\widehat{\Gamma}_{\rm mean}$	acc. diff. (pp)	elasticity to next
1	0.416 $\pm$ 0.023	21.3	2.60 $\pm$ 0.45	4.47 $\pm$ 0.07	$+$ 0.66 $\pm$ 1.12	0.83
5	0.206 $\pm$ 0.021	11.9	1.44 $\pm$ 0.10	2.71 $\pm$ 1.70	$+$ 5.25 $\pm$ 1.50	0.88
10	0.172 $\pm$ 0.019	10.2	1.22 $\pm$ 0.03	2.68 $\pm$ 2.00	$+$ 5.47 $\pm$ 0.90	1.09
20	0.157 $\pm$ 0.019	9.20	1.09 $\pm$ 0.06	2.61 $\pm$ 2.10	$+$ 2.06 $\pm$ 0.14	0.95
50	0.147 $\pm$ 0.017	8.63	1.02 $\pm$ 0.08	2.62 $\pm$ 2.30	$+$ 1.68 $\pm$ 0.27	N/A

Appendix

Table 13: Image budget at Dir-0.1 (DM, official schedule; partitions p0–p2

\times

three initialization seeds at every budget).

Dataset	Level	$n$	$\mathcal{C}_{\rm loc}$	$\widehat{\Gamma}_{\rm mean}$	accuracy gain (pp)
CIFAR-10	IID	5	0.016	0.08 $\pm$ 0.06	$+$ 5.28 $\pm$ 0.92
CIFAR-10	Dir-0.1	5	7.72	2.33 $\pm$ 1.40	$+$ 5.35 $\pm$ 0.75
SVHN	IID	5	0.013	$-$ 0.04 $\pm$ 0.04	$+$ 3.24 $\pm$ 1.10
SVHN	Dir-0.1	5	10.5	1.47 $\pm$ 0.95	$+$ 5.48 $\pm$ 1.45
CIFAR-10	no split	3	0 (exact)	0.08 $\pm$ 0.21	$+$ 6.55 $\pm$ 0.42
SVHN	no split	3	0 (exact)	0.09 $\pm$ 0.16	$+$ 2.16 $\pm$ 1.27

Appendix

Table 14: Cross-dataset comparison (DM, official schedule, IPC 10 per source; five partitions per split level on CIFAR-10 and SVHN, three on CIFAR-100, three replicates for the no-split controls).

			ConvNet		ResNet-18
Dataset	Level	$n$	joint	gain	union	joint	gain
CIFAR-10	IID	5	51.6 $\pm$ 0.6	$+$ 5.28 $\pm$ 0.92	40.3 $\pm$ 0.6	42.1 $\pm$ 0.5	$+$ 1.81 $\pm$ 0.85
CIFAR-10	Dir-1.0	5	51.9 $\pm$ 0.5	$+$ 5.72 $\pm$ 0.64	40.2 $\pm$ 0.6	42.3 $\pm$ 0.6	$+$ 2.05 $\pm$ 0.77
CIFAR-10	Dir-0.5	5	51.7 $\pm$ 0.5	$+$ 5.86 $\pm$ 1.24	39.2 $\pm$ 1.1	42.6 $\pm$ 0.4	$+$ 3.31 $\pm$ 0.84
CIFAR-10	Dir-0.1	5	51.1 $\pm$ 1.1	$+$ 5.35 $\pm$ 0.75	37.8 $\pm$ 1.1	41.2 $\pm$ 0.8	$+$ 3.44 $\pm$ 0.95
CIFAR-10	no split	3	52.6 $\pm$ 0.4	$+$ 6.55 $\pm$ 0.42	41.1 $\pm$ 0.8	42.2 $\pm$ 0.1	$+$ 1.13 $\pm$ 0.79

Appendix

Table 15: E14: accuracy of the same synthetic sets on the ConvNet and on ResNet-18 (DM, official evaluation protocol, three evaluation seeds). Percent, mean

\pm

SD over partitions; gain is joint minus union within partition.

Level	$n$	eff. rank, separate (unioned)	eff. rank, joint	cross/within
IID	2	8.91 $\pm$ 0.11	10.27 $\pm$ 0.02	0.942 $\pm$ 0.000
Dirichlet-0.1	5	7.39 $\pm$ 0.42	9.79 $\pm$ 0.40	1.288 $\pm$ 0.097
No split (E7)	3	8.91 $\pm$ 0.05	10.29 $\pm$ 0.11	0.941 $\pm$ 0.001

Appendix

Table 16: Effective rank per class and cross/within distance ratio in a frozen feature space. Mean

\pm

SD over partitions (

n

as listed; two IID partitions, five Dir-0.1 partitions, three no-split replicates); image budgets matched within partition. Dir-1.0 and Dir-0.5 were not measured.

Local-law network at $\eta_{\rm ref}=7.8\times 10^{-4}$
Network	Level	$\eta L$	$r_{H}$	$\delta_{H}$
ReLU, 2,000-sample subset	IID / Dir-1.0 / Dir-0.1	0.112 / 0.116 / 0.126	—	0.29 $\pm$ 0.08
ReLU, $10^{4}$ subset	IID / Dir-0.1	0.096 / 0.108	—	—
SiLU, $10^{4}$ subset	IID / Dir-0.1	0.047 / 0.066	—	0.14 $\pm$ 0.02
tanh, $10^{4}$ subset	IID / Dir-0.1	0.039 / 0.060	—	0.20 $\pm$ 0.02
Distillation network (ReLU) at $\eta=6.25\times 10^{-3}$

Appendix

Table 17: Assumption diagnostics.

Shift	label TV	$n$	$P_{\rm comp}$	$\sqrt{P_{\rm comp}}$	$\mathcal{C}_{\rm loc}$	$\mathcal{C}_{\rm loc}/\sqrt{P_{\rm comp}}$	$\widehat{\Gamma}_{\rm mean}$ †
IID	0.000	5	$4.13\times 10^{-12}$	0.002	0.016	(8.1)	$-$ 0.377
Feature skew f	0.000	8	$3.95\times 10^{-7}$	0.63	0.652	(1.04)	$-$ 0.478
Dirichlet-1.0	0.227	5	$2.30\times 10^{-5}$	4.61	4.54	1.01	$+$ 0.928
Dirichlet-0.5	0.300	5	$5.25\times 10^{-5}$	7.12	5.48	0.78	not run
Dirichlet-0.1	0.392	5	$1.42\times 10^{-4}$	11.7	7.72	0.67	$+$ 1.96

Appendix

Table 18: Type of heterogeneity changes the empirical outcome.

Figure 17: Better constituents do not always yield a better union. Ratio of the error with stored semantic mass to the error with uniform weights, for the individual sources and for their composed union (percentages: change relative to uniform weights). (a) At Dir-1.0 the individual error falls by 60–71% while the composed error is unchanged. (b) At Dir-0.1 the improvement partly survives union.

	$m=2$	$m=5$	ratio
Real partition and network only
$\mathcal{C}_{\rm loc}$ ( $\times 10^{-3}$ )	8.09 $\pm$ 1.17	14.1 $\pm$ 2.3	1.75
$\sqrt{P_{\rm comp}}$ ( $\times 10^{-3}$ )	12.3 $\pm$ 2.1	25.8 $\pm$ 2.2	2.09
Distilled sets (IPC 10 per source; total budget grows with $m$ )
$\widehat{\Gamma}_{\rm RMS}$ ( $\times 10^{-3}$ )	2.35 $\pm$ 1.44	5.65 $\pm$ 1.56	2.41
accuracy gain (pp)	$+$ 5.35 $\pm$ 0.75	$+$ 7.34 $\pm$ 1.34	1.37

Appendix

Table 19: Two versus five sources at Dir-0.1 (DM, official schedule, IPC 10 per source). Mean

\pm

SD over

n=5

partitions; within a partition the endpoint quantities are RMS over the three initialization seeds 100–102 and the accuracy gain is their mean, the convention of Table 7 , whose Dir-0.1 row the

m=2

column reproduces. The first block depends only on the real partition and the network; the second describes the distilled sets, whose total budget is not matched between

m=2

and

m=5

, so

\mathcal{C}_{\rm ind}

and its ratio to

\mathcal{C}_{\rm loc}

confound source count with budget and are not interpreted.

Explore similar work

Mar 16, 2026cs.LG

Dataset Distillation Efficiently Encodes Low-Dimensional Representations from Gradient-Based Learning of Non-Linear Tasks

Dataset distillation, a training-aware data compression technique, has recently attracted increasing attention as an effective tool for mitigating costs of optimization and data storage. However, progress remains largely empirical. Mechanisms underlying the extraction of task-relevant information from the training process and the efficient encoding of such information into synthetic data points remain elusive. In this paper, we theoretically analyze practical algorithms of dataset distillation applied to the gradient-based training of two-layer neural networks with width

L

. By focusing on a non-linear task structure called multi-index model, we prove that the low-dimensional structure of the problem is efficiently encoded into the resulting distilled data. This dataset reproduces a model with high generalization ability for a required memory complexity of

\tildeΘ$$(r^2d+L)

, where

d

and

r

are the input and intrinsic dimensions of the task. To the best of our knowledge, this is one of the first theoretical works that include a specific task structure, leverage its intrinsic dimensionality to quantify the compression rate and study dataset distillation implemented solely via gradient-based algorithms.

Yuri Kinoshita, Naoki Nishikawa, Taro Toyoizumi

The University of Tokyo, Tokyo, Japan · Laboratory for Neural Computation and Adaptation, RIKEN Center for Brain Science, Wako, Japan · RIKEN Center for Advanced Intelligence Project, Nihonbashi, Japan

Diffusion-Based Dataset Distillation Dataset Distillation Task-Aware Compression Intrinsic Dimensionality Gradient-Based Optimization Two-Layer Neural Networks+1

Ask PDF

Apr 30, 2026cs.LG

Fair Dataset Distillation via Cross-Group Barycenter Alignment

Dataset Distillation aims to compress a large dataset into a small synthetic one while maintaining predictive performance. We show that as different demographic groups exhibit distinct predictive patterns, the distillation process struggles to simultaneously preserve informative signals for all subgroups, regardless of whether group sizes are mildly or severely imbalanced. Consequently, models trained on distilled data can experience substantial performance drops for certain subgroups, leading to fairness gaps. Crucially, these gaps do not disappear by merely correcting group imbalance, since they stem from fundamental mismatches in subgroup predictive patterns rather than from sample-size disparities alone. We therefore formally analyze the interaction between these two sources of bias and cast the solution as identifying a group-imbalance-agnostic barycenter of the predictive information that induces similar representations across all subgroups. By distilling toward this shared aggregate representation, we show that group fairness concerns can be reduced. Our approach is compatible with existing distillation methods, and empirical results show that it substantially reduces bias introduced by dataset distillation. Code is available at https://github.com/mhmoslemi/COBRA.

Mohammad Hossein Moslemi, Nima Hosseini Dashtbayaz, Zhimin Mei +2

Western University, London, Ontario · Vector Institute, Toronto, Ontario · Ivey Business School, London, Ontario +1

Diffusion-Based Dataset Distillation Dataset Distillation Algorithmic Fairness Disparities Class Imbalance

Ask PDF

Jun 19, 2026cs.CV

Structural Assessment for Understanding and Guiding Dataset Distillation in Discrete Token Space

Dataset distillation (DD) has proven to reduce training cost while preserving accuracy. While promising, the factors that make one distilled dataset more effective than another remain poorly understood. In this work, we investigate this question through the lens of discrete visual tokenizers. Whereas many prior DD efforts emphasize matching global data distributions, we suggest that the effectiveness depends on which semantic concepts are captured and how they are composed. Discrete visual tokenizers provide a finite vocabulary that enables direct statistical analysis of such compositional structure. Through quantitative analysis of token-level statistics, we introduce the structural score to measure the adequacy of token compositions. We observe that distilled datasets with balanced token composition yield higher validation performance. On the other hand, divergence from the original data does not necessarily harm performance. We further show that samples with high structural scores in the discrete token space can effectively guide diffusion-based DD. Our findings highlight the importance of token composition in dataset effectiveness, offering a principled complement to distributional similarity considerations in DD.

Yue Cao, Jianyang Gu, Vyacheslav Kungurtsev +4

The University of British Columbia · The Ohio State University · Czech Technical University in Prague +1

Diffusion-Based Dataset Distillation Dataset Distillation Visual Tokenizers Token-Level Uncertainty

Ask PDF