cs.LGSep 28, 2026

The Composition Gap in Dataset Distillation

Authors: Guang Li, Takahiro Ogawa, Miki Haseyama

Organizations: Hokkaido University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Dataset distillation compresses a training set into a small synthetic set, usually evaluated one at a time. In federated and data-governance settings, several parties distill their own data and a user trains on their union. We ask whether the union of separately distilled sets reproduces training on the union of the real data composability and show that it can fail even when every source is distilled exactly and the total budget admits an exact joint distillate. Compressing a training trajectory into fewer steps transforms the source statistics nonlinearly, so averaging compressed sources differs from compressing their average. For quadratic objectives we derive the exact composition error for two-to-one step compression in terms of the source-Hessian variance and the linear terms of the losses, and on a smooth network at small step sizes this prediction captures the local endpoint discrepancy in magnitude and direction. For learned synthetic sets, however, the composed error decomposes exactly into this local discrepancy and an aggregate source residual. Under endpoint matching the residual exceeds the structural term by more than an order of magnitude, and under distribution matching the two terms partly cancel. Joint distillation also retains an accuracy advantage when both sets are distilled from the same dataset, where the local discrepancy is exactly zero. Training fidelity and downstream accuracy are therefore distinct requirements, neither established by evaluating each set on its own.

Figures & tables

Figure 1: Compression and mixing do not commute. Distilling each source and then mixing the distillates (orange) applies the finite-step compression map fpf_{p} to each source Hessian before averaging. Distilling the union (blue) averages first and compresses once. The two paths can yield different training maps, and their difference is the composition discrepancy. The diagram is a schematic of exact quadratic compression at a general ratio pp for sources with a shared optimum; the local endpoint discrepancy ΔΦ\Delta\Phi measured in Section 6 is its two-step instance on real data.
Figure 2: Exact constituents compose with an error set by curvature disagreement (Theorem 4.2 , quadratic models with a shared optimum at zero). (a) Two sources with different curvature ellipses. The real two-step and joint one-step endpoints coincide (blue), and the composed endpoint is displaced by η2VHθ0\eta^{2}V_{H}\theta_{0} (orange). (b) Endpoint discrepancies of gradient-descent runs on random source pairs against the closed form. (c) Identical spectra with rotated eigenbases still yield a nonzero discrepancy, since VHV_{H} includes eigenvector alignment.
Figure 3: The observed error of a union of distillates is the net of the local discrepancy and the source residual. Exact decomposition dind=ΔΦ+rsrcd_{\rm ind}=\Delta\Phi+r_{\rm src} on the distribution-matching sets of Section 6 , with the step calibration described there (CIFAR-10, SVHN and CIFAR-100; level means over partitions, Appendix Table 8 ). (a) RMS norms of the local discrepancy ΔΦ\Delta\Phi , the aggregate source residual rsrcr_{\rm src} and the observed error dindd_{\rm ind} , with the quadratic prediction η2rcomp\eta^{2}r_{\rm comp} marked; the local discrepancy is exactly zero in the no-split control and 0.0130.013 – 0.018×10−30.018\times 10^{-3} at IID, below the resolution of the axis. The residual is the larger term at every level, and ∥dind∥<∥rsrc∥\|d_{\rm ind}\|<\|r_{\rm src}\| under skew. (b) Cosines at the skewed levels: the two terms are anti-aligned, the prediction is partly aligned with ΔΦ\Delta\Phi at this step size ( ηL≈1\eta L\approx 1 ) and nearly orthogonal to dindd_{\rm ind} .
Figure 4: The excess endpoint error grows with label skew, whereas the accuracy gain of joint distillation persists in the no-split control. Independent versus joint distillation on CIFAR-10 with distribution matching at ten images per class per source. Each skew level has five partitions (circles), and squares with bars give their mean and SD. Within a partition, endpoint errors (after the per-set step calibration of Section 6 ) are RMS over three initializations and accuracies average three evaluation seeds. (a) RMS endpoint error of the separately distilled union and of the joint set against the real-union two-step map. (b) Their difference, the excess endpoint error. (c) Accuracy gain of joint over separate-then-union distillation. The no-split control (red) has three replicates of the full dataset, and its bar is the SD over replicates.
Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

SymbolMeaning
Maps
ΦDK,η\Phi_{D}^{K,\eta}KK -step full-batch GD map at step size η\eta on dataset DD
Φind, ΦR\Phi_{\rm ind},\ \Phi_{R}composed synthetic one-step map; real-union two-step map
fp(H)f_{p}(H)time-compression transform (I−(I−ηH)p)/η(I-(I-\eta H)^{p})/\eta , p=T/Kp=T/K
Errors
E, C=E\mathcal{E},\ \mathcal{C}=\sqrt{\mathcal{E}}squared endpoint error over θ0∼P0\theta_{0}\sim P_{0} ; its RMS form
Appendix
Table 1: Notation.
Figure 5: Concavity of the finite-step clock. gpg_{p} is concave in the source curvature, so the average of two compressed sources (compress, then mix) lies below the compression of their average (mix, then compress); under gradient flow (dashed) the two coincide.
Figure 6: Scalar reweighting repairs one curvature mode, not two. Composition gap (normalized) as a function of the synthetic weight β\beta : for a single mode there is a β∗≠α\beta^{*}\neq\alpha at which the gap is zero; for two modes with different βj∗\beta^{*}_{j} the minimum over β\beta is a residual floor.
nsynn_{\rm syn}rank of Gramrel. Frobenius errorendpoint lossmeasured/closed-form defectlog-log corr.
4448.0%4.8×10−14.8\times 10^{-1}11.7 ±\pm 1.40.12
6631.7%2.0×10−12.0\times 10^{-1}7.6 ±\pm 0.90.16
8819.9%8.9×10−28.9\times 10^{-2}4.8 ±\pm 0.60.16
1212<10−6<10^{-6}1.3×10−311.3\times 10^{-31}1.000 ±\pm 0.0001.00
2412<10−6<10^{-6}1.6×10−311.6\times 10^{-31}1.000 ±\pm 0.0001.00
Appendix
Table 2: E5: endpoint matching recovers the transformed statistic once the budget reaches the dimension.
Figure 7: Where the arbitrary-ratio lower bound is certified. The zero-crossing of Theorem E.2 ’s sufficient-separation bracket as a function of effective defect rank, for several compression ratios; the bound is certified below each curve, and larger pp , step size, or rank can make it uninformative even though the exact composition gap remains nonnegative. The dotted line marks ηL≈0.12\eta L\approx 0.12 , the value of the ReLU 2,000-example subset diagnostic at ηref\eta_{\rm ref} (the calibration cells of Figure 9 lie at ηL=0.04\eta L=0.04 – 0.110.11 ); the independent-versus-joint comparison operates at ηL≈1\eta L\approx 1 , outside the range, where only the exact p=2p=2 identity is invoked.
Figure 8: The detectability window (schematic). The structural term grows as η2\eta^{2} , constituent error as η\eta , nonlinear error as η3\eta^{3} ; the structural term is certifiably visible only where it exceeds their sum, the shaded interval of Corollary G.2 .
ComponentSignatureDiagnosticMinimum repair required
Global clockcommon ratioendpoint speedscalar retiming
Source clockcoefficients cic_{i}source HVP scaleper-source reweighting
Curvature varianceVH≠0V_{H}\neq 0PcompP_{\rm comp}mode-dependent if the required retimings differ ( H.1 )
Affine conflictVHμ0−w≠0V_{H}\mu_{0}-w\neq 0mean shiftscalar retiming only if it matches the linear part and the affine offset simultaneously (the scalar example of Section 4 is repaired by the factor 3.6/3.53.6/3.5 ); otherwise mode-dependent
Missing subspacerank deficitpersistent residualnot repairable by retiming
Appendix
Table 3: Theory-derived repairability taxonomy.
PredictorSpearman ρ\rhopartial ρ\rho (adj. label div.)partial ρ\rho (adj. grad. dis.)
PcompP_{\rm comp}0.991 [0.960, 0.999]0.869 [0.568, 0.989]0.701 [0.210, 0.971]
PisoP_{\rm iso}0.955 [0.857, 0.984]0.316 [ −- 0.316, 0.785]−- 0.001 [ −- 0.262, 0.496]
Label divergence0.964 [0.907, 0.985]N/AN/A
Gradient disagreement0.983 [0.937, 0.995]N/AN/A
Appendix
Table 4: E1: which statistic predicts the observed discrepancy.
Figure 9: Local-law validation with all activations and both levels. (a) Squared discrepancy ∥ΔΦ(θ0)∥2\|\Delta\Phi(\theta_{0})\|^{2} against ηL\eta L at Dir-0.1, with exponents fitted inside the shaded window ηL<1\eta L<1 and dotted lines at ηref\eta_{\rm ref} and ηL=1\eta L=1 . (b) Calibration ratio ∥ΔΦ(θ0)∥/(η2∥rcomp(θ0)∥)\|\Delta\Phi(\theta_{0})\|/(\eta^{2}\|r_{\rm comp}(\theta_{0})\|) against ηL\eta L (solid: Dir-0.1, dashed: IID, bands over partitions), with the small-step regime enlarged in the inset. At IID the ReLU structural term is at numerical zero, so its ratio is not meaningful.
Figure 10: Dense E1 sweep on the ReLU network. (a) Partition-level mean RMS discrepancy against ηL\eta L at nine step sizes (five partitions per level with three initializations each, error bars are SDs over partitions); filled markers are the eight rates inside the strict window and lines are the fitted power laws. (b) Fitted exponents of Cloc2\mathcal{C}_{\rm loc}^{2} in η\eta with 95% intervals from a bootstrap over the five partitions; both skewed levels lie 3–6% below four, and the IID discrepancy is not governed by the structural term.
BankIter.Bank lossHeld-outUpw\mathcal{U}_{\rm pw}Cind\mathcal{C}_{\rm ind}Cjoint\mathcal{C}_{\rm joint}Γ^RMS\widehat{\Gamma}_{\rm RMS}Upw/Cstruct\mathcal{U}_{\rm pw}/\mathcal{C}_{\rm struct}ineq.
641,1000.1160.380±0.0080.380\pm 0.0083.811.250.990.2638.50/96
643,0000.1140.379±0.0080.379\pm 0.0083.791.250.990.2638.40/96
5121,1000.1320.372±0.0080.372\pm 0.0083.731.200.950.2437.70/96
5123,0000.1300.370±0.0080.370\pm 0.0083.711.200.950.2537.50/96
Appendix
Table 5: E10: attribution of the source-error plateau. Dir-0.1, ten images per class per source, (2,1)(2,1) at ηref\eta_{\rm ref} , SiLU, three partitions, all conditions scored on the same 32 held-out initializations. Bank loss is the relative matching loss of the source sets on their training bank, held-out is the relative source error on the evaluation initializations (mean and SD over partitions), errors are in units of 10−310^{-3} , and Γ^RMS=Cind−Cjoint\widehat{\Gamma}_{\rm RMS}=\mathcal{C}_{\rm ind}-\mathcal{C}_{\rm joint} is computed from the unrounded RMS values. These runs were archived with Upw\mathcal{U}_{\rm pw} , the RMS over initializations of ∑iαi∥ei(θ0)∥\sum_{i}\alpha_{i}\|e_{i}(\theta_{0})\| , in place of Usrc\mathcal{U}_{\rm src} of Table 6 ; on the cell common to both tables the two differ by 0.2%0.2\% ( 3.8083.808 against 3.8143.814 ). Ineq. counts initializations as in Table 6 .
Figure 11: Endpoint-matching distillates on the SiLU network (E10; Dir-0.1 partition 0 of the E1 subset, source 0, which holds the seven classes shown). Images are initialized from real images of the class and optimized for 1,1001{,}100 Adam iterations on the one-step-versus-two-step endpoint discrepancy; the columns show the initialization and the learned sets at one, five and ten images per class (up to five per class displayed). The learned images move far from their initialization (median relative displacement 1.11.1 ), since the objective constrains the induced training step and not the appearance.
Figure 12: At small step sizes the local prediction is accurate, but the endpoint error of learned synthetic sets lies far above the local discrepancy. Endpoint-matching distillates on the SiLU network at Dir-0.1 (three partitions, RMS over 3232 held-out initializations, mean and SD over partitions). (a) Weighted source error, error of the union of the learned sets and error of the jointly learned set against the image budget at ηref\eta_{\rm ref} ; the local reference Cloc\mathcal{C}_{\rm loc} and the structural composition error Cstruct\mathcal{C}_{\rm struct} closely agree and lie far below. (b) The same quantities against the step size at five images per class, with the cosine between ΔΦ\Delta\Phi and η2rcomp\eta^{2}r_{\rm comp} at each step size.
LevelSettingIPCUsrc\mathcal{U}_{\rm src}rel.Cind\mathcal{C}_{\rm ind}Cjoint\mathcal{C}_{\rm joint}Cloc\mathcal{C}_{\rm loc}Cstruct\mathcal{C}_{\rm struct}cos⁡(ΔΦ,s^)\cos(\Delta\Phi,\hat{s})cos⁡(dind,s^)\cos(d_{\rm ind},\hat{s})ineq.
IID(2,1)(2,1) , ηref\eta_{\rm ref}12.030.511.841.510.000090.000091.0000.050/96
IID(2,1)(2,1) , ηref\eta_{\rm ref}51.100.271.041.010.000090.000091.0000.040/96
IID(2,1)(2,1) , ηref\eta_{\rm ref}101.020.250.990.990.000090.000091.0000.040/96
IID(2,1)(2,1) , 4ηref4\eta_{\rm ref}54.110.263.863.710.00140.00140.9970.030/96
IID(2,1)(2,1) , 16ηref16\eta_{\rm ref}512.90.2311.610.60.0210.0220.955−0.04-0.040/96
IID(4,2)(4,2) , ηref\eta_{\rm ref}52.160.272.041.970.00050.00020.9990.040/96
Appendix
Table 6: E10: endpoint-matching distillates on the network (SiLU unless stated). Per cell, RMS over 32 held-out initializations; mean over three partitions. Errors in units of 10−310^{-3} . Usrc=∑iαi∥ΦSiK,η−ΦDiT,η∥L2(P0)\mathcal{U}_{\rm src}=\sum_{i}\alpha_{i}\|\Phi^{K,\eta}_{S_{i}}-\Phi^{T,\eta}_{D_{i}}\|_{L^{2}(P_{0})} is the weighted source error of Section 6 at the row’s (T,K)(T,K) , rel. is the weighted mean over sources of each set’s endpoint error divided by its own real TT -step displacement, s^=K(p2)η2rcomp\hat{s}=K\binom{p}{2}\eta^{2}r_{\rm comp} is the structural prediction (coefficient 11 for (2,1)(2,1) and 22 for (4,2)(4,2) , Theorem E.2 ), and dindd_{\rm ind} is the per-initialization error of the union of the learned sets against the real-union endpoint. The last column (ineq.) is evaluated before aggregation: it counts initializations θ0\theta_{0} at which ∥s^(θ0)∥>∑iαi∥ei(θ0)∥\|\hat{s}(\theta_{0})\|>\sum_{i}\alpha_{i}\|e_{i}(\theta_{0})\| , with ei(θ0)=ΦSiK,η(θ0)−ΦDiT,η(θ0)e_{i}(\theta_{0})=\Phi^{K,\eta}_{S_{i}}(\theta_{0})-\Phi^{T,\eta}_{D_{i}}(\theta_{0}) .
 Endpoint errorsTest accuracy
Source splitreference Cloc\mathcal{C}_{\rm loc}separate Cind\mathcal{C}_{\rm ind}joint Cjoint\mathcal{C}_{\rm joint}excess Γ^RMS\widehat{\Gamma}_{\rm RMS}separatejointgain
Heterogeneity sweep (five partitions per level)
IID0.016±00.016\pm 0 .001)7.41±07.41\pm 0 .04)7.34±07.34\pm 0 .08)0.06±00.06\pm 0 .10)46.3±046.3\pm 0 .9)51.6±051.6\pm 0 .6)+5.3±0+5.3\pm 0 .9)
Dir-1.04.85±04.85\pm 0 .99)8.72±08.72\pm 0 .29)7.34±07.34\pm 0 .08)1.38±01.38\pm 0 .25)46.2±046.2\pm 0 .6)51.9±051.9\pm 0 .5)+5.7±0+5.7\pm 0 .6)
Dir-0.55.58±05.58\pm 0 .76)8.71±08.71\pm 0 .31)7.33±07.33\pm 0 .06)1.38±01.38\pm 0 .27)45.8±145.8\pm 1 .1)51.7±051.7\pm 0 .5)+5.9±1+5.9\pm 1 .2)
Dir-0.18.09±18.09\pm 1 .17)9.85±19.85\pm 1 .33)7.50±07.50\pm 0 .26)2.35±12.35\pm 1 .44)45.8±045.8\pm 0 .7)51.1±151.1\pm 1 .1)+5.3±0+5.3\pm 0 .7)
Appendix
Table 7: Separate-then-union versus joint distillation on CIFAR-10 (distribution matching, official schedule, 10 images per class per source). Operator quantities are RMS endpoint distances to the real-union two-step map, in units of 10−310^{-3} . Accuracies in percent.
Figure 13: Independent versus joint distillation: excess endpoint error versus the local reference Cloc\mathcal{C}_{\rm loc} on CIFAR-10. Each point represents one partition (five per level, three seeds each); the line is a descriptive pooled no-intercept fit across the skewed levels (slope 0.28). Level means appear in Table 7 .
Figure 14: Synthetic images on one Dir-0.1 partition of CIFAR-10 (partition 0, DM at the official schedule; up to five images per class per set are shown). Left: the union of the two independently distilled sets (E2), with the frame colour marking the source of each image (blue, source 1, which holds five classes; orange, source 2, which holds all ten); each source stores ten images per class it holds, so the union has twenty images in the shared classes and ten in the others. Middle: the jointly distilled set at the same total budget of 150150 images, 1515 per class. Right: the class-allocation-matched joint set of E11 (Appendix J.8 ), which places 2020 images in the five shared classes and 1010 in the five classes held by one source, the per-class counts of the union.
DatasetLevel∥dind∥\|d_{\rm ind}\|∥djoint∥\|d_{\rm joint}\|∥ΔΦ∥\|\Delta\Phi\|∥rsrc∥\|r_{\rm src}\|∥s^∥\|\hat{s}\|cos⁡(ΔΦ,rsrc)\cos(\Delta\Phi,r_{\rm src})cos⁡(ΔΦ,s^)\cos(\Delta\Phi,\hat{s})cos⁡(dind,s^)\cos(d_{\rm ind},\hat{s})
CIFAR-10IID7.417.340.0167.410.002−0.01-0.010.10−0.04-0.04
CIFAR-10Dir-1.08.727.344.859.154.84−0.35-0.350.410.06
CIFAR-10Dir-0.58.717.335.589.847.26−0.48-0.480.510.01
CIFAR-10Dir-0.19.857.508.0910.412.3−0.46-0.460.600.04
CIFAR-10no split7.437.3407.430–––
SVHNIID6.086.130.0136.080.002−0.02-0.020.080.09
Appendix
Table 8: E9: exact decomposition of the observed discrepancy on the E2 sets. Norms in units of 10−310^{-3} , RMS over seeds within partition and mean over partitions; cosines are means over seeds and partitions. s^=η2rcomp\hat{s}=\eta^{2}r_{\rm comp} .
DatasetLevel∥dind∥\|d_{\rm ind}\|∥djoint∥\|d_{\rm joint}\|∥rsrc∥\|r_{\rm src}\|cos⁡(ΔΦ,rsrc)\cos(\Delta\Phi,r_{\rm src})excess
CIFAR-10IID7.41 / 13.47.34 / 13.47.41 / 13.4−0.01-0.01 / +0.02+0.02+0.07±0.09+0.07\pm 0.09 / −0.00±0.09-0.00\pm 0.09
CIFAR-10Dir-1.08.72 / 13.47.34 / 13.49.15 / 13.6−0.35-0.35 / −0.20-0.20+1.42±0.25+1.42\pm 0.25 / +0.08±0.08+0.08\pm 0.08
CIFAR-10Dir-0.58.71 / 13.57.33 / 13.39.84 / 13.9−0.48-0.48 / −0.27-0.27+1.42±0.27+1.42\pm 0.27 / +0.17±0.11+0.17\pm 0.11
CIFAR-10Dir-0.19.85 / 13.77.50 / 13.410.4 / 14.4−0.46-0.46 / −0.36-0.36+2.38±1.45+2.38\pm 1.45 / +0.28±0.08+0.28\pm 0.08
SVHNDir-0.17.65 / 11.06.18 / 10.712.1 / 12.8−0.78-0.78 / −0.56-0.56+1.47±0.95+1.47\pm 0.95 / +0.37±0.16+0.37\pm 0.16
CIFAR-100Dir-0.12.51 / 7.831.96 / 7.822.87 / 7.85−0.48-0.48 / −0.11-0.11+0.56±0.03+0.56\pm 0.03 / +0.01±0.03+0.01\pm 0.03
Appendix
Table 9: E9 without the per-set scalar calibration. Each entry gives the calibrated value of Table 8 followed by the uncalibrated value (class-mass weights, ci=1c_{i}=1 ). Norms in units of 10−310^{-3} ; the excess is Γ^mean\widehat{\Gamma}_{\rm mean} , the signed per-initialization difference ∥dind∥−∥djoint∥\|d_{\rm ind}\|-\|d_{\rm joint}\| averaged within partition, mean ±\pm SD over partitions, which differs from Γ^RMS=Cind−Cjoint\widehat{\Gamma}_{\rm RMS}=\mathcal{C}_{\rm ind}-\mathcal{C}_{\rm joint} of Table 7 by a few percent.
DatasetLevelWeightingCind\mathcal{C}_{\rm ind}Cjoint\mathcal{C}_{\rm joint}removed (ind / joint)excess
CIFAR-10IIDpaper7.41 ±\pm 0.047.34 ±\pm 0.08–++ 0.06 ±\pm 0.10
  bank7.82 ±\pm 0.057.83 ±\pm 0.08−- 0.11 / −- 0.14−- 0.01 ±\pm 0.12
  oracle5.83 ±\pm 0.055.76 ±\pm 0.040.38 / 0.38++ 0.07 ±\pm 0.04
CIFAR-10Dir-1.0paper8.72 ±\pm 0.297.34 ±\pm 0.08–++ 1.38 ±\pm 0.25
  bank7.93 ±\pm 0.087.83 ±\pm 0.080.17 / −- 0.14++ 0.10 ±\pm 0.08
  oracle6.00 ±\pm 0.105.76 ±\pm 0.040.53 / 0.38++ 0.24 ±\pm 0.10
Appendix
Table 10: E12: endpoint error of the saved DM sets under three class weightings. RMS over the three evaluation initializations within partition, mean ±\pm SD over five partitions, in units of 10−310^{-3} . Removed is the fraction of squared error removed relative to the paper’s weighting; excess is Cind−Cjoint\mathcal{C}_{\rm ind}-\mathcal{C}_{\rm joint} under the same weighting.
SetIterationsAccuracyvs. unioneffective rank
union of the independent sets046.44 ±\pm 0.50–8.91
warm start from the union50046.16 ±\pm 0.51−- 0.28 ±\pm 0.439.00
warm start from the union2,00046.88 ±\pm 0.67++ 0.44 ±\pm 0.479.15
cold start from real images2,00048.27 ±\pm 0.54++ 1.83 ±\pm 0.659.66
joint set (official schedule)20,00052.59 ±\pm 0.36++ 6.15 ±\pm 0.6610.29
Appendix
Table 11: E13: joint distillation warm-started from the union of the no-split sets (CIFAR-10, DM, 200 images). Accuracy in percent, mean ±\pm SD over three replicates; paired differences are within replicate; effective rank is the per-class participation ratio of the frozen embedding.
  Matched jointCanonical joint MatchedCanonical
Partition (images)Cind\mathcal{C}_{\rm ind}Cjoint\mathcal{C}_{\rm joint}Γ^RMS\widehat{\Gamma}_{\rm RMS}Cjoint\mathcal{C}_{\rm joint}Γ^RMS\widehat{\Gamma}_{\rm RMS}Acc. indep.Acc. jointGainAcc. jointGain
0 (150)9.487.921.557.951.5345.4±0.745.4\pm 0.746.4±0.646.4\pm 0.6+1.0+1.049.9±0.549.9\pm 0.5+4.5+4.5
1 (170)8.887.821.067.351.5346.1±1.046.1\pm 1.049.5±0.549.5\pm 0.5+3.4+3.451.4±0.451.4\pm 0.4+5.4+5.4
2 (200)12.187.294.907.294.9046.5±0.846.5\pm 0.852.5±0.852.5\pm 0.8+6.0+6.052.8±0.752.8\pm 0.7+6.3+6.3
Appendix
Table 12: E11: class-allocation-matched joint set at Dir-0.1 on CIFAR-10. The image count is the budget of both joint sets on that partition. Endpoint quantities are RMS over three initialization seeds in units of 10−310^{-3} , accuracy is the mean and SD over five evaluation seeds in percent, and the gain is joint minus independent.
Figure 15: Composed error follows constituent error (DM, Dirichlet-0.1, partition p0). (a) Relative constituent error at the snapshot checkpoints {0,1000,2500,5000,10000,15000,20000} of the official schedule; it plateaus after 5,000 iterations. (b) The composed error Cind\mathcal{C}_{\rm ind} tracks it with unit elasticity and settles above the local reference Cloc\mathcal{C}_{\rm loc} (dashed), at 1.15×1.15\times for this partition and about 1.2×1.2\times over the five partitions of the official sweep.
Figure 16: Image budget at Dir-0.1 (the same three partitions at every budget). (a) Excess endpoint error Γ^mean\widehat{\Gamma}_{\rm mean} of the separately distilled union, aggregated as the mean over seeds of the per-seed endpoint-distance difference as in Table 13 , with the local reference Cloc\mathcal{C}_{\rm loc} , which the partitions fix. (b) Accuracy gain of joint over separate-then-union distillation.
IPCϵrel\epsilon_{\rm rel}Cind\mathcal{C}_{\rm ind}Cind/Cloc\mathcal{C}_{\rm ind}/\mathcal{C}_{\rm loc}Γ^mean\widehat{\Gamma}_{\rm mean}acc. diff. (pp)elasticity to next
10.416 ±\pm 0.02321.32.60 ±\pm 0.454.47 ±\pm 0.07++ 0.66 ±\pm 1.120.83
50.206 ±\pm 0.02111.91.44 ±\pm 0.102.71 ±\pm 1.70++ 5.25 ±\pm 1.500.88
100.172 ±\pm 0.01910.21.22 ±\pm 0.032.68 ±\pm 2.00++ 5.47 ±\pm 0.901.09
200.157 ±\pm 0.0199.201.09 ±\pm 0.062.61 ±\pm 2.10++ 2.06 ±\pm 0.140.95
500.147 ±\pm 0.0178.631.02 ±\pm 0.082.62 ±\pm 2.30++ 1.68 ±\pm 0.27N/A
Appendix
Table 13: Image budget at Dir-0.1 (DM, official schedule; partitions p0–p2 ×\times three initialization seeds at every budget).
DatasetLevelnnCloc\mathcal{C}_{\rm loc}Γ^mean\widehat{\Gamma}_{\rm mean}accuracy gain (pp)
CIFAR-10IID50.0160.08 ±\pm 0.06++ 5.28 ±\pm 0.92
CIFAR-10Dir-0.157.722.33 ±\pm 1.40++ 5.35 ±\pm 0.75
SVHNIID50.013−- 0.04 ±\pm 0.04++ 3.24 ±\pm 1.10
SVHNDir-0.1510.51.47 ±\pm 0.95++ 5.48 ±\pm 1.45
CIFAR-10no split30 (exact)0.08 ±\pm 0.21++ 6.55 ±\pm 0.42
SVHNno split30 (exact)0.09 ±\pm 0.16++ 2.16 ±\pm 1.27
Appendix
Table 14: Cross-dataset comparison (DM, official schedule, IPC 10 per source; five partitions per split level on CIFAR-10 and SVHN, three on CIFAR-100, three replicates for the no-split controls).
   ConvNetResNet-18
DatasetLevelnnjointgainunionjointgain
CIFAR-10IID551.6 ±\pm 0.6++ 5.28 ±\pm 0.9240.3 ±\pm 0.642.1 ±\pm 0.5++ 1.81 ±\pm 0.85
CIFAR-10Dir-1.0551.9 ±\pm 0.5++ 5.72 ±\pm 0.6440.2 ±\pm 0.642.3 ±\pm 0.6++ 2.05 ±\pm 0.77
CIFAR-10Dir-0.5551.7 ±\pm 0.5++ 5.86 ±\pm 1.2439.2 ±\pm 1.142.6 ±\pm 0.4++ 3.31 ±\pm 0.84
CIFAR-10Dir-0.1551.1 ±\pm 1.1++ 5.35 ±\pm 0.7537.8 ±\pm 1.141.2 ±\pm 0.8++ 3.44 ±\pm 0.95
CIFAR-10no split352.6 ±\pm 0.4++ 6.55 ±\pm 0.4241.1 ±\pm 0.842.2 ±\pm 0.1++ 1.13 ±\pm 0.79
Appendix
Table 15: E14: accuracy of the same synthetic sets on the ConvNet and on ResNet-18 (DM, official evaluation protocol, three evaluation seeds). Percent, mean ±\pm SD over partitions; gain is joint minus union within partition.
Levelnneff. rank, separate (unioned)eff. rank, jointcross/within
IID28.91 ±\pm 0.1110.27 ±\pm 0.020.942 ±\pm 0.000
Dirichlet-0.157.39 ±\pm 0.429.79 ±\pm 0.401.288 ±\pm 0.097
No split (E7)38.91 ±\pm 0.0510.29 ±\pm 0.110.941 ±\pm 0.001
Appendix
Table 16: Effective rank per class and cross/within distance ratio in a frozen feature space. Mean ±\pm SD over partitions ( nn as listed; two IID partitions, five Dir-0.1 partitions, three no-split replicates); image budgets matched within partition. Dir-1.0 and Dir-0.5 were not measured.
NetworkLevelηL\eta LrHr_{H}δH\delta_{H}
Local-law network at ηref=7.8×10−4\eta_{\rm ref}=7.8\times 10^{-4}
ReLU, 2,000-sample subsetIID / Dir-1.0 / Dir-0.10.112 / 0.116 / 0.126—0.29 ±\pm 0.08
ReLU, 10410^{4} subsetIID / Dir-0.10.096 / 0.108——
SiLU, 10410^{4} subsetIID / Dir-0.10.047 / 0.066—0.14 ±\pm 0.02
tanh, 10410^{4} subsetIID / Dir-0.10.039 / 0.060—0.20 ±\pm 0.02
Distillation network (ReLU) at η=6.25×10−3\eta=6.25\times 10^{-3}
Appendix
Table 17: Assumption diagnostics.
Shiftlabel TVnnPcompP_{\rm comp}Pcomp\sqrt{P_{\rm comp}}Cloc\mathcal{C}_{\rm loc}Cloc/Pcomp\mathcal{C}_{\rm loc}/\sqrt{P_{\rm comp}}Γ^mean\widehat{\Gamma}_{\rm mean} †
IID0.00054.13×10−124.13\times 10^{-12}0.0020.016(8.1)−- 0.377
Feature skew f0.00083.95×10−73.95\times 10^{-7}0.630.652(1.04)−- 0.478
Dirichlet-1.00.22752.30×10−52.30\times 10^{-5}4.614.541.01++ 0.928
Dirichlet-0.50.30055.25×10−55.25\times 10^{-5}7.125.480.78not run
Dirichlet-0.10.39251.42×10−41.42\times 10^{-4}11.77.720.67++ 1.96
Appendix
Table 18: Type of heterogeneity changes the empirical outcome.
Figure 17: Better constituents do not always yield a better union. Ratio of the error with stored semantic mass to the error with uniform weights, for the individual sources and for their composed union (percentages: change relative to uniform weights). (a) At Dir-1.0 the individual error falls by 60–71% while the composed error is unchanged. (b) At Dir-0.1 the improvement partly survives union.
 m=2m=2m=5m=5ratio
Real partition and network only
Cloc\mathcal{C}_{\rm loc} ( ×10−3\times 10^{-3} )8.09 ±\pm 1.1714.1 ±\pm 2.31.75
Pcomp\sqrt{P_{\rm comp}} ( ×10−3\times 10^{-3} )12.3 ±\pm 2.125.8 ±\pm 2.22.09
Distilled sets (IPC 10 per source; total budget grows with mm )
Γ^RMS\widehat{\Gamma}_{\rm RMS} ( ×10−3\times 10^{-3} )2.35 ±\pm 1.445.65 ±\pm 1.562.41
accuracy gain (pp)++ 5.35 ±\pm 0.75++ 7.34 ±\pm 1.341.37
Appendix
Table 19: Two versus five sources at Dir-0.1 (DM, official schedule, IPC 10 per source). Mean ±\pm SD over n=5n=5 partitions; within a partition the endpoint quantities are RMS over the three initialization seeds 100–102 and the accuracy gain is their mean, the convention of Table 7 , whose Dir-0.1 row the m=2m=2 column reproduces. The first block depends only on the real partition and the network; the second describes the distilled sets, whose total budget is not matched between m=2m=2 and m=5m=5 , so Cind\mathcal{C}_{\rm ind} and its ratio to Cloc\mathcal{C}_{\rm loc} confound source count with budget and are not interpreted.

Explore similar work

CardsList

  1. Dataset Distillation Efficiently Encodes Low-Dimensional Representations from Gradient-Based Learning of Non-Linear Tasks

    Mar 16, 2026Yuri Kinoshita, Naoki Nishikawa, Taro ToyoizumiDiffusion-Based Dataset DistillationDataset Distillation

    The University of Tokyo, Tokyo, Japan · Laboratory for Neural Computation and Adaptation, RIKEN Center for Brain Science, Wako, Japan · RIKEN Center for Advanced Intelligence Project, Nihonbashi, Japan

  2. Fair Dataset Distillation via Cross-Group Barycenter Alignment

    Apr 30, 2026Mohammad Hossein Moslemi, Nima Hosseini Dashtbayaz, Zhimin Mei +2Diffusion-Based Dataset DistillationDataset Distillation

    Western University, London, Ontario · Vector Institute, Toronto, Ontario · Ivey Business School, London, Ontario +1

  3. Structural Assessment for Understanding and Guiding Dataset Distillation in Discrete Token Space

    Jun 19, 2026Yue Cao, Jianyang Gu, Vyacheslav Kungurtsev +4Diffusion-Based Dataset DistillationDataset Distillation

    The University of British Columbia · The Ohio State University · Czech Technical University in Prague +1