Information-Theoretic Scaling Laws for Generative Diffusion Training

Druv Pai; Dejan Stancevic; Yi Ma; Luca Ambrogioni

cs.LGOct 23, 2026

Information-Theoretic Scaling Laws for Generative Diffusion Training

Authors: Druv Pai, Dejan Stancevic, Yi Ma, Luca Ambrogioni

Organizations: UC Berkeley · Radboud University · HKU, UC Berkeley

Abstract

Diffusion models trained on increasingly large datasets sampled from the same data distribution will converge to the same learned distribution. However, in practice, a model trained on finite data depends on the particular training dataset; this dependence is reflected in properties of the model's outputs, such as memorization and sample quality. To understand this dependence through an information-theoretic lens, we study model information, a computationally tractable measure of how much samples from the trained model depend on the particular training dataset. We explore its scaling behavior as model size and training dataset size vary. Specifically, we uncover two qualitatively distinct scaling regimes: (1) a nonlinear model information regime marking the transition from pure memorization to imperfect generalization, and (2) a generalization regime characterized by linear scaling of the model information and progressively improved generalization and sample quality. We then posit an information-theoretically principled scaling law to quantify these two regimes, and show that this law yields induced scaling laws for memorization and sample quality. We finally demonstrate that these induced scaling laws extrapolate well outside the fit's domain. Our results demonstrate information-theoretic principles for memorization and sample quality scaling laws in diffusion models. Our code is publicly available on GitHub.

Figures & tables

Figure 1: Model information distinguishes generalization from memorization. A source distribution

p_{\phi_{\star}}

generates independent finite datasets

\mathcal{D}_{i}

, which are used to train models

\Phi_{i}

. In the generalization regime, the trained models have nearly indistinguishable output distributions, so a sample

\widehat{X}

carries little information about which model (or training dataset) produced it. In the memorization regime, each trained model’s outputs concentrate around its own training data, so generated samples reveal the training dataset.

Panel of Figure 2 — Figure 2: Variable dependence for the model information. The data source $\phi_{\star}$ generates the training dataset $\mathcal{D}$ , whose randomness induces the distribution $\Pi$ over learned models. A model $\Phi$ trained on $\mathcal{D}$ generates $\hat{X}$ , and the forward channel produces $\hat{X}_{t}$ .

Panel of Figure 3 — Figure 3: Model information scaling on ImageNet. Model information as a function of training-set size for four model sizes (written in terms of depth $L$ ), with the fit of ( 14 ). The right panel shows the $N^{-1}$ regime.

Panel of Figure 4 — Figure 4: The estimate depends weakly on the number of trained models. Fashion-MNIST at depth $L=4$ , from four to ten trained models per $(P,N)$ , has nearly identical model information across different scales (Appendix D.9 ).

Panel of Figure 5 — Figure 5: Memorization occurs below $N=N_{\mathrm{c}}(P)$ , while sample quality improves in the $N^{-1}$ regime. The sample size is normalized by the critical training-set size $N_{\mathrm{c}}(P)$ of ( 14 ). The dark band, $N/N_{\mathrm{c}}(P)\in[e^{-2\sigma},e^{2\sigma}]$ , contains the sizes at which most groups stop memorizing; the lighter band is the $N^{-1}$ regime. Below, we show random samples from models at depth $L=5$ at three training-set sizes $N$ . At (d) the model does not copy its training images, and its samples are new but blurred. At (e) the samples are more realistic, and FID is still decreasing in $N$ .

Panel of Figure 6 — Figure 6: Model information is correlated with memorization and with FID. (a) The memorization ratio at all $(P,N)$ pairs against model information divided by $\hat{I}_{\max}(P)-F(P)=\sup_{N}E(P,N)$ , computed from the fitted scaling law ( 14 ), with a logistic fit. (b) FID against $E(P,N)$ of the fitted scaling law in the $N^{-1}$ regime. The lines are one linear fit of $\log\mathrm{FID}$ on $\log E$ and $\log P$ , evaluated at each model size.

Panel of Figure 7 — Figure 7: Predicting model information, memorization, and FID at a new model size. All fits use only the models at depths $L\in\{2,3,4\}$ (faint markers and curves). Open markers are the measurements at $L=5$ , the dashed curves are the predictions for $L=5$ , and the star is the new pair $(L=5,N=250{,}000)$ , outside the range of our experiments. (a) Model information and the fitted scaling law ( 14 ). (b) Memorization ratio. (c) FID, predicted in the $N^{-1}$ regime; the shaded region is below the predicted start of this regime for $L=5$ .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Model information per dimension in the linear-Gaussian model in the $d\to\infty$ limit, shown as a function of $N/d$ . The darker and lighter shaded regions indicate the memorization and generalization regimes.

dataset	resolution	$d$	$\sigma_{\mathrm{data}}$	$N$ values	$N_{\max}$	$N_{\mathrm{full}}$	depths $L$	$M$	$K_{\mathrm{train}}$ / $K_{\mathrm{val}}$
ImageNet	$64\times 64$	12288	0.55	12	100,000	1,271,167	2,3,4,5	4	$\{1,2\}$ / 2
CIFAR-10	$32\times 32$	3072	0.52	14	50,000	50,000	1,2,3,4,5	4	$\{1,2\}$ / 2
Fashion-MNIST	$32\times 32$	3072	0.80	7	60,000	60,000	1,2,4	10	$\{2,3,4,5\}$ / 5
FFHQ	$64\times 64$	12288	0.55	7	60,000	60,000	1,2,4	4	$\{1,2\}$ / 2

Appendix

Table 1: The four image datasets.

M

is the number of models trained per

(P,N)

pair, split into

K_{\mathrm{train}}

reference models that fit the mixture score and

K_{\mathrm{val}}

held-out models that are evaluated against it. We sweep over the listed values of

K_{\mathrm{train}}

in Appendix D.9 .

N_{\max}

is the largest training-set size in the list of values of

N

, and

N_{\mathrm{full}}

is the size of the full dataset from which every training set is drawn; on ImageNet we also train models on the full dataset at each model size.

dataset	$(P,N)$ pairs	$P_{\mathrm{ref}}$	$k$	$b$	$N_{\mathrm{c}}(P_{\mathrm{ref}})$	$a$	$\sigma$	$\ell$	$c_{0}$	$c_{1}$	$\beta$	log RMSE	$F(P)$ error
ImageNet UNet	48	$3.2{\times}10^{7}$	7.51	0.483	603	0.345	0.622	13.75	220.1	40.4	0.80	0.112	0.088
CIFAR-10 UNet	65	$2.0{\times}10^{7}$	0.911	0.518	1330	0.409	0.503	16.59	110.0	$9.9{\times}10^{-3}$	3.00	0.101	0.066
Fashion-MNIST UNet	18	$6.7{\times}10^{6}$	0.0942	0.378	1093	0.314	0.642	148.41	101.4	0.223	$-3.00$	0.074	0.746
FFHQ UNet	18	$6.7{\times}10^{6}$	4.41	0.598	485	0.488	0.571	36.18	182.5	484	0.24	0.109	0.297

Appendix

Table 2: Scaling-law coefficients. The nine parameters of ( 14 ): those of

E(P,N)

(k,b,N_{\mathrm{c}}(P_{\mathrm{ref}}),a,\sigma,\ell)

, and those of

F(P)

(c_{0},c_{1},\beta)

P_{\mathrm{ref}}

is the median model size of the dataset. The fit bounds are

|\beta|\leq 3

and

\ell\leq e^{5}\approx 148.4

. The last two columns are the log RMSE over all

(P,N)

pairs, and the log RMSE of the fitted

F(P)

against the estimates at

N=N_{\mathrm{full}}

, which are not part of the fit.

variant	ImageNet	CIFAR-10	Fashion-MNIST	FFHQ
( 14 )	0.112	0.101	0.074	0.109
single group size, $\sigma=0$	0.249	0.159	—	—
$\ell$ fixed	0.242	0.246	0.161	0.224
correction switched	0.112	0.103	0.074	0.108

Appendix

Table 3: Log RMSE of the scaling law and of restricted versions of it. “

\ell

fixed” sets

\ell=\log(\sigma_{\mathrm{data}}/\sigma_{t_{\min}})

and refits the other eight parameters. “Correction switched” adds the finite-population correction of Appendix C.4 on ImageNet and removes it on the other three datasets. On ImageNet, the fit with

\sigma=0

is off by up to a factor of two at the

(P,N)

pairs around

N=N_{\mathrm{c}}(P)

. A dash marks a variant that we did not run.

dataset	$d$	predicted $\Lambda$	fitted $\hat{I}_{\max}(P)-F(P)$	ratio	$b-a$
ImageNet	12288	$5.50{\times}10^{4}$	$6.63$ – $9.37{\times}10^{4}$	1.21–1.70	0.14
CIFAR-10	3072	$1.36{\times}10^{4}$	$1.95$ – $3.13{\times}10^{4}$	1.44–2.30	0.11
Fashion-MNIST	3072	$1.49{\times}10^{4}$	$1.66$ – $2.10{\times}10^{4}$	1.12–1.41	0.06
FFHQ	12288	$5.50{\times}10^{4}$	$7.39$ – $11.08{\times}10^{4}$	1.34–2.01	0.11

Appendix

Table 4: Predicted and fitted maximum of model information. The prediction

\Lambda=d\log(\sigma_{\mathrm{data}}/\sigma_{t_{\min}})

\sigma_{t_{\min}}=6.266\times 10^{-3}

, against

\hat{I}_{\max}(P)-F(P)

of the fitted scaling law ( 14 ) over the model sizes of each dataset; the prediction contains no fitted quantity. The ratio is fitted over predicted. The fitted

\hat{I}_{\max}(P)-F(P)

is proportional to

\widetilde{P}^{a-b}

dataset	architecture	$P$	$N_{\mathrm{c}}(P)$	$N_{\mathrm{c}}/h$	$\hat{I}_{\max}-F$	$F(P)$	$\hat{I}_{\mathrm{m}}(P,N_{\mathrm{full}})$	$\hat{I}_{\mathrm{m}}(P,N_{\mathrm{full}})/(\hat{I}_{\max}-F)$	$\hat{I}_{\mathrm{m}}(P,N_{\mathrm{full}})/d$
ImageNet UNet	$h=64,L=2$	$6.7{\times}10^{6}$	350	5.47	93683	361.7	330.1	0.4%	0.027
ImageNet UNet	$h=96,L=3$	$2.0{\times}10^{7}$	511	5.33	80611	279.2	266.1	0.3%	0.022
ImageNet UNet	$h=128,L=4$	$4.4{\times}10^{7}$	673	5.26	72264	251.4	225.7	0.3%	0.018
ImageNet UNet	$h=160,L=5$	$8.3{\times}10^{7}$	836	5.23	66303	239.1	218.2	0.3%	0.018
CIFAR-10 UNet	$h=32,L=1$	$1.1{\times}10^{6}$	410	12.82	31282	165.8	148.2	0.5%	0.048
CIFAR-10 UNet	$h=64,L=2$	$6.7{\times}10^{6}$	850	13.28	25731	110.2	109.8	0.4%	0.036

Appendix

Table 5: Fitted critical training-set sizes and

F(P)

. The critical training-set size

N_{\mathrm{c}}(P)=N_{\mathrm{c}}(P_{\mathrm{ref}})\widetilde{P}^{a}

, its ratio to the width

h

, and

\hat{I}_{\max}(P)-F(P)

. Then the fitted

F(P)

and the estimate

\hat{I}_{\mathrm{m}}(P,N_{\mathrm{full}})

, where all models share one dataset and which is not part of the fit, followed by this estimate divided by

\hat{I}_{\max}(P)-F(P)

and by the input dimension

d

(in nats).

dataset	form	params	log RMSE	BIC	$F(P)$	largest $P$	large $N$
ImageNet	MP	9	0.112	$\mathbf{-175.0}$	0.088	0.183 / 0.143	0.950 / 0.072
ImageNet	sigmoid–log	9	0.141	$-153.0$	0.544	0.221 / 0.173	1.529 / 0.085
ImageNet	sigmoid–power	10	0.142	$-148.9$	2.407	0.223 / 0.170	1.500 / 0.089
CIFAR-10	MP	9	0.101	$\mathbf{-260.8}$	0.066	0.189 / 0.123	0.105 / 0.068
CIFAR-10	sigmoid–log	9	0.102	$-259.3$	0.047	0.200 / 0.142	0.205 / 0.069
CIFAR-10	sigmoid–power	10	0.104	$-252.3$	0.074	0.204 / 0.146	0.189 / 0.071

Appendix

Table 6: Comparison of functional forms. MP is ( 14 ); sigmoid–log is ( 157 ); sigmoid–power is ( 158 ). Errors are root-mean-square in log-information. Column

F(P)

compares the fitted

F(P)

with the estimates at

N=N_{\mathrm{full}}

, which no fit uses. The last two columns give the error on withheld points / the error on the same points when they are included in the fit. In “largest

P

” we withhold every measurement at the largest model size, and in “large

N

” every dataset size above the median. A dash marks a test with fewer than four remaining dataset sizes. Lower is better in every column, and the best value on each dataset is in bold. On ImageNet the sigmoid–power form has fitted exponent

\alpha=0.15

, a log-log slope of

-0.15

dataset	memorization ratio $\tfrac{1}{2}$	FID maximum	info–mem. correlation
ImageNet	0.22–0.28	1.1–1.5	0.927
CIFAR-10	0.36–0.42	0.84–2.2	0.965
Fashion-MNIST	0.29–0.36	1.7 ( $L=4$ )	0.952
FFHQ	0.14–0.24	0.64–1.6	0.844

Appendix

Table 7: Memorization and FID across the transition. The training-set sizes, in units of

N_{\mathrm{c}}(P)

, at which the memorization ratio is one half and at which FID has its local maximum, as ranges over model sizes (Table 11 gives them per model size on ImageNet). The smallest

N

with a memorization ratio below

1\%

lies between

0.64\,N_{\mathrm{c}}(P)

and

3.4\,N_{\mathrm{c}}(P)

. On ImageNet the local maximum of FID is at this

N

. On ImageNet the memorization ratio at

N=10

is between

0.84

and

0.89

. The last column is the Pearson correlation between model information and the memorization ratio over all

(P,N)

pairs.

statistic	value
Pearson correlation, model information and memorization ratio (48 pairs)	0.93
mean absolute error of the logistic fit to the memorization ratio	0.020
Spearman correlation, model information and FID, $N^{-1}$ regime (24 pairs), each model size	$\geq 0.94$
$R^{2}$ of ( 159 )	0.95
$\kappa$ , $\xi$ in ( 159 )	0.12, $-0.17$
$R^{2}$ of ( 159 ) with measured $\log I_{\mathrm{m}}$ in place of $\log E$	0.91

Appendix

Table 8: Model information, memorization, and FID on ImageNet. Statistics behind Figure 6 and the predictions of Section 4.3 . The logistic function has two parameters and takes the logarithm of model information divided by

\hat{I}_{\max}(P)-F(P)

. The last three rows are the log RMSE of the predicted FID at

L=5

, over the five training-set sizes in the

N^{-1}

regime, for three regressions fitted on

L\in\{2,3,4\}

$N$	model information (nats)	memorization ratio	FID
10	$64{,}800$ / $74{,}700$	0.85 / 0.88	—
400	$42{,}600$ / $41{,}700$	0.22 / 0.17	—
630	$31{,}700$ / $20{,}100$	0.033 / 0.027	—
1000	$19{,}200$ / $14{,}200$	0.001 / 0.004	—
4000	$1219$ / $1098$	0 / 0	58.2 / 59.9
16,000	$357$ / $395$	0 / 0	46.1 / 46.5

Appendix

Table 9: Predictions at depth

L=5

from fits to

L\in\{2,3,4\}

on ImageNet. Each entry is predicted / measured. The fit to

L\in\{2,3,4\}

has

N_{\mathrm{c}}=965

L=5

, against

836

in the fit to all model sizes, so we predict FID for

N\geq e^{2\sigma}N_{\mathrm{c}}=3124

. The last two rows are the new training-set size

N=250{,}000\approx 299\,N_{\mathrm{c}}(P)

, outside the range of every fit, predicted from the fit to

L\in\{2,3,4\}

and from the fit to all model sizes.

dataset	model information	memorization (MAE)	correlation	FID (pairs)
ImageNet	0.183 / 0.143	0.009 / 0.008	0.999	0.066 / 0.052 (5)
CIFAR-10	0.189 / 0.123	0.022 / 0.009	0.991	0.120 / 0.098 (7)
Fashion-MNIST	0.278 / 0.078	0.039 / 0.033	0.989	0.569 / $0^{*}$ (1)
FFHQ	0.400 / 0.110	0.026 / 0.017	0.997	0.075 / 0.066 (1)
all four	—	0.021	0.993	—

Appendix

Table 10: Predictions at the withheld largest model size. Model information: log RMSE of the prediction / log RMSE of the same points in the fit to all model sizes. Memorization: mean absolute error of the predicted memorization ratio on the withheld

(P,N)

pairs / on the pairs used in the fit, and Pearson correlation between the predicted and measured ratio on the withheld pairs. FID: log RMSE on the withheld pairs with

N\geq e^{2\sigma}N_{\mathrm{c}}(P)

/ on the pairs used in the fit, and the number of withheld pairs in this range. Fashion-MNIST and FFHQ have three model sizes, so their withheld fits use two. ∗ On Fashion-MNIST the FID regression has three parameters and is fitted on three pairs, so it fits them exactly. The last row pools the

37

withheld pairs of the four datasets.

$L$	$P$	$N_{\mathrm{c}}(P)$	fitted $F(P)$	estimate at $N_{\mathrm{full}}$	slope	mem. ratio $\tfrac{1}{2}$	FID peak
2	$6.7{\times}10^{6}$	350	362	330	$-0.96$	0.26	1.14
3	$2.0{\times}10^{7}$	511	279	266	$-0.91$	0.28	1.23
4	$4.4{\times}10^{7}$	673	251	226	$-0.94$	0.26	1.49
5	$8.3{\times}10^{7}$	836	239	218	$-1.02$	0.22	1.20

Appendix

Table 11: The fitted scaling law, memorization, and FID at each model size on ImageNet.

F(P)

and the estimate at

N=N_{\mathrm{full}}

are in nats. “Slope” is the log-log slope of the measured model information minus its estimate at

N=N_{\mathrm{full}}

, between

N=4000

and

N=40{,}000

. The last two columns are the training-set sizes at which the memorization ratio is one half and at which FID has its local maximum, in units of

N_{\mathrm{c}}(P)

Panel of Figure 9 — Figure 9: Model information scaling on CIFAR-10. Model information as a function of training-set size for five model sizes, together with the fit of ( 14 ). The largest dataset size is the full dataset, $N_{\max}=N_{\mathrm{full}}=50{,}000$ , where all models share the same training data and the remaining estimate is due to the variation between training seeds and the bias of the estimator; these points are not part of the fit. The right panel magnifies the boxed region, where $E(P,N)$ decays to zero as $N^{-1}-N_{\mathrm{full}}^{-1}$ .

Panel of Figure 10 — Figure 10: The two regimes on CIFAR-10. Layout and shading follow Figure 5 . The sample size is normalized by the critical training-set size $N_{\mathrm{c}}(P)$ ; the dark band is $N/N_{\mathrm{c}}(P)\in[e^{-2\sigma},e^{2\sigma}]$ , and the lighter band the $N^{-1}$ regime. Below, we show random samples from models at depth $L=5$ at three training set sizes $N$ .

Panel of Figure 11 — Figure 11: The two regimes on Fashion-MNIST and FFHQ. Layout and shading follow Figure 10 : the darker band is $N/N_{\mathrm{c}}(P)\in[e^{-2\sigma},e^{2\sigma}]$ , where most groups cross $N=d_{z}$ , and the lighter band beyond it is the $N^{-1}$ regime. (a,c) model information for three model sizes with the fitted scaling law. (b,d) the memorization ratio (solid, left axis) and clean-FID (dashed, right axis), with standard errors across the trained models at each $(P,N)$ . On both datasets the results are as in the main text: memorization ends near $N=N_{\mathrm{c}}(P)$ , and FID falls monotonically through the $N^{-1}$ regime. On FFHQ, FID increases by a factor of $1.3$ to $1.7$ between $N=43$ and $N=180$ at $L=1$ , and between $N=180$ and $N=780$ at $L=2$ and $L=4$ . On Fashion-MNIST at $L=4$ it increases from $15.2$ at $N=780$ to $21.6$ at $N=3300$ .

Panel of Figure 12 — Figure 12: Samples across the transition on Fashion-MNIST (top) and FFHQ (bottom). All six panels come from models at $L=4$ . (a,d) ten training images, repeated verbatim. (b,e) near $N=N_{\mathrm{c}}(P)$ , where the memorization ratio has just dropped to nearly zero and FID has its local maximum: the images are new, but the least sharp of the three. (c,f) the full dataset, where no copies are detected and FID is $3.5$ and $6.6$ respectively.

Panel of Figure 13 — Figure 13: The number of reference models changes the estimates by a nearly constant factor. (a) Fashion-MNIST at $L=4$ with $K_{\mathrm{train}}$ from two to five; the four curves lie on top of one another. (b) every smaller $K_{\mathrm{train}}$ on every dataset and model size, as a ratio to the estimate at the largest $K_{\mathrm{train}}$ for that dataset: seventeen curves, all within $[0.99,1.12]$ and almost all above one. With fewer reference models the estimates are larger, which is the sign of the error term ( 155 ).

dataset	estimator	$K_{\mathrm{train}}$	$a$	$\sigma$	$\ell$	$b$	$\hat{I}_{\max}-F$	log RMSE
ImageNet	model-free	1	0.348	0.619	—	—	—	—
ImageNet	model-free	2	0.345	0.622	13.75	0.483	$7.556{\times}10^{4}$	0.112
CIFAR-10	model-free	1	0.402	0.502	15.85	0.502	$2.388{\times}10^{4}$	0.1015
CIFAR-10	model-free	2	0.409	0.500	16.20	0.519	$2.282{\times}10^{4}$	0.1009
FFHQ	model-free	1	0.508	0.503	15.39	0.631	$9.576{\times}10^{4}$	0.1304
FFHQ	model-free	2	0.509	0.505	15.10	0.625	$9.108{\times}10^{4}$	0.1205

Appendix

Table 12: Refitting the scaling law at every

K_{\mathrm{train}}

. We fit ( 14 ) independently at each

K_{\mathrm{train}}

, on the same

(P,N)

pairs. The column

\hat{I}_{\max}-F

\hat{I}_{\max}(P)-F(P)

P=P_{\mathrm{ref}}

. Across

K_{\mathrm{train}}

N_{\mathrm{c}}(P)

changes by at most

2.1\%

on Fashion-MNIST,

2.8\%

on CIFAR-10,

0.5\%

on FFHQ, and

1.3\%

on ImageNet. On ImageNet the fit excludes the estimates at

N=N_{\mathrm{full}}

; with

K_{\mathrm{train}}=1

instead of

2

, the median ratio of the estimates is

1.016

85\%

of the ratios are at most

1.05

, and the largest is

1.19

, at

N=10

and

L=2

. The fits to the two classifier estimators are stable in

K_{\mathrm{train}}

; the fit to the distillation estimator is close to them only at the largest

K_{\mathrm{train}}

Panel of Figure 14 — Figure 14: Four and ten models per $(P,N)$ pair give nearly the same estimates. Fashion-MNIST with $K_{\mathrm{train}}=K_{\mathrm{val}}=k$ , for a total of $2k$ trained models per $(P,N)$ pair; four models is the CIFAR-10, FFHQ, and ImageNet configuration. (a) $L=4$ for each number of models, as in Figure 4 . (b) the estimates with four, six, and eight models at every model size, as a ratio to the estimates with ten models: the average ratio is above one and decreases toward one as the number of models grows.

models per $(P,N)$	mean ratio	$a$	$\sigma$	$\ell$	$b$	$\hat{I}_{\max}-F$	log RMSE
4	1.07	0.383	0.510	11.50	0.449	$1.963{\times}10^{4}$	0.1296
6	1.04	0.382	0.529	12.05	0.441	$1.935{\times}10^{4}$	0.1276
8	1.02	0.397	0.519	11.80	0.472	$1.872{\times}10^{4}$	0.1312
10	1	0.358	0.513	11.64	0.420	$1.872{\times}10^{4}$	0.1026

Appendix

Table 13: Refitting the scaling law for every number of trained models. The model-free estimator on Fashion-MNIST with

K_{\mathrm{train}}=K_{\mathrm{val}}=k

, indexed by the total number

2k

of trained models per

(P,N)

pair. The ten-model row is the configuration of the other Fashion-MNIST experiments. “Mean ratio” is the average ratio of the estimates to those with ten models; individual ratios lie in

[0.90,1.18]

, and with four models the median deviation is

6.2\%

. With four models against ten,

N_{\mathrm{c}}(P)

changes by at most

8.5\%

Panel of Figure 15 — Figure 15: Samples across the memorization transition. Both panels show samples from the same CIFAR-10 $L=5$ architecture, with $N_{\mathrm{c}}(P)\approx 2380$ . The $N=710$ model sometimes repeats sharp training-set images and scores better on FID. The $N=2900$ model has stopped memorizing and scores substantially worse. At $N=6500$ FID is $23.1$ , and on the full dataset it is $7.8$ .

Diffusion Models

Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.

See all 1,176 papers

Oct 13, 2026

Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models

Understanding the mechanism behind why diffusion models memorize their training data remains an open question in generative modeling. In this paper, we examine this question from two perspectives: (a) the model-level perspective concerning the onset of memorization in a model during training, and (b) the sample-level perspective concerning the proclivity of an individual sample to be memorized by a given model. We show that the so-called input loss curvature, or the Frobenius norm of the Hessian of the diffusion model's training loss, is an effective metric for quantifying both perspectives on memorization. In particular, we empirically demonstrate that input curvatures averaged across training and validation samples can identify when a model is likely to start regurgitating its training data when tracked over training time. However, this metric is not sufficient for measuring an individual sample's likelihood of being memorized. To address this perspective, we introduce a counterfactual definition of memorization in diffusion models, extending the analogous notion from supervised learning proposed by [Feldman, 2019]. We prove that this counterfactual memorization is controlled by the input curvature of the leave-one-out model, thus establishing a direct theoretical link between a sample's local loss landscape and its generation likelihood. We validate our derived bounds through several empirical experiments and show that input-dependent curvature provides a proxy for memorization.

Shashank Sule, Xinyue Yu, Boonam Shin +3

Department of Mathematics, University of California, Los Angeles, CA 90095, USA. · Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA.

Memorization in Generative Models Diffusion Models

Ask PDF

May 7, 2026

Understanding diffusion models requires rethinking (again) generalization

This position paper argues that understanding generalization in diffusion models requires fundamentally new theoretical frameworks that go beyond both classical statistical learning theory and the benign overfitting paradigm developed for supervised learning. In diffusion models, unlike in supervised learning, memorization of training data and generalization to novel samples are incompatible: a model that has fully memorized its training set generates copies rather than novel data. Several theoretical explanations for why practical diffusion models nevertheless generalize have been proposed, based on capacity limitations, implicit regularization from optimization, or architectural inductive biases, but their interactions remain unclear. We argue that the field should pivot from explaining why the diffusion models do not memorize to investigating what the model actually learns during pre-memorization phase. To highlight our stance, we conduct empirical study of diffusion models trained on CIFAR-10, and we distill the findings into concrete open questions that we believe are key to improve understanding of generalization in diffusion models.

Pierre Marion, Yu-Han Wu

Inria, École Normale Supérieure, PSL Research University · LPSM, Sorbonne University & Google DeepMind

Memorization in Generative Models Diffusion Models

Ask PDF

Jul 9, 2026

An exact information theory of generalization phase transitions in Bayesian diffusion models

How diffusion models circumvent the curse of dimensionality to learn complex distributions over high dimensional spaces from a finite training set, instead of memorizing it, remains a fundamental mystery. To address this, we introduce analytically tractable Bayesian information restricted diffusion (BIRD) models, in which each pixel observes restricted information about noisy data. A BIRD model time-reverses diffusion by inferring which past training sample produced its current restricted observation using the Bayesian posterior. This model class generalizes existing analytical diffusion models that use spatially local information restriction. We show that spatially local BIRD models closely approximate trained diffusion models early in training, across different architectures such as UNets and DiTs. Under minimal assumptions on the data distribution, we identify an information-theoretic phase boundary between memorization and generalization in the joint space of amount of training data, time in the reverse generative process, and amount of information restriction: a BIRD model memorizes when the mutual information between its restricted noisy observations and the training data exceeds the log number of training points, and it generalizes otherwise. Experiments across a range of datasets confirm our theoretically predicted location for the transition. We find that generation proceeds near the edge of memorization: both spatially local BIRD models and early-training diffusion models track the memorization-generalization phase boundary by increasingly restricting information over time. Overall, our results reveal a fundamental role for information restriction in generative AI to circumvent the curse of dimensionality.

Henry Hunt, Mason Kamb, Surya Ganguli

Department of Applied Physics Stanford University Stanford, CA 94305

Memorization in Generative Models Diffusion Models Generalization Bounds

Ask PDF

Abstract

Diffusion Models

Explore similar work

Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models

Understanding diffusion models requires rethinking (again) generalization

An exact information theory of generalization phase transitions in Bayesian diffusion models