Denoising Is Stein: Moment Access and a Resource Hierarchy for Learning Hidden Structure in Diffusion Models

Rui Wang

cs.LGNov 2, 2026

Denoising Is Stein: Moment Access and a Resource Hierarchy for Learning Hidden Structure in Diffusion Models

Authors: Rui Wang

Organizations: Vancouver, BC, Canada

Abstract

We study score learning from denoising pairs when the data contain a hidden non-Gaussian component. If covariance reveals the component, its subspace can be recovered directly. If covariance is isotropic, the first distinguishing moment has order $k^\ast$ , and the learner must extract that moment from clean points and their noisy views. Correlations in the denoising loss are Stein discrepancies against a baseline; for polynomial fields they determine exactly the corresponding moment differences. Averaging fresh noisy views conditional on a clean point gives an exact moment query on the clean distribution. In the covariance-matched regime, orientation-independent dictionaries satisfying an $L^4/L^2$ condition need order $d^{k^\ast-1}$ features, and the stated dot-product kernel ridge models need the same power in samples. Small-step online SGD on one unit has this exponent up to logarithms. A two-stage learner uses the curvature of a smoothed denoising loss to recover the hidden direction, then sharpens it and fits a scalar readout. For every even $k^\ast$ , it obtains a fixed score gain over the Gaussian baseline from $O(d^{k^\ast/2}\log d)$ clean points. In the covariance-visible regime, the frozen-feature threshold occurs at $d^J/J!$ , while projection onto principal components makes the fitted-model, score-risk and generated-law distributions independent of ambient dimension. In the covariance-matched experiments, the measured partial-trace overlaps are consistent with the proposed spectral transition. A width-8 network trained with clean-point reuse shows a $d^2$ -like training-set threshold over $d=64$ -- $192$ , with fixed validation and calibration overhead.

Figures & tables

Figure 1: Two hidden-component data regimes in two dimensions. (a) Clean points $X_{0}=uZ$ lie on a line; blue points are their noisy views at $t=0.3$ , and short arrows connect selected pairs. The line is visible in the clean covariance. (b) The clean covariance is $I_{2}$ . Blue and orange mark the mixture components only for illustration; their labels are not observed by the learner. (c) In the covariance-matched example, the clean marginal along $u$ (orange) differs from its moment-matched Gaussian (dashed black) even though their means and variances agree. Diffusion at $t=0.3$ weakens that difference (blue).

Learner	Resource	Cost	Status
Covariance-matched regime, generative exponent $k^{\ast}$ (even where stated)
Polynomial fields of degree $<k^{\ast}-1$	any	no gain at any size	proved (Cor. 3.5 )
Orientation-blind dictionary with bounded $L^{4}/L^{2}$ constant	$M$	$\Theta(d^{k^{\ast}-1})$	proved under the $L^{4}/L^{2}$ hypothesis (Thm. 4.21 ); open for random-feature spans
Dot-product kernel ridge regression on noisy inputs	$n$	$\Omega(d^{k^{\ast}-1})$	proved for ridge $\gg d^{-(k^{\ast}-1)}$ ; at every ridge for the exponential kernel (Thm. 5.2 , Lem. 5.3 )
Online spherical SGD, one denoiser unit, small steps	$n=$ steps	$d^{k^{\ast}-1}$ up to $\log^{2}d$	proved via Ben Arous et al. (2021) (Cor. 6.2 )
$L^{2}$ statistical queries	tolerance	$d^{-k^{\ast}/4+o(1)}$ ; the sample analogue $n\gtrsim d^{k^{\ast}/2}$ is heuristic	tolerance bound proved under an output-norm assumption (Prop. 6.3 )

Table 1: The resource hierarchy for improving the score at a fixed diffusion time by a constant fraction of the Gaussian baseline’s excess risk, with

k=1

hidden direction.

M

: dictionary size;

n

: distinct clean points;

r

: noisy views per clean point. Costs are up to constants depending on the latent law and the time, but not on

d

. Every theorem row counts all clean points the learner uses; the two-stage learner’s total is

O(d^{k^{\ast}/2}\log d)+O(d)+O(1)

. The empirical row counts training points only; that learner also uses a fixed set of

65{,}536

validation and calibration points. Lower bounds hold for the stated class and under the stated hypotheses. “Proved” rows are theorems of this paper unless marked as imported.

Figure 2: The covariance-matched rows of Table 1 on one scale, for

k^{\ast}=4

and the two-component mixture with

\mu=0.8

. Lines are the leading terms of the theorems, with the constants

2/\tilde{\kappa}^{2}

(full reuse) and

v_{t}/\bar{\kappa}_{t}^{2}

(one view, at

t^{\ast}

) of Theorem 6.5 ; the logarithm is normalized to one at

d=64

. The red line is the dictionary-size threshold

d^{3}/6

of the frozen barrier, whose exponent the kernel and online-SGD lower bounds share. Markers are measured 50% training-set thresholds from Section 7.2 ; they exclude the fixed

65{,}536

validation and calibration points, and cover only

d=64

–

192

Figure 3: Moment access in the covariance-matched regime (

\mu=0.8

\sigma_{0}=1

). (a) Hermite moments

e_{j}(t)

of the hidden coordinate. The Gaussian baseline matches degrees one to three, so they carry no signal (Theorem 3.4 ); diffusion multiplies degree

j

\lambda_{t}^{j}

(Theorem 3.7 ), so it never creates a lower-degree signal. (b) The signal-to-noise ratio

S_{t}

of Theorem 3.8 peaks at

t^{\ast}

, which depends only on

k^{\ast}

and

\sigma_{0}

(dotted lines). (c) The noise inflation factor

1+(\iota_{t}-1)/r

of Theorem 6.5 : one view per clean point pays at least

64/27

t^{\ast}

, and reusing each clean point under many fresh views removes the factor at every time.

Figure 4: Frozen representations. (a) Density ratio of the hidden coordinate to its Gaussian reference at

t=0.6

. In the covariance-visible regime it is bounded below by

c_{t}

(Lemma 4.3 ), which gives the barrier of Theorem 4.5 ; in the covariance-matched regime its infimum is zero, and the one-sided change of measure (Lemma 4.12 ) combined with hypercontractivity (Lemma 4.20 ) replaces the floor. (b) The staircase in the covariance-visible regime at

t=0.6

, computed from the exact constants. Solid: the best lower bound on the score risk of any orientation-blind dictionary of size

M

(maximum of Theorems 4.5 and 4.13 and the level-5 floor). Dashed: the risk attained by polynomial dictionaries of degree 1, 3 and 5 (Theorem 4.9 ). Dotted:

N_{d,3}

. Below

d^{3}/6

the lower bound approaches the Gaussian control as

d

grows (exact height); the threshold sits at

d^{3}/6

(sharp location); between consecutive thresholds the height is known only within the gap shown.

			$d=64$			$d=256$			$d=1024$
$t$	$c_{t}$	$F_{\infty}/R_{G}$	cubic	expl.	KF	cubic	expl.	KF	cubic	expl.	KF
0.1	0.030	0.03	0.03	0.00	0.00	0.03	0.00	0.00	0.03	0.00	0.00
0.3	0.284	0.15	0.14	0.00	0.00	0.15	0.00	0.51	0.15	0.23	0.89
0.6	0.615	0.31	0.28	0.09	0.33	0.31	0.90	0.92	0.31	0.98	0.98
1.0	0.833	0.51	0.46	0.72	0.74	0.50	0.97	0.97	0.51	0.99	0.99

Table 2: Lower bounds on the orientation-averaged score risk as fractions of the Gaussian control

R_{G}(t)

, for

\mu=1.5

\tau=0.35

k=1

L=3

bandwidths and width

p=1024

, so

D=3075+d+1

F_{\infty}/R_{G}

: the large-

d

limit of Theorem 4.5 . Cubic : the Ky Fan floor of Theorem 4.5 . Expl. and KF : the explicit and Ky Fan forms of Theorem 4.13 , whose large-

d

limit is

1

. Values are computed from the bounds stated in the theorems.

$t$	$c_{t}$	$\kappa_{t}$	$F_{\infty}(t)$	Gaussian control	ratio
0.1	0.0305	0.0995	0.0755	2.7069	0.028
0.3	0.2835	0.1042	0.0797	0.5241	0.152
0.6	0.6149	0.0467	0.0177	0.0565	0.313
1.0	0.8334	0.0108	0.0016	0.0032	0.510

Table 3: Exact constants for

\mu=1.5

\tau=0.35

and

k=1

. Top:

F_{\infty}

is the large-

d

limit of the explicit barrier for dictionaries of size

o(d^{3})

; the Gaussian control is the score risk of the population moment-matched Gaussian denoiser. Middle: the staircase levels of Corollary 4.10 in score-risk units, i.e. multiplied by

(a_{t}/h_{t})^{2}

; lower levels

c_{t}\kappa_{J,t}

apply below

d^{J}/J!

, upper levels

\varepsilon_{J}

are attained at

\binom{d+J}{J}

, and

\bar{\jmath}_{t}=J_{t}/\kappa_{t}

. Bottom: the reference width at which the explicit barrier vanishes with

L=3

bandwidths. Values use the exact formulas of the preceding results.

Figure 5: Smoothing, computed on simulated clean data (

d=48

n=6n_{c}

points, full reuse). (a) The reuse correlation

\widehat{C}_{\infty}(w)

of one unit on the plane spanned by the hidden direction

u

and a random

v\perp u

: it is a quartic, flat to third order at the origin, so a small random start sees no gradient toward

u

. (b) After Gaussian smoothing with

\sigma=1

, the quadratic term

6\sigma^{2}w^{\top}\widehat{H}w

, proportional to the partial-trace statistic

\bar{Y}_{n}

, creates curvature at the origin aligned with

u

(Proposition 6.7 ; the constant

3\sigma^{4}\tr\widehat{H}

is omitted). (c) Squared overlap under the normalized smoothed-gradient map from eight random starts. Without smoothing the iteration stays near the equator. With

\sigma=\infty

it is the power method on

\bar{Y}_{n}

and stops at the spectral overlap, close to

1-n_{c}/n

(Remark 6.6 ). At moderate

\sigma

the quartic term also acts once the iterate is warm, which is the sharpening step of the two-stage learner.

Figure 6: The first stage of the two-stage learner: eigenvalues of the full-reuse partial-trace matrix

\bar{Y}_{n}

for a simulated covariance-matched mixture with

\mu=0.8

d=128

u=e_{1}

and

k^{\ast}=4

. The two panels use

n=0.5n_{c}

and

n=4n_{c}

, where

n_{c}=2d^{2}/\kappa_{4}^{2}

. The dotted line is a null bulk-edge heuristic, and the dashed line in the right panel is the corresponding outlier prediction. The orange line marks the observed leading eigenvalue. Its eigenvector has little squared overlap with

u

below the transition and substantial overlap above it. Theorem 6.5 supplies a finite-sample recovery bound; these edge and outlier locations are heuristic.

Figure 7: Best-in-span score risk of orientation-blind dictionaries against the normalized dimension

(M+d+1)/N_{d,3}

(covariance-visible regime). Circles: affine functions, full lower levels and a random fraction of the cubic Hermite level; crosses: tanh random features of equal size; dashed: the explicit barrier. The curves for

d=6

–

32

collapse, and the full cubic level reaches the dimension-free level of Theorem 4.9 .

Learner	$d=64$	$d=128$	$d=192$	slope
X4 reuse SGD, width 8 (GPU)	2.68 [2.03, 3.66]	2.84 [2.29, 3.52]	2.18 [1.74, 2.71]	1.84 [1.48, 2.18]
Two-stage, full reuse, 1 sharpening step	5.96 [5.24, 7.07]	4.98 [4.18, 5.92]	6.29 [5.85, 6.73]	2.02 [1.85, 2.18]
Two-stage, full reuse, 1 step, smaller overhead †	7.70 [6.49, 8.75]	10.87 [9.53, 12.34]	15.34 [13.46, 17.42]	2.61 [2.43, 2.80]
Two-stage, full reuse, 3 sharpening steps	4.04 [3.26, 5.01]	3.68 [3.15, 4.36]	4.28 [3.87, 4.76]	2.03 [1.83, 2.24]
Two-stage, one view, 3 sharpening steps	9.17 [7.08, 11.41]	9.53 [8.01, 11.33]	10.96 [10.19, 12.09]	2.15 [1.91, 2.39]
Spectral only, full reuse	15.49 [14.43, 17.11]	12.59 [11.72, 13.47]	14.97 [13.46, 16.53]	1.94 [1.81, 2.06]

Table 4: Fifty-percent training-set thresholds

n_{50}/d^{2}

with 95% cell-bootstrap intervals, and slopes of

\log n_{50}

\log d

with intervals, all on the frozen X4 protocol: eight seeds, gain threshold

0.25

. Unless marked

\dagger

, every learner also uses

32{,}768

validation and

32{,}768

calibration clean points, which the thresholds do not include; the overhead rows give these fixed costs in the same units, to be added for the total clean-point budget. The

\dagger

row uses

8{,}192+8{,}192

points. GPU rows are the frozen X4 study; the other rows are the explicit learners of Section 6.5 with full reuse or one view and with or without the sharpening step. Reference transitions:

n_{c}/d^{2}=2.98

(full reuse) and

7.06

(one view).

Figure 8: Covariance-matched regime,

k^{\ast}=4

. (a) Squared overlap of the leading eigenvector of the clean partial trace against

n/n_{c}

with

n_{c}=2d^{2}/\kappa_{4}^{2}

: means of eight seeds with

\pm 2

standard errors; the curve is the heuristic prediction ( 6.1 ). (b) Training-set thresholds

n_{50}/d^{2}

of Table 4 with 95% cell-bootstrap intervals; horizontal lines mark the reuse and single-view transitions. (c) Success fractions against

n/d^{2}

(training points) at

d=192

Ratio of training-set thresholds	$d=64$	$d=128$	$d=192$
X4 network / two-stage (1 step)	0.45 [0.32, 0.64]	0.57 [0.44, 0.74]	0.35 [0.27, 0.45]
X4 network / two-stage (3 steps)	0.66 [0.47, 0.95]	0.77 [0.60, 1.00]	0.51 [0.39, 0.64]
one view / full reuse, two-stage (3 steps)	2.27 [1.68, 3.07]	2.59 [2.08, 3.22]	2.56 [2.22, 2.92]
one view / full reuse, spectral only	2.13 [1.87, 2.46]	2.82 [2.54, 3.25]	2.37 [2.13, 2.83]
smaller / standard overhead, two-stage (1 step)	1.29 [1.04, 1.59]	2.18 [1.75, 2.71]	2.44 [2.00, 2.77]

Table 5: Contrasts between training-set thresholds: ratios of

n_{50}

with 95% intervals from independent cell-bootstrap draws (unpaired). Learners in each ratio share the overhead, except in the last row, which compares the one-step learner with

8{,}192+8{,}192

and with

32{,}768+32{,}768

validation and calibration points.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Result	Content	Page	Proof	Main tools
Appendix A : moment calculus (Section 3 )
Thm 3.1	Stein–Tweedie identity	3.1	A.1 , p. A.1	Gaussian integration by parts; Tweedie’s formula
Thm 3.2	polynomial fields $\Leftrightarrow$ moments	3.2	A.2 , p. A.2	Ornstein–Uhlenbeck generator in the Hermite basis
Lem 3.3	conditional Stein identity	3.3	A.3 , p. A.3	Stein’s lemma at fixed $x_{0}$ ; Mehler’s formula; Hermite eigen-equation
Thm 3.4	baseline hides matched moments	3.4	A.4 , p. A.4	binomial expansion of moment tensors; Theorem 3.2
Cor 3.5	exact polynomial barrier	3.5	A.4 , p. A.4	tower property; Theorem 3.4

Appendix

Table 14

Symbol	Meaning	Defined
$a_{t},\;h_{t}$	$e^{-t}$ and $1-e^{-2t}$ , signal and noise variances of the forward process	§ 2.1
$X_{0},\;X_{t},\;P_{t},\;p_{t}$	clean point, noisy view $a_{t}X_{0}+\sqrt{h_{t}}G$ , law and density of $X_{t}$	§ 2.1
$m_{t}$	denoiser $\mathbb{E}[X_{0}\mid X_{t}=\cdot]$	§ 2.1
$s_{t},\;\widehat{s}$	score $\nabla\log p_{t}$ and a score estimate $(a_{t}\widehat{m}-x)/h_{t}$ . In App. B and Lemma 4.2 , $s$ without index is the mixture-component scale $(a_{t}^{2}\tau^{2}+h_{t})^{1/2}$	§ 2.1
$\bar{\sigma}_{t}$	coordinate scale $(a_{t}^{2}\sigma_{0}^{2}+h_{t})^{1/2}$ of $X_{t}$ in the covariance-matched regime ( $\bar{\sigma}$ when $t$ is fixed)	Ass. 3
$Q_{0},\;Q_{t},\;q_{t},\;m^{Q}_{t}$	baseline law, its diffused law and density, its denoiser	§ 2.1

Appendix

Table 15

$d$	$p$	Gaussian	ambient RF	Ky Fan floor	PCA RF
8	64	0.5248	0.2887 (0.022)	0.01083	0.003556 (0.00047)
8	256	0.5248	0.1334 (0.0064)	0.003055	0.002977 (0.00058)
8	1024	0.5248	0.09092 (0.0039)	0.001052	0.003348 (0.00053)
16	64	0.5257	0.4923 (0.0095)	0.05666	0.003046 (0.00068)
16	256	0.5257	0.3458 (0.015)	0.01845	0.002585 (0.0005)
16	1024	0.5257	0.2279 (0.0066)	0.01478	0.002971 (0.00077)

Appendix

Table 8: Score risk at

t=0.3

k=1

: mean (standard error) over replicates, and the Ky Fan floor for the ambient dictionary.

$d$	$t$	$p_{1/2}$	$p_{N}$	half-gap threshold	largest floor
6	0.3	64	14	0.262	0.0097
8	0.3	64	34	0.262	0.0381
12	0.3	256	112	0.263	0.0589
16	0.3	512	260	0.263	0.0662
24	0.3	2048	850	0.261	0.0718
32	0.3	4096	1972	0.263	0.0741

Appendix

Table 9: Half-gap width

p_{1/2}

of the ambient estimator, reference width

p_{N}

, the half-gap risk threshold and the largest Ky Fan floor over the widths tested.

Figure 9: Ambient risk normalized by

B_{d}(t)

against

L(p+1)/N_{d,3}

; the dashed line is the explicit barrier.

$k$	$n$	$t=0.1$	$t=0.3$	$t=0.6$	$t=1.0$
1	64	–/–	24/29	–/–	–/–
1	128	–/–	18/19	21/23	–/–
1	256	35/91	18/18	18/19	–/–
1	512	20/20	17/17	17/18	25/31
1	1024	18/19	17/17	17/17	20/20
1	2048	17/17	17/17	17/17	17/18

Appendix

Table 10: Certified

d_{0}

p=256

, as per-cell / simultaneous: for every

d\geq d_{0}

the explicit floor of the ambient estimator exceeds the upper confidence limit of the PCA risk (Lemma 4.17 ). Per-cell limits are one-sided normal

99\%

limits; simultaneous limits are Bonferroni–

t

over all cells. Both are approximate. A dash means no certificate.

latent	Gaussian	ambient RF	PCA RF	KDE	exact score	fresh
four, $k=2$	0.473 (0.08)	0.473 (0.08)	0.178 (0.03)	0.176 (0.02)	0.125 (0.02)	0.127
product, $k=1$	0.662 (0.007)	0.662 (0.007)	0.179 (0.04)	0.218 (0.02)	0.112 (0.03)	0.145
product, $k=2$	0.929 (0.006)	0.929 (0.006)	0.31 (0.02)	0.385 (0.01)	0.279 (0.05)	0.266

Appendix

Table 11: Latent

W_{2}

, mean (standard error), at

d=512

and

n=1024

; the last column is the fresh-versus-fresh reference value.

Figure 10: Latent

W_{2}

(mean and standard error over replicates) against

d

$t$	$J$	$d$	dimension	Hermite	tanh	$k(a/h)^{2}\varepsilon_{J}$
0.3	3	6	63	0.186	0.211	0.189
0.3	3	8	129	0.189	0.152	0.189
0.3	3	12	377	0.189	0.147	0.189
0.3	3	16	833	0.193	0.163	0.189
0.3	3	24	2625	0.192	0.193	0.189
0.3	3	32	6017	0.196	0.19	0.189

Appendix

Table 12: Best-in-span score risk (mean over replicates) of the full Hermite dictionary at level

J

(dimension

\binom{d+J}{J}

) and of tanh random features of the same dimension, against the dimension-free level

k(a/h)^{2}\varepsilon_{J}

of Theorem 4.9 .

Figure 11: Best-in-span score risk of orientation-blind dictionaries against the normalized dimension

(M+d+1)/N_{d,3}

. Circles: affine functions, full lower Hermite levels and a uniformly random subspace of the next level; crosses: tanh random features of the same dimension. Horizontal lines: the Gaussian control and the dimension-free levels of Theorem 4.9 ; the vertical line marks

N_{d,3}

$\sigma_{0}$	$d$	$n$	$\sin^{2}\Theta$	reference	Gaussian	ambient	barrier	PCA
0.1	8	256	8.6e-05	0.00013	0.513	0.169	0.00298	0.00855
0.1	128	256	0.0019	0.0021	0.539	0.54	0.077	0.0367
0.1	512	256	0.0083	0.0084	0.619	0.622	0.0781	0.116
0.1	8	4096	5.2e-06	8.3e-06	0.508	0.12	0.00298	0.000892
0.1	128	4096	0.00014	0.00013	0.512	0.512	0.077	0.00251
0.1	512	4096	0.00054	0.00053	0.515	0.515	0.0781	0.00736

Appendix

Table 13: Data

UZ+\sigma_{0}G_{0}

k=1

p=256

t=0.3

: subspace error

\left\lVert\sin\Theta\right\rVert_{F}^{2}

(measured / spiked-covariance reference) and score risks (mean). The barrier column is

\lambda_{t}^{-2}F^{\mathrm{KF}}_{d}(L(p+1);t^{\prime})

Figure 12: Score risk at

t=0.3

against

d

for data off the subspace. PCA random features for three training sizes, ambient random features at

n=1024

and the time-changed barrier for the ambient dictionary.

$\sigma_{0}$	$d$	Gaussian	ambient RF	PCA RF	KDE	exact score	fresh
0.0	32	0.67	0.402	0.267	0.3	0.131	0.127
0.0	512	0.653	0.653	0.326	0.284	0.297	0.16
0.1	32	0.66	0.407	0.263	0.291	0.124	0.122
0.1	512	0.643	0.642	0.302	0.266	0.286	0.156
0.3	32	0.593	0.449	0.202	0.244	0.0907	0.0938
0.3	512	0.574	0.573	0.223	0.218	0.225	0.126

Appendix

Table 14: Latent

W_{2}

of generated samples (mean over replicates) for data off the subspace,

k=1

n=1024

; the reference law is that of

U^{\top}X_{0}^{\sigma}=Z+\sigma_{0}\xi

Score-Based Generative Modeling

Score-based generative modeling learns data-distribution gradients to generate samples through reverse diffusion, flows, and related stochastic dynamics.

See all 120 papers

Mar 24, 2026

Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data

We study the theoretical behavior of denoising score matching--the learning task associated to diffusion models--when the data distribution is supported on a low-dimensional manifold and the score is parameterized using a random feature neural network. We derive asymptotically exact expressions for the test, train, and score errors in the high-dimensional limit. Our analysis reveals that, for linear manifolds the sample complexity required to learn the score function scales linearly with the intrinsic dimension of the manifold, rather than with the ambient dimension. Perhaps surprisingly, the benefits of low-dimensional structure starts to diminish once we have a non-linear manifold. These results indicate that diffusion models can benefit from structured data; however, the dependence on the specific type of structure is subtle and intricate.

Anand Jerry George, Nicolas Macris

École Polytechnique Fédérale de Lausanne (EPFL), Lab for Statistical Mechanics of Inference in Large Systems (SMILS), CH-1015 Lausanne, Switzerland

Intrinsic Dimensionality Random Feature Methods Learning Curves Score-Based Generative Modeling Statistical Learning Theory Diffusion Models

Ask PDF

Oct 14, 2026

Denoising Score Matching in Diffusion Models: The Effect of Rank Constraints on Generalization

Architectural biases restrict the class of functions that a diffusion score network can represent, but their effect on generalization remains poorly understood. We study denoising score matching with score networks having a rank constrained non-linear term and a linear skip connection. Numerical experiments with trainable three-layer networks on data supported on an embedded sphere show that increasing rank lowers training error, but its effect on test error depends on sample size and data dimension. To explain this behavior, we analyze a random-feature score model under rank constraint with Gaussian data supported on a low-dimensional subspace. We characterize the optimal rank-constrained estimator through a truncated singular value decomposition and derive asymptotically precise limiting training and test errors when sample size

(n)

, feature width

(p)

, data dimension

(D)

, and rank

(r)

scale proportionally with the ambient dimension

(d)

. The resulting spectral characterization distinguishes eigenmodes that improve the empirical fit from those that improve population accuracy. It also identifies the dimension of the orthogonal complement of the data subspace as a natural rank threshold. The theoretical learning curves qualitatively reproduce the observed rank-dependent overfitting, providing a tractable account of how bottlenecks or rank constraints can regularize score estimation and why their effectiveness depends on data dimension and sample size.

Anand Jerry George, Nicolas Macris

École Polytechnique Fédérale de Lausanne (EPFL), Lab for Statistical Mechanics of Inference in Large Systems (SMILS), CH-1015 Lausanne, Switzerland

Score-Based Generative Modeling Neural Network Generalization

Ask PDF

Mar 13, 2026

A theory of learning data statistics in diffusion models, from easy to hard

While diffusion models have emerged as a powerful class of generative models, their learning dynamics remain poorly understood. We address this issue first by empirically showing that standard diffusion models trained on natural images exhibit a distributional simplicity bias, learning simple, pair-wise input statistics before specializing to higher-order correlations. We reproduce this behaviour in simple denoisers trained on a minimal data model, the mixed cumulant model, where we precisely control both pair-wise and higher-order correlations of the inputs. We identify a scalar invariant of the model that governs the sample complexity of learning pair-wise and higher-order correlations that we call the diffusion information exponent, in analogy to related invariants in different learning paradigms. Using this invariant, we prove that the denoiser learns simple, pair-wise statistics of the inputs at linear sample complexity, while more complex higher-order statistics, such as the fourth cumulant, require at least cubic sample complexity. We also prove that the sample complexity of learning the fourth cumulant is linear if pair-wise and higher-order statistics share a correlated latent structure. Our work describes a key mechanism for how diffusion models can learn distributions of increasing complexity.

Lorenzo Bardone, Claudia Merger, Sebastian Goldt

Statistical Physics of Computation Laboratory, EPFL, Lausanne, Switzerland · Theoretical and Scientific Data Science Group, SISSA, Trieste, Italy

Diffusion Models Statistical Learning Theory

Ask PDF

Abstract

Score-Based Generative Modeling

Explore similar work

Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data

Denoising Score Matching in Diffusion Models: The Effect of Rank Constraints on Generalization

A theory of learning data statistics in diffusion models, from easy to hard