First Learn, Then Memorize: The Spectral Bias of Diffusion Models

Raphaël Urfin; Tony Bonnaire; Giulio Biroli; Marc Mézard

cs.LGSep 28, 2026

First Learn, Then Memorize: The Spectral Bias of Diffusion Models

Authors: Raphaël Urfin, Tony Bonnaire, Giulio Biroli, Marc Mézard

Organizations: Laboratoire de Physique de l’École normale supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité, F-75005 Paris, France · Université Paris-Saclay, CNRS, Institut d’Astrophysique Spatiale, 91405 Orsay, France · Department of Computing Sciences, Bocconi University, Milano, Italy

Abstract

Diffusion models trained on a finite dataset first learn to generate novel, high-quality samples and only much later collapse onto their training set. We identify the mechanism behind this separation of timescales and the object that probes it. The training dynamics of the score function are governed---exactly, and at any width---by the Gram matrix of the Neural Tangent Kernel (NTK) evaluated on the noisy training data, so the timescales of generalization and of memorization must be encoded in its spectrum. We show that they are, and that the structure responsible has no analogue in standard kernel settings. The use of multiple noise realizations per sample ( $m$ noised copies at a fixed noise level) in the score-matching loss is what restructures the Gram matrix spectrum into two distinct parts. The first, of large eigenvalues, carries the global features of the target distribution and is present already for $m=1$ . The second, which the repeated noising creates, consists of the smallest eigenvalues and is supported on eigenvectors aligned with the sample-specific noise directions; it sets a memorization timescale parametrically larger in the training set size $n$ . We establish this picture on two fronts. Analytically, we solve the spectrum in the lazy high-dimensional limit for both linear ( $n \asymp d$ ) and polynomial ( $n \asymp d^k$ ) sample complexities, and prove through a bias--variance decomposition that the first bulk minimizes the approximation error while the second drives the error associated with memorization. Empirically, we show the same two-bulk structure in Convolutional NTKs on CelebA and in finite-width U-Nets trained well beyond the lazy regime, and we make the link causal: truncating the Gram matrix at rank $r$ tunes the generalization--memorization transition, and an $L_2$ penalty targeting the second bulk suppresses memorization in feature-learning U-Nets.

Figures & tables

Figure 1 : Gram matrix spectrum. Eigenvalues $\lambda_{r}$ in decreasing order against their rank. ( Left ) Sketch of the spectrum predicted by Theorem 2.3 for the Gram matrix $\bm{\mathrm{G}}^{\nu\alpha,\mu\beta}=f(\bm{\mathrm{Y}}^{\nu\alpha\top}\bm{\mathrm{Y}}^{\mu\beta}/d)$ . ( Middle ) Spectrum of $\bm{\mathrm{G}}$ with isotropic $\bm{\Sigma}=\sigma^{2}\bm{\mathrm{I}}_{d}$ , $t=0.1$ , $m=4$ , $\psi_{n}=4$ and $f$ the NTK of a two-layer ReLU network (Appendix A.8 ) in the linear regime at several values of $d$ ; the dashed black curve is the linear equivalent $\bm{\mathrm{G}}_{\mathrm{lin}}$ of ( 6 ) at $d=512$ . ( Right ) Quadratic regime $n=2d^{2}$ at $d=64$ ; the dashed black curve is the polynomial equivalent of Theorem 2.3 . Details in Appendix D.7 .

Figure 2 : Bias–variance decomposition.

\mathcal{B}^{2}

\mathcal{V}

and

\mathcal{B}^{2}+\mathcal{V}

for the kernel

f(u)=\mathbb{E}[\tanh(z_{1})\tanh(z_{2})]

, with

(z_{1},z_{2})

unit-variance Gaussians of correlation

u

(Appendix A.8 ), at

\sigma^{2}=1

m=4

and

t=0.1

. ( Left ) Vs. the rescaled ridge

\tilde{\gamma}

\psi_{n}=8

, axis reversed to show the correspondence with training time; the inset shows the same vs. the training time

\tau

of ( 5 ). Solid lines: Theorem 2.2 ; markers: experiments at

d=256

; in the inset, markers and dotted line are the finite-

d

gradient flow. ( Middle )

\mathcal{B}^{2}

(top) and

\mathcal{V}

(bottom) vs.

\psi_{n}

, for the optimal ridge

\tilde{\gamma}^{*}=\arg\min_{\tilde{\gamma}}\mathcal{L}_{\mathrm{test}}

(solid) and the ridgeless limit

\tilde{\gamma}\to 0^{+}

(dashed). ( Right )

\mathcal{B}^{2}

vs.

\tilde{\gamma}

for several

\psi_{n}

, dots marking

\tilde{\gamma}^{*}

; the inset shows the same vs.

\psi_{n}\tilde{\gamma}

. Details in Appendix D.10 .

Figure 3 : Structure of the generalization and memorization bulks in the CNTK spectrum on CelebA. (Left) Ordered eigenvalues for several $n$ at $m=4$ . The dotted lines indicate $r=n$ . Inset: Same, rescaled by $n$ . (Right) Eigenvalue histogram with counts rescaled by $n$ . Inset: collapse of the top eigenvalues under rescaling by $n$ . $s=0.05$ , averaged over 10 training sets.

Figure 4 : Spectrally truncated kernel regression and generalization–memorization transition. (Left) FID (solid) and $f_{\mathrm{mem}}$ (dashed) vs. truncation rank $r$ for several $n$ . Inset: $f_{\mathrm{mem}}$ vs. $r/n$ . (Middle) Time-averaged $L_{2}$ training (solid) and test (dashed) losses; the inset shows the same curves against $r/n$ . (Right) Top row: training images. Rows 2–6: generated samples ( $n=3072$ ) starting from a fixed training noise associated with the top-row clean image at $r\in\{2048,\,4096,\,8192,\,16384,\,N\!=\!nm\}$ .

Figure 5 : Training-time-dependent empirical Gram eigenspectrum of a U-Net. Eigenvalues of the empirical Gram matrix computed for several training times

\tau

s=0.23

for (Left) the unregularized dynamics, and (Right) the

L_{2}

-regularized dynamics.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6 : Empirical histogram of the eigenvalues of

\bm{\mathrm{G}}_{\mathrm{lin}}

for

d=100

(blue) averaged over 10 runs and the analytical prediction obtained by solving the equations on the Stieltjes transform for

m=5,\psi_{n}=4.0

and

t=0.1

for

\rho_{\bm{\Sigma}}(\lambda)=\frac{1}{2}\delta(\lambda-1)+\frac{1}{2}\delta(\lambda-0.1)

and for the kernel parameters

\mu_{1}=1,\mu_{I}=0.1,\mu_{B}=0.2

. The left and right panels corresponds to different ranges of

\lambda

Figure 7 : Comparison between the linear equivalent and the non-linear NTK spectrum in finite dimension for

d=128,\ \psi_{n}=4,\ m=4,

and

t=0.1

Figure 8 : Analytical solutions of the rescaled bulks

\rho_{1},\rho_{2}

for different values of

\psi_{n}

for

m=5,\ t=0.1,\bm{\Sigma}=\bm{\mathrm{I}}_{d},\ \mu_{1}=1.0,\ \mu_{B}=0.2,\ \mu_{I}=0.1,\ \mu_{0}=0.0

and the asymptotic limit

\delta(\lambda-1)

Figure 9 : Bias and variance against the sample complexity.

\mathcal{B}^{2}

\mathcal{V}

and

\mathcal{B}^{2}+\mathcal{V}

against

m\psi_{n}=mn/d

for the ridgeless estimator

\tilde{\gamma}\to 0^{+}

, at

t=0.1

\sigma^{2}=1

, with the kernel of Figure 2 , for

m=4

( left ) and

m=1

( right ). Solid lines are the equations of Theorem 2.2 ; markers are experiments at

d=256

Figure 10 : The ridge as a proxy for early stopping.

\mathcal{L}_{\mathrm{test}}

\mathcal{B}^{2}

and

\mathcal{V}

against the training time

\tau

, at

m=1

( top ) and

m=4

( bottom ). Black: the theory of Theorem 2.2 evaluated at

\tilde{\gamma}=1/\tau

. Green: kernel gradient flow of ( 5 ) measured at finite

d=256

\psi_{n}=8

, averaged over

16

(

m=1

) and

8

(

m=4

) realizations of the dataset and

512

test points. Dotted verticals are

\tau_{\mathrm{gen}}

and

\tau_{\mathrm{mem}}

. Panels are scaled independently: at

m=1

the bias vanishes whereas at

m=4

it saturates at

\Theta(1)

Figure 11 : Scalings with

t

\mathcal{B}^{2}

( top ) and

\mathcal{V}

( bottom ) against

t

\psi_{n}=10^{3}

and

m=4

, where

1/(m\psi_{n})\ll t\ll 1

, for the optimally ridged estimator

\tilde{\gamma}^{*}

(solid) and the ridgeless one

\tilde{\gamma}\to 0^{+}

(dashed), with the kernel of Fig. 2 . Lines are the equations of Theorem 2.2 .

Figure 12 : Ordered eigenvalues for several $m$ at fixed $n=256$ and their rescaling by $m$ in the inset. Spectra obtained at $s=0.05$ and averaged over 10 realizations of the training set.

Figure 13 : Training and test losses along training for the (Left) unregularized and (Right) regularized dynamics of Fig. 5 .

Explore similar work

Smoothing the Score Function to Enhance Generalization in Diffusion Models
Date pendingXinyu Zhou, Jiawei Zhang, Stephen J. WrightScore-Based Diffusion Model Gumbel-Softmax Relaxation
Department of Computer Sciences, University of Wisconsin Madison, WI, USA.
Memorisation, convergence and generalisation in generative models
May 20, 2026Antoine Maillard, Sebastian GoldtGenerative Models Generalisation
1INRIA Paris & DI ENS, PSL University, Paris, France · 2International School of Advanced Studies (SISSA), Trieste, Italy
Understanding diffusion models requires rethinking (again) generalization
May 7, 2026Pierre Marion, Yu-Han WuOverfitting Diffusion Models
Inria, École Normale Supérieure, PSL Research University · LPSM, Sorbonne University & Google DeepMind

Date pending, cs.LGJ/K move · Enter open · S save

Smoothing the Score Function to Enhance Generalization in Diffusion Models

Xinyu Zhou, Jiawei Zhang, Stephen J. Wright

Department of Computer Sciences, University of Wisconsin Madison, WI, USA.

Score-Based Diffusion Model Gumbel-Softmax Relaxation Uncertainty Score

Open paper AskPDF arXiv

Diffusion models achieve remarkable generation quality, yet face a fundamental challenge known as memorization, where generated samples can replicate training samples exactly. We develop a theoretical framework to explain this phenomenon by showing that the empirical score function (the score function corresponding to the empirical distribution) is a weighted sum of the score functions of Gaussian distributions, in which the weights are sharp softmax functions. This structure causes individual training samples to dominate the score function, resulting in sampling collapse. In practice, approximating the empirical score function with a neural network can partially alleviate this issue and improve generalization. Our theoretical framework explains why: In training, the neural network learns a smoother approximation of the weighted sum, allowing the sampling process to be influenced by local manifolds rather than single points. Leveraging this insight, we propose two novel methods to further enhance generalization: (1) Noise Unconditioning enables each training sample to adaptively determine its score function weight to increase the effect of more training samples, thereby preventing single-point dominance and mitigating collapse. (2) Temperature Smoothing introduces an explicit parameter to control the smoothness. By increasing the temperature in the softmax weights, we naturally reduce the dominance of any single training sample and mitigate memorization. Experiments across multiple datasets validate our theoretical analysis and demonstrate the effectiveness of the proposed methods in improving generalization while maintaining high generation quality.

Abstract

Explore similar work

Smoothing the Score Function to Enhance Generalization in Diffusion Models

Memorisation, convergence and generalisation in generative models

Understanding diffusion models requires rethinking (again) generalization