Generalization, memorization, and overfitting for diffusion models trained in the lazy high-dimensional regime

Hugo Latourelle-Vigeant; Sinho Chewi; Aram-Alexandre Pooladian; John Sous; Theodor Misiakiewicz

stat.MLAug 25, 2026

Generalization, memorization, and overfitting for diffusion models trained in the lazy high-dimensional regime

Authors: Hugo Latourelle-Vigeant, Sinho Chewi, Aram-Alexandre Pooladian, John Sous, Theodor Misiakiewicz

Organizations: Department of Statistics and Data Science, Yale University · Institute for Foundations of Data Science, Yale University · Department of Applied Physics, Yale University

Abstract

Modern score-based generative models have achieved remarkable empirical success in high-dimensional tasks such as image, audio, and video synthesis. These models reduce distribution learning to a sequence of regression problems that, if solved exactly on finite data, would ultimately reproduce the training samples. Their ability to generalize must therefore arise from the implicit or explicit regularization during training. In this work, we develop a generative counterpart to the theory of benign overfitting and algorithmic regularization for overparameterized neural networks in the supervised lazy-training regime. We study denoising score matching in a vector-valued reproducing kernel Hilbert space with an inner-product kernel. In the proportional high-dimensional regime $n\asymp d$ , we derive exact risk trajectories under gradient flow training. These trajectories exhibit three phases governed by qualitatively distinct estimators: a spectral estimator that generalizes, a pure-noise score with localized peaks that interpolate the training objective, and an empirical Bayes estimator that memorizes the data. We then analyze how these estimators combine along the reverse-time SDE and characterize the distribution of the resulting samples. The analysis reveals familiar mechanisms from supervised learning, including kernel linearization and self-induced regularization from the nonlinear part of the kernel, but also reveals a distinct phenomenology specific to generative modeling.

Figures & tables

Panel of Figure 1 — Figure 1: Stylized evolution of the empirical denoising loss ${\mathcal{L}}(\widehat{\bm{f}}_{\mathsf{t}};0)$ and population risk $\mathcal{R}(\widehat{\bm{f}}_{\mathsf{t}})$ along gradient flow at fixed noise level $(\rho,\sigma)$ , with $\tau_{\bm{\Sigma}}={\rm Tr}({\bm{\Sigma}})/d$ . As training time increases, the score estimator transitions from a spectral shrinkage estimator at $\mathsf{t}=O(d)$ (Theorem 1 ), to fitting the pure-noise score at polynomial times $\mathsf{t}=d^{1+\Theta(1)}$ , with localized bumps that drive the training loss to zero while remaining invisible on typical test inputs (Theorem 2 ), and finally to the empirical Bayes denoiser which memorizes the training samples at super-polynomial times (Theorem 3 ). In this figure, generalization means that the train and test objectives are close.

Figure 2: Training and test errors as functions of the normalized gradient-flow time

\mathsf{t}/d

for

n=240

, diffusion time

t=0.2

, and Gaussian data

\pi_{0}=\mathcal{N}(0,\mathbf{I}_{d})

. Dashed and solid lines correspond, respectively, to the training and test errors of the finite-dimensional linearized estimator obtained by solving ( 39 ) with gradient flow. Open and filled circles show the corresponding deterministic equivalents from Theorem 1 . The dotted horizontal line indicates the asymptotic test-error plateau

\tfrac{\rho^{2}}{\sigma^{2}}\frac{{\rm Tr}({\bm{\Sigma}})}{d}

. Left: We use

h(x)=x+x^{4}/12

and vary

d/n

while holding

n

, and hence

\delta_{n}

, fixed. This changes the relaxation rates

\gamma_{-}

, shifting the onset of overfitting. Right: We fix

d=120

and vary the effective linearization parameter

\delta_{n}

. When

\delta_{n}=0

, the estimator is linear and cannot interpolate the training loss. By contrast, whenever

\delta_{n}>0

, the training error converges to zero and the test error approaches the dotted interpolation plateau. Because

\gamma_{-}

increases with

\delta_{n}

, larger

\delta_{n}

leads to an earlier transition to overfitting.

Figure 3: Recovery of two covariance scales along the linearized reverse process for

d=720

n=800

T=3

t_{0}=10^{-3}

h(x)=x+0.005x^{5}

and

\pi_{0}=\mathcal{N}(0,{\bm{\Sigma}})

where

{\bm{\Sigma}}={\rm diag}(\frac{17}{12}\mathbf{I}_{504},\frac{1}{36}\mathbf{I}_{216})

has condition number

51

and

\tau_{\bm{\Sigma}}=1

. Left: A schedule

t\mapsto\mathsf{t}_{t}

specifies the gradient-flow training time in ( 26 ) at every diffusion time

t

. We consider the pointwise DSM-optimal schedule

\mathsf{t}_{t}^{\star}\in\arg\min_{\mathsf{t}/d\in[10^{-1},10^{2}]}\mathcal{R}_{\mathrm{det},t}^{(\mathsf{t})}

. We compare it with the constant schedule

\mathsf{t}_{t}^{\mathrm{const}}=\mathsf{t}_{t_{0}}^{\star}

and an exponential schedule

\mathsf{t}_{t}^{\mathrm{exp}}

interpolating between

\mathsf{t}_{T}^{\star}

and

\mathsf{t}_{t_{0}}^{\star}

on a logarithmic diffusion-time scale. Middle: Solid and dashed lines show, respectively, the average projected covariances

{\rm Tr}({\mathsf{P}}_{\pm}\widetilde{{\bm{\Sigma}}}_{t})/\operatorname{rank}({\mathsf{P}}_{\pm})

along the dynamics in ( 56 ), where

{\mathsf{P}}_{+}

and

{\mathsf{P}}_{-}

denote the projections onto the large- and small-eigenvalue eigenspaces of

{\bm{\Sigma}}

, respectively. Filled and open circles show the corresponding deterministic predictions from Theorem 5 , while gray dashed lines mark the population eigenvalues. Upper right: Solid curves show the conditional normalized Gaussian divergence

\KL(\mathcal{N}(0,{\bm{\Sigma}})\,\|\,\mathcal{N}(0,\widetilde{{\bm{\Sigma}}}_{t}))/d

. Markers show a natural spectral plug-in prediction using ( 58 ). Lower right: Solid curves show the cumulative population score MSE

\int_{t}^{T}(\mathcal{R}_{s}(\widehat{\bm{f}}^{\mathrm{lin}}_{\mathsf{t}_{s}})-\mathcal{R}_{s}^{\star})/\sigma_{s}^{2}\textup{d}s

, where

\widehat{\bm{f}}^{\mathrm{lin}}_{\mathsf{t}_{s}}

is the linearized estimator from Lemma 30 and

\mathcal{R}^{\star}_{s}

is the Bayes population DSM risk at diffusion time

s

. Markers show the deterministic prediction from Theorem 1 .

Generative Modeling

Generative modeling learns data distributions to create new samples, from images and text to molecules, simulations, and structured records.

See all 492 papers

Benign Overfitting Does Not Occur in Diffusion Models
Jul 2, 2026Tyler Farghly, Benjamin Dupuis, Alain Durmus +1Double Descent Neural Network Generalization
1INRIA - École Normale Supérieure - PSL Research University - CNRS, France · 2 École Polytechnique - CMAP - IP Paris, France
Double Descent and Malign Overfitting in Diffusion Models
Sep 22, 2026Raphaël Urfin, Tony Bonnaire, Giulio Biroli +1Double Descent Benign Overfitting
Laboratoire de Physique de l’École normale supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité, F-75005 Paris, France · Université Paris-Saclay, CNRS, Institut d’Astrophysique Spatiale, 91405 Orsay, France · Department of Computing Sciences, Bocconi University, Milano, Italy
First Learn, Then Memorize: The Spectral Bias of Diffusion Models
Sep 28, 2026Raphaël Urfin, Tony Bonnaire, Giulio Biroli +1Neural Network Generalization Memorization in Generative Models
Laboratoire de Physique de l’École normale supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité, F-75005 Paris, France · Université Paris-Saclay, CNRS, Institut d’Astrophysique Spatiale, 91405 Orsay, France · Department of Computing Sciences, Bocconi University, Milano, Italy

Jul 2, 2026J/K move · Enter open · S save

Benign Overfitting Does Not Occur in Diffusion Models

Tyler Farghly, Benjamin Dupuis, Alain Durmus, Umut Simsekli

1INRIA - École Normale Supérieure - PSL Research University - CNRS, France · 2 École Polytechnique - CMAP - IP Paris, France

Double Descent Neural Network Generalization Benign Overfitting Diffusion Models

Open paper AskPDF arXiv

Benign overfitting and double descent have come to shape our understanding of generalization in deep learning, establishing that overfitting is not only compatible with good generalization but can actively benefit it. Diffusion models share much of the machinery of standard deep learning, so it is natural to assume that they also exhibit these properties. In this work, we show that this assumption is largely incorrect. We first establish fundamental impossibility results showing that, unless the sample size grows exponentially with the data dimension, overfitting and good generalization cannot occur simultaneously. Consequently, the population loss follows a classical U-shaped curve in model complexity rather than exhibiting double descent. Analyzing a simplified setting, we identify a key difference between regression and score matching: regression benefits from an alignment between the target and the empirical covariance; score matching admits no such alignment, leaving overfitting irreparably harmful. We further identify implicit regularization stemming from time-smoothness of the score and early stopping during training as mechanisms that prevent such overfitting and verify our findings with high-dimensional image generation experiments. Our results reveal that generalization in diffusion models is governed by mechanisms distinct from those of traditional regression, motivating the development of new theory.

Abstract

Generative Modeling

Explore similar work

Benign Overfitting Does Not Occur in Diffusion Models

Double Descent and Malign Overfitting in Diffusion Models

First Learn, Then Memorize: The Spectral Bias of Diffusion Models