cs.LGNov 2, 2026

The role of injected noise in diffusion model generalization

Authors: Vincent Emmerling, Felix Günther, David Kappel

Organizations: CITEC, Bielefeld University Bielefeld, Germany

Abstract

Diffusion models trained on distinct datasets can produce highly similar samples when prompted with the same noise input, suggesting that they learn a generalized data distribution. We investigate the role of noise injected during the sampling process, and show that it plays an important role in making or breaking this strong generalization effect. Based on this finding we investigate the robustness of sample similarity to changes in the injected noise, and find that its impact is largest in the sampling regime where broad structure already exists but the precise identity of the sample is not yet determined.

Figures & tables

Explore similar work

Mar 12, 2026

Beyond Sample Copying: Structural Memorization in Diffusion Models

Diffusion models generalize well in practice. Paradoxically, an optimal diffusion model fully memorizes the training data and therefore fails to generalize, raising the question of what induces generalization in a real diffusion model. We show that diffusion models progressively overfit the denoising training objective, creating a generalization gap between validation and training performance at intermediate noise levels. In a fully analytic 2D toy model with a controlled denoising error, we trace this gap to the interaction between model error and the density of the data distribution's support. The optimal denoising flow field localizes sharply around individual training points, whereas model error suppresses exact recall of training points, yielding a smooth, generalizing flow field. Finally, we examine how training-time overfitting manifests along inference trajectories. We find that predictions made from intermediate trajectory states occupy a distinct feature-space regime from predictions made from noised training and validation images. As training progresses and model size increases, these predictions develop greater relative affinity to the training data, despite the absolute similarity to the validation data not decreasing. Together, these findings show that the denoising objective and inference trajectories express structural memorization differently.
Mar 24, 2026

Manifold Generalization Provably Proceeds Memorization in Diffusion Models

Diffusion models often generate novel samples even when the learned score is only coarse -- a phenomenon not accounted for by the standard view of diffusion training as density estimation. In this paper, we show that, under the manifold hypothesis, this behavior can instead be explained by coarse scores capturing the geometry of the data while discarding the fine-scale distributional structure of the population measure~μdataμ_{\scriptscriptstyle\mathrm{data}}. Concretely, whereas estimating the full data distribution μdataμ_{\scriptscriptstyle\mathrm{data}} supported on a kk-dimensional manifold is known to require the classical minimax rate O~(N−1/k)\tilde{\mathcal{O}}(N^{-1/k}), we prove that diffusion models trained with coarse scores can exploit the regularity of the manifold support and attain a near-parametric rate toward a different target distribution. This target distribution has density uniformly comparable to that of~μdataμ_{\scriptscriptstyle\mathrm{data}} throughout any O~(N−β/(4k))\tilde{\mathcal{O}}\bigl(N^{-β/(4k)}\bigr)-neighborhood of the manifold, where ββ denotes the manifold regularity. Our guarantees therefore depend only on the smoothness of the underlying support, and are especially favorable when the data density itself is irregular, for instance non-differentiable. In particular, when the manifold is sufficiently smooth, we obtain that generalization -- formalized as the ability to generate novel, high-fidelity samples -- occurs at a statistical rate strictly faster than that required to estimate the full population distribution~μdataμ_{\scriptscriptstyle\mathrm{data}}.
May 7, 2026

Understanding diffusion models requires rethinking (again) generalization

This position paper argues that understanding generalization in diffusion models requires fundamentally new theoretical frameworks that go beyond both classical statistical learning theory and the benign overfitting paradigm developed for supervised learning. In diffusion models, unlike in supervised learning, memorization of training data and generalization to novel samples are incompatible: a model that has fully memorized its training set generates copies rather than novel data. Several theoretical explanations for why practical diffusion models nevertheless generalize have been proposed, based on capacity limitations, implicit regularization from optimization, or architectural inductive biases, but their interactions remain unclear. We argue that the field should pivot from explaining why the diffusion models do not memorize to investigating what the model actually learns during pre-memorization phase. To highlight our stance, we conduct empirical study of diffusion models trained on CIFAR-10, and we distill the findings into concrete open questions that we believe are key to improve understanding of generalization in diffusion models.