Beyond Sample Copying: Structural Memorization in Diffusion Models

Tim Kaiser; Markus Kollmann

cs.LGMar 12, 2026

Beyond Sample Copying: Structural Memorization in Diffusion Models

Authors: Tim Kaiser, Markus Kollmann

Organizations: Heinrich-Heine-University Düsseldorf

Abstract

Diffusion models generalize well in practice. Paradoxically, an optimal diffusion model fully memorizes the training data and therefore fails to generalize, raising the question of what induces generalization in a real diffusion model. We show that diffusion models progressively overfit the denoising training objective, creating a generalization gap between validation and training performance at intermediate noise levels. In a fully analytic 2D toy model with a controlled denoising error, we trace this gap to the interaction between model error and the density of the data distribution's support. The optimal denoising flow field localizes sharply around individual training points, whereas model error suppresses exact recall of training points, yielding a smooth, generalizing flow field. Finally, we examine how training-time overfitting manifests along inference trajectories. We find that predictions made from intermediate trajectory states occupy a distinct feature-space regime from predictions made from noised training and validation images. As training progresses and model size increases, these predictions develop greater relative affinity to the training data, despite the absolute similarity to the validation data not decreasing. Together, these findings show that the denoising objective and inference trajectories express structural memorization differently.

Figures & tables

Figure 1: Qualitative summary of contributions. (a) State-of-the-art diffusion models exhibit a growing generalization gap: the difference in reconstruction error (denoising training objective) between training and validation data increases as training progresses. (b) This gap is modulated by model error: Increasing model error smoothes the learned flow field, suppressing the generalization gap. For low model error (top), the flow field localizes sharply around training points, while for high model error (bottom), it more smoothly approaches the data manifold. (c) Similarly, during inference, trajectories become increasingly biased towards the training data, relative to the validation data, despite increasing their absolute similarity to both.

Panel of Figure 2 — Figure 2: The relative generalization gap $G(\sigma)$ increases with images seen during training (colorbar in millions) and with model size in relation to dataset size. The black line indicates the beginning of overfitting (validation error starts increasing) for each noise level $\sigma$ .

Panel of Figure 3 — Figure 3: Flow field lines of the optimal target predictor $\bm{y}$ at different noise levels $\sigma=28,2.8,0.63$ . Blue dots show predictions $\bm{y}^{*}(\bm{x},\sigma)$ for $\bm{x}=\bm{y}+\sigma\bm{n},\bm{n}\sim\mathcal{N}(0,I),\bm{y}\sim D_{\text{train}}$ . Color indicates the magnitude of $\bm{y}^{*}(\bm{x},\sigma)-\bm{x}$ . Field geometry is (a) global, predictions tend towards the superposition of all training points, (b) moderately localized around training points, predictions approximate the data manifold, and (c) highly localized around each training point, predictions replicate the training points.

Panel of Figure 4 — Figure 4: Relative generalization gap $G(\sigma)$ in our 2D toy model with circular data manifold (a) for different settings of the model error parameter $\delta$ (colorbar), (b) for different number of training points $N$ (colorbar), and (c) computed using a single point on the data manifold as the validation set, parametrized by its angle ( $\sigma\approx 1.1$ ). The resulting gap is zero at training points, and largest between training points, with magnitude related inversely to $\delta$ .

Panel of Figure 5 — Figure 5: (a) and (b) Average cosine similarity between predictions of query images $\bm{q}\in Q$ and clean images from the reference sets $\cal D_{\text{train/val}}$ ( Eq. 11 ). As the noise level $\sigma$ decreases, overall similarity to the reference sets increases. At low noise, both noisy training and noisy validation images as queries (blue and orange lines) exceed intermediate trajectory states (green line) in absolute similarity to both reference sets. (c) The relative train affinity ( Eq. 12 ) of intermediate trajectory states (green line) increases alongside the absolute similarities to the reference sets. All plots show EDM2-S on ImageNet64.

Panel of Figure 6 — Figure 6: (a) Average cosine similarity between predictions of query images $\bm{q}\in Q$ and clean images from the reference sets $\cal D_{\text{train/val}}$ ( Eq. 11 ). As training time increases, the absolute similarity of intermediate trajectory states (green line) to both reference sets increases. (b) So does their relative train-affinity ( Eq. 12 , green line). (c) Their positive relative train-affinity is robust across choices of $k$ . All plots show EDM2-S on ImageNet64.

Figure 7: The relative generalization gap $G(\sigma)$ with average L2-reconstruction error increases and shifts towards higher noise levels when the number of classes (colorbar) for a class-conditioned model increases. Each line corresponds to a re-trained EDM model on CIFAR-10.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Metric	Sample size	GPU-h	Wall-clock
ImageNet-64	$L({\cal D},\sigma)$	8192	4m - 4m	1m - 1m
	Standard FD	50.000	56m - 4h20m	14m - 1h5m
	$S_{k}(Q,R;\sigma)$	10.000	4m - 4m	1m - 1m
ImageNet-512	$L({\cal D},\sigma)$	8192	16m - 16m	4m - 4m
	Standard FD	50.000	2h28m - 7h44m	37m - 1h56m
	$S_{k}(Q,R;\sigma)$	10.000	-	-

Appendix

Table 1: Compute requirements for metric evaluation on ImageNet-64, for one model (checkpoint) and a single noise level

\sigma

. GPU-h and wall-clock time for ImageNet range from smallest to largest model size. They are approximate and refer to a server with 4xNVIDIA-A100 (45GB).

Panel of Figure 8 — Figure 8: Relative generalization gaps ( Eq. 1 ) with EDM2-S on ImageNet-64. $w=0$ corresponds to no guidance. An increasing guidance weight amplifies the gap, but this trend reverses for moderate to high guidance weights. (c) fixes $\sigma$ close to the peak of the generalization gaps in (a) and (b), indicated by the dotted black line.

Panel of Figure 9 — Figure 9: (a) Decreasing the receptive field mitigates the relative generalization gap ( Eq. 1 ) with $M_{\text{L2}}$ . Colorbar shows the sliding window size. 64 corresponds to regular inference. (b) During inference, the model shifts from long-range correlations to short-range correlations. The correlation score is averaged over pixels and images. (c) Edge pixels tend to have longer-range dependencies. $\sigma\approx 1.2$ . RGB shows the pixel-wise correlation score, agg is aggregated over channels. All plots show ImageNet-64, EDM2-S.

Figure 10: Edge pixels tend to have longer-range dependencies. Shown is the pixel-wise correlation score, averaged over 128 images and aggregated over channels, computed with EDM2-S on ImageNet-64 at $\sigma=1$ .

Panel of Figure 11 — Figure 11: Flow field lines and predictions of the optimal target predictor $\bm{y}^{*}(\bm{x},\sigma)$ at different noise levels $\sigma=28,2.8,0.63$ . Color indicates the magnitude of the prediction error $\bm{y}^{*}(\bm{x},\sigma)-x$ . Field geometry is (a) global, predictions tend towards the superposition of all training points. (b) moderately localized around training points, predictions approximate the data manifold (c) highly localized around each training point, predictions replicate the training points.

Panel of Figure 12 — Figure 12: Relative generalization gap ( Eq. 1 ) with $M_{\text{L2}}$ in our 2D toy model with random data on a circular manifold and (a) different settings of the model error parameter $\delta$ , (b) different number of training points, and (c) different settings for the guidance weight $w$ . $w=0$ corresponds to no guidance.

Panel of Figure 13 — Figure 13: Relative generalization gap ( Eq. 1 ) with EDM2 on ImageNet-64 for various model sizes, plotted vs $\sigma$ . Colorbar shows images seen during training. Dotted black lines indicate $\sigma$ values used in Fig. 14 .

Panel of Figure 14 — Figure 14: Relative generalization gap ( Eq. 1 ) and training and validation results, with EDM2 on ImageNet-64 for various model sizes. For each model size, $\sigma$ is fixed at the peak of the generalization gap in Fig. 13 .

Panel of Figure 15 — Figure 15: Relative generalization gap ( Eq. 1 ) with EDM2 on ImageNet-512 for various model sizes, plotted against the noise level $\sigma$ . Colorbar shows images seen during training. Dotted black lines indicate $\sigma$ values used in Fig. 16 .

Panel of Figure 16 — Figure 16: Relative generalization gap ( Eq. 1 ) and training and validation results, with EDM2 on ImageNet-64 for various model sizes. For each model size, $\sigma$ is fixed at the peak of the generalization gap in Fig. 15 .

Panel of Figure 17 — Figure 17: Relative generalization gap ( Eq. 1 ) with EDM on CIFAR-10/100. Colorbar shows images seen during training. Dotted black lines indicate $\sigma$ values used in Fig. 18

Panel of Figure 18 — Figure 18: Relative generalization gap ( Eq. 1 ) and training and validation results with EDM on CIFAR-10/100 for model sizes. $\sigma$ is fixed at the peak of the generalization gap, see Fig. 17 .

Panel of Figure 19 — Figure 19: The gap between the absolute similarity to the references between predictions from intermediate states of trajectories (green line) and predictions from noised data points (blue and orange lines) closes with increasing model size. All plots show fully trained models. First column: EDM2-XS. Second column: EDM2-XL.

Panel of Figure 20 — Figure 20: Qualitative trends in the metrics of Section 4 are robust to the choice of $k$ . All plots show fully trained models.

Panel of Figure 21 — Figure 21: The gap between the absolute similarity to the references between predictions from intermediate states of trajectories (green line) and predictions from noised data points (blue and orange lines) closes with increasing model size. All plots show fully trained models. First column: EDM2-XS. Second column: EDM2-XL.

Explore similar work

Understanding diffusion models requires rethinking (again) generalization
May 7, 2026Pierre Marion, Yu-Han WuMemorization in Generative Models Diffusion Models
Inria, École Normale Supérieure, PSL Research University · LPSM, Sorbonne University & Google DeepMind
Early Signatures of Memorization in Diffusion Models via Basin Geometry and Cyclic Denoising
Oct 8, 2026Nikhil Verma, Siddharthan Dileep, Anoop Singh +4Memorization in Generative Models Diffusion Models
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Computer Science and Engineering, Indian Institute of Technology Delhi · Jawaharlal Nehru Centre for Advanced Scientific Research +2
On the Memorization of Consistency Distillation for Diffusion Models
Apr 26, 2026Bingqing Jiang, Difan ZouMemorization in Generative Models Consistency Distillation
School of Computing & Data Science, The University of Hong Kong. · School of Computing & Data Science and Institute of Data Science, The University of Hong Kong.

May 7, 2026, cs.LGJ/K move · Enter open · S save

Understanding diffusion models requires rethinking (again) generalization

Pierre Marion, Yu-Han Wu

Inria, École Normale Supérieure, PSL Research University · LPSM, Sorbonne University & Google DeepMind

Memorization in Generative Models Diffusion Models

Open paper AskPDF arXiv

This position paper argues that understanding generalization in diffusion models requires fundamentally new theoretical frameworks that go beyond both classical statistical learning theory and the benign overfitting paradigm developed for supervised learning. In diffusion models, unlike in supervised learning, memorization of training data and generalization to novel samples are incompatible: a model that has fully memorized its training set generates copies rather than novel data. Several theoretical explanations for why practical diffusion models nevertheless generalize have been proposed, based on capacity limitations, implicit regularization from optimization, or architectural inductive biases, but their interactions remain unclear. We argue that the field should pivot from explaining why the diffusion models do not memorize to investigating what the model actually learns during pre-memorization phase. To highlight our stance, we conduct empirical study of diffusion models trained on CIFAR-10, and we distill the findings into concrete open questions that we believe are key to improve understanding of generalization in diffusion models.

Abstract

Explore similar work

Understanding diffusion models requires rethinking (again) generalization

Early Signatures of Memorization in Diffusion Models via Basin Geometry and Cyclic Denoising

On the Memorization of Consistency Distillation for Diffusion Models