Diffusion probabilistic models have become a cornerstone of modern generative AI, yet the mechanisms underlying their generalization remain poorly understood. In fact, if these models were perfectly minimizing their training loss, they would just generate data belonging to their training set, i.e., memorize, as empirically found in the overparameterized regime. We revisit this view by showing that, in highly overparameterized diffusion models, generalization in natural data domains is progressively achieved during training before the onset of memorization. Our results, ranging from image to language diffusion models, systematically support the empirical law that memorization time is proportional to the dataset size, consistent with a kernel-regression bound on the time required to fit the empirical score at low noise. Generalization vs. memorization is then best understood as a competition between time scales. We show that this phenomenology is recovered in diffusion models learning a simple probabilistic context-free grammar with random rules, where generalization corresponds to the hierarchical acquisition of deeper grammar rules as training time grows, and the generalization cost of early stopping can be characterized. We summarize these results in a phase diagram. Overall, our results support that a principled early-stopping criterion - scaling with dataset size - can effectively optimize generalization while avoiding memorization, with direct implications for hyperparameter transfer and privacy-sensitive applications.
Figures & tables
Figure 1: Memorization dynamics in vision diffusion models. Left: Train loss, validation loss, and fraction of copied images as a function of training steps τ for iDDPM models trained on CIFAR-10 with varying training set sizes P . Both losses decrease initially, indicating generalization, but diverge at the onset of memorization ( τmem , vertical dashed lines), where the models start copying training data. Larger training sets delay τmem , scaling approximately linearly with P (insets, in which curves are plotted against τ/P ). Lines: mean across 3 independent runs; shaded bands: ±1 std. Right, top: Samples generated by the model trained on P=16,384 images at early stopping ( τ≈τmem ) and well past memorization ( τ≫τmem ), each shown above their nearest neighbor in the training set: at τmem the generations are novel, while at long training they reproduce training examples. Right, bottom: FID against the union of CIFAR-10 train and test splits as a function of τ for the same model; the FID is non-monotonic, with a minimum at τ≈τmem .
Figure 2: Progressive generalization in vision diffusion models. Cosine similarity between images generated by two diffusion models trained on disjoint subsets of CelebA of size P=2,048 , as a function of training steps τ . Throughout the pre-memorization window ( τ<τmem ), the two models generate nearly identical images, indicating they are approximating the same underlying score function – a property of the population distribution and not of the training set used. After τmem , the models diverge, generating images increasingly similar to their own training sets.
Figure 3: Memorization dynamics in language diffusion models. Train loss, validation loss, and fraction of copied text as a function of training steps for GPT-based MD4 models trained on text8 and varying training set sizes P . Both losses initially decrease, indicating generalization, but diverge at the onset of memorization ( τmem ), where the models start copying training text. τmem grows linearly with P (insets). Lines: mean across 3 independent runs; shaded bands: ±1 std.
Figure 4: Memorization vs. generalization on the RHM. For training set size P=256 , the diffusion model generates valid data only when it is memorizing. For P=16,384 , instead, the model generalizes, approximately at τ∗ , before starting to memorize. The memorization time scales linearly in P (insets). Data for RHM parameters v=16 , m=4 , L=3 , s=2 .
Figure 5: Diffusion models achieve partial generalization in the RHM before memorizing. (a) The diffusion model learns progressively deeper RHM rules during training. However, the rules at the deepest level L=5 are never learned, and the corresponding error decreases only when memorization occurs, since P=1,024 is smaller than the sample complexity PL∗∼104 . (b) Two diffusion models trained on disjoint training sets learn the same score function before the onset of memorization at τmem . Data for RHM parameters v=16 , m=3 , L=5 , s=2 .
Figure 6: Phase diagram of generalization vs. memorization indicating different regimes as a function of training time τ and sample complexity P : partial generalization, (full) generalization and memorization. Note that in the RHM, learning proceeds through well-defined steps, while it is smoother for natural data (or more realistic versions of the RHM ( Cagnetta et al., 2025b ) ).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Memorization dynamics in Stable Diffusion. (a) Training and validation losses as a function of training step τ for Stable Diffusion fine-tuned on different subsets of LAION-10k with P training points. Both losses initially decrease, indicating generalization, and diverge at the memorization onset time τmem . The memorization time τmem is linear in the training set size P . (b) Cosine similarity scores between SSCD ResNet embedding for generated images and their nearest training neighbor at early stopping ( τ=8,192 ) and final training ( τ=262,144 ). The dashed line indicates the mean similarity score between the closest LAION-10k samples. The sharp increase at late training signals memorization.
Figure 8: Replicates generated by Stable Diffusion. Example generations (left) from the final training checkpoint ( τ=262,144 ) with similarity score >0.5 to their nearest neighbor in the training set (right), confirming memorization.
Figure 9: Scaling of memorization time with dataset size. For each dataset size P , we linearly interpolate the train and validation losses as a function of the training step τ (on a logarithmic grid) and define the memorization time τmem as the first step at which the interpolated loss gap Lval(τ)−Ltrain(τ) exceeds a fixed threshold (different colors). We then plot τmem as a function of P for (a) iDDPMs trained on CIFAR-10 and (b) MD4 language diffusion models trained on text8. In both cases, the data are well described by a power-law fit τmem∝Pβ (dashed lines) with exponents β close to one, indicating an approximately linear growth of memorization time with dataset size across modalities. Error bars indicate the sample standard deviation across three independent runs.
Figure 10: CIFAR-10 samples generated with early-stopped model. Additional samples from the iDDPM trained on 16,384 CIFAR-10 images, generated at the early stopping point before memorization. The model produces diverse and high-quality images without replicating the training data.
Figure 11: Examples of copies on CIFAR-10. Top: samples generated by the iDDPM trained on 8,192 CIFAR-10 images at the end of training. Bottom: nearest neighbors from the training set. The model reproduces specific training examples, indicating memorization.
Figure 12: Examples of copies on text8. Left: Diffusion-generated text with MD4 trained on 1,024 training sequences of the text8 dataset for 524,288 SGD steps. Right: the corresponding nearest training sequences. The generated samples are copies of the training set, up to a few character-level mistakes.
Figure 13: Sampling of RHM production rules. Mean occurrence ( left ) and centered covariance ( right ) of the production rules sampled by a diffusion model trained on P=16,384 strings ( v=16 , m=4 , L=3 , s=2 ). The model, trained with early stopping ( τ=32,768 ), samples all RHM rules with a mean occurrence that is approximately uniform (up to sampling noise). Likewise, the correlations among the co-occurrences of sampled rules show that they are sampled approximately independently.
Figure 14: Effect of the optimizer on the onset of memorization in the RHM. Fraction of copies as a function of training step for a D3PM trained on P=2,048 sequences sampled from an RHM with L=3 , s=3 , v=24 , m=12 , using Adam and SGD, each with its maximal stable learning rate (largest learning rate yielding convergent training loss). The curves nearly coincide, and the onset of memorization occurs at approximately the same number of training steps for both optimizers.
Figure 15: Neural Tangent Kernel (NTK) initialization: one-hidden-layer ReLU neural network (width 8192 ) learning the empirical score at fixed diffusion noise variance σ2 , trained with full-batch gradient descent. Training points sampled from a Gaussian distribution in d=64 dimensions. Left : at fixed training set size P=128 , training and test loss diverge at a timescale ( τmem ) depending on σ (inset), which scales as σ−1 (main). Right : at fixed σ2=3.2⋅10−2 , τmem increases with P (inset), consistently with the scaling τmem∝P (main).
Figure 16: Feature learning (mean-field) initialization, same setting as Figure 15 . Also in this case, τmem is compatible with the scaling τmem∼σ−1 at fixed P ( left ), and τmem∝P at fixed σ ( right ).
Figure 17: Effect of changing batch size B , same setting as Figs. 15 and 16 (fixed σ2=3.2⋅10−2 , P=128 ). Varying the batch size B of training, both with the NTK ( left ) and feature learning ( right ) initialization, does not affect τmem .
Figure 18: Optimizer dependence. Training loss as a function of training step for a one-hidden-layer ReLU neural network (NTK initialization) trained with Adam and SGD at multiple dataset sizes P . For each optimizer, we choose its maximal stable learning rate, defined as the largest learning rate for which the training loss reliably converges. For both optimizers, the curves at different P start decaying at the same point when plotted against τ/P , indicating the same linear scaling of the characteristic training time with P . Adam and SGD differ only by a horizontal shift, corresponding to an OP(1) change in the constant of proportionality. The overall scaling and decay of the loss remain essentially identical. The inset shows the same data as a function of the unscaled training step τ .
This position paper argues that understanding generalization in diffusion models requires fundamentally new theoretical frameworks that go beyond both classical statistical learning theory and the benign overfitting paradigm developed for supervised learning. In diffusion models, unlike in supervised learning, memorization of training data and generalization to novel samples are incompatible: a model that has fully memorized its training set generates copies rather than novel data. Several theoretical explanations for why practical diffusion models nevertheless generalize have been proposed, based on capacity limitations, implicit regularization from optimization, or architectural inductive biases, but their interactions remain unclear. We argue that the field should pivot from explaining why the diffusion models do not memorize to investigating what the model actually learns during pre-memorization phase. To highlight our stance, we conduct empirical study of diffusion models trained on CIFAR-10, and we distill the findings into concrete open questions that we believe are key to improve understanding of generalization in diffusion models.
Pierre Marion, Yu-Han Wu
Inria, École Normale Supérieure, PSL Research University · LPSM, Sorbonne University & Google DeepMind
Diffusion models generalize well in practice. Paradoxically, an optimal diffusion model fully memorizes the training data and therefore fails to generalize, raising the question of what induces generalization in a real diffusion model. We show that diffusion models progressively overfit the denoising training objective, creating a generalization gap between validation and training performance at intermediate noise levels. In a fully analytic 2D toy model with a controlled denoising error, we trace this gap to the interaction between model error and the density of the data distribution's support. The optimal denoising flow field localizes sharply around individual training points, whereas model error suppresses exact recall of training points, yielding a smooth, generalizing flow field. Finally, we examine how training-time overfitting manifests along inference trajectories. We find that predictions made from intermediate trajectory states occupy a distinct feature-space regime from predictions made from noised training and validation images. As training progresses and model size increases, these predictions develop greater relative affinity to the training data, despite the absolute similarity to the validation data not decreasing. Together, these findings show that the denoising objective and inference trajectories express structural memorization differently.
Diffusion models trained on increasingly large datasets sampled from the same data distribution will converge to the same learned distribution. However, in practice, a model trained on finite data depends on the particular training dataset; this dependence is reflected in properties of the model's outputs, such as memorization and sample quality. To understand this dependence through an information-theoretic lens, we study model information, a computationally tractable measure of how much samples from the trained model depend on the particular training dataset. We explore its scaling behavior as model size and training dataset size vary. Specifically, we uncover two qualitatively distinct scaling regimes: (1) a nonlinear model information regime marking the transition from pure memorization to imperfect generalization, and (2) a generalization regime characterized by linear scaling of the model information and progressively improved generalization and sample quality. We then posit an information-theoretically principled scaling law to quantify these two regimes, and show that this law yields induced scaling laws for memorization and sample quality. We finally demonstrate that these induced scaling laws extrapolate well outside the fit's domain. Our results demonstrate information-theoretic principles for memorization and sample quality scaling laws in diffusion models. Our code is publicly available on GitHub.
Druv Pai, Dejan Stancevic, Yi Ma +1
UC Berkeley · Radboud University · HKU, UC Berkeley