Organizations: Department of Mathematics, University of California, Los Angeles, CA 90095, USA. · Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA.
Understanding the mechanism behind why diffusion models memorize their training data remains an open question in generative modeling. In this paper, we examine this question from two perspectives: (a) the model-level perspective concerning the onset of memorization in a model during training, and (b) the sample-level perspective concerning the proclivity of an individual sample to be memorized by a given model. We show that the so-called input loss curvature, or the Frobenius norm of the Hessian of the diffusion model's training loss, is an effective metric for quantifying both perspectives on memorization. In particular, we empirically demonstrate that input curvatures averaged across training and validation samples can identify when a model is likely to start regurgitating its training data when tracked over training time. However, this metric is not sufficient for measuring an individual sample's likelihood of being memorized. To address this perspective, we introduce a counterfactual definition of memorization in diffusion models, extending the analogous notion from supervised learning proposed by [Feldman, 2019]. We prove that this counterfactual memorization is controlled by the input curvature of the leave-one-out model, thus establishing a direct theoretical link between a sample's local loss landscape and its generation likelihood. We validate our derived bounds through several empirical experiments and show that input-dependent curvature provides a proxy for memorization.
Figures & tables
Figure 1: (A) Train vs. validation input curvature for CelebA, n=128, (B) Val. curvature starts to diverge from train curvature when fmem(τ) starts accelerating, (C) fmem(τ) values for n=128,10240 on CelebA, (D) and (E) Same experiment for CelebA, n=10240. Note that this model undergoes generalization, and memorization sets in much later at τ≈105 , (F) Curvature of an individual sample is not a good predictor of its copy count.
Figure 2: (A) Input ( 11 ) curvatures consistently rise but validation ( 12 ) curvatures reach a local minimum at approximately peak memorization for DDPMs on different data sizes, (B) Sizes of gradients of validation loss also display local minima, (C) Input curvature on training data tracks the largest frequency of memorized samples, (D) Relative error between train and validation curvatures exhibits a local maximum, leading to an early stopping criterion, (E) Memorization cannot be detected just based on validation loss, (F) Memorization rises as models train further.
Figure 3: Loss difference and curvature for Imagenet (n=128) DDPM noise prediction - leave-10-out training : (left) difference between per sample loss of full model and leave-10-out model vs. training step for each of the 10 left-out training samples, (right) per sample curvature of the leave-10-out model vs. training step for each of the 10 left-out training samples.
Figure 4: Loss difference and curvature for Imagenet (n=1024) DDPM noise prediction - leave-10-out-training : (left) difference between per sample loss of full model and leave-10-out model vs. training step for each of the 10 left-out training samples, (right) per sample curvature of the leave-10-out model vs. training step for each of the 10 left-out training samples.
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Generated samples vs. nearest training image for CelebA (n=128) DDPM : generated samples at the pre-memorization, start of memorization, and full memorization stages alongside their nearest neighbor in the training set.
Figure 6: Generated samples vs. nearest training image for CelebA (n=128) flow matching : generated samples at the pre-memorization, start of memorization, and full memorization stages alongside their nearest neighbor in the training set.
Figure 7: Generated samples vs. nearest training image for CelebA (n=10240) DDPM : generated samples at the pre-memorization, start of memorization, and full memorization stages alongside their nearest neighbor in the training set.
Figure 8: Generated samples vs. nearest training image for CelebA (n=10240) flow matching : generated samples at the pre-memorization, start of memorization, and full memorization stages alongside their nearest neighbor in the training set.
Figure 9: Generated samples vs. nearest training image for Imagenet (n=128) DDPM leave-10-out model : generated samples at the pre-memorization, start of memorization, and full memorization stages alongside their nearest neighbor in the training set.
Figure 10: Generated samples vs. nearest training image for Imagenet (n=1024) DDPM leave-10-out model : generated samples at the pre-memorization, start of memorization, and full memorization stages alongside their nearest neighbor in the training set.
Figure 11: Generated samples vs. nearest training image for Imagenet (n=128) flow matching leave-10-out model : generated samples at the pre-memorization, start of memorization, and full memorization stages alongside their nearest neighbor in the training set.
Figure 12: Generated samples vs. nearest training image for Imagenet (n=1024) flow matching leave-10-out model : generated samples at the pre-memorization, start of memorization, and full memorization stages alongside their nearest neighbor in the training set.
Figure 13: Generated samples vs. nearest training image for ImageNet ( n=128 ) DDPM. pre_mem: step 2,000 , fmem=0 , five random samples. start_mem: step 3,000 , fmem=0.10% (10 copies of 104 ), five randomly selected copies. full_mem: step 150,000 , fmem=99.8% , five randomly selected copies.
Figure 14: Generated samples vs. nearest training image for ImageNet ( n=256 ) DDPM. pre_mem: step 3,000 , fmem=0 , five random samples. start_mem: step 5,000 , fmem=0.04% (4 copies of 104 ); the four copies, followed in the rightmost pair by the non-copy with the smallest gap ratio. full_mem: step 200,000 , fmem=95.5% , five randomly selected copies.
Figure 15: Generated samples vs. nearest training image for ImageNet ( n=512 ) DDPM. pre_mem: step 5,000 , fmem=0 , five random samples. start_mem: step 7,000 , fmem=0.03% (3 copies of 104 ); the three copies, followed in the two rightmost pairs by the non-copies with the smallest gap ratios. peak_mem: step 2,000,000 , fmem=85.0% , five randomly selected copies. This model never reaches the full-memorization threshold.
Figure 16: Generated samples vs. nearest training image for ImageNet ( n=1024 ) DDPM. pre_mem: step 15,000 , fmem=0 , five random samples. start_mem: step 20,000 , fmem=0.10% (10 copies of 104 ), five randomly selected copies. peak_mem: step 2,000,000 , fmem=19.9% , five randomly selected copies. This model never reaches the full-memorization threshold.
Figure 17: Generated samples vs. nearest training image for ImageNet ( n=2048 ) DDPM. pre_mem: step 70,000 , fmem=0 , five random samples. start_mem: step 100,000 , fmem=0.02% (2 copies of 104 ); the two copies, followed in the three rightmost pairs by the non-copies with the smallest gap ratios. There is no full_mem row: fmem peaks at 0.57% .
Figure 18: Generated samples vs. nearest training image for ImageNet ( n=4096 ) DDPM. pre_mem: step 300,000 , fmem=0 , five random samples. start_mem: step 500,000 , an isolated checkpoint with fmem=0.03% (3 copies of 104 ); the three copies, followed in the two rightmost pairs by the non-copies with the smallest gap ratios. Here start_mem labels the isolated copy event. No sustained memorization is observed, and there is no full_mem row.
Figure 19: Generated samples vs. nearest training image for ImageNet ( n=128 ) flow matching. pre_mem: step 1,000 , fmem=0 , five random samples. start_mem: step 2,000 , fmem=0.10% (10 copies of 104 ), five randomly selected copies. full_mem: step 10,000 , fmem=95.7% , five randomly selected copies.
Figure 20: Generated samples vs. nearest training image for ImageNet ( n=1024 ) flow matching. pre_mem: step 10,000 , fmem=0 , five random samples. start_mem: step 15,000 , fmem=0.46% (46 copies of 104 ), five randomly selected copies. full_mem: step 500,000 , fmem=96.2% , five randomly selected copies.
Figure 21: Curvature and memorization for CelebA (n=128) : average input loss curvature over diffusion time and training set and fraction memorized (%) with memorization threshold k=1/3 vs. training step for (a) DDPM noise prediction and (b) flow matching.
Figure 22: Curvature and memorization for CelebA (n=10240) : average input loss curvature over diffusion time and training set and fraction memorized (%) with memorization threshold k=1/3 vs. training step for (a) DDPM noise prediction and (b) flow matching.
Figure 23: Curvature and memorization for Imagenet (n=128) : average input loss curvature over diffusion time and training set and fraction memorized (%) with memorization threshold k=1/3 vs. training step for (a) DDPM noise prediction and (b) flow matching.
Figure 24: Curvature and memorization for Imagenet (n=1024) : average input loss curvature over diffusion time and training set and fraction memorized (%) with memorization threshold k=1/3 vs. training step for (a) DDPM noise prediction and (b) flow matching.
Figure 25: Curvature and memorization for Imagenet (n=10240) : average input loss curvature over diffusion time and training set and fraction memorized (%) with memorization threshold k=1/3 vs. training step for (a) DDPM noise prediction and (b) flow matching.
Figure 26: Training vs. validation curvature for CelebA (n=128) flow matching : (left) average input loss curvature over diffusion time and training (blue) / validation (orange) set, (right) relative training/validation curvature gap calculated as ∣Ex∼train[∥∇x2ℓ(ϕ(τ),x)∥F2]−Ex∼val.[∥∇x2ℓ(ϕ(τ),x)∥F2]∣/Ex∼train[∥∇x2ℓ(ϕ(τ),x)∥F2] (blue) and ∣Ex∼train[∥∇x2ℓ(ϕ(τ),x)∥F2]−Ex∼val.[∥∇x2ℓ(ϕ(τ),x)∥F2]∣/Ex∼val[∥∇x2ℓ(ϕ(τ),x)∥F2] (orange).
Figure 27: Memorization for Imagenet (n=10240) : fraction memorized (%) with memorization threshold k∈{1/3,0.8,0.9} vs. training step for (a) DDPM noise prediction and (b) flow matching.
Figure 28: Curvature and memorization for Imagenet (n=10240) DDPM Sample Prediction : (a) average input loss curvature over diffusion time and training set and fraction memorized (%) with memorization threshold k=1/3 vs. training step, and (b) fraction memorized (%) with memorization threshold k∈{1/3,0.8,0.9} vs. training step.
Figure 29: CelebA vs. Imagenet (n=10240) Curvature : average input loss curvature over diffusion time and training set vs. training step for (a) DDPM noise prediction and (b) flow matching.
Figure 30: Memorization and curvature for Imagenet (n=1024) DDPM noise prediction - full vs. leave-10-out training : (a) fraction memorized (%) with memorization threshold k=1/3 vs. training step and (b) average input loss curvature over diffusion time and training set vs. training step for full n=1024 training set and leave-10-out training set with the top 10 memorized training samples removed.
Figure 31: Memorization and curvature for Imagenet (n=1024) flow matching - full vs. leave-10-out training : (a) fraction memorized (%) with memorization threshold k=1/3 vs. training step and (b) average input loss curvature over diffusion time and training set vs. training step for full n=1024 training set and leave-10-out training set with the top 10 memorized training samples removed.
Figure 32: Memorization and curvature for Imagenet (n=128) DDPM noise prediction - full vs. leave-10-out training : (a) fraction memorized (%) with memorization threshold k=1/3 vs. training step and (b) average input loss curvature over diffusion time and training set vs. training step for full n=128 training set and leave-10-out training set with the top 10 memorized training samples removed.
Figure 33: Memorization and curvature for Imagenet (n=128) flow matching - full vs. leave-10-out training : (a) fraction memorized (%) with memorization threshold k=1/3 vs. training step and (b) average input loss curvature over diffusion time and training set vs. training step for full n=128 training set and leave-10-out training set with the top 10 memorized training samples removed.
Figure 34: Loss difference and curvature for Imagenet (n=128) flow matching - leave-10-out-training : (left) difference between per sample loss of full model and leave-10-out model vs. training step for each of the 10 left-out training samples, (right) per sample curvature of the leave-10-out model vs. training step for each of the 10 left-out training samples.
Figure 35: Loss difference and curvature for Imagenet (n=1024) flow matching - leave-10-out-training : (left) difference between per sample loss of full model and leave-10-out model vs. training step for each of the 10 left-out training samples, (right) per sample curvature of the leave-10-out model vs. training step for each of the 10 left-out training samples.
Figure 36: Memorization and curvature for Imagenet (n=128) flow matching - full vs. leave-10-out training : (a) fraction memorized (%) with memorization threshold k=1/3 vs. training step and (b) average input loss curvature over diffusion time and training set vs. training step for full n=128 training set and leave-10-out training set with the top 10 memorized training samples removed.
Figure 37: Curvature of copied and non-copied training images. Each panel shows one ImageNet DDPM training size. Orange and blue curves average the input-Hessian trace over images with F(i,τ)>0 and F(i,τ)=0 , respectively. Group membership is determined separately at each checkpoint; “never copied” in the legend refers only to that checkpoint’s generations. Shading shows one standard error across images in the group, and orange numbers give the copied-group size. A group is plotted only when it contains at least ten images. Dashed vertical lines mark the first evaluated checkpoint with fmem>1% , used as a reference threshold.
Figure 38: Maximum copy count and relative curvature gap. Each point represents one evaluated checkpoint of an ImageNet DDPM; colors indicate training-set size. The horizontal coordinate is maxiF(i,τ) among 10,000 generations. The vertical coordinate is ∣Ctrain(τ)−Cval(τ)∣/∣Ctrain(τ)∣ , where the two means use all training images and the same 1,024 held-out images, respectively. Curvature is the input-Hessian trace estimated as described above. The horizontal scale is logarithmic above one and linear near zero to retain checkpoints with no observed copies.
Memorization in generative models studies when models retain training examples, how to detect their recall, and how to limit it without losing utility.
Generative models have a persistent limitation: their tendency to memorize training data can create legal liabilities and erode creative diversity. Understanding which samples are memorized in whole or in part, and under what conditions, therefore remains an important open problem. Here we answer the question "Are atypical or rare samples memorized first?" in the negative. We train diffusion models on strings generated according to the production rules of the Random Hierarchy Model (RHM), and find that samples composed of common substrings are preferentially memorized. This holds true even if the training data consists of entirely unique samples, indicating that deduplication at the data point level does not provide a meaningful privacy guarantee. Correspondingly we predict, then observe, delayed memorization for fat-tailed datasets (i.e., those with more atypical samples). This effect is amplified when fat-tails are introduced into high-level production rules. These together suggest that dataset diversity, particularly at higher levels of abstraction, plays an important role in staving off memorization. Finally, we identify an intermediate regime of partial memorization in which common substrings are learned first and subsequently overproduced during generation. If training is stopped in this regime, models will exhibit the reversion-to-the-mean blandness often derided as "slop".
Marta Aparicio Rodriguez, Anastasia Borovykh, Grigorios A. Pavliotis +1
Department of Mathematics, Imperial College London, UK · ML Lab, Capital Fund Management, France · Department of Physics, École Polytechnique Fédérale de Lausanne (EPFL), Switzerland
Diffusion models can unintentionally memorize training samples, raising concerns about privacy and copyright. While recent methods can detect memorization, they often rely on global or model-specific signals and provide limited insight into where memorization appears within a generated image. We provide a geometric characterization of local memorization as a coordinate-wise variance collapse. However, such collapse can also arise from intrinsic data constraints rather than overfitting. To isolate overfitting-driven memorization, we propose curvature-difference methods that subtract the curvature of an underfitted baseline, either the unconditional model or a less-trained version of itself. We further derive a score-difference proxy that provides a geometric explanation for the widely used score-difference-based detection metric. Experiments on Stable Diffusion, evaluated against ground-truth memorization masks, show that our method outperforms the prior attention-based localization method. Code is available at https://github.com/Gwangho99/mem-curv-diff.
Gwangho Kim, Sungyoon Lee
Department of Computer Science, Hanyang University, Seoul, South Korea.
Diffusion models generalize well in practice. Paradoxically, an optimal diffusion model fully memorizes the training data and therefore fails to generalize, raising the question of what induces generalization in a real diffusion model. We show that diffusion models progressively overfit the denoising training objective, creating a generalization gap between validation and training performance at intermediate noise levels. In a fully analytic 2D toy model with a controlled denoising error, we trace this gap to the interaction between model error and the density of the data distribution's support. The optimal denoising flow field localizes sharply around individual training points, whereas model error suppresses exact recall of training points, yielding a smooth, generalizing flow field. Finally, we examine how training-time overfitting manifests along inference trajectories. We find that predictions made from intermediate trajectory states occupy a distinct feature-space regime from predictions made from noised training and validation images. As training progresses and model size increases, these predictions develop greater relative affinity to the training data, despite the absolute similarity to the validation data not decreasing. Together, these findings show that the denoising objective and inference trajectories express structural memorization differently.