Two Calm Ends and the Wild Middle: A Geometric Picture of Memorization in Diffusion Models

Nick Dodson; Xinyu Gao; Qingsong Wang; Yusu Wang; Zhengchao Wan

cs.LGFeb 19, 2026

Two Calm Ends and the Wild Middle: A Geometric Picture of Memorization in Diffusion Models

Authors: Nick Dodson, Xinyu Gao, Qingsong Wang, Yusu Wang, Zhengchao Wan

Organizations: Department of Mathematics, University of Missouri, Columbia, Missouri, USA · Halıcıo˘glu Data Science Institute, University of California San Diego, La Jolla, California, USA

Abstract

Diffusion models generate high-quality samples but can also memorize training data, raising serious privacy concerns. Understanding the mechanisms governing when memorization versus generalization occurs remains an active area of research. In particular, it is unclear where along the noise schedule memorization is induced, how data geometry influences it, and how phenomena at different noise scales interact. We introduce a geometric framework that partitions the noise schedule into three regimes based on the coverage properties of training data by Gaussian shells and the concentration behavior of the posterior, which we argue are two fundamental objects governing memorization and generalization in diffusion models. This perspective reveals that memorization risk is highly non-uniform across noise levels. We further identify a danger zone at medium noise levels where memorization is most pronounced. In contrast, both the small and large noise regimes resist memorization, but through fundamentally different mechanisms: small noise avoids memorization due to limited training coverage, while large noise exhibits low posterior concentration and admits a provably near linear Gaussian denoising behavior. For the medium noise regime, we identify geometric conditions through which we propose a geometry-informed targeted intervention that mitigates memorization.

Figures & tables

Figure 1: MSE to Clean Image. Comparison of denoising quality across noise levels. Solid lines: training data; dotted lines: test data. EDM-1K shows a generalization gap in the mid-

\sigma

region.

Figure 2: Per-Noise-Level Memorization Rate. Fraction of denoised test images classified as memorized at each noise level.

EDM-1K $\to$ EDM-50K swap:
Condition	Noise Range ( $\sigma$ )	Mem. Rate
EDM-1K (default sample)	$[0.002,80]$	92.2%
EDM-50K (default sample)	$[0.002,80]$	0.0%
large	$\sigma>8.4$	93.0%
medium	$[0.14,8.4]$	0.0%
small	$\sigma<0.14$	91.0%

Table 1: Memorization Rates in Denoiser Swapping. Memorization measured as fraction of 256 samples where

d_{1\mathrm{NN}}<d_{2\mathrm{NN}}/3

. Swapping in medium region flips behavior.

Figure 3: Gaussian shell coverage and max posterior weight on CIFAR-10. We plot empirical estimates of the Gaussian shell coverage

C_{\sigma}(p,\mathcal{D})

and the max posterior weight

W_{\sigma}(\mathcal{D})

as functions of the noise level

\sigma

, using a

1\mathrm{k}

CIFAR-10 training subset and

1\mathrm{k}

held-out test images.

Figure 4: Comparison of anti-memorization methods. Training curves showing FID score (solid lines) and memorization ratio (dashed lines) for three methods: 2k-Baseline, 2k-Dummy, and 2k-Gap. Gap training achieves the lowest memorization (0.7%) and best FID (2.35) at final checkpoint.

Figure 5: Danger zone and gap configurations. The intersection of coverage and max weight curves identifies the danger zone. Memorization rates for different gap configurations are overlaid, showing that gaps targeting this region minimize memorization.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Empirical shell coverage on CIFAR-10. We plot the lower bound, coverage, and upper bound from Theorem 4.8 . The lower bound curve (blue) is nearly indistinguishable from the coverage curve across the entire range of $\sigma$ , as it almost perfectly overlaps with it.

Figure 7: Max vs. Sample Weight on 1K CIFAR-10. We plot average values of $\max_{1\leq i\leq N}w_{i}(x_{1}+\sigma\bm{Z},\sigma)$ and $w_{1}(x_{1}+\sigma\bm{Z})$ across a range of $\sigma$ . The two curves show that the max posterior weight can usually be well estimated by the sample weight, particularly at smaller noise levels where they coincide.

Figure 8: Coverage and max posterior weight vs noise level for 2k car training set and 1k car test set. The danger zone is where both metrics are high, spanning

\sigma\in[1,5]

with intersection near

\sigma\approx 1.9

Figure 9: Pareto frontier showing the trade-off between image quality (FID) and memorization (FMEM%). Each point represents a different timestep gap configuration. The dashed line connects non-dominated solutions forming the Pareto frontier.

Figure 10: Denoiser Swapping Samples. Top half: EDM-1K base with EDM-50K swapped in different regions. Bottom half: EDM-50K base with EDM-1K swapped in. Red boxes indicate memorized samples (where $d_{1\mathrm{NN}}<d_{2\mathrm{NN}}/3$ ). Swapping in the medium region (steps 6-13) completely flips memorization behavior, while swapping only in the danger zone (steps 7-10) shows partial effects.

$\sigma$	80.00	57.59	40.79	28.37	19.35	12.91	8.40	5.32	3.26	1.92	1.09	0.59	0.30	0.14	0.06	0.02	0.01	0.00

Appendix

Table 2: Time Parameterization Conventions for EDM Sampling. EDM uses 18 sampling steps for CIFAR-10 with noise scales

\sigma

(top row) and EDM time coordinate Red denotes initial (high noise) steps, blue denotes middle steps, green denotes end (low noise) steps.

Figure 11: One-Step Denoising at $\sigma=8.4$ . From top: initial test image, noisy input, EDM-1K output, 1-NN of EDM-1K , EDM-50K output, 1-NN of EDM-50K , EMP-1K output, 1-NN of EMP-1K . At this high noise level, all denoisers perform similarly with no memorization.

Figure 12: One-Step Denoising at $\sigma=5.3$ . From top: initial test image, noisy input, EDM-1K output, 1-NN of EDM-1K , EDM-50K output, 1-NN of EDM-50K , EMP-1K output, 1-NN of EMP-1K . Denoising begins to show model differences as EDM-1K starts exhibiting some preference for training images.

Figure 13: One-Step Denoising at $\sigma=3.2$ . From top: initial test image, noisy input, EDM-1K output, 1-NN of EDM-1K , EDM-50K output, 1-NN of EDM-50K , EMP-1K output, 1-NN of EMP-1K . EDM-1K memorization becomes pronounced, with outputs matching training nearest neighbors. This is near the peak memorization region.

Figure 14: One-Step Denoising at $\sigma=1.9$ . From top: initial test image, noisy input, EDM-1K output, 1-NN of EDM-1K , EDM-50K output, 1-NN of EDM-50K , EMP-1K output, 1-NN of EMP-1K . EDM-1K outputs are much closer to training images than EDM-50K at this critical mid- $\sigma$ level.

Figure 15: One-Step Denoising at $\sigma=0.6$ . From top: initial test image, noisy input, EDM-1K output, 1-NN of EDM-1K , EDM-50K output, 1-NN of EDM-50K , EMP-1K output, 1-NN of EMP-1K . At this lower noise level, EDM-1K memorization weakens while EMP-1K shows stronger nearest-neighbor retrieval.

Figure 16: One-Step Denoising at $\sigma=0.14$ . From top: initial test image, noisy input, EDM-1K output, 1-NN of EDM-1K , EDM-50K output, 1-NN of EDM-50K , EMP-1K output, 1-NN of EMP-1K . At this low noise level, EDM-1K shows minimal memorization while EMP-1K exhibits nearly perfect nearest-neighbor behavior.

Figure 17: 2k-Baseline detailed visualization. This 8 $\times$ 8 grid shows 64 image pairs, each arranged as [Generated — 1-NN from training]. In each pair, the generated sample appears on the left and its nearest neighbor from the 2,000-image training set appears on the right . Red borders indicate memorized pairs. The high prevalence of red borders confirms severe memorization in the baseline case.

Figure 18: 2k-Dummy detailed visualization. Same layout as Figure 17 : each of the 64 pairs shows [Generated — 1-NN]. Red borders mark memorized pairs. Conditional training with an auxiliary noise class substantially reduces memorization. Most generated samples now deviate meaningfully from their nearest neighbors, though several memorized samples remain (red borders).

Figure 19: 2k-Gap detailed visualization. Same layout: 64 pairs of [Generated — 1-NN]. Gap training (excluding $\sigma\in[1.0,5.0]$ from training) achieves minimal memorization. Generated samples maintain semantic similarity to cars while introducing novel variations in color, viewpoint, and detail. This demonstrates that targeted undertraining in the medium- $\sigma$ danger zone enables robust generalization.

Figure 20: Baseline (no gap). 74.97% memorization. The high prevalence of red borders confirms severe memorization in this low-data regime.

Figure 21: Gap [100–200]. 42.93% memorization. Skipping 100 timesteps in the early danger zone reduces memorization substantially.

Figure 22: Gap [100–275]. 13.45% memorization. Extending the gap further into the danger zone dramatically reduces memorization.

Figure 23: Gap [100–350]. 9.24% memorization. The widest gap achieves the lowest memorization rate, though at the cost of increased FID (26.74 vs 13.09 baseline).

Memorization in Generative Models

Memorization in generative models studies when models retain training examples, how to detect their recall, and how to limit it without losing utility.

See all 86 papers

Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models
Oct 13, 2026Shashank Sule, Xinyue Yu, Boonam Shin +3Memorization in Generative Models Diffusion Models
Department of Mathematics, University of California, Los Angeles, CA 90095, USA. · Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA.
Beyond Sample Copying: Structural Memorization in Diffusion Models
Mar 12, 2026Tim Kaiser, Markus KollmannDiffusion Models
Heinrich-Heine-University Düsseldorf
Localizing Memorized Regions in Diffusion Models via Coordinate-Wise Curvature Differences
May 26, 2026Gwangho Kim, Sungyoon LeeMemorization in Generative Models Diffusion Models
Department of Computer Science, Hanyang University, Seoul, South Korea.

Oct 13, 2026J/K move · Enter open · S save

Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models

Shashank Sule, Xinyue Yu, Boonam Shin, Yuxuan Liu+2

Department of Mathematics, University of California, Los Angeles, CA 90095, USA. · Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA.

Memorization in Generative Models Diffusion Models

Open paper AskPDF arXiv

Understanding the mechanism behind why diffusion models memorize their training data remains an open question in generative modeling. In this paper, we examine this question from two perspectives: (a) the model-level perspective concerning the onset of memorization in a model during training, and (b) the sample-level perspective concerning the proclivity of an individual sample to be memorized by a given model. We show that the so-called input loss curvature, or the Frobenius norm of the Hessian of the diffusion model's training loss, is an effective metric for quantifying both perspectives on memorization. In particular, we empirically demonstrate that input curvatures averaged across training and validation samples can identify when a model is likely to start regurgitating its training data when tracked over training time. However, this metric is not sufficient for measuring an individual sample's likelihood of being memorized. To address this perspective, we introduce a counterfactual definition of memorization in diffusion models, extending the analogous notion from supervised learning proposed by [Feldman, 2019]. We prove that this counterfactual memorization is controlled by the input curvature of the leave-one-out model, thus establishing a direct theoretical link between a sample's local loss landscape and its generation likelihood. We validate our derived bounds through several empirical experiments and show that input-dependent curvature provides a proxy for memorization.

Abstract

Memorization in Generative Models

Explore similar work

Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models

Beyond Sample Copying: Structural Memorization in Diffusion Models

Localizing Memorized Regions in Diffusion Models via Coordinate-Wise Curvature Differences