From Modes to Memories: Characterizing the Scale-Space Dynamics of Diffusion Models

Cristina López Amado; Marco Fumero; Francesco Locatello

cs.LGSep 30, 2026

From Modes to Memories: Characterizing the Scale-Space Dynamics of Diffusion Models

Authors: Cristina López Amado, Marco Fumero, Francesco Locatello

Organizations: Institute of Science and Technology Austria (ISTA)

Abstract

Diffusion models are typically viewed as stochastic processes that transform noise into data. We take a complementary perspective: a diffusion model defines a family of deterministic dynamical systems indexed by noise scale. At each fixed scale $σ$ , we treat the denoiser as a self-map and study its dynamics. For an exact denoiser, fixed points correspond to critical points of the smoothed data density, while attractors correspond to its modes; as $σ$ increases, sample-level modes merge into progressively coarser ones. This suggests a geometric view of memorization: examples that receive excess probability mass due to duplication or overfitting, as well as outliers, should remain distinguishable under stronger smoothing than ordinary examples. We quantify this persistence by the critical scale $σ_c$ , the largest noise scale at which an example is retained by the fixed-scale dynamics. In conditional models, the same construction extends naturally to image--caption pairs. Experiments in controlled settings and on large-scale models show that $σ_c$ tracks memorization arising from duplication, overfitting, and outliers, and identifies both memorized and partially memorized examples in Stable Diffusion. Moreover, $σ_c$ yields interpretable measures of the image spatial distribution and caption dependence of memorization.

Figures & tables

Figure 1: A diffusion model as a family of dynamical systems. Fixing the noise level $\sigma$ and feeding the denoiser its own output defines a self map $M_{\sigma}$ ( left ). A memorized training image is a fixed point of this map: its orbit stays at the image, whereas the orbit of a control image drifts away and crosses the escape radius $r$ ( bottom left ). Repeating this retention test at every $\sigma$ ( right ) yields the critical scale $\sigma_{\mathrm{c}}$ , the largest noise level at which the image is still returned; it is $6.7\times$ larger for the memorized image, effectively detecting it.

Figure 2: One model, a family of dynamical systems. Orbits of the exact map

M_{\sigma}

(Eq. 1 ) for samples from an 8-mode mixture of gaussians. Left: at small

\sigma

each datum carries its own attractor. Middle: at intermediate

\sigma

the attractors are the 8 population modes. Right: at large

\sigma

a single attractor remains, at the data mean.

Figure 3: Scale space of attractors. Exact map $M_{\sigma}$ on a 2-D training set: three groups of three sub-clusters (black dots), plus one datum repeated $15$ times (red star). Top: basins of attraction at four scales, coloured by the reached attractor ( $\times$ ); arrows are the drift $M_{\sigma}(x)-x$ . Bottom left: merge tree of the attractors reached from the data (width $=$ number of data collected). Single data lose their attractor at small $\sigma$ ; the duplicated datum (red) keeps it until merging with the neighbouring sub-clusters. Bottom right: number of attractors. A denoiser trained on the same data reproduces this scale space, basins, merge tree and counts alike (Appendix C , Fig. 8 ).

Figure 4: Duplication on full CIFAR-10. (a) $\sigma_{\mathrm{c}}$ of each image, grouped by multiplicity, in the model trained with duplicates (circles) and in the released model trained on the same images without duplicates (triangles). (b) $\sigma_{\mathrm{c}}$ as a function of the iteration budget $K$ (median per group): constant for images with $m\geq 8$ , which are fixed points, and decreasing as $1/K$ for held-out and $m=1$ images, which are transients. (c) Orbits $M_{\sigma}^{k}(x)$ of a memorized training image ( $m=400$ ) and of a training image seen once ( $m=1$ ), with $\sigma$ fixed along each row and $k$ along the columns; the dashed line marks the $\sigma_{\mathrm{c}}$ of each image ( $K=50$ , $r=0.5$ ; $1.62$ and $0.14$ ). The memorized image is still reproduced after $50$ iterations at $\sigma=1$ , where the other one has already vanished.

Figure 5: Rarity: detection thresholds of

\sigma_{\mathrm{c}}

and of sampling. Fraction of added images detected as a function of their multiplicity

m

, for sources of increasing atypicality. Orange : the image appears at least once among

10^{5}

samples; the threshold is

m\approx 8

for every source. Blue :

\sigma_{\mathrm{c}}

exceeds the

95

th percentile of the images of the same source not added to the model; the threshold decreases to

m=1

as the images become more atypical. The shaded region is memorization detected by

\sigma_{\mathrm{c}}

but not by sampling.

Score	MV vs. control	MV vs. swap	TV vs. control	TV vs. swap	MV vs. TV
$\\|\epsilon_{c}-\epsilon_{\varnothing}\\|$ Wen et al. (2024)	$\mathbf{0.999}$	$0.606$	$\mathbf{0.999}$	$0.184$	$0.902$
$\\|\epsilon_{c}(x_{0})-\epsilon_{\varnothing}(x_{0})\\|$	$\mathbf{0.999}$	$0.887$	$\mathbf{0.996}$	$0.696$	$0.868$
$\sigma_{\mathrm{c}}$ , $K=16$	$0.917$	$0.904$	$0.695$	$0.803$	$0.909$
$\Delta\log\sigma_{\mathrm{c}}$ , $K=16$	$0.8569$	$\mathbf{0.908}$	$0.824$	$\mathbf{0.939}$	$0.820$
$\|\Delta\log\sigma_{\mathrm{c}}\|$ , $K=16$	$0.984$	$0.860$	$0.879$	$0.354$	$0.905$
local $\|\Delta\log\sigma_{\mathrm{c}}\|$ , $K=16$	$\mathbf{0.997}$	$0.865$	$\mathbf{0.993}$	$0.458$	$\mathbf{0.935}$

Table 1: Memorization-detection on Stable Diffusion v1.4. AUC for distinguishing MV and TV examples from controls, caption-swapped pairs, and MV from TV. Our global scores use

r=0.25

, and local

|\Delta\log\sigma_{\mathrm{c}}|

uses

r=0.1

. For both variants of Wen et al. (2024) , we report its best-performing configuration (Appendix F ).

Panel of Figure 6 — Figure 6: Critical-scale gaps reveal spatial and image-driven memorization. Left: A TV example: the training image, generations from its training caption, and the local gap $|\Delta\log\sigma_{c}^{u}|$ . The gap is large where all generations copy the training image (the room) and near zero where they vary (the curtain pattern). Right: A wrong caption exposes images already retained by the unconditional model. Center: wrong-caption gap $\Delta\log\sigma_{c}$ against unconditional inpainting RMSE (top) and unconditional $\sigma_{c}$ (bottom); bars include std deviation over 5 captions. Side: a negative-gap image is retained by unconditional dynamics but lost when conditioning on a wrong caption; unconditional inpainting restores the masked region.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Definition 1 on Stable Diffusion. Top: the map $M_{\sigma}$ . The image is encoded once into the latent $x_{0}$ ; the U-Net is evaluated with the caption at noise level $\sigma$ , and its output $x_{k}-\sigma\hat{\epsilon}$ is fed back as the next input. Bottom: each row is an orbit $(M_{\sigma}^{k}(x))_{k\geq 0}$ , decoded for display, of a memorized image under its own caption ( $w=1$ ). First row: started at the image, the orbit does not move; the image is a fixed point $x^{\ast}$ . Second row: started with part of the image replaced by a grey square, the orbit restores it and converges to the same $x^{\ast}$ . The occluded image lies in the basin $\mathcal{B}_{\sigma}(x^{\ast})$ , so $x^{\ast}$ is an attractor. Third row: the same start at a larger $\sigma$ , which defines a different map in the family $\{M_{\sigma}\}_{\sigma>0}$ . There the start is no longer in the basin, and the orbit leaves.

Figure 8: A trained denoiser reproduces the scale space of Fig. 3 . Same training set and same panels as Fig. 3 , with the exact map $M_{\sigma}$ replaced by $M_{\sigma}(x)=D_{\sigma}(x)$ for an EDM-preconditioned MLP trained on the $141$ points. Top: basins at the four scales of Fig. 3 , coloured by the attractor ( $\times$ ) reached from the data; grey marks the grid points that reach none of them, in the corners of the domain where the network saw no training data. Bottom left: merge tree of the trained map. Bottom right: its attractor count against that of the exact map. The two maps agree: the counts are equal at $80\%$ of the $90$ scales, the partitions of the training set they induce have adjusted Rand index $0.99$ on average ( $0.76$ at worst, at the smallest scales where single data split), and the duplicated datum keeps its attractor over the same range of scales in both.

Figure 9: The exact denoiser follows the law.

\sigma_{\mathrm{c}}^{\mathrm{emp}}

of the added images of the rarity experiment against their kernel distance

d_{\mathrm{kern}}

to the training set (Eq. 28 ), for images added once (grey) and

32

times (red), under the escape test with

K=1

r=0.2

. The lines are the law

\sigma_{\mathrm{c}}=d_{\mathrm{kern}}/\sqrt{2\log(N/m)}

of Theorem 5 , drawn without fitting. Doubling the distance doubles the critical scale, whereas

32

copies increase it by a factor

1.26

intervention	group	$\sigma_{\mathrm{c}}^{\mathrm{emp}}$	$\sigma_{\mathrm{c}}$	$\kappa$
duplication	$m=1$	$4.99$	$0.142$	$0.028$
	$m=8$	$5.60$	$1.036$	$0.194$
	$m=800$	$9.91$	$1.431$	$0.145$
	$m=800$ , control model	$9.91$	$0.269$	$0.024$
overfitting	$N=5\cdot 10^{4}$	$4.93$	$0.056$	$0.012$
	$N=2000$	$6.11$	$0.243$	$0.043$

Appendix

Table 2: The three interventions on a common scale.

\sigma_{\mathrm{c}}^{\mathrm{emp}}

is the critical scale of the exact empirical denoiser of the training set of each model under the same escape test, equation 40 ;

\kappa=\sigma_{\mathrm{c}}/\sigma_{\mathrm{c}}^{\mathrm{emp}}

is the fraction of this solution that the network implements. Group medians, training images only: the ceiling varies little, and the effect is carried by

\kappa

Figure 10: Orbits across scales, one image per group. Orbits of the model trained with duplicates, with $\sigma$ fixed along each row and the iteration index $k$ along each column, for a memorized image ( $m=800$ ), a training image with $m=1$ and a held-out image, each chosen at the median $\sigma_{\mathrm{c}}$ of its group. The dashed line marks the $\sigma_{\mathrm{c}}$ of the image ( $K=64$ , $r=0.5$ ). The memorized image is unchanged after $64$ iterations at $\sigma=1.43$ , whereas the other two converge to a uniform colour at $\sigma=0.24$ , with nearly identical critical scales ( $0.122$ and $0.118$ ).

Figure 11: Overfitting: the same images across training-set sizes. (a)

\sigma_{\mathrm{c}}

of the

100

training images shared by all models (blue) and of

100

held-out images (orange): medians with interquartile range, individual images in the background. Images that escape at the lower end of the bisection interval are drawn at that end, so the held-out medians for

N\leq 500

are upper bounds. (b) Fraction of generated samples that are copies (orange, left axis; when no copy is found the point is drawn at the

95\%

upper bound) and AUC of

\sigma_{\mathrm{c}}

between training and held-out images (blue, right axis). At

N=2000

sampling no longer detects memorization while

\sigma_{\mathrm{c}}

still does.

Figure 12: Paired comparison on three images that are never sampled. One added image per source, chosen at the median of its group. Each row shows $M_{\sigma}(x)$ , one denoising step applied to the clean image at noise level $\sigma$ , in the model trained on the image (top, blue) and in the model not trained on it (bottom, grey), together with the relative displacement $\|M_{\sigma}(x)-x\|/\|x\|$ whose crossing of $r$ defines $\sigma_{\mathrm{c}}$ . The model not trained on the image moves it towards what it has learned (blur, generic texture) at a much smaller $\sigma$ .

Panel of Figure 13 — Figure 13: Examples removed when filtering the memorization dataset. (a) Each row shows a group of duplicate examples, of which we retain only one. (b) The first image in each row is the stored training image, followed by three generations from its training caption. We remove cases in which the caption does not reproduce the stored image.

Figure 14: The full $(\sigma,k)$ grid behind Figure 1 : the critical scale on Stable Diffusion v1.4, for the same memorized caption–image pair (left) and control pair (middle), each run with its own caption at guidance scale $1$ . Each row iterates $M_{\sigma}$ from the image $x$ at one noise scale $\sigma$ ; a framed iterate $M_{\sigma}^{k}(x)$ is still within $r\|x\|$ of $x$ . At small $\sigma$ the model keeps returning $x$ , which therefore sits in a basin of its own; at large $\sigma$ the orbit drifts away towards content shared with other data. The critical scale $\sigma_{\mathrm{c}}$ (dashed line) separates the two regimes: the memorized image withstands far more noise than the control before it escapes. Right: the drift $\|M_{\sigma}^{k}(x)-x\|/\|x\|$ along the orbit, one curve per scale coloured by $\sigma$ ; the thick curve is a scale above $\sigma_{\mathrm{c}}$ and the dot marks where it crosses $r$ . Here $K=50$ and $r=0.5$ .

Figure 15: Ablation on the number of iterations

\bm{K}

. Detection performance as a function of

K

for escape radii

r=0.1

and

r=0.25

. The first four columns report AUC for the different critical-scale scores; the final column reports the mean critical scale (solid line)

\pm

one standard deviation (shaded region). Performance is stable around the primary choice

K=16,r=0.25

Figure 16: Ablation on the escape radius

r

. Detection performance (AUC) as a function of

r

using

K=16

. For this ablation,

\sigma_{c}

is approximated over a logarithmic grid of 24 noise levels rather than estimated by log-space bisection. Performance is stable around the primary choice

K=16,r=0.25

(global measures),

r=0.1

(local

|\Delta\log\sigma_{c}|

Panel of Figure 17 — Figure 17: Critical-scale score distributions across evaluation groups. Histograms of the scores used to separate the different groups. Left: Local $|\Delta\log\sigma_{c}|$ for MV, TV, and control examples ( $K=16$ , $r=0.1$ ). Center-Right: Signed $\Delta\log\sigma_{c}$ for MV and TV pairs with their original and swapped captions ( $K=16$ , $r=0.25$ ). Swapped pairs frequently have negative gaps.

Panel of Figure 18 — Figure 18: Additional examples of localized memorization. Each panel shows a TV training image, generations from its caption, and the corresponding local $|\Delta\log\sigma_{c}^{u}|$ map. Large gaps align with regions reproduced consistently across generations, whereas variable regions generally have gaps near zero.

Panel of Figure 19 — Figure 19: Near-zero gaps are ambiguous. The two examples have similar mismatched-caption gaps but different absolute retention. Top: Both branches retain the image, and unconditional reconstruction succeeds. Bottom: Neither branch retains the image, and unconditional reconstruction fails.

Panel of Figure 20 — Figure 20: Additional examples with negative mismatched-caption gaps. Both examples exhibit unconditional retention and successful inpainting. The more negative gap in the top example corresponds to more faithful reconstructions than the less negative gap in the bottom example.

Panel of Figure 21 — Figure 21: Trajectory and critical scale MV vs control. Top: critical scale on for a totally memorized caption–image pair ( left ) and a control pair ( right ), each run with its own caption at guidance scale $1$ . Each row iterates $M_{\sigma}$ from the image $x$ at one noise scale $\sigma$ ; a framed iterate $M_{\sigma}^{k}(x)$ is still within $r\|x\|$ of $x$ . At small $\sigma$ the model keeps returning $x$ , which therefore sits in a basin of its own; at large $\sigma$ the orbit drifts away towards content shared with other data. The critical scale $\sigma_{\mathrm{c}}$ (dashed colored line) separates the two regimes: the memorized image withstands far more noise than the control before it escapes. Bottom: the drift $\|M_{\sigma}^{k}(x)-x\|/\|x\|$ along the orbit, one curve per scale coloured by $\sigma$ ; the thick curve is a scale above $\sigma_{\mathrm{c}}$ and the dot marks where it crosses $r$ .

Panel of Figure 22 — Figure 22: Trajectory and critical scale TV vs control. Top: critical scale on for a partially memorized caption–image pair ( left ) and a control pair ( right ), each run with its own caption at guidance scale $1$ . Each row iterates $M_{\sigma}$ from the image $x$ at one noise scale $\sigma$ ; a framed iterate $M_{\sigma}^{k}(x)$ is still within $r\|x\|$ of $x$ . At small $\sigma$ the model keeps returning $x$ , which therefore sits in a basin of its own; at large $\sigma$ the orbit drifts away towards content shared with other data. The critical scale $\sigma_{\mathrm{c}}$ (dashed colored line) separates the two regimes: the memorized image withstands far more noise than the control before it escapes. Bottom: the drift $\|M_{\sigma}^{k}(x)-x\|/\|x\|$ along the orbit, one curve per scale coloured by $\sigma$ ; the thick curve is a scale above $\sigma_{\mathrm{c}}$ and the dot marks where it crosses $r$ .

Panel of Figure 23 — Figure 23: Trajectory and critical scale MV vs TV. Top: critical scale on for a totally memorized caption–image pair ( left ) and a partially memorized pair ( right ), each run with its own caption at guidance scale $1$ . Each row iterates $M_{\sigma}$ from the image $x$ at one noise scale $\sigma$ ; a framed iterate $M_{\sigma}^{k}(x)$ is still within $r\|x\|$ of $x$ . At small $\sigma$ the model keeps returning $x$ , which therefore sits in a basin of its own; at large $\sigma$ the orbit drifts away towards content shared with other data. The critical scale $\sigma_{\mathrm{c}}$ (dashed colored line) separates the two regimes: the memorized image withstands far more noise than the control before it escapes. Bottom: the drift $\|M_{\sigma}^{k}(x)-x\|/\|x\|$ along the orbit, one curve per scale coloured by $\sigma$ ; the thick curve is a scale above $\sigma_{\mathrm{c}}$ and the dot marks where it crosses $r$ .

Panel of Figure 24 — Figure 24: $\bm{\Delta\log\sigma_{c}}$ for MV and control images. The top row shows an MV image–caption pair and the bottom row a control. The left and center panels show fixed-scale trajectories across noise levels under training-caption and unconditional conditioning, respectively. The right panels plot $\max_{k\leq K}||M_{\sigma}^{k}(x)-x||/||x||$ for both branches. Their crossings with the dotted threshold $r$ define the conditional and unconditional critical scales, whose log difference is $\Delta\log\sigma_{c}$ . The MV pair has a large gap because its caption substantially increases retention, whereas the control has nearly equal critical scales and a gap close to zero.

Panel of Figure 25 — Figure 25: Fixed-scale trajectories across noise levels. Each comparison shows the iterate $M_{\sigma}^{K}(x)$ across noise scales (top) and the maximum normalized displacement $\max_{k\leq K}|M_{\sigma}^{k}(x)-x|/|x|$ as a function of $\sigma$ (bottom). The dotted line marks the escape radius $r$ ; its crossing determines $\sigma_{c}$ . The upper panel compares an MV, a TV, and a control image, ordered from longest to shortest retention. The lower panel compares two TV images with different amounts of memorized content. The image with more memorized content is retained longer.

Explore similar work

May 26, 2026cs.LG

Localizing Memorized Regions in Diffusion Models via Coordinate-Wise Curvature Differences

Diffusion models can unintentionally memorize training samples, raising concerns about privacy and copyright. While recent methods can detect memorization, they often rely on global or model-specific signals and provide limited insight into where memorization appears within a generated image. We provide a geometric characterization of local memorization as a coordinate-wise variance collapse. However, such collapse can also arise from intrinsic data constraints rather than overfitting. To isolate overfitting-driven memorization, we propose curvature-difference methods that subtract the curvature of an underfitted baseline, either the unconditional model or a less-trained version of itself. We further derive a score-difference proxy that provides a geometric explanation for the widely used score-difference-based detection metric. Experiments on Stable Diffusion, evaluated against ground-truth memorization masks, show that our method outperforms the prior attention-based localization method. Code is available at https://github.com/Gwangho99/mem-curv-diff.

Gwangho Kim, Sungyoon Lee

Department of Computer Science, Hanyang University, Seoul, South Korea.

Diffusion Models Localization

Ask PDF

May 21, 2026cs.CV

Broken Memories: Detecting and Mitigating Memorization in Diffusion Models with Degraded Generations

While diffusion models excel at generating high-quality images, their tendency to memorize training data poses significant privacy and copyright risks. In this work, we for the first time identify that memorization induces internal numerical instability, often manifesting as visually ``broken'' artifacts. Inspired by stability analysis in numerical methods, we introduce empirical stability regions based on latent update norms to quantitatively characterize stable behavior during generation. Leveraging this, we propose a principled, on-the-fly framework for step-wise detection and adaptive mitigation. Our approach suppresses memorization without altering prompts or guidance, thereby preserving semantic fidelity and image quality. Extensive experiments on Stable Diffusion 1.4 demonstrate that our method achieves an AUC

>0.999

detection performance and a

0.0\%

memorization rate after mitigation with negligible overhead (

\approx0.01

s per image).

Yuanmin Huang, Mi Zhang, Chen Chen +4

Fudan University Shanghai, China · East China University of Science and Technology Shanghai, China

Diffusion Models Memorization High-Resolution

Ask PDF

May 28, 2026cs.LG

Diffusion Models Preferentially Memorize Prototypical Examples or: Why Does My Diffusion Model Love Slop?

Generative models have a persistent limitation: their tendency to memorize training data can create legal liabilities and erode creative diversity. Understanding which samples are memorized in whole or in part, and under what conditions, therefore remains an important open problem. Here we answer the question "Are atypical or rare samples memorized first?" in the negative. We train diffusion models on strings generated according to the production rules of the Random Hierarchy Model (RHM), and find that samples composed of common substrings are preferentially memorized. This holds true even if the training data consists of entirely unique samples, indicating that deduplication at the data point level does not provide a meaningful privacy guarantee. Correspondingly we predict, then observe, delayed memorization for fat-tailed datasets (i.e., those with more atypical samples). This effect is amplified when fat-tails are introduced into high-level production rules. These together suggest that dataset diversity, particularly at higher levels of abstraction, plays an important role in staving off memorization. Finally, we identify an intermediate regime of partial memorization in which common substrings are learned first and subsequently overproduced during generation. If training is stopped in this regime, models will exhibit the reversion-to-the-mean blandness often derided as "slop".

Marta Aparicio Rodriguez, Anastasia Borovykh, Grigorios A. Pavliotis +1

Department of Mathematics, Imperial College London, UK · ML Lab, Capital Fund Management, France · Department of Physics, École Polytechnique Fédérale de Lausanne (EPFL), Switzerland

Diffusion Models Memorization Generative Models Long-Tailed Distribution Rare Events

Ask PDF