cs.LGSep 30, 2026
SaveFrom Modes to Memories: Characterizing the Scale-Space Dynamics of Diffusion Models
Organizations: Institute of Science and Technology Austria (ISTA)
Abstract
Diffusion models are typically viewed as stochastic processes that transform noise into data. We take a complementary perspective: a diffusion model defines a family of deterministic dynamical systems indexed by noise scale. At each fixed scale , we treat the denoiser as a self-map and study its dynamics. For an exact denoiser, fixed points correspond to critical points of the smoothed data density, while attractors correspond to its modes; as increases, sample-level modes merge into progressively coarser ones. This suggests a geometric view of memorization: examples that receive excess probability mass due to duplication or overfitting, as well as outliers, should remain distinguishable under stronger smoothing than ordinary examples. We quantify this persistence by the critical scale , the largest noise scale at which an example is retained by the fixed-scale dynamics. In conditional models, the same construction extends naturally to image--caption pairs. Experiments in controlled settings and on large-scale models show that tracks memorization arising from duplication, overfitting, and outliers, and identifies both memorized and partially memorized examples in Stable Diffusion. Moreover, yields interpretable measures of the image spatial distribution and caption dependence of memorization.
Figures & tables
Figure 1: A diffusion model as a family of dynamical systems. Fixing the noise level and feeding the denoiser its own output defines a self map ( left ). A memorized training image is a fixed point of this map: its orbit stays at the image, whereas the orbit of a control image drifts away and crosses the escape radius ( bottom left ). Repeating this retention test at every ( right ) yields the critical scale , the largest noise level at which the image is still returned; it is larger for the memorized image, effectively detecting it.
Figure 2: One model, a family of dynamical systems. Orbits of the exact map (Eq. 1 ) for samples from an 8-mode mixture of gaussians. Left: at small each datum carries its own attractor. Middle: at intermediate the attractors are the 8 population modes. Right: at large a single attractor remains, at the data mean.
Figure 3: Scale space of attractors. Exact map on a 2-D training set: three groups of three sub-clusters (black dots), plus one datum repeated times (red star). Top: basins of attraction at four scales, coloured by the reached attractor ( ); arrows are the drift . Bottom left: merge tree of the attractors reached from the data (width number of data collected). Single data lose their attractor at small ; the duplicated datum (red) keeps it until merging with the neighbouring sub-clusters. Bottom right: number of attractors. A denoiser trained on the same data reproduces this scale space, basins, merge tree and counts alike (Appendix C , Fig. 8 ).
Figure 4: Duplication on full CIFAR-10. (a) of each image, grouped by multiplicity, in the model trained with duplicates (circles) and in the released model trained on the same images without duplicates (triangles). (b) as a function of the iteration budget (median per group): constant for images with , which are fixed points, and decreasing as for held-out and images, which are transients. (c) Orbits of a memorized training image ( ) and of a training image seen once ( ), with fixed along each row and along the columns; the dashed line marks the of each image ( , ; and ). The memorized image is still reproduced after iterations at , where the other one has already vanished.
Figure 5: Rarity: detection thresholds of and of sampling. Fraction of added images detected as a function of their multiplicity , for sources of increasing atypicality. Orange : the image appears at least once among samples; the threshold is for every source. Blue : exceeds the th percentile of the images of the same source not added to the model; the threshold decreases to as the images become more atypical. The shaded region is memorization detected by but not by sampling.
| Score | MV vs. control | MV vs. swap | TV vs. control | TV vs. swap | MV vs. TV |
|---|---|---|---|---|---|
| Wen et al. (2024) | |||||
| , | |||||
| , | |||||
| , | |||||
| local , |
Table 1: Memorization-detection on Stable Diffusion v1.4. AUC for distinguishing MV and TV examples from controls, caption-swapped pairs, and MV from TV. Our global scores use , and local uses . For both variants of Wen et al. (2024) , we report its best-performing configuration (Appendix F ).
Figure 6: Critical-scale gaps reveal spatial and image-driven memorization. Left: A TV example: the training image, generations from its training caption, and the local gap . The gap is large where all generations copy the training image (the room) and near zero where they vary (the curtain pattern). Right: A wrong caption exposes images already retained by the unconditional model. Center: wrong-caption gap against unconditional inpainting RMSE (top) and unconditional (bottom); bars include std deviation over 5 captions. Side: a negative-gap image is retained by unconditional dynamics but lost when conditioning on a wrong caption; unconditional inpainting restores the masked region.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Definition 1 on Stable Diffusion. Top: the map . The image is encoded once into the latent ; the U-Net is evaluated with the caption at noise level , and its output is fed back as the next input. Bottom: each row is an orbit , decoded for display, of a memorized image under its own caption ( ). First row: started at the image, the orbit does not move; the image is a fixed point . Second row: started with part of the image replaced by a grey square, the orbit restores it and converges to the same . The occluded image lies in the basin , so is an attractor. Third row: the same start at a larger , which defines a different map in the family . There the start is no longer in the basin, and the orbit leaves.
Figure 8: A trained denoiser reproduces the scale space of Fig. 3 . Same training set and same panels as Fig. 3 , with the exact map replaced by for an EDM-preconditioned MLP trained on the points. Top: basins at the four scales of Fig. 3 , coloured by the attractor ( ) reached from the data; grey marks the grid points that reach none of them, in the corners of the domain where the network saw no training data. Bottom left: merge tree of the trained map. Bottom right: its attractor count against that of the exact map. The two maps agree: the counts are equal at of the scales, the partitions of the training set they induce have adjusted Rand index on average ( at worst, at the smallest scales where single data split), and the duplicated datum keeps its attractor over the same range of scales in both.
Figure 9: The exact denoiser follows the law. of the added images of the rarity experiment against their kernel distance to the training set (Eq. 28 ), for images added once (grey) and times (red), under the escape test with , . The lines are the law of Theorem 5 , drawn without fitting. Doubling the distance doubles the critical scale, whereas copies increase it by a factor .
| intervention | group | |||
|---|---|---|---|---|
| duplication | ||||
| , control model | ||||
| overfitting | ||||
Table 2: The three interventions on a common scale. is the critical scale of the exact empirical denoiser of the training set of each model under the same escape test, equation 40 ; is the fraction of this solution that the network implements. Group medians, training images only: the ceiling varies little, and the effect is carried by .
Figure 10: Orbits across scales, one image per group. Orbits of the model trained with duplicates, with fixed along each row and the iteration index along each column, for a memorized image ( ), a training image with and a held-out image, each chosen at the median of its group. The dashed line marks the of the image ( , ). The memorized image is unchanged after iterations at , whereas the other two converge to a uniform colour at , with nearly identical critical scales ( and ).
Figure 11: Overfitting: the same images across training-set sizes. (a) of the training images shared by all models (blue) and of held-out images (orange): medians with interquartile range, individual images in the background. Images that escape at the lower end of the bisection interval are drawn at that end, so the held-out medians for are upper bounds. (b) Fraction of generated samples that are copies (orange, left axis; when no copy is found the point is drawn at the upper bound) and AUC of between training and held-out images (blue, right axis). At sampling no longer detects memorization while still does.
Figure 12: Paired comparison on three images that are never sampled. One added image per source, chosen at the median of its group. Each row shows , one denoising step applied to the clean image at noise level , in the model trained on the image (top, blue) and in the model not trained on it (bottom, grey), together with the relative displacement whose crossing of defines . The model not trained on the image moves it towards what it has learned (blur, generic texture) at a much smaller .
Figure 13: Examples removed when filtering the memorization dataset. (a) Each row shows a group of duplicate examples, of which we retain only one. (b) The first image in each row is the stored training image, followed by three generations from its training caption. We remove cases in which the caption does not reproduce the stored image.
Figure 14: The full grid behind Figure 1 : the critical scale on Stable Diffusion v1.4, for the same memorized caption–image pair (left) and control pair (middle), each run with its own caption at guidance scale . Each row iterates from the image at one noise scale ; a framed iterate is still within of . At small the model keeps returning , which therefore sits in a basin of its own; at large the orbit drifts away towards content shared with other data. The critical scale (dashed line) separates the two regimes: the memorized image withstands far more noise than the control before it escapes. Right: the drift along the orbit, one curve per scale coloured by ; the thick curve is a scale above and the dot marks where it crosses . Here and .
Figure 15: Ablation on the number of iterations . Detection performance as a function of for escape radii and . The first four columns report AUC for the different critical-scale scores; the final column reports the mean critical scale (solid line) one standard deviation (shaded region). Performance is stable around the primary choice .
Figure 16: Ablation on the escape radius . Detection performance (AUC) as a function of using . For this ablation, is approximated over a logarithmic grid of 24 noise levels rather than estimated by log-space bisection. Performance is stable around the primary choice (global measures), (local ).
Figure 17: Critical-scale score distributions across evaluation groups. Histograms of the scores used to separate the different groups. Left: Local for MV, TV, and control examples ( , ). Center-Right: Signed for MV and TV pairs with their original and swapped captions ( , ). Swapped pairs frequently have negative gaps.
Figure 18: Additional examples of localized memorization. Each panel shows a TV training image, generations from its caption, and the corresponding local map. Large gaps align with regions reproduced consistently across generations, whereas variable regions generally have gaps near zero.
Figure 19: Near-zero gaps are ambiguous. The two examples have similar mismatched-caption gaps but different absolute retention. Top: Both branches retain the image, and unconditional reconstruction succeeds. Bottom: Neither branch retains the image, and unconditional reconstruction fails.
Figure 20: Additional examples with negative mismatched-caption gaps. Both examples exhibit unconditional retention and successful inpainting. The more negative gap in the top example corresponds to more faithful reconstructions than the less negative gap in the bottom example.
Figure 21: Trajectory and critical scale MV vs control. Top: critical scale on for a totally memorized caption–image pair ( left ) and a control pair ( right ), each run with its own caption at guidance scale . Each row iterates from the image at one noise scale ; a framed iterate is still within of . At small the model keeps returning , which therefore sits in a basin of its own; at large the orbit drifts away towards content shared with other data. The critical scale (dashed colored line) separates the two regimes: the memorized image withstands far more noise than the control before it escapes. Bottom: the drift along the orbit, one curve per scale coloured by ; the thick curve is a scale above and the dot marks where it crosses .
Figure 22: Trajectory and critical scale TV vs control. Top: critical scale on for a partially memorized caption–image pair ( left ) and a control pair ( right ), each run with its own caption at guidance scale . Each row iterates from the image at one noise scale ; a framed iterate is still within of . At small the model keeps returning , which therefore sits in a basin of its own; at large the orbit drifts away towards content shared with other data. The critical scale (dashed colored line) separates the two regimes: the memorized image withstands far more noise than the control before it escapes. Bottom: the drift along the orbit, one curve per scale coloured by ; the thick curve is a scale above and the dot marks where it crosses .
Figure 23: Trajectory and critical scale MV vs TV. Top: critical scale on for a totally memorized caption–image pair ( left ) and a partially memorized pair ( right ), each run with its own caption at guidance scale . Each row iterates from the image at one noise scale ; a framed iterate is still within of . At small the model keeps returning , which therefore sits in a basin of its own; at large the orbit drifts away towards content shared with other data. The critical scale (dashed colored line) separates the two regimes: the memorized image withstands far more noise than the control before it escapes. Bottom: the drift along the orbit, one curve per scale coloured by ; the thick curve is a scale above and the dot marks where it crosses .
Figure 24: for MV and control images. The top row shows an MV image–caption pair and the bottom row a control. The left and center panels show fixed-scale trajectories across noise levels under training-caption and unconditional conditioning, respectively. The right panels plot for both branches. Their crossings with the dotted threshold define the conditional and unconditional critical scales, whose log difference is . The MV pair has a large gap because its caption substantially increases retention, whereas the control has nearly equal critical scales and a gap close to zero.
Figure 25: Fixed-scale trajectories across noise levels. Each comparison shows the iterate across noise scales (top) and the maximum normalized displacement as a function of (bottom). The dotted line marks the escape radius ; its crossing determines . The upper panel compares an MV, a TV, and a control image, ordered from longest to shortest retention. The lower panel compares two TV images with different amounts of memorized content. The image with more memorized content is retained longer.