Text-to-image diffusion models often memorize training data, revealing a fundamental failure to generalize beyond the training set. Current mitigation strategies typically sacrifice image quality or prompt alignment to reduce memorization. To address this, we propose Reachability-Aware Diffusion Steering (RADS), an inference-time framework that prevents memorization while preserving generation fidelity. RADS models the diffusion denoising process as a dynamical system and applies concepts from reachability analysis to approximate the "backward reachable tube"--the set of intermediate states that inevitably evolve into memorized samples. We then formulate mitigation as a constrained reinforcement learning (RL) problem, where a policy learns to steer the trajectory away from memorization via minimal perturbations in the caption embedding space. Empirical evaluations show that RADS achieves a superior Pareto frontier between generation diversity (SSCD), quality (FID), and alignment (CLIP) compared to state-of-the-art baselines. Crucially, RADS provides robust mitigation without modifying the diffusion backbone, offering a plug-and-play solution for safe generation. Our website is available at: https://s-karnik.github.io/rads-memorization-project-page/.
Figures & tables
Figure 1: rads outperforms prior work. Generated images for the prompt Design Art Beautiful View of Paris Paris Eiffel Towerunder Red Sky Ultra Glossy Cityscape Circle Wall Art . (a) Memorized target image from the training set. (b) Mitigated results produced by prior methods; 1 – 4 correspond to Wen et al. (2024) ; Ren et al. (2024) ; Hintersdorf et al. (2024) ; Jain et al. (2025) , respectively. (c) Mitigated result produced by rads (ours).
Figure 2: Overview of rads . This diagram illustrates a real example of how rads prevents memorization in the Stable Diffusion v1.4 model, while the baseline (no mitigation) generates an image that closely resembles the training sample. rads does so by modeling the “attraction basin” of memorization as a backward reachable tube (BRT) and learning a policy πϕ for steering the caption embedding inputs to the diffusion model using reachability-constrained reinforcement learning.
Figure 3: Memorization happens early. Images generated with text guidance enabled for only the first k steps ( k∈{0,1,2} ). Just 2 steps of guidance (c) are sufficient to reproduce the memorized image (d). Caption: The No Limits Business Woman Podcast .
Figure 4: Pareto Frontier Analysis: Quality and Alignment vs. Replication. We compare rads (Ours) against various state-of-the-art mitigation methods on the Webster (2023) dataset. The top row shows Quality ( −log10(FID)↑ ), while the bottom row displays Alignment (CLIP Score ↑ ). The x-axis shows the (1−SSCDtarget)↑ scores. rads consistently occupies the upper-right region of the frontier, maintaining high utility and semantic alignment while significantly reducing memorization.
Figure 5: Jain et al. (2025) produces mitigated samples that closely resemble one another. Generated images using the same initial latent xT . (a) Generated image without text guidance ( g=0 ). (b–d) Generated images produced using different prompts.
Figure 6: Zero-Shot Generalization. We compare rads (Ours) in a zero-shot evaluation against state-of-the-art mitigation methods on the MemBench dataset ( Hong et al., 2024 ) . The top row shows Quality ( −log10(FID)↑ ), while the bottom row displays Alignment (CLIP Score ↑ ). The x-axis shows the (1−SSCDtarget)↑ scores. rads consistently occupies the upper-right region of the frontier, maintaining high utility and semantic alignment while significantly reducing memorization. Note: The Hintersdorf et al. (2024) baseline is omitted due to its computational infeasibility (see Appendix A.6 ).
Figure 7: rads achieves mitigation with the highest generation diversity. Generated images for the prompt Michael Fassbender to Star In <i>Assassin’s Creed</i> Movie .
Figure 8: rads achieves mitigation on challenging prompts. Generated images for the prompt <em>Bloodborne</em> Video: Sony Explains the Game’s Procedurally Generated Dungeons .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: rads achieves mitigation with high image quality. Generated images for the prompt Anna Kendrick is Writing a Collection of Funny, Personal Essays . (a) Generated image without mitigation. (b-e) Mitigated results produced by prior methods. (f) Mitigated result produced by rads (ours).
Figure 10: rads can achieve mitigation with better alignment to the generation intent. Generated images for the prompt Shaw Floors Value Collections Fyc Tt II Net Dinner With Friends (t) 732T_5E022 . (a) Generated image without mitigation. (b-e) Mitigated results produced by prior methods. (f) Mitigated result produced by rads (ours).
Table 1: RADS strikes the best Pareto frontier between image diversity, quality, and alignment. We compare rads with prior methods and a no-mitigation baseline across different datasets, models, and denoising samplers. Lower indicates better performance for SSCDseeds , SSCDprompts , and FID , while higher indicates better performance for CLIP . Best results are in bold. While Jain et al. (2025) achieves the lowest SSCDseeds , it suffers from severe quality degradation, as evidenced by the FID and CLIP scores.
Figure 11: Dynamics of Memorization Steering. Guidance L2 -norm during denoising. Standard diffusion (blue) spikes as it collapses into the memorization basin. rads (orange) steers away from this peak.
Method
Time (s)
No mitigation
2.2950 ± 0.1580
Wen et al. (2024)
2.8983 ± 0.0459
Ren et al. (2024)
2.7544 ± 0.0404
Hintersdorf et al. (2024)
104.0895 ± 48.9586
Jain et al. (2025)
2.2322 ± 0.0368
rads (ours)
2.9262 ± 0.0335
Appendix
Table 2: Average Inference Time Comparison
Figure 12: Loss of Semantic Alignment. Generated images for the prompt Mothers influence on her young hippo . (a) Generated image without mitigation. (b-e) Mitigated results produced by rads (ours).
Memorization in generative models studies when models retain training examples, how to detect their recall, and how to limit it without losing utility.
Text-to-image diffusion models have achieved remarkable progress in image synthesis, yet can exhibit memorization by closely reproducing individual training examples. Effective mitigation must preserve useful prompt information to guide alternative depictions. We introduce a training-free method that redistributes cross-attention with Gaussian smoothing before reinforcing content-token contributions and attenuating padding contributions, without additional denoiser evaluations. With this intervention, stronger content conditioning can improve prompt alignment at comparable training-image similarity. A local analysis identifies when reinforcement preserves shared value information while redistribution reduces localized attention mass. On Stable Diffusion v1.4 and v2.0, all evaluated smoothing widths lie on the empirical Pareto frontiers for training-image similarity versus both prompt alignment and image preference. A configuration selected on Stable Diffusion reduces template reproduction in DeepFloyd IF without further tuning. These findings support jointly controlling conditioning allocation and strength to generate prompt-consistent alternatives.
Generative models have been shown to "memorize" certain training data, leading to verbatim or near-verbatim generating images, which may cause privacy concerns or copyright infringement. We introduce Guidance Using Attractive-Repulsive Dynamics (GUARD), a novel framework for memorization mitigation in text-to-image diffusion models. GUARD adjusts the image denoising process to guide the generation away from an original training image and towards one that is distinct from training data while remaining aligned with the prompt, guarding against reproducing training data, without hurting image generation quality. We propose a concrete instantiation of this framework, where the positive target that we steer towards is given by a novel method for (cross) attention attenuation based on (i) a novel statistical mechanism that automatically identifies the prompt positions where cross attention must be attenuated and (ii) attenuating cross-attention in these per-prompt locations. The resulting GUARD offers a surgical, dynamic per-prompt inference-time approach that, we find, is by far the most robust method in terms of consistently producing state-of-the-art results for memorization mitigation across two architectures and for both verbatim and template memorization, while also improving upon or yielding comparable results in terms of image quality.
Kairan Zhao, Eleni Triantafillou, Peter Triantafillou
While diffusion models excel at generating high-quality images, their tendency to memorize training data poses significant privacy and copyright risks. In this work, we for the first time identify that memorization induces internal numerical instability, often manifesting as visually ``broken'' artifacts. Inspired by stability analysis in numerical methods, we introduce empirical stability regions based on latent update norms to quantitatively characterize stable behavior during generation. Leveraging this, we propose a principled, on-the-fly framework for step-wise detection and adaptive mitigation. Our approach suppresses memorization without altering prompts or guidance, thereby preserving semantic fidelity and image quality. Extensive experiments on Stable Diffusion 1.4 demonstrate that our method achieves an AUC >0.999 detection performance and a 0.0% memorization rate after mitigation with negligible overhead (≈0.01s per image).
Yuanmin Huang, Mi Zhang, Chen Chen +4
Fudan University Shanghai, China · East China University of Science and Technology Shanghai, China