cs.CVFeb 24, 2026

Steering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion

Authors: Sathwik Karnik, Juyeop Kim, Sanmi Koyejo, Jong-Seok Lee, Somil Bansal

Organizations: Stanford University, Stanford, CA, USA · Yonsei University, Seoul, Korea

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Text-to-image diffusion models often memorize training data, revealing a fundamental failure to generalize beyond the training set. Current mitigation strategies typically sacrifice image quality or prompt alignment to reduce memorization. To address this, we propose Reachability-Aware Diffusion Steering (RADS), an inference-time framework that prevents memorization while preserving generation fidelity. RADS models the diffusion denoising process as a dynamical system and applies concepts from reachability analysis to approximate the "backward reachable tube"--the set of intermediate states that inevitably evolve into memorized samples. We then formulate mitigation as a constrained reinforcement learning (RL) problem, where a policy learns to steer the trajectory away from memorization via minimal perturbations in the caption embedding space. Empirical evaluations show that RADS achieves a superior Pareto frontier between generation diversity (SSCD), quality (FID), and alignment (CLIP) compared to state-of-the-art baselines. Crucially, RADS provides robust mitigation without modifying the diffusion backbone, offering a plug-and-play solution for safe generation. Our website is available at: https://s-karnik.github.io/rads-memorization-project-page/.

Figures & tables

Figure 1: rads outperforms prior work. Generated images for the prompt Design Art Beautiful View of Paris Paris Eiffel Towerunder Red Sky Ultra Glossy Cityscape Circle Wall Art . (a) Memorized target image from the training set. (b) Mitigated results produced by prior methods; 1 – 4 correspond to Wen et al. (2024) ; Ren et al. (2024) ; Hintersdorf et al. (2024) ; Jain et al. (2025) , respectively. (c) Mitigated result produced by rads (ours).
Figure 2: Overview of rads . This diagram illustrates a real example of how rads prevents memorization in the Stable Diffusion v1.4 model, while the baseline (no mitigation) generates an image that closely resembles the training sample. rads does so by modeling the “attraction basin” of memorization as a backward reachable tube (BRT) and learning a policy πϕ\pi_{\phi} for steering the caption embedding inputs to the diffusion model using reachability-constrained reinforcement learning.
Figure 3: Memorization happens early. Images generated with text guidance enabled for only the first kk steps ( k∈{0,1,2}k\in\{0,1,2\} ). Just 2 steps of guidance (c) are sufficient to reproduce the memorized image (d). Caption: The No Limits Business Woman Podcast .
Figure 4: Pareto Frontier Analysis: Quality and Alignment vs. Replication. We compare rads (Ours) against various state-of-the-art mitigation methods on the Webster (2023) dataset. The top row shows Quality ( −log⁡10(FID)↑-\log_{10}(\mathrm{FID})\uparrow ), while the bottom row displays Alignment (CLIP Score ↑\uparrow ). The x-axis shows the (1−SSCDtarget)↑(1-\textbf{SSCD}_{\textbf{target}})\uparrow scores. rads consistently occupies the upper-right region of the frontier, maintaining high utility and semantic alignment while significantly reducing memorization.
Figure 5: Jain et al. (2025) produces mitigated samples that closely resemble one another. Generated images using the same initial latent xT\mathbf{x}_{T} . (a) Generated image without text guidance ( g=0g=0 ). (b–d) Generated images produced using different prompts.
Figure 6: Zero-Shot Generalization. We compare rads (Ours) in a zero-shot evaluation against state-of-the-art mitigation methods on the MemBench dataset ( Hong et al., 2024 ) . The top row shows Quality ( −log⁡10(FID)↑-\log_{10}(\mathrm{FID})\uparrow ), while the bottom row displays Alignment (CLIP Score ↑\uparrow ). The x-axis shows the (1−SSCDtarget)↑(1-\textbf{SSCD}_{\textbf{target}})\uparrow scores. rads consistently occupies the upper-right region of the frontier, maintaining high utility and semantic alignment while significantly reducing memorization. Note: The Hintersdorf et al. (2024) baseline is omitted due to its computational infeasibility (see Appendix A.6 ).
Figure 7: rads achieves mitigation with the highest generation diversity. Generated images for the prompt Michael Fassbender to Star In <i>Assassin’s Creed</i> Movie .
Figure 8: rads achieves mitigation on challenging prompts. Generated images for the prompt <em>Bloodborne</em> Video: Sony Explains the Game’s Procedurally Generated Dungeons .
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: rads achieves mitigation with high image quality. Generated images for the prompt Anna Kendrick is Writing a Collection of Funny, Personal Essays . (a) Generated image without mitigation. (b-e) Mitigated results produced by prior methods. (f) Mitigated result produced by rads (ours).
Figure 10: rads can achieve mitigation with better alignment to the generation intent. Generated images for the prompt Shaw Floors Value Collections Fyc Tt II Net Dinner With Friends (t) 732T_5E022 . (a) Generated image without mitigation. (b-e) Mitigated results produced by prior methods. (f) Mitigated result produced by rads (ours).
Table 1: RADS strikes the best Pareto frontier between image diversity, quality, and alignment. We compare rads with prior methods and a no-mitigation baseline across different datasets, models, and denoising samplers. Lower indicates better performance for SSCDseeds\textbf{SSCD}_{\textbf{seeds}} , SSCDprompts\textbf{SSCD}_{\textbf{prompts}} , and FID , while higher indicates better performance for CLIP . Best results are in bold. While Jain et al. (2025) achieves the lowest SSCDseeds\textbf{SSCD}_{\textbf{seeds}} , it suffers from severe quality degradation, as evidenced by the FID and CLIP scores.
Figure 11: Dynamics of Memorization Steering. Guidance L2L^{2} -norm during denoising. Standard diffusion (blue) spikes as it collapses into the memorization basin. rads (orange) steers away from this peak.
MethodTime (s)
No mitigation2.2950 ± 0.1580
Wen et al. (2024)2.8983 ± 0.0459
Ren et al. (2024)2.7544 ± 0.0404
Hintersdorf et al. (2024)104.0895 ± 48.9586
Jain et al. (2025)2.2322 ± 0.0368
rads (ours)2.9262 ± 0.0335
Appendix
Table 2: Average Inference Time Comparison
Figure 12: Loss of Semantic Alignment. Generated images for the prompt Mothers influence on her young hippo . (a) Generated image without mitigation. (b-e) Mitigated results produced by rads (ours).
More in this topic

Memorization in Generative Models

Memorization in generative models studies when models retain training examples, how to detect their recall, and how to limit it without losing utility.

See all 85 papers

Explore similar work

CardsList

  1. Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioning

    Oct 1, 2026Hyungjun Joo, Sehwan Kim, Hyeonggeun Han +2Diffusion Model AlignmentAttention Control in Diffusion Models

    Seoul National University · Konkuk University

  2. You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models

    Feb 23, 2026Kairan Zhao, Eleni Triantafillou, Peter TriantafillouMemorization in Generative ModelsT2I Diffusion Models

    University of Warwick, UK · Google DeepMind, UK

  3. Broken Memories: Detecting and Mitigating Memorization in Diffusion Models with Degraded Generations

    May 21, 2026Yuanmin Huang, Mi Zhang, Chen Chen +4Memorization in Generative ModelsDiffusion Models

    Fudan University Shanghai, China · East China University of Science and Technology Shanghai, China