You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models

Kairan Zhao; Eleni Triantafillou; Peter Triantafillou

cs.CVFeb 23, 2026

You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models

Authors: Kairan Zhao, Eleni Triantafillou, Peter Triantafillou

Organizations: University of Warwick, UK · Google DeepMind, UK

Abstract

Generative models have been shown to "memorize" certain training data, leading to verbatim or near-verbatim generating images, which may cause privacy concerns or copyright infringement. We introduce Guidance Using Attractive-Repulsive Dynamics (GUARD), a novel framework for memorization mitigation in text-to-image diffusion models. GUARD adjusts the image denoising process to guide the generation away from an original training image and towards one that is distinct from training data while remaining aligned with the prompt, guarding against reproducing training data, without hurting image generation quality. We propose a concrete instantiation of this framework, where the positive target that we steer towards is given by a novel method for (cross) attention attenuation based on (i) a novel statistical mechanism that automatically identifies the prompt positions where cross attention must be attenuated and (ii) attenuating cross-attention in these per-prompt locations. The resulting GUARD offers a surgical, dynamic per-prompt inference-time approach that, we find, is by far the most robust method in terms of consistently producing state-of-the-art results for memorization mitigation across two architectures and for both verbatim and template memorization, while also improving upon or yielding comparable results in terms of image quality.

Figures & tables

Panel of Figure 1 — Figure 1: CA patterns. (a) and (b) : the CA mass across tokens, in the first and last inference steps for verbatim and template memorization (VM and TM). For clarity, we exclude the first token (position 0) from the plots, as it consistently receives the majority of CA across both memorized and non-memorized examples, which would dominate the scale and obscure differences among the remaining tokens. (c) and (d) : the CA mass on EOT across steps, for a memorized and a non-memorized prompt. (c) : VM on SD v1.4. (d) : TM on SD v2.0.

Panel of Figure 2 — Figure 2: Overview of the CA-in-GUARD denoising process.

	SD v1.4 – verbatim memorization			SD v1.4 – template memorization			SD v2.0 – template memorization
Method	SSCD ( $\downarrow$ )	CLIP ( $\uparrow$ )	FID ( $\downarrow$ )	SSCD ( $\downarrow$ )	CLIP ( $\uparrow$ )	FID ( $\downarrow$ )	SSCD ( $\downarrow$ )	CLIP ( $\uparrow$ )	FID ( $\downarrow$ )
No mitigation	0.875 ${\scriptstyle\pm 0.001}$	0.346 ${\scriptstyle\pm 0.001}$	243.056	0.776 ${\scriptstyle\pm 0.017}$	0.219 ${\scriptstyle\pm 0.007}$	258.976	0.735 ${\scriptstyle\pm 0.011}$	0.215 ${\scriptstyle\pm 0.005}$	303.266
RTA	0.328 ${\scriptstyle\pm 0.007}$	0.263 ${\scriptstyle\pm 0.002}$	175.866	0.617 ${\scriptstyle\pm 0.043}$	0.187 ${\scriptstyle\pm 0.010}$	218.343	0.543 ${\scriptstyle\pm 0.048}$	0.183 ${\scriptstyle\pm 0.009}$	233.580
Wen et al.	0.115 ${\scriptstyle\pm 0.011}$	0.267 ${\scriptstyle\pm 0.003}$	162.848	0.545 ${\scriptstyle\pm 0.038}$	0.188 ${\scriptstyle\pm 0.008}$	209.719	0.260 ${\scriptstyle\pm 0.026}$	0.183 ${\scriptstyle\pm 0.008}$	188.914
Ren et al.	0.113 ${\scriptstyle\pm 0.007}$	0.258 ${\scriptstyle\pm 0.005}$	164.638	0.602 ${\scriptstyle\pm 0.033}$	0.184 ${\scriptstyle\pm 0.007}$	222.066	0.356 ${\scriptstyle\pm 0.024}$	0.188 ${\scriptstyle\pm 0.007}$	208.416
Han et al.	0.191 ${\scriptstyle\pm 0.016}$	0.256 ${\scriptstyle\pm 0.008}$	166.551	0.479 ${\scriptstyle\pm 0.033}$	0.188 ${\scriptstyle\pm 0.006}$	210.839	0.401 ${\scriptstyle\pm 0.024}$	0.186 ${\scriptstyle\pm 0.005}$	208.852

Table 1: Evaluation results across metrics, architectures and memorization types. For each setting (architecture and memorization type), we select for each method the configuration that yields the best SSCD, since this is the primary metric for memorization mitigation (in Figure 4 , and in more detail in Table 5(c) in the Appendix, we show results with different selection criteria too). For each prompt, we generate four images and report the mean

{\scriptstyle\pm(}

95% confidence interval) across four generations. Additional evaluation results on SD v3.0 are reported in Table 9 in A.8 .

Panel of Figure 3 — Figure 3: The best achievable SSCD, CLIP, and FID of different methods . We plot the best value a method can achieve on each metric individually , using the configuration that yields best results on that specific metric. In each subplot, a different configuration may be used (we pick the hyperparameter setting that yields the best SSCD, best CLIP and best FID, respectively), so this plot does not speak to the ability of a method to do well on all metrics jointly , nor to trade-offs between these metrics, which we investigate separately.

Figure 4: Comparison of the SSCD of different methods for the hyperparameter configuration that works best for CLIP .

Panel of Figure 5 — Figure 5: Qualitative examples on SD v1.4 under verbatim (top row) and template memorization (bottom row). No mitigation often reproduces the training example closely. CA-in-GUARD can significantly mitigate memorization while preserving prompt-relevant content and image quality. Additional examples are provided in A.9 .

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Table 2: Hyperparameter search ranges for CA attenuation and CA-in-GUARD across different settings.

	SD v1.4			SD v2.0
Method	SSCD	CLIP	FID	SSCD	CLIP	FID
No mitigation	$0.071\,{\scriptstyle\pm 0.006}$	$0.299\,{\scriptstyle\pm 0.010}$	141.947	$0.074\,{\scriptstyle\pm 0.006}$	$0.322\,{\scriptstyle\pm 0.006}$	141.350
CA attenuation	$0.069\,{\scriptstyle\pm 0.006}$	$0.298\,{\scriptstyle\pm 0.006}$	142.898	$0.072\,{\scriptstyle\pm 0.006}$	$0.320\,{\scriptstyle\pm 0.005}$	139.417

Appendix

Table 3: Comparison of no mitigation and CA attenuation on non-memorized prompts for Stable Diffusion v1.4 and v2.0. Results are reported as mean values with 95% confidence intervals.

Panel of Figure 6 — Figure 6: The best achievable SSCD, CLIP, and FID of CA-in-GUARD (our default) versus semantic-in-GUARD (ablation), evaluated across 3 settings: SD v1.4 with verbatim memorization, template memorization, and SD v2.0 with template memorization.

Panel of Figure 7 — Figure 7: Cross-attention mass in different U-Net blocks over inference steps, by comparing three type of prompts: (i) a memorized prompt, (ii) a counterfactual non-memorized prompt obtained by paraphrasing the memorized prompt (which already substantially reduces SSCD), and (iii) an unrelated non-memorized prompt

Panel of Figure 8 — Figure 8: SSCD–FID and SSCD–CLIP Pareto frontiers comparing CA attenuation using selective blocks (our default) vs. all blocks (ablation). Lower SSCD and FID and higher CLIP indicate better performance; methods closer to the bottom-right corner are optimal in both the left and right subfigures.

Panel of Figure 9 — Figure 9: SSCD-FID and SSCD-CLIP Pareto frontiers comparing CA attenuation over all heads (our default) vs. top k% heads (ablation) . Lower SSCD and FID and higher CLIP indicate better performance; thus, methods closer to the bottom-right corner are optimal for both left and right subfigures.

Panel of Figure 10 — Figure 10: SSCD-FID and SSCD-CLIP Pareto frontiers comparing CA attenuation over all timesteps (our default) vs. other variants (ablation) . Lower SSCD and FID and higher CLIP indicate better performance; thus, methods closer to the bottom-right corner are optimal for both left and right subfigures.

Panel of Figure 11 — Figure 11: SSCD-FID and SSCD-CLIP Pareto frontiers for CA-in-GUARD ablations across sampler choice , inference steps , scheduler choice , and CFG scale . Lower SSCD and FID, together with higher CLIP, indicate better performance; methods closer to the bottom-right corner achieve a more favorable trade-off.

Method	SSCD	CLIP	FID
No mitigation	$0.571\pm 0.068$	$0.280\pm 0.019$	$194.697$
Wen et al.	$0.569\pm 0.064$	$0.284\pm 0.016$	$188.374$
CA-in-GUARD	$0.403\pm 0.071$	$0.243\pm 0.019$	$184.982$

Appendix

Table 4: Quantitative results for the CFG scale ablation in Figure 11(d) , reported at CFG scale

s=1.0

, with each method shown under the configuration with the best SSCD (our primary memorization-mitigation metric).

Table 5: Comparison of memorization mitigation methods across SD v1.4 and SD v2.0 under verbatim and template memorization settings. Results are evaluated using SSCD, CLIP, and FID. Each subtable shows results when selecting the best configuration for one target metric (SSCD, CLIP, or FID). For each prompt, we generate four images and report the mean

\pm

(95% confidence interval) across the four generations.

Panel of Figure 12 — Figure 12: The best achievable SSCD, CLIP, and FID of our methods compared to the prior state-of-the-art (and most related) baseline of Ren et al. . We plot the best achievable value on each metric individually , using the configuration that yields best results on that metric. Because in each subplot, a different configuration may be used for a given method (the one that yields the best SSCD, best CLIP and best FID, respectively), this plot does not speak to the ability to do well on all metrics jointly , nor to trade-offs between these metrics, which we investigate later.

Panel of Figure 13 — Figure 13: SSCD–FID and SSCD–CLIP Pareto frontiers comparing our methods with existing baselines under three memorization settings. Lower SSCD and FID and higher CLIP indicate better performance; methods closer to the bottom-right corner represent more favorable trade-offs in both subfigures. The reference CLIP score indicates the target semantic fidelity that methods are expected to attain (see Section 6 – Hyperparameter selection for details on how this reference is defined).

Method	SD v1.4 – template memorization			SD v2.0 – template memorization
Method	SSCD ( $\downarrow$ )	DINO@1 ( $\downarrow$ )	DINO@5 ( $\downarrow$ )	SSCD ( $\downarrow$ )	DINO@1 ( $\downarrow$ )	DINO@5 ( $\downarrow$ )
No mitigation	$0.776{\pm}0.017$	$0.924{\pm}0.006$	$0.906{\pm}0.013$	$0.735{\pm}0.011$	$0.903{\pm}0.006$	$0.895{\pm}0.007$
RTA	$0.617{\pm}0.043$	$0.836{\pm}0.030$	$0.813{\pm}0.035$	$0.543{\pm}0.048$	$0.794{\pm}0.038$	$0.778{\pm}0.039$
Wen et al.	$0.545{\pm}0.038$	$0.823{\pm}0.024$	$0.790{\pm}0.028$	$0.260{\pm}0.026$	$0.655{\pm}0.023$	$0.637{\pm}0.024$
Ren et al.	$0.602{\pm}0.033$	$0.858{\pm}0.015$	$0.832{\pm}0.020$	$0.356{\pm}0.024$	$0.711{\pm}0.020$	$0.695{\pm}0.020$
Han et al.	$0.479{\pm}0.033$	$0.832{\pm}0.015$	$0.789{\pm}0.019$	$0.401{\pm}0.024$	$0.745{\pm}0.017$	$0.732{\pm}0.018$

Appendix

Table 6: SSCD and DINO results for template memorization settings, as an expansion of Table 1 . Lower values indicate less similarity to the training set and therefore less memorization.

Method	SD v1.4			SD v2.0
Method	SSCD ( $\downarrow$ )	CLIP ( $\uparrow$ )	FID ( $\downarrow$ )	SSCD ( $\downarrow$ )	CLIP ( $\uparrow$ )	FID ( $\downarrow$ )
No mitigation	$0.524{\pm}0.028$	$0.257{\pm}0.006$	$198.081$	$0.333{\pm}0.025$	$0.260{\pm}0.005$	$146.527$
RTA	$0.366{\pm}0.026$	$0.223{\pm}0.007$	$153.361$	$0.274{\pm}0.023$	$0.223{\pm}0.006$	$135.615$
Wen et al.	$0.255{\pm}0.019$	$0.220{\pm}0.006$	$137.464$	$0.152{\pm}0.011$	$0.219{\pm}0.006$	$126.339$
Ren et al.	$0.350{\pm}0.024$	$0.223{\pm}0.006$	$144.830$	$0.200{\pm}0.016$	$0.218{\pm}0.006$	$130.757$
Han et al.	$0.239{\pm}0.018$	$0.220{\pm}0.006$	$143.297$	$0.176{\pm}0.023$	$0.221{\pm}0.006$	$125.894$

Appendix

Table 7: Full-spectrum evaluation results across metrics and architectures. For each setting, we select for each method the configuration that yields the best SSCD, since SSCD is the primary metric for memorization mitigation.

Method	SD v1.4		SD v2.0
Method	Time (s)	Memory (GiB)	Time (s)	Memory (GiB)
No mitigation	7.575	6.172	6.880	6.580
RTA	7.563	6.172	6.884	6.590
Wen et al.	9.210	12.697	8.709	13.102
Ren et al.	15.302	6.213	10.125	6.695
Han et al.	12.107	14.373	10.901	14.785

Appendix

Table 8: Average wall-clock runtime per example (in seconds) and GPU memory usage (in GiB) of each method across both model architectures.

Method	SSCD ( $\downarrow$ )	CLIP ( $\uparrow$ )	FID ( $\downarrow$ )
No mitigation	$0.337{\pm}0.011$	$0.213{\pm}0.021$	$336.888$
RTA	$0.286{\pm}0.021$	$0.182{\pm}0.023$	$275.955$
Wen et al.	$0.303{\pm}0.031$	$0.182{\pm}0.022$	$373.299$
Han et al.	$0.200{\pm}0.037$	$0.182{\pm}0.022$	$219.990$
CA attenuation	$0.204{\pm}0.032$	$0.185{\pm}0.027$	$215.708$
CA-in-GUARD	$0.168{\pm}0.017$	$0.181{\pm}0.020$	$212.728$

Appendix

Table 9: SD v3.0 evaluation results, extending Table 1 . For each method, we report the configuration that achieves the best SSCD.

Panel of Figure 14 — Figure 14: Additional qualitative examples across model architectures and memorization settings. We compare the training image with generations from no mitigation, prior mitigation methods, CA attenuation, and CA-in-GUARD. CA-in-GUARD consistently moves generations away from memorized training images while preserving prompt-relevant visual content.

Memorization in Generative Models

Memorization in generative models studies when models retain training examples, how to detect their recall, and how to limit it without losing utility.

See all 85 papers

Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioning
Oct 1, 2026Hyungjun Joo, Sehwan Kim, Hyeonggeun Han +2Diffusion Model Alignment Attention Control in Diffusion Models
Seoul National University · Konkuk University
Steering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion
Feb 24, 2026Sathwik Karnik, Juyeop Kim, Sanmi Koyejo +2Memorization in Generative Models T2I Diffusion Models
Stanford University, Stanford, CA, USA · Yonsei University, Seoul, Korea
Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection
Mar 13, 2026Yunzhuo Chen, Jordan Vice, Naveed Akhtar +2T2I Diffusion Models T2I Generation
The University of Western Australia, Perth, Australia · The University of Melbourne, Melbourne, Australia · Curtin University, Perth, Australia

Oct 1, 2026J/K move · Enter open · S save

Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioning

Hyungjun Joo, Sehwan Kim, Hyeonggeun Han, Sangwoo Hong+1

Seoul National University · Konkuk University

Diffusion Model Alignment Attention Control in Diffusion Models Memorization in Generative Models T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image diffusion models have achieved remarkable progress in image synthesis, yet can exhibit memorization by closely reproducing individual training examples. Effective mitigation must preserve useful prompt information to guide alternative depictions. We introduce a training-free method that redistributes cross-attention with Gaussian smoothing before reinforcing content-token contributions and attenuating padding contributions, without additional denoiser evaluations. With this intervention, stronger content conditioning can improve prompt alignment at comparable training-image similarity. A local analysis identifies when reinforcement preserves shared value information while redistribution reduces localized attention mass. On Stable Diffusion v1.4 and v2.0, all evaluated smoothing widths lie on the empirical Pareto frontiers for training-image similarity versus both prompt alignment and image preference. A configuration selected on Stable Diffusion reduces template reproduction in DeepFloyd IF without further tuning. These findings support jointly controlling conditioning allocation and strength to generate prompt-consistent alternatives.

Abstract

Memorization in Generative Models

Explore similar work

Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioning

Steering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion

Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection