Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection

Yunzhuo Chen; Jordan Vice; Naveed Akhtar; Nur Al Hasan Haldar; Ajmal Mian

cs.CVMar 13, 2026

Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection

Authors: Yunzhuo Chen, Jordan Vice, Naveed Akhtar, Nur Al Hasan Haldar, Ajmal Mian

Organizations: The University of Western Australia, Perth, Australia · The University of Melbourne, Melbourne, Australia · Curtin University, Perth, Australia

Abstract

State-of-the-art text-to-image diffusion models can produce impressive visuals but may memorize and reproduce training images, creating copyright and privacy risks. Existing prompt perturbations applied at inference time, such as random token insertion or embedding noise, may lower copying but often harm image-prompt alignment and overall fidelity. To address this, we introduce two complementary methods. First, Region-Aware Prompt Augmentation (RAPTA) uses an object detector to find salient regions and turn them into semantically grounded prompt variants, which are randomly sampled during training to increase diversity, while maintaining semantic alignment. Second, Attention-Driven Multimodal Copy Detection (ADMCD) aggregates local patch, global semantic, and texture cues with a lightweight transformer to produce a fused representation, and applies simple thresholded decision rules to detect copying without training with large annotated datasets. Experiments show that RAPTA reduces overfitting while maintaining high synthesis quality, and that ADMCD reliably detects copying, outperforming single-modal metrics.

Figures & tables

Fig. 1: Overview of our framework. (a) RAPTA augments the training caption in an image-grounded way: a detector provides region proposals with classes and coarse positions; a small template set instantiates region-aware variants from the base prompt. One variant is sampled per iteration to condition the conditional diffusion model, injecting grounded diversity while preserving prompt–image alignment. (b) ADMCD is an inference-time, two-step detector. For a generated image $G$ , we compare it against the training set using fused-feature cosine similarity $S_{\mathrm{fus}}$ ; pairs with $S_{\mathrm{fus}}>\tau_{1}$ are flagged as Copy. A similarity-weighted score over three streams (visual, CLIP, texture) with threshold $\tau_{2}$ assigns the copy type.

Fig. 2: RAPTA pipeline. (a) A pretrained detector highlights saliency-based boxes with class/confidence; we keep the top- $M$ with $S_{i}>\tau_{b}$ and discretize each box center to a $3\times 3$ grid $\mathcal{G}$ (top-left $\cdots$ bottom-right), yielding position tokens $\mathrm{pos}_{i}$ . (b) From the base prompt $p$ , detected class $c_{i}$ and position $\mathrm{pos}_{i}$ , a small template set $\{T_{j}\}_{j=1}^{J}$ instantiates region-aware variants $V=\{p\}\cup\{\,T_{j}(p,c_{i},\mathrm{pos}_{i})\,\}$ . (c) For image $I$ and each $v\in V$ , CLIP consistency $S_{v}=\cos\!\big(f_{I},f_{v}\big)$ is converted to weights $w_{v}=S_{v}^{\gamma}$ and normalized to $\pi(v)=w_{v}/\sum_{u\in V}w_{u}$ ; we sample a single prompt $\tilde{p}\sim\pi(\cdot)$ to condition training. This preserves prompt–image alignment while injecting grounded diversity.

Fig. 3: ADMCD pipeline. (a) For an image $X\in\{G,R\}$ , three complementary embeddings are extracted: a patch-level visual descriptor $\mathbf{f}^{\mathrm{vis}}(X)$ , a global CLIP descriptor $\mathbf{f}^{\mathrm{clip}}(X)$ , and a texture descriptor $\mathbf{f}^{\mathrm{tex}}(X)$ . After linear projection, the streams are fused by a lightweight Transformer and $\ell_{2}$ -normalized to yield $\hat{\mathbf{f}}_{\mathrm{fus}}(X)$ . (b) Copy decision uses fused cosine similarity $S_{\mathrm{fus}}$ ; we flag Copy if $S_{\mathrm{fus}}>\tau_{1}$ . (c) For flagged pairs we compute stream-wise similarities $S_{\mathrm{vis}},S_{\mathrm{clip}},S_{\mathrm{tex}}$ and the similarity-weighted score $\bar{S}$ ; Retrieve/Exact if $\bar{S}>\tau_{2}$ , otherwise Style . Thresholds $(\tau_{1},\tau_{2})$ and weights $(\omega_{1},\omega_{2},\omega_{3})$ are fixed from validation sweeps.

Panel of Table 4 — TABLE I: Result comparison on one query: a generated image $G$ and the Top-5 retrieved references $R_{1..5}$ under different methods.

Method	ADMCD (Ours)	FID	CLIP Score	KID
DCR [ 14 ]	3.2	7.9	30.5	2.9
LDM–T2I [ 24 ]	5.3	10.4	33.2	3.1
SD2.1-base [ 25 ]	7.4	8.3	27.8	3.3
RAPTA(Ours)	2.6	8.1	23.1	1.6

TABLE II: Detection vs. conditional diffusion backbones. Copy Rate is computed by ADMCD at threshold

\tau_{1}

; generation quality is reported on the same sample set.

Method	Original	Gauss N.	Gauss B.	Poisson	S&P	Speckle
LPIPS [ 20 ]	0.233	0.444	0.335	0.375	0.612	0.569
ORB [ 18 ]	0.276	0.242	0.155	0.282	0.172	0.184
SSIM [ 19 ]	0.677	0.504	0.664	0.591	0.389	0.407
SSCD [ 5 ]	0.680	0.594	0.443	0.429	0.485	0.407
DreamSim [ 22 ]	0.857	0.781	0.714	0.691	0.689	0.707
ADMCD(Ours)	0.974	0.923	0.940	0.929	0.871	0.894

TABLE III: Similarity between a generated image

G

and a fixed reference

R

under noise perturbations. Column Original reports the score for

(G,R)

without any perturbation; the other columns report scores after perturbing only

G

with Gaussian noise, Gaussian blur, Poisson noise, salt–pepper, and speckle. Higher is better except LPIPS (distance).

Method	Crop20%	Flip H	Flip V	Occlude10%	Rotate $30^{\circ}$
LPIPS [ 20 ]	0.335	0.373	0.519	0.315	0.615
ORB [ 18 ]	0.232	0.221	0.243	0.110	0.276
SSIM [ 19 ]	0.570	0.556	0.427	0.642	0.207
SSCD [ 5 ]	0.577	0.404	0.464	0.391	0.489
DreamSim [ 22 ]	0.617	0.524	0.564	0.691	0.689
ADMCD(Ours)	0.970	0.886	0.857	0.748	0.939

TABLE IV: Similarity between a generated image

G

and a fixed reference

R

under geometric perturbations applied only to

G

: 20% crop, horizontal/vertical flips, 10% occlusion, and

30^{\circ}

rotation. Higher is better except LPIPS (distance).

G

and

R

are the same as in Table III .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 4: Accuracy vs. threshold on fused similarity $S_{\mathrm{fus}}$ . Best accuracy is achieved at $\tau=0.938$ .

Fig. 5: Accuracy heatmap over $(w_{\mathrm{vis}},w_{\mathrm{clip}})$ with $w_{\mathrm{tex}}=1-w_{\mathrm{vis}}-w_{\mathrm{clip}}$ . Peak at $(0.24,0.38,0.38)$ .

Fig. 6: Robustness examples. Top: generated query $G$ . Bottom: the clean retrieved copy $R$ and ten corrupted variants—Gaussian noise, Gaussian blur, Poisson noise, salt–pepper, speckle, 20% crop, horizontal/vertical flips, 10% occlusion, and $30^{\circ}$ rotation. Our fused-feature similarity $S_{\mathrm{fus}}$ remains high and continues to select the correct match across both photometric and geometric perturbations. Numerical scores for each method appear in Tables III and IV .

Fig. 7: Per-image robustness under ten common image-based attacks. For a query $G$ and its retrieved copy $R$ , we apply Gaussian noise/blur, Poisson, salt–pepper, speckle, 20% crop, horizontal/vertical flips, 10% occlusion, and a $30^{\circ}$ rotation.

T2I Diffusion Models

Text-to-image (T2I) diffusion models generate images from prompts by learning to reverse noise under text conditioning.

See all 394 papers

Steering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion
Feb 24, 2026Sathwik Karnik, Juyeop Kim, Sanmi Koyejo +2Memorization in Generative Models T2I Diffusion Models
Stanford University, Stanford, CA, USA · Yonsei University, Seoul, Korea
You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models
Feb 23, 2026Kairan Zhao, Eleni Triantafillou, Peter TriantafillouMemorization in Generative Models T2I Diffusion Models
University of Warwick, UK · Google DeepMind, UK
Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioning
Oct 1, 2026Hyungjun Joo, Sehwan Kim, Hyeonggeun Han +2Diffusion Model Alignment Attention Control in Diffusion Models
Seoul National University · Konkuk University

Feb 24, 2026J/K move · Enter open · S save

Steering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion

Sathwik Karnik, Juyeop Kim, Sanmi Koyejo, Jong-Seok Lee+1

Stanford University, Stanford, CA, USA · Yonsei University, Seoul, Korea

Memorization in Generative Models T2I Diffusion Models RL for Diffusion Models

Open paper AskPDF arXiv

Text-to-image diffusion models often memorize training data, revealing a fundamental failure to generalize beyond the training set. Current mitigation strategies typically sacrifice image quality or prompt alignment to reduce memorization. To address this, we propose Reachability-Aware Diffusion Steering (RADS), an inference-time framework that prevents memorization while preserving generation fidelity. RADS models the diffusion denoising process as a dynamical system and applies concepts from reachability analysis to approximate the "backward reachable tube"--the set of intermediate states that inevitably evolve into memorized samples. We then formulate mitigation as a constrained reinforcement learning (RL) problem, where a policy learns to steer the trajectory away from memorization via minimal perturbations in the caption embedding space. Empirical evaluations show that RADS achieves a superior Pareto frontier between generation diversity (SSCD), quality (FID), and alignment (CLIP) compared to state-of-the-art baselines. Crucially, RADS provides robust mitigation without modifying the diffusion backbone, offering a plug-and-play solution for safe generation. Our website is available at: https://s-karnik.github.io/rads-memorization-project-page/.

Abstract

T2I Diffusion Models

Explore similar work

Steering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion

You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models

Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioning