Organizations: Southeast University · Engineering Research Center of Blockchain Application, Supervision and Management (Southeast University), Ministry of Education · Beijing Institute of Technology
Diffusion models can memorize and reproduce training samples during inference, raising serious privacy and copyright concerns. Existing inference-time mitigation methods often suppress conditional information globally, which may reduce memorization but can also harm semantic alignment and image quality. In this paper, we revisit memorization from the perspective of cross-attention heads. Through systematic analysis and targeted ablation, we find that memorization is mainly driven by a small subset of shallow-layer attention heads with abnormally strong semantic control, rather than by the entire cross-attention mechanism. Motivated by this finding, we propose Projection-based Top-k Attention Head Suppression, a targeted inference-time mitigation method. Our method first identifies the top-k attention heads most related to memorization using a semantic-control score, and then removes their condition-specific components through directional projection between conditional and unconditional representations. This design selectively weakens memorization-related features while preserving useful semantic information. Extensive experiments on different diffusion models and datasets show that our method effectively reduces memorization and achieves a better trade-off between memorization mitigation and image quality than existing approaches.
Figures & tables
Figure 1: Examples of semantic-control scores across attention heads. Memorized images tend to activate a few distinctly prominent attention heads, whose semantic response strengths are substantially higher than those of other attention heads. In contrast, non-memorized images exhibit a much smoother and more uniform distribution of semantic strength across attention heads.
Figure 2: Overview of our framework. The left panel illustrates the computation of multi-head attention. Based on this process, we introduce two modifications on the right panel : (1) Identifying attention heads with strong semantic control through attention map analysis. (2) Applying projection-based suppression to mitigate memorization-related components in the selected heads.
Figure 3: Layer-wise analysis of attention heads with strong semantic control. Memorized images exhibit a larger number of high semantic-control attention heads in the shallow layers, especially layers 0 ∼ 3. In deeper layers, the statistics of memorized and non-memorized samples become similar.
Figure 4: Ablation study on attention head groups. We evaluate different suppression factors ϵ∈{0.0,0.2,0.4,0.6} . Suppressing the Top-3 heads produces the largest reduction in SSCD Score, while suppressing the Bottom-3 heads has little effect.
Figure 5: SSCD-CLIP trade-off curve comparison with baselines. Our method consistently lies in the lower-right region compared with all baselines. This demonstrates that, under the same SSCD level, our approach achieves higher CLIP scores. In other words, given comparable semantic alignment, our method provides the most effective suppression of memorization.
Figure 6: Qualitative comparison with baselines. The first column shows the memorized training images, while the remaining seven columns present the results of baselines and our method. Our method achieves a superior balance between memorization mitigation, semantic consistency, and image quality compared with existing baselines.
Layers
SD v1.4
SD v2.0
SSCD ↓
CLIP ↑
SSCD ↓
CLIP ↑
None
0.5122
0.3009
0.6306
0.2947
0–1
0.3742
0.2887
0.5975
0.2973
0–3
0.1578
0.2590
0.2662
0.2730
0–5
0.1301
0.2446
0.2067
0.2564
0–7
0.1098
0.2264
0.1602
0.2465
Table 1: Progressive Layers ablation study for mitigation. Suppressing layers 0–3 provides the most substantial mitigation, while extending suppression to deeper layers yields diminishing benefits at the cost of semantic fidelity.
Layers
SD v1.4
SD v2.0
SSCD ↓
CLIP ↑
SSCD ↓
CLIP ↑
None
0.5122
0.3009
0.6306
0.2947
0 ∼ 3
0.1578
0.2590
0.2662
0.2730
4 ∼ 7
0.4260
0.2690
0.3978
0.2667
8 ∼ 11
0.4412
0.2934
0.6165
0.2588
12 ∼ 15
0.3962
0.2969
0.6264
0.2922
Table 2: Segmented layers ablation study for mitigation. Early layers (0 ∼ 3) contribute most significantly to mitigation, while suppression in deeper layers yields limited benefits.
λ
k=2
k=3
k=4
SSCD ↓
CLIP ↑
SSCD ↓
CLIP ↑
SSCD ↓
CLIP ↑
1.00
0.1692
0.2610
0.1578
0.2590
0.1534
0.2576
0.95
0.1697
0.2603
0.1590
0.2592
0.1554
0.2581
0.90
0.1718
0.2607
0.1580
0.2593
0.1558
0.2589
0.85
0.1752
0.2611
0.1602
0.2606
0.1566
0.2592
0.80
0.1768
0.2609
0.1626
0.2613
0.1594
0.2604
Table 3: Ablation study of k and λ on SD v1.4. The stable performance across different settings demonstrates the robustness of our method to hyperparameter choices.
Method
RTA
IND
DOG
PEA
CALS
SALI
Ours
Time (s) ↓
2765
2385
5420
1490
1090
2075
1115
Table 4: Runtime comparison of memorization mitigation methods on SD v1.4 for 500 samples. Our method achieves the second-fastest runtime, with only a minor overhead compared with CALS.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: SSCD-CLIP trade-off on MS-COCO fine-tuned SD v1.4. Our method achieves a better balance between memorization mitigation and semantic alignment than baselines.
Method
SD v1.4
SD v2.0
AUC ↑
T@F=1% ↑
AUC ↑
T@F=1% ↑
Wen et al.
94.32
34.32
91.43
20.63
Ren et al.
95.67
42.66
84.82
10.50
Jeon et al.
99.01
82.80
96.88
50.20
Ours
98.41
78.12
96.23
54.21
Appendix
Table 5: Memorization detection results on SD v1.4 and SD v2.0. The competitive performance further supports our finding that memorization is associated with shallow-layer attention heads exhibiting large conditional–unconditional attention discrepancies.
Figure 8: Temporal distribution of outlier semantic-control attention heads across denoising timesteps. Each row corresponds to a layer, and each column corresponds to one generated sample. In each subplot, the x -axis denotes the denoising timestep, and the y -axis denotes the attention head index. Each cyan point indicates that the corresponding attention head is identified as an outlier semantic-control head at that timestep. The horizontal and temporally continuous patterns show that memorization-related semantic control is not random or transient, but is mainly carried by a stable subset of attention heads that persist across most denoising timesteps.
Table 6: Comparison between naive suppression and our proposed suppression strategy on SD v1.4 and SD v2.0. Our method achieves comparable memorization mitigation while better preserving semantic fidelity on both models.
Method
SD v1.4
SD v2.0
CLIP ( ↑ )
FID ( ↓ )
CLIP ( ↑ )
FID ( ↓ )
No Mitigation
0.3386
85.47
0.3509
86.73
Ours
0.3390
85.49
0.3503
86.20
Appendix
Table 7: Generation quality on non-memorized prompts. Our method preserves semantic alignment and distributional quality on non-memorized inputs.
Method
SD 3
SSCD ( ↓ )
CLIP ( ↑ )
No Mitigation
0.3372
0.2625
RTA
0.2252
0.2367
PEA
0.2292
0.2438
IND
0.2442
0.2451
DOG
0.2289
0.2219
Appendix
Table 8: Memorization mitigation results on Stable Diffusion 3. Our method achieves the best balance between memorization mitigation and semantic alignment.
Method
SDv1.4 – AM
SDv1.4 – VM
SDv1.4 – TM
SDv2.0 – TM
SSCD ( ↓ )
CLIP ( ↑ )
SSCD ( ↓ )
CLIP ( ↑ )
SSCD ( ↓ )
CLIP ( ↑ )
SSCD ( ↓ )
CLIP ( ↑ )
No Mitigation
0.5122
0.3009
0.7895
0.3371
0.4322
0.2904
0.6306
0.2947
RTA
0.1045
0.1959
0.0888
0.2053
0.1090
0.1932
0.1583
0.2093
0.1906
0.2394
0.1849
0.2552
0.1923
0.2348
0.2152
0.2463
0.2407
0.2487
0.2660
0.2727
0.2334
0.2418
0.2866
0.2633
0.2873
0.2545
0.3173
0.2516
0.2786
0.2553
0.4547
0.2824
Appendix
Table 9: Mitigation results on different memorization types. AM denotes all memorized samples, VM denotes verbatim memorization, and TM denotes template memorization. We report SSCD and CLIP scores under SD v1.4 and SD v2.0, with four hyperparameter settings for each method. Lower SSCD indicates better memorization mitigation, while higher CLIP indicates better prompt-image alignment.
Memorization in generative models studies when models retain training examples, how to detect their recall, and how to limit it without losing utility.