cs.CVNov 14, 2026

Memorization Lives in a Few Attention Heads: Mitigating Memorization in Diffusion Models via Projection-Based Suppression

Authors: Puwei Lian, Yujun Cai, Songze Li, Yuezu Lv, Guanghui Wen

Organizations: Southeast University · Engineering Research Center of Blockchain Application, Supervision and Management (Southeast University), Ministry of Education · Beijing Institute of Technology

Abstract

Diffusion models can memorize and reproduce training samples during inference, raising serious privacy and copyright concerns. Existing inference-time mitigation methods often suppress conditional information globally, which may reduce memorization but can also harm semantic alignment and image quality. In this paper, we revisit memorization from the perspective of cross-attention heads. Through systematic analysis and targeted ablation, we find that memorization is mainly driven by a small subset of shallow-layer attention heads with abnormally strong semantic control, rather than by the entire cross-attention mechanism. Motivated by this finding, we propose Projection-based Top-k Attention Head Suppression, a targeted inference-time mitigation method. Our method first identifies the top-k attention heads most related to memorization using a semantic-control score, and then removes their condition-specific components through directional projection between conditional and unconditional representations. This design selectively weakens memorization-related features while preserving useful semantic information. Extensive experiments on different diffusion models and datasets show that our method effectively reduces memorization and achieves a better trade-off between memorization mitigation and image quality than existing approaches.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models

    Feb 23, 2026Kairan Zhao, Eleni Triantafillou, Peter TriantafillouMemorization in Generative ModelsT2I Diffusion Models

  2. Localizing Memorized Regions in Diffusion Models via Coordinate-Wise Curvature Differences

    May 26, 2026Gwangho Kim, Sungyoon LeeMemorization in Generative ModelsDiffusion Models

  3. Broken Memories: Detecting and Mitigating Memorization in Diffusion Models with Degraded Generations

    May 21, 2026Yuanmin Huang, Mi Zhang, Chen Chen +4Memorization in Generative ModelsDiffusion Models