cs.CVFeb 5, 2026

Consistency-Preserving Concept Erasure via Unsafe-Safe Pairing and Directional Fisher-Weighted Adaptation

Authors: Yongwoo Kim, Sungmin Cha, Hyunsoo Kim, Jaewon Lee, Donghyun Kim

Organizations: Korea University · New York University · Korea Institute of Science and Technology

Abstract

With the increasing versatility of text-to-image diffusion models, the ability to selectively erase undesirable concepts (e.g., harmful content) has become indispensable. However, existing concept erasure approaches primarily focus on removing unsafe concepts without providing guidance toward corresponding safe alternatives, which often leads to failure in preserving the structural and semantic consistency between the original and erased generations. In this paper, we propose a novel framework, PAIRed Erasing (PAIR), which reframes concept erasure from simple removal to consistency-preserving semantic realignment using unsafe-safe pairs. We first generate safe counterparts from unsafe inputs while preserving structural and semantic fidelity, forming paired unsafe-safe multimodal data. Leveraging these pairs, we introduce two key components: (1) Paired Semantic Realignment, a guided objective that uses unsafe-safe pairs to explicitly map target concepts to semantically aligned safe anchors; and (2) Fisher-Weighted Initialization for DoRA, which initializes parameter-efficient low-rank adaptation matrices using unsafe-safe pairs, encouraging the generation of safe alternatives while selectively suppressing unsafe concepts. Together, these components enable fine-grained erasure that removes only the targeted concepts while maintaining overall semantic consistency. Extensive experiments demonstrate that our approach achieves the best overall trade-off among baselines, providing effective concept erasure while preserving structural integrity, semantic coherence, and generation quality.

Figures & tables

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Closed-Form Concept Erasure via Double Projections

    Apr 11, 2026Chi Zhang, Jingpu Cheng, Zhixian Wang +1Concept ErasureDiffusion Model Unlearning

  2. GRACE: Adaptive Concept Erasure with Geometry-Guided Retention in Diffusion Models

    Sep 11, 2026Qinghui Gong, Yihuai Liang, Jinshan Lai +3Diffusion Model Fine-TuningDiffusion Model Unlearning

  3. Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models

    Jul 6, 2026Parth Upman, Nishita Jain, Shreyank N GowdaAttention Control in Diffusion ModelsDiffusion-Based Image Editing