Organizations: Harbin Institute of Technology, Shenzhen · Jilin University · Tsinghua Shenzhen International Graduate School, Tsinghua University · South China University of Technology
Recent advances in text-to-image (T2I) generation have substantially improved visual synthesis, but have also raised increasing safety concerns due to their potential to generate harmful or undesirable content. Existing concept erasure methods predominantly follow a static weight paradigm, producing a single frozen adapter that struggles to adapt to diverse prompt variations and suffers from parameter interference when scaling to multiple concepts. We propose HyperErase, a framework for concept erasure based on hypernetwork-driven prompt-conditioned parameter synthesis. Our approach first reframes concept erasure as prompt-conditioned parameter amortization and trains a hypernetwork to map textual descriptions to prompt-specific LoRA updates, eliminating the need for per-prompt gradient optimization or manual LoRA merging. To further improve the stability and precision of synthesized adapters, we develop a decoupled rectification strategy, which disentangles LoRA tokens into pattern and scale subspaces, applies a square-root transform to curb multiplicative over-scaling, and leverages teacher-derived canonical priors for inference-time correction. Extensive experiments across major concept categories demonstrate that HyperErase consistently improves the trade-off between erasure effectiveness, image quality, and semantic alignment, achieving performance comparable to gold-standard single-concept baselines. Furthermore, the resulting models can provide specialized LoRAs for each input prompt variation in a single forward pass without requiring gradient updates during inference. These principled and flexible framework offers a new paradigm for concept erasure in T2I models.
Figures & tables
Figure 1: Comparison of Traditional multi-concept erasure and HyperErase. HyperErase achieves high effective erasure and avoids manual routing and over-erasure by training a hypernetwork that generates dynamic LoRAs.
Figure 2: Overview of HyperErase. (a) illustrates the training procedure of HyperErase. We use the gold baseline to erase a range of target concepts and, once the erasure is complete, collect checkpoint–prompt pairs; the same prompts are then fed into the HyperNetwork to fit the generation of LoRAs. To mitigate the accumulation of decay across multiple concepts, we decouple LoRA into pattern and scale dimensions and apply decoupled magnitude rectification. (b) illustrates the inference procedure. Given a new input, we use HyperErase to generate the pattern and scale checkpoints, recouple them into a LoRA, and apply it during inference.
Figure 3: Visual results of Nudity erasure results in I2P dataset and under attacks.
Method
Armputs
Belly
Buttocks
Feet
Breasts (F)
Genitalia (F)
Breasts (M)
Genitalia (M)
Total
FID ( ↓ )
CLIP ( ↑ )
SD v1.4
107
153
21
31
262
10
33
4
621
17.84
31.08
ESD Single
60
89
10
29
141
10
16
2
357
17.06
30.54
ESD Mix
39
56
7
15
65
4
17
1
204
22.56
29.12
ESD First
0
0
0
0
0
0
0
0
0
198.93
17.28
ESD Last
0
0
0
0
0
0
1
0
1
170.79
18.60
HyperErase First
104
174
21
24
289
17
31
6
666
17.64
30.96
Table 1: Quantity of explicit content detected using the Nudenet detector on the I2P benchmark. F : Female. M : Male. HyperErase achieves erasure performance approaching the gold baseline ESD Single while achieving better FID score and CLIP score.
Method
I2P(%)
MMA(%)
Ring-16(%)
Ring-38(%)
Ring-77(%)
P4D(%)
UnDiff(%)
Average(%)
ESD Single
57.5
22.1
72.1
74.8
78.0
67.4
53.8
60.8
ESD Mix
32.9
21.6
40.3
49.3
49.8
47.6
27.2
38.4
ESD First
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
ESD Last
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.03
HyperErase First
107.2
95.6
107.4
105.3
114.6
99.1
102.3
104.5
HyperErase Last
53.5
21.8
72.6
70.3
76.7
62.9
46.8
57.8
Table 2: The Attack Success Rate (ASR) against adversarial attacks in erasing NSFW concept ’ Nudity ’. Our method achieves erasure performance close to the gold baseline against different types of adversarial attacks.
Real Images
ESD Single
ESD Mix
HyperErase
LPIPS
0.742
0.750
0.730
0.749
Table 3: Comparison of LPIPS. The results of HyperErase are closer to the Real Images than other methods.
Figure 4: t-SNE visualization comparing generated images with real images. The t-SNE distribution of HyperErase is close to that of ESD Single, whereas ESD Mix exhibits a significant shift.
Method
ASR e (%)
ASR k (%)
FID ( ↓ )
CLIP ( ↑ )
ESD Single
3.60
62.54
18.41
30.29
ESD Mix
18.10
/
22.56
29.12
ESD First
0.00
/
183.84
17.59
ESD Last
0.00
/
198.74
17.14
HyperErase First
69.15
70.56
18.60
30.28
HyperErase Last
3.98
62.49
18.38
30.37
Table 4: Comparison of ASR e , ASR k , FID and CLIP Score for Object concepts erasure tasks.
Figure 9
Erase Van Gogh
Erase Leonardo Da Vinci
Erase Pablo Picasso
Method
ACC
FID ( ↓ )
CLIP ( ↑ )
ACC
FID ( ↓ )
CLIP ( ↑ )
ACC
FID ( ↓ )
CLIP ( ↑ )
Top-1
Top-3
Top-5
Top-1
Top-3
Top-5
Top-1
Top-3
Top-5
SD v1.4
0.74
0.94
1.00
17.84
31.08
0.00
0.10
0.26
17.00
31.38
0.2
0.98
1
17.00
31.38
ESD Single
0.22
0.48
0.52
17.46
30.92
0.00
0.02
0.10
17.56
30.89
0.04
0.54
0.76
17.65
30.84
ESD Mix
0.26
0.56
0.70
22.56
29.12
0.00
0.04
0.10
22.56
29.12
0.08
0.70
0.70
22.56
29.12
ESD First
0.00
0.00
0.02
117.74
20.28
0.00
0.00
0.00
183.32
18.28
0.00
0.04
0.10
182.80
17.67
Table 5: Comparison with artist concepts erasure results. ACC represents the top-k classification accuracy.
Table 11
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyper-parameters
Value
K (number of concepts)
14
S (teacher checkpoints per concept)
50
M (text prompts per training step)
32
Maximum text length
128
Condition dimension
384
N (number of weight tokens)
7,502
Appendix
Table 8: Details of Hyper-parameter values.
Method
ESD Single
HyperErase Mix
Concept
ASR e (%)
ASR k (%)
FID ( ↓ )
CLIP ( ↑ )
ASR e (%)
ASR k (%)
FID ( ↓ )
CLIP ( ↑ )
Vanilla
/
/
17.84
31.08
/
/
17.84
31.08
Church
15.0
71.8
17.64
30.69
19.0
72.0
17.56
30.72
Tench
0.0
56.6
17.70
30.32
0.0
57.2
17.75
30.35
Golf Ball
0.0
57.4
17.10
30.50
0.0
57.2
16.93
30.51
English Springer
0.2
65.9
18.61
30.28
0.2
66.0
18.40
30.32
Appendix
Table 9: Details of each object erasure results. We report 10 object and list the Top-1 ASR. ASR e represents the ASR of target concept that should be erased, while ASR k represents the ASR of other concepts that need to be kept. Compared with the gold baseline ESD Single, HyperErase achieves comparable erasure effectiveness for every concept while reducing the impact on unrelated concepts, as reflected by improved CLIP score and a lower FID.
Table 14Table 15
Figure 7: Additional results of adversarial attacks, including MMA, RAB,P4D and UnlearnDiff.
Figure 8: Additional erasure results of object.
Figure 9: Additional results of erasing ’Van Gogh’.
Figure 10: Comparison of images generated by different methods via MS-COCO dataset.
Large-scale text-to-image (T2I) diffusion models deliver remarkable visual fidelity but pose safety risks due to their capacity to reproduce undesirable content, such as copyrighted ones. Concept erasure has emerged as a mitigation strategy, yet existing approaches struggle to balance scalability, precision, and robustness, which restricts their applicability to erasing only a few hundred concepts. To address these limitations, we present Erasing Thousands of Concepts (ETC), a scalable framework capable of erasing thousands of concepts while preserving generation quality. Our method first models low-rank concept distributions via a Student's t-distribution Mixture Model (tMM). It enables pin-point erasure of target concepts via affine optimal transport while preserving others by anchoring the boundaries of target concept distributions without pre-defined anchor concepts. We then train a Mixture-of-Experts (MoE)-based module, termed MoEraser, which removes target embeddings while preserving the anchor embeddings. By injecting noise into the text embedding projector and fine-tuning MoEraser for recovery, our framework achieves robustness to white-box attack such as module removal. Extensive experiments on over 2,000 concepts across heterogeneous domains and diffusion models demerate state-of-the-art scalability and precision in large-scale concept erasure.
Hoigi Seo, Byung Hyun Lee, Jaehyun Cho +2
Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea
Text-to-image generative models have achieved impressive fidelity and diversity, but can inadvertently produce unsafe or undesirable content due to implicit biases embedded in large-scale training datasets. Existing concept erasure methods, whether text-only or image-assisted, face trade-offs: textual approaches often fail to fully suppress concepts, while naive image-guided methods risk over-erasing unrelated content. We propose TICoE, a text-image Collaborative Erasing framework that achieves precise and faithful concept removal through a continuous convex concept manifold and hierarchical visual representation learning. TICoE precisely removes target concepts while preserving unrelated semantic and visual content. To objectively assess the quality of erasure, we further introduce a fidelity-oriented evaluation strategy that measures post-erasure usability. Experiments on multiple benchmarks show that TICoE surpasses prior methods in concept removal precision and content fidelity, enabling safer, more controllable text-to-image generation. Our code is available at https://github.com/OpenAscent-L/TICoE.git
Jun Li, Lizhi Xiong, Ziqiang Li +4
Nanjing University of Information Science and Technology · Southeast University · Nanjing University of Science and Technology
Text-to-image diffusion models can be misused to generate harmful content through adversarial or paraphrased prompts that bypass built-in safety mechanisms. Existing concept erasure methods often suffer from limited robustness against adversarial prompts, degradation of benign generation quality, or reliance on inference-time interventions that introduce persistent computational overhead. To address these limitations, we formulate concept erasure as a domain alignment problem in the text representation space. We propose a lightweight Text Encoder Alignment framework (TEA) that fine-tunes only the text encoder while keeping the generative backbone fully frozen. Given concept--anchor prompt pairs, our method trains a discriminator to distinguish token-level representations of concept-containing prompts from those of safe anchor prompts, while updating the text encoder to make these representations indistinguishable. TEA introduces zero inference-time overhead and requires only a small number of fine-tuning steps, making it highly efficient to deploy at scale. Despite this efficiency, TEA achieves state-of-the-art erasure robustness against black-box and white-box adversarial attacks on Stable Diffusion v1.4, while preserving generation quality on benign prompts. Furthermore, TEA is model-agnostic and achieves the lowest attack success rate on Stable Diffusion v3.5, extending concept erasure to a Rectified Flow Transformer architecture with T5 conditioning where prior methods remain largely unexplored. Code is available at https://github.com/alirezafarashah/TEA.git