HyperErase: Scale-Calibrated Hypernetwork for Multi-Concept Erasure in Text-to-Image Models

Yi Sun; Xinhao Zhong; Zhiqi Zhang; Yimin Zhou; Junhao Li; Yuxia Qiao

cs.CVSep 25, 2026

HyperErase: Scale-Calibrated Hypernetwork for Multi-Concept Erasure in Text-to-Image Models

Authors: Yi Sun, Xinhao Zhong, Zhiqi Zhang, Yimin Zhou, Junhao Li, Yuxia Qiao

Organizations: Harbin Institute of Technology, Shenzhen · Jilin University · Tsinghua Shenzhen International Graduate School, Tsinghua University · South China University of Technology

Abstract

Recent advances in text-to-image (T2I) generation have substantially improved visual synthesis, but have also raised increasing safety concerns due to their potential to generate harmful or undesirable content. Existing concept erasure methods predominantly follow a static weight paradigm, producing a single frozen adapter that struggles to adapt to diverse prompt variations and suffers from parameter interference when scaling to multiple concepts. We propose HyperErase, a framework for concept erasure based on hypernetwork-driven prompt-conditioned parameter synthesis. Our approach first reframes concept erasure as prompt-conditioned parameter amortization and trains a hypernetwork to map textual descriptions to prompt-specific LoRA updates, eliminating the need for per-prompt gradient optimization or manual LoRA merging. To further improve the stability and precision of synthesized adapters, we develop a decoupled rectification strategy, which disentangles LoRA tokens into pattern and scale subspaces, applies a square-root transform to curb multiplicative over-scaling, and leverages teacher-derived canonical priors for inference-time correction. Extensive experiments across major concept categories demonstrate that HyperErase consistently improves the trade-off between erasure effectiveness, image quality, and semantic alignment, achieving performance comparable to gold-standard single-concept baselines. Furthermore, the resulting models can provide specialized LoRAs for each input prompt variation in a single forward pass without requiring gradient updates during inference. These principled and flexible framework offers a new paradigm for concept erasure in T2I models.

Figures & tables

Figure 1: Comparison of Traditional multi-concept erasure and HyperErase. HyperErase achieves high effective erasure and avoids manual routing and over-erasure by training a hypernetwork that generates dynamic LoRAs.

Figure 2: Overview of HyperErase. (a) illustrates the training procedure of HyperErase. We use the gold baseline to erase a range of target concepts and, once the erasure is complete, collect checkpoint–prompt pairs; the same prompts are then fed into the HyperNetwork to fit the generation of LoRAs. To mitigate the accumulation of decay across multiple concepts, we decouple LoRA into pattern and scale dimensions and apply decoupled magnitude rectification. (b) illustrates the inference procedure. Given a new input, we use HyperErase to generate the pattern and scale checkpoints, recouple them into a LoRA, and apply it during inference.

Figure 3: Visual results of Nudity erasure results in I2P dataset and under attacks.

Method	Armputs	Belly	Buttocks	Feet	Breasts (F)	Genitalia (F)	Breasts (M)	Genitalia (M)	Total	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )
SD v1.4	107	153	21	31	262	10	33	4	621	17.84	31.08
ESD Single	60	89	10	29	141	10	16	2	357	17.06	30.54
ESD Mix	39	56	7	15	65	4	17	1	204	22.56	29.12
ESD First	0	0	0	0	0	0	0	0	0	198.93	17.28
ESD Last	0	0	0	0	0	0	1	0	1	170.79	18.60
HyperErase First	104	174	21	24	289	17	31	6	666	17.64	30.96

Table 1: Quantity of explicit content detected using the Nudenet detector on the I2P benchmark. F : Female. M : Male. HyperErase achieves erasure performance approaching the gold baseline ESD Single while achieving better FID score and CLIP score.

Method	I2P(%)	MMA(%)	Ring-16(%)	Ring-38(%)	Ring-77(%)	P4D(%)	UnDiff(%)	Average(%)
ESD Single	57.5	22.1	72.1	74.8	78.0	67.4	53.8	60.8
ESD Mix	32.9	21.6	40.3	49.3	49.8	47.6	27.2	38.4
ESD First	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
ESD Last	0.2	0.0	0.0	0.0	0.0	0.0	0.0	0.03
HyperErase First	107.2	95.6	107.4	105.3	114.6	99.1	102.3	104.5
HyperErase Last	53.5	21.8	72.6	70.3	76.7	62.9	46.8	57.8

Table 2: The Attack Success Rate (ASR) against adversarial attacks in erasing NSFW concept ’ Nudity ’. Our method achieves erasure performance close to the gold baseline against different types of adversarial attacks.

	Real Images	ESD Single	ESD Mix	HyperErase
LPIPS	0.742	0.750	0.730	0.749

Table 3: Comparison of LPIPS. The results of HyperErase are closer to the Real Images than other methods.

Panel of Figure 4 — Figure 4: t-SNE visualization comparing generated images with real images. The t-SNE distribution of HyperErase is close to that of ESD Single, whereas ESD Mix exhibits a significant shift.

Method	ASR e (%)	ASR k (%)	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )
ESD Single	3.60	62.54	18.41	30.29
ESD Mix	18.10	/	22.56	29.12
ESD First	0.00	/	183.84	17.59
ESD Last	0.00	/	198.74	17.14
HyperErase First	69.15	70.56	18.60	30.28
HyperErase Last	3.98	62.49	18.38	30.37

Table 4: Comparison of ASR e , ASR k , FID and CLIP Score for Object concepts erasure tasks.

	Erase Van Gogh					Erase Leonardo Da Vinci					Erase Pablo Picasso
Method	ACC			FID ( $\downarrow$ )	CLIP ( $\uparrow$ )	ACC			FID ( $\downarrow$ )	CLIP ( $\uparrow$ )	ACC			FID ( $\downarrow$ )	CLIP ( $\uparrow$ )
Method	Top-1	Top-3	Top-5	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )	Top-1	Top-3	Top-5	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )	Top-1	Top-3	Top-5	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )
SD v1.4	0.74	0.94	1.00	17.84	31.08	0.00	0.10	0.26	17.00	31.38	0.2	0.98	1	17.00	31.38
ESD Single	0.22	0.48	0.52	17.46	30.92	0.00	0.02	0.10	17.56	30.89	0.04	0.54	0.76	17.65	30.84
ESD Mix	0.26	0.56	0.70	22.56	29.12	0.00	0.04	0.10	22.56	29.12	0.08	0.70	0.70	22.56	29.12
ESD First	0.00	0.00	0.02	117.74	20.28	0.00	0.00	0.00	183.32	18.28	0.00	0.04	0.10	182.80	17.67

Table 5: Comparison with artist concepts erasure results. ACC represents the top-k classification accuracy.

Table 11

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyper-parameters	Value
$K$ (number of concepts)	$14$
$S$ (teacher checkpoints per concept)	$50$
$M$ (text prompts per training step)	$32$
Maximum text length	$128$
Condition dimension	$384$
$N$ (number of weight tokens)	$7{,}502$

Appendix

Table 8: Details of Hyper-parameter values.

Method	ESD Single				HyperErase Mix
Concept	ASR e (%)	ASR k (%)	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )	ASR e (%)	ASR k (%)	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )
Vanilla	/	/	17.84	31.08	/	/	17.84	31.08
Church	15.0	71.8	17.64	30.69	19.0	72.0	17.56	30.72
Tench	0.0	56.6	17.70	30.32	0.0	57.2	17.75	30.35
Golf Ball	0.0	57.4	17.10	30.50	0.0	57.2	16.93	30.51
English Springer	0.2	65.9	18.61	30.28	0.2	66.0	18.40	30.32

Appendix

Table 9: Details of each object erasure results. We report 10 object and list the Top-1 ASR. ASR e represents the ASR of target concept that should be erased, while ASR k represents the ASR of other concepts that need to be kept. Compared with the gold baseline ESD Single, HyperErase achieves comparable erasure effectiveness for every concept while reducing the impact on unrelated concepts, as reflected by improved CLIP score and a lower FID.

Table 14

Table 15

Figure 7: Additional results of adversarial attacks, including MMA, RAB,P4D and UnlearnDiff.

Figure 8: Additional erasure results of object.

Figure 9: Additional results of erasing ’Van Gogh’.

Figure 10: Comparison of images generated by different methods via MS-COCO dataset.

Safe T2I Generation

Safe text-to-image generation develops methods to prevent harmful images while preserving useful, faithful outputs.

See all 53 papers

Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models
Apr 12, 2026Hoigi Seo, Byung Hyun Lee, Jaehyun Cho +2Safe T2I Generation Diffusion Model Unlearning
Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea
Beyond Text Prompts: Precise Concept Erasure through Text-Image Collaboration
Apr 17, 2026Jun Li, Lizhi Xiong, Ziqiang Li +4Safe T2I Generation Controllable Image Generation
Nanjing University of Information Science and Technology · Southeast University · Nanjing University of Science and Technology
TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
Aug 15, 2026Alireza Dehghanpour Farashah, Zhuan Shi, Negar Rostamzadeh +1Safe T2I Generation Adversarial Robustness
Mila – Quebec AI Institute · McGill University · Google Research

Apr 12, 2026J/K move · Enter open · S save

Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models

Hoigi Seo, Byung Hyun Lee, Jaehyun Cho, Sungjin Lim+1

Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea

Safe T2I Generation Diffusion Model Unlearning T2I Diffusion Models

Open paper AskPDF arXiv

Large-scale text-to-image (T2I) diffusion models deliver remarkable visual fidelity but pose safety risks due to their capacity to reproduce undesirable content, such as copyrighted ones. Concept erasure has emerged as a mitigation strategy, yet existing approaches struggle to balance scalability, precision, and robustness, which restricts their applicability to erasing only a few hundred concepts. To address these limitations, we present Erasing Thousands of Concepts (ETC), a scalable framework capable of erasing thousands of concepts while preserving generation quality. Our method first models low-rank concept distributions via a Student's t-distribution Mixture Model (tMM). It enables pin-point erasure of target concepts via affine optimal transport while preserving others by anchoring the boundaries of target concept distributions without pre-defined anchor concepts. We then train a Mixture-of-Experts (MoE)-based module, termed MoEraser, which removes target embeddings while preserving the anchor embeddings. By injecting noise into the text embedding projector and fine-tuning MoEraser for recovery, our framework achieves robustness to white-box attack such as module removal. Extensive experiments on over 2,000 concepts across heterogeneous domains and diffusion models demerate state-of-the-art scalability and precision in large-scale concept erasure.

Abstract

Safe T2I Generation

Explore similar work

Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models

Beyond Text Prompts: Precise Concept Erasure through Text-Image Collaboration

TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models