Localized Concept Erasure in Text-to-Image Diffusion Models via High-Level Representation Misdirection

Uichan Lee; Jeonghyeon Kim; Sangheum Hwang

cs.CVFeb 23, 2026

Localized Concept Erasure in Text-to-Image Diffusion Models via High-Level Representation Misdirection

Authors: Uichan Lee, Jeonghyeon Kim, Sangheum Hwang

Organizations: Department of Data Science, Seoul National University of Science and Technology

Abstract

Recent advances in text-to-image (T2I) diffusion models have seen rapid and widespread adoption. However, their powerful generative capabilities raise concerns about potential misuse for synthesizing harmful, private, or copyrighted content. To mitigate such risks, concept erasure techniques have emerged as a promising solution. Prior works have primarily focused on fine-tuning the denoising component (e.g., the U-Net backbone). However, recent causal tracing studies suggest that visual attribute information is localized in the early self-attention layers of the text encoder, indicating a potential alternative for concept erasing. Building on this insight, we conduct preliminary experiments and find that directly fine-tuning early layers can suppress target concepts but often degrades the generation quality of non-target concepts. To overcome this limitation, we propose High-Level Representation Misdirection (HiRM), which misdirects high-level semantic representations of target concepts in the text encoder toward designated vectors such as random directions or semantically defined directions (e.g., supercategories), while updating only early layers that contain causal states of visual attributes. Our decoupling strategy enables precise concept removal with minimal impact on unrelated concepts, as demonstrated by strong results on UnlearnCanvas and NSFW benchmarks across diverse targets (e.g., objects, styles, nudity). HiRM also preserves generative utility at low training cost, transfers to state-of-the-art architectures such as Flux without additional training, and shows synergistic effects with denoiser-based concept erasing methods.

Figures & tables

Methods	I2P-931 ↓	COCO-1k
Methods	I2P-931 ↓	LPIPS ↓	CLIP↑
SD	24.81%	-	0.310
Diff-Q ( Basu et al., 2023 )	7.09%	0.34	0.273
Diff-Q *	0.75%	0.60	0.187

Table 1: Evaluation of Diff-QuickFix on I2P-931 (a subset of I2P) and COCO-1k.

Panel of Figure 1 — Figure 1: Overview of HiRM. (a) HiRM updates only the first block of the text encoder while steering the final-layer representations of target concepts (e.g., “Van Gogh”) toward designated directions, effectively decoupling update location and erasure target. (b) Compared to existing methods, HiRM achieves a better balance between concept erasure and utility preservation. (c) HiRM demonstrates favorable trade-offs in terms of training time and overall performance, measured as the average of erasure and retention scores across style, object, and robustness settings.

Method	Weight	Training	Style erasing (%)				Object erasing (%)				Efficiency ( s, GB )
Method	Modification	based	UA $\uparrow$	IRA $\uparrow$	CRA $\uparrow$	AA $\uparrow$	UA $\uparrow$	IRA $\uparrow$	CRA $\uparrow$	AA $\uparrow$	Time $\downarrow$	Mem. $\downarrow$
SAFREE ( Yoon et al., 2024 )	✗	✗	53.50	80.66	97.86	77.34	29.85	98.82	95.58	74.75	-	-
TraSCE ( Jain et al., 2024 )	✗	✗	72.35	76.06	96.84	81.75	49.85	97.87	95.59	81.10	-	-
UCE ( Gandikota et al., 2024 )	✓	✗	98.40	60.22	47.71	68.78	94.31	39.35	34.67	56.11	-	6.47
RECE ( Gong et al., 2024 )	✓	✗	78.40	97.63	98.48	91.50	84.20	98.39	97.73	93.44	-	19.32
Diff-Q ( Basu et al., 2023 )	✓	✗	96.40	93.91	97.13	95.81	94.00	98.37	96.21	96.19	-	1.60

Table 2: Evaluation of erasing methods on the UnlearnCanvas benchmark.

Method	Weight	Training	Ring-A-Bell (%) $\downarrow$			Unlearn-	MMA-	I2P (%) $\downarrow$	COCO
Method	Modification	based	K-16	K-38	K-77	DiffAtk (%) $\downarrow$	Diffusion (%) $\downarrow$	I2P (%) $\downarrow$	FID ↓	CLIP ↑
SD1.4	-	-	95.79	95.79	89.47	78.17	67.90	17.80	-	0.315
SAFREE ( Yoon et al., 2024 )	✗	✗	47.37	40.00	26.32	40.85	30.30	0.64	6.77	0.312
TraSCE ( Jain et al., 2024 )	✗	✗	4.21	3.16	0.00	16.90	15.60	0.45	10.39	0.305
UCE ( Gandikota et al., 2024 )	✓	✗	6.32	3.16	6.32	35.21	11.60	0.89	9.77	0.306
RECE ( Gong et al., 2024 )	✓	✗	1.05	0.00	1.05	21.83	0.40	0.57	29.65	0.277

Table 3: Evaluation of erasing methods on various adversarial prompt benchmarks and COCO.

Panel of Figure 2 — Figure 2: Comparison of t-SNE visualizations of HiRM-R. Each figure compares token embeddings before and after erasure, where blue circles represent the original embeddings and orange X markers represent the embeddings after erasure. The left columns of each figure visualize embeddings from the final transformer block, and the right columns show embeddings from the first block.

Method	Ring-16 ↓	Ring-38 ↓	Ring-77 ↓	MMA ↓	I2P ↓	COCO-1k (CLIP) ↑
Flux1.dev ( Labs, 2024 )	88.42	87.37	87.37	25.40	4.49	0.308
ESD ( Gandikota et al., 2023 )	41.05	33.68	32.63	7.10	2.93	0.307
CA ( Kumari et al., 2023 )	11.58	11.58	6.32	3.20	1.83	0.302
UCE ( Gandikota et al., 2024 )	65.26	66.32	62.11	9.30	3.27	0.309
EraseAnything ( Gao et al., 2025 )	29.47	24.21	26.32	6.60	2.64	0.305
Diff-Q ( Basu et al., 2023 )	57.89	51.58	64.21	12.20	4.44	0.306

Table 4: Evaluation of various methods across multiple benchmarks based on the Flux1.dev architecture.

Method	Ring-16 ↓	Ring-38 ↓	Ring-77 ↓	MMA ↓	I2P ↓	COCO-1k (CLIP) ↑
ESD ( Gandikota et al., 2023 )	41.05	33.68	32.63	7.10	2.93	0.307
+ HiRM-R (Ours)	12.63	7.37	4.21	3.30	2.51	0.306
CA ( Kumari et al., 2023 )	11.58	11.58	6.32	3.20	1.83	0.302
+ HiRM-R (Ours)	3.16	2.11	2.11	4.40	1.11	0.302
EraseAnything ( Gao et al., 2025 )	29.47	24.21	26.32	6.60	2.64	0.305
+ HiRM-R (Ours)	3.16	1.05	3.16	2.50	1.57	0.305

Table 5: Synergistic effects of combining HiRM-R with denoiser-based concept erasing methods on the Flux1.dev architecture.

	Adversarial Robustness					Multi Concept		COCO-10k
Method	Ring-16↓	Ring-38↓	Ring-77↓	MMA↓	I2P↓	Acc e ↓	Acc r ↑	FID↓	CLIP↑
SPEED ( Li et al., 2025 )	-	-	-	-	-	3.46	88.48	-	-
HiRM-S (Ours)	1.05	1.05	0.00	3.30	0.66	-	-	6.75	0.306
S-HiRM-S (w/ Ours)	1.05	1.05	2.11	1.70	0.43	3.64	79.30	7.43	0.304

Table 6: Synergistic effects of combining HiRM-S (text encoder) and SPEED (U-Net) for adversarial robustness and multi-concept erasure.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Style Concepts

"Abstractionism" , "Artist_Sketch" , "Blossom_Season" , "Bricks" , "Byzantine" , "Cartoon" , "Cold_Warm" , "Color_Fantasy" , "Comic_Etch" , "Crayon" , "Cubism" , "Dadaism" , "Dapple" , "Defoliation" , "Early_Autumn" , "Expressionism" , "Fauvism" , "French" , "Glowing_Sunset" , "Gorgeous_Love" , "Greenfield" , "Impressionism" , "Ink_Art" , "Joy" , "Liquid_Dreams" , "Magic_Cube" , "Meta_Physics" , "Meteor_Shower" , "Monet" , "Mosaic" , "Neon_Lines" , "On_Fire" , "Pastel" , "Pencil_Drawing" , "Picasso" , "Pop_Art" , "Red_Blue_Ink" , "Rust" , "Seed_Images" , "Sketch" , "Sponge_Dabbed" , "Structuralism" , "Superstring" , "Surrealism" , "Ukiyoe" , "Van_Gogh" , "Vibrant_Flow" , "Warm_Love" , "Warm_Smear" , "Watercolor" , "Winter"

Object Concepts

"Architectures" , "Bears" , "Birds" , "Butterfly" , "Cats" , "Dogs" , "Fishes" , "Flame" , "Flowers" , "Frogs" , "Horses" , "Human" , "Jellyfish" , "Rabbits" , "Sandwiches" , "Sea" , "Statues" , "Towers" , "Trees" , "Waterfalls"

Appendix

Table 7: Full list of style and object concepts used for concept erasing evaluation.

Figure 3: MLLM-as-a-judge prompt example.

Figure 4: HiRM-R sample response.

Figure 5: Stable Diffusion (SD) sample response.

	Metric	UCE	ESD-x	ESD-u	GA	TaskVec	STEREO	RECE	HiRM-S
Diffusion Completion $\downarrow$	CLIP (t=5)	27.7	27.2	26.9	24.0	23.8	23.9	28.8	25.8
	Class Acc. (%)	42.7	37.8	32.5	1.1	2.4	3.2	36.5	9.4
Noise-Based Probing $\downarrow$	CLIP	27.8	28.0	27.7	26.1	26.5	24.6	27.0	24.6
	Class Acc. (%)	21.9	30.7	27.7	2.67	11.0	1.1	13.0	7.0
Unrelated Concept $\uparrow$	CLIP	31.2	30.8	30.7	28.8	29.4	29.0	30.5	31.0
	Class Acc. (%)	75.0	71.3	70.4	52.2	60.4	52.8	71.7	81.1

Appendix

Table 8: Evaluation of different erasing methods under diffusion-completion and noise-based probing attacks on object and style concepts, including UCE ( Gandikota et al., 2024 ) , ESD ( Gandikota et al., 2023 ) , GA, TaskVec ( Pham et al., 2024 ) , STEREO ( Srivatsan et al., 2025 ) , RECE ( Gong et al., 2024 ) , and HiRM-S (ours).

Metric	Accuracy (%)
Average Unlearning Accuracy	88.33
In-domain Accuracy	65.56
Cross-domain Accuracy	98.80

Appendix

Table 9: Quantitative results on multi-concept erasure (6 styles) using simple weight averaging.

Concept	HiRM-S (Keyword)			HiRM-S (Template)
Concept	UA $\uparrow$	IRA $\uparrow$	CRA $\uparrow$	UA $\uparrow$	IRA $\uparrow$	CRA $\uparrow$
Bricks	100.0	94.6	98.3	100.0	89.4	97.9
Cartoon	100.0	91.4	97.4	100.0	92.5	97.3
Crayon	100.0	92.2	97.6	100.0	91.4	97.2
Van Gogh	95.0	94.2	97.7	100.0	96.0	98.4
Dapple	100.0	94.3	99.0	100.0	96.4	98.8

Appendix

Table 10: Ablation of keyword- vs. template-gated pipelines on style concepts from UnlearnCanvas.

Method	Ring-16 ↑	Ring-38 ↑	Ring-77 ↑	Avg ↑	MLLM-as-a-judge Rank ↓	NudeNet Rank ↓
SD 1.4	1.13	1.16	1.25	1.18	13	13
SAFREE ( Yoon et al., 2024 )	2.15	2.33	2.32	2.27	11	10
TraSCE ( Jain et al., 2024 )	4.26	4.11	4.15	4.17	8	5
UCE ( Gandikota et al., 2024 )	4.26	4.53	4.20	4.33	7	7
RECE ( Gong et al., 2024 )	4.77	4.77	4.66	4.73	3	1
Diff-Q ( Basu et al., 2023 )	4.63	4.53	4.47	4.54	5	7

Appendix

Table 11: Evaluation results across Ring-A-Bell benchmarks, average score, and ranking. Scores are derived from the MLLM-as-a-judge evaluation and correspond to the average safety score on a 1–5 scale for each dataset.

Target layer	I2P-931 ↓	COCO-1k
Target layer	I2P-931 ↓	CLIP↑	LPIPS↓
11-mlp-fc2	13.10%	29.99	0.25
11- $W_{\text{out}}$	2.47%	30.14	0.24
10-mlp-fc2	4.83%	29.61	0.27
10- $W_{\text{out}}$	3.54%	29.52	0.27
9-mlp-fc2	11.60%	29.99	0.25

Appendix

Table 12: Comparison of target layers on I2P-931(a subset of I2P) and COCO-1k datasets.

Seed	I2P-931	COCO-1k
SD1.4	–	30.99
42	2.47%	30.14
52	2.26%	30.14
62	3.11%	29.64
82	3.33%	29.66
72	2.04%	29.81

Appendix

Table 13: Comparison across different seed values

Coefficient	I2P-931	COCO-1k
SD1.4	–	31.00
300	2.04%	30.14
400	1.93%	29.90
500	2.47%	30.14
600	1.18%	30.08
700	1.18%	30.03

Appendix

Table 14: Comparison across different coefficient values

Figure 6: Qualitative comparison of concept erasure methods on the removal of the Vibrant Flow style from the UnlearnCanvas benchmark. The top row shows generations from prompts containing the target concept. The second and third rows depict in-domain ( Ink Art , same style domain) and cross-domain ( Frog , object domain) prompts, respectively, which are semantically unrelated to the erased concept.

Figure 7: Qualitative comparison of concept erasure methods on the removal of the Vibrant Flow style from the UnlearnCanvas benchmark. Each row’s images are generated from prompts such as “A Trees in Vibrant Flow style”, “A Rabbits image in Cartoon style”, “A Trees image in Liquid Dreams style”, “A Waterfall image in Ukiyoe style”, and “A Frogs image in Van Gogh style”.

Figure 8: Qualitative comparison of concept erasure methods on the removal of the Ink Art style from the UnlearnCanvas benchmark. Each row’s images are generated from prompts such as “A Dogs in Ink Art style”, “A Rabbits image in Cartoon style”, “A Trees image in Liquid Dreams style”, “A Waterfalls image in Ukiyoe style”, and “A Frogs image in Van Gogh style”.

Figure 9: Qualitative comparison of concept erasure methods on the removal of the Artist Sketch style from the UnlearnCanvas benchmark. Each row’s images are generated from prompts such as “A Birds in Artist Sketch style”, “A Dogs image in Dapple style”, “A Architectures image in Ink Art style”, “A Butterfly image in Neon Lines style”, and “A Towers image in Pastel style”.

Figure 10: Qualitative comparison of concept erasure methods on the removal of the Meta Physics style from the UnlearnCanvas benchmark. Each row’s images are generated from prompts such as “A Flowers in Meta Physics style”, “A Rabbits image in Sketch style”, “A Trees image in Picasso style”, “A Waterfalls image in French style”, and “A Frogs image in Meteor Shower style”.

Figure 11: Qualitative comparison of concept erasure methods on the removal of the Towers object from the UnlearnCanvas benchmark. Each row’s images are generated from prompts such as “A Towers in Bricks style”, “A Rabbits image in Cartoon style”, “A Trees image in Liquid Dreams style”, “A Waterfalls image in Ukiyoe style”, and “A Frogs image in Van Gogh style”.

Figure 12: Qualitative comparison of concept erasure methods on the removal of the Trees object from the UnlearnCanvas benchmark. Each row’s images are generated from prompts such as “A Trees in Crayon style”, “A Bears image in Sketch style”, “A Rabbits image in Picasso style”, “A Horses image in French style”, and “A Flame image in Meteor Shower style”.

Figure 13: Qualitative comparison of concept erasure methods on the removal of the Sandwiches object from the UnlearnCanvas benchmark. Each row’s images are generated from prompts such as “A Sandwiches in Meteor Shower style”, “A Jellyfish image in Monet style”, “A Trees image in On Fire style”, “A Flowers image in Red Blue Ink style”, and “A Fishes image in Warm Smear style”.

Figure 14: Qualitative comparison of concept erasure methods on Flux1.dev using the Adversarial benchmark.

Figure 15: Qualitative results showing the synergistic effect of combining denoiser-based methods with HiRM-R on Flux1.dev using the Adversarial benchmark.

Panel of Figure 16 — Figure 16: Comparison of t-SNE visualizations of HiRM-R. Each figure compares token embeddings before and after erasure, where blue circles represent the original embeddings and orange X markers represent the embeddings after erasure. The left columns of each figure visualize embeddings from the final transformer block, and the right columns show embeddings from the first block.

Panel of Figure 17 — Figure 17: Comparison of t-SNE visualizations of HiRM-S. Each figure compares token embeddings before and after erasure, where blue circles represent the original embeddings and orange X markers represent the embeddings after erasure. The left columns of each figure visualize embeddings from the final transformer block, and the right columns show embeddings from the first block.

Panel of Figure 18 — Figure 18: Analysis of neuron activation shifts using Jaccard Similarity. We visualize the overlap of the Top-50 activated neurons between the original and HiRM-unlearned text encoders across 51 style concepts. The target concept (’Van Gogh’, red bar) exhibits a drastic drop in similarity in both (a) the first embedding and (b) the last embedding, indicating a fundamental disruption of its neural representation. In contrast, non-target concepts (blue bars) maintain high stability, demonstrating that HiRM selectively erases the target while preserving unrelated attributes.

Figure 19: Qualitative comparison of concept erasure methods under LoRA-tuned U-Net settings. Each column shows images generated from the prompt “An image of beautiful Object ” using different erasure methods. SD1.5 uses the original text encoder with a LoRA-tuned U-Net. UCE and MACE operate on erased U-Nets combined with LoRA weights. DIFF-Q and HiRM-R modify only the text encoder, preserving the original U-Net and LoRA weights. Unlike UCE and MACE, our method (HiRM-R) successfully removes the target concept (i.e., “dog”) while preserving unrelated generations (e.g., “lake”, “cat”), demonstrating strong transferability with LoRA-based downstream adaptation.

Figure 20: Qualitative comparison of concept erasure methods under LoRA-tuned U-Net settings. Each column shows images generated from the prompt “An image of beautiful Object ” using different erasure methods. SD1.5 uses the original text encoder with a LoRA-tuned U-Net. UCE and MACE operate on erased U-Nets combined with LoRA weights. DIFF-Q and HiRM-R modify only the text encoder, preserving the original U-Net and LoRA weights. Unlike UCE and MACE, our method (HiRM-R) successfully removes the target concept (i.e., “frog”) while preserving unrelated generations (e.g., “sea”, “rabbit”), demonstrating strong transferability with LoRA-based downstream adaptation.

Diffusion Model Unlearning

Diffusion model unlearning removes specified concepts or training examples from generative models while preserving unrelated generation.

See all 75 papers

Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models
Apr 12, 2026Hoigi Seo, Byung Hyun Lee, Jaehyun Cho +2Safe T2I Generation Diffusion Model Unlearning
Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea
VisualErase: Dual-Branch Visual Trajectory Redirection for Robust Concept Erasure in Text-to-Image Diffusion Models
Oct 4, 2026Qianlong Xiang, Miao Zhang, Kun Wang +3Diffusion Model Unlearning T2I Diffusion Models
Harbin Institute of Technology (Shenzhen) · City University of Hong Kong · Shenzhen Loop Area Institute +2
Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models
Mar 27, 2026Zhuan Shi, Alireza Dehghanpour Farashah, Rik de Vries +1Concept Erasure T2I Diffusion Models
McGill University · Mila - Quebec AI Institute · EPFL

Apr 12, 2026J/K move · Enter open · S save

Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models

Hoigi Seo, Byung Hyun Lee, Jaehyun Cho, Sungjin Lim+1

Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea

Safe T2I Generation Diffusion Model Unlearning T2I Diffusion Models

Open paper AskPDF arXiv

Large-scale text-to-image (T2I) diffusion models deliver remarkable visual fidelity but pose safety risks due to their capacity to reproduce undesirable content, such as copyrighted ones. Concept erasure has emerged as a mitigation strategy, yet existing approaches struggle to balance scalability, precision, and robustness, which restricts their applicability to erasing only a few hundred concepts. To address these limitations, we present Erasing Thousands of Concepts (ETC), a scalable framework capable of erasing thousands of concepts while preserving generation quality. Our method first models low-rank concept distributions via a Student's t-distribution Mixture Model (tMM). It enables pin-point erasure of target concepts via affine optimal transport while preserving others by anchoring the boundaries of target concept distributions without pre-defined anchor concepts. We then train a Mixture-of-Experts (MoE)-based module, termed MoEraser, which removes target embeddings while preserving the anchor embeddings. By injecting noise into the text embedding projector and fine-tuning MoEraser for recovery, our framework achieves robustness to white-box attack such as module removal. Extensive experiments on over 2,000 concepts across heterogeneous domains and diffusion models demerate state-of-the-art scalability and precision in large-scale concept erasure.

Abstract

Diffusion Model Unlearning

Explore similar work

Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models

VisualErase: Dual-Branch Visual Trajectory Redirection for Robust Concept Erasure in Text-to-Image Diffusion Models

Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models