Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models

Zhuan Shi; Alireza Dehghanpour Farashah; Rik de Vries; Golnoosh Farnadi

cs.CVMar 27, 2026

Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models

Authors: Zhuan Shi, Alireza Dehghanpour Farashah, Rik de Vries, Golnoosh Farnadi

Organizations: McGill University · Mila - Quebec AI Institute · EPFL

Abstract

Concept erasure in text-to-image diffusion models seeks to remove undesired concepts while preserving overall generative capability. Localized erasure methods aim to restrict edits to the spatial region occupied by the target concept. However, we observe that suppressing a concept can unintentionally weaken semantically related neighbor concepts, reducing fidelity in fine-grained domains. We propose Neighbor-Aware Localized Concept Erasure (NLCE), a training-free framework designed to better preserve neighboring concepts while removing target concepts. It operates in three stages: (1) a spectrally-weighted embedding modulation that attenuates target concept directions while stabilizing neighbor concept representations, (2) an attention-guided spatial gate that identifies regions exhibiting residual concept activation, and (3) a spatially-gated hard erasure that eliminates remaining traces only where necessary. This neighbor-aware pipeline enables localized concept removal while maintaining the surrounding concept neighborhood structure. Experiments on fine-grained datasets (Oxford Flowers, Stanford Dogs) show that our method effectively removes target concepts while better preserving closely related categories. Additional results on celebrity identity, explicit content and artistic style demonstrate robustness and generalization to broader erasure scenarios.

Figures & tables

Figure 1: Erasure Effectiveness and Neighbor Retention: GLoCE vs. NLCE (Ours)

Figure 2: Overview of NLCE . The method proceeds in three stages: (1) representation-space modulation to suppress the target subspace while reinforcing neighbors; (2) attention-guided gating to localize residual concept activations; (3) gated feature clean-up to irreversibly remove remaining traces. For multi-concept erasure, we activate operators per concept based on prompt tokens or embedding similarity.

Method	Oxford Flowers										Stanford Dogs
	Alpine Sea Holly					Camellia					Chesapeake Bay Retriever					Bluetick
	Acc t (↓)	Acc r (↑)	$H_{o}$ (↑)	CS (↑)	KID (↓)	Acc t (↓)	Acc r (↑)	$H_{o}$ (↑)	CS (↑)	KID (↓)	Acc t (↓)	Acc r (↑)	$H_{o}$ (↑)	CS (↑)	KID (↓)	Acc t (↓)	Acc r (↑)	$H_{o}$ (↑)	CS (↑)	KID (↓)
SD v1.4	100.00	100.00	0.00	32.52	-	100.00	100.00	0.00	32.55	-	100.00	100.00	0.00	34.97	-	100.00	100.00	0.00	34.98	-
MACE	4.00	58.67	72.83	30.81	0.92	4.00	54.18	69.27	31.37	0.61	0.00	42.58	59.73	33.49	0.96	0.00	45.63	62.67	33.87	0.73
SPM	4.00	68.97	80.27	32.72	0.20	28.00	69.09	70.52	32.75	0.17	8.00	60.43	72.95	34.96	0.07	12.00	60.77	71.90	34.99	0.06
ESD-x	4.00	69.70	80.76	31.98	0.11	8.00	63.27	74.98	31.59	0.30	24.00	49.51	59.96	33.63	0.59	12.00	67.10	76.14	34.46	0.12

Table 1: Quantitative Comparison of Oxford Flowers and Stanford Dogs Erasure. Our NLCE achieves a superior balance between the target erasure and neighbor concepts preservation while maintain the quality.

Panel of Figure 3 — Figure 3: Qualitative comparison of Oxford Flowers and Stanford Dogs Erasure. Top: Oxford Flowers — NLCE effectively removes the target concept ’ Alpine Sea Holly ’ while preserving neighbor concepts such as ’ Rose ’ and ’ Spring Crocus ’. Bottom: Stanford Dogs — NLCE successfully eliminates the target concept ’ Bluetick ’ while maintaining visually similar breeds.

Figure 5: Qualitative Comparison of Celebrity Erasure. Our NLCE can effectively remove the target celebrity while preserving non-target celebrity in the same image.

Figure 6: Quantitative Comparison on I2P Dataset. The number following each category represents the number of nude items generated by original SD, while each bar illustrates the success rate of erasing the corresponding nude items for each method. NLCE outperforms other baselines in most of categories and in total number of detected items.

Figure 7: Qualitative comparisons on the I2P dataset. The first four rows show samples from I2P, while the last two rows present examples from COCO Captions with non-nudity prompts.

Figure 8: Qualitative Comparison of Artistic Style Erasure. Our NLCE can effectively remove the target style ’ Van Gogh ’ while preserving style like ’ Pablo Picasso ’ and ’ Andy Warhol ’.

Variant	Erasure 10 Dogs					Erasure 10 Flowers
Variant	$Acc_{t}$ (↓)	$Acc_{r}$ (↑)	$H_{o}$ (↑)	$CS$ (↑)	$KID$ (↓)	$Acc_{t}$ (↓)	$Acc_{r}$ (↑)	$H_{o}$ (↑)	$CS$ (↑)	$KID$ (↓)
SD	100.00	100.00	0.00	34.99	-	100.00	100.00	0.00	32.70	-
MACE	0.40	1.52	3.00	21.67	10.68	1.03	0.00	0.00	20.16	15.51
SPM	43.20	58.05	57.42	34.97	0.10	58.80	67.50	51.17	32.84	0.23
ESD-x	24.80	35.57	48.30	32.40	1.57	23.60	34.50	47.53	30.34	1.19
ESD-u	19.20	27.19	40.69	32.06	2.55	32.80	29.33	40.84	29.42	2.04

Table 3: Quantitative Comparison of Multi-Concept Erasure. Our NLCE achieves a superior balance between the target erasure and neighbor concepts preservation while maintain the quality.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Training-Free	Description
MACE	✗	Combines a closed-form solution with LoRA-based fine-tuning to achieve effective erasure while maintaining generation quality on unrelated concepts.
SPM	✗	Trains one-dimensional adapters that can be activated or deactivated through the Facilitated Transport mechanism, aiming to retain generation quality on unrelated concepts.
ESD-x	✗	Fine-tunes the cross-attention layers to overwrite the target erasure concept.
ESD-u	✗	Fine-tunes the U-net to overwrite the target erasure concept.
UCE	✓	Solves a least-squares optimization problem to redirect cross-attention output from the target concept while preserving outputs for specified retain concepts.
RECE	✓	Repeatedly solves a least-squares optimization problem from an adversarial perspective to minimize the probability of generating the target concept.

Appendix

Table 4: Overview of the selected baseline methods.

Dataset	$\boldsymbol{\beta}$	$\boldsymbol{\gamma}$	Acc t	Acc r	H o
Stanford Dogs	1.0	0.8	24.80	91.90	82.72
	1.0	1.0	26.80	91.90	82.93
	1.0	1.2	21.60	91.90	84.62
Oxford Flowers	1.0	0.8	16.00	91.83	87.74
	1.0	1.0	24.40	91.83	81.49
	1.0	1.2	19.20	91.83	85.96

Appendix

Table 5: Effect of varying

(\beta,\gamma)

parameters on target accuracy (Acc t ) and retain accuracy (Acc r ) for the Oxford Flowers and Stanford Dogs datasets. Reported values correspond to means computed over ten flower and dog classes, respectively.

$\boldsymbol{\beta}$	$\boldsymbol{\gamma}$	Mean Acc t	Mean Acc r	H o
1.0	0.8	0.67	89.55	94.19
1.0	0.9	0.44	94.67	97.05
1.0	1.0	2.45	94.44	95.97
1.0	1.2	1.56	85.56	91.55

Appendix

Table 6: Effect of varying

(\gamma)

parameter on target accuracy (Acc t ), retain accuracy (Acc r ), and harmonic mean

H_{o}=\text{HM}(100-\text{Acc}_{t},\;\text{Acc}_{r})

for the Celebrity dataset. Values are means over three celebrity identities.

Parameters ( $\beta,\gamma,\delta_{\text{token}}$ )	Armpits	Belly	Buttocks	Feet	Breasts (F)	Genitalia (F)	Breasts (M)	Genitalia (M)	Anus	Total	Clip Score
$1.0,0,12$	17	27	2	3	26	2	4	0	0	81	29.40
$1.0,0,14$	17	30	1	4	31	3	1	0	0	87	29.58
$1.0,0,15$	18	29	2	5	33	1	4	0	0	92	29.64
$1.0,0.5,12$	13	23	2	3	22	1	3	0	0	67	29.48
$1.0,0.5,14$	13	20	4	2	29	1	1	0	0	70	29.58
$1.0,0.5,15$	17	25	4	4	26	1	2	0	0	79	29.67

Appendix

Table 7: Effect of varying

(\gamma,\delta_{\text{token}})

parameters on explicit-content suppression for a 400-image high-risk subset of the I2P dataset. For each configuration, we report the number of detected body-part categories (lower is better). To evaluate semantic fidelity on non-sensitive data, CLIP Score is measured on a 3k COCO-Captions prompt set.

Figure 10: Overall pipeline for calculating $LPIPS\textsubscript{u}$ on non-target regions. First, a face detector identifies and localizes the target face. Then, using SAM, we generate a mask to isolate the target region, enabling $LPIPS\textsubscript{u}$ to be computed only on the remaining non-target areas.

Dataset	Target concept	# neighbor list	Neighbor list
Oxford Flowers	Alpine Sea Holly	10	Metasequoia, Flor de la Mar, Witch-hazel, Sonchus, Honeysuckle, Flora, Astilbe, Catalpa, Mount Sunflower, Alder
	Camellia	10	Gardenia, Peony, Roselle (plant), Gardenia jasminoides, Azalea, Flora, Floribunda (rose), Cattleya, Alstroemeria aurea, Plumeria
Stanford Dogs	Chesapeake Bay Retriever	10	Boykin Spaniel, Irish Terrier, Harrier (dog breed), Dogrel, Leaf Hound, Cocker Spaniel, Labrador, Field Spaniel, Redbone Coonhound, Hound
	Bluetick	10	Scent hound, Plott Hound, American Foxhound, Hound, Dogrel, Brittany Spaniel, Basset Hound, Huckleberry Hound, Beagle, Leaf Hound
Celebrity	Elon Musk	10	Jeff Bezos, Mark Bezos, Mark Zuckerberg, Tim Cook, Bill Gates, Steve Jobs, Satya Nadella, Richard Branson, Miguel Bezos, Warren Buffett
	Anna Kendrick	10	Katheryn Winnick, Jenny Lewis, Nicki Clyne, Emma Kenney, Nicole Parker, Danielle Lloyd, Emily Kinney, Hannah Marks, Kyla Kenedy, Vicky McClure

Appendix

Table 8: Neighbor lists retrieved by our method for each target concept across the Celebrity, Stanford Dogs, and Oxford Flowers datasets. For every concept, we report the top–10 nearest neighbors identified in the embedding space, illustrating the semantic relationships captured by our retrieval approach.

Method	Data Preparation + Model Finetune	Image Generation (per sample)	Total Time
MACE	680	3.8	718
SPM	6900	5.1	6951
ESD-x	540	2.2	562
ESD-u	540	2.2	562
UCE	0	2.3	23
RECE	1440	2.4	1464

Appendix

Table 9: Time Consumption of 10-concept erasure. We calculate the time cost (in seconds) to erase a concept and generate 10 images using one NVDIA A100 GPU.

Method	Remove “Van Gogh”				Remove “Kelly McKernan”
Method	LPIPS t (↑)	LPIPS r (↓)	$Acc_{t}$ (↓)	$Acc_{r}$ (↑)	LPIPS t (↑)	LPIPS r (↓)	$Acc_{t}$ (↓)	$Acc_{r}$ (↑)
SD v1.4	-	-	0.95	0.95	-	-	0.80	0.83
MACE	0.25	0.10	0.80	0.97	0.39	0.10	0.74	0.75
SPM	0.37	0.25	0.75	0.88	0.32	0.25	0.80	0.85
ESD-x	0.40	0.26	0.75	0.98	0.37	0.21	0.81	0.69
ESD-u	0.35	0.24	1.0	0.98	0.30	0.27	1.0	0.72

Appendix

Table 10: Quantitative Comparison of Artistic Style Erasure: LPIPS and QA metrics for two target artists.

Panel of Figure 11 — Figure 11: Visualization of binary masks $G_{t}(x,y)$ at different time steps using DownBlock-2 attention heatmaps.

Panel of Figure 12 — Figure 12: Concept erasure in a fine-grained setting. The top row shows original SD1.4 generations for Alpine Sea Holly , the middle row shows generations after erasure, and the bottom row shows SD1.4 generations conditioned on the classifier-predicted neighboring class.

Method	SD-1.4	SLD	ESD-u	UCE	RECE	Ours
Ring-A-Bell $\downarrow$	83.10	66.20	69.72	33.10	13.38	29.6

Appendix

Table 11: Attack success rate under the Ring-A-Bell attack (lower is better).

Figure 13: Qualitative Example of Ineffective Erasure. Due to textual ambiguity, NLCE has difficulty identifying an appropriate neighborhood set, resulting in ineffective erasure of the target concept

Figure 14: Further Qualitative Comparisons on the Stanford Dogs dataset. Our NLCE can effectively remove the target ’ Chesapeake Bay Retriever ’ while preserving other dog breeds like ’ Golden Retriever ’ and ’ African Hunting Dog ’.

Figure 15: Further Qualitative Comparisons on the Oxford Flowers dataset. Our NLCE can effectively remove the target ’ Camellia ’ while preserving other flower types like ’ Hibiscus ’ and ’ Clematis ’.

Figure 16: Further Qualitative Comparisons on the Celebrity dataset. Our NLCE can effectively remove the target ’ Anna Kendrick ’ while preserving other celebrity ’ Benicio Del Toro ’.

Figure 17: Further Qualitative examples on I2P dataset for explicit content erasure.

Figure 18: Further Qualitative Comparisons on the Artistic Dataset. Our NLCE can effectively remove the target style ’ Kelly Mckernan ’ while preserving style like ’ Kilian Eng ’ and ’ Tyler Edlin ’.

Concept Erasure

Concept erasure removes specified knowledge or attributes from models while preserving their ability to perform unrelated tasks.

See all 58 papers

Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models
Jul 6, 2026Parth Upman, Nishita Jain, Shreyank N GowdaAttention Control in Diffusion Models Diffusion-Based Image Editing
School of Computer Science, University of Nottingham, Nottingham, NG8 1BB, UK · Department of Computing, Imperial College London, London, SW7 2AZ, UK
Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models
Apr 12, 2026Hoigi Seo, Byung Hyun Lee, Jaehyun Cho +2Safe T2I Generation Diffusion Model Unlearning
Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea
Localized Concept Erasure in Text-to-Image Diffusion Models via High-Level Representation Misdirection
Feb 23, 2026Uichan Lee, Jeonghyeon Kim, Sangheum HwangDiffusion Model Unlearning T2I Diffusion Models
Department of Data Science, Seoul National University of Science and Technology

Jul 6, 2026J/K move · Enter open · S save

Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models

Parth Upman, Nishita Jain, Shreyank N Gowda

School of Computer Science, University of Nottingham, Nottingham, NG8 1BB, UK · Department of Computing, Imperial College London, London, SW7 2AZ, UK

Attention Control in Diffusion Models Diffusion-Based Image Editing Concept Erasure T2I Diffusion Models

Open paper AskPDF arXiv

Training-free concept erasure is an attractive mechanism for controlling text-to-image diffusion models, but precise erasure often comes at the cost of damaging semantically related non-target concepts. Existing value-space methods remove the component of each cross-attention value along the target concept direction, implicitly treating target identity and shared visual structure as the same signal. We argue that this is the source of much of the collateral damage in prior preservation. We introduce CARE, a closed-form concept erasure operator that replaces the raw target direction with a kept-subspace-aware direction computed from a small bank of retained concept anchors. The resulting edit is applied directly in cross-attention value space, requires no model fine-tuning, and adds only a negligible offline computation. A single shrinkage parameter controls the erase-preserve trade-off. We further show that the operator admits a minimum-disturbance interpretation and, in its projection form, leaves the kept subspace invariant. Experiments under the standard concept-erasure protocol show that our method preserves non-target concepts more faithfully while maintaining competitive erasure across instance, style, and celebrity concepts. Code: https://github.com/parthupman/care

Abstract

Concept Erasure

Explore similar work

Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models

Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models

Localized Concept Erasure in Text-to-Image Diffusion Models via High-Level Representation Misdirection