EGLOCE: Training-Free Energy-Guided Latent Optimization for Concept Erasure

Junyeong Ahn; Seojin Yoon; Sungyong Baik

cs.CVApr 10, 2026

EGLOCE: Training-Free Energy-Guided Latent Optimization for Concept Erasure

Authors: Junyeong Ahn, Seojin Yoon, Sungyong Baik

Organizations: KAIST AI · Hanyang University

Abstract

As text-to-image diffusion models grow increasingly prevalent, the ability to remove specific concepts-mostly explicit content and many copyrighted characters or styles-has become essential for safety and compliance. Existing unlearning approaches often require costly re-training, modify parameters at the cost of degradation of unrelated concept fidelity, or depend on indirect inference-time adjustment that compromise the effectiveness of concept erasure. Inspired by the success of energy-guided sampling for preservation of the condition of diffusion models, we introduce Energy-Guided Latent Optimization for Concept Erasure (EGLOCE), a training-free approach that removes unwanted concepts by re-directing noisy latent during inference. Our method employs a dual-objective framework: a repulsion energy that steers generation away from target concepts via gradient descent in latent space, and a retention energy that preserves semantic alignment to the original prompt. Combined with previous approaches that either require erroneous modified model weights or provide weak inference-time guidance, EGLOCE operates entirely at inference and enhances erasure performance, enabling plug-and-play integration. Extensive experiments demonstrate that EGLOCE improves concept removal while maintaining image quality and prompt alignment across baselines, even with adversarial attacks. To the best of our knowledge, our work is the first to establish a new paradigm for safe and controllable image generation through dual energy-based guidance during sampling.

Figures & tables

Figure 2: Overview of the EGLOCE framework. The flowchart illustrates how repulsion and retention energies are applied during diffusion steps to erase undesired concepts while preserving others.

	Adversarial					COCO
Method	I2P ↓	P4D ↓	Ring-A-Bell ↓	MMA-Diffusion ↓	UnlearnDiffAtk ↓	FID ↓	CLIP ↑
ESD [ 12 ]	0.234	0.583	0.291	0.406	0.296	13.77	30.43
ESD + Ours	0.209	0.589	0.228	0.417	0.218	13.74	31.40
SLD [ 40 ]	0.396	0.920	0.557	0.912	0.570	18.77	30.79
SLD + Ours	0.374	0.900	0.519	0.892	0.570	17.40	31.56
RECE [ 14 ]	0.141	0.427	0.127	0.557	0.148	15.07	30.95

Table 1: Attack success rate (ASR), CLIP, and FID scores for nudity removal with each baseline and its variant augmented with our method (baseline + ours). For each baseline, the best score for each metric is highlighted in bold.

Figure 3: Qualitative results of the proposed method. $\left(\text{A}\right)$ When applied on top of the baselines, our method performs fine-grained refinements that remove residual nudity left in their outputs. $\left(\text{B}\right)$ It further erases the target object while largely preserving the surrounding content. $\left(\text{C}\right)$ Applying EGLOCE makes the generated images follow the input prompts more faithfully and improves their perceptual quality.

Figure 4: Qualitative results on nudity erasure across four baselines.

	Remove “Van Gogh”				Remove “Kelly McKernan”
Method	$\textbf{LPIPS}_{e}$ ↑	$\textbf{LPIPS}_{u}$ ↓	$\textbf{Acc}_{e}$ ↓	$\textbf{Acc}_{u}$ ↑	$\textbf{LPIPS}_{e}$ ↑	$\textbf{LPIPS}_{u}$ ↓	$\textbf{Acc}_{e}$ ↓	$\textbf{Acc}_{u}$ ↑
SD-v1.4	–	–	1.00	1.00	–	–	1.00	1.00
ESD	0.36	0.16	0.20	1.00	0.41	0.21	0.20	1.00
ESD + Ours	0.35	0.16	0.25	1.00	0.41	0.19	0.20	1.00
SLD	0.62	0.52	1.00	1.00	0.54	0.55	1.00	1.00
SLD + Ours	0.61	0.52	1.00	1.00	0.53	0.54	1.00	1.00

Table 2: LPIPS and classification accuracy for artist-style removal with each baseline and its variant augmented with our method (baseline+ours).

Figure 5: Failure cases show that, in some cases, integrating our method to the baseline brings little visible change over the original results, or does not fully remove the targeted content.

	Church		English Springer
Method	$\textbf{Acc}_{e}$ ↓	$\textbf{Acc}_{u}$ ↑	$\textbf{Acc}_{e}$ ↓	$\textbf{Acc}_{u}$ ↑
ESD	0.70	1.00	0.00	1.00
ESD + Ours	0.70	1.00	0.00	1.00
SLD	0.90	1.00	0.40	1.00
SLD + Ours	0.70	1.00	0.10	1.00
RECE	0.00	0.98	0.00	0.93

Table 3: Classification accuracy for object removal with each baseline and its variant augmented with our method.

		Adversarial		COCO
Method	K	P4D ↓	MMA ↓	FID ↓	CLIP ↑
SAFREE	–	0.400	0.529	20.68	30.70
SAFREE	3	0.360	0.496	19.85	31.38
+ Ours	5	0.300	0.464	19.56	31.62
	7	0.267	0.445	19.11	31.89

Table 4: Ablation study on the number of EGLOCE update iterations

K

. The table reports how the attack success rate (ASR), CLIP, and FID scores change as

K

varies.

		Adversarial		COCO
Method	R	P4D ↓	MMA ↓	FID ↓	CLIP ↑
SAFREE	N	0.327	0.473	20.06	30.37
+ Ours	Y	0.360	0.496	19.85	31.38

Table 5: Ablation study on the effect of the EGLOCE retention term. The column R (retain) indicates whether the retention component is used: models with R = Y include the retention term, while models with R = N do not.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	Interval		Adversarial					COCO
Method	$\textbf{t}_{\text{start}}$	$\textbf{t}_{\text{end}}$	I2P ↓	P4D ↓	Ring-A-Bell ↓	MMA ↓	UnlearnDiffAtk ↓	FID ↓	CLIP ↑
SAFREE	20	45	0.066	0.258	0.114	0.359	0.141	21.46	30.93
+ Ours	20	40	0.101	0.384	0.114	0.495	0.211	20.72	30.81
	20	35	0.106	0.397	0.165	0.529	0.211	20.68	30.70

Appendix

Table A : Attack success rate (ASR), CLIP, and FID scores for nudity removal using SAFREE augmented with our method (SAFREE + ours) under different

t_{\text{start}}

settings (all other hyperparameters fixed).

	Interval		Adversarial					COCO
Method	$\textbf{t}_{\text{start}}$	$\textbf{t}_{\text{end}}$	I2P ↓	P4D ↓	Ring-A-Bell ↓	MMA ↓	UnlearnDiffAtk ↓	FID ↓	CLIP ↑
SAFREE	20	35	0.106	0.397	0.165	0.529	0.211	20.68	30.70
+ Ours	10	35	0.105	0.358	0.165	0.529	0.211	20.50	31.68
	0	35	0.101	0.358	0.177	0.509	0.211	20.39	33.82

Appendix

Table B : Attack success rate (ASR), CLIP, and FID scores for nudity removal using SAFREE augmented with our method (SAFREE + ours) under different

t_{\text{end}}

settings (all other hyperparameters fixed).

Figure A : Qualitative effects under different $t_{\text{start}}$ settings.

Figure B : Qualitative effects under different $t_{\text{end}}$ settings.

	Scale		Adversarial					COCO
Method	$\lambda_{\text{rep}}$	$\lambda_{\text{ret}}$	I2P ↓	P4D ↓	Ring-A-Bell ↓	MMA ↓	UnlearnDiffAtk ↓	FID ↓	CLIP ↑
SAFREE	6	3	0.118	0.417	0.228	0.537	0.225	20.78	30.42
+ Ours	10	5	0.106	0.397	0.165	0.529	0.211	20.68	30.70
	14	7	0.101	0.397	0.127	0.492	0.197	20.52	30.88
	18	9	0.086	0.338	0.114	0.467	0.197	20.41	31.04

Appendix

Table C : Attack success rate (ASR), CLIP, and FID scores for nudity removal using SAFREE augmented with our method (SAFREE + ours) under different scale settings (all other hyperparameters fixed).

Figure C : Qualitative effects under different scale settings.

Figure D : Qualitative results of nudity removal with ESD + ours. Example outputs illustrating how augmenting ESD with our method leads to more effective removal of nude content.

Figure E : Qualitative results of nudity removal with SLD + ours. Example outputs illustrating how augmenting SLD with our method leads to more effective removal of nude content.

Figure F : Qualitative results of nudity removal with RECE + ours. Example outputs illustrating how augmenting RECE with our method leads to more effective removal of nude content.

Figure G : Qualitative results of nudity removal with SAFREE + ours. Example outputs illustrating how augmenting SAFREE with our method leads to more effective removal of nude content.

Figure H : Qualitative results of object removal with SLD + ours. Example outputs illustrating how augmenting SLD with our method leads to more effective removal of church or English springer content.

Figure I : Qualitative results of object removal with SAFREE + ours. Example outputs illustrating how augmenting SAFREE with our method leads to more effective removal of church or English springer content.

Figure J : Qualitative improvements across baselines. Example generations from three models and their variants augmented with our method, demonstrating improved image quality and better adherence to the input prompts.

Figure K : Failure cases of concept removal with baseline + ours. Example outputs where adding our method to the baseline brings little visible change over the original results, or does not fully remove the targeted content.

Diffusion Model Unlearning

Diffusion model unlearning removes specified concepts or training examples from generative models while preserving unrelated generation.

See all 77 papers

Prototype-Guided Concept Erasure in Diffusion Models
Mar 9, 2026Yuze Cai, Jiahao Lu, Hongxiang Shi +2Concept Erasure Safe T2I Generation
Fudan University · National University of Singapore
TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
Aug 15, 2026Alireza Dehghanpour Farashah, Zhuan Shi, Negar Rostamzadeh +1Safe T2I Generation Adversarial Robustness
Mila – Quebec AI Institute · McGill University · Google Research
Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models
Apr 12, 2026Hoigi Seo, Byung Hyun Lee, Jaehyun Cho +2Safe T2I Generation Diffusion Model Unlearning
Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea

Mar 9, 2026J/K move · Enter open · S save

Prototype-Guided Concept Erasure in Diffusion Models

Yuze Cai, Jiahao Lu, Hongxiang Shi, Yichao Zhou+1

Fudan University · National University of Singapore

Concept Erasure Safe T2I Generation T2I Diffusion Models Diffusion Model Unlearning Diffusion Model Guidance

Open paper AskPDF arXiv

Concept erasure is extensively utilized in image generation to prevent text-to-image models from generating undesired content. Existing methods can effectively erase narrow concepts that are specific and concrete, such as distinct intellectual properties (e.g. Pikachu) or recognizable characters (e.g. Elon Musk). However, their performance degrades on broad concepts such as sexual'' or violent'', whose wide scope and multi-faceted nature make them difficult to erase reliably. To overcome this limitation, we exploit the model's intrinsic embedding geometry to identify latent embeddings that encode a given concept. By clustering these embeddings, we derive a set of concept prototypes that summarize the model's internal representations of the concept, and employ them as negative conditioning signals during inference to achieve precise and reliable erasure. Extensive experiments across multiple benchmarks show that our approach achieves substantially more reliable removal of broad concepts while preserving overall image quality, marking a step towards safer and more controllable image generation.

Abstract

Diffusion Model Unlearning

Explore similar work

Prototype-Guided Concept Erasure in Diffusion Models

TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models

Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models