Concept Acupuncturist: An Efficient and Precise Concept Erasure Method for Diffusion Models via Sparse Autoencoders

Yiran Xiahou; Siliang Qin; Kai Chen; Shengzhi Zhang; Renyang Liu

cs.CVOct 23, 2026

Concept Acupuncturist: An Efficient and Precise Concept Erasure Method for Diffusion Models via Sparse Autoencoders

Authors: Yiran Xiahou, Siliang Qin, Kai Chen, Shengzhi Zhang, Renyang Liu

Organizations: Institute of Information Engineering, Chinese Academy of Sciences, China · Department of Computer Science, Metropolitan College, Boston University, USA · Institute of Data Science, NationalUniversity of SingaporeSingapore, Singapore

Abstract

Diffusion-based image generation models produce high-quality images but may generate harmful or policy-violating content, motivating machine unlearning (MU) to selectively remove target concepts while preserving general capabilities. However, existing MU approaches often struggle with the distributed and entangled nature of concept representations, leading to inefficient optimization and imprecise edits when applied through large-scale fine-tuning over full diffusion trajectories. To bridge this gap, we propose Concept Acupuncturist (ConceptACP), a novel MU framework that integrates interpretability-guided analysis with inference-free fine-tuning for efficient and precise concept removal. By establishing mappings between semantic concepts and cross-attention blocks in the U-Net denoiser and localizing the most concept-relevant block for targeted optimization, ConceptACP avoids full diffusion trajectory optimization, thereby improving fine-tuning efficiency. To mitigate concept entanglement, we propose a feature-level correction mechanism based on Sparse Autoencoders (SAEs), where SAE-derived sparse, semantically meaningful feature activations are manipulated to construct targeted correction signals, which are then projected back to the output space of the corresponding block and used as supervision for precise concept removal. Consequently, ConceptACP eliminates the need for external inference-time modules while achieving efficient and fine-grained unlearning. Instantiated on SDXL-Turbo, ConceptACP improves the forgetting success rate by 36.34 percentage points in the standard one-step generation setting on average over the best-performing baseline, while preserving image quality and prompt fidelity, and generalizes effectively across other Stable Diffusion backbones. Our code is available at https://github.com/Eran-Xia/ConceptACP.

Figures & tables

Fig. 1: Visualization of the top-6 activated SAE features across different cross-attention blocks. The black rectangle visually masks the region containing nudity.

Method	SAEUron	TraSCE	SAFREE
Inference Overhead	+73.3%	+125.8%	+14.4%

TABLE I: Inference overhead comparison of inference-time concept forgetting methods on SDXL-Turbo.

Fig. 2: Discrepancy between classifier-based evaluation and human perception after concept unlearning.

Fig. 3: Overview of ConceptACP . Given a target concept, we first automatically identify relevant cross-attention blocks and associated features, then such features are selectively modified and mapped to the corresponding cross-attention outputs to guide fine-tuning.

Fig. 4: Effect of suppressing key features on generated images. Suppressing #4913 removes nudity, while no single feature fully removes the Church concept.

Fig. 5: Effects of suppressing feature #4913 (nudity) and enhancing feature #4998 (cartoon style) on generated images.

Method	Metric	T1 (one-step generation)					T4 (four-step generation)
Method		Nudity	VanGogh	Church	Parachute	Avg.	Nudity	VanGogh	Church	Parachute	Avg.
	FSR	7.44	57.68	1.60	1.60	17.80	11.60	7.68	0.80	2.00	5.52
SDXL-Turbo	CLIP Acc.	96.40	89.28	95.20	94.00	93.72	95.36	96.56	95.20	93.20	95.08
	FID	-	-	-	-	-	-	-	-	-	-
	FSR	7.44	58.96	0.80	1.60	17.20	25.12	1.12	3.60	82.80	28.16
ESD	CLIP Acc.	96.40	88.72	95.20	94.00	93.58	96.96	96.96	87.20	67.60	87.18

TABLE II: Comparison of unlearning performance across four concepts under T1 and T4. FSR (%,

\uparrow

), CLIP Acc. (%,

\downarrow

) and FID (

\downarrow

) are reported. The best FSR and CLIP Acc. are in bold , and the second-best ones are underlined .

Panel of Figure 8 — Fig. 6: Qualitative comparison of forgetting performance across four concepts on SDXL-Turbo. T1 denotes generation in one timestep, and T4 denotes generation in four timesteps.

(a) SDXL
Method	Metric	Nudity	VanGogh	Church	Parachute	Avg
ESD	FSR	61.60	92.80	2.00	10.00	41.60
	FID	63.48	164.06	46.80	49.80	81.04
UCE	FSR	51.60	92.40	2.00	34.00	45.00
	FID	138.88	63.07	90.26	94.70	96.73
Concept Prune	FSR	100.00	93.60	42.00	76.00	77.90

TABLE III: Unlearning performance on different diffusion backbones. FSR (%,

\uparrow

) and FID (

\downarrow

) are reported.

Method	Score	Concept
Method	Score	Nudity	VanGogh	Church	Parachute	Avg.
ESD	FO	0.68	0.68	0.75	0.93	0.76
ESD	FAC	0.60	0.73	0.60	0.97	0.73
UCE	FO	0.60	0.60	0.53	0.55	0.57
UCE	FAC	0.51	0.56	0.51	0.55	0.53
ConceptPrune	FO	0.58	0.60	0.70	0.43	0.58

TABLE IV: Comparison of unlearning precision. FO Score (

\uparrow

) and FAC Score (

\uparrow

) are reported, with

K=20

for FO evaluation.

Method	I2P	MMA	UnlearnDiffAtk	Avg.
SDXL-Turbo	13.32	42.00	36.61	30.64
ESD	12.89	38.20	30.83	27.31
UCE	9.45	36.70	30.44	25.53
ConceptPrune	0.75	6.30	1.35	2.80
SAEUron	4.62	11.50	18.11	11.41
ConceptACP	3.33	16.60	5.97	8.63

TABLE V: Comparison of robustness on Nudity evaluated by ASR (%,

\downarrow

) with 4 denoising steps.

Method	Nudity	VanGogh	Church	Parachute	Avg.
SDXL-Turbo	36.61	14.14	44.32	29.17	31.06
ESD	30.83	26.34	45.45	22.50	31.28
UCE	30.44	7.87	27.27	5.83	17.85
ConceptPrune	1.35	3.53	15.91	12.50	8.32
SAEUron	18.11	12.54	35.23	34.17	25.01
ConceptACP	5.97	7.98	15.91	21.67	12.88

TABLE VI: Robustness on four unlearning tasks under UnlearnDiffAtk evaluated by ASR

\downarrow

(%) with 4 denoising steps. Bold = best (minimum); underline = second best.

Panel of Figure 13 — Fig. 7: Generated images from different unlearning methods on I2P, MMA and UnlearnDiffAtk prompts.

Scenario	Concept	FSR (%) $\uparrow$	FID $\downarrow$
Different Blocks	Nudity (single)	98.00	89.80
	Van Gogh (single)	92.40	77.34
	Nudity (multi)	98.40(+0.40)	106.08(+16.28)
	Van Gogh (multi)	92.80(+0.40)	86.19(+8.85)
Same Block	Church (single)	80.00	47.12
	Parachute (single)	40.00	50.56

TABLE VII: Multi-concept unlearning results.

Panel of Figure 15 — Fig. 8: Effect of $\lambda$ on FSR and FID for four unlearning tasks.

Setting	Total Time	Avg. / Epoch	Dataset
Latent I/O	670.3s	6.7s	$\sim$ 72MB
Prompt $\rightarrow$ Latent	1,249.7s	12.5s	$\sim$ 29MB

TABLE VIII: Resource usage of fine-tuning. Times exclude data loading and I/O. Latent I/O denotes

\langle D_{\text{in}}[l],D_{\text{out}}[l]\rangle

pairs, while Prompt

\rightarrow

Latent denotes prompt-to-latent supervision.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

S(c,\mathcal{D},i,f)

Appendix

Algorithm 1 Feature Localization for Concept

c

Fig. 10: The unlearning effects of our one-step fine-tuned model when applied to single-step to 30-step generation.

Method	Mod. Param.	Infer. Time( $\Delta$ )
SDXL-Turbo	NA	0.6848 s
ESD	545.67 MB (21.25%)	0.6840 s (-0.12%)
UCE	340.79 MB (13.27%)	0.6864 s (+0.23%)
ConceptPrune	6.41 MB (0.25%)	0.6802 s (-0.68%)
SAEUron	13.12 MB (2.04%)	1.1872 s (+73.3%)
ConceptACP	350.15 MB (13.64%)	0.6768 s (-1.17%)

Appendix

TABLE IX: Efficiency comparison for single-concept unlearning on SDXL-Turbo. Mod. Param. reports the number of modified U-Net parameters, with the percentage indicating the proportion of total U-Net parameters. Infer. Time measures runtime for generating one image using 4 denoising steps, with relative change with respect to the original model.

Fig. 11: Forgetting effect across four unlearning tasks under different $\lambda$ values with a=-5. Generation step: 4.

Panel of Figure 21 — Fig. 12: Forgetting effect across four unlearning tasks under different $a$ values. Rows correspond to different categories, while columns correspond to $a$ values. Generation steps: 4.

Fig. 13: Results of single- and multi-concept unlearning on SDXL-Turbo. Generation steps: 4.

Model	Concept	Location	Feature IDs
SDXL -Turbo	Nudity	down.2.1	#4913 , #4001
	VanGogh	up.0.1	#3166 , #3950
	Church	down.2.1	#3799 , #2495 , #2005 , #2831 , #3551 , #4256 , #4008 , #1292
	Parachute	down.2.1	#642 , #3260 , #4525 , #3453 , #4241 , #3851 , #4966 , #4882 , #4764
SDXL	Nudity	down.2.1	#4913 , #4001
	VanGogh	up.0.1	#4950 , #3166 , #2008 , #4197 , #1566

Appendix

TABLE X: The related features for the four main concepts identified by ConceptACP on SDXL-Turbo and SDXL.

Panel of Figure 24 — Fig. 14: Comparison of forgetting performance across four unlearning tasks under two generation steps.

Diffusion Model Unlearning

Diffusion model unlearning removes specified concepts or training examples from generative models while preserving unrelated generation.

See all 77 papers

No Concept Escapes the Audit: Auditing-Aware Unlearning for Verifiable Concept Erasure in Diffusion Models
Oct 4, 2026Kaiyuan Deng, Yuchen Li, Gen Li +5Diffusion Model Unlearning T2I Diffusion Models
The University of Arizona · Peking University · Clemson University +2
Disentangled Sparse Representations for Concept-Separated Diffusion Unlearning
May 12, 2026Hyeonjin Kim, Hangyeol Jung, Heechan Yun +2Disentangled Representation Learning Sparse Autoencoders
Yonsei University · KAIST
Compensation-free Machine Unlearning in Text-to-Image Diffusion Models by Eliminating the Mutual Information
Mar 1, 2026Xinwen Cheng, Jingyuan Zhang, Zhehao Huang +2Diffusion Model Unlearning Concept Erasure
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University

Oct 4, 2026J/K move · Enter open · S save

No Concept Escapes the Audit: Auditing-Aware Unlearning for Verifiable Concept Erasure in Diffusion Models

Kaiyuan Deng, Yuchen Li, Gen Li, Yang Xiao+4

The University of Arizona · Peking University · Clemson University+2

Diffusion Model Unlearning T2I Diffusion Models Diffusion Models

Open paper AskPDF arXiv

Text-to-image diffusion models can generate prohibited content, which motivates concept erasure through machine unlearning. Most erasure methods intervene at the text interface, through prompt modification or localized updates to text-conditioning weights, and they are evaluated by what the model outputs for given prompts. Such evaluation cannot see what the network still encodes. Latent-space auditing, which bypasses text conditioning and probes the denoising network directly, shows that erased concepts remain recoverable from internal representations. We find that this also holds for methods built to be robust against adversarial prompts, and that the problem grows with the number of erased concepts. We propose Auditing-Aware Unlearning for Verifiable Concept Erasure in Diffusion Models (AVCE), a framework that grounds erasure in the model's latent representations. AVCE audits the embedding neighborhood of each concept and condenses the discovered vulnerable directions into an anchor at the weakest geometric point. It edits cross-attention and self-attention projections in closed form at this anchor, then fine-tunes the two pathways with pathway-level auditing losses, using orthogonal gradient projection to consolidate multiple concepts. Experiments on SD v1.5, SDXL, and Flux 1.0 across object, explicit-content, and artistic-style unlearning show that AVCE reduces attack success rates by 5.07x and improves auditing scores by 3.84x over the strongest baseline, while preserving competitive generation quality.

Abstract

Diffusion Model Unlearning

Explore similar work

No Concept Escapes the Audit: Auditing-Aware Unlearning for Verifiable Concept Erasure in Diffusion Models

Disentangled Sparse Representations for Concept-Separated Diffusion Unlearning

Compensation-free Machine Unlearning in Text-to-Image Diffusion Models by Eliminating the Mutual Information