Concept Acupuncturist: An Efficient and Precise Concept Erasure Method for Diffusion Models via Sparse Autoencoders
Authors: Yiran Xiahou, Siliang Qin, Kai Chen, Shengzhi Zhang, Renyang Liu
Organizations: Institute of Information Engineering, Chinese Academy of Sciences, China · Department of Computer Science, Metropolitan College, Boston University, USA · Institute of Data Science, NationalUniversity of SingaporeSingapore, Singapore
Diffusion-based image generation models produce high-quality images but may generate harmful or policy-violating content, motivating machine unlearning (MU) to selectively remove target concepts while preserving general capabilities. However, existing MU approaches often struggle with the distributed and entangled nature of concept representations, leading to inefficient optimization and imprecise edits when applied through large-scale fine-tuning over full diffusion trajectories. To bridge this gap, we propose Concept Acupuncturist (ConceptACP), a novel MU framework that integrates interpretability-guided analysis with inference-free fine-tuning for efficient and precise concept removal. By establishing mappings between semantic concepts and cross-attention blocks in the U-Net denoiser and localizing the most concept-relevant block for targeted optimization, ConceptACP avoids full diffusion trajectory optimization, thereby improving fine-tuning efficiency. To mitigate concept entanglement, we propose a feature-level correction mechanism based on Sparse Autoencoders (SAEs), where SAE-derived sparse, semantically meaningful feature activations are manipulated to construct targeted correction signals, which are then projected back to the output space of the corresponding block and used as supervision for precise concept removal. Consequently, ConceptACP eliminates the need for external inference-time modules while achieving efficient and fine-grained unlearning. Instantiated on SDXL-Turbo, ConceptACP improves the forgetting success rate by 36.34 percentage points in the standard one-step generation setting on average over the best-performing baseline, while preserving image quality and prompt fidelity, and generalizes effectively across other Stable Diffusion backbones. Our code is available at https://github.com/Eran-Xia/ConceptACP.
Figures & tables
Fig. 1: Visualization of the top-6 activated SAE features across different cross-attention blocks. The black rectangle visually masks the region containing nudity.
Method
SAEUron
TraSCE
SAFREE
Inference Overhead
+73.3%
+125.8%
+14.4%
TABLE I: Inference overhead comparison of inference-time concept forgetting methods on SDXL-Turbo.
Fig. 2: Discrepancy between classifier-based evaluation and human perception after concept unlearning.
Fig. 3: Overview of ConceptACP . Given a target concept, we first automatically identify relevant cross-attention blocks and associated features, then such features are selectively modified and mapped to the corresponding cross-attention outputs to guide fine-tuning.
Fig. 4: Effect of suppressing key features on generated images. Suppressing #4913 removes nudity, while no single feature fully removes the Church concept.
Fig. 5: Effects of suppressing feature #4913 (nudity) and enhancing feature #4998 (cartoon style) on generated images.
Method
Metric
T1 (one-step generation)
T4 (four-step generation)
Nudity
VanGogh
Church
Parachute
Avg.
Nudity
VanGogh
Church
Parachute
Avg.
FSR
7.44
57.68
1.60
1.60
17.80
11.60
7.68
0.80
2.00
5.52
SDXL-Turbo
CLIP Acc.
96.40
89.28
95.20
94.00
93.72
95.36
96.56
95.20
93.20
95.08
FID
-
-
-
-
-
-
-
-
-
-
FSR
7.44
58.96
0.80
1.60
17.20
25.12
1.12
3.60
82.80
28.16
ESD
CLIP Acc.
96.40
88.72
95.20
94.00
93.58
96.96
96.96
87.20
67.60
87.18
TABLE II: Comparison of unlearning performance across four concepts under T1 and T4. FSR (%, ↑ ), CLIP Acc. (%, ↓ ) and FID ( ↓ ) are reported. The best FSR and CLIP Acc. are in bold , and the second-best ones are underlined .
Fig. 6: Qualitative comparison of forgetting performance across four concepts on SDXL-Turbo. T1 denotes generation in one timestep, and T4 denotes generation in four timesteps.
(a) SDXL
Method
Metric
Nudity
VanGogh
Church
Parachute
Avg
ESD
FSR
61.60
92.80
2.00
10.00
41.60
FID
63.48
164.06
46.80
49.80
81.04
UCE
FSR
51.60
92.40
2.00
34.00
45.00
FID
138.88
63.07
90.26
94.70
96.73
Concept Prune
FSR
100.00
93.60
42.00
76.00
77.90
TABLE III: Unlearning performance on different diffusion backbones. FSR (%, ↑ ) and FID ( ↓ ) are reported.
Method
Score
Concept
Nudity
VanGogh
Church
Parachute
Avg.
ESD
FO
0.68
0.68
0.75
0.93
0.76
FAC
0.60
0.73
0.60
0.97
0.73
UCE
FO
0.60
0.60
0.53
0.55
0.57
FAC
0.51
0.56
0.51
0.55
0.53
ConceptPrune
FO
0.58
0.60
0.70
0.43
0.58
TABLE IV: Comparison of unlearning precision. FO Score ( ↑ ) and FAC Score ( ↑ ) are reported, with K=20 for FO evaluation.
Method
I2P
MMA
UnlearnDiffAtk
Avg.
SDXL-Turbo
13.32
42.00
36.61
30.64
ESD
12.89
38.20
30.83
27.31
UCE
9.45
36.70
30.44
25.53
ConceptPrune
0.75
6.30
1.35
2.80
SAEUron
4.62
11.50
18.11
11.41
ConceptACP
3.33
16.60
5.97
8.63
TABLE V: Comparison of robustness on Nudity evaluated by ASR (%, ↓ ) with 4 denoising steps.
Method
Nudity
VanGogh
Church
Parachute
Avg.
SDXL-Turbo
36.61
14.14
44.32
29.17
31.06
ESD
30.83
26.34
45.45
22.50
31.28
UCE
30.44
7.87
27.27
5.83
17.85
ConceptPrune
1.35
3.53
15.91
12.50
8.32
SAEUron
18.11
12.54
35.23
34.17
25.01
ConceptACP
5.97
7.98
15.91
21.67
12.88
TABLE VI: Robustness on four unlearning tasks under UnlearnDiffAtk evaluated by ASR ↓ (%) with 4 denoising steps. Bold = best (minimum); underline = second best.
Fig. 7: Generated images from different unlearning methods on I2P, MMA and UnlearnDiffAtk prompts.
Scenario
Concept
FSR (%) ↑
FID ↓
Different Blocks
Nudity (single)
98.00
89.80
Van Gogh (single)
92.40
77.34
Nudity (multi)
98.40(+0.40)
106.08(+16.28)
Van Gogh (multi)
92.80(+0.40)
86.19(+8.85)
Same Block
Church (single)
80.00
47.12
Parachute (single)
40.00
50.56
TABLE VII: Multi-concept unlearning results.
Fig. 8: Effect of λ on FSR and FID for four unlearning tasks.
Setting
Total Time
Avg. / Epoch
Dataset
Latent I/O
670.3s
6.7s
∼ 72MB
Prompt → Latent
1,249.7s
12.5s
∼ 29MB
TABLE VIII: Resource usage of fine-tuning. Times exclude data loading and I/O. Latent I/O denotes ⟨Din[l],Dout[l]⟩ pairs, while Prompt → Latent denotes prompt-to-latent supervision.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
S(c,D,i,f)
Appendix
Algorithm 1 Feature Localization for Concept c
Fig. 10: The unlearning effects of our one-step fine-tuned model when applied to single-step to 30-step generation.
Method
Mod. Param.
Infer. Time( Δ )
SDXL-Turbo
NA
0.6848 s
ESD
545.67 MB (21.25%)
0.6840 s (-0.12%)
UCE
340.79 MB (13.27%)
0.6864 s (+0.23%)
ConceptPrune
6.41 MB (0.25%)
0.6802 s (-0.68%)
SAEUron
13.12 MB (2.04%)
1.1872 s (+73.3%)
ConceptACP
350.15 MB (13.64%)
0.6768 s (-1.17%)
Appendix
TABLE IX: Efficiency comparison for single-concept unlearning on SDXL-Turbo. Mod. Param. reports the number of modified U-Net parameters, with the percentage indicating the proportion of total U-Net parameters. Infer. Time measures runtime for generating one image using 4 denoising steps, with relative change with respect to the original model.
Fig. 11: Forgetting effect across four unlearning tasks under different λ values with a=-5. Generation step: 4.
Fig. 12: Forgetting effect across four unlearning tasks under different a values. Rows correspond to different categories, while columns correspond to a values. Generation steps: 4.
Fig. 13: Results of single- and multi-concept unlearning on SDXL-Turbo. Generation steps: 4.
Text-to-image diffusion models can generate prohibited content, which motivates concept erasure through machine unlearning. Most erasure methods intervene at the text interface, through prompt modification or localized updates to text-conditioning weights, and they are evaluated by what the model outputs for given prompts. Such evaluation cannot see what the network still encodes. Latent-space auditing, which bypasses text conditioning and probes the denoising network directly, shows that erased concepts remain recoverable from internal representations. We find that this also holds for methods built to be robust against adversarial prompts, and that the problem grows with the number of erased concepts. We propose Auditing-Aware Unlearning for Verifiable Concept Erasure in Diffusion Models (AVCE), a framework that grounds erasure in the model's latent representations. AVCE audits the embedding neighborhood of each concept and condenses the discovered vulnerable directions into an anchor at the weakest geometric point. It edits cross-attention and self-attention projections in closed form at this anchor, then fine-tunes the two pathways with pathway-level auditing losses, using orthogonal gradient projection to consolidate multiple concepts. Experiments on SD v1.5, SDXL, and Flux 1.0 across object, explicit-content, and artistic-style unlearning show that AVCE reduces attack success rates by 5.07x and improves auditing scores by 3.84x over the strongest baseline, while preserving competitive generation quality.
Kaiyuan Deng, Yuchen Li, Gen Li +5
The University of Arizona · Peking University · Clemson University +2
Unlearning specific concepts in text-to-image diffusion models has become increasingly important for preventing undesirable content generation. Among prior approaches, sparse autoencoder (SAE)-based methods have attracted attention due to their ability to suppress target concepts through lightweight manipulation of latent features, without modifying model parameters. However, SAEs trained primarily for sparse reconstruction do not explicitly enforce concept-wise separation, allowing semantically related concepts to share latent features. To address this, we propose SAEParate, which explicitly separates semantically overlapping concepts in the SAE latent space through concept-aware contrastive learning, supported by cross-attention-guided spatial aggregation that emphasizes concept-relevant regions. In addition, we enhance the encoder with a nonlinear transformation to increase its expressive capacity under this separation objective, enabling more concept-discriminative latent representations. Experiments on UnlearnCanvas demonstrate state-of-the-art performance, with particularly strong gains in joint style-object unlearning, a challenging setting where existing methods suffer from severe interference between target and non-target concepts.
The powerful generative capabilities of diffusion models have raised growing privacy and safety concerns regarding generating sensitive or undesired content. In response, machine unlearning (MU) -- commonly referred to as concept erasure (CE) in diffusion models -- has been introduced to remove specific knowledge from model parameters meanwhile preserving innocent knowledge. Despite recent advancements, existing unlearning methods often suffer from excessive and indiscriminate removal, which leads to substantial degradation in the quality of innocent generations. To preserve model utility, prior works rely on compensation, i.e., re-assimilating a subset of the remaining data or explicitly constraining the divergence from the pre-trained model on remaining concepts. However, we reveal that generations beyond the compensation scope still suffer, suggesting such post-remedial compensations are inherently insufficient for preserving the general utility of large-scale generative models. Therefore, in this paper, we advocate for developing compensation-free concept erasure operations, which precisely identify and eliminate the undesired knowledge such that the impact on other generations is minimal. In technique, we propose to MiM-MU, which is to unlearn a concept by minimizing the mutual information with a delicate design for computational effectiveness and for maintaining sampling distribution for other concepts. Extensive evaluations demonstrate that our proposed method achieves effective concept removal meanwhile maintaining high-quality generations for other concepts, and remarkably, without relying on any post-remedial compensation for the first time.
Xinwen Cheng, Jingyuan Zhang, Zhehao Huang +2
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University