Push-Pull Attentional Anchoring for Diffusion Concept Erasure

Nattanat Chatthee; Tagon Sompong; Ekapol Chuangsuwanich; Supasorn Suwajanakorn

cs.CVOct 20, 2026

Push-Pull Attentional Anchoring for Diffusion Concept Erasure

Authors: Nattanat Chatthee, Tagon Sompong, Ekapol Chuangsuwanich, Supasorn Suwajanakorn

Organizations: VISTEC, Thailand · Chulalongkorn University, Thailand

Abstract

Rapid advances in diffusion models have raised concerns about privacy, harmful content, and copyright violations. Concept erasure methods aim to address these risks by removing undesirable concepts from pretrained models without full retraining. However, existing approaches often degrade non-target generation quality through heuristic strength scaling (e.g., classifier-free guidance or negative prompt scaling), leading to uncontrolled and excessive semantic drift. To address this, we propose Push-Pull Attentional Anchoring (PPAA), a mechanism in the cross-attention space that displaces target representations while anchoring the scale of erasure via bounded cosine-similarity constraints. By using a relative similarity ratio instead of an absolute difference, our adaptive threshold represents a meaningful percentage of semantic retention, facilitating a single value to be applied uniformly across layers. We conduct extensive experiments across multiple concept categories, including celebrities, artistic styles, nudity, and intellectual property. Our method achieves favorable trade-offs against prior approaches on Stable Diffusion v1.4 and demonstrates its applicability to modern architectures such as FLUX.

Figures & tables

Figure 1: Given a textual concept, our method erases the target concept while preserving non-target concepts , achieving a favorable erasure–preservation trade-off with controllable erasure strength and applicability to modern architectures such as FLUX.

Figure 2: Erasure–Preservation score across concept categories. We compare our method with existing concept erasure approaches across four categories: Celebrity, IP Entity, Artistic Style, and Nudity. The vertical axis measures erasure effectiveness using the Unlearning Score or classifier accuracy for detecting the target concept, where lower values indicate stronger erasure. The horizontal axis measures preservation effectiveness using CMMD ( $\times 1000$ ), where lower values indicate better preservation of non-target concepts. Each point denotes a hyperparameter setting; baselines vary their erasure-strength parameter when available, whereas our method varies only $m_{\text{push}}$ . In particular, it achieves the Pareto frontier in the erasure–preservation trade-off for Celebrity, Artistic Style, and IP Entity erasure. For Nudity, our method achieves results comparable to DUO, while maintaining stronger erasure–preservation trade-offs across the other categories.

Figure 3: Qualitative comparison of erasure methods. Our method erases the target concepts (e.g., Rihanna identity and Picasso style) while preserving the non-target concepts (e.g., Margot Robbie identity and Claude Monet style). Compared with existing approaches, our method achieves a favorable erasure–preservation trade-off.

Method	Pre-ASR $\downarrow$	ASR $\downarrow$	Avg Time (mins) $\uparrow$
ESD	0.26	0.92	9.89
UCE	0.41	0.92	8.35
MACE	0.55	0.92	7.11
AC	0.20	0.88	9.98
DUO	0.02	0.47	13.45
SUMA	0.16	0.83	11.80

Table 1: Success rate of recovering the erased nudity concept via UnlearnDiff [ 40 ] . Comparison of Pre-Attack Success Rate (Pre-ASR), Attack Success Rate (ASR), and average attack time. Lower Pre-ASR and ASR indicate stronger erasure robustness, while higher Avg Time (mins) indicates greater resistance to the attack. Although our method is not explicitly optimized for robustness, it remains competitive among methods without robustness-focused objectives (MACE, UCE, AC, and ESD).

Method	ASR $\downarrow$	CMMD $\downarrow$
ESD	0.69	13.94
UCE	0.79	4.05
DUO	0.01	6.44
STEREO	0.04	139.47
Ours	0.20	1.43

Table 2: Robustness against CCE attacks [ 27 ] and preservation under nudity erasure. Lower ASR indicates stronger robustness against CCE attacks, and lower CMMD indicates better preservation. Our method achieves a favorable robustness–preservation trade-off. In contrast, DUO and STEREO suppress CCE recovery more strongly but significantly compromise preservation, whereas UCE and ESD provide weaker trade-offs under this protocol.

Figure 4: Concept erasure and preservation results on FLUX. (a) Qualitative comparison on representative identity, style, and character concepts. Our method erases target concepts, such as Barack Obama , Van Gogh , and Mickey Mouse , while preserving non-target concepts, such as Will Smith , Picasso , and Donald Duck . ESD causes substantial semantic shifts in non-target concepts, whereas UCE fails to semantically erase Mickey Mouse . (b) Quantitative results. Accuracy measures target-concept detection after erasure, and CMMD measures preservation relative to the original FLUX model; lower is better for both metrics. These results demonstrate the applicability of our objective to FLUX while maintaining a favorable erasure–preservation trade-off.

Figure 5: Controllable semantic deviation induced by semantic push. As $m_{\text{push}}$ decreases from 0.80 to 0.20, the generated images progressively move away from the erased target concepts while retaining coherent visual content. Examples include artistic style, celebrity identity, and IP entity erasure, illustrating the controllable semantic separation induced by $\mathcal{L}_{\text{push}}$ .

Figure 6: Ablation Study on Design Choices. Ours denotes the full method, which uses cosine distance in $L_{\text{push}}$ , $L_{\text{pull}}$ , and $L_{\text{preserve}}$ , together with the relative-margin formulation and preservation losses ( $L_{\text{preserve}}$ , $L_{\text{denoise}}$ ). Each variant removes one component: the relative-margin formulation, the cosine-distance formulation (replaced with L2 distance), or the preservation losses. Each point corresponds to a different push margin. The x-axis measures same-domain preservation (CMMD $\times 1000$ , lower is better) and the y-axis measures unlearning score (detection accuracy, lower is better). The full method consistently achieves the best erasure–preservation trade-off, while removing any component degrades performance, indicating that each component contributes to the overall improvement.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

	Generic / Related anchor				Random anchor
Setting	Avg.	Std.	Min.	Max.	Avg.	Std.	Min.	Max.
Main experiments (13 concepts)	0.659	0.166	0.247	0.942	0.390	0.186	0.050	0.834
50 celebrities	0.482	0.161	0.126	0.881	0.367	0.184	0.055	0.817
50 artistic styles	0.651	0.124	0.337	0.942	0.344	0.195	0.030	0.848
80 COCO classes	0.606	0.142	0.182	0.927	0.511	0.174	0.130	0.926
600 OpenImage classes	0.574	0.150	0.133	0.927	0.489	0.181	0.047	0.991

Appendix

Table 3: Statistics of the pull-loss denominator

\cossim(W_{0}p_{e},W_{0}p_{g})

on SD1.4 across evaluated concepts. The observed denominators remain positive for both semantically related generic anchors and random anchors sampled from the 600 OpenImage classes, supporting the numerical stability of the relative pull ratio.

Category

Same-Domain Preservation Concepts

Celebrity

Maya Angelou, Gillian Anderson, Jim Morrison, Jennifer Connelly, Benicio Del Toro, Avril Lavigne, Aaron Paul, Bill Murray, Kim Jong Un, Justin Bieber, David Bowie, Barry Manilow, Judy Garland, Betty White, Denise Richards, Gal Gadot, Pierce Brosnan, Julianne Moore, David Tennant, Jackie Chan, Natalie Portman, Rachel Dratch, Liv Tyler, Gordon Ramsay, Patrick Stewart, Doris Day, Matthew McConaughey, Amy Schumer, Hayley Atwell, Niall Horan, Neil deGrasse Tyson, Heath Ledger, Kristen Stewart, Amy Poehler, Kirsten Dunst, Matt Damon, Joan Rivers, Bill Nye, Britney Spears, Lizzy Caplan, Emma Roberts, Clint Eastwood, Rachel McAdams, Harry Dean Stanton, Krysten Ritter, Aretha Franklin, Kate Upton, George Takei, Christina Hendricks, Andy Samberg

Artistic Style

Abed Abdi, Albert Marquet, Alberto Vargas, Adrian Tomine, Albert Edelfelt, A. J. Casson, Abbott Fuller Graves, Alan Kenny, Alberto Sughi, Aaron Siskind, Adriaen van Outrecht, Albert Goodwin, Alberto Biasi, Albert Bloch, Adrian Donoghue, Akos Major, Alberto Burri, Ai Weiwei, Alastair Magnaldo, Albert Watson, Agnes Martin, Aleksey Savrasov, Adolph Gottlieb, Adrianus Eversen, Affandi, Alessandro Allori, Alec Soth, Abigail Larson, Alan Parry, Akira Toriyama, Adonna Khare, Alena Aenami, Agnes Cecile, Albert Lynch, Albrecht Dürer, Alejandro Jodorowsky, Alex Alemany, Alex Colville, Albert Eckhout, Abram Efimovich Arkhipov, Adolph Menzel, Aleksi Briclot, Agostino Tassi, Albert Bierstadt, Aaron Douglas, Albert Tucker, Abraham Pether, Alberto Magnelli, Adam Hughes, Alan Schaller

Appendix

Table 4: Same-domain preservation concepts used in evaluation. For Celebrity and Artistic Style Erasure, we randomly sample 50 celebrities and 50 artistic styles from the list provided by [ 24 ] . These concepts measure preservation performance within the same semantic domain as the erased concept.

Figure 7: Erasure–Preservation Trade-off Across Concept Categories. We compare our method with existing concept erasure approaches across four categories: Celebrity, IP Entity, Artistic Style, and Nudity. For Celebrity and Artistic, we provide both same-domain and generic-domain preservation scores. Our method consistently performs well across categories. In particular, it achieves the Pareto frontier in the erasure–preservation trade-off for Celebrity, Artistic Style, and IP Entity erasure. For Nudity, our method performs comparably to DUO. However, DUO performs poorly in the other categories.

Figure 8: Training-Free Trajectory Probing Results. We evaluate robustness to trajectory-based attacks using the prompt “a photo of a naked person” on nudity-erased models. Base shows images generated without applying the attack procedure. Attacked (Best) denotes the image most aligned with the attack concept according to the CLIP score among all generated samples during the probing trajectory. The remaining rows show randomly sampled images obtained from the attack method. Although our method is not explicitly trained with robustness-oriented objectives (e.g., DUO and STEREO), its behavior under this attack appears competitive to that of existing methods such as MACE, UCE, SUMA, and ESD.

Figure 9: Exploratory token selection for semantic push. We compare three token-selection variants for $\mathcal{L}_{\mathrm{push}}$ : applying it to both the concept-relevant token(s) and the EOS token, only to the EOS token, and only to the concept-relevant token(s). The concept-token-only variant localizes the semantic change more strongly to concept-specific information, indicating a promising refinement for reducing unintended changes to non-target attributes such as pose and background.

Figure 10: Concept erasure and preservation results on the Z-Image model. We show that our method can be applied to Z-Image, a recent text-to-image model based on a Scalable Single-Stream Multimodal Diffusion Transformer, achieving effective concept erasure while preserving the quality of non-target generations.

Figure 11: Additional Qualitative Comparison on Celebrity Erasure. Prompts corresponding to the target concepts are shown in red, while prompts corresponding to the non-target concepts are shown in blue and are randomly sampled. Images in the same row are generated using the same random seed across all methods for fair visual comparison.

Figure 12: Additional Qualitative Comparison on Celebrity Erasure. Prompts corresponding to the target concepts are shown in red, while prompts corresponding to the non-target concepts are shown in blue. Images in the same row are generated using the same random seed across all methods for fair visual comparison.

Figure 13: Additional Qualitative Comparison on Artistic Style Erasure. Prompts corresponding to the target concepts are shown in red, while prompts corresponding to the non-target concepts are shown in blue. Images in the same row are generated using the same random seed across all methods for fair visual comparison.

Figure 14: Additional Qualitative Comparison on Artistic Style Erasure. Prompts corresponding to the target concepts are shown in red, while prompts corresponding to the non-target concepts are shown in blue. Images in the same row are generated using the same random seed across all methods for fair visual comparison.

Figure 15: Additional Qualitative Comparison on Intellectual Property Entity Erasure. Prompts corresponding to the target concepts are shown in red, while prompts corresponding to the non-target concepts are shown in blue. Images in the same row are generated using the same random seed across all methods for fair visual comparison.

Figure 16: Additional Qualitative Comparison on Intellectual Property Entity Erasure. Prompts corresponding to the target concepts are shown in red, while prompts corresponding to the non-target concepts are shown in blue. Images in the same row are generated using the same random seed across all methods for fair visual comparison.

Figure 17: Additional Qualitative Comparison on Nudity Erasure. Prompts corresponding to the target concepts are shown in red, while prompts corresponding to the non-target concepts are shown in blue. Images in the same row are generated using the same random seed across all methods for fair visual comparison.

Concept Erasure

Concept erasure removes specified knowledge or attributes from models while preserving their ability to perform unrelated tasks.

See all 61 papers

Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models
May 11, 2026Nicola Novello, Andrea M. TonelloCross-Attention Concept Erasure
University of Klagenfurt Klagenfurt, Austria
Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models
Jul 6, 2026Parth Upman, Nishita Jain, Shreyank N GowdaAttention Control in Diffusion Models Diffusion-Based Image Editing
School of Computer Science, University of Nottingham, Nottingham, NG8 1BB, UK · Department of Computing, Imperial College London, London, SW7 2AZ, UK
Closed-Form Concept Erasure via Double Projections
Apr 11, 2026Chi Zhang, Jingpu Cheng, Zhixian Wang +1Concept Erasure Diffusion Model Unlearning
National University of Singapore · Shanghai Jiao Tong University · University of Nevada, Reno

May 11, 2026J/K move · Enter open · S save

Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models

Nicola Novello, Andrea M. Tonello

University of Klagenfurt Klagenfurt, Austria

Cross-Attention Concept Erasure T2I Diffusion Models

Open paper AskPDF arXiv

Erasing specific concepts from text-to-image diffusion models is essential for avoiding the generation of copyrighted and explicit content. Closed-form concept erasure methods offer a fast alternative to backpropagation-based techniques, but they become less effective when scaling from smaller models such as Stable Diffusion 1.5 to larger models like Stable Diffusion XL. To maintain erasure effectiveness in these larger-scale architectures, we propose SParse cross-Attention-based Concept Erasure (SPACE). SPACE iteratively modifies the cross-attention parameters of a model with a closed-form update that jointly induces sparsity and erases target concepts. By concentrating the concept mapping to a lower-dimensional subspace, SPACE achieves superior erasure efficacy compared to dense baselines. Extensive experimental results show improvements in erasure effectiveness and robustness against adversarial prompts. Furthermore, SPACE achieves 80%-90% cross-attention sparsity, reducing the storage requirements for saving the modified parameters by 70%, demonstrating its memory efficiency.

Abstract

Concept Erasure

Explore similar work

Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models

Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models

Closed-Form Concept Erasure via Double Projections