TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models

Alireza Dehghanpour Farashah; Zhuan Shi; Negar Rostamzadeh; Golnoosh Farnadi

cs.CVAug 15, 2026

TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models

Authors: Alireza Dehghanpour Farashah, Zhuan Shi, Negar Rostamzadeh, Golnoosh Farnadi

Organizations: Mila – Quebec AI Institute · McGill University · Google Research

Abstract

Text-to-image diffusion models can be misused to generate harmful content through adversarial or paraphrased prompts that bypass built-in safety mechanisms. Existing concept erasure methods often suffer from limited robustness against adversarial prompts, degradation of benign generation quality, or reliance on inference-time interventions that introduce persistent computational overhead. To address these limitations, we formulate concept erasure as a domain alignment problem in the text representation space. We propose a lightweight Text Encoder Alignment framework (TEA) that fine-tunes only the text encoder while keeping the generative backbone fully frozen. Given concept--anchor prompt pairs, our method trains a discriminator to distinguish token-level representations of concept-containing prompts from those of safe anchor prompts, while updating the text encoder to make these representations indistinguishable. TEA introduces zero inference-time overhead and requires only a small number of fine-tuning steps, making it highly efficient to deploy at scale. Despite this efficiency, TEA achieves state-of-the-art erasure robustness against black-box and white-box adversarial attacks on Stable Diffusion v1.4, while preserving generation quality on benign prompts. Furthermore, TEA is model-agnostic and achieves the lowest attack success rate on Stable Diffusion v3.5, extending concept erasure to a Rectified Flow Transformer architecture with T5 conditioning where prior methods remain largely unexplored. Code is available at https://github.com/alirezafarashah/TEA.git

Figures & tables

Figure 1: Overview of TEA . The text encoder is fine-tuned adversarially to align concept prompt representations with those of anchor prompts, while a preservation loss keeps anchor embeddings close to the original encoder output.

Panel of Figure 5 — Figure 5: PAD curves across encoder layers for SD 1.4 . Solid lines (sensitive vs. anchor): our method progressively reduces separability. Dashed lines (sensitive vs. COCO): all models retain near-maximum PAD ( $\approx 2.0$ ). Results are mean $\pm$ std over 10 splits.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Adversarial loss and discriminator accuracy during training for all text encoders of SD-v3.5.

Aligned positions	Attack Success Rate (%) $\downarrow$				COCO-Caption
Aligned positions	MMA	Ring-A-Bell	P4D	Avg.	FID $\downarrow$	CLIP $\uparrow$
SD-v1.4	93.20	98.13	86.40	92.58	–	31.51
Concept spans only	28.30	0.00	6.99	11.76	42.89	30.88
All tokens ( TEA )	1.20	0.00	2.21	1.14	46.50	30.65

Appendix

Table 3: Effect of restricting alignment to concept token spans on SD-v1.4.

Panel of Figure 7 — Figure 7: Reduction in NudeNet detections on the I2P benchmark relative to the original SD-v1.4.

$\lambda_{\mathrm{pres}}$	$\lambda_{\mathrm{adv}}$	Attack Success Rate (%) $\downarrow$				COCO-Caption
$\lambda_{\mathrm{pres}}$	$\lambda_{\mathrm{adv}}$	MMA	Ring-A-Bell	P4D	Avg.	FID $\downarrow$	CLIP $\uparrow$
–	–	93.20	98.13	86.40	92.58	–	31.51
0.5	0.5	1.90	0.00	1.84	1.25	48.13	30.34
0.5	1.0	0.50	0.00	0.74	0.41	52.65	30.01
1.0	0.5	1.20	0.00	2.21	1.14	46.50	30.65
1.0	1.0	1.90	0.00	1.47	1.12	46.41	30.54

Appendix

Table 4: Ablation over

\lambda_{\mathrm{adv}}

and

\lambda_{\mathrm{pres}}

on SD-v1.4.

Method	Attack Success Rate (%) $\downarrow$
Method	MMA	Ring-A-Bell	P4D	Avg.
FLUX.1-dev	67.90	72.15	60.53	66.86
DES	13.00	65.82	17.54	32.12
Ours ( TEA )	12.20	48.10	14.04	24.78

Appendix

Table 5: Results of explicit concept erasure on FLUX.1-dev.

Figure 8: FLUX.1-dev generations counted as successful attacks by NudeNet after erasure with TEA .

Benchmark	Method	NudeNet (%) $\downarrow$	LlavaGuard (%) $\downarrow$
MMA	FLUX.1-dev	67.90	11.50
MMA	Ours ( TEA )	12.20	0.30
Ring-A-Bell	FLUX.1-dev	72.15	46.84
Ring-A-Bell	Ours ( TEA )	48.10	16.46
P4D	FLUX.1-dev	60.53	21.93
P4D	Ours ( TEA )	14.04	0.88

Appendix

Table 6: NudeNet and LlavaGuard evaluated on the same FLUX.1-dev generations.

Anchor set	Attack Success Rate (%) $\downarrow$				COCO-Caption
Anchor set	MMA	Ring-A-Bell	P4D	Avg.	FID $\downarrow$	CLIP $\uparrow$
SD-v1.4	93.20	98.13	86.40	92.58	–	31.51
Original anchors ( TEA )	1.20	0.00	2.21	1.14	46.50	30.65
Deletion anchors	4.00	0.93	2.57	2.50	45.36	30.65

Appendix

Table 7: Original anchors against pure deletion anchors on SD-v1.4.

Method	Training Time $\downarrow$	Inference Overhead $\downarrow$
SPM	$\sim$ 1.2 h	$\sim$ 1.05 $\times$
ESD	$\sim$ 0.7 h	1 $\times$
SLD	0	$\sim$ 1.5 $\times$
UCE	$\sim$ 2 s	$\sim$ 1.05 $\times$
SAFREE	0	$\sim$ 1.5 $\times$
GLoCE	$\sim$ 120 s	$\sim$ 1.2 $\times$

Appendix

Table 8: Computational cost comparison.

Figure 9: Additional qualitative examples of explicit content erasure on Stable Diffusion 3.5 using Ring-A-Bell adversarial prompts.

Figure 10: Additional qualitative examples of explicit content erasure on Stable Diffusion 3.5 using MMA and P4D adversarial prompts.

Figure 11: Qualitative examples of explicit content erasure on Stable Diffusion 1.4.

Figure 12: Qualitative examples of artistic style erasure on Stable Diffusion 1.4.

Concept Prompt	Anchor Prompts
topless woman in garden	1. woman in floral summer dress in garden
	2. lady enjoying flowers in a long robe
	3. portrait of a woman picking flowers
	4. woman wearing a sunhat and scarf in garden
	5. casual outdoor photo of a woman in spring attire
female nude ballet dancer	1. ballet dancer in leotard stretching

Appendix

Table 9: Examples of concept–anchor prompt pairs from the training dataset. The full training set contains 50 concept prompts, each paired with five anchor prompts, for 250 pairs in total; ten concept prompts are shown here.

Safe T2I Generation

Safe text-to-image generation develops methods to prevent harmful images while preserving useful, faithful outputs.

See all 52 papers

Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models
Apr 12, 2026Hoigi Seo, Byung Hyun Lee, Jaehyun Cho +2Safe T2I Generation Diffusion Model Unlearning
Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea
PEAK: Precise and Persistent Concept Erasure via k-Sparse Autoencoders
Aug 11, 2026Man Jiang, Ouxiang Li, Weibao Xue +4Diffusion Model Unlearning T2I Diffusion Models
Hefei University of Technology · University of Science and Technology of China · University of Macau
Continual Concept Erasure in Diffusion Models by Suppressing Cross-Edit Interference
Oct 1, 2026Yongliang Wu, Haori Lu, Jinqi Luo +3Diffusion Model Unlearning Image Editing
University of Illinois Urbana-Champaign · University of Pennsylvania · National University of Singapore

Apr 12, 2026J/K move · Enter open · S save

Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models

Hoigi Seo, Byung Hyun Lee, Jaehyun Cho, Sungjin Lim+1

Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea

Safe T2I Generation Diffusion Model Unlearning T2I Diffusion Models

Open paper AskPDF arXiv

Large-scale text-to-image (T2I) diffusion models deliver remarkable visual fidelity but pose safety risks due to their capacity to reproduce undesirable content, such as copyrighted ones. Concept erasure has emerged as a mitigation strategy, yet existing approaches struggle to balance scalability, precision, and robustness, which restricts their applicability to erasing only a few hundred concepts. To address these limitations, we present Erasing Thousands of Concepts (ETC), a scalable framework capable of erasing thousands of concepts while preserving generation quality. Our method first models low-rank concept distributions via a Student's t-distribution Mixture Model (tMM). It enables pin-point erasure of target concepts via affine optimal transport while preserving others by anchoring the boundaries of target concept distributions without pre-defined anchor concepts. We then train a Mixture-of-Experts (MoE)-based module, termed MoEraser, which removes target embeddings while preserving the anchor embeddings. By injecting noise into the text embedding projector and fine-tuning MoEraser for recovery, our framework achieves robustness to white-box attack such as module removal. Extensive experiments on over 2,000 concepts across heterogeneous domains and diffusion models demerate state-of-the-art scalability and precision in large-scale concept erasure.

Abstract

Safe T2I Generation

Explore similar work

Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models

PEAK: Precise and Persistent Concept Erasure via k-Sparse Autoencoders

Continual Concept Erasure in Diffusion Models by Suppressing Cross-Edit Interference