VisualErase: Dual-Branch Visual Trajectory Redirection for Robust Concept Erasure in Text-to-Image Diffusion Models

Qianlong Xiang; Miao Zhang; Kun Wang; Yupeng Hu; Junhui Hou; Liqiang Nie

cs.CVOct 4, 2026

VisualErase: Dual-Branch Visual Trajectory Redirection for Robust Concept Erasure in Text-to-Image Diffusion Models

Authors: Qianlong Xiang, Miao Zhang, Kun Wang, Yupeng Hu, Junhui Hou, Liqiang Nie

Organizations: Harbin Institute of Technology (Shenzhen) · City University of Hong Kong · Shenzhen Loop Area Institute · National University of Singapore · Shandong University

Abstract

Concept erasure is essential for the safe deployment of text-to-image diffusion models, as they may reproduce harmful, copyrighted, or privacy-sensitive content learned from unconstrained large-scale data. Existing methods typically erase unwanted concepts while preserving general generation capability by redirecting target-related text-to-image mappings. However, recent studies show that erased models may still retain visual generative trajectories of target concepts, leaving them vulnerable to adversarial recovery attacks and revealing a fundamental gap between redirecting text-to-image mappings and truly removing visual knowledge. To bridge this gap, we propose VisualErase, a new paradigm that redirects concept-bearing visual generative trajectories toward explicitly defined concept-removed outcomes. To enable this redirection, we use structure-preserving image editing to construct content-aligned, concept-removed counterparts for source images, providing explicit visual endpoints that retain non-target content. We then derive a denoising target from each source-to-counterpart pair and use a dual-branch redirection loss to align both text-conditioned and unconditional predictions with this target, since conditional supervision alone does not explicitly constrain generation without textual guidance. To mitigate the adverse effects of concept erasure on non-target generation, we jointly optimize the redirection loss with a counterpart retention loss that matches denoising predictions from the frozen pretrained model. Across style, celebrity, and nudity erasure, VisualErase limits the maximum attack success rate over seven attacks to 0%, 8%, and 0.1%, respectively, while retaining general generation quality. These results highlight the importance of visual trajectory redirection for robust concept erasure beyond text-to-image mappings alone.

Figures & tables

Figure 1: Conceptual comparison of text-to-image redirection and visual trajectory redirection. (a) Text-to-image redirection steers generation under a target text condition toward an alternative concept, but does not necessarily remove the underlying ability to generate the target concept. (b) VisualErase instead supervises denoising from source visual states at different timesteps toward a content-aligned, concept-removed counterpart that defines the desired visual endpoint. Red arrows indicate the respective redirection objectives.

Figure 2: Overview of VisualErase . (a) Paired Data Construction creates a content-aligned counterpart in which the target concept is absent. (b) For visual trajectory erasure, Target Derivation converts the pair into a denoising target, which Dual-Branch Redirection applies under both the source and empty text conditions. Counterpart Retention is described in the text.

Erasure method	ACC $\downarrow$	Attack Success Rate (%) $\downarrow$							Generation utility
Erasure method	ACC $\downarrow$	PEZ	MMA	RAB	P4D	UDA	CCE	TINA+	FID $\downarrow$	CLIP $\uparrow$
SD1.4	76.0	40.0	40.0	90.0	94.0	96.0	60.0	76.0	14.0	26.6
ESD	2.0	2.0	0.0	6.0	30.0	32.0	8.0	60.0	14.5	25.9
FMN	10.0	0.0	2.0	6.0	54.0	56.0	18.0	70.0	13.8	26.5
AC	12.0	0.0	6.0	14.0	68.0	77.0	14.0	72.0	14.0	26.5
MACE	4.0	0.0	0.0	4.0	42.0	56.0	26.0	72.0	12.4	23.5

Table 1: Van Gogh style erasure.

Figure 3: Qualitative comparison across style, celebrity, and nudity erasure under standard generation and TINA+. Each setting compares SD1.4, ESD, STEREO, and VisualErase on the same evaluation cases. Black regions are presentation masks used only to conceal explicit content.

Erasure method	ACC $\downarrow$	Attack Success Rate (%) $\downarrow$							Generation utility
Erasure method	ACC $\downarrow$	PEZ	MMA	RAB	P4D	UDA	CCE	TINA+	FID $\downarrow$	CLIP $\uparrow$
SD1.4	35.6	31.4	34.8	70.5	40.7	39.8	3.5	42.4	14.0	26.6
ESD	2.5	0.8	0.7	20.0	3.5	5.6	46.5	39.0	13.6	25.6
FMN	33.9	30.5	31.1	69.5	38.0	38.7	15.5	44.9	13.8	26.3
UCE	0.0	4.2	6.9	8.4	7.0	7.7	14.8	41.5	14.3	26.4
MACE	0.0	1.7	0.1	0.0	5.6	7.7	17.6	43.2	12.8	24.2

Table 3: Nudity erasure under the VLM-based semantic evaluation protocol.

Variant	TINA+ ASR $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
Cond.	66.0	16.0	26.8
Uncond.	2.0	25.3	26.5
Cond. + Uncond.	0.0	22.0	26.5
Cond. + Uncond. + Retent. ( VisualErase )	0.0	14.7	26.5

Table 4: Objective ablation on Van Gogh style erasure.

Trainable scope	Updated (M)	UDA ASR $\downarrow$	TINA+ ASR $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
Cross-KV	19.2	2.0	46.0	21.1	26.1
Cross-Attn	44.0	0.0	2.0	17.4	26.7
Self-Attn	49.6	6.0	0.0	14.2	26.6
All-Attn ( VisualErase )	93.5	0.0	0.0	14.7	26.5
Non-Cross	815.4	2.0	0.0	14.5	26.5
All Linear/Conv	859.3	0.0	0.0	14.4	26.5

Table 5: Trainable U-Net parameter scope on Van Gogh style erasure under the complete VisualErase objective. Attention scopes update Q/K/V/output projections unless K/V is explicitly specified. Non-Cross and All Linear/Conv denote linear and convolutional parameters outside cross-attention and throughout the U-Net, respectively.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Comparison of paired-data construction strategies. Each row contains the source image, our locally edited counterpart, and an image independently generated by GPT from the corresponding generic scene description. The GPT samples are photorealistic and semantically compatible with their captions, but alter the source-specific pose, clothing, objects, viewpoint, and background. Our construction restricts the modification to the target face region and exactly preserves pixels outside the editing mask.

Figure 6: Representative NudeNet false positives without recognizable explicit anatomy.

Erasure method	ACC $\downarrow$	Attack Success Rate (%) $\downarrow$							Generation utility
Erasure method	ACC $\downarrow$	PEZ	MMA	RAB	P4D	UDA	CCE	TINA+	FID $\downarrow$	CLIP $\uparrow$
SD1.4	94.1	66.1	70.9	96.8	100.0	100.0	32.4	100.0	14.0	26.6
ESD	21.2	11.9	13.1	50.5	69.0	76.1	74.7	86.4	13.6	25.6
FMN	88.1	62.7	67.0	97.9	97.9	97.9	54.9	100.0	13.8	26.3
UCE	24.6	25.4	32.6	29.5	76.1	78.9	49.3	97.5	14.3	26.4
MACE	10.2	8.5	6.0	6.3	75.4	81.7	50.0	93.2	12.8	24.2

Appendix

Table 6: Classifier-only NudeNet evaluation for Nudity erasure at a threshold of 0.45.

Trainable scope	Active	Updated (M)	ACC $\downarrow$	TINA+ $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
All-Attn	100%	93.5	0.0	6.8	15.5	26.8
Non-Cross	100%	815.4	0.0	0.8	16.7	26.3
All Linear/Conv	100%	859.3	0.0	0.8	16.3	26.4
SG Non-Cross	50%	407.7	0.0	0.0	16.7	26.4
SG Non-Cross	25%	203.9	0.8	0.8	16.0	26.5
SG Non-Cross ( VisualErase )	10%	81.5	0.0	0.0	15.5	26.6

Appendix

Table 7: Trainable scope and saliency ratio for Nudity erasure. All rows use natural clothed counterparts and 5,000 updates. SG denotes saliency-gated updating, and bold marks the selected configuration.

Method	Time (min)	Relative time
STEREO	30.0	2.12 $\times$
VisualErase	14.2	1.00 $\times$

Appendix

Table 8: Wall-clock training time for Van Gogh erasure on one H100.

Figure 7: VisualErase on Tench object erasure under standard target prompts and TINA+ attacks. The generated images do not contain a recognizable Tench.

Figure 8: Tench classifier predictions. All examples are predicted as Tench. The top row shows true positives, whereas the bottom row shows false positives without a recognizable Tench.

Explore similar work

Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models
Jul 6, 2026Parth Upman, Nishita Jain, Shreyank N GowdaConcept Erasure Text-To-Image Diffusion Models
School of Computer Science, University of Nottingham, Nottingham, NG8 1BB, UK · Department of Computing, Imperial College London, London, SW7 2AZ, UK
Continual Concept Erasure in Diffusion Models by Suppressing Cross-Edit Interference
Oct 1, 2026Yongliang Wu, Haori Lu, Jinqi Luo +3Concept Erasure Text-To-Image Diffusion Models
University of Illinois Urbana-Champaign · University of Pennsylvania · National University of Singapore
GRACE: Adaptive Concept Erasure with Geometry-Guided Retention in Diffusion Models
Sep 14, 2026Qinghui Gong, Yihuai Liang, Yuanlun Xie +3Concept Erasure Text-To-Image Diffusion Models
School of Information Science and Technology, Southwest Jiaotong University, Chengdu 611756, China · School of Electronic Information and Electrical Engineering, Chengdu University, Chengdu, China · Key Laboratory of Intelligent Control and Optimization for Industrial Equipment of Ministry of Education, Dalian University of Technology, Dalian, China +1

Jul 6, 2026, cs.CVJ/K move · Enter open · S save

Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models

Parth Upman, Nishita Jain, Shreyank N Gowda

School of Computer Science, University of Nottingham, Nottingham, NG8 1BB, UK · Department of Computing, Imperial College London, London, SW7 2AZ, UK

Concept Erasure Text-To-Image Diffusion Models Diffusion Models Cross-Attention Semantic Anchor Structural Damage+1

Open paper AskPDF arXiv

Training-free concept erasure is an attractive mechanism for controlling text-to-image diffusion models, but precise erasure often comes at the cost of damaging semantically related non-target concepts. Existing value-space methods remove the component of each cross-attention value along the target concept direction, implicitly treating target identity and shared visual structure as the same signal. We argue that this is the source of much of the collateral damage in prior preservation. We introduce CARE, a closed-form concept erasure operator that replaces the raw target direction with a kept-subspace-aware direction computed from a small bank of retained concept anchors. The resulting edit is applied directly in cross-attention value space, requires no model fine-tuning, and adds only a negligible offline computation. A single shrinkage parameter controls the erase-preserve trade-off. We further show that the operator admits a minimum-disturbance interpretation and, in its projection form, leaves the kept subspace invariant. Experiments under the standard concept-erasure protocol show that our method preserves non-target concepts more faithfully while maintaining competitive erasure across instance, style, and celebrity concepts. Code: https://github.com/parthupman/care

Abstract

Explore similar work

Erasing Without Collateral Damage: Precise Concept Removal in Diffusion Models

Continual Concept Erasure in Diffusion Models by Suppressing Cross-Edit Interference

GRACE: Adaptive Concept Erasure with Geometry-Guided Retention in Diffusion Models