Rethinking Negative Prompting for Concept Suppression: Unconditional Anchoring and Localized Guidance

Anubhav Jain; Takashi Shibuya; Yuya Kobayashi; Yuhta Takida; Nasir Memon; Julian Togelius; Yuki Mitsufuji

cs.CVDec 10, 2024

Rethinking Negative Prompting for Concept Suppression: Unconditional Anchoring and Localized Guidance

Authors: Anubhav Jain, Takashi Shibuya, Yuya Kobayashi, Yuhta Takida, Nasir Memon, Julian Togelius, Yuki Mitsufuji

Organizations: New York University · Sony AI · Sony Group Corporation

Abstract

Text-to-image diffusion models have rapidly transitioned from research prototypes to widely deployed tools. Along with their impressive capabilities, these models can also generate harmful or undesirable content (e.g., NSFW imagery), motivating practical mechanisms for removing such concepts. Negative prompting, which uses the negative-prompt prediction as the guidance anchor, is among the most widely used mechanisms for steering sampling trajectories away from unwanted concepts. However, for jailbreak prompts, it can fail in a regime we refer to as guidance collapse, where the model's denoising predictions for the prompt and negative prompt become similar, so that the guidance difference driving negative prompting vanishes and the guided prediction degenerates to the negative-prompt prediction, undermining suppression. To fix this issue, we propose TraSCE (Trajectory Steering for ConcEpt Suppression), a training-free inference-time method that makes negative prompting reliable under guidance collapse. TraSCE combines two complementary components: (i) unconditional anchoring, which makes the fallback under guidance collapse a neutral unconditional prediction rather than the negative-prompt prediction, and (ii) an alignment-localized steering step, which deliberately induces this collapse for adversarial prompts and steers the sample to the unconditional fallback, while remaining minimally intrusive on benign prompts. We evaluate TraSCE on adversarial prompt benchmarks targeting nudity and violence, and further demonstrate its applicability to suppressing artistic styles and objects. Across these settings, TraSCE substantially improves the robustness of negative prompting to jailbreak prompts without any training, weight updates, or pre-collected concept-specific data (prompts or images), while keeping the impact on general image quality small.

Figures & tables

Panel of Figure 2 — Figure 2: Empirical evidence of guidance collapse under jailbreak prompts. (Left) Per-timestep alignment gap $d_{t}=\|\hat{\bm{\epsilon}}_{\textnormal{p}}-\hat{\bm{\epsilon}}_{\textnormal{np}}\|$ for benign (blue) versus adversarial (red) prompts; adversarial prompts yield consistently smaller gaps. (Right) Probability $P(d_{t}<\delta)$ that the alignment gap falls below a threshold; adversarial prompts fall below every threshold far more often than benign prompts.

Method	Ring-A-Bell			MMA-Diffusion $\downarrow$	P4D $\downarrow$	UnLearnDiffAtk $\downarrow$	I2P $\downarrow$	COCO
Method	K77 $\downarrow$	K38 $\downarrow$	K16 $\downarrow$	MMA-Diffusion $\downarrow$	P4D $\downarrow$	UnLearnDiffAtk $\downarrow$	I2P $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
SDv1.4	85.26	87.37	93.68	95.7	98.7	69.7	17.8	16.71	0.304
SA ( Heng & Soh, 2024 )	63.15	56.84	56.84	9.30	47.68	12.68	2.81	25.80	0.297
CA ( Kumari et al., 2023 )	86.32	91.58	89.47	9.90	10.60	5.63	1.04	24.12	0.301
ESD ( Gandikota et al., 2023 )	20.00	29.47	35.79	12.70	9.27	15.49	2.87	18.18	0.302
MACE ( Lu et al., 2024 )	2.10	0.0	0.0	0.50	2.72	2.82	1.51	16.80	0.287

Table 1: Comparison with baseline defenses under adversarial prompts targeting nudity generation: Ring-A-Bell ( Tsai et al., 2023 ) , MMA-Diffusion ( Yang et al., 2024 ) , P4D ( Chin et al., 2024 ) , UnLearnDiffAtk ( Zhang et al., 2024c ) , and I2P ( Schramowski et al., 2023 ) . We report the attack success rate (ASR; lower is better). Nudity detection is performed using NudeNet ( Bedapudi, 2019 ) with a threshold of

0.45

, using the same detector and threshold for all methods. See Appendix B for detector versions and benchmark-specific settings. Rows are grouped by deployment constraint: gray = training-based fine-tuning (requires training and weight updates); blue = training-free weight-editing (updates weights without training); green = inference-time (no training and no weight updates).

Figure 3: Qualitative comparisons under jailbreak prompts targeting nudity generation. TraSCE reduces nudity content while better preserving prompt semantics and visual quality. Additional examples are provided in Appendix Figure 12 .

Method	ASR $\downarrow$
SDv1.4	99.6
FNM ( Zhang et al., 2024a )	98.8
CA ( Kumari et al., 2023 )	100
ESD ( Gandikota et al., 2023 )	86.0
UCE ( Gandikota et al., 2024 )	89.8
RECE ( Gong et al., 2024 )	89.2

Table 2: Results on suppressing violence on the adversarial Ring-A-Bell-Union ( Tsai et al., 2023 ) prompt set. We report attack success rate (ASR; lower is better) measured by the Q16 detector ( Schramowski et al., 2022 ) , using the same detector and settings for all methods (see Appendix B for details). COCO-30K FID/CLIP for all methods are reported in Table 1 . Note that SLD-Max attains lower ASR here but with substantially degraded general image quality.

Figure 4: Comparison under adversarial prompts targeting Van Gogh style found by Ring-A-Bell ( Tsai et al., 2023 ) . TraSCE reduces Van Gogh-style artifacts while preserving the prompt content.

Method	Remove “Van Gogh”		Remove “Kelly McKernan”
Method	$\text{ACC}_{\text{e}}\downarrow$	$\text{ACC}_{\text{u}}\uparrow$	$\text{ACC}_{\text{e}}\downarrow$	$\text{ACC}_{\text{u}}\uparrow$
SDv1.4	100.0	94.93	70.0	67.08
CA ( Kumari et al., 2023 )	85.00	96.00	55.00	69.23
ESD-x ( Gandikota et al., 2023 )	100.0	89.18	81.25	69.33
RECE ( Gong et al., 2024 )	75.00	92.40	30.00	68.75
UCE ( Gandikota et al., 2024 )	80.00	91.13	55.00	63.75

Table 3: Results on artistic style suppression while maintaining unrelated styles.

\text{ACC}_{\text{e}}

is the rate at which the erased style is still recognized as the target style (lower is better), and

\text{ACC}_{\text{u}}

is the recognition accuracy for unrelated styles (higher is better). Style classification is performed by GPT-4o following ( Yoon et al., 2024 ) .

Figure 5: Qualitative results on suppressing the object “car” under an adversarial prompt generated by Ring-A-Bell ( Tsai et al., 2023 ) . TraSCE suppresses the target object while preserving overall image quality.

Panel of Figure 6 — Figure 6: Trajectory analysis of TraSCE (object erasure example). Prompt: “photo of a guitar”; negative concept: “guitar”. (a) Distance to the negative-concept direction, $\|\hat{\bm{\epsilon}}-\hat{\bm{\epsilon}}_{\textnormal{np}}\|_{2}^{2}$ , across timesteps, where $\hat{\bm{\epsilon}}$ denotes the guided noise prediction used by each method. The unconditional-anchored formulation (Eq. ( 3 )) keeps the guided direction farther from the negative-concept direction than standard negative prompting (Eq. ( 2 )) and SLD-Max, especially in later steps. (b) Loss values $\mathcal{L}_{t}$ (Eq. ( 4 )) with/without localized guidance, showing that the loss-based step reduces the discrepancy between $\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t,\bm{e}_{\textnormal{p}})$ and $\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t,\bm{e}_{\textnormal{np}})$ .

Negative prompting		Steering	Ring-A-Bell (ASR)			FID $\downarrow$
Std. NP	UA-NP	step	K77 $\downarrow$	K38 $\downarrow$	K16 $\downarrow$	FID $\downarrow$
✓			17.89	28.42	34.74	18.33
✓		✓	10.63	10.63	13.82	18.45
	✓		4.21	10.52	11.57	18.59
	✓	✓	1.05	2.10	2.10	17.41

Table 4: Component ablation of TraSCE on Ring-A-Bell (nudity) and COCO-30K. We compare standard negative prompting (Std. NP; Eq. ( 2 )) and unconditional-anchored negative prompting (UA-NP; Eq. ( 3 )), with/without alignment-localized steering. Combining unconditional anchoring with localized guidance reduces ASR while keeping FID close to the base model.

Method	Ring-A-Bell			MMA-Diffusion $\downarrow$	P4D $\downarrow$	UnLearnDiffAtk $\downarrow$	COCO
Method	K77 $\downarrow$	K38 $\downarrow$	K16 $\downarrow$	MMA-Diffusion $\downarrow$	P4D $\downarrow$	UnLearnDiffAtk $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
FLUX.1-dev	82.11	82.11	83.16	11.80	49.01	23.94	20.41	0.309
TraSCE (ours)	7.37	3.16	9.47	0.80	2.65	2.11	21.41	0.309

Table 5: Evaluation on FLUX.1-dev under adversarial prompt benchmarks. Ring-A-Bell and MMA-Diffusion are black-box prompt sets, while P4D and UnLearnDiffAtk are static prompt sets produced by white-box attacks. We report ASR (lower is better) and COCO-30K FID/CLIP to assess the trade-off between safety and general image quality.

Method	Detected Nudity (Quantity)				COCO
Method	Common	Female	Male	Total $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
FLUX.1-dev	406	161	38	605	20.41	30.87
CA (Model-based) ( Kumari et al., 2023 )	253	65	26	344	22.66	29.05
CA (Noise-based) ( Kumari et al., 2023 )	290	72	28	390	23.07	28.73
ESD ( Gandikota et al., 2023 )	329	145	32	506	23.08	28.44
UCE ( Gandikota et al., 2024 )	122	39	12	173	30.71	24.56

Table 6: Assessment of nudity suppression on FLUX.1-dev. (Left) Number of images flagged as explicit by NudeNet on I2P ( Schramowski et al., 2023 ) (lower is better). (Right) FID and CLIP score on COCO-30K ( Lin et al., 2014 ) . Following Gao et al. (2025) , we use a NudeNet threshold of 0.6.

Method	Inference time (s)
SDv1.4 w/ Std. NP	5.45
SLD	6.81
SAFREE	6.90
TraSCE (UA-NP; no steering)	6.80
TraSCE (UA-NP + steering)	14.29

Table 7: Comparing inference time (A100 GPU).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Ring-A-Bell (ASR)			FID $\downarrow$
Method	K77 $\downarrow$	K38 $\downarrow$	K16 $\downarrow$	FID $\downarrow$
Classifier guidance
w/ CLIP	13.82	23.40	24.46	19.82
w/ NudeNet	7.44	10.63	19.14	51.67
Alignment-localized steering	10.63	10.63	13.82	18.45

Appendix

Table 8: Ablation on guidance strategies for reducing nudity generations on Ring-A-Bell Tsai et al. (2023) . To isolate the effect of the guidance signal, all methods use the same negative prompting formulation (Eq. 2 ) and differ only in the additional guidance term. We report ASR (lower is better) and COCO-30K FID (lower is better). NudeNet-based classifier guidance achieves lower ASR but severely degrades image quality, whereas our alignment-localized steering improves robustness with much smaller FID degradation.

Figure 7: Visual comparison between directly minimizing an MSE loss and using our localized exponential loss (Eq. ( 4 )). The exponential loss better preserves perceptual quality by reducing unnecessary interference when the two predictions are far apart.

Method	Ring-A-Bell (ASR)			FID $\downarrow$	CLIP $\uparrow$
Method	K16 $\downarrow$	K38 $\downarrow$	K77 $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
Hard gate ( $\tau=0.5$ )	17.80	15.79	8.42	19.64	0.298
Hard gate ( $\tau=1.0$ )	9.47	15.79	7.37	18.54	0.298
Hard gate ( $\tau=1.5$ )	5.26	8.42	6.32	16.51	0.296
Hard gate ( $\tau=3.0$ )	0.00	0.00	1.05	25.18	0.265
Soft gate (ours)	1.05	2.10	2.10	17.41	0.299

Appendix

Table 9: Comparison between hard-threshold gating (sweeping

\tau

) and our Gaussian soft gate on Ring-A-Bell (nudity) and COCO-30K. Lower ASR and lower FID are better; higher CLIP is better.

Method	Ring-A-Bell (ASR)			FID $\downarrow$	CLIP $\uparrow$
Method	K16 $\downarrow$	K38 $\downarrow$	K77 $\downarrow$	FID $\downarrow$	CLIP $\uparrow$
Std. NP ( $s=7.5$ )	17.89	28.42	34.74	18.33	0.301
Std. NP ( $s=20$ )	57.89	49.47	42.11	25.33	0.299
TraSCE (ours)	1.05	2.10	2.10	17.41	0.299

Appendix

Table 10: Comparison against simply increasing the classifier-free guidance (CFG) scale on Ring-A-Bell (nudity) and COCO-30K. Increasing the CFG scale does not defend against jailbreak prompts. The repulsion from the negative-prompt prediction has magnitude

s\delta

, so it vanishes as the alignment gap

\delta\to 0

under jailbreak prompts, no matter how large the scale

s

is (Proposition 1 ). Lower ASR and FID are better; higher CLIP is better.

Class name	Accuracy of erased class (%) $\downarrow$						Accuracy of other classes (%) $\uparrow$
Class name	SD	ESD-u	UCE	RECE	SD-NP	TraSCE (ours)	SD	ESD-u	UCE	RECE	SD-NP	TraSCE (ours)
Cassette Player	15.6	0.6	0.0	0.0	4.6	0.0	85.1	64.5	90.3	90.3	64.1	62.8
Chain Saw	66.0	6.0	0.0	0.0	25.2	0.2	79.6	68.2	76.1	76.1	50.9	49.7
Church	73.8	54.2	8.4	2.0	21.2	0.2	78.7	71.6	80.2	80.5	58.4	57.5
English Springer	92.5	6.2	0.2	0.0	0.0	0.0	76.6	62.6	78.9	77.8	63.6	62.8
French Horn	99.6	0.4	0.0	0.0	0.0	0.0	75.8	49.4	77.0	77.0	58.0	54.9

Appendix

Table 11: Results on suppressin objects from Imagenette Howard (2019) , evaluated using a pretrained ResNet-50 classifier. Prompts follow the standard template “Image of a {Object}”. We report (i) accuracy on the erased class (lower is better) and (ii) average accuracy on the remaining classes (higher is better). Note that ESD, UCE, and RECE update model weights, while TraSCE is training-free and does not modify weights.

Panel of Figure 8 — Figure 8: Hyperparameter sensitivity analysis.

Panel of Figure 9 — Figure 9: Minimal-invasiveness study using the same examples as Fig. 1 in Amara et al. (2025) . Rows 1 & 3 use prompts containing the suppressed concept (“koala”, “Sun”); Rows 2 & 4 use benign prompts describing closely related concepts (a sloth for “koala”, the moon for “Sun”). TraSCE preserves the benign concept while still suppressing the target, indicating higher specificity.

Panel of Figure 10 — Figure 10: Concept Suppression with semantically related negative prompts. We suppress a target concept using a negative prompt that is semantically related but not an exact match (“swimsuit” vs. “bikini”) on SDv1.4. TraSCE reduces the target concept even when the prompt and the negative prompt use different but closely related terms (e.g., broader/narrower terms).

Panel of Figure 11 — Figure 11: Multi-concept suppression on FLUX.1-dev. Prompt: “A depiction of a starry night over a quiet town, reminiscent of Van Gogh’s famous painting”.

Category	TraSCE	ESD	SLD
Least Resembles Van Gogh Style	45.56%/62.5%	17.93%/0.0%	36.41%/37.5%
Best Image Quality	50.54%/100.0%	33.15%/0.0%	16.30%/0.0%

Appendix

Table 12: Human evaluation of artistic style suppression (23 participants; 8 prompts). For each prompt, participants compare outputs from TraSCE, ESD ( Gandikota et al., 2023 ) , and SLD ( Schramowski et al., 2023 ) and vote for (i) the image that least resembles Van Gogh style while preserving prompt semantics, and (ii) the best image quality. We report total vote share across all prompts / percentage of prompts where the method received the most votes .

Figure 12: Additional qualitative comparisons under adversarial prompts aimed at generating nudity content. Many jailbreak prompts are semantically ill-formed (e.g., dominated by random token sequences), for which the intended output is not well-defined; in such cases, TraSCE prioritizes suppression and may produce less semantically specific or less coherent images.

Figure 13: Qualitative results on suppressing the artistic style of Kelly McKernan for the prompt “Whimsical fairy tale scene by Kelly McKernan”, while preserving unrelated styles (Thomas Kinkade, Kilian Eng, and Ajin: Demi-Human). TraSCE better preserves unrelated styles and prompt semantics while suppressing the target style.

Figure 14: Impact on general image generation quality on COCO-30K prompts. TraSCE preserves overall image quality and prompt alignment on non-adversarial prompts.

Diffusion Model Guidance

Diffusion model guidance steers denoising toward desired conditions, rewards, constraints, or away from unwanted concepts.

See all 362 papers

Concept Removal Guidance: Evidence-Calibrated Negative Guidance for Safe Diffusion Sampling
Jun 29, 2026Yoonseok Choi, Chaeyoung Oh, Hyunjun Choi +2Diffusion Model Guidance Diffusion Guidance
Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · Center for Humanoid Research, KIST, Seoul, Republic of Korea
Orthogonal Negative Guidance in Attention Feature Space for Text-to-Image Generation
May 28, 2026Jungmin Ko, Jungwon Park, Jimyeong Kim +3Diffusion Model Guidance T2I Generation
Interdisciplinary Program in Artificial Intelligence, Seoul National University · Research Institute for Convergence Science, Seoul National University · Daegu Gyeongbuk Institute of Science and Technology +3
EGLOCE: Training-Free Energy-Guided Latent Optimization for Concept Erasure
Apr 10, 2026Junyeong Ahn, Seojin Yoon, Sungyong BaikDiffusion Model Unlearning Safe T2I Generation
KAIST AI · Hanyang University

Jun 29, 2026J/K move · Enter open · S save

Concept Removal Guidance: Evidence-Calibrated Negative Guidance for Safe Diffusion Sampling

Yoonseok Choi, Chaeyoung Oh, Hyunjun Choi, Seokin Seo+1

Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · Center for Humanoid Research, KIST, Seoul, Republic of Korea

Diffusion Model Guidance Diffusion Guidance Adversarial Robustness Concept Erasure T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image diffusion models remain vulnerable to adversarial prompts that elicit disallowed content, motivating reliable inference-time controls. A popular approach is negative guidance, which subtracts a negative prompt direction with a fixed weight. However, it often forces a safety-fidelity trade-off, causing artifacts or prompt drift when over-applied and failing under attacks when under-applied. Dynamic variants reweight guidance using posterior-odds signals, which can be brittle for open-vocabulary compositional prompts, while lightweight similarity-based methods ignore the evolving image evidence along the denoising trajectory. We introduce Concept Removal Guidance (CRG), a training-free method that estimates unwanted-concept presence at each diffusion step from the model's noise predictions, and adaptively calibrates negative guidance via a closed-form constrained update enforcing a target presence threshold while minimally perturbing the conditional trajectory. Across red-teaming benchmarks, CRG reduces attack success rates while preserving benign fidelity, and extends to additional suppression targets such as artist style and violence without fine-tuning or external classifiers.

Abstract

Diffusion Model Guidance

Explore similar work

Concept Removal Guidance: Evidence-Calibrated Negative Guidance for Safe Diffusion Sampling

Orthogonal Negative Guidance in Attention Feature Space for Text-to-Image Generation

EGLOCE: Training-Free Energy-Guided Latent Optimization for Concept Erasure