SafeCtrl: Region-Aware Safety Control for Text-to-Image Diffusion via Detect-Then-Suppress

Lingyun Zhang; Yu Xie; Zhongli Fang; Yu Liu; Ping Chen

cs.CVApr 5, 2026

SafeCtrl: Region-Aware Safety Control for Text-to-Image Diffusion via Detect-Then-Suppress

Authors: Lingyun Zhang, Yu Xie, Zhongli Fang, Yu Liu, Ping Chen

Organizations: School of Computer Science and Technology, Fudan University, China · Institute of Big Data, Fudan University, China · Nanjing University of Information Science and Technology, China

Abstract

The widespread deployment of text-to-image diffusion models is significantly challenged by the generation of visually harmful content, such as sexually explicit content, violence, and horror imagery. Common safety interventions, ranging from input filtering to model concept erasure, often suffer from two critical limitations: (1) a severe trade-off between safety and context preservation, where removing unsafe concepts degrades the fidelity of the safe content, and (2) vulnerability to adversarial attacks, where safety mechanisms are easily bypassed. To address these challenges, we propose SafeCtrl, a Region-Aware safety control framework operating on a Detect-Then-Suppress paradigm. Unlike global safety interventions, SafeCtrl first employs an attention-guided Detect module to precisely localize specific risk regions. Subsequently, a localized Suppress module, optimized via image-level Direct Preference Optimization (DPO), neutralizes harmful semantics only within the detected areas, effectively transforming unsafe objects into safe alternatives while leaving the surrounding context intact. Extensive experiments across multiple risk categories demonstrate that SafeCtrl achieves a superior trade-off between safety and fidelity compared to state-of-the-art methods. Crucially, our approach exhibits improved resilience against adversarial prompt attacks, offering a precise and robust solution for responsible generation.

Figures & tables

Fig. 1: Solving the safety and context preservation trade-off. (a) The original Stable Diffusion generates unsafe content (e.g., sexually explicit content) from a harmful prompt. (b) Prior Global Fine-tuning methods (e.g., ESD) remove the concept but suffer from severe context degradation, altering the background, lighting, and identity. (c) SafeCtrl (Ours) employs a Region-Aware “Detect-Then-Suppress” paradigm. By keeping the base model frozen (Blue blocks) and using a external module (Green blocks) trained via DPO, SafeCtrl precisely localizes and neutralizes risk, ensuring safety while strictly preserving the original context and artistic intent.

Fig. 2: The overall architecture of the SafeCtrl framework. (Left) SafeCtrl is instantiated as a set of external modules that operate in parallel with frozen U-Net. The safety control is adaptively activated within specific diffusion stages: the Detection Timestep Window $[T_{\text{start}},T_{\text{switch}}]$ and the Suppression Timestep Window $[T_{\text{switch}},0]$ . (Right) A detailed view inside a single Safety Control Module, illustrating the Detect-Then-Suppress paradigm. The Risk Detection component first analyzes image features to compute risk mask.The suppression process is governed by an activation trigger ( $A$ ). Only if a risk is detected does the Preference-Aligned Suppression component activate, computing a safety-guided attention output ( $V_{safe}$ ) to the original cross-attention layer.

Fig. 3: Detailed comparison of NudeNet detection counts on the I2P benchmark. Each subplot corresponds to a different safety method, showing the number of generated images flagged for various nudity-related categories (lower is better).

Fig. 4: Impact of Timesteps on Detection Accuracy. mIoU scores across concepts consistently improve as the denoising step

t

decreases (1000

\to

0). This empirically justifies our Dynamic Scheduling. We activate detection around

t\in[600,800]

where semantic structures stabilize, avoiding computation on early noisy steps.

Fig. 5: Qualitative Comparison on Nudity Suppression. Columns show results using the same seed. SLD often fail to ensure safety, whereas ESD, SPM, AlignGuard and RDM, achieve safety at the cost of altering the original background and identity, leading to context degradation. Although CR operates locally, its hard replacement can appear rigid artifacts. SafeCtrl combines precise localization (red masks) with natural DPO-based suppression, consistently generating safe outputs while preserving the original artistic intent and background details.

Method	Overall $\downarrow$	FID $\downarrow$	CLIP $\uparrow$	H-Score $\uparrow$
Original SD	0.40	14.30	0.2626	0.750
SLD (CVPR’23)	0.19	18.22	0.2543	0.640
ESD (ICCV’23)	0.18	17.34	0.2381	0.402
SPM (CVPR’24)	0.34	14.77	0.2581	0.751
AlignGuard (ICCV’25)	0.14	20.80	0.2554	0.501
RDM (WWW’25)	0.12	15.13	0.2588	0.869

TABLE I: Quantitative Comparison on I2P and COCO-30K. We report the Overall Unsafe Ratio (lower

\downarrow

is better), FID (lower

\downarrow

), and CLIP (higher

\uparrow

). The H-Score (higher

\uparrow

) quantifies the trade-off, calculated as the harmonic mean of Safety (

1-\text{Overall}

) and normalized Utility. SafeCtrl achieves the best safety–utility trade-off.

Method	Pascal-Car	CelebA-HQ	Pascal-Horse	Avg.
ReGAN	52.2	69.9	54.3	58.8
SLiMe	68.3	75.7	63.3	69.1
SegDDPM*	-	78.0	58.7	-
CR	69.3	78.1	64.3	70.6
Ours (SafeCtrl)	72.1	78.3	65.7	72.0

TABLE II: Localization Accuracy (mIoU

\uparrow

) on Proxy Benchmarks. Evaluating under a 10-shot setting. SafeCtrl outperforms baselines including the heavyweight CR, verifying precise region awareness.

Method	SD	SLD	ESD	SPM	AlignG.	RDM	CR	Ours
Unsafe $\downarrow$	0.95	0.97	0.54	0.65	0.63	0.32	0.27	0.28

TABLE III: Robustness against Adversarial Prompts (Ring-A-Bell). We report the Unsafe Ratio (

\downarrow

). SafeCtrl maintains robust defense, achieving performance on par with CR.

Method	Total Added Params $\downarrow$	Latency (s) $\downarrow$
Original SD	-	10.63
CR	$\sim$ 860 M	12.70
Ours (SafeCtrl)	$\sim$ 75 M	11.77

TABLE IV: Efficiency Analysis. Unlike CR which duplicates the U-Net (

\sim

860M params), SafeCtrl is a lightweight plugin (

\sim

75M), offering comparable safety with significantly lower overhead.

Fig. 6: Generalization to Diverse Visual Risks. SafeCtrl effectively generalizes beyond nudity to mitigate other distinct visual threats. Suppression of abstract horror concepts (e.g., ghosts, zombies, monsters) into benign objects (e.g., dolls, humans) or suppression of concrete violence/weapons (e.g., knives, guns) into safe alternatives.

Safe T2I Generation

Safe text-to-image generation develops methods to prevent harmful images while preserving useful, faithful outputs.

See all 52 papers

Disciplined Diffusion: Text-to-Image Diffusion Model against NSFW Generation
May 1, 2026Chi Zhang, Changjia Zhu, Xiaowen Li +2Adversarial Robustness Safe T2I Generation
Bellini College of Artificial Intelligence, Cybersecurity and Computing, University of South Florida, Tampa, FL, USA · Department of Electrical and Computer Engineering, College of Engineering, University of South Florida, Tampa, FL, USA
Introspective Attention Modulation for Safe Text-to-Image Generation
Jul 16, 2026Basim Azam, Hossein Rahmani, Naveed AkhtarDiffusion Transformer T2I Generation
The University of Melbourne, Australia · Lancaster University, United Kingdom
Modular Energy Steering for Safe Text-to-Image Generation with Foundation Models
Apr 2, 2026Yaoteng Tan, Zikui Cai, M. Salman AsifSafe T2I Generation T2I Diffusion Models
University of California Riverside · University of Maryland

May 1, 2026J/K move · Enter open · S save

Disciplined Diffusion: Text-to-Image Diffusion Model against NSFW Generation

Chi Zhang, Changjia Zhu, Xiaowen Li, Yao Liu+1

Bellini College of Artificial Intelligence, Cybersecurity and Computing, University of South Florida, Tampa, FL, USA · Department of Electrical and Computer Engineering, College of Engineering, University of South Florida, Tampa, FL, USA

Adversarial Robustness Safe T2I Generation T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image (T2I) diffusion models have the ability to build high-quality pictures from text prompts, but they pose safety concerns because they can generate offensive or disturbing imagery when provided with harmful inputs. Existing safety filters typically rely on text-based classifiers or image-based checkers that completely block the output upon detecting a threat, issuing an explicit allow/block feedback signal to the user. This binary strategy leaves models vulnerable to adversarial attacks that alter keywords to bypass detection, and it causes high false-alarm rates that degrade the experience for benign users. To address such vulnerabilities, we propose Disciplined Diffusion (DDiffusion), a novel robust text-to-image diffusion that counters Not Safe For Work (NSFW) generation by uncovering implicit malicious semantics in prompt embeddings. DDiffusion leverages a semantic retrieval mechanism to evaluate prompts against concept distributions rather than relying on brittle pairwise similarity. Furthermore, it employs a localization method during the diffusion process to selectively edit only the harmful regions of the generated image. By returning locally sanitized images instead of applying uniform blocking, DDiffusion suppresses malicious content while preserving generation fidelity for benign prompts and avoiding the binary allow-deny signal on which existing probing attacks rely.

Abstract

Safe T2I Generation

Explore similar work

Disciplined Diffusion: Text-to-Image Diffusion Model against NSFW Generation

Introspective Attention Modulation for Safe Text-to-Image Generation

Modular Energy Steering for Safe Text-to-Image Generation with Foundation Models