Modular Energy Steering for Safe Text-to-Image Generation with Foundation Models

Yaoteng Tan; Zikui Cai; M. Salman Asif

cs.CVApr 2, 2026

Modular Energy Steering for Safe Text-to-Image Generation with Foundation Models

Authors: Yaoteng Tan, Zikui Cai, M. Salman Asif

Organizations: University of California Riverside · University of Maryland

Abstract

Controlling the behavior of text-to-image generative models is critical for safe and practical deployment. Existing safety approaches typically rely on model fine-tuning or curated datasets, which can degrade generation quality or limit scalability. We propose an inference-time steering framework that leverages gradient feedback from frozen pretrained foundation models to guide the generation process without modifying the underlying generator. Our key observation is that vision-language foundation models encode rich semantic representations that can be repurposed as off-the-shelf supervisory signals during generation. By injecting such feedback through clean latent estimates at each sampling step, our method formulates safety steering as an energy-based sampling problem. This design enables modular, training-free safety control that is compatible with both diffusion and flow-matching models and can generalize across diverse visual concepts. Experiments demonstrate state-of-the-art robustness against NSFW red-teaming benchmarks and effective multi-target steering, while preserving high generation quality on benign non-targeted prompts. Our framework provides a principled approach for utilizing foundation models as semantic energy estimators, enabling reliable and scalable safety control for text-to-image generation.

Figures & tables

Figure 1: Overview of the proposed steering framework. We take off-the-shelf foundation models such as CLIP or a VLM as a plug-in semantic energy estimator at inference-time to steer unwanted generations. Our method is training-free that does not perform any model weight update, and dataset-free that does not require any domain-specific image datasets beyond user provided blacklist of target concepts in textual formations.

Figure 2: Visual examples of

\hat{\mathbf{x}}_{0|t}

across different generation steps for identity-related prompts. For running a fixed number of

50

inference steps, the identity can be visually observed at step

10

(i.e.,

t=0.8T

), where semantic attributes emerge in early generation stage that motivates us to utilize them as semantic energy to manipulate generation, before the generation collapsing into high-fidelity rendering.

Figure 3: NSFW classification probability in

\hat{\mathbf{x}}_{0|t}

across generation steps

t

. The red and green curves represent classifier scores for unsafe and safe prompts averaged over a dataset (100 prompts). High probabilities at early stages suggest that pretrained CLIP and VLM provide effective energy signals for steering targeted concepts (e.g., nudity).

Safety Methods	Method Properties		Attack Success Rate ( $\downarrow$ )				COCO-30K
Safety Methods	Weight modified	External dataset	P4D	Ring- A-Bell	MMA- Diffusion	Unlearn- DiffAtk	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )
Original SD ( Rombach et al., 2022 )	–	–	0.934	0.747	0.952	0.676	20.50	31.48 $\pm$ 2.96
SAFREE ( Yoon et al., 2025 )	✗	✗	0.404	0.088	0.590	0.254	23.51	31.09 $\pm$ 3.09
CURE ⋆ ( Biswas et al., 2025 )	✓	✗	0.107	0.013	0.169	0.281	–	31.18
SafeDenoiser ⋆ ( Kim et al., 2025 )	✗	✓	–	0.127	0.469	0.207	22.55	30.66
SGF ⋆ ( Kim et al., 2026 )	✗	✓	–	0.051	0.297	0.164	23.73	30.36

Table 1: Quantitative evaluation on NSFW generation steering. ASR evaluates effectiveness on suppressing NSFW content generation. COCO-30K evaluates generation quality of models after safety method is applied. Best performance values are highlighted for each column. (Methods that are marked with

\star

are reported values from original paper due to unavailable or incomplete open-source code.)

Panel of Figure 4 — Figure 4: Quantitative evaluation of steering single ID. The heatmap colors represent the average classification probability scores, where darker regions indicate lower ID verifier confidence in the targeted identity and lighter regions indicate higher confidence. The high contrast between the diagonal and off-diagonal elements indicates our steering framework achieves precise targeted ID suppression and non-targeted ID preservations.

Figure 5: Quantitative evaluation of joint steering multi-ID. The heatmap colors represent the average classification probability scores, where darker regions indicate lower confidence and lighter regions indicate higher confidence in the targeted IDs. As shown in the joint steering of 2 up to 9 IDs, our method effectively steers multiple targeted IDs simultaneously while ensuring that untargeted IDs are generated without compromise.

Safety Method	Attack Success Rate ( $\downarrow$ )				COCO-30K
Safety Method	P4D	Ring-A-Bell	MMA-Diffusion	Unlearn-DiffAtk	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )
Original SD ( Esser et al., 2024 )	0.701	0.620	0.639	0.570	29.04	32.01
SAFREE ⋆ ( Yoon et al., 2025 )	0.271	0.430	0.165	0.302	–	31.55
Ours (CLIP)	0.224	0.215	0.137	0.176	56.26	31.26

Table 2: Quantitative evaluation on NSFW generation steering on SD-v3. ASR evaluates effectiveness on suppressing NSFW content generation. COCO-30K evaluates generation quality of models after safety method is applied. Best performance values are highlighted for each column. Our framework is scalable to advanced T2I models. (Methods that are marked with ⋆ are reported values from original paper due to incomplete open-source code.)

Figure 7: Visual examples of celebrity ID and object category steering with CLIP-based steering. Images are generated with prompt “A photo of [target ID/object]”; each column corresponds to a steered concept. Our framework precisely suppresses targeted concepts while preserving non-targeted generations.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model Variant	Attack Success Rate ( $\downarrow$ )				COCO-1K
Model Variant	P4D	Ring-A-Bell	MMA-Diffusion	Unlearn-DiffAtk	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )
ViT-B-32	0.054	0.013	0.078	0.070	75.13	31.00
ViT-L-14	0.068	0.025	0.063	0.077	74.91	31.00
ViT-H-14	0.061	0.051	0.109	0.141	87.53	30.78
ViT-bigG-14	0.088	0.025	0.074	0.063	81.77	30.82
InternVL2-1B	0.041	0.051	0.086	0.014	143.31	29.98

Appendix

Table 4: Evaluation of different CLIP and VLM architectures used as safety energy estimators. ASR evaluates effectiveness on suppressing NSFW content generation. COCO-1K evaluates generation quality with safety steering applied.

Panel of Figure 9 — Figure 9: Multi-identity steering evaluation. Heatmaps of identity classification probabilities when jointly steering different numbers of identities (1, 10, 11, and 18). Bright diagonals indicate correct identity generation, while darker off-diagonal entries indicate suppression of non-target identities.

Safety Method	Attack Success Rate ( $\downarrow$ )				COCO-30K
Safety Method	P4D	Ring-A-Bell	MMA-Diffusion	Unlearn-DiffAtk	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )
Original SD	0.585	0.684	0.719	0.458	25.60	32.15
Ours (CLIP)	0.082	0.127	0.042	0.099	35.28	31.53

Appendix

Table 5: Quantitative evaluation on NSFW generation steering on SD-v3.5. ASR evaluates effectiveness on suppressing NSFW content generation. COCO-30K evaluates generation quality of models after safety method is applied.

Safe T2I Generation

Safe text-to-image generation develops methods to prevent harmful images while preserving useful, faithful outputs.

See all 52 papers

Introspective Attention Modulation for Safe Text-to-Image Generation
Jul 16, 2026Basim Azam, Hossein Rahmani, Naveed AkhtarDiffusion Transformer T2I Generation
The University of Melbourne, Australia · Lancaster University, United Kingdom
SafeCtrl: Region-Aware Safety Control for Text-to-Image Diffusion via Detect-Then-Suppress
Apr 5, 2026Lingyun Zhang, Yu Xie, Zhongli Fang +2Safe T2I Generation Adversarial Robustness
School of Computer Science and Technology, Fudan University, China · Institute of Big Data, Fudan University, China · Nanjing University of Information Science and Technology, China
Safe Image Generation via Reinforcement Learning
Oct 5, 2026Eungyeol Han, Jong-Seok LeeT2I Generation Safe T2I Generation
Yonsei University School of Integrated Technology

Jul 16, 2026J/K move · Enter open · S save

Introspective Attention Modulation for Safe Text-to-Image Generation

Basim Azam, Hossein Rahmani, Naveed Akhtar

The University of Melbourne, Australia · Lancaster University, United Kingdom

Diffusion Transformer T2I Generation Safe T2I Generation Attention Control in Diffusion Models Controllable Image Generation T2I Diffusion Models+1

Open paper AskPDF arXiv

State-of-the-art flow based text-to-image (T2I) models exhibit remarkable generative abilities but remain vulnerable to producing unsafe content. Prior safety efforts range from concept erasure and prompt filtering to classifier-based gating. However, simple techniques like parameter efficient adaptations of the models easily bypass such guardrails. We introduce a unique principled approach that achieves safety by regulating the model's attention dynamics through inference-time introspection, exhibiting intrinsic robustness. Our method analyzes and rebalances attention activations throughout image synthesis, steering generations away from unsafe concepts while preserving semantic alignment. This introspective control ensures safety of deployed models. Across standard and adversarial safety benchmarks, our approach achieves remarkable safety scores while maintaining or even improving alignment and perceptual quality. Our results reveal that attention-space regulation offers a considerably more promising path to safer diffusion transformer based image generation than the existing concept erasing mechanism.Our code can be accessed at https://basim-azam.github.io/iam/

Abstract

Safe T2I Generation

Explore similar work

Introspective Attention Modulation for Safe Text-to-Image Generation

SafeCtrl: Region-Aware Safety Control for Text-to-Image Diffusion via Detect-Then-Suppress

Safe Image Generation via Reinforcement Learning