Text-to-Image (T2I) diffusion models have demonstrated strong generation ability, but their potential to generate unsafe content raises significant safety concerns. Existing inference-time defense methods typically perform category-agnostic token-level intervention in the text embedding space, which fails to capture malicious semantics distributed across the full token sequence and remains vulnerable to adversarial prompts. In this paper, we propose DTVI, a dual-stage inference-time defense framework for safe T2I generation. Unlike existing methods that intervene on specific token embeddings, our method introduces category-aware sequence-level intervention on the full prompt embedding to better capture distributed malicious semantics, and further attenuates the remaining unsafe influences during the visual generation stage. Experimental results on real-world unsafe prompts, adversarial prompts, and multiple harmful categories show that our method achieves effective and robust defense while preserving reasonable generation quality on benign prompts, obtaining an average Defense Success Rate (DSR) of 94.43% across sexual-category benchmarks and 88.56 across seven unsafe categories, while maintaining generation quality on benign prompts.
Figures & tables
Figure 1. Qualitative results of DTVI across seven unsafe categories. The top row shows images generated by the undefended model, and the bottom row shows the corresponding outputs after applying DTVI. Our method effectively suppresses unsafe visual content across diverse harmful categories while preserving the overall compositional structure of the generated images.
Figure 2. Overall pipeline of DTVI. Given a user prompt, it performs a dual-stage inference-time intervention to suppress unsafe generation while preserving benign intent. In the textual stage, the prompt embedding is purified by removing unsafe semantic components and steering it away from the unsafe region. In the visual stage, unsafe visual tendencies are further suppressed during denoising by reducing feature responses aligned with the visual steering direction, thus improving safety with minimal disruption to benign generation.
Method
I2P-Sexual ↑
Adversarial Prompts
Avg. ↑
COCO
SneakyPrompt ↑
Ring-A-Bell ↑
MMA-Diffusion ↑
P4D ↑
CLIP ↑
FID ↓
LPIPS ↓
SD-V1.5
-
-
-
-
-
-
31.32
-
-
ESD
-12.33%
31.25%
1.52%
5.62%
7.14%
7.92%
30.94
14.37
0.33
UCE
8.00%
40.00%
16.24%
16.25%
-4.35%
14.98%
30.77
30.89
0.60
SafeGen
-37.64%
37.04%
N/A
78.85%
29.82%
27.02%
31.19
17.04
0.46
SLD-MAX
84.51%
87.50%
50.00%
43.12%
52.25%
66.85%
29.28
24.87
0.53
Table 1. Defense Success Rate (DSR) and generation quality comparison with baselines. Notably, our experiments cover seven unsafe categories including hate, harassment, violence, self-harm, sexual, shocking, illegal activity. For fair comparison with SafeGen and SafeRedir, whose publicly available checkpoints only cover the sexual category, we report results on the sexual subset of I2P (I2P-Sexual). Ring-A-Bell contains violence-category adversarial prompts and is therefore not applicable to these two methods. Avg. is computed as the macro-average over the shared unsafe benchmarks, i.e., I2P-Sexual, SneakyPrompt, MMA-Diffusion, and P4D, all of which belong to the sexual category.
Method
Hate ↑
Harassment ↑
Violence ↑
Self-harm ↑
Sexual ↑
Shocking ↑
Illegal Activity ↑
Avg. ↑
ESD
37.74%
-2.86%
3.95%
1.75%
-12.33%
-3.45%
15.38%
5.74%
UCE
41.18%
8.57%
24.00%
31.58%
8.00%
20.69%
14.81%
21.26%
SLD-MAX
79.25%
68.57%
75.00%
82.14%
84.51%
64.29%
81.48%
76.46%
DTVI (Ours)
100.00%
77.78%
89.33%
82.46%
94.74%
82.76%
92.86%
88.56%
Table 2. DSR comparison across seven unsafe categories with baselines applicable to multi-category defense. We reproduce ESD using the official code by erasing seven unsafe concepts, including hate, harassment, violence, self-harm, sexual, shocking, and illegal activity. UCE follows the same setting as ESD, and SLD uses the official implementation with the MAX hyperparameter setting.
Method
Nn
Nu
Nu/Nn
ESD
68
45
66%
UCE
20
12
60%
SafeGen
36
20
56%
Table 3. Manual inspection results on negative-DSR samples. Nn denotes the number of inspected negative-DSR samples, Nu denotes the number of cases judged unsafe after defense.
Figure 3. Some examples of negative defense cases on ESD, UCE, and SafeGen. In these cases, the defended models generate images with more explicit unsafe content than the undefended model, leading to negative DSR values.
Figure 4. Parameter sensitivity analysis. Effect of varying λ (left) and ϵf (right) on defense performance (DSR, left axis) and image quality (FID, right axis). The higher DSR indicates stronger defense, lower FID indicates better image fidelity.
Figure 5. We use NudeNet ( notAI-tech, 2024 ) to obtain average nude scores under single-module ablations on three representative unsafe benchmarks. Both modules reduce unsafe visual content compared with the undefended setting.
Method
T
V
I2P-Sexual ↑
Adversarial Prompts
Avg. ↑
COCO
MMA-Diffusion ↑
P4D ↑
CLIP ↑
FID ↓
LPIPS ↓
SD-V1.5
✗
✗
-
-
-
-
31.32
-
-
DTVI (Ours)
✓
✗
80.00%
55.62%
26.96%
54.19%
30.86
19.30
0.40
✗
✓
73.97%
90.00%
68.75%
77.57%
31.03
12.66
0.26
✓
✓
94.74%
97.50%
86.73%
92.99%
30.66
20.54
0.43
Table 4. Ablation studies on DTVI. T denotes the textual intervention and V denotes the visual intervention.
Figure 6. Qualitative comparison in the ablation studies. Columns correspond to the undefended model, textual-only intervention, visual-only intervention, and joint textual-visual intervention. The joint setting produces safer and less explicit outputs than using either intervention alone.