cs.CVFeb 24, 2026
SaveWhen Safety Collides: Resolving Multi-Category Harmful Conflicts in Text-to-Image Diffusion via Adaptive Safety Guidance
Organizations: Sydney AI Centre, The University of Sydney · City University of Hong Kong · TMLR Group, Hong Kong Baptist University
Abstract
Text-to-Image (T2I) diffusion models have demonstrated significant advancements in generating high-quality images, while raising potential safety concerns regarding harmful content generation. Safety-guidance-based methods have been proposed to mitigate harmful outputs by steering generation away from harmful zones, where the zones are averaged across multiple harmful categories based on predefined keywords. However, these approaches fail to capture the complex interplay among different harm categories, leading to "harmful conflicts" where mitigating one type of harm may inadvertently amplify another, thus increasing overall harmful rate. To address this issue, we propose Conflict-aware Adaptive Safety Guidance (CASG), a training-free framework that dynamically identifies and applies the category-aligned safety direction during generation. CASG is composed of two components: (i) Conflict-aware Category Identification (CaCI), which identifies the harmful category most aligned with the model's evolving generative state, and (ii) Conflict-resolving Guidance Application (CrGA), which applies safety steering solely along the identified category to avoid multi-category interference. CASG can be applied to both latent-space and text-space safeguards. Experiments on T2I safety benchmarks demonstrate CASG's state-of-the-art performance, reducing the harmful rate by up to 15.4% compared to existing methods.
Figures & tables
Figure 1: We demonstrate the safety performance of SLD on different harmful keywords and analyze the harmful conflicts. (a) shows SLD effectively steers the prompt guidance away from the harmful zone when harmful keywords precisely match the prompt’s harmful category ( sex ). (b) illustrates keyword mismatch scenarios where harmful conflicts arise when attempting to steer away from the hate harmful zone while inadvertently moving toward the sexual harmful zone. (c) demonstrates the performance degradation when applying multiple-categories keywords hate, sexual . More analysis are presented in Section 3 .
Figure 2: Cross-Category Directional Conflict in latent space. Each arrow represents a category-wise safety direction projected into the top three PCA dimensions. Directions from different categories intersect or oppose one another, and these relationships evolve across timesteps, indicating dynamic harmful conflicts.
Figure 3: Aggregated Directional Attenuation in latent space. The horizontal axis shows diffusion timesteps, and the vertical axis lists harmful categories. Color intensity indicates category-wise directional retention (darker means higher retention). The fluctuating patterns reveal strong cross-category attenuation. More results are shown in Appendix A.2 .
| Safeguard Category | Harmful Rate on sexual prompt | Harmful Rate on violence prompt |
| Baseline (SD 1.5) | 67.2 | 52.8 |
| Safety Misalignment Degradation | ||
| hate | 72.4 (+5.2) | 34.6 (-18.2) |
| sexual content | 3.2 (-64.0) | 26.4 (-26.4) |
| violence | 59.4 (-7.8) | 6.2 (-46.6) |
| illegal activity | 64.6 (-2.6) | 18.6 (-34.2) |
Table 1: Safety Degradation. Harmful Rates (%) under SLD when applying different harmful categories. Values in brackets denote the change relative to the baseline, and the best safety performance is underlined. More result shown in Appendix A.3 .
Figure 4: Overview of Conflict-aware Adaptive Safety Guidance: CASG identifies the harmful category most aligned with the current state and applies safety guidance specifically along that category to mitigate harmful conflict. In text space, alignment is estimated via the residual magnitude after orthogonal projection of the prompt embedding; in latent space, by measuring the angle between harmful and prompt guidance directions.
| Method | Conflict- Aware | Harmful Rate % | FID | CLIP | |||
| I2P | T2VSafetyBench | UD | CoProv2 | COCO | |||
| SD-v1.5 [ 38 ] | - | 42.2 | 58.3 | 52.3 | 28.2 | - | 31.43 |
| ESD [ 12 ] | 42.0 (-0.2) | 57.4 (-0.9) | 50.6 (-1.7) | 28.1 (-0.1) | 38.15 | 31.35 | |
| UCE [ 13 ] | 26.7 (-15.5) | 28.2 (-30.1) | 33.0 (-19.3) | 19.4 (-8.8) | 77.41 | 29.12 | |
| RECE [ 14 ] | 21.5 (-20.7) | 18.9 (-39.4) | 22.3 (-30.0) | 8.6 (-19.6) | 67.35 | 27.67 | |
| SafetyDPO [ 28 ] | 13.7 (-28.5) | 24.0 (-34.3) | 16.7 (-35.6) | 4.2 (-24.0) | 49.64 | 30.61 | |
Table 2: Comparison of text-to-image safeguard methods. Harmful rates ( , lower is better; brackets show change relative to SDv1.5) are evaluated on four benchmarks. Image quality on COCO is measured by FID ( , lower is better) and CLIP score ( , higher is better). Methods requiring model modification are shown in gray ; the best results are in bold .
Figure 5: Comparison of T2I safety methods across different categories of harmful content. The rows show generation results for prompts related to violence and inappropriate content. Methods marked with * require parameter tuning or model modifications.
| Method | I2P | T2VSafetyBench | UD |
| SLD | 12.7 | 25.2 | 15.7 |
| GPT+SLD | 11.6 | 12.3 | 20.1 |
| QwenGuard+SLD | 14.0 | 21.1 | 23.3 |
| CASG+SLD | 10.2 | 9.8 | 9.8 |
Table 3: Comparison of CASG+SLD and LLM-assisted SLD variants. We present the harmful rate to measure the safety performance, where lower values indicate better safety performance.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Cross-Category Directional Conflict in latent space under sexual and hate prompts. Each arrow represents a category-wise safety direction projected into the top three PCA dimensions. Directions from different categories intersect or oppose one another, and these relationships evolve across timesteps, indicating dynamic harmful conflicts.
Figure 7: Cross-Category Directional Conflict in text space under sexual and hate prompts. Each arrow represents a category-wise safety direction projected into the top three PCA dimensions. Directions from different categories intersect or oppose one another.
Figure 8: Aggregated Directional Attenuation in latent space. The horizontal axis shows diffusion timesteps, the vertical axis lists harmful categories. Color intensity indicates category-wise directional retention (CDRR, darker means higher retention
| Default Keywords | Coarse Keywords | Fine-Grained Keywords | ||||
| Safeguard Category | Prompt with sexual | Prompt with violence prompt | Prompt with sexual | Prompt with violence prompt | Prompt with sexual | Prompt with violence prompt |
| Baseline (SDv1.5 [ 38 ] ) | 67.2 | 52.8 | 67.2 | 52.8 | 67.2 | 52.8 |
| Safety Misalignment Degradation | ||||||
| hate | 72.4 (+5.2) | 34.6 (-18.2) | 72.7 (+5.5) | 34.6 (-18.2) | 67.4 (+0.2) | 20.4 (-32.4) |
| sexual | 3.2 (-64.0) | 26.4 (-26.4) | 12.2 (-55.0) | 34.8 (-18.0) | 2.8 (-64.4) | 24.6 (-28.2) |
| violence | 59.4 (-7.8) | 6.2 (-46.6) | 32.9 (-34.3) | 12.2 (-40.6) | 58.2 (-9.0) | 5.8 (-47.0) |
Table 4: Safety degradation under three harmful keyword settings : Default (20 keywords), Coarse (7 keywords), and Fine-grained (42 GPT-generated keywords). We report harmful rates on sexual and violence prompts. Values in brackets denote the change relative to the baseline, and the best safety performance is underlined.
| Safeguard Category | Harmful Rate on sexual prompt | Harmful Rate on violence prompt |
| Baseline (SDv1.5) | 67.2 | 52.8 |
| Safety Misalignment Degradation | ||
| hate | 72.6 (+5.4) | 41.0 (-11.8) |
| sexual content | 32.6 (-34.6) | 34.6 (-18.2) |
| violence | 68.8 (+1.6) | 22.0 (-30.8) |
| illegal activity | 67.6 (+0.4) | 31.0 (-21.8) |
Table 5: Safety degradation under SAFREE (text-space safeguard) when applying different harmful categories. Values in brackets denote the change relative to the baseline, and the best safety performance is underlined.
| Safeguard Category | Harmful Rate on sexual prompt | Harmful Rate on violence prompt |
| Baseline (SDv3 [ 10 ] ) | 51.0 | 49.6 |
| Safety Misalignment Degradation | ||
| hate | 50.2 (-0.8) | 30.4 (-19.2) |
| sexual content | 7.8 (-43.2) | 33.0 (-16.6) |
| violence | 40.4 (-10.6) | 22.8 (-26.8) |
| illegal activity | 43.8 (-7.2) | 33.4 (-16.2) |
Table 6: Safety degradation under SDv3 (DiT-based structure) when applying different harmful categories. Values in brackets denote the change relative to the baseline, and the best safety performance is underlined.
| Method | Conflict- Aware | Harmful Rate % | FID | CLIP | |||
| I2P | T2VSafetyBench | UD | CoProv2 | COCO | |||
| SD-v1.5 [ 38 ] | - | 44.9 | 59.5 | 54.2 | 27.0 | - | 31.52 |
| ESD [ 12 ] | 42.3 (-2.6) | 57.2 (-2.3) | 50.6 (-3.2) | 26.3 (-0.7) | 39.38 | 31.39 | |
| UCE [ 13 ] | 28.4 (-16.5) | 29.2 (-30.3) | 31.1 (-23.1) | 20.7 (-6.3) | 78.72 | 29.09 | |
| RECE [ 14 ] | 21.0 (-23.9) | 19.9 (-39.6) | 24.3 (-29.9) | 9.4 (-17.6) | 65.94 | 28.11 | |
| SafetyDPO [ 28 ] | 16.2 (-28.7) | 25.2 (-34.3) | 19.2 (-35.0) | 5.5 (-21.5) | 51.20 | 30.59 | |
Table 7: Comparison of text-to-image safeguard methods (guidance scale = 10). Harmful rates ( , lower is better; brackets show change relative to SDv1.5) are evaluated on four benchmarks. Image quality on COCO is measured by FID ( , lower is better) and CLIP score ( , higher is better). Methods requiring model modification are shown in gray ; the best results are in bold .
| Keyword set | SAFREE | CASG+SAFREE | SLD | CASG+SLD |
| Default | 41.5 | 37.5 (-4.0) | 25.2 | 9.8 (-15.4) |
| Synonyms | 36.2 | 32.3 (-3.9) | 14.8 | 10.6 (-4.2) |
| Abstract | 49.4 | 40.7 (-8.7) | 26.9 | 18.9 (-8.0) |
| Detailed | 41.1 | 36.8 (-4.3) | 19.1 | 7.9 (-11.2) |
Table 8: ASR (%) under different predefined harmful keyword variants on T2VSafetyBench. Values in brackets denote the change relative to the base safeguard.
| Category | CASG+SAFREE | CASG+SLD |
| hate | 15.6 (-5.5) | 10.8 (-16.9) |
| harassment | 15.3 (-0.7) | 10.1 (-1.0) |
| violence | 19.2 (-9.4) | 16.8 (-14.3) |
| self-harm | 14.0 (-10.3) | 6.4 (-19.0) |
| sexual content | 22.4 (-9.3) | 6.1 (-61.2) |
| shocking | 29.7 (-3.9) | 15.2 (-13.6) |
Table 9: We present category-wise harmful rate analysis of the harmful rate reduction in I2P datasets. Values in brackets denote the relative drop compared with SAFREE and SLD.
Figure 9: Harmful cosine in latent space with sexual-related prompt. The horizontal axis shows diffusion timesteps, and the vertical axis lists harmful categories. Color intensity indicates harmful cosine (darker means stronger harmfulness
Figure 10: Comparison of T2I safety methods using prompts with different harmful content. Methods marked with * require parameter tuning or model modifications.
| Method | Inference Time (s/sample) |
| SAFREE | 3.9 |
| CASG+SAFREE | 3.98 (1.02 ) |
| SLD | 4.0 |
| CASG+SLD (k=2) | 5.0 (1.25 ) |
| CASG+SLD (k=3) | 6.2 (1.55 ) |
| CASG+SLD (k=4) | 7.1 (1.78 ) |
Table 10: Inference Efficiency Comparison. denotes the number of predefined harmful categories. Values in brackets denote the inference time multiples relative to SAFREE and SLD.
Figure 11: Qualitative comparison between CASG and the base safeguard on benign prompts (COCO).
More in this topic
Safe T2I Generation
Safe text-to-image generation develops methods to prevent harmful images while preserving useful, faithful outputs.
See all 53 papers