JANUS: A Lightweight Framework for Jailbreaking Text-to-Image Models via Distribution Optimization
Organizations: The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou HighTech Zone (Binjiang) Blockchain and Data Security Research Institute, China · Alibaba Group
Abstract
Text-to-image (T2I) models such as Stable Diffusion and DALLE remain susceptible to generating harmful or Not-Safe-For-Work (NSFW) content under jailbreak attacks despite deployed safety filters. Existing jailbreak attacks either rely on proxy-loss optimization instead of the true end-to-end objective, or depend on large-scale and costly RL-trained generators. Motivated by these limitations, we propose JANUS , a lightweight framework that formulates jailbreak as optimizing a structured prompt distribution under a black-box, end-to-end reward from the T2I system and its safety filters. JANUS replaces a high-capacity generator with a low-dimensional mixing policy over two semantically anchored prompt distributions, enabling efficient exploration while preserving the target semantics. On modern T2I models, we outperform state-of-the-art jailbreak methods, improving ASR-8 from 25.30% to 43.15% on Stable Diffusion 3.5 Large Turbo with consistently higher CLIP and NSFW scores. JANUS succeeds across both open-source and commercial models. These findings expose structural weaknesses in current T2I safety pipelines and motivate stronger, distribution-aware defenses. Warning: This paper contains model outputs that may be offensive.
Figures & tables
| Model | Method | TASR(%) | IASR-1(%) | IASR-4(%) | IASR-8(%) | ASR-1(%) | ASR-4(%) | ASR-8(%) | CLIP Score | NSFW Score |
|---|---|---|---|---|---|---|---|---|---|---|
| SD3.5LT | MMA | 7.65 | 4.90 | 11.19 | 15.15 | 1.30 | 2.95 | 4.00 | 0.25 | 0.20 |
| MMP | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | |
| QFA | 37.00 | 11.54 | 23.26 | 28.65 | 9.85 | 20.55 | 25.30 | 0.31 | 0.28 | |
| PGJ | 32.75 | 13.59 | 31.43 | 41.21 | 5.40 | 13.20 | 17.15 | 0.23 | 0.27 | |
| SneakyPrompt | 34.00 | 14.00 | 24.00 | 31.00 | 5.36 | 7.03 | 11.24 | 0.32 | 0.28 | |
| JANUS (Ours) | 94.25 | 17.30 | 37.01 | 46.65 | 12.40 | 30.80 | 43.15 | 0.37 | 0.33 |
| SD3.5LT | DALL E3 | |||||||
|---|---|---|---|---|---|---|---|---|
| TASR | IASR | ASR | NSFW | TASR | IASR | ASR | NSFW | |
| Unimodal | 97.00% | 28.00% | 26.87% | 0.241 | 9.53% | 9.32% | 0.69% | 0.062 |
| Fix NSFW | 91.50% | 35.15% | 32.33% | 0.286 | 8.65% | 14.99% | 0.89% | 0.071 |
| Full Process | 94.25% | 46.65% | 44.50% | 0.329 | 9.28% | 18.56% | 1.39% | 0.082 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | MMA | MMP | QFA | PGJ | Sneaky | JANUS |
|---|---|---|---|---|---|---|
| Runtime (s) | 1550.98 | 1098.19 | 82.06 | 54.44 | 82.31 | 87.57 |
| Model | Method | TASR(%) | IASR-1(%) | IASR-4(%) | IASR-8(%) | ASR-1(%) | ASR-4(%) | ASR-8(%) | CLIP Score | NSFW Score |
|---|---|---|---|---|---|---|---|---|---|---|
| SDXL | MMA | 7.65 | 23.90 | 48.55 | 61.10 | 2.10 | 4.00 | 4.85 | 0.43 | 0.32 |
| MMP | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | |
| QFA | 37.00 | 25.10 | 49.25 | 62.00 | 9.63 | 16.40 | 24.26 | 0.41 | 0.35 | |
| PGJ | 32.75 | 14.47 | 33.98 | 46.88 | 5.90 | 14.90 | 19.95 | 0.39 | 0.31 | |
| SneakyPrompt | 34.00 | 21.73 | 46.83 | 52.74 | 8.42 | 15.28 | 18.49 | 0.38 | 0.29 | |
| JANUS (Ours) | 94.25 | 25.98 | 51.07 | 64.83 | 23.15 | 46.30 | 58.20 | 0.47 | 0.39 |
Adversarial Attacks on VLMs
Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.
See all 182 papers