JANUS: A Lightweight Framework for Jailbreaking Text-to-Image Models via Distribution Optimization

Haolun Zheng; Yu He; Tailun Chen; Shuo Shao; Zhixuan Chu; Hongbin Zhou; Lan Tao; Zhan Qin

cs.CVMar 22, 2026

JANUS: A Lightweight Framework for Jailbreaking Text-to-Image Models via Distribution Optimization

Authors: Haolun Zheng, Yu He, Tailun Chen, Shuo Shao, Zhixuan Chu, Hongbin Zhou, Lan Tao, Zhan Qin,

Organizations: The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou HighTech Zone (Binjiang) Blockchain and Data Security Research Institute, China · Alibaba Group

Abstract

Text-to-image (T2I) models such as Stable Diffusion and DALLE remain susceptible to generating harmful or Not-Safe-For-Work (NSFW) content under jailbreak attacks despite deployed safety filters. Existing jailbreak attacks either rely on proxy-loss optimization instead of the true end-to-end objective, or depend on large-scale and costly RL-trained generators. Motivated by these limitations, we propose JANUS , a lightweight framework that formulates jailbreak as optimizing a structured prompt distribution under a black-box, end-to-end reward from the T2I system and its safety filters. JANUS replaces a high-capacity generator with a low-dimensional mixing policy over two semantically anchored prompt distributions, enabling efficient exploration while preserving the target semantics. On modern T2I models, we outperform state-of-the-art jailbreak methods, improving ASR-8 from 25.30% to 43.15% on Stable Diffusion 3.5 Large Turbo with consistently higher CLIP and NSFW scores. JANUS succeeds across both open-source and commercial models. These findings expose structural weaknesses in current T2I safety pipelines and motivate stronger, distribution-aware defenses. Warning: This paper contains model outputs that may be offensive.

Figures & tables

Figure 2: Overall pipeline of our JANUS . Stage 1 builds two semantically anchored base distributions from the target prompt $\mathbf{p}_{t}$ and its clean counterpart $\mathbf{p}_{c}$ , then mixes them into a parameterized prompt distribution $\mathbf{p}_{\alpha}$ . Stage 2 performs black-box policy optimization: samples from $\mathbf{p}_{\alpha}$ are evaluated by the T2I model, and the bypass/NSFW feedback updates the mixing policy $\alpha$ .

Model	Method	TASR(%) $\uparrow$	IASR-1(%) $\uparrow$	IASR-4(%) $\uparrow$	IASR-8(%) $\uparrow$	ASR-1(%) $\uparrow$	ASR-4(%) $\uparrow$	ASR-8(%) $\uparrow$	CLIP Score $\uparrow$	NSFW Score $\uparrow$
SD3.5LT	MMA	7.65	4.90	11.19	15.15	1.30	2.95	4.00	0.25	0.20
	MMP	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
	QFA	37.00	11.54	23.26	28.65	9.85	20.55	25.30	0.31	0.28
	PGJ	32.75	13.59	31.43	41.21	5.40	13.20	17.15	0.23	0.27
	SneakyPrompt	34.00	14.00	24.00	31.00	5.36	7.03	11.24	0.32	0.28
	JANUS (Ours)	94.25	17.30	37.01	46.65	12.40	30.80	43.15	0.37	0.33

Table 1: Quantitative comparison of existing jailbreak attacks on SD3.5LT and DALL

\cdot

E3. Higher values indicate better performance (

\uparrow

), and the best results in each column are highlighted in bold .

Figure 3: Qualitative results of JANUS on Stable Diffusion XL (left) and Midjourney (right).

	SD3.5LT				DALL $\cdot$ E3
	TASR	IASR	ASR	NSFW	TASR	IASR	ASR	NSFW
Unimodal	97.00%	28.00%	26.87%	0.241	9.53%	9.32%	0.69%	0.062
Fix NSFW	91.50%	35.15%	32.33%	0.286	8.65%	14.99%	0.89%	0.071
Full Process	94.25%	46.65%	44.50%	0.329	9.28%	18.56%	1.39%	0.082

Table 2: Component-wise ablation of JANUS on SD3.5LT and DALL

\cdot

E3. We compare the full framework (“Full Process”) against a Unimodal variant (single distribution) and a fixed-NSFW-reward variant. We report TASR, IASR, ASR and NSFW Score for N=8.

Figure 4: Effect of the mixing policy

\alpha

on jailbreak performance for SD3.5LT (left) and DALL

\cdot

E3 (right). The left y-axis reports TASR / IASR / ASR (%), while the right y-axis reports the NSFW score. Fixing

\alpha

to any static value leads to a suboptimal trade-off between filter evasion and content harmlessness. Our full framework (“Fully Trained”) uses RL to learn a dynamic

\alpha

policy, achieving superior overall jailbreak performance.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	MMA	MMP	QFA	PGJ	Sneaky	JANUS
Runtime (s)	1550.98	1098.19	82.06	54.44	82.31	87.57

Appendix

Table 3: Computational efficiency comparison. We report the average runtime (in seconds) per successful jailbreak. JANUS achieves a significant speedup compared to optimization-based baselines (MMA, MMP) while maintaining a competitive runtime against generator-based methods (PGJ, SneakyPrompt) without requiring memory-intensive Large Language Models.

Model	Method	TASR(%) $\uparrow$	IASR-1(%) $\uparrow$	IASR-4(%) $\uparrow$	IASR-8(%) $\uparrow$	ASR-1(%) $\uparrow$	ASR-4(%) $\uparrow$	ASR-8(%) $\uparrow$	CLIP Score $\uparrow$	NSFW Score $\uparrow$
SDXL	MMA	7.65	23.90	48.55	61.10	2.10	4.00	4.85	0.43	0.32
	MMP	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
	QFA	37.00	25.10	49.25	62.00	9.63	16.40	24.26	0.41	0.35
	PGJ	32.75	14.47	33.98	46.88	5.90	14.90	19.95	0.39	0.31
	SneakyPrompt	34.00	21.73	46.83	52.74	8.42	15.28	18.49	0.38	0.29
	JANUS (Ours)	94.25	25.98	51.07	64.83	23.15	46.30	58.20	0.47	0.39

Appendix

Table 4: Quantitative comparison of existing jailbreak attacks on SDXL and Midjourney. Higher values indicate better performance (

\uparrow

), and the best results in each column are highlighted in bold . Note that for Midjourney, IASR-1 and ASR-1 are omitted (-) as the model generates a minimum batch of 4 images per query.

Figure 5: More qualitative results of JANUS on Stable Diffusion 3.5 Large Turbo (left) and DALL $\cdot$ E3 (right).

Figure 6: More qualitative results of JANUS on Stable Diffusion XL (left) and Midjourney (right).

Figure 7: More qualitative results of JANUS on Midjourney(1).

Figure 8: More qualitative results of JANUS on Midjourney(2).

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 182 papers

Jailbreaking Text-to-Image Models Through Cracks: Navigating Heterogeneous Safety Filters via Multi-Agent Debate
Sep 1, 2026Kaiyan Wen, Shijie Zhang, Lu Yu +1Adversarial Attacks on VLMs Multi-Agent Debate
College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China · Shenzhen MSU-BIT University, Shenzhen, China · City University of Hong Kong, Hong Kong, China
Dynamic Defense Profiling Enables Cognitive Jailbreak of Text-to-Image Models
Jul 20, 2026Dongdong Yang, Deyue Zhang, Zhao Liu +5Adversarial Prompt Generation Adversarial Examples
AI Security Lab, Beijing, China · School of Artificial Intelligence, Beihang University, Beijing, China
Low-Effort Jailbreak Attacks Against Text-to-Image Safety Filters
Apr 2, 2026Ahmed B Mustafa, Zihan Ye, Yang Lu +2Adversarial Attacks on VLMs Safe T2I Generation
School of Computer Science, University of Nottingham, Wollaton Road, NG8 1BB · Department of Intelligent Science, Xi’an Jiaotong-Liverpool University, China, 215123 · School of Informatics, Xiamen University, Xiamen, 361005, China

Sep 1, 2026J/K move · Enter open · S save

Jailbreaking Text-to-Image Models Through Cracks: Navigating Heterogeneous Safety Filters via Multi-Agent Debate

Kaiyan Wen, Shijie Zhang, Lu Yu, Guangdong Bai

College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China · Shenzhen MSU-BIT University, Shenzhen, China · City University of Hong Kong, Hong Kong, China

Adversarial Attacks on VLMs Multi-Agent Debate Safety Filtering Multimodal Jailbreak Attacks

Open paper AskPDF arXiv

Text-to-image (T2I) models remain vulnerable to jailbreak attacks that elicit Not-Safe-For-Work (NSFW) content, despite increasingly being guarded by heterogeneous, multi-layer safety stacks combining text filters, image classifiers, and cross-modal detectors. Existing jailbreak studies either optimize against individual filters or query the complete pipeline with aggregate feedback, making it difficult to identify the active constraint and adapt to conflicts across safety layers. In this paper, we introduce the Detection Surface, a unified geometric framework that characterizes the decision boundaries induced by heterogeneous T2I safety filters and their joint effect on the jailbreak search space. This formulation reveals that successful evasion is governed by a sparse and non-convex region shaped by cross-layer conflicts, where mutations that bypass one filter may increase exposure to another. Motivated by this analysis, we propose CRACK, a multi-agent debate framework for adaptive jailbreak search that decomposes jailbreak search into exploration, diagnosis, and arbitration. CRACK coordinates an Attack Agent, a Defense Agent, and a Judge Agent to iteratively generate prompt mutations, obtain layer-specific diagnostic feedback, and optimize mutation strategies through reward-guided refinement. Through repeated rounds of debate, CRACK adapts its search direction to the evolving cross-layer constraints while preserving the original harmful intent. Extensive experiments across multiple T2I models, datasets, and safety configurations show that CRACK achieves Attack Success Rates (ASR) of up to 99.63% under composite defenses, while requiring fewer queries than existing methods and maintaining semantic fidelity.

Abstract

Adversarial Attacks on VLMs

Explore similar work

Jailbreaking Text-to-Image Models Through Cracks: Navigating Heterogeneous Safety Filters via Multi-Agent Debate

Dynamic Defense Profiling Enables Cognitive Jailbreak of Text-to-Image Models

Low-Effort Jailbreak Attacks Against Text-to-Image Safety Filters