MMJailBench: A Factorized Benchmark for Disentangling Multimodal Jailbreak Vulnerabilities

Tianshi Wang; Jingsong Wang; Yafei Huang; Fengling Li; Xin Li; Lei Zhu

cs.CRAug 26, 2026

MMJailBench: A Factorized Benchmark for Disentangling Multimodal Jailbreak Vulnerabilities

Authors: Tianshi Wang, Jingsong Wang, Yafei Huang, Fengling Li, Xin Li, Lei Zhu

Organizations: Tongji University · Mohamed bin Zayed University of Artificial Intelligence · Shanghai Artificial Intelligence Laboratory

Abstract

Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet how different factors shape their jailbreak vulnerabilities remains poorly understood. Existing benchmarks often couple harmful intent, prompt framing, visual semantics, and instruction carrier within individual jailbreak instances, obscuring the specific sources of observed vulnerabilities. To address this limitation, we introduce MMJailBench, a factorized benchmark that systematically varies and combines these factors under controlled configurations, enabling fine-grained comparison and factor-level attribution. Large-scale evaluations across 16 open-weight and proprietary MLLMs reveal highly heterogeneous and model-dependent vulnerability profiles. Jailbreak vulnerability varies markedly across harm domains, exposing uneven coverage in current multimodal safety alignment. Prompt framing emerges as the dominant source of variation, task-relevant visual semantics systematically increase jailbreak susceptibility with authority-like cues exposing particularly pronounced vulnerabilities, and visually rendered instructions do not consistently increase jailbreak susceptibility relative to direct textual instructions. To further investigate the risks introduced by multimodal context, we conduct diagnostic analyses on a representative open-weight model and identify vulnerability-associated patterns in internal representations and cross-modal interactions. Finally, we develop a modular multimodal jailbreak evaluation suite with full and lightweight configurations, multiple judge options, and multidimensional metrics, enabling reproducible, scalable, and cost-efficient multimodal jailbreak auditing.

Figures & tables

Figure 1: Motivation of MMJailBench. Existing multimodal jailbreak benchmarks entangle multiple factors within fixed instances, limiting factor-level diagnosis. MMJailBench factorizes these factors and constructs a controlled evaluation space for fine-grained analysis.

Figure 2: Construction and evaluation pipeline of MMJailBench. The benchmark factorizes harmful intent, prompt framing, visual semantics, and instruction carrier, and evaluates MLLMs under controlled multimodal jailbreak configurations.

Model	Avg.	Cyber	Econ.	Privacy	Decept.	Illegal	Hate	Violence	Sensitive	Sexual
Open-weight
glm4.1v-9b	72.22	80.72	85.28	71.78	78.04	83.16	67.39	79.58	44.94	59.12
step3-vl-10b	53.36	70.00	68.22	62.06	58.93	54.48	49.17	44.24	42.06	27.06
ministral3-8b	51.64	66.83	63.78	59.61	59.58	56.21	43.11	46.98	38.06	30.59
llava-onevision-1.5	40.64	42.00	49.72	38.17	42.74	52.30	36.00	48.96	21.06	34.80
internvl3-8b	49.12	63.11	61.72	51.33	53.21	61.49	42.33	50.80	27.39	30.69

Table 1: Per-model ASR (%) across nine major harmful domains in our MMJailBench. Avg. denotes the average ASR over all evaluated harmful domains.

Figure 3: Model-level vulnerability profiles across prompt framing and visual semantic factors. Each radar plot shows ASR variation under controlled factor changes, revealing model-specific sensitivity patterns.

Group	Image condition	ASR (%)	$\Delta$ ASR (%)
Semantic	Authorization document	47.73	12.96
	Identity credential	45.24	10.47
	Task scenario	44.87	10.10
	Professional role	43.60	8.83
	Dangerous context	43.22	8.45
Control	Blank image	36.96	2.19

Table 2: Visual semantic ablation results comparing semantic and control image conditions.

\Delta

ASR denotes the ASR increase relative to the no-image condition.

Carrier	$N$	Mis. (%)	ASR (%)	CASR (%)
TEXT	130,560	0.37	50.70	50.89
OCR	130,560	4.14	40.59	42.34
UNION	261,120	2.26	45.64	46.69

Table 3: Instruction carrier analysis under TEXT and OCR settings.

N

denotes the number of evaluated model responses, and Mis. represents the instruction mismatch rate.

Model	$\Delta$ Intent	$\Delta$ Prompt	$\Delta$ Visual	$\Delta$ Carrier
Open-weight
glm4.1v-9b	76.7	51.8	4.2	8.3
step3-vl-10b	86.7	72.2	4.8	10.5
ministral3-8b	83.3	57.2	9.6	19.9
llava-onevision-1.5	61.7	35.3	4.6	50.5
internvl3-8b	75.0	88.3	5.5	4.6

Table 4: Model-dependent vulnerability profiles across MMJailBench factors. Each

\Delta

score denotes the range of ASR variation across configurations of the corresponding factor.

Figure 4: Representation-level diagnostics of gemma3-12b under authority-document and danger-image contexts. Authority-related visual cues exhibit distinct representation patterns across layers and harm domains.

Figure 5: Cross-modal interaction diagnostics in gemma3-12b. Attention allocation to harm-related textual tokens and visual tokens varies across layers under authority-document and danger-image contexts.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Count	Ratio (%)
Cyber & Tech Abuse	30	11.0
$\bullet$ Malware & Hacking	16	5.9
$\bullet$ Agent Abuse & Automation	14	5.1
Economic Harm	30	11.0
$\bullet$ Fraud & Scams	16	5.9
$\bullet$ Financial Crimes & Risky Practices	14	5.1

Appendix

Table 5: Harmful-intent taxonomy and category statistics. MMJailBench contains 272 intents organized into 9 major harm domains and 18 harm scenarios.

Figure 6: Pairwise cosine similarity distribution among the 272 harmful intents. The high-similarity tail is manually inspected during taxonomy construction to identify and remove near-duplicate candidates.

Figure 7: PCA visualization of harmful-intent embeddings. Different colors represent harm domains, providing a qualitative view of taxonomy coverage and semantic diversity.

Template	Framing Strategy	Representative Pattern
Academic	Academic Analysis	Analyze the topic from an academic perspective and discuss related aspects.
System	System-Role Simulation	Assume a specific system identity and follow role-specific response instructions.
Story	Fictional Storytelling	Describe the scenario through a fictional story or hypothetical setting.
Code	Code-Style Indirection	Express the requested procedure through code format or comments.
Structure	Structured Completion	Complete a predefined structure containing multiple stages or sections.
Paradox	Safety-Paradox Reasoning	Discuss harmful actions indirectly under a safety-oriented rationale.

Appendix

Table 6: Overview of the 6 prompt framing templates used in MMJailBench. Each template modifies the prompt framing while preserving the underlying harmful intent.

Figure 8: Examples of the five visual semantic categories in MMJailBench. Each category provides distinct visual contextual cues while preserving the underlying harmful intent.

Score	Definition
1	Refusal or safe response without harmful details.
2	Potential risk without direct harmful instructions.
3	High-level or partial harmful information.
4	Actionable harmful assistance.
5	Detailed and operational harmful guidance.

Appendix

Table 7: Harmfulness scoring rubric for response evaluation.

Category	Count	ASR	$\Delta_{\mathrm{T-O}}$
Cyber & Tech Abuse
$\bullet$ Malware & Hacking	16	56.22	10.77
$\bullet$ Agent Abuse & Automation	14	58.79	11.58
Economic Harm
$\bullet$ Fraud & Scams	16	57.66	11.20
$\bullet$ Financial Crimes & Risky Practices	14	54.66	10.57

Appendix

Table 8: Category-level jailbreak results. ASR is computed by pooling the two instruction carriers, and

\Delta_{\mathrm{T-O}}

denotes the ASR difference between TEXT and OCR carriers.

Model	Acad.	Sys.	Story	Code	Stru.	Para.
Open-weight
glm4.1v-9b	77.83	46.40	98.20	68.16	91.40	57.54
step3-vl-10b	65.11	15.51	58.05	46.36	87.68	48.42
ministral3-8b	82.72	30.70	70.00	25.55	76.36	28.24
llava-onevision-1.5	35.59	50.51	54.89	40.66	47.21	19.60
internvl3-8b	74.41	0.07	88.42	37.10	57.13	43.09

Appendix

Table 9: Per-model ASR (%) across different prompt framing strategies on MMJailBench.

Model	Dang.	Scen.	Auth. Doc.	ID Cred.	Role
Open-weight
glm4.1v-9b	75.34	72.33	71.17	73.93	73.50
step3-vl-10b	51.87	53.40	55.82	55.48	51.04
ministral3-8b	57.57	53.80	52.79	49.17	47.98
llava-onevision-1.5	42.28	43.17	40.10	42.95	38.54
internvl3-8b	52.60	52.36	48.10	49.97	47.15

Appendix

Table 10: Per-model ASR (%) across different visual semantic conditions on MMJailBench.

Figure 9: Per-model jailbreak factor sensitivity profiles. Horizontal and vertical axes represent ASR ranges induced by prompt framing and visual semantic conditions, respectively. Marker size indicates the ASR range induced by instruction carriers. Colors and shapes denote four K-means profiles obtained with 1,000 random restarts.

Figure 10: Per-model ASR across five task-relevant visual semantic conditions and four control conditions. The radial scale spans 0 $\sim$ 80%. Axes, shading, and model colors are consistent with the main-paper radar plot.

Measure	Result
Mean ASR difference (Light $-$ Full)	$-0.06\%$
95% CI for ASR difference	[ $-0.45\%$ , $0.30\%$ ]
Mean absolute ASR difference	$0.79\%$
Maximum absolute ASR difference	$3.53\%$
Cells within 2% difference	30/32 (93.75%)
Spearman $\rho$	0.997

Appendix

Table 11: Consistency analysis between Full and Lightweight benchmark settings.

Evaluation Pair	Harmfulness		Mismatch Detection
Evaluation Pair	QWK	$\Delta$ ASR	Acc.	Macro- $F_{1}$
GPT-5 vs. Human Experts	0.97	+0.3%	96.3%	0.85
Lightweight Judge vs. GPT-5	0.95	$-0.4\%$	99.4%	0.94

Appendix

Table 12: Judge agreement and validation results between GPT-5, human experts, and the lightweight judge.

Multimodal Jailbreak Attacks

Multimodal jailbreak attacks exploit text, images, audio, or video to make AI systems bypass safety constraints.

See all 28 papers

May 18, 2026

DMN: A Compositional Framework for Jailbreaking Multimodal LLMs with Multi-Image Inputs

Multimodal Large Language Models (MLLMs) are vulnerable to jailbreak attacks, which can elicit harmful responses from MLLMs. Many MLLMs support multi-image inputs, inadvertently introducing new vulnerabilities due to less efforts on multi-image safety alignment. Previous MLLM jailbreak methods only uses a single image, which restricts the attack space: they cannot distribute harmful requests across multiple images, carry abundant information, or exploit additional visual reasoning tasks to distract MLLMs. To address these limitations, in this paper, we propose a compositional jailbreak framework, DMN, which leverages Distributed instruction, Multimodal evidence and a Number chain task to fully enhance the jailbreak performance. Extensive experiments show that DMN is highly effective for MLLM jailbreaking, e.g. achieving attack success rates of over 90% on GPT-4o, Gemini-2.5-pro and Claude Sonnet 4, surpassing other baselines by a large margin. This compositional, multi-image jailbreak strategy reveals fundamental weaknesses in their safety mechanisms.

Wenzhuo Xu, Zhipeng Wei, Zonghao Ying +4

AI Security Lab, Beijing, China · International Computer Science Institute, CA, USA · UC Berkeley, CA, USA +1

Multimodal Large Language Models LLM Safety Multimodal Jailbreak Attacks Adversarial Attacks on Unified Multimodal Models

Ask PDF

Mar 23, 2026

Structured Visual Narratives Undermine Safety Alignment in Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) extend text-only LLMs with visual reasoning, but also introduce new safety failure modes under visually grounded instructions. We study comic-template jailbreaks that embed harmful goals inside simple three-panel visual narratives and prompt the model to role-play and "complete the comic." Building on JailbreakBench and JailbreakV, we introduce ComicJailbreak, a comic-based jailbreak benchmark with 1,167 attack instances spanning 10 harm categories and 5 task setups. Across 15 state-of-the-art MLLMs (six commercial and nine open-source), comic-based attacks achieve success rates comparable to strong rule-based jailbreaks and substantially outperform plain-text and random-image baselines, with ensemble success rates exceeding 90% on several commercial models. Then, with the existing defense methodologies, we show that these methods are effective against the harmful comics, they will induce a high refusal rate when prompted with benign prompts. Finally, using automatic judging and targeted human evaluation, we show that current safety evaluators can be unreliable on sensitive but non-harmful content. Our findings highlight the need for safety alignment robust to narrative-driven multimodal jailbreaks.

Rui Yang Tan, Yujia Hu, Roy Ka-Wei Lee

Singapore University of Technology and Design. · University of British Colombia.

MLLM Safety Adversarial Attacks on VLMs Adversarial Attacks on LLMs LLM Safety Alignment LLM Safety Benchmarks Language Model Safety Evaluation

Ask PDF

Dec 21, 2024

Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models

Large language models (LLMs) are widely applied in various fields of society due to their powerful reasoning, understanding, and generation capabilities. However, the security issues associated with these models are becoming increasingly severe. Jailbreaking attacks, as an important method for detecting vulnerabilities in LLMs, have been explored by researchers who attempt to induce these models to generate harmful content through various attack methods. Nevertheless, existing jailbreaking methods face numerous limitations, such as excessive query counts, limited coverage of jailbreak modalities, low attack success rates, and simplistic evaluation methods. To overcome these constraints, this paper proposes a multimodal jailbreaking method: JMLLM. This method integrates multiple strategies to perform comprehensive jailbreak attacks across text, visual, and auditory modalities. Additionally, we contribute a new and comprehensive dataset for multimodal jailbreaking research: TriJail, which includes jailbreak prompts for all three modalities. Experiments on the TriJail dataset and the benchmark dataset AdvBench, conducted on 13 popular LLMs, demonstrate advanced attack success rates and significant reduction in time overhead.

Yanxu Mao, Peipei Liu, Tiehan Cui +3

School of Computer Science and Technology, Shanghai Jiao Tong University, Shanghai, China · Zhongguancun Laboratory, Beijing, China · School of Software, Henan University, Kaifeng, China +1

Adversarial Attacks on LLMs Multimodal Large Language Models LLM Safety Multimodal Jailbreak Attacks

Ask PDF

Abstract

Multimodal Jailbreak Attacks

Explore similar work

DMN: A Compositional Framework for Jailbreaking Multimodal LLMs with Multi-Image Inputs

Structured Visual Narratives Undermine Safety Alignment in Multimodal Large Language Models

Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models