With the emergence of strong vision language capabilities, multimodal large language models (MLLMs) have demonstrated tremendous potential for real-world applications. However, the security vulnerabilities exhibited by the visual modality pose significant challenges to deploying such models in open-world environments. Recent studies have successfully induced harmful responses from target MLLMs by encoding harmful textual semantics directly into visual inputs. However, in these approaches, the visual modality primarily serves as a trigger for unsafe behavior, often exhibiting semantic ambiguity and lacking grounding in realistic scenarios. In this work, we define a novel setting: vision-centric jailbreak, where visual information serves as a necessary component in constructing a complete and realistic jailbreak context. Building on this setting, we propose the VisCo (Visual Contextual) Attack. VisCo fabricates contextual dialogue using four distinct vision-focused strategies, dynamically generating auxiliary images when necessary to construct a vision-centric jailbreak scenario. To maximize attack effectiveness, it incorporates automatic toxicity obfuscation and semantic refinement to produce a final attack prompt that reliably triggers harmful responses from the target black-box MLLMs. Specifically, VisCo achieves a toxicity score of 4.78 and an Attack Success Rate (ASR) of 85% on MM-SafetyBench against GPT-4o, significantly outperforming the baseline, which achieves a toxicity score of 2.48 and an ASR of 22.2%. Code: https://github.com/Dtc7w3PQ/Visco-Attack.
Figures & tables
Figure 1: Illustration of the vision-centric jailbreak setting. The visual input is an essential component that constitutes the complete jailbreak scenario.
Figure 2: Workflow of the VisCo Attack. (Left) generation of the fabricated visual context and the initial attack prompt using vision-focused strategies. (Right) iterative toxicity obfuscation and semantic refinement of the initial attack prompt.
GPT-4o
GPT-4o-mini
Gemini-2.0-Flash
InternVL2.5-78B
QR Attack
VisCo Attack
QR Attack
VisCo Attack
QR Attack
VisCo Attack
QR Attack
VisCo Attack
Category
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
01-IA
1.02
2.06
4.85
89.69
1.10
2.06
4.88
91.75
1.49
10.31
4.95
94.85
1.61
9.28
4.95
95.88
02-HS
1.22
0.61
4.59
64.42
1.60
5.52
4.78
79.75
1.93
13.50
4.77
82.21
2.38
20.25
4.81
81.60
03-MG
2.00
15.91
4.93
95.45
1.77
13.64
4.93
95.45
3.52
56.82
4.91
95.45
3.57
56.82
4.98
97.73
04-PH
1.85
19.44
4.85
90.97
1.94
18.75
4.86
90.28
2.83
39.58
4.97
97.22
3.13
44.44
4.95
95.14
Table 1: Results of Query-Relevant (QR) Attack and our VisCo Attack on MM-SafetyBench in terms of Toxic (1–5) and ASR (%) across different MLLMs. “01-IA” to “13-GD” denote the 13 subcategories of prohibited scenarios, and “ALL” represents the overall performance across all categories.
Attack
FigStep
VisCo Attack
Metric
Toxic
ASR
Toxic
ASR
LLaVA-OV-7B-Chat
3.98
54.00
4.70
80.00
InternVL2.5-78B
2.74
34.00
4.84
88.00
Qwen2.5-VL-72B-Instruct
4.18
64.00
4.82
86.00
Gemini-2.0-Flash
3.86
54.00
4.68
80.00
GPT-4o-mini
3.02
40.00
4.76
86.00
Table 2: Comparison of FigStep and VisCo Attack across different MLLMs on SafeBench-Tiny in terms of Toxic (1–5) and ASR (%).
Setting
Toxic
ASR
VisCo Attack
3.72
50.00
w/o Context
3.34
36.00
w/o Refinement
3.68
42.00
2 Rounds
3.84
42.00
4 Rounds
3.98
54.00
Table 3: Ablation study of VisCo Attack on SafeBench-Tiny using GPT-4o in terms of Toxic (1–5) and ASR (%).
Figure 3: Results of VisCo Attack with different red team assistants ( πred ) on SafeBench-Tiny using GPT-4o as the target model, in terms of Toxic (1–5) and ASR (%).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Category
GPT-4o
GPT-4o-mini
Gemini-2.0-Flash
InternVL2.5-78B
QR
VS
QR
VS
QR
VS
QR
VS
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
01-IA
0.86
2.06
4.90
89.69
1.11
2.06
4.95
94.85
1.58
12.37
4.93
92.78
1.69
12.37
4.94
94.85
02-HS
1.53
4.29
4.60
65.64
1.91
8.59
4.75
79.14
2.45
24.54
4.80
80.98
2.82
32.52
4.78
79.14
03-MG
2.34
34.09
4.95
97.73
2.30
29.55
4.93
97.73
3.55
47.73
4.95
97.73
3.91
61.36
4.93
97.73
04-PH
1.77
18.75
4.95
95.83
1.97
18.06
4.97
97.22
2.86
38.89
4.99
99.31
3.15
45.83
4.99
99.31
Appendix
Table 4: Attack results for Enhanced QR (SD+Typo, on regenerated images) and our VisCo attack (VS: Image-Grounded Scenario Simulation) on MM-SafetyBench in terms of Toxic (1–5) and ASR (%) across different MLLMs. “01–IA” to “13–GD” denote the 13 subcategories of prohibited scenarios, and “ALL” represents the overall performance across all categories.
Category
GPT-4o
GPT-4o-mini
Gemini-2.0-Flash
InternVL2.5-78B
VM
VH
VM
VH
VM
VH
VM
VH
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
01-IA
4.51
75.26
4.94
94.85
4.59
79.38
4.94
94.85
4.94
95.88
4.99
98.97
4.80
88.66
4.94
95.88
02-HS
4.33
48.47
4.68
74.23
4.67
70.55
4.88
90.18
4.75
84.05
4.85
88.34
4.79
81.60
4.91
92.02
03-MG
4.95
95.45
5.00
100.00
5.00
100.00
5.00
100.00
5.00
100.00
5.00
100.00
5.00
100.00
5.00
100.00
04-PH
4.72
82.64
4.92
93.75
4.83
84.03
4.95
95.14
4.92
93.06
4.93
95.83
4.92
92.36
4.96
96.53
Appendix
Table 5: Results of our VisCo attack strategies (VM: Image Multi-Perspective Analysis; VH: Exploiting Image Hallucination) on MM-SafetyBench in terms of Toxic (1–5) and ASR (%) across different MLLMs. “01–IA” to “13–GD” denote the 13 subcategories of prohibited scenarios, and “ALL” represents the overall performance across all categories.
Model
VS
VM
VH
Toxic
ASR
Toxic
ASR
Toxic
ASR
LLaVA-OV-7B-Chat
4.72
80.00
4.72
76.00
4.92
92.00
InternVL2.5-78B
4.80
84.00
4.84
90.00
4.92
92.00
Qwen2.5-VL-72B-Instruct
4.82
86.00
4.76
82.00
4.96
96.00
Gemini-2.0-Flash
4.82
86.00
4.80
84.00
4.88
92.00
GPT-4o-mini
4.70
82.00
4.64
78.00
4.88
92.00
Appendix
Table 6: Results of our VisCo attack strategies (VS, VM, VH) on FigStep-SafeBench (SafeBench-Tiny) in terms of Toxic (1–5) and ASR (%) across different MLLMs.
Model
VS
VM
VI
VH
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
LLaVA-OV-7B-Chat
4.93
93.64
4.89
90.91
4.94
94.55
4.93
93.64
InternVL2.5-78B
4.94
93.64
4.79
88.18
4.91
93.64
4.95
96.36
Qwen2.5-VL-72B-Instruct
4.94
94.55
4.93
95.45
4.96
96.36
4.95
96.36
Gemini-2.0-Flash
4.95
94.55
4.85
92.73
4.93
94.55
4.95
97.27
GPT-4o-mini
4.82
86.36
4.62
76.36
4.82
87.27
4.85
90.91
Appendix
Table 7: Results of our VisCo attack strategies (VS, VM, VI, VH) on HarmBench (multimodal_behavior) in terms of Toxic (1–5) and ASR (%) across different MLLMs.
Figure 4: Case Study (VS: Image-Grounded Scenario Simulation)
Figure 5: Case Study (VM: Image Multi-Perspective Analysis)
Figure 6: Case Study (VI: Iterative Image Interrogation)
Figure 7: Case Study (VH: Exploiting Image Hallucination)
Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.
Multimodal Large Language Models (MLLMs) have achieved remarkable performance but remain vulnerable to jailbreak attacks that can induce harmful content and undermine their secure deployment. Previous studies have shown that introducing additional inference steps, which disrupt security attention, can make MLLMs more susceptible to being misled into generating malicious content. However, these methods rely on single-image masking or isolated visual cues, which only modestly extend reasoning paths and thus achieve limited effectiveness, particularly against strongly aligned commercial closed-source models. To address this problem, in this paper, we propose Multi-Image Dispersion and Semantic Reconstruction (MIDAS), a multimodal jailbreak framework that decomposes harmful semantics into risk-bearing subunits, disperses them across multiple visual clues, and leverages cross-image reasoning to gradually reconstruct the malicious intent, thereby bypassing existing safety mechanisms. The proposed MIDAS enforces longer and more structured multi-image chained reasoning, substantially increases the model's reliance on visual cues while delaying the exposure of malicious semantics and significantly reducing the model's security attention, thereby improving the performance of jailbreak against advanced MLLMs. Extensive experiments across different datasets and MLLMs demonstrate that the proposed MIDAS outperforms state-of-the-art jailbreak attacks for MLLMs and achieves an average attack success rate of 81.46% across 4 closed-source MLLMs. Our code is available at this link.
Yilian Liu, Xiaojun Jia, Guoshun Nan +6
Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore · Guilin University of Electronic Technology, China
Existing jailbreak attacks have exposed serious safety weaknesses in multimodal large language models (MLLMs). Most prior attacks rely on specially constructed visual inputs to carry or support harmful intent. In this paper, we uncover a different vulnerability: task-relevant scene semantics alone can shift an MLLM from refusal to compliance, even when the visual input introduces no additional attack instructions and the original task intent remains unchanged. We formalize this failure as a scene-grounded visual safety vulnerability and introduce SceneSafetyBench to investigate it across eight risk categories and nine white- and black-box MLLMs. Our analysis reveals that this failure is associated with how task-relevant visual evidence influences downstream safety representations, suggesting that visual grounding alone cannot fully explain the observed safety shift. Building on this finding, we propose Scene-Grounded Attacks (SGA), which preserve the original task while constructing validated scene-grounded queries and combining their independent responses. Across nine target MLLMs, SGA achieves an average attack success rate (ASR) of 86.17%, outperforming the strongest baseline by 8.55 percentage points and achieving the highest ASR on eight models. Our results identify scene semantics as an overlooked attack surface in MLLMs and highlight a growing tension between stronger visual grounding and robust multimodal safety alignment.
Tongxi Wu, Yuyang Cheng, Jiacheng Xian +5
Nanjing University · The Hong Kong University of Science and Technology (Guangzhou) · Independent Researcher +2
Multimodal Large Language Models (MLLMs) are vulnerable to jailbreak attacks, which can elicit harmful responses from MLLMs. Many MLLMs support multi-image inputs, inadvertently introducing new vulnerabilities due to less efforts on multi-image safety alignment. Previous MLLM jailbreak methods only uses a single image, which restricts the attack space: they cannot distribute harmful requests across multiple images, carry abundant information, or exploit additional visual reasoning tasks to distract MLLMs. To address these limitations, in this paper, we propose a compositional jailbreak framework, DMN, which leverages Distributed instruction, Multimodal evidence and a Number chain task to fully enhance the jailbreak performance. Extensive experiments show that DMN is highly effective for MLLM jailbreaking, e.g. achieving attack success rates of over 90% on GPT-4o, Gemini-2.5-pro and Claude Sonnet 4, surpassing other baselines by a large margin. This compositional, multi-image jailbreak strategy reveals fundamental weaknesses in their safety mechanisms.
Wenzhuo Xu, Zhipeng Wei, Zonghao Ying +4
AI Security Lab, Beijing, China · International Computer Science Institute, CA, USA · UC Berkeley, CA, USA +1