With the emergence of strong vision language capabilities, multimodal large language models (MLLMs) have demonstrated tremendous potential for real-world applications. However, the security vulnerabilities exhibited by the visual modality pose significant challenges to deploying such models in open-world environments. Recent studies have successfully induced harmful responses from target MLLMs by encoding harmful textual semantics directly into visual inputs. However, in these approaches, the visual modality primarily serves as a trigger for unsafe behavior, often exhibiting semantic ambiguity and lacking grounding in realistic scenarios. In this work, we define a novel setting: vision-centric jailbreak, where visual information serves as a necessary component in constructing a complete and realistic jailbreak context. Building on this setting, we propose the VisCo (Visual Contextual) Attack. VisCo fabricates contextual dialogue using four distinct vision-focused strategies, dynamically generating auxiliary images when necessary to construct a vision-centric jailbreak scenario. To maximize attack effectiveness, it incorporates automatic toxicity obfuscation and semantic refinement to produce a final attack prompt that reliably triggers harmful responses from the target black-box MLLMs. Specifically, VisCo achieves a toxicity score of 4.78 and an Attack Success Rate (ASR) of 85% on MM-SafetyBench against GPT-4o, significantly outperforming the baseline, which achieves a toxicity score of 2.48 and an ASR of 22.2%. Code: https://github.com/Dtc7w3PQ/Visco-Attack.
Figures & tables
Figure 1: Illustration of the vision-centric jailbreak setting. The visual input is an essential component that constitutes the complete jailbreak scenario.
Figure 2: Workflow of the VisCo Attack. (Left) generation of the fabricated visual context and the initial attack prompt using vision-focused strategies. (Right) iterative toxicity obfuscation and semantic refinement of the initial attack prompt.
GPT-4o
GPT-4o-mini
Gemini-2.0-Flash
InternVL2.5-78B
QR Attack
VisCo Attack
QR Attack
VisCo Attack
QR Attack
VisCo Attack
QR Attack
VisCo Attack
Category
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
01-IA
1.02
2.06
4.85
89.69
1.10
2.06
4.88
91.75
1.49
10.31
4.95
94.85
1.61
9.28
4.95
95.88
02-HS
1.22
0.61
4.59
64.42
1.60
5.52
4.78
79.75
1.93
13.50
4.77
82.21
2.38
20.25
4.81
81.60
03-MG
2.00
15.91
4.93
95.45
1.77
13.64
4.93
95.45
3.52
56.82
4.91
95.45
3.57
56.82
4.98
97.73
04-PH
1.85
19.44
4.85
90.97
1.94
18.75
4.86
90.28
2.83
39.58
4.97
97.22
3.13
44.44
4.95
95.14
Table 1: Results of Query-Relevant (QR) Attack and our VisCo Attack on MM-SafetyBench in terms of Toxic (1–5) and ASR (%) across different MLLMs. “01-IA” to “13-GD” denote the 13 subcategories of prohibited scenarios, and “ALL” represents the overall performance across all categories.
Attack
FigStep
VisCo Attack
Metric
Toxic
ASR
Toxic
ASR
LLaVA-OV-7B-Chat
3.98
54.00
4.70
80.00
InternVL2.5-78B
2.74
34.00
4.84
88.00
Qwen2.5-VL-72B-Instruct
4.18
64.00
4.82
86.00
Gemini-2.0-Flash
3.86
54.00
4.68
80.00
GPT-4o-mini
3.02
40.00
4.76
86.00
Table 2: Comparison of FigStep and VisCo Attack across different MLLMs on SafeBench-Tiny in terms of Toxic (1–5) and ASR (%).
Setting
Toxic
ASR
VisCo Attack
3.72
50.00
w/o Context
3.34
36.00
w/o Refinement
3.68
42.00
2 Rounds
3.84
42.00
4 Rounds
3.98
54.00
Table 3: Ablation study of VisCo Attack on SafeBench-Tiny using GPT-4o in terms of Toxic (1–5) and ASR (%).
Figure 3: Results of VisCo Attack with different red team assistants ( πred ) on SafeBench-Tiny using GPT-4o as the target model, in terms of Toxic (1–5) and ASR (%).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Category
GPT-4o
GPT-4o-mini
Gemini-2.0-Flash
InternVL2.5-78B
QR
VS
QR
VS
QR
VS
QR
VS
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
01-IA
0.86
2.06
4.90
89.69
1.11
2.06
4.95
94.85
1.58
12.37
4.93
92.78
1.69
12.37
4.94
94.85
02-HS
1.53
4.29
4.60
65.64
1.91
8.59
4.75
79.14
2.45
24.54
4.80
80.98
2.82
32.52
4.78
79.14
03-MG
2.34
34.09
4.95
97.73
2.30
29.55
4.93
97.73
3.55
47.73
4.95
97.73
3.91
61.36
4.93
97.73
04-PH
1.77
18.75
4.95
95.83
1.97
18.06
4.97
97.22
2.86
38.89
4.99
99.31
3.15
45.83
4.99
99.31
Appendix
Table 4: Attack results for Enhanced QR (SD+Typo, on regenerated images) and our VisCo attack (VS: Image-Grounded Scenario Simulation) on MM-SafetyBench in terms of Toxic (1–5) and ASR (%) across different MLLMs. “01–IA” to “13–GD” denote the 13 subcategories of prohibited scenarios, and “ALL” represents the overall performance across all categories.
Category
GPT-4o
GPT-4o-mini
Gemini-2.0-Flash
InternVL2.5-78B
VM
VH
VM
VH
VM
VH
VM
VH
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
01-IA
4.51
75.26
4.94
94.85
4.59
79.38
4.94
94.85
4.94
95.88
4.99
98.97
4.80
88.66
4.94
95.88
02-HS
4.33
48.47
4.68
74.23
4.67
70.55
4.88
90.18
4.75
84.05
4.85
88.34
4.79
81.60
4.91
92.02
03-MG
4.95
95.45
5.00
100.00
5.00
100.00
5.00
100.00
5.00
100.00
5.00
100.00
5.00
100.00
5.00
100.00
04-PH
4.72
82.64
4.92
93.75
4.83
84.03
4.95
95.14
4.92
93.06
4.93
95.83
4.92
92.36
4.96
96.53
Appendix
Table 5: Results of our VisCo attack strategies (VM: Image Multi-Perspective Analysis; VH: Exploiting Image Hallucination) on MM-SafetyBench in terms of Toxic (1–5) and ASR (%) across different MLLMs. “01–IA” to “13–GD” denote the 13 subcategories of prohibited scenarios, and “ALL” represents the overall performance across all categories.
Model
VS
VM
VH
Toxic
ASR
Toxic
ASR
Toxic
ASR
LLaVA-OV-7B-Chat
4.72
80.00
4.72
76.00
4.92
92.00
InternVL2.5-78B
4.80
84.00
4.84
90.00
4.92
92.00
Qwen2.5-VL-72B-Instruct
4.82
86.00
4.76
82.00
4.96
96.00
Gemini-2.0-Flash
4.82
86.00
4.80
84.00
4.88
92.00
GPT-4o-mini
4.70
82.00
4.64
78.00
4.88
92.00
Appendix
Table 6: Results of our VisCo attack strategies (VS, VM, VH) on FigStep-SafeBench (SafeBench-Tiny) in terms of Toxic (1–5) and ASR (%) across different MLLMs.
Model
VS
VM
VI
VH
Toxic
ASR
Toxic
ASR
Toxic
ASR
Toxic
ASR
LLaVA-OV-7B-Chat
4.93
93.64
4.89
90.91
4.94
94.55
4.93
93.64
InternVL2.5-78B
4.94
93.64
4.79
88.18
4.91
93.64
4.95
96.36
Qwen2.5-VL-72B-Instruct
4.94
94.55
4.93
95.45
4.96
96.36
4.95
96.36
Gemini-2.0-Flash
4.95
94.55
4.85
92.73
4.93
94.55
4.95
97.27
GPT-4o-mini
4.82
86.36
4.62
76.36
4.82
87.27
4.85
90.91
Appendix
Table 7: Results of our VisCo attack strategies (VS, VM, VI, VH) on HarmBench (multimodal_behavior) in terms of Toxic (1–5) and ASR (%) across different MLLMs.
Figure 4: Case Study (VS: Image-Grounded Scenario Simulation)
Figure 5: Case Study (VM: Image Multi-Perspective Analysis)
Figure 6: Case Study (VI: Iterative Image Interrogation)
Figure 7: Case Study (VH: Exploiting Image Hallucination)
Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.
Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore · Guilin University of Electronic Technology, China