Organizations: School of Computer Science, Wuhan University, Wuhan, China · Zhongguancun Academy, Beijing, China · Tianjin University, Tianjin, China · University of Chinese Academy of Sciences, Beijing, China
Vision-Language Models (VLMs) expand the attack surface of safety-aligned systems by coupling visual perception with text generation. Existing multimodal jailbreak attacks primarily rely on crafted visual content, adversarial perturbations, or image-specific attack strategies, leaving the potential of reusable visual anchors in benign natural images largely unexplored. To address the problem, we introduce MemJack, a memory-augmented multi-agent framework for automated VLM red-teaming with natural images. MemJack combines visual-anchor discovery, visual-semantic camouflage, response evaluation, reflection-guided repair, and dynamic replanning within a closed-loop attack pipeline. Beyond attack generation, MemJack automatically transforms public images into attack anchors, enables the construction of MemJack-Bench, a dataset of over 113,000 interactive multimodal jailbreak trajectories for safety evaluation and defensive alignment. Extensive empirical evaluations across full, unmodified COCO val2017 images demonstrate that MemJack achieves a 71.48% attack success rate (ASR) against Qwen3-VL-Plus, scaling to 90% under extended budgets. Compared with representative multimodal jailbreak baselines under the same natural-image evaluation setting, MemJack achieves the highest ASR while requiring fewer mean rounds to success, demonstrating superior jailbreak effectiveness on VLMs. These results demonstrate that benign natural images can act as transferable jailbreak anchors and reveal substantial vulnerabilities in current safety-aligned VLMs.
Figures & tables
Figure 1: Overview of MemJack. The planner maps visual anchors to attack goals; the attacker generates visual-semantic prompts; the evaluator scores responses and triggers reflection or replanning. The memory module persists across images for strategy transfer.
Figure 2: Progressive attack performance. (a) cumulative ASR and moving average on COCO-val. (b) cumulative MRS and moving average on COCO-Jailbreak.
Table 3: MemJack generalization on diverse malicious and benign image datasets.
Victim Model
Type
ASR
MRS
Qwen3-VL-Plus
API
72
5.38
Gemini-3-Flash
API
35
6.62
Mistral-Medium-3
API
82
4.45
Llama-3.2-11B-Vision
Local
68
6.25
Qwen3-VL-8B-Instruct
Local
53
5.89
GLM-4.6-V-Flash
Local
63
8.33
Table 4: ASR across victim models.
Method
Text Operation
Image Operation
ASR (%)
Avg. Queries
Text-only attacks
GCG [ 30 ]
Suffix optimization
Unchanged
0.62
13.95
AutoDAN-Turbo [ 15 ]
Strategy rewriting
Unchanged
21.25
17.65
PAIR [ 1 ]
Iterative prompt
Unchanged
51.25
17.71
Multimodal attacks
HADES [ 12 ]
Template query
Toxified/typographic image
3.75
19.79
Table 5: Comparison of jailbreak attack baselines on AdvBench-M. Text operation and Image operation make the attacked channel explicit. Avg. Queries is averaged over all completed samples under a 20 victim-query budget.
Table 8
Model
Advbench-M (N=729)
MM-Safetybench (N=260)
SIUO (N=167)
FigStep (N=500)
VLBreakBench (N=916)
JailbreakV (N=100)
COCO-Jailbreak (N=3574)
Qwen3-VL-Plus
0.00%
0.77%
0.00%
0.00%
1.64%
0.00%
61.78%
Mistral-Medium-3
1.10%
10.00%
0.00%
16.20%
27.40%
4.00%
49.69%
Gemini-3-Flash
0.00%
1.54%
0.60%
4.60%
4.37%
5.00%
14.55%
Claude-Haiku-4.5
0.00%
0.77%
0.00%
1.80%
0.22%
0.00%
2.69%
GPT-5-Mini
0.00%
0.77%
0.00%
2.80%
0.22%
0.00%
0.25%
DeepSeek-V3.2
0.14%
0.77%
0.00%
5.00%
6.00%
1.00%
48.07%
Table 8: Comprehensive ASR evaluation results across jailbreak benchmarks
The safety of large vision-language models is increasingly stress-tested by multimodal jailbreaks, yet existing attacks remain largely static at the meta level: template-based attacks freeze the image-text layout, while iterative attacks adapt only the image-text content with fixed attack strategies and frozen attacker parameters. We propose Meta-Adaptive Multimodal Jailbreaking (MAMJ), which instead optimizes the attacker itself along two axes: an attack strategy prompt (ASP) governing attack iteration and attacker model weights determining attack effectiveness. Across groups of multimodal attack trajectories, an LLM-based critique first refines the ASP, after which group-aggregated attack success rate (ASR) rewards update those weights. On MM-SafetyBench, MAMJ achieves 81.0%, 78.9%, and 82.3% ASR against GPT-4o, Gemini-3-Pro-Preview, and Seed 2.0, respectively, outperforming the strongest sample-level baseline by up to 24.1 percentage points. The learned attacker, comprising the optimized ASP and attacker weights, also transfers without retraining to unseen victims and remains effective under representative defenses. These results reveal a systemic vulnerability of frontier VLMs to meta-adaptive jailbreaks and motivate defenses against meta-level adversaries. Code is available at https://github.com/Alibaba-VELLDEPTH/MetaJailbreak-VLM.
Benlei Cui, Shen Pang, Yuke Wang +7
1Yuvion Team, Alibaba Group · Laboratory for Statistical Monitoring and Intelligent Governance of Common Prosperity, School of Statistics and Data Science, Zhejiang Gongshang University
Large vision-language models (VLMs) often exhibit weakened safety alignment with the integration of the visual modality. Even when text prompts contain explicit harmful intent, adding an image can substantially increase jailbreak success rates. In this paper, we observe that VLMs can clearly distinguish benign inputs from harmful ones in their representation space. Moreover, even among harmful inputs, jailbreak samples form a distinct internal state that is separable from refusal samples. These observations suggest that jailbreaks do not arise from a failure to recognize harmful intent. Instead, the visual modality shifts representations toward a specific jailbreak state, thereby leading to a failure to trigger refusal. To quantify this transition, we identify a jailbreak direction and define the jailbreak-related shift as the component of the image-induced representation shift along this direction. Our analysis shows that the jailbreak-related shift reliably characterizes jailbreak behavior, providing a unified explanation for diverse jailbreak scenarios. Finally, we propose a defense method that enhances VLM safety by removing the jailbreak-related shift (JRS-Rem) at inference time. Experiments show that JRS-Rem provides strong defense across multiple scenarios while preserving performance on benign tasks.
Zhihua Wei, Qiang Li, Jian Ruan +4
School of Computer Science and Technology, Tongji University · Shanghai Artificial Intelligence Laboratory
The visual modality of vision-language models (VLMs) is an underexplored attack surface for bypassing safety alignment. We introduce four jailbreak attacks exploiting the vision component: (1) encoding harmful instructions as visual symbol sequences with a decoding legend, (2) replacing harmful objects with benign substitutes (e.g., bomb -> banana) then prompting for harmful actions using the substitute term, (3) replacing harmful text in images (e.g., on book covers) with benign words while visual context preserves the original meaning, and (4) visual analogy puzzles whose solution requires inferring a prohibited concept. Evaluating across six frontier VLMs, our visual attacks bypass safety alignment and expose a cross-modality alignment gap: text-based safety training does not automatically generalize to harmful intent conveyed visually. For example, our visual cipher achieves 40.9% attack success on Claude-Haiku-4.5 versus 10.7% for an equivalent textual cipher. To further our insight into the attack mechanism, we present preliminary interpretability and mitigation results. These findings highlight that robust VLM alignment requires treating vision as a first-class target for safety post-training.
Aharon Azulay, Jan Dubiński, Zhuoyun Li +2
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3