Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models

In Chong Choi; Jiacheng Zhang; Feng Liu; Yiliao Song

cs.CVFeb 16, 2026

Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models

Authors: In Chong Choi, Jiacheng Zhang, Feng Liu, Yiliao Song

Organizations: The University of Melbourne · The University of Adelaide

Abstract

Multi-turn jailbreak attacks have proven effective against text-only large language models (LLMs), where malicious content is gradually introduced to bypass safety alignment. However, effectively extending such attacks to large vision-language models (LVLMs) remains underexplored. In this paper, we find that naively incorporating visual inputs can make multi-turn jailbreaks easier to defend against; for example, overly malicious visual content will easily trigger the defense mechanism in safety-aligned LVLMs, resulting in more conservative responses. Based on this finding, we propose multi-turn adaptive prompting attack (MAPA) that 1) at each turn, alternates text-vision attack actions to elicit the most malicious response; and 2) across turns, adjusts the attack trajectory through iterative back-and-forth refinement to gradually amplify response maliciousness. This two-level design enables MAPA to consistently outperform state-of-the-art methods, improving attack success rates by 15-30% on recent benchmarks against LLaVA-v1.6-Mistral-7B, Qwen2.5-VL-7B-Instruct, Llama-3.2-Vision-11B-Instruct and GPT-4o-mini. Our code is available at: https://github.com/thomaschoi143/MAPA.

Figures & tables

Figure 1: Multi-turn dialogue examples. We attack a LLaVA-V1.6-Mistral-7B model using text input from a state-of-the-art multi-turn jailbreak method [ 34 ] and query-related images generated by the Stable Diffusion [ 25 ] . It can be seen that naively incorporating images (Sub-figure b) fails to jailbreak the model, whereas carefully selecting less-defensible attack actions across turns progressively elicits harmful responses (Sub-figure c). Further dialogue details are provided in Appendix A . We also validate this aspect by the ablation study in Table 4 .

Figure 2: The flowchart of MAPA . Right: at each turn, MAPA alternates text-vision attack actions to elicit the most malicious response calculated by semantic correlations. Left : across turns, MAPA adjusts the attack trajectory through iterative back-and-forth refinement, thereby effectively amplifying the maliciousness. Reflection: if attack attempt fails, a new attempt begins by regenerating a chain attack using information from prior failures.

Category	Method	LLaVA	Qwen	Llama	Average
Single-turn	VRP	66.67	5.00	65.00	45.56
Single-turn	MML	20.00	65.00	00.00	28.33
Multi-turn	CoA	75.00	73.33	63.33	70.55
	CoA+visual(SD)	68.33	71.67	31.67	57.22
	CoA+visual(UltraBreak)	76.67	66.67	58.33	67.11
	ActorAttack	25.00	25.00	31.67	27.22

Table 1: Attack success rate (%) on HarmBench : MAPA vs. baselines against various target LVLMs. We show the most successful attack in underline bold and the second-best in underline .

Benchmark	Method	LLaVA	Qwen	Llama	Average
JailbreakBench	VRP	00.00	00.00	00.00	00.00
	MML	00.00	00.00	00.00	00.00
	CoA	78.33	70.00	55.00	67.78
	ActorAttack	27.59	35.00	24.56	29.05
	FootInTheDoor	66.66	41.67	56.66	55.00
	MAPA (ours)	93.33	93.33	86.67	91.11

Table 2: Attack success rate (%) on more benchmarks : MAPA vs. multi-turn baselines against various target LVLMs, i.e. , LLaVA-V1.6-Mistral-7B (LLaVA), Qwen2.5-VL-7B-Instruct (Qwen) and Llama-3.2-Vision-11B-Instruct (Llama). We show the most successful attack in underline bold and the second-best in underline .

Victim: GPT-4o-mini	CoA	ActorAttack	FootInTheDoor	MAPA (ours)
Default Judge LLM	53.33	45.76	41.66	88.33
Advanced Judge LLM	73.33	64.40	60.00	93.33
( improvement )	( +20.00 )	( +18.24 )	( +18.34 )	( +5.00 )

Table 3: Success rate of attacking GPT-4o-mini adopting the fine-tuned Llama-2-13B model as Default Judge LLM or GPT-4o-mini as Advanced Judge LLM .

Cross-Turn	*Within-Turn	LLaVA	Qwen	Llama	Average
MAPA (default setting)		93.33	86.66	86.67	88.89
MAPA w/o Reflection		83.33	86.66	70.00	80.00
		(- 10.00)	(- 00.00)	(- 16.67)	(- 8.89)
MAPA repeat w/o Reflection		90.00	86.66	76.66	84.44
		(- 3.33)	(- 00.00)	(- 10.00)	(- 4.45)
MAPA w/o Policy Adjustment		86.66	83.33	60.00	76.66

Table 4: Ablation study. Performance degradations (%) are in red . *Attack Action 1-3 refers to attacking by Unconnected Text , Unconnected Text + Vision or Connected Text + Vision separately. For efficiency, we randomly sample 5 behaviors from per HarmBench’s category, resulting in 30 harmful tasks.

Method	LLaVA	Qwen	Llama
CoA	83.33% (#20.17)	86.66% (#20.00)	85.00% (#34.33)
ActorAttack	65.00%(#22.47)	48.33%(#18.48)	71.67%(#35.43)
FootInTheDoor	75.00%(#24.47)	50.00%(#26.87)	85.00%(#34.50)
MAPA (ours)	98.33% (#17.88)	100.00% (#16.47)	91.66% (#34.30)

Table 5: The values in each

\cdot

%(#

\cdot

) indicate the attack success rate and the average number of target queries of methods on HarmBench(sampled), given the budget of total target queries limited to a roughly similar level.

Figure 3: Distribution of attack actions across turns of MAPA ’s successful jailbreaks on HarmBench (sampled).

Defense	LLaVA	Qwen	Llama	Average
AdaShield-Static	78.33	58.33	71.66	69.44
Llama-Guard-3-Vision-11B	78.33	78.33	68.33	75.00

Table 6: Attack success rates (%) against defended LVLMs on the HarmBench(sampled) evaluation set.

Hyperparameter( default )	Perturbation	LLaVA	Qwen	Llama	Average
temperature redteam ( 0.3 )	$-20\%$	+ 0.00	+ 6.67	+ 6.67	+ 4.45
temperature redteam ( 0.3 )	$+20\%$	+ 0.00	+ 6.67	+ 3.33	+ 3.33
temperature victim ( 0.0 )	0.3	+ 0.00	+ 6.67	+ 6.66	+ 4.44
max_token redteam ( 2000 )	$-20\%$	- 6.67	+ 3.33	+ 3.33	+0.00
max_token redteam ( 2000 )	$+20\%$	- 3.34	+ 3.33	- 3.34	-1.12
max_token victim ( 300 )	$-33\%$	- 3.34	+ 6.67	+ 0.00	+ 1.11

Table 7: Hyperparameter sensitivity analysis.

Model	Avg. SEM (Successful)	Avg. SEM (Failed)
Llama	0.6738	0.5773
LLaVA	0.6740	0.6572
Qwen	0.6735	0.6490

Table 8: Average SEM of successful and failed jailbreaks on HarmBench (sampled).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: An example of attacking with text-only prompts.

Figure 5: An example of attacking with text prompts and query-related images.

Figure 6: An example of attacking with connected text prompts and query-related images.

Figure 7: A successfully jailbroken example by combining previous input types.

Figure 8: Change in the response’s semantic correlation across turns on LLaVA-V1.6-Mistral-7B, sampled from five successful and five failed attempts from our method. Success cases exhibit a more pronounced upward trend than failure cases do.

Figure 9: System prompt to generate the attack chain without reflection for the attacker in Python.

Figure 10: System prompt to generate the attack chain with reflection for the attacker in Python.

Figure 11: System prompt to generate the next unconnected attack prompt in Advance policy for the attacker in Python.

Figure 12: System prompt to regenerate the unconnected attack prompt of the current turn in Regen policy for the attacker in Python.

Figure 13: System prompt to generate the connected text prompt and image prompt for the connector in Python.

Figure 14: Prompt to generate a malicious image according to the image prompt for the diffusion model in Python.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 179 papers

Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models
Aug 27, 2026Benlei Cui, Shen Pang, Yuke Wang +7VLM Robustness Adversarial Attacks on VLMs
1Yuvion Team, Alibaba Group · Laboratory for Statistical Monitoring and Intelligent Governance of Common Prosperity, School of Statistics and Data Science, Zhejiang Gongshang University
Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
Apr 11, 2026Jingru Li, Wei Ren, Tianqing ZhuJailbreak Attacks Adversarial Attacks on VLMs
China University of Geosciences, Wuhan · City University of Macau
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3

Aug 27, 2026J/K move · Enter open · S save

Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models

Benlei Cui, Shen Pang, Yuke Wang, Xuemei Dong+6

1Yuvion Team, Alibaba Group · Laboratory for Statistical Monitoring and Intelligent Governance of Common Prosperity, School of Statistics and Data Science, Zhejiang Gongshang University

VLM Robustness Adversarial Attacks on VLMs Multimodal Jailbreak Attacks LLM Safety Evaluation

Open paper AskPDF arXiv

The safety of large vision-language models is increasingly stress-tested by multimodal jailbreaks, yet existing attacks remain largely static at the meta level: template-based attacks freeze the image-text layout, while iterative attacks adapt only the image-text content with fixed attack strategies and frozen attacker parameters. We propose Meta-Adaptive Multimodal Jailbreaking (MAMJ), which instead optimizes the attacker itself along two axes: an attack strategy prompt (ASP) governing attack iteration and attacker model weights determining attack effectiveness. Across groups of multimodal attack trajectories, an LLM-based critique first refines the ASP, after which group-aggregated attack success rate (ASR) rewards update those weights. On MM-SafetyBench, MAMJ achieves 81.0%, 78.9%, and 82.3% ASR against GPT-4o, Gemini-3-Pro-Preview, and Seed 2.0, respectively, outperforming the strongest sample-level baseline by up to 24.1 percentage points. The learned attacker, comprising the optimized ASP and attacker weights, also transfers without retraining to unseen victims and remains effective under representative defenses. These results reveal a systemic vulnerability of frontier VLMs to meta-adaptive jailbreaks and motivate defenses against meta-level adversaries. Code is available at https://github.com/Alibaba-VELLDEPTH/MetaJailbreak-VLM.

Abstract

Adversarial Attacks on VLMs

Explore similar work

Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models

Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking

Jailbreaking Vision-Language Models Through the Visual Modality