Multimodal Resource-Exhaustion Attacks on Vision-Language Models via Joint Pixel-Prompt Optimization

Zhaoxiong Ni; Yatie Xiao; Chi-Man Pun; Fei Peng; Qingxiao Guan; Keke Tang

cs.AISep 5, 2026

Multimodal Resource-Exhaustion Attacks on Vision-Language Models via Joint Pixel-Prompt Optimization

Authors: Zhaoxiong Ni, Yatie Xiao, Chi-Man Pun, Fei Peng, Qingxiao Guan, Keke Tang

Organizations: Guangzhou University School of Computer Science and Cyber Engineering Guangzhou, China · University of Macau Department of Computer and Information Science Macau, China · Guangzhou University School of Artificial Intelligence Guangzhou, China · Guangzhou University Cyberspace Institute of Advanced Technology Guangzhou, China

Abstract

Resource-exhaustion attacks against autoregressive vision-language models (VLMs) typically assume unimodal threat models, treating the image branch as the primary optimization surface while holding user-visible prompts fixed. Even recent loop-centric variants remain confined to this single-channel paradigm, leaving the exploitation of availability unexplored as a cross-modal optimization problem over jointly controllable input surfaces. We introduce Joint Pixel-Prompt Optimization (JPPO), the first compound adversarial framework elevating the visible prompt to a first-class adversarial variable alongside image perturbations. Under a restricted joint-input threat model, JPPO performs coupled, stagewise optimization over both the pixel and prompt surfaces. This produces synergistic cost amplification, mechanistically distinct from loop-dependent failures, exhibiting negligible loop incidence in our experiments. Evaluating five open-source VLM families on MS COCO and ImageNet under an 8/255 infinity-norm budget, JPPO achieves over 4.6x latency and 5.3x energy amplification on Qwen2.5-VL-7B, and over 36.6x latency with 32.7x energy amplification on BLIP-2. This represents the strongest cost amplification among directly compared baselines while requiring substantially fewer optimization iterations. Ablations confirm this amplification arises from multimodal coordination rather than prompt length or isolated modalities. These findings reveal structural blind spots in current VLM serving defenses, motivating cost-aware robustness evaluation as a first-class security requirement for multimodal deployments.

Figures & tables

Figure 1. Comparison between a benign inference (top) and the proposed JPPO attack (bottom), along with the resulting cost curve (right). While the clean input yields a concise response, our joint optimization of bounded pixel perturbations and a budget-constrained interface-visible prompt context drives the model toward prolonged decoding, resulting in substantially higher cumulative latency and estimated energy under the same serving configuration. Benign versus JPPO inference comparison The benign inference case is shown at the top and the JPPO inference case at the bottom. The benign input produces a short response, while the JPPO-adversarial input produces a much longer continuation. A cost plot on the right indicates substantially larger cumulative latency and estimated energy for the attacked case under the same serving configuration.

Figure 2. Overview of JPPO. JPPO is a two-stage multimodal availability attack. In Stage I, the attacker maintains three prompt paths, namely visual, spatial, and semantic, and iteratively merges them into a diverse warm-up prompt while jointly updating a bounded pixel perturbation. The strongest measured Stage-I state, consisting of its paired warm-up prompt and perturbation, initializes Stage II, in which JPPO performs generation-driven prompt refinement through selective accumulation, together with further perturbation updates. Across both stages, the attack is guided by three availability-oriented objectives, and the final adversarial state is selected based on the realized resource amplification. JPPO pipeline overview A flow diagram illustrates the two-stage JPPO pipeline. Stage I builds a warm-up prompt from visual, spatial, and semantic path memories while jointly updating a bounded pixel perturbation. Stage II starts from the warm-up prompt and the strongest Stage I perturbation, then further refines the running prompt and pixel perturbation using model-generated feedback. The final adversarial state is selected based on the realized resource amplification.

Model	Method	MS COCO			ImageNet
Model	Method	Len.	$\widehat{E}$ (J)	Lat. (s)	Len.	$\widehat{E}$ (J)	Lat. (s)
BLIP-2	Clean	8.36	67.38	0.42	7.07	81.50	0.55
	Noise	8.23	65.65	0.44	7.22	60.28	0.43
	NICGSlowDown	86.76	567.03	4.46	103.41	663.15	5.01
	Verbose Images	211.62	1432.61	12.07	231.31	1592.69	12.48
	JPPO	373.23	2324.29	19.58	361.60	2665.96	20.16

Table 1. Full comparison on MS COCO and ImageNet. We report absolute output length, estimated-energy proxy

\widehat{E}

, and latency. Means are over three repeated inference runs and three dataset seeds; image-level standard deviations are reported in the appendix. Bold marks the largest non-Clean value in each setting.

Figure 3. Latency amplification across victim models, aggregated over MS COCO and ImageNet. Bars show mean amplification over the two datasets, with error bars indicating cross-dataset variation. Latency amplification bar chart A grouped bar chart compares latency amplification across BLIP-2, Qwen2.5-VL, LLaVA-NeXT, InstructBLIP, and MiniGPT-4. JPPO is the tallest bar for each model among the directly evaluated baselines. BLIP-2 shows the largest latency amplification, while LLaVA-NeXT shows comparatively smaller amplification. Error bars show variation between the two datasets.

Figure 4. White-box-to-black-box latency survivability. For each target, light-blue circles denote the four off-diagonal transfers, dark-blue circles their arithmetic mean, and orange diamonds the exact-model white-box result. Panels show MS COCO and ImageNet on a shared logarithmic scale; the dashed line marks the matched Clean baseline (

1\times

). White-box-to-black-box latency survivability Two side-by-side point plots compare JPPO latency amplification on MS COCO and ImageNet. For each of five target models, four light-blue points show transfer from the other source models, a dark-blue point shows their mean, and an orange diamond shows the exact-model white-box result. All transfer means remain above the Clean baseline but below their exact-model references, showing partial and asymmetric black-box target transfer.

Source Model	BLIP-2	Qwen2.5-VL	LLaVA-NeXT	InstructBLIP	MiniGPT-4
BLIP-2	19.58 / 20.16	16.98 / 18.74	22.49 / 24.29	12.33 / 13.02	17.71 / 8.94
Qwen2.5-VL	4.40 / 7.36	24.03 / 21.16	18.92 / 21.61	16.32 / 16.56	14.61 / 12.35
LLaVA-NeXT	3.73 / 3.38	23.68 / 21.30	27.53 / 27.18	14.73 / 12.80	17.97 / 19.19
InstructBLIP	0.92 / 1.65	15.97 / 17.04	18.45 / 18.46	17.58 / 21.43	12.22 / 12.86
MiniGPT-4	3.64 / 3.31	10.69 / 8.83	16.82 / 11.45	7.52 / 8.02	23.64 / 23.10

Table 2. JPPO latency across source–target pairs (s; MS COCO / ImageNet). Off-diagonal cells use no target-side adaptation; shaded diagonals are exact-model white-box references. Bold marks the highest latency for each target model and dataset.

Model	Method	Construction		Serving
Model	Method	Time (s)	$\widehat{E}$ (kJ)	$\widehat{E}$ (J)	Lat. (s)
BLIP-2	VI	316.04	41.95	1512.65	12.28
BLIP-2	JPPO	582.26	53.04	2495.13	19.87
Qwen2.5-VL	VI	4813.97	183.85	2831.95	10.65
Qwen2.5-VL	JPPO	3354.70	146.72	5163.02	22.60
LLaVA-NeXT	VI	8645.60	658.26	3214.15	11.91

Table 3. Construction and serving costs averaged across datasets. Values are equal-weight means over MS COCO and ImageNet. VI denotes Verbose Images. For JPPO, construction cost includes Stage I and one Stage-II trajectory initialized from the highest-energy Stage-I candidate; serving cost measures only the final selected request.

Method	Rel.	Info.	Red.	Use.
Clean	4.49	3.66	1.43	4.05
Neutral-long-prompt	4.29	4.01	2.04	3.92
Prompt-only	3.72	3.07	3.45	2.98
Verbose Images	2.82	2.15	4.08	2.91
JPPO	3.59	3.44	3.32	3.38

Table 4. Original-task output quality. GPT-5.5 scores for 450,000 candidate outputs are reported as equal-weight means across the ten model–dataset settings. Each output is evaluated against the corresponding original image and benign instruction, irrespective of the method-specific input used during generation. Rel., Info., Red., and Use. denote relevance, informativeness, redundancy, and usefulness; higher is better except for Red.

Model	Loop Inc. (%)	Longest Span (%)	Tail Cov. (%)
BLIP-2	1.90 / 1.01	2.04 / 1.26	1.31 / 0.52
Qwen2.5-VL	0.00 / 0.00	0.00 / 0.00	0.00 / 0.00
LLaVA-NeXT	0.00 / 0.00	0.47 / 0.58	0.00 / 0.00
InstructBLIP	0.00 / 0.00	0.01 / 0.13	0.00 / 0.00
MiniGPT-4	1.60 / 0.00	1.82 / 1.58	0.93 / 0.59

Table 5. Loop characterization of JPPO on MS COCO and ImageNet. Entries are reported as MS COCO / ImageNet. Inc., Span, and Cov. denote loop incidence, longest repeated-span ratio, and tail loop coverage, respectively. Lower values are better.

Figure 5. Optimization-space ablation on BLIP-2. Clean-normalized output-length, estimated-energy, and latency amplification for Pixel-only, Prompt-only, and Joint optimization on MS COCO and ImageNet. Optimization-space ablation bar chart Two bar charts compare Pixel-only, Prompt-only, and Joint optimization on MS COCO and ImageNet. Each panel reports amplification over clean for length, energy, and latency. Prompt-only is generally stronger than Pixel-only, while Joint optimization is strongest across all metrics and both datasets.

Model	Modality		Single Loss			Joint
Model	Pixel	Prompt	$\boldsymbol{\mathcal{L}_{\mathrm{eos}}}$	$\boldsymbol{\mathcal{L}_{\mathrm{align}}}$	$\boldsymbol{\mathcal{L}_{\mathrm{bot}}}$	Joint
BLIP-2	11.02	20.19	29.68	22.52	22.72	34.50
Qwen2.5-VL	3.91	3.36	4.12	4.68	4.71	5.77
LLaVA-NeXT	3.04	3.21	2.96	3.18	3.39	4.13
InstructBLIP	2.37	3.06	4.89	4.31	4.57	5.62
MiniGPT-4	4.27	4.33	5.11	5.46	5.72	6.80

Table 6. Cross-model ablation on MS COCO. Estimated-energy amplification over Clean. Bold marks the largest value per model.

Model	JPPO	Verbose Images	NICG SlowDown	Lingo Loop
(a) Nine-defense mean
BLIP-2	11.25	13.05	7.91	25.12
Qwen2.5-VL	2.78	4.80	2.03	6.04
LLaVA-NeXT	5.14	3.72	3.32	14.28
InstructBLIP	1.24	14.92	4.29	15.97
MiniGPT-4	4.69	21.95	10.98	25.93

Table 7. Cross-model defense degradation (%). Panel (a) averages the nine non-EarlyStop defenses, and Panel (b) reports EarlyStop. Values are equal-weight means over datasets and serving-cost metrics; lower means less mitigation. Bold marks the smallest degradation per model and the smallest overall average.

Figure 6. Per-defense degradation across five VLMs (%). Rows denote defenses and columns denote attacks. Each cell is an equal-weight mean over the five models, MS COCO and ImageNet, and the three serving-cost metrics (output length, estimated energy, and latency). Darker blue indicates greater degradation; lower means less mitigation, and negative values indicate increased cost. Per-defense degradation heatmap A heatmap with ten defenses as rows and four attack methods as columns. Each cell is annotated with the corresponding degradation percentage. Lighter blue indicates lower degradation and darker blue indicates higher degradation, with the color bar spanning approximately minus 2 percent to 25 percent. Negative values are displayed directly in the corresponding cells.

Model	Method	P95	Overload	GPU-h/1K
MS COCO
Qwen2.5-VL	VI	1.82 $\times$	[9%,10%]	1.76
Qwen2.5-VL	JPPO	1.94 $\times$	[4%,5%]	6.60
BLIP-2	VI	37.1 $\times$	[0.75%,1.00%]	3.05
BLIP-2	JPPO	55.9 $\times$	[0.50%,0.75%]	4.92
ImageNet

Table 8. Uncontrolled four-worker FIFO impact at 80% benign utilization. VI denotes Verbose Images. Each overload bracket reports the largest stable and smallest overloaded offered attack shares. Bold marks the stronger effect within each model–dataset setting: higher P95 inflation and GPU-h/1K, and a lower overload threshold.

Model	Policy	JPPO Admission	Attacker Worker Share	Load $\boldsymbol{\rho}$	Benign API P95 E2E Inflation
Qwen2.5-VL	Open	100.00% / 100.00%	23.5% / 25.3%	1.04 / 1.07	42.9 $\times$ / 47.6 $\times$
	Request-RL	91.34% / 87.80%	22.1% / 23.6%	1.01 / 1.02	13.1 $\times$ / 16.2 $\times$
	Safety+Request-RL	90.15% / 86.83%	21.8% / 23.2%	1.00 / 1.02	14.5 $\times$ / 18.0 $\times$
	EarlyStop+Request-RL	91.34% / 87.80%	21.7% / 22.9%	1.00 / 1.01	12.4 $\times$ / 15.4 $\times$
BLIP-2	Open	100.00% / 100.00%	27.2% / 24.1%	1.10 / 1.05	108.6 $\times$ / 94.2 $\times$
	Request-RL	82.74% / 89.55%	24.4% / 22.3%	1.04 / 1.01	80.7 $\times$ / 76.8 $\times$

Table 9. Fixed-trace local API evaluation under serving controls. Entries are MS COCO / ImageNet. Admission is measured before target-model execution, while P95 reports benign end-to-end latency inflation.

Model	Method	MS COCO				ImageNet
Model	Method	Len.	Tokens	$\widehat{E}$ (J)	Lat (s)	Len.	Tokens	$\widehat{E}$ (J)	Lat (s)
Qwen2.5-VL-7B	Clean	70.51	82.13	825.21	3.16	74.13	88.47	893.15	3.68
	LingoLoop	435.29	501.94	5056.13	18.75	439.18	500.18	5014.32	18.01
	JPPO	456.15	510.78	5528.41	23.61	440.68	502.32	5184.18	22.46
InstructBLIP	Clean	53.84	65.35	675.21	3.61	59.17	71.56	857.44	5.08
	LingoLoop	417.61	495.20	5001.62	34.23	409.92	488.36	4751.66	33.71

Table 10. Matched comparison with LingoLoop. All methods use greedy decoding and a 512-token cap; we report word and token lengths, estimated energy, and latency. Bold marks the largest non-Clean value for each model, dataset, and metric.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Dataset	Build Time (s)	Build Energy (kJ)	Online Energy (J)	Online Latency (s)
BLIP-2	MS COCO	576.41 / 304.92	50.14 / 38.30	2324.29 / 1432.61	19.58 / 12.07
BLIP-2	ImageNet	588.10 / 327.16	55.93 / 45.60	2665.96 / 1592.69	20.16 / 12.48
Qwen2.5-VL	MS COCO	3554.83 / 4896.92	154.02 / 186.28	5458.59 / 2898.41	24.03 / 10.88
Qwen2.5-VL	ImageNet	3154.57 / 4731.01	139.42 / 181.41	4867.44 / 2765.49	21.16 / 10.41
LLaVA-NeXT	MS COCO	7574.42 / 8538.11	597.51 / 655.17	6520.12 / 3219.71	27.53 / 11.78
LLaVA-NeXT	ImageNet	7416.33 / 8753.08	520.00 / 661.35	5706.21 / 3208.59	27.18 / 12.04

Appendix

Table 11. Dataset-specific offline construction and online serving costs. Values are reported as JPPO / Verbose Images. Construction metrics are attacker-side; online metrics measure the final selected request. Main-text Table 3 reports equal-weight means over MS COCO and ImageNet.

Defense	BLIP-2	Qwen2.5-VL	LLaVA-NeXT	InstructBLIP	MiniGPT-4
JPEG (Q=95)	-0.46	-0.71	2.52	3.62	2.59
JPEG (Q=75)	15.70	0.11	4.63	4.06	9.55
JPEG (Q=50)	25.79	5.62	9.01	1.81	0.78
Resize restoration	16.89	4.68	14.58	-1.96	9.12
5-bit quantization	-4.28	-1.06	4.92	-9.01	-1.82
4-bit quantization	30.50	0.45	1.69	4.62	-0.12

Appendix

Table 12. Model-level defense degradation for JPPO (%). Each entry averages both datasets and the three serving-cost metrics. For each model, the mean of the first nine rows and the EarlyStop row correspond to the JPPO entries in Panels (a) and (b), respectively, of Table 7 .

Defense setting	Len.	$\widehat{E}$	Lat.
Nine non-EarlyStop defenses	5.19	4.96	4.92
EarlyStop	2.55	5.81	2.63

Appendix

Table 13. Metric-level defense degradation for JPPO (%). The non-EarlyStop row averages across nine defenses, five target models, and both datasets, while EarlyStop is averaged across five target models and both datasets. The equal-weight means across output length, estimated energy, and latency are 5.02% and 3.66%, respectively, matching the JPPO averages reported in Table 7 .

Model	Method	MS COCO			ImageNet
Model	Method	Len.	$\widehat{E}$ (J)	Lat (s)	Len.	$\widehat{E}$ (J)	Lat (s)
BLIP-2	Clean	8.36	67.38	0.42	7.07	81.50	0.55
	Noise	8.23	65.65	0.44	7.22	60.28	0.43
	NICGSlowDown	86.76	567.03	4.46	103.41	663.15	5.01
	Verbose Images	211.62	1432.61	12.07	231.31	1592.69	12.48
	Hidden Tail	52.05	351.52	2.31	50.13	309.45	2.29

Appendix

Table 14. Expanded full comparison including Hidden Tail under the unified low-perturbation top-

p

/512-token protocol. This table is provided for completeness; the main text excludes Hidden Tail from the directly compared baseline table because its original evaluation regime is not directly aligned with the main low-perturbation comparison. Bold marks the largest non-Clean value for each model, dataset, and metric.

Schedule	Len.	$\widehat{E}$	Lat.
Random	4.46 $\times$	4.96 $\times$	4.28 $\times$
Phase-based	5.95 $\times$	6.06 $\times$	5.40 $\times$

Appendix

Table 15. Stage-I schedule comparison. Values are averaged over the five evaluated models and both datasets under the main evaluation protocol.

Model	Strategy	Traj.	Build Time (s)	Build $\widehat{E}$ (kJ)	Final $\widehat{E}$ (J)	$\Delta$ Final $\widehat{E}$
Qwen2.5-VL	Highest-energy candidate	1	3354.70	146.72	5163.02	$0.00\%$
	Random top-3	1	3381.46	147.95	5201.64	$+0.75\%$
	Random top-5	1	3319.28	145.18	4937.83	$-4.36\%$
	Refine all top-3	3	7186.91	311.27	5384.76	$+4.30\%$
	Refine all top-5	5	11046.38	488.11	5427.19	$+5.12\%$
BLIP-2	Highest-energy candidate	1	582.26	53.04	2495.13	$0.00\%$

Appendix

Table 16. Stage-II candidate-selection ablation. Values are equal-weight means over MS COCO and ImageNet. Build cost includes Stage I once and all Stage-II trajectories executed by each strategy; final

\widehat{E}

measures only the selected request.

Model	Method	MS COCO			ImageNet
		Len.	$\widehat{E}$ (J)	Lat (s)	Len.	$\widehat{E}$ (J)	Lat (s)
BLIP-2	Original	8.36	67.38	0.42	7.07	81.50	0.55
	Neutral	21.93	137.21	1.01	24.17	152.75	1.11
	JPPO	373.23	2324.29	19.58	361.60	2665.96	20.16
Qwen2.5-VL	Original	80.24	945.97	4.61	81.52	912.43	4.53
	Neutral	102.54	1257.80	5.09	115.12	1468.24	5.27

Appendix

Table 17. Effect of different prompt settings and attack methods on multiple models. Each model row shows the results for Original Prompt, Neutral Prompt, and JPPO under the same evaluation protocol. The Neutral Prompt uses the fixed length-control prompt described in Appendix C.3 and is included to test whether JPPO’s gains can be explained by prompt length alone. Bold marks the largest value for each model, dataset, and metric.

Method	Len.	$\widehat{E}$ (J)	Lat. (s)
MS COCO
Clean	8.36	67.38	0.42
Pixel-only	82.59	742.61	5.73
Prompt-only	212.43	1360.29	14.82
JPPO (Joint)	373.23	2324.29	19.58
ImageNet

Appendix

Table 18. Exact values for the optimization-space ablation on BLIP-2. This table complements the amplification visualization in Figure 5 .

Loss Components			MS COCO			ImageNet
$\boldsymbol{\mathcal{L}_{\mathrm{eos}}}$	$\boldsymbol{\mathcal{L}_{\mathrm{align}}}$	$\boldsymbol{\mathcal{L}_{\mathrm{bot}}}$	Len.	$\widehat{E}$ (J)	Lat. (s)	Len.	$\widehat{E}$ (J)	Lat. (s)
			212.43	1360.29	14.82	192.60	1248.02	14.76
✓			282.03	2000.06	18.16	313.67	1958.06	18.51
	✓		239.67	1517.39	16.24	223.90	1471.54	15.47
		✓	244.07	1530.97	16.12	230.67	1539.71	16.85
✓	✓		283.07	2165.65	18.29	275.53	1863.73	17.17

Appendix

Table 19. Ablation analysis of the three availability-oriented loss terms on BLIP-2:

\mathcal{L}_{\mathrm{eos}}

for prefix stop suppression,

\mathcal{L}_{\mathrm{align}}

for cross-modal alignment suppression, and

\mathcal{L}_{\mathrm{bot}}

for bottleneck regularization. We report generation length, estimated-energy proxy

\widehat{E}

, and latency across MS COCO and ImageNet. The row with no checked loss component corresponds to prompt-only refinement without pixel-space availability losses.

Source Model	BLIP-2	Qwen2.5-VL	LLaVA-NeXT	InstructBLIP	MiniGPT-4
MS COCO
BLIP-2	373.23	361.21	345.36	209.59	226.40
Qwen2.5-VL	111.02	441.23	298.74	293.08	222.87
LLaVA-NeXT	69.20	446.50	369.10	217.91	256.93
InstructBLIP	26.90	343.22	277.93	250.92	198.30
MiniGPT-4	76.70	274.51	274.46	104.40	385.35

Appendix

Table 20. JPPO output length across all source–target model pairs. Rows are construction models, columns are evaluation targets, and shaded diagonal cells denote exact-model white-box references. Bold marks the highest output length for each target model and dataset.

Source Model	BLIP-2	Qwen2.5-VL	LLaVA-NeXT	InstructBLIP	MiniGPT-4
MS COCO
BLIP-2	2324.29	3509.62	5178.87	2296.14	3032.69
Qwen2.5-VL	642.49	5458.59	4191.65	2844.30	2838.93
LLaVA-NeXT	435.85	5083.85	6520.12	2309.90	3605.19
InstructBLIP	101.26	3254.06	3751.56	3476.18	2695.34
MiniGPT-4	328.93	2391.89	3912.61	1037.25	5378.77

Appendix

Table 21. JPPO estimated energy across all source–target model pairs (J). Rows are construction models, columns are evaluation targets, and shaded diagonal cells denote exact-model white-box references. Bold marks the highest estimated energy for each target model and dataset.

Figure 7. Mean token-level novelty ratio before and after JPPO across five evaluated VLMs. The statistic is computed on normalized generated tokens using a sliding window of 32. Higher values indicate more novel continuation behavior, whereas lower values indicate stronger repetition. JPPO reduces novelty relative to benign generation, but the attacked outputs remain well above collapse-to-loop behavior on most models, consistent with the two-dataset loop-incidence results in Table 5 . Novelty-ratio comparison before and after attack A bar chart compares the mean novelty ratio before and after JPPO across five VLMs. For every model, the attacked outputs have lower novelty than the clean outputs, but the post-attack values remain moderate rather than collapsing to near-zero repetition.

Figure 8. Sensitivity of JPPO to the bottleneck dimensionality ratio on BLIP-2 across MS COCO and ImageNet. The ratio

\rho

determines the compressed bottleneck dimension

d^{\prime}

in Eq. ( 13 ). The left column shows the low-ratio regime, and the right column shows the broader global trend. The top row reports output-length amplification, while the bottom row reports latency amplification. Bottleneck-ratio sensitivity across two datasets A four-panel line chart showing sensitivity to the bottleneck dimensionality ratio on BLIP-2 across MS COCO and ImageNet. The left column shows the low-ratio regime, and the right column shows the broader global trend. The top row plots output-length amplification, and the bottom row plots latency amplification.

$\rho$	MS COCO			ImageNet
$\rho$	Len.	$\widehat{E}$ (J)	Lat (s)	Len.	$\widehat{E}$ (J)	Lat (s)
Clean	8.36	67.38	0.42	7.07	81.50	0.55
0.02	294.09	1971.62	18.24	248.62	1641.15	16.21
0.04	371.82	2414.18	19.49	369.11	2410.97	19.42
0.06	277.87	1962.28	17.89	249.18	1601.92	15.94
0.08	288.91	2026.17	17.90	283.01	1968.11	17.53

Appendix

Table 22. Full sensitivity results for the bottleneck dimensionality ratio

\rho

on BLIP-2 across MS COCO and ImageNet. The ratio determines the compressed representation size

d^{\prime}=\max(\lfloor\rho D\rfloor,1)

in the bottleneck regularization objective. We report mean output length, estimated energy, and latency under each setting.

Figure 9. Gradient-based evidence localization for clean and JPPO-adversarial inputs. Each row shows a representative BLIP-2 example. From left to right, we present the clean image, the Grad-CAM overlay on the clean image, the adversarial image obtained by applying the bounded pixel perturbation, and the Grad-CAM overlay on the adversarial image. Compared with the clean input, the adversarial counterpart alters the spatial distribution of model-relevant visual evidence, often making it weaker, more diffuse, or more fragmented, rather than tightly concentrated in compact, object-centric regions. For readability, only a portion of the generated output is shown. Grad-CAM comparison for clean and attacked inputs Several BLIP-2 examples are shown in rows. Each row contains a clean image, its Grad-CAM overlay, an adversarial image, and its Grad-CAM overlay. Relative to the clean cases, the attacked cases typically show weaker, more diffuse, or more fragmented evidence localization rather than compact object-centered hotspots.

$\epsilon$	MS COCO			ImageNet
$\epsilon$	Len.	$\widehat{E}$ (J)	Lat (s)	Len.	$\widehat{E}$ (J)	Lat (s)
Clean	8.36	67.38	0.42	7.07	81.50	0.55
2/255	196.83	1286.49	10.07	189.30	1317.21	11.72
4/255	278.20	1852.19	14.92	277.47	1904.15	15.67
8/255	373.23	2324.29	19.58	361.60	2665.96	20.16
16/255	318.83	2072.89	16.72	363.26	2357.01	19.37

Appendix

Table 23. Impact of perturbation magnitude

\epsilon

on attack effectiveness on BLIP-2 over the MS COCO and ImageNet datasets. Bold marks the largest value for each dataset and metric.

Stage 1 Weights	Stage 2 Weights	MS COCO			ImageNet
Stage 1 Weights	Stage 2 Weights	Len.	$\widehat{E}$ (J)	Lat (s)	Len.	$\widehat{E}$ (J)	Lat (s)
–	–	8.36	67.38	0.42	7.07	81.50	0.55
$(1,1,1)$	$(1,1,1)$	360.90	2040.80	19.77	344.62	2052.23	19.03
$(2,2,2)$	$(2,2,2)$	318.13	2091.37	18.46	301.94	1939.41	17.96
$(2,1,1)$	$(1,1,1)$	240.07	1601.58	15.87	242.09	1625.15	16.34
$(2,2,1)$	$(1,1,1)$	306.93	1872.58	18.06	321.02	1911.64	18.84

Appendix

Table 24. Sensitivity of JPPO to the loss-weight configuration on BLIP-2. Each row reports the Stage-I and Stage-II weights for

(\mathcal{L}_{\mathrm{eos}},\mathcal{L}_{\mathrm{align}},\mathcal{L}_{\mathrm{bot}})

. The shaded row with “–” weights is the main-evaluation Clean reference from Table 1 ; the shaded row with

(2,1,1)

and

(2,3,1)

is the main-evaluation Default JPPO reference. All unshaded rows report a separate small-scale weight-sensitivity sweep. Bold marks the highest estimated energy within each dataset among the tested weight configurations.

Stage 1	Stage 2	MS COCO			ImageNet
Stage 1	Stage 2	Len.	$\widehat{E}$ (J)	Lat (s)	Len.	$\widehat{E}$ (J)	Lat (s)
50	50	338.27	2030.03	16.72	343.93	1985.02	16.38
50	100	307.69	1964.21	16.20	270.32	1748.71	13.74
100	100	306.04	2067.13	17.62	321.42	1899.38	15.43
100	150	293.57	1999.24	16.23	339.14	2170.09	18.37
100	200	260.07	1633.25	13.07	293.57	1932.44	17.08

Appendix

Table 25. Effect of prompt length limits on generation length, estimated energy, and latency on BLIP-2 over the MS COCO and ImageNet datasets. Bold marks the largest value for each dataset and metric.

Dataset	Metric	32	64	128	256	512	1024
MS COCO	Len	30.50	56.77	87.57	179.43	373.23	726.83
	$\widehat{E}$ (J)	206.01	352.29	619.63	1283.68	2324.29	5493.30
	Lat (s)	1.64	2.58	4.29	8.64	19.58	39.35
ImageNet	Len.	31.21	52.80	101.16	191.64	361.60	756.83
	$\widehat{E}$ (J)	216.02	358.18	669.23	1168.57	2665.96	5601.88
	Lat (s)	1.28	2.14	4.55	7.95	20.16	41.92

Appendix

Table 26. Ablation of maximum generation caps for JPPO on the MS COCO and ImageNet datasets.

Stage 1	Stage 2	MS COCO			ImageNet
Stage 1	Stage 2	Len.	$\widehat{E}$ (J)	Lat (s)	Len.	$\widehat{E}$ (J)	Lat (s)
0	0	8.36	67.38	0.42	7.07	81.50	0.55
0	100	225.00	1359.96	9.78	202.79	1445.34	10.92
0	200	226.87	1424.19	10.28	205.83	1472.13	11.72
100	0	277.43	1812.72	14.54	250.03	1767.54	13.98
200	0	313.62	1968.65	16.09	202.81	1474.07	11.22

Appendix

Table 27. Ablation study on stage-wise iterations for BLIP-2 on MS COCO and ImageNet. Bold marks the largest value for each dataset and metric.

Stage 1	Stage 2	MS COCO			ImageNet
Stage 1	Stage 2	Len.	$\widehat{E}$ (J)	Lat (s)	Len.	$\widehat{E}$ (J)	Lat (s)
0	0	8.36	67.38	0.42	7.07	81.50	0.55
10	10	147.97	923.87	7.37	77.07	447.99	4.94
20	20	194.43	1169.28	9.81	246.65	1560.44	12.01
50	50	238.27	1710.68	14.12	224.37	1530.08	12.11
50	100	298.73	2137.28	17.39	322.37	2119.76	17.48

Appendix

Table 28. Resource usage under different stage-wise iteration schedules for BLIP-2 on MS COCO and ImageNet.

Decoding	Method	MS COCO			ImageNet
Decoding	Method	Len.	$\widehat{E}$ (J)	Lat. (s)	Len.	$\widehat{E}$ (J)	Lat. (s)
Greedy	Clean	7.72	65.26	0.37	7.81	66.14	0.39
	Noise	7.81	66.16	0.41	7.34	63.35	0.38
	NICGSlowDown	171.98	1097.11	7.71	226.18	1358.81	8.00
	Verbose Images	303.84	2086.49	10.06	328.89	2073.88	10.91
	Hidden Tail	76.10	564.16	3.88	89.01	549.19	3.45

Appendix

Table 29. Comparison of different decoding strategies on BLIP-2. We report absolute generation length, estimated-energy proxy

\widehat{E}

, and latency on MS COCO and ImageNet. Bold marks the largest non-Clean value for each decoding strategy, dataset, and metric.

Model	Method	MS COCO			ImageNet
Model	Method	Len.	$\widehat{E}$ (J)	Lat (s)	Len.	$\widehat{E}$ (J)	Lat (s)
BLIP-2	Clean	2.23	13.26	0.07	1.86	16.71	0.11
	Noise	2.10	12.79	0.09	1.87	12.91	0.08
	NICGSlowDown	145.91	209.41	4.05	155.94	357.09	4.81
	Verbose Images	176.07	494.18	3.89	172.10	572.65	4.69
	JPPO	97.62	316.74	2.19	88.01	443.13	3.11

Appendix

Table 30. Standard deviation of resource exhaustion attacks on MS COCO and ImageNet. We report the sample standard deviation across image-level means for generation length, estimated energy, and latency, where each image-level value is first averaged over three repeated runs with the same input-method pair.

Model	Method	Attack Effect			Caption Quality
Model	Method	Len.	$\widehat{E}$ (J)	Lat (s)	BLEU-1	BLEU-2	BLEU-3	BLEU-4	CIDEr
BLIP-2	NICGSlowDown	86.76	567.03	4.46	0.011	0.004	0.002	0.001	0.023
	Verbose Images	211.62	1432.61	12.07	0.027	0.013	0.003	0.002	0.072
	JPPO	373.23	2324.29	19.58	0.022	0.010	0.002	0.001	0.045
Qwen2.5-VL	NICGSlowDown	146.12	1561.60	7.28	0.014	0.007	0.003	0.001	0.025
	Verbose Images	252.46	2898.41	10.88	0.040	0.018	0.006	0.002	0.061

Appendix

Table 31. Resource-exhaustion effectiveness and caption quality on MS COCO for BLIP-2 and Qwen2.5-VL. We report generation length, estimated energy, and latency together with BLEU-1/2/3/4 and CIDEr. Clean rows are omitted, so the table compares caption-quality metrics among attack outputs rather than directly reporting clean-versus-attack degradation. All methods are evaluated under the same protocol. Bold marks the largest value for each model and metric.

Dataset	Model	NICGSlowDown	Verbose Images	Hidden Tail	JPPO
MS COCO	LLaVA-NeXT	0.970	0.972	0.962	0.978
	BLIP-2	0.973	0.971	0.958	0.965
	Qwen2.5-VL	0.955	0.941	0.948	0.962
	InstructBLIP	0.979	0.970	0.962	0.952
	MiniGPT-4	0.974	0.969	0.968	0.968
ImageNet	LLaVA-NeXT	0.971	0.973	0.977	0.988

Appendix

Table 32. Representation-space similarity measured by cosine similarity between CLIP image embeddings of the original and adversarial images.

Figure 10. Qualitative clean-versus-attacked example for BLIP-2. For the same image and task prompt, the clean input yields a relatively concise output, whereas the attacked input induces a substantially longer continuation. The figure additionally reports the realized Length, Estimated Energy, and Latency for this case. Qualitative BLIP-2 clean versus attacked example A qualitative BLIP-2 example compares clean and attacked outputs for the same image and task prompt. The attacked case produces a noticeably longer response and higher reported serving-cost metrics than the clean case.

Figure 11. Qualitative clean-versus-attacked example for Qwen2.5-VL. For the same image and task prompt, the attacked input drives the model toward a longer, more resource-intensive decoding trajectory than the clean input does. The figure additionally reports the realized Length, Energy, and Latency for this case. Qualitative Qwen2.5-VL clean versus attacked example A qualitative Qwen2.5-VL example compares clean and attacked outputs for the same image and task prompt. The attacked case shows a longer continuation and higher reported serving-cost metrics than the clean case.

Figure 12. Qualitative clean-versus-attacked example for LLaVA-NeXT. Compared with the clean input, the attacked input produces a visibly more persistent continuation under the same image and task prompt. The figure additionally reports the realized Length, Energy, and Latency for this case. Qualitative LLaVA-NeXT clean versus attacked example A qualitative LLaVA-NeXT example compares clean and attacked outputs for the same image and task prompt. The attacked case generates a more persistent continuation and higher reported serving-cost metrics than the clean case.

Figure 13. Qualitative clean-versus-attacked example for InstructBLIP. For the same image and task prompt, the attacked input yields a substantially longer response than the clean input and incurs a higher realized serving cost. The figure additionally reports the realized Length, Energy, and Latency for this case. Qualitative InstructBLIP clean versus attacked example A qualitative InstructBLIP example compares clean and attacked outputs for the same image and task prompt. The attacked case produces a substantially longer response and higher reported serving-cost metrics than the clean case.

Figure 14. Qualitative clean-versus-attacked example for MiniGPT-4. The attacked input induces a longer and more expensive decoding trajectory than the clean input for the same image and task prompt. The figure additionally reports the realized Length, Energy, and Latency for this case. Qualitative MiniGPT-4 clean versus attacked example A qualitative MiniGPT-4 example compares clean and attacked outputs for the same image and task prompt. The attacked case shows a longer continuation and higher reported serving-cost metrics than the clean case.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 179 papers

A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation
May 15, 2026Hao Yang, Zhuo Ma, Yang Liu +3VLM Robustness Adversarial Attacks on VLMs
Xidian University Xi’an, Shaanxi, China
Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
May 26, 2026Xiang Fang, Wanlong Fang, Changshuo WangVLM Robustness Adversarial Attacks on VLMs
School of Software Engineering, Huazhong University of Science and Technology · Nanyang Technological University, Singapore · University College London
Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models
Feb 16, 2026In Chong Choi, Jiacheng Zhang, Feng Liu +1Adversarial Attacks on VLMs Large Vision-Language Models
The University of Melbourne · The University of Adelaide

May 15, 2026J/K move · Enter open · S save

A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation

Hao Yang, Zhuo Ma, Yang Liu, Yilong Yang+2

Xidian University Xi’an, Shaanxi, China

VLM Robustness Adversarial Attacks on VLMs Large Vision-Language Models

Open paper AskPDF arXiv

Large vision-language models (LVLMs) have emerged as a powerful paradigm for multimodal intelligence, but their growing deployment also expands the attack surface of prompt injection. Despite this growing concern, existing attacks still suffer from a critical limitation: the injected prompt for one modality only steers the model's interpretation of that singular input. Alternatively, these attacks remain multimodal but fail to achieve cross-modal prompt perturbation. To bridge this gap, we introduce a novel cross-modal prompt injection attack CrossMPI, which can steer the model's interpretation of both textual and visual inputs via image-only prompt injection. Our design is underpinned by the following key breakthroughs. First, we turn the focus of the injected prompt perturbation optimization from the visual embedding space (typically with only

10^5

parameters) to the model hidden state space (for multimodal information integration and with

10^7

parameters). Then, two strategies are adopted to mitigate the optimization challenges posed by the larger parameter space. To constrain the optimized model parameter space, we introduce a layer selection strategy that identifies the layers most critical to multimodal integration. Interestingly, deviating from the past experience, our analysis reveals that the optimal layers for LVLM prompt perturbation reside in the middle of the model rather than the last. To constrain the image perturbation space, we propose a new distance-decremental perturbation budget assignment strategy that allocates budgets decrementally as the pixel distance to semantic-critical regions increases. Extensive experiments across multiple LVLMs and datasets show that our method significantly outperforms baseline approaches.

Abstract

Adversarial Attacks on VLMs

Explore similar work

A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation

Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization

Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models