Organizations: Guangzhou University School of Computer Science and Cyber Engineering Guangzhou, China · University of Macau Department of Computer and Information Science Macau, China · Guangzhou University School of Artificial Intelligence Guangzhou, China · Guangzhou University Cyberspace Institute of Advanced Technology Guangzhou, China
Resource-exhaustion attacks against autoregressive vision-language models (VLMs) typically assume unimodal threat models, treating the image branch as the primary optimization surface while holding user-visible prompts fixed. Even recent loop-centric variants remain confined to this single-channel paradigm, leaving the exploitation of availability unexplored as a cross-modal optimization problem over jointly controllable input surfaces. We introduce Joint Pixel-Prompt Optimization (JPPO), the first compound adversarial framework elevating the visible prompt to a first-class adversarial variable alongside image perturbations. Under a restricted joint-input threat model, JPPO performs coupled, stagewise optimization over both the pixel and prompt surfaces. This produces synergistic cost amplification, mechanistically distinct from loop-dependent failures, exhibiting negligible loop incidence in our experiments. Evaluating five open-source VLM families on MS COCO and ImageNet under an 8/255 infinity-norm budget, JPPO achieves over 4.6x latency and 5.3x energy amplification on Qwen2.5-VL-7B, and over 36.6x latency with 32.7x energy amplification on BLIP-2. This represents the strongest cost amplification among directly compared baselines while requiring substantially fewer optimization iterations. Ablations confirm this amplification arises from multimodal coordination rather than prompt length or isolated modalities. These findings reveal structural blind spots in current VLM serving defenses, motivating cost-aware robustness evaluation as a first-class security requirement for multimodal deployments.
Figures & tables
Figure 1. Comparison between a benign inference (top) and the proposed JPPO attack (bottom), along with the resulting cost curve (right). While the clean input yields a concise response, our joint optimization of bounded pixel perturbations and a budget-constrained interface-visible prompt context drives the model toward prolonged decoding, resulting in substantially higher cumulative latency and estimated energy under the same serving configuration. Benign versus JPPO inference comparison The benign inference case is shown at the top and the JPPO inference case at the bottom. The benign input produces a short response, while the JPPO-adversarial input produces a much longer continuation. A cost plot on the right indicates substantially larger cumulative latency and estimated energy for the attacked case under the same serving configuration.
Figure 2. Overview of JPPO. JPPO is a two-stage multimodal availability attack. In Stage I, the attacker maintains three prompt paths, namely visual, spatial, and semantic, and iteratively merges them into a diverse warm-up prompt while jointly updating a bounded pixel perturbation. The strongest measured Stage-I state, consisting of its paired warm-up prompt and perturbation, initializes Stage II, in which JPPO performs generation-driven prompt refinement through selective accumulation, together with further perturbation updates. Across both stages, the attack is guided by three availability-oriented objectives, and the final adversarial state is selected based on the realized resource amplification. JPPO pipeline overview A flow diagram illustrates the two-stage JPPO pipeline. Stage I builds a warm-up prompt from visual, spatial, and semantic path memories while jointly updating a bounded pixel perturbation. Stage II starts from the warm-up prompt and the strongest Stage I perturbation, then further refines the running prompt and pixel perturbation using model-generated feedback. The final adversarial state is selected based on the realized resource amplification.
Model
Method
MS COCO
ImageNet
Len.
E (J)
Lat. (s)
Len.
E (J)
Lat. (s)
BLIP-2
Clean
8.36
67.38
0.42
7.07
81.50
0.55
Noise
8.23
65.65
0.44
7.22
60.28
0.43
NICGSlowDown
86.76
567.03
4.46
103.41
663.15
5.01
Verbose Images
211.62
1432.61
12.07
231.31
1592.69
12.48
JPPO
373.23
2324.29
19.58
361.60
2665.96
20.16
Table 1. Full comparison on MS COCO and ImageNet. We report absolute output length, estimated-energy proxy E , and latency. Means are over three repeated inference runs and three dataset seeds; image-level standard deviations are reported in the appendix. Bold marks the largest non-Clean value in each setting.
Figure 3. Latency amplification across victim models, aggregated over MS COCO and ImageNet. Bars show mean amplification over the two datasets, with error bars indicating cross-dataset variation. Latency amplification bar chart A grouped bar chart compares latency amplification across BLIP-2, Qwen2.5-VL, LLaVA-NeXT, InstructBLIP, and MiniGPT-4. JPPO is the tallest bar for each model among the directly evaluated baselines. BLIP-2 shows the largest latency amplification, while LLaVA-NeXT shows comparatively smaller amplification. Error bars show variation between the two datasets.
Figure 4. White-box-to-black-box latency survivability. For each target, light-blue circles denote the four off-diagonal transfers, dark-blue circles their arithmetic mean, and orange diamonds the exact-model white-box result. Panels show MS COCO and ImageNet on a shared logarithmic scale; the dashed line marks the matched Clean baseline ( 1× ). White-box-to-black-box latency survivability Two side-by-side point plots compare JPPO latency amplification on MS COCO and ImageNet. For each of five target models, four light-blue points show transfer from the other source models, a dark-blue point shows their mean, and an orange diamond shows the exact-model white-box result. All transfer means remain above the Clean baseline but below their exact-model references, showing partial and asymmetric black-box target transfer.
Source Model
BLIP-2
Qwen2.5-VL
LLaVA-NeXT
InstructBLIP
MiniGPT-4
BLIP-2
19.58 / 20.16
16.98 / 18.74
22.49 / 24.29
12.33 / 13.02
17.71 / 8.94
Qwen2.5-VL
4.40 / 7.36
24.03 / 21.16
18.92 / 21.61
16.32 / 16.56
14.61 / 12.35
LLaVA-NeXT
3.73 / 3.38
23.68 / 21.30
27.53 / 27.18
14.73 / 12.80
17.97 / 19.19
InstructBLIP
0.92 / 1.65
15.97 / 17.04
18.45 / 18.46
17.58 / 21.43
12.22 / 12.86
MiniGPT-4
3.64 / 3.31
10.69 / 8.83
16.82 / 11.45
7.52 / 8.02
23.64 / 23.10
Table 2. JPPO latency across source–target pairs (s; MS COCO / ImageNet). Off-diagonal cells use no target-side adaptation; shaded diagonals are exact-model white-box references. Bold marks the highest latency for each target model and dataset.
Model
Method
Construction
Serving
Time (s)
E (kJ)
E (J)
Lat. (s)
BLIP-2
VI
316.04
41.95
1512.65
12.28
JPPO
582.26
53.04
2495.13
19.87
Qwen2.5-VL
VI
4813.97
183.85
2831.95
10.65
JPPO
3354.70
146.72
5163.02
22.60
LLaVA-NeXT
VI
8645.60
658.26
3214.15
11.91
Table 3. Construction and serving costs averaged across datasets. Values are equal-weight means over MS COCO and ImageNet. VI denotes Verbose Images. For JPPO, construction cost includes Stage I and one Stage-II trajectory initialized from the highest-energy Stage-I candidate; serving cost measures only the final selected request.
Method
Rel.
Info.
Red.
Use.
Clean
4.49
3.66
1.43
4.05
Neutral-long-prompt
4.29
4.01
2.04
3.92
Prompt-only
3.72
3.07
3.45
2.98
Verbose Images
2.82
2.15
4.08
2.91
JPPO
3.59
3.44
3.32
3.38
Table 4. Original-task output quality. GPT-5.5 scores for 450,000 candidate outputs are reported as equal-weight means across the ten model–dataset settings. Each output is evaluated against the corresponding original image and benign instruction, irrespective of the method-specific input used during generation. Rel., Info., Red., and Use. denote relevance, informativeness, redundancy, and usefulness; higher is better except for Red.
Model
Loop Inc. (%)
Longest Span (%)
Tail Cov. (%)
BLIP-2
1.90 / 1.01
2.04 / 1.26
1.31 / 0.52
Qwen2.5-VL
0.00 / 0.00
0.00 / 0.00
0.00 / 0.00
LLaVA-NeXT
0.00 / 0.00
0.47 / 0.58
0.00 / 0.00
InstructBLIP
0.00 / 0.00
0.01 / 0.13
0.00 / 0.00
MiniGPT-4
1.60 / 0.00
1.82 / 1.58
0.93 / 0.59
Table 5. Loop characterization of JPPO on MS COCO and ImageNet. Entries are reported as MS COCO / ImageNet. Inc., Span, and Cov. denote loop incidence, longest repeated-span ratio, and tail loop coverage, respectively. Lower values are better.
Figure 5. Optimization-space ablation on BLIP-2. Clean-normalized output-length, estimated-energy, and latency amplification for Pixel-only, Prompt-only, and Joint optimization on MS COCO and ImageNet. Optimization-space ablation bar chart Two bar charts compare Pixel-only, Prompt-only, and Joint optimization on MS COCO and ImageNet. Each panel reports amplification over clean for length, energy, and latency. Prompt-only is generally stronger than Pixel-only, while Joint optimization is strongest across all metrics and both datasets.
Model
Modality
Single Loss
Joint
Pixel
Prompt
Leos
Lalign
Lbot
BLIP-2
11.02
20.19
29.68
22.52
22.72
34.50
Qwen2.5-VL
3.91
3.36
4.12
4.68
4.71
5.77
LLaVA-NeXT
3.04
3.21
2.96
3.18
3.39
4.13
InstructBLIP
2.37
3.06
4.89
4.31
4.57
5.62
MiniGPT-4
4.27
4.33
5.11
5.46
5.72
6.80
Table 6. Cross-model ablation on MS COCO. Estimated-energy amplification over Clean. Bold marks the largest value per model.
Model
JPPO
Verbose Images
NICG SlowDown
Lingo Loop
(a) Nine-defense mean
BLIP-2
11.25
13.05
7.91
25.12
Qwen2.5-VL
2.78
4.80
2.03
6.04
LLaVA-NeXT
5.14
3.72
3.32
14.28
InstructBLIP
1.24
14.92
4.29
15.97
MiniGPT-4
4.69
21.95
10.98
25.93
Table 7. Cross-model defense degradation (%). Panel (a) averages the nine non-EarlyStop defenses, and Panel (b) reports EarlyStop. Values are equal-weight means over datasets and serving-cost metrics; lower means less mitigation. Bold marks the smallest degradation per model and the smallest overall average.
Figure 6. Per-defense degradation across five VLMs (%). Rows denote defenses and columns denote attacks. Each cell is an equal-weight mean over the five models, MS COCO and ImageNet, and the three serving-cost metrics (output length, estimated energy, and latency). Darker blue indicates greater degradation; lower means less mitigation, and negative values indicate increased cost. Per-defense degradation heatmap A heatmap with ten defenses as rows and four attack methods as columns. Each cell is annotated with the corresponding degradation percentage. Lighter blue indicates lower degradation and darker blue indicates higher degradation, with the color bar spanning approximately minus 2 percent to 25 percent. Negative values are displayed directly in the corresponding cells.
Model
Method
P95
Overload
GPU-h/1K
MS COCO
Qwen2.5-VL
VI
1.82 ×
[9%,10%]
1.76
JPPO
1.94 ×
[4%,5%]
6.60
BLIP-2
VI
37.1 ×
[0.75%,1.00%]
3.05
JPPO
55.9 ×
[0.50%,0.75%]
4.92
ImageNet
Table 8. Uncontrolled four-worker FIFO impact at 80% benign utilization. VI denotes Verbose Images. Each overload bracket reports the largest stable and smallest overloaded offered attack shares. Bold marks the stronger effect within each model–dataset setting: higher P95 inflation and GPU-h/1K, and a lower overload threshold.
Model
Policy
JPPO Admission
Attacker Worker Share
Load ρ
Benign API P95 E2E Inflation
Qwen2.5-VL
Open
100.00% / 100.00%
23.5% / 25.3%
1.04 / 1.07
42.9 × / 47.6 ×
Request-RL
91.34% / 87.80%
22.1% / 23.6%
1.01 / 1.02
13.1 × / 16.2 ×
Safety+Request-RL
90.15% / 86.83%
21.8% / 23.2%
1.00 / 1.02
14.5 × / 18.0 ×
EarlyStop+Request-RL
91.34% / 87.80%
21.7% / 22.9%
1.00 / 1.01
12.4 × / 15.4 ×
BLIP-2
Open
100.00% / 100.00%
27.2% / 24.1%
1.10 / 1.05
108.6 × / 94.2 ×
Request-RL
82.74% / 89.55%
24.4% / 22.3%
1.04 / 1.01
80.7 × / 76.8 ×
Table 9. Fixed-trace local API evaluation under serving controls. Entries are MS COCO / ImageNet. Admission is measured before target-model execution, while P95 reports benign end-to-end latency inflation.
Model
Method
MS COCO
ImageNet
Len.
Tokens
E (J)
Lat (s)
Len.
Tokens
E (J)
Lat (s)
Qwen2.5-VL-7B
Clean
70.51
82.13
825.21
3.16
74.13
88.47
893.15
3.68
LingoLoop
435.29
501.94
5056.13
18.75
439.18
500.18
5014.32
18.01
JPPO
456.15
510.78
5528.41
23.61
440.68
502.32
5184.18
22.46
InstructBLIP
Clean
53.84
65.35
675.21
3.61
59.17
71.56
857.44
5.08
LingoLoop
417.61
495.20
5001.62
34.23
409.92
488.36
4751.66
33.71
Table 10. Matched comparison with LingoLoop. All methods use greedy decoding and a 512-token cap; we report word and token lengths, estimated energy, and latency. Bold marks the largest non-Clean value for each model, dataset, and metric.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Dataset
Build Time (s)
Build Energy (kJ)
Online Energy (J)
Online Latency (s)
BLIP-2
MS COCO
576.41 / 304.92
50.14 / 38.30
2324.29 / 1432.61
19.58 / 12.07
ImageNet
588.10 / 327.16
55.93 / 45.60
2665.96 / 1592.69
20.16 / 12.48
Qwen2.5-VL
MS COCO
3554.83 / 4896.92
154.02 / 186.28
5458.59 / 2898.41
24.03 / 10.88
ImageNet
3154.57 / 4731.01
139.42 / 181.41
4867.44 / 2765.49
21.16 / 10.41
LLaVA-NeXT
MS COCO
7574.42 / 8538.11
597.51 / 655.17
6520.12 / 3219.71
27.53 / 11.78
ImageNet
7416.33 / 8753.08
520.00 / 661.35
5706.21 / 3208.59
27.18 / 12.04
Appendix
Table 11. Dataset-specific offline construction and online serving costs. Values are reported as JPPO / Verbose Images. Construction metrics are attacker-side; online metrics measure the final selected request. Main-text Table 3 reports equal-weight means over MS COCO and ImageNet.
Defense
BLIP-2
Qwen2.5-VL
LLaVA-NeXT
InstructBLIP
MiniGPT-4
JPEG (Q=95)
-0.46
-0.71
2.52
3.62
2.59
JPEG (Q=75)
15.70
0.11
4.63
4.06
9.55
JPEG (Q=50)
25.79
5.62
9.01
1.81
0.78
Resize restoration
16.89
4.68
14.58
-1.96
9.12
5-bit quantization
-4.28
-1.06
4.92
-9.01
-1.82
4-bit quantization
30.50
0.45
1.69
4.62
-0.12
Appendix
Table 12. Model-level defense degradation for JPPO (%). Each entry averages both datasets and the three serving-cost metrics. For each model, the mean of the first nine rows and the EarlyStop row correspond to the JPPO entries in Panels (a) and (b), respectively, of Table 7 .
Defense setting
Len.
E
Lat.
Nine non-EarlyStop defenses
5.19
4.96
4.92
EarlyStop
2.55
5.81
2.63
Appendix
Table 13. Metric-level defense degradation for JPPO (%). The non-EarlyStop row averages across nine defenses, five target models, and both datasets, while EarlyStop is averaged across five target models and both datasets. The equal-weight means across output length, estimated energy, and latency are 5.02% and 3.66%, respectively, matching the JPPO averages reported in Table 7 .
Model
Method
MS COCO
ImageNet
Len.
E (J)
Lat (s)
Len.
E (J)
Lat (s)
BLIP-2
Clean
8.36
67.38
0.42
7.07
81.50
0.55
Noise
8.23
65.65
0.44
7.22
60.28
0.43
NICGSlowDown
86.76
567.03
4.46
103.41
663.15
5.01
Verbose Images
211.62
1432.61
12.07
231.31
1592.69
12.48
Hidden Tail
52.05
351.52
2.31
50.13
309.45
2.29
Appendix
Table 14. Expanded full comparison including Hidden Tail under the unified low-perturbation top- p /512-token protocol. This table is provided for completeness; the main text excludes Hidden Tail from the directly compared baseline table because its original evaluation regime is not directly aligned with the main low-perturbation comparison. Bold marks the largest non-Clean value for each model, dataset, and metric.
Schedule
Len.
E
Lat.
Random
4.46 ×
4.96 ×
4.28 ×
Phase-based
5.95 ×
6.06 ×
5.40 ×
Appendix
Table 15. Stage-I schedule comparison. Values are averaged over the five evaluated models and both datasets under the main evaluation protocol.
Model
Strategy
Traj.
Build Time (s)
Build E (kJ)
Final E (J)
Δ Final E
Qwen2.5-VL
Highest-energy candidate
1
3354.70
146.72
5163.02
0.00%
Random top-3
1
3381.46
147.95
5201.64
+0.75%
Random top-5
1
3319.28
145.18
4937.83
−4.36%
Refine all top-3
3
7186.91
311.27
5384.76
+4.30%
Refine all top-5
5
11046.38
488.11
5427.19
+5.12%
BLIP-2
Highest-energy candidate
1
582.26
53.04
2495.13
0.00%
Appendix
Table 16. Stage-II candidate-selection ablation. Values are equal-weight means over MS COCO and ImageNet. Build cost includes Stage I once and all Stage-II trajectories executed by each strategy; final E measures only the selected request.
Model
Method
MS COCO
ImageNet
Len.
E (J)
Lat (s)
Len.
E (J)
Lat (s)
BLIP-2
Original
8.36
67.38
0.42
7.07
81.50
0.55
Neutral
21.93
137.21
1.01
24.17
152.75
1.11
JPPO
373.23
2324.29
19.58
361.60
2665.96
20.16
Qwen2.5-VL
Original
80.24
945.97
4.61
81.52
912.43
4.53
Neutral
102.54
1257.80
5.09
115.12
1468.24
5.27
Appendix
Table 17. Effect of different prompt settings and attack methods on multiple models. Each model row shows the results for Original Prompt, Neutral Prompt, and JPPO under the same evaluation protocol. The Neutral Prompt uses the fixed length-control prompt described in Appendix C.3 and is included to test whether JPPO’s gains can be explained by prompt length alone. Bold marks the largest value for each model, dataset, and metric.
Method
Len.
E (J)
Lat. (s)
MS COCO
Clean
8.36
67.38
0.42
Pixel-only
82.59
742.61
5.73
Prompt-only
212.43
1360.29
14.82
JPPO (Joint)
373.23
2324.29
19.58
ImageNet
Appendix
Table 18. Exact values for the optimization-space ablation on BLIP-2. This table complements the amplification visualization in Figure 5 .
Loss Components
MS COCO
ImageNet
Leos
Lalign
Lbot
Len.
E (J)
Lat. (s)
Len.
E (J)
Lat. (s)
212.43
1360.29
14.82
192.60
1248.02
14.76
✓
282.03
2000.06
18.16
313.67
1958.06
18.51
✓
239.67
1517.39
16.24
223.90
1471.54
15.47
✓
244.07
1530.97
16.12
230.67
1539.71
16.85
✓
✓
283.07
2165.65
18.29
275.53
1863.73
17.17
Appendix
Table 19. Ablation analysis of the three availability-oriented loss terms on BLIP-2: Leos for prefix stop suppression, Lalign for cross-modal alignment suppression, and Lbot for bottleneck regularization. We report generation length, estimated-energy proxy E , and latency across MS COCO and ImageNet. The row with no checked loss component corresponds to prompt-only refinement without pixel-space availability losses.
Source Model
BLIP-2
Qwen2.5-VL
LLaVA-NeXT
InstructBLIP
MiniGPT-4
MS COCO
BLIP-2
373.23
361.21
345.36
209.59
226.40
Qwen2.5-VL
111.02
441.23
298.74
293.08
222.87
LLaVA-NeXT
69.20
446.50
369.10
217.91
256.93
InstructBLIP
26.90
343.22
277.93
250.92
198.30
MiniGPT-4
76.70
274.51
274.46
104.40
385.35
Appendix
Table 20. JPPO output length across all source–target model pairs. Rows are construction models, columns are evaluation targets, and shaded diagonal cells denote exact-model white-box references. Bold marks the highest output length for each target model and dataset.
Source Model
BLIP-2
Qwen2.5-VL
LLaVA-NeXT
InstructBLIP
MiniGPT-4
MS COCO
BLIP-2
2324.29
3509.62
5178.87
2296.14
3032.69
Qwen2.5-VL
642.49
5458.59
4191.65
2844.30
2838.93
LLaVA-NeXT
435.85
5083.85
6520.12
2309.90
3605.19
InstructBLIP
101.26
3254.06
3751.56
3476.18
2695.34
MiniGPT-4
328.93
2391.89
3912.61
1037.25
5378.77
Appendix
Table 21. JPPO estimated energy across all source–target model pairs (J). Rows are construction models, columns are evaluation targets, and shaded diagonal cells denote exact-model white-box references. Bold marks the highest estimated energy for each target model and dataset.
Figure 7. Mean token-level novelty ratio before and after JPPO across five evaluated VLMs. The statistic is computed on normalized generated tokens using a sliding window of 32. Higher values indicate more novel continuation behavior, whereas lower values indicate stronger repetition. JPPO reduces novelty relative to benign generation, but the attacked outputs remain well above collapse-to-loop behavior on most models, consistent with the two-dataset loop-incidence results in Table 5 . Novelty-ratio comparison before and after attack A bar chart compares the mean novelty ratio before and after JPPO across five VLMs. For every model, the attacked outputs have lower novelty than the clean outputs, but the post-attack values remain moderate rather than collapsing to near-zero repetition.
Figure 8. Sensitivity of JPPO to the bottleneck dimensionality ratio on BLIP-2 across MS COCO and ImageNet. The ratio ρ determines the compressed bottleneck dimension d′ in Eq. ( 13 ). The left column shows the low-ratio regime, and the right column shows the broader global trend. The top row reports output-length amplification, while the bottom row reports latency amplification. Bottleneck-ratio sensitivity across two datasets A four-panel line chart showing sensitivity to the bottleneck dimensionality ratio on BLIP-2 across MS COCO and ImageNet. The left column shows the low-ratio regime, and the right column shows the broader global trend. The top row plots output-length amplification, and the bottom row plots latency amplification.
ρ
MS COCO
ImageNet
Len.
E (J)
Lat (s)
Len.
E (J)
Lat (s)
Clean
8.36
67.38
0.42
7.07
81.50
0.55
0.02
294.09
1971.62
18.24
248.62
1641.15
16.21
0.04
371.82
2414.18
19.49
369.11
2410.97
19.42
0.06
277.87
1962.28
17.89
249.18
1601.92
15.94
0.08
288.91
2026.17
17.90
283.01
1968.11
17.53
Appendix
Table 22. Full sensitivity results for the bottleneck dimensionality ratio ρ on BLIP-2 across MS COCO and ImageNet. The ratio determines the compressed representation size d′=max(⌊ρD⌋,1) in the bottleneck regularization objective. We report mean output length, estimated energy, and latency under each setting.
Figure 9. Gradient-based evidence localization for clean and JPPO-adversarial inputs. Each row shows a representative BLIP-2 example. From left to right, we present the clean image, the Grad-CAM overlay on the clean image, the adversarial image obtained by applying the bounded pixel perturbation, and the Grad-CAM overlay on the adversarial image. Compared with the clean input, the adversarial counterpart alters the spatial distribution of model-relevant visual evidence, often making it weaker, more diffuse, or more fragmented, rather than tightly concentrated in compact, object-centric regions. For readability, only a portion of the generated output is shown. Grad-CAM comparison for clean and attacked inputs Several BLIP-2 examples are shown in rows. Each row contains a clean image, its Grad-CAM overlay, an adversarial image, and its Grad-CAM overlay. Relative to the clean cases, the attacked cases typically show weaker, more diffuse, or more fragmented evidence localization rather than compact object-centered hotspots.
ϵ
MS COCO
ImageNet
Len.
E (J)
Lat (s)
Len.
E (J)
Lat (s)
Clean
8.36
67.38
0.42
7.07
81.50
0.55
2/255
196.83
1286.49
10.07
189.30
1317.21
11.72
4/255
278.20
1852.19
14.92
277.47
1904.15
15.67
8/255
373.23
2324.29
19.58
361.60
2665.96
20.16
16/255
318.83
2072.89
16.72
363.26
2357.01
19.37
Appendix
Table 23. Impact of perturbation magnitude ϵ on attack effectiveness on BLIP-2 over the MS COCO and ImageNet datasets. Bold marks the largest value for each dataset and metric.
Stage 1 Weights
Stage 2 Weights
MS COCO
ImageNet
Len.
E (J)
Lat (s)
Len.
E (J)
Lat (s)
–
–
8.36
67.38
0.42
7.07
81.50
0.55
(1,1,1)
(1,1,1)
360.90
2040.80
19.77
344.62
2052.23
19.03
(2,2,2)
(2,2,2)
318.13
2091.37
18.46
301.94
1939.41
17.96
(2,1,1)
(1,1,1)
240.07
1601.58
15.87
242.09
1625.15
16.34
(2,2,1)
(1,1,1)
306.93
1872.58
18.06
321.02
1911.64
18.84
Appendix
Table 24. Sensitivity of JPPO to the loss-weight configuration on BLIP-2. Each row reports the Stage-I and Stage-II weights for (Leos,Lalign,Lbot) . The shaded row with “–” weights is the main-evaluation Clean reference from Table 1 ; the shaded row with (2,1,1) and (2,3,1) is the main-evaluation Default JPPO reference. All unshaded rows report a separate small-scale weight-sensitivity sweep. Bold marks the highest estimated energy within each dataset among the tested weight configurations.
Stage 1
Stage 2
MS COCO
ImageNet
Len.
E (J)
Lat (s)
Len.
E (J)
Lat (s)
50
50
338.27
2030.03
16.72
343.93
1985.02
16.38
50
100
307.69
1964.21
16.20
270.32
1748.71
13.74
100
100
306.04
2067.13
17.62
321.42
1899.38
15.43
100
150
293.57
1999.24
16.23
339.14
2170.09
18.37
100
200
260.07
1633.25
13.07
293.57
1932.44
17.08
Appendix
Table 25. Effect of prompt length limits on generation length, estimated energy, and latency on BLIP-2 over the MS COCO and ImageNet datasets. Bold marks the largest value for each dataset and metric.
Dataset
Metric
32
64
128
256
512
1024
MS COCO
Len
30.50
56.77
87.57
179.43
373.23
726.83
E (J)
206.01
352.29
619.63
1283.68
2324.29
5493.30
Lat (s)
1.64
2.58
4.29
8.64
19.58
39.35
ImageNet
Len.
31.21
52.80
101.16
191.64
361.60
756.83
E (J)
216.02
358.18
669.23
1168.57
2665.96
5601.88
Lat (s)
1.28
2.14
4.55
7.95
20.16
41.92
Appendix
Table 26. Ablation of maximum generation caps for JPPO on the MS COCO and ImageNet datasets.
Stage 1
Stage 2
MS COCO
ImageNet
Len.
E (J)
Lat (s)
Len.
E (J)
Lat (s)
0
0
8.36
67.38
0.42
7.07
81.50
0.55
0
100
225.00
1359.96
9.78
202.79
1445.34
10.92
0
200
226.87
1424.19
10.28
205.83
1472.13
11.72
100
0
277.43
1812.72
14.54
250.03
1767.54
13.98
200
0
313.62
1968.65
16.09
202.81
1474.07
11.22
Appendix
Table 27. Ablation study on stage-wise iterations for BLIP-2 on MS COCO and ImageNet. Bold marks the largest value for each dataset and metric.
Stage 1
Stage 2
MS COCO
ImageNet
Len.
E (J)
Lat (s)
Len.
E (J)
Lat (s)
0
0
8.36
67.38
0.42
7.07
81.50
0.55
10
10
147.97
923.87
7.37
77.07
447.99
4.94
20
20
194.43
1169.28
9.81
246.65
1560.44
12.01
50
50
238.27
1710.68
14.12
224.37
1530.08
12.11
50
100
298.73
2137.28
17.39
322.37
2119.76
17.48
Appendix
Table 28. Resource usage under different stage-wise iteration schedules for BLIP-2 on MS COCO and ImageNet.
Decoding
Method
MS COCO
ImageNet
Len.
E (J)
Lat. (s)
Len.
E (J)
Lat. (s)
Greedy
Clean
7.72
65.26
0.37
7.81
66.14
0.39
Noise
7.81
66.16
0.41
7.34
63.35
0.38
NICGSlowDown
171.98
1097.11
7.71
226.18
1358.81
8.00
Verbose Images
303.84
2086.49
10.06
328.89
2073.88
10.91
Hidden Tail
76.10
564.16
3.88
89.01
549.19
3.45
Appendix
Table 29. Comparison of different decoding strategies on BLIP-2. We report absolute generation length, estimated-energy proxy E , and latency on MS COCO and ImageNet. Bold marks the largest non-Clean value for each decoding strategy, dataset, and metric.
Model
Method
MS COCO
ImageNet
Len.
E (J)
Lat (s)
Len.
E (J)
Lat (s)
BLIP-2
Clean
2.23
13.26
0.07
1.86
16.71
0.11
Noise
2.10
12.79
0.09
1.87
12.91
0.08
NICGSlowDown
145.91
209.41
4.05
155.94
357.09
4.81
Verbose Images
176.07
494.18
3.89
172.10
572.65
4.69
JPPO
97.62
316.74
2.19
88.01
443.13
3.11
Appendix
Table 30. Standard deviation of resource exhaustion attacks on MS COCO and ImageNet. We report the sample standard deviation across image-level means for generation length, estimated energy, and latency, where each image-level value is first averaged over three repeated runs with the same input-method pair.
Model
Method
Attack Effect
Caption Quality
Len.
E (J)
Lat (s)
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
BLIP-2
NICGSlowDown
86.76
567.03
4.46
0.011
0.004
0.002
0.001
0.023
Verbose Images
211.62
1432.61
12.07
0.027
0.013
0.003
0.002
0.072
JPPO
373.23
2324.29
19.58
0.022
0.010
0.002
0.001
0.045
Qwen2.5-VL
NICGSlowDown
146.12
1561.60
7.28
0.014
0.007
0.003
0.001
0.025
Verbose Images
252.46
2898.41
10.88
0.040
0.018
0.006
0.002
0.061
Appendix
Table 31. Resource-exhaustion effectiveness and caption quality on MS COCO for BLIP-2 and Qwen2.5-VL. We report generation length, estimated energy, and latency together with BLEU-1/2/3/4 and CIDEr. Clean rows are omitted, so the table compares caption-quality metrics among attack outputs rather than directly reporting clean-versus-attack degradation. All methods are evaluated under the same protocol. Bold marks the largest value for each model and metric.
Dataset
Model
NICGSlowDown
Verbose Images
Hidden Tail
JPPO
MS COCO
LLaVA-NeXT
0.970
0.972
0.962
0.978
BLIP-2
0.973
0.971
0.958
0.965
Qwen2.5-VL
0.955
0.941
0.948
0.962
InstructBLIP
0.979
0.970
0.962
0.952
MiniGPT-4
0.974
0.969
0.968
0.968
ImageNet
LLaVA-NeXT
0.971
0.973
0.977
0.988
Appendix
Table 32. Representation-space similarity measured by cosine similarity between CLIP image embeddings of the original and adversarial images.
Figure 10. Qualitative clean-versus-attacked example for BLIP-2. For the same image and task prompt, the clean input yields a relatively concise output, whereas the attacked input induces a substantially longer continuation. The figure additionally reports the realized Length, Estimated Energy, and Latency for this case. Qualitative BLIP-2 clean versus attacked example A qualitative BLIP-2 example compares clean and attacked outputs for the same image and task prompt. The attacked case produces a noticeably longer response and higher reported serving-cost metrics than the clean case.
Figure 11. Qualitative clean-versus-attacked example for Qwen2.5-VL. For the same image and task prompt, the attacked input drives the model toward a longer, more resource-intensive decoding trajectory than the clean input does. The figure additionally reports the realized Length, Energy, and Latency for this case. Qualitative Qwen2.5-VL clean versus attacked example A qualitative Qwen2.5-VL example compares clean and attacked outputs for the same image and task prompt. The attacked case shows a longer continuation and higher reported serving-cost metrics than the clean case.
Figure 12. Qualitative clean-versus-attacked example for LLaVA-NeXT. Compared with the clean input, the attacked input produces a visibly more persistent continuation under the same image and task prompt. The figure additionally reports the realized Length, Energy, and Latency for this case. Qualitative LLaVA-NeXT clean versus attacked example A qualitative LLaVA-NeXT example compares clean and attacked outputs for the same image and task prompt. The attacked case generates a more persistent continuation and higher reported serving-cost metrics than the clean case.
Figure 13. Qualitative clean-versus-attacked example for InstructBLIP. For the same image and task prompt, the attacked input yields a substantially longer response than the clean input and incurs a higher realized serving cost. The figure additionally reports the realized Length, Energy, and Latency for this case. Qualitative InstructBLIP clean versus attacked example A qualitative InstructBLIP example compares clean and attacked outputs for the same image and task prompt. The attacked case produces a substantially longer response and higher reported serving-cost metrics than the clean case.
Figure 14. Qualitative clean-versus-attacked example for MiniGPT-4. The attacked input induces a longer and more expensive decoding trajectory than the clean input for the same image and task prompt. The figure additionally reports the realized Length, Energy, and Latency for this case. Qualitative MiniGPT-4 clean versus attacked example A qualitative MiniGPT-4 example compares clean and attacked outputs for the same image and task prompt. The attacked case shows a longer continuation and higher reported serving-cost metrics than the clean case.
Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.
Large vision-language models (LVLMs) have emerged as a powerful paradigm for multimodal intelligence, but their growing deployment also expands the attack surface of prompt injection. Despite this growing concern, existing attacks still suffer from a critical limitation: the injected prompt for one modality only steers the model's interpretation of that singular input. Alternatively, these attacks remain multimodal but fail to achieve cross-modal prompt perturbation. To bridge this gap, we introduce a novel cross-modal prompt injection attack CrossMPI, which can steer the model's interpretation of both textual and visual inputs via image-only prompt injection. Our design is underpinned by the following key breakthroughs. First, we turn the focus of the injected prompt perturbation optimization from the visual embedding space (typically with only 105 parameters) to the model hidden state space (for multimodal information integration and with 107 parameters). Then, two strategies are adopted to mitigate the optimization challenges posed by the larger parameter space. To constrain the optimized model parameter space, we introduce a layer selection strategy that identifies the layers most critical to multimodal integration. Interestingly, deviating from the past experience, our analysis reveals that the optimal layers for LVLM prompt perturbation reside in the middle of the model rather than the last. To constrain the image perturbation space, we propose a new distance-decremental perturbation budget assignment strategy that allocates budgets decrementally as the pixel distance to semantic-critical regions increases. Extensive experiments across multiple LVLMs and datasets show that our method significantly outperforms baseline approaches.
Large Vision-Language Models (LVLMs) have transformed multi-modal understanding, excelling in tasks like image captioning and visual question answering by integrating visual and textual inputs. However, their robustness against adversarial attacks, particularly those exploiting both modalities, remains underexplored, posing risks to critical applications like autonomous driving and content moderation. Existing attacks focus on single modalities or require impractical white-box access, limiting their real-world relevance. In this paper, we introduce Multi-Modal Adversarial Synergy, a groundbreaking framework that crafts universal, black-box multi-modal attacks against LVLMs. MMAS simultaneously generates a texture scale-constrained universal adversarial perturbation for images and a learnable prompt perturbation for text, optimized jointly using only model queries. The image perturbation leverages wavelet-based texture constraints to ensure imperceptibility and robustness across diverse visual inputs. The text perturbation, constrained by an L-norm in the embedding space, maintains semantic coherence while steering outputs toward a target. A novel cross-modal regularization term aligns the perturbations' gradient directions, enhancing their synergistic impact and transferability across tasks and models. Extensive experiments show the strong universal adversarial capabilities of our proposed attack with prevalent LVLMs.
Xiang Fang, Wanlong Fang, Changshuo Wang
School of Software Engineering, Huazhong University of Science and Technology · Nanyang Technological University, Singapore · University College London
Multi-turn jailbreak attacks have proven effective against text-only large language models (LLMs), where malicious content is gradually introduced to bypass safety alignment. However, effectively extending such attacks to large vision-language models (LVLMs) remains underexplored. In this paper, we find that naively incorporating visual inputs can make multi-turn jailbreaks easier to defend against; for example, overly malicious visual content will easily trigger the defense mechanism in safety-aligned LVLMs, resulting in more conservative responses. Based on this finding, we propose multi-turn adaptive prompting attack (MAPA) that 1) at each turn, alternates text-vision attack actions to elicit the most malicious response; and 2) across turns, adjusts the attack trajectory through iterative back-and-forth refinement to gradually amplify response maliciousness. This two-level design enables MAPA to consistently outperform state-of-the-art methods, improving attack success rates by 15-30% on recent benchmarks against LLaVA-v1.6-Mistral-7B, Qwen2.5-VL-7B-Instruct, Llama-3.2-Vision-11B-Instruct and GPT-4o-mini. Our code is available at: https://github.com/thomaschoi143/MAPA.
In Chong Choi, Jiacheng Zhang, Feng Liu +1
The University of Melbourne · The University of Adelaide