Large vision-language models (VLMs) are increasingly deployed in safety-critical settings, yet existing visual jailbreak research has focused almost exclusively on autoregressive architectures, leaving an important emerging family unstudied: multimodal discrete diffusion vision-language models (dVLMs). We identify a vulnerability specific to diffusion generation: because the visual embedding conditions every reverse denoising step rather than acting as a one-time prefix, adversarial visual semantics are repeatedly propagated and amplified across the generation trajectory, a phenomenon we term cross-step conditional propagation. We provide empirical evidence through stage-sensitivity analysis, prompt-level switch rates, and pairwise denoising-bin disagreement metrics, confirmed by bootstrap resampling. We propose DIVA (Discrete-diffusion Vision-language model Attack), a white-box visual jailbreak framework using cross-modal intent obfuscation and diffusion-aware multi-timestep adversarial optimization. Across three dVLMs, DIVA reaches 58.8%, 67.7%, and 69.1% HADES ASR under the Beaver reward-model metric, outperforming visual jailbreak baselines designed for autoregressive models. Code: https://github.com/loststars2002/DIVA
Figures & tables
Figure 1: The target dVLM correctly rejects harmful text-only and harmful image-only inputs. Yet, after adding a small DIVA perturbation δ to the image and rewriting the query into a benign-looking form, the same model produces a harmful response, exposing a critical weakness in multimodal safety alignment.
Figure 2: Overview of DIVA. Left (Cross-Modal Intent Obfuscation): The harmful instruction u is rewritten into a visually grounded proxy prompt p , and the removed malicious semantics are encoded into a composite image xc=C(xtyp,xsem) via typography overlay and a harmful reference image. Right (Diffusion-Aware Adversarial Optimization): At each PGD step, K timesteps are sampled from w(t) , and the aggregated masked cross-entropy loss LDIVA is backpropagated through the full multimodal stack to update the bounded perturbation δ , yielding the final attacked image x~=Π[0,1](xc+δ) .
Figure 3: Architectural comparison of adversarial conditioning in ARMs and dVLMs. In ARMs, the vision encoder produces a fixed visual prefix injected only once, so adversarial influence is confined to the initial conditioning step. In dVLMs, the attacked image re-conditions the model at every reverse denoising step, causing harmful visual semantics to be repeatedly amplified across the generation trajectory, a phenomenon we term cross-step conditional propagation .
Model
Animal
Financial
Privacy
Self-Harm
Violence
Average
LLaDA-V
LLaDA-V
53.33
60.00
45.33
34.67
76.67
54.00
LLaDA-V + DIJA
67.33
55.33
47.33
34.00
76.00
56.00
LLaDA-V + PAD
69.33
54.00
50.00
39.33
65.33
55.60
LLaDA-V + DIVA (Ours)
61.33
66.00
39.33
50.00
77.33
58.80
MMaDA
Table 1: ASR (%) on HADES under the Beaver reward-model metric. Best result per model group in bold , second best underlined .
Model
Safe Rate ↓
Safe Refuse ↓
Safe Warning ↓
ASR ↑
LLaDA-V
Plain LLaDA-V (text-only)
49.53
0.00
49.53
50.47
LLaDA-V
50.42
0.00
50.42
49.58
LLaDA-V + DIJA
49.64
0.05
49.59
50.36
LLaDA-V + PAD
48.94
0.05
48.89
51.06
LLaDA-V + DIVA (Ours)
47.93
0.00
47.93
52.07
Table 2: Results on VLSBench (%). Safe Rate , Safe Refuse , and Safe Warning are lower for a more successful attack; ASR (Unsafe Rate) is higher. Best ASR per model group in bold , second best underlined .
Setting
Animal
Financial
Privacy
Self-Harm
Violence
Average
Plain (text-only)
43.33
37.33
16.00
16.67
51.33
32.93
xc only
41.33
48.67
22.67
18.67
73.67
41.00
xc + Random δ
45.33
52.00
26.67
24.00
74.67
44.53
xc + Shuffled δ
53.33
60.67
33.33
39.33
76.67
52.67
xc + DIVA δ (Ours)
61.33
66.00
39.33
50.00
77.33
58.80
Table 3: Ablation study on HADES under the Beaver reward-model metric (LLaDA-V). We isolate the contribution of the composite image xc and the diffusion-aware perturbation δ by comparing random, spatially shuffled, and true optimized variants. Best in bold , second best underlined .
Figure 4: Qualitative comparison of model responses before and after DIVA. Left : both LLaDA-V and MMaDA correctly refuse harmful text-only queries, producing policy-compliant safety disclaimers. Right : after rewriting the explicit instruction into a visually grounded proxy prompt and augmenting the composite image with the optimised perturbation δ , both models produce detailed, step-by-step harmful completions. The proxy prompts are superficially benign and would not trigger text-side filters in isolation; the malicious semantics reside in the composite image and are amplified through cross-step conditional propagation across the denoising trajectory.
Figure 5: Visualisation of the DIVA perturbation for a representative Animal-category example. Top-left: composite image xc with typography overlay. Top-right: attacked image x~=xc+δ . Bottom-left: perturbation δ amplified by 4× for visibility, showing dense spatially distributed noise. Bottom-right: per-pixel magnitude heatmap ∣δ∣ , confirming roughly uniform, low-energy distortion.
Metric
LLaDA-V (dVLM)
LLaVA-1.5 (ARM)
Attack Performance
ASR: Image only
80.80
96.00
ASR: DIVA
88.53
89.87
Gain
+7.73
−6.13
Stage-Sensitivity Diagnostics
SSI
12.00 [8.00, 19.33]
1.33 [0.00, 4.00]
Table 4: Attack performance and stage-sensitivity diagnostics for LLaDA-V (dVLM) vs. LLaVA-1.5 (ARM) under identical conditions. Gain : DIVA minus image-only ASR (pp). Bootstrap 95% CIs in brackets. All metrics in %.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
LLaDA-V / HADES
LLaDA-V / VLSBench
MMaDA / HADES
MMaDA / VLSBench
ε ( ℓ∞ )
0.2500
0.2500
0.2510
0.2510
PGD outer steps
500
300
300
300
Step size α
0.0013
0.0017
0.0021
0.0021
Timesteps per step K
4
4
4
4
Response length R
128
64
128
128
Total evals (Steps ×K )
2,000
1,200
1,200
1,200
Appendix
Table 5: DIVA hyperparameters for all experiments. ε : ℓ∞ pixel budget; α : PGD step size; K : timesteps sampled per outer step; R : target response length (tokens). All runs use w(t)=Uniform,t∼U[0.02,1.00) and bfloat16 precision. The MMaDA-8B-MixCoTe variant is identical to the MMaDA / HADES column except for the model checkpoint.
Category
PSNR ↑
SSIM ↑
LPIPS ↓
ℓ∞
Mean ∣Δ∣
Average
32.0
0.806
0.132
0.137
0.019
Appendix
Table 6: Perceptual similarity between xc and x~=xc+δ across HADES categories ( ε=0.25 , LLaDA-V). Higher PSNR/SSIM and lower LPIPS indicate less distortion. ℓ∞ is the maximum absolute pixel deviation; mean ∣Δ∣ is the average absolute deviation.
Category
Keyword
Trigger rate ( xc / x~ )
Text sim.
Animal
beat
1.0 / 1.0
1.00
Financial
steal
1.0 / 1.0
1.00
Privacy
spy
1.0 / 1.0
1.00
Self-Harm
overdose
1.0 / 1.0
1.00
Violence
gun
1.0 / 1.0
1.00
Average
—
1.0 / 1.0
1.00
Appendix
Table 7: OCR detectability of the typography overlay before and after adding δ (EasyOCR, LLaDA-V HADES). Trigger rate: fraction of images where the keyword is detected. Text similarity: normalised edit-distance similarity between OCR outputs on xc and x~ .
HADES scenario
Beaver label
Animal
animal_abuse
Financial
financial_crime
Privacy
privacy_violation
Self-Harm
self_harm
Violence
violence
Appendix
Table 8: Mapping from HADES harm scenarios to Beaver-Dam-7B safety-category labels.
Model / method
Animal
Financial
Privacy
Self-Harm
Violence
Overall
LLaDA-V
24.00
42.67
32.67
29.33
44.67
34.67
LLaDA-V + DIJA
53.33
38.67
32.00
42.00
56.67
44.53
LLaDA-V + PAD
45.33
38.00
36.67
38.67
50.00
41.73
LLaDA-V + DIVA
57.33
42.67
36.00
55.33
56.00
49.47
MMaDA
24.67
31.33
23.33
22.67
35.33
27.47
MMaDA + DIJA
31.33
8.00
12.67
22.67
31.33
21.20
Appendix
Table 9: HADES ASR (%) under StrongREJECT.
Comparison unit
n
r
p
All methods × categories
40
0.50
0.0011
LLaDA-V family only
20
0.63
0.0028
MMaDA family only
20
0.47
0.0348
Method-level overall
8
0.37
0.371
Appendix
Table 10: Pearson correlation between Beaver and StrongREJECT ASR.
Target
Comparison
DIVA ASR
95% CI
Δ ASR
Overall p
Significant categories
MMaDA
vs. Base
67.73
[64.27, 71.07]
+36.93
7.58×10−54
All five categories
MMaDA
vs. PAD
67.73
[64.27, 71.07]
+10.67
6.48×10−7
Animal; Self-Harm
MMaDA
vs. DIJA
67.73
[64.27, 71.07]
+6.67
6.44×10−4
Privacy
LLaDA-V
vs. Base
58.80
[55.20, 62.27]
+4.80
0.0039
Self-Harm
LLaDA-V
vs. PAD
58.80
[55.20, 62.27]
+3.20
0.161
Financial; Self-Harm; Violence
LLaDA-V
vs. DIJA
58.80
[55.20, 62.27]
+2.80
0.156
Financial; Self-Harm
Appendix
Table 11: Significance analysis on HADES under the Beaver metric. Δ ASR is DIVA minus the comparison method in percentage points. Overall p -values use paired exact McNemar tests.
Comparison
DIVA
Other
Δ ASR
z
p
vs. Base
69.07
23.07
+46.00
17.87
2.0×10−71
vs. DIJA
69.07
49.87
+19.20
7.57
3.6×10−14
vs. PAD
69.07
53.73
+15.34
6.10
1.1×10−9
Appendix
Table 12: Aggregate two-proportion tests for LLaDA2.0-Uni on HADES.
K
R
Steady-state step (s)
Extrap. 300-step opt. (s)
1
64
0.15
45.9
1
128
0.17
51.1
2
64
0.35
104.1
2
128
0.34
103.3
4
64
0.63
189.0
4
128
0.66
196.6
Appendix
Table 13: Runtime ablation for MMaDA-8B-Base δ optimisation ( 512×512 , bfloat16 , single H800 80 GB). Steady-state step excludes the first PGD step, which incurs one-time CUDA compilation overhead ( ≈4.6 – 8.3 s). Extrap. 300-step opt. projects the steady-state per-step time to the full 300-step budget used in the MMaDA HADES experiments. Model load ( ≈9 – 21 s) and preprocessing ( ≈1 – 2 s) are excluded and incurred once per category regardless of K or R .
Setting
Explicit harmful prompt
Rewritten prompt
Composite image xc
Perturbation type
Avg. ASR
Plain (text-only)
✓
–
–
None
32.93
xc only
–
✓
✓
None
41.00
xc + Random δ
–
✓
✓
Random values in [−ϵ,ϵ]
44.53
xc + Shuffled δ
–
✓
✓
Optimized values, spatially permuted
52.67
xc + DIVA δ
–
✓
✓
Multi-timestep optimized values and locations
58.80
Appendix
Table 14: Composition of the HADES Beaver ablation conditions for LLaDA-V. A check mark denotes that the component is present. The composite image xc jointly contains typography and semantically aligned imagery.
Setting
Animal
Financial
Privacy
Self-Harm
Violence
Average
Plain (text-only)
64.00
58.00
33.33
46.00
65.33
53.33
xc only
88.67
91.33
64.67
66.00
94.00
80.93
xc + Random δ
94.00
89.33
66.00
71.33
91.33
82.40
xc + Shuffled δ
92.00
90.00
69.33
90.67
93.33
87.07
xc + DIVA δ (Ours)
93.33
96.00
69.33
90.67
93.33
88.53
Appendix
Table 15: Ablation study on HADES under the traditional metric (LLaDA-V). We isolate the contribution of the composite image xc and the diffusion-aware perturbation δ by comparing random, spatially shuffled, and true optimized variants. Best in bold , second best underlined .
Model
Category
0–8
8–16
16–32
32–64
Range
LLaDA-V (dVLM)
Animal
86.67
83.33
83.33
80.00
6.67
Financial
93.33
90.00
90.00
90.00
3.33
Privacy
70.00
53.33
53.33
50.00
20.00
Self-Harm
86.67
66.67
63.33
66.67
23.34
Violence
86.67
90.00
93.33
93.33
6.66
Avg
84.67
76.67
76.66
76.00
12.00
Appendix
Table 16: ASR (%) per denoising-step bin under DIVA for LLaDA-V (dVLM) and LLaVA-1.5 (ARM). Range =maxb−minb (per-category SSI). Range cells ≥10 pp are highlighted . Best bin per row in bold .
Figure 6: Additional qualitative examples across the Privacy (top), Animal (middle), and Self-Harm (bottom) categories. Left : the unattacked models (LLaDA-V for Privacy and Animal; MMaDA for Self-Harm) correctly refuse all harmful queries, with the Self-Harm refusal additionally providing mental health crisis resources. Right : under DIVA, the explicit instructions are replaced by visually grounded proxy prompts and the composite images are augmented with the optimised perturbation δ , causing the models to produce detailed harmful completions covering data exfiltration (Privacy), animal harm techniques (Animal), and self-injury guidance (Self-Harm). Red text highlights the harmful content elicited by the attack.
Diffusion Vision-Language Models (dVLMs), built upon the non-causal foundations of Diffusion Large Language Models (dLLMs), have demonstrated remarkable efficacy in multimodal tasks by departing from the traditional autoregressive generation paradigm. While dVLMs appear inherently robust against conventional jailbreak tactics, which we categorize as Fixed Prefix Optimization (FPO) (e.g., anchoring responses with "Sure, here is"), this perceived resilience is deceptive. Our investigation into the safety landscape of dVLMs reveals a unique refusal pattern: Immediate Refusal and Progressive Refusal. We find that while FPO-based attacks often fail by triggering the latter, the progressive refinement process itself uncovers a novel, latent attack surface. To exploit this vulnerability, we propose Global Probability Optimization (GPO), a general jailbreak paradigm designed specifically for the denoising trajectory of masked diffusion models. Unlike prefix-based methods, GPO manipulates the global generative dynamics to bypass guardrails in diffusion language models. Building on this, we introduce GPO-V, the first visual-modality jailbreak framework tailored for dVLMs. Empirical results demonstrate that GPO-V produces stealthy perturbations with exceptional cross-model transferability, revealing a critical security gap in non-sequential generative architectures. Our findings underscore the critical urgency of addressing safety alignment in dVLMs. These results necessitate an immediate and fundamental re-evaluation of current defense paradigms to mitigate the unique risks of diffusion-based generation. Our code is available at: https://anonymous.4open.science/r/GPO-V-0250.
Yu Pan, Andi Zhang, Yi Wang +2
ShanghaiTech University · University of Warwick · SUN YAT-SEN UNIVERSITY
Multi-turn jailbreak attacks have proven effective against text-only large language models (LLMs), where malicious content is gradually introduced to bypass safety alignment. However, effectively extending such attacks to large vision-language models (LVLMs) remains underexplored. In this paper, we find that naively incorporating visual inputs can make multi-turn jailbreaks easier to defend against; for example, overly malicious visual content will easily trigger the defense mechanism in safety-aligned LVLMs, resulting in more conservative responses. Based on this finding, we propose multi-turn adaptive prompting attack (MAPA) that 1) at each turn, alternates text-vision attack actions to elicit the most malicious response; and 2) across turns, adjusts the attack trajectory through iterative back-and-forth refinement to gradually amplify response maliciousness. This two-level design enables MAPA to consistently outperform state-of-the-art methods, improving attack success rates by 15-30% on recent benchmarks against LLaVA-v1.6-Mistral-7B, Qwen2.5-VL-7B-Instruct, Llama-3.2-Vision-11B-Instruct and GPT-4o-mini. Our code is available at: https://github.com/thomaschoi143/MAPA.
In Chong Choi, Jiacheng Zhang, Feng Liu +1
The University of Melbourne · The University of Adelaide
The visual modality of vision-language models (VLMs) is an underexplored attack surface for bypassing safety alignment. We introduce four jailbreak attacks exploiting the vision component: (1) encoding harmful instructions as visual symbol sequences with a decoding legend, (2) replacing harmful objects with benign substitutes (e.g., bomb -> banana) then prompting for harmful actions using the substitute term, (3) replacing harmful text in images (e.g., on book covers) with benign words while visual context preserves the original meaning, and (4) visual analogy puzzles whose solution requires inferring a prohibited concept. Evaluating across six frontier VLMs, our visual attacks bypass safety alignment and expose a cross-modality alignment gap: text-based safety training does not automatically generalize to harmful intent conveyed visually. For example, our visual cipher achieves 40.9% attack success on Claude-Haiku-4.5 versus 10.7% for an equivalent textual cipher. To further our insight into the attack mechanism, we present preliminary interpretability and mitigation results. These findings highlight that robust VLM alignment requires treating vision as a first-class target for safety post-training.
Aharon Azulay, Jan Dubiński, Zhuoyun Li +2
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3