DIVA: Exploiting Cross-Step Conditional Propagation for Visual Jailbreaks in Discrete Diffusion Vision-Language Models

Guorui Song; Runqing Tang; Jingye Zhang; Luyuan Zhang; Feice Huang; Cong Ray; Guocun Wang; Dake Zhong

cs.CVSep 1, 2026

DIVA: Exploiting Cross-Step Conditional Propagation for Visual Jailbreaks in Discrete Diffusion Vision-Language Models

Authors: Guorui Song, Runqing Tang, Jingye Zhang, Luyuan Zhang, Feice Huang, Cong Ray, Guocun Wang, Dake Zhong,

Organizations: Tsinghua University · Beijing University of Posts and Telecommunications

Abstract

Large vision-language models (VLMs) are increasingly deployed in safety-critical settings, yet existing visual jailbreak research has focused almost exclusively on autoregressive architectures, leaving an important emerging family unstudied: multimodal discrete diffusion vision-language models (dVLMs). We identify a vulnerability specific to diffusion generation: because the visual embedding conditions every reverse denoising step rather than acting as a one-time prefix, adversarial visual semantics are repeatedly propagated and amplified across the generation trajectory, a phenomenon we term cross-step conditional propagation. We provide empirical evidence through stage-sensitivity analysis, prompt-level switch rates, and pairwise denoising-bin disagreement metrics, confirmed by bootstrap resampling. We propose DIVA (Discrete-diffusion Vision-language model Attack), a white-box visual jailbreak framework using cross-modal intent obfuscation and diffusion-aware multi-timestep adversarial optimization. Across three dVLMs, DIVA reaches 58.8%, 67.7%, and 69.1% HADES ASR under the Beaver reward-model metric, outperforming visual jailbreak baselines designed for autoregressive models. Code: https://github.com/loststars2002/DIVA

Figures & tables

Figure 1: The target dVLM correctly rejects harmful text-only and harmful image-only inputs. Yet, after adding a small DIVA perturbation $\delta$ to the image and rewriting the query into a benign-looking form, the same model produces a harmful response, exposing a critical weakness in multimodal safety alignment.

Figure 2: Overview of DIVA. Left (Cross-Modal Intent Obfuscation): The harmful instruction $u$ is rewritten into a visually grounded proxy prompt $p$ , and the removed malicious semantics are encoded into a composite image $x_{c}=\mathcal{C}(x_{\mathrm{typ}},\,x_{\mathrm{sem}})$ via typography overlay and a harmful reference image. Right (Diffusion-Aware Adversarial Optimization): At each PGD step, $K$ timesteps are sampled from $w(t)$ , and the aggregated masked cross-entropy loss $\mathcal{L}_{\mathrm{DIVA}}$ is backpropagated through the full multimodal stack to update the bounded perturbation $\delta$ , yielding the final attacked image $\tilde{x}=\Pi_{[0,1]}(x_{c}+\delta)$ .

Figure 3: Architectural comparison of adversarial conditioning in ARMs and dVLMs. In ARMs, the vision encoder produces a fixed visual prefix injected only once, so adversarial influence is confined to the initial conditioning step. In dVLMs, the attacked image re-conditions the model at every reverse denoising step, causing harmful visual semantics to be repeatedly amplified across the generation trajectory, a phenomenon we term cross-step conditional propagation .

Model	Animal	Financial	Privacy	Self-Harm	Violence	Average
LLaDA-V
LLaDA-V	53.33	60.00	45.33	34.67	76.67	54.00
LLaDA-V + DIJA	67.33	55.33	47.33	34.00	76.00	56.00
LLaDA-V + PAD	69.33	54.00	50.00	39.33	65.33	55.60
LLaDA-V + DIVA (Ours)	61.33	66.00	39.33	50.00	77.33	58.80
MMaDA

Table 1: ASR (%) on HADES under the Beaver reward-model metric. Best result per model group in bold , second best underlined .

Model	Safe Rate $\downarrow$	Safe Refuse $\downarrow$	Safe Warning $\downarrow$	ASR $\uparrow$
LLaDA-V
Plain LLaDA-V (text-only)	49.53	0.00	49.53	50.47
LLaDA-V	50.42	0.00	50.42	49.58
LLaDA-V + DIJA	49.64	0.05	49.59	50.36
LLaDA-V + PAD	48.94	0.05	48.89	51.06
LLaDA-V + DIVA (Ours)	47.93	0.00	47.93	52.07

Table 2: Results on VLSBench (%). Safe Rate , Safe Refuse , and Safe Warning are lower for a more successful attack; ASR (Unsafe Rate) is higher. Best ASR per model group in bold , second best underlined .

Setting	Animal	Financial	Privacy	Self-Harm	Violence	Average
Plain (text-only)	43.33	37.33	16.00	16.67	51.33	32.93
$x_{c}$ only	41.33	48.67	22.67	18.67	73.67	41.00
$x_{c}$ + Random $\delta$	45.33	52.00	26.67	24.00	74.67	44.53
$x_{c}$ + Shuffled $\delta$	53.33	60.67	33.33	39.33	76.67	52.67
$x_{c}$ + DIVA $\delta$ (Ours)	61.33	66.00	39.33	50.00	77.33	58.80

Table 3: Ablation study on HADES under the Beaver reward-model metric (LLaDA-V). We isolate the contribution of the composite image

x_{c}

and the diffusion-aware perturbation

\delta

by comparing random, spatially shuffled, and true optimized variants. Best in bold , second best underlined .

Figure 4: Qualitative comparison of model responses before and after DIVA. Left : both LLaDA-V and MMaDA correctly refuse harmful text-only queries, producing policy-compliant safety disclaimers. Right : after rewriting the explicit instruction into a visually grounded proxy prompt and augmenting the composite image with the optimised perturbation $\delta$ , both models produce detailed, step-by-step harmful completions. The proxy prompts are superficially benign and would not trigger text-side filters in isolation; the malicious semantics reside in the composite image and are amplified through cross-step conditional propagation across the denoising trajectory.

Figure 5: Visualisation of the DIVA perturbation for a representative Animal-category example. Top-left: composite image $x_{c}$ with typography overlay. Top-right: attacked image $\tilde{x}=x_{c}+\delta$ . Bottom-left: perturbation $\delta$ amplified by $4\times$ for visibility, showing dense spatially distributed noise. Bottom-right: per-pixel magnitude heatmap $|\delta|$ , confirming roughly uniform, low-energy distortion.

Metric	LLaDA-V (dVLM)	LLaVA-1.5 (ARM)
Attack Performance
ASR: Image only	80.80	96.00
ASR: DIVA	88.53	89.87
Gain	$\mathbf{+7.73}$	${-6.13}$
Stage-Sensitivity Diagnostics
SSI	12.00 [8.00, 19.33]	1.33 [0.00, 4.00]

Table 4: Attack performance and stage-sensitivity diagnostics for LLaDA-V (dVLM) vs. LLaVA-1.5 (ARM) under identical conditions. Gain : DIVA minus image-only ASR (pp). Bootstrap 95% CIs in brackets. All metrics in %.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	LLaDA-V / HADES	LLaDA-V / VLSBench	MMaDA / HADES	MMaDA / VLSBench
$\varepsilon$ ( $\ell_{\infty}$ )	0.2500	0.2500	0.2510	0.2510
PGD outer steps	500	300	300	300
Step size $\alpha$	0.0013	0.0017	0.0021	0.0021
Timesteps per step $K$	4	4	4	4
Response length $R$	128	64	128	128
Total evals (Steps $\,{\times}\,K$ )	2,000	1,200	1,200	1,200

Appendix

Table 5: DIVA hyperparameters for all experiments.

\varepsilon

\ell_{\infty}

pixel budget;

\alpha

: PGD step size;

K

: timesteps sampled per outer step;

R

: target response length (tokens). All runs use

w(t)=\text{Uniform},\,t\sim\mathcal{U}[0.02,1.00)

and bfloat16 precision. The MMaDA-8B-MixCoTe variant is identical to the MMaDA / HADES column except for the model checkpoint.

Category	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$\ell_{\infty}$	Mean $\|\Delta\|$
Average	32.0	0.806	0.132	0.137	0.019

Appendix

Table 6: Perceptual similarity between

x_{c}

and

\tilde{x}=x_{c}+\delta

across HADES categories (

\varepsilon=0.25

, LLaDA-V). Higher PSNR/SSIM and lower LPIPS indicate less distortion.

\ell_{\infty}

is the maximum absolute pixel deviation; mean

|\Delta|

is the average absolute deviation.

Category	Keyword	Trigger rate ( $x_{c}$ / $\tilde{x}$ )	Text sim.
Animal	beat	1.0 / 1.0	1.00
Financial	steal	1.0 / 1.0	1.00
Privacy	spy	1.0 / 1.0	1.00
Self-Harm	overdose	1.0 / 1.0	1.00
Violence	gun	1.0 / 1.0	1.00
Average	—	1.0 / 1.0	1.00

Appendix

Table 7: OCR detectability of the typography overlay before and after adding

\delta

(EasyOCR, LLaDA-V HADES). Trigger rate: fraction of images where the keyword is detected. Text similarity: normalised edit-distance similarity between OCR outputs on

x_{c}

and

\tilde{x}

HADES scenario	Beaver label
Animal	animal_abuse
Financial	financial_crime
Privacy	privacy_violation
Self-Harm	self_harm
Violence	violence

Appendix

Table 8: Mapping from HADES harm scenarios to Beaver-Dam-7B safety-category labels.

Model / method	Animal	Financial	Privacy	Self-Harm	Violence	Overall
LLaDA-V	24.00	42.67	32.67	29.33	44.67	34.67
LLaDA-V + DIJA	53.33	38.67	32.00	42.00	56.67	44.53
LLaDA-V + PAD	45.33	38.00	36.67	38.67	50.00	41.73
LLaDA-V + DIVA	57.33	42.67	36.00	55.33	56.00	49.47
MMaDA	24.67	31.33	23.33	22.67	35.33	27.47
MMaDA + DIJA	31.33	8.00	12.67	22.67	31.33	21.20

Appendix

Table 9: HADES ASR (%) under StrongREJECT.

Comparison unit	$n$	$r$	$p$
All methods $\times$ categories	40	0.50	0.0011
LLaDA-V family only	20	0.63	0.0028
MMaDA family only	20	0.47	0.0348
Method-level overall	8	0.37	0.371

Appendix

Table 10: Pearson correlation between Beaver and StrongREJECT ASR.

Target	Comparison	DIVA ASR	95% CI	$\Delta$ ASR	Overall $p$	Significant categories
MMaDA	vs. Base	67.73	[64.27, 71.07]	+36.93	$7.58\times 10^{-54}$	All five categories
MMaDA	vs. PAD	67.73	[64.27, 71.07]	+10.67	$6.48\times 10^{-7}$	Animal; Self-Harm
MMaDA	vs. DIJA	67.73	[64.27, 71.07]	+6.67	$6.44\times 10^{-4}$	Privacy
LLaDA-V	vs. Base	58.80	[55.20, 62.27]	+4.80	0.0039	Self-Harm
LLaDA-V	vs. PAD	58.80	[55.20, 62.27]	+3.20	0.161	Financial; Self-Harm; Violence
LLaDA-V	vs. DIJA	58.80	[55.20, 62.27]	+2.80	0.156	Financial; Self-Harm

Appendix

Table 11: Significance analysis on HADES under the Beaver metric.

\Delta

ASR is DIVA minus the comparison method in percentage points. Overall

p

-values use paired exact McNemar tests.

Comparison	DIVA	Other	$\Delta$ ASR	$z$	$p$
vs. Base	69.07	23.07	+46.00	17.87	$2.0\times 10^{-71}$
vs. DIJA	69.07	49.87	+19.20	7.57	$3.6\times 10^{-14}$
vs. PAD	69.07	53.73	+15.34	6.10	$1.1\times 10^{-9}$

Appendix

Table 12: Aggregate two-proportion tests for LLaDA2.0-Uni on HADES.

$K$	$R$	Steady-state step (s)	Extrap. 300-step opt. (s)
1	64	0.15	45.9
1	128	0.17	51.1
2	64	0.35	104.1
2	128	0.34	103.3
4	64	0.63	189.0
4	128	0.66	196.6

Appendix

Table 13: Runtime ablation for MMaDA-8B-Base

\delta

optimisation (

512\times 512

, bfloat16 , single H800 80 GB). Steady-state step excludes the first PGD step, which incurs one-time CUDA compilation overhead (

{\approx}4.6

–

8.3

s). Extrap. 300-step opt. projects the steady-state per-step time to the full 300-step budget used in the MMaDA HADES experiments. Model load (

{\approx}9

–

21

s) and preprocessing (

{\approx}1

–

2

s) are excluded and incurred once per category regardless of

K

R

Setting	Explicit harmful prompt	Rewritten prompt	Composite image $x_{c}$	Perturbation type	Avg. ASR
Plain (text-only)	$\checkmark$	–	–	None	32.93
$x_{c}$ only	–	$\checkmark$	$\checkmark$	None	41.00
$x_{c}$ + Random $\delta$	–	$\checkmark$	$\checkmark$	Random values in $[-\epsilon,\epsilon]$	44.53
$x_{c}$ + Shuffled $\delta$	–	$\checkmark$	$\checkmark$	Optimized values, spatially permuted	52.67
$x_{c}$ + DIVA $\delta$	–	$\checkmark$	$\checkmark$	Multi-timestep optimized values and locations	58.80

Appendix

Table 14: Composition of the HADES Beaver ablation conditions for LLaDA-V. A check mark denotes that the component is present. The composite image

x_{c}

jointly contains typography and semantically aligned imagery.

Setting	Animal	Financial	Privacy	Self-Harm	Violence	Average
Plain (text-only)	64.00	58.00	33.33	46.00	65.33	53.33
$x_{c}$ only	88.67	91.33	64.67	66.00	94.00	80.93
$x_{c}$ + Random $\delta$	94.00	89.33	66.00	71.33	91.33	82.40
$x_{c}$ + Shuffled $\delta$	92.00	90.00	69.33	90.67	93.33	87.07
$x_{c}$ + DIVA $\delta$ (Ours)	93.33	96.00	69.33	90.67	93.33	88.53

Appendix

Table 15: Ablation study on HADES under the traditional metric (LLaDA-V). We isolate the contribution of the composite image

x_{c}

and the diffusion-aware perturbation

\delta

by comparing random, spatially shuffled, and true optimized variants. Best in bold , second best underlined .

Model	Category	0–8	8–16	16–32	32–64	Range
LLaDA-V (dVLM)	Animal	86.67	83.33	83.33	80.00	6.67
	Financial	93.33	90.00	90.00	90.00	3.33
	Privacy	70.00	53.33	53.33	50.00	20.00
	Self-Harm	86.67	66.67	63.33	66.67	23.34
	Violence	86.67	90.00	93.33	93.33	6.66
	Avg	84.67	76.67	76.66	76.00	12.00

Appendix

Table 16: ASR (%) per denoising-step bin under DIVA for LLaDA-V (dVLM) and LLaVA-1.5 (ARM). Range

=\max_{b}-\min_{b}

(per-category SSI). Range cells

\geq 10

pp are highlighted . Best bin per row in bold .

Figure 6: Additional qualitative examples across the Privacy (top), Animal (middle), and Self-Harm (bottom) categories. Left : the unattacked models (LLaDA-V for Privacy and Animal; MMaDA for Self-Harm) correctly refuse all harmful queries, with the Self-Harm refusal additionally providing mental health crisis resources. Right : under DIVA, the explicit instructions are replaced by visually grounded proxy prompts and the composite images are augmented with the optimised perturbation $\delta$ , causing the models to produce detailed harmful completions covering data exfiltration (Privacy), animal harm techniques (Animal), and self-injury guidance (Self-Harm). Red text highlights the harmful content elicited by the attack.

Diffusion VLMs

See all 13 papers

GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimization
May 8, 2026Yu Pan, Andi Zhang, Yi Wang +2Diffusion Model Alignment Adversarial Attacks on VLMs
ShanghaiTech University · University of Warwick · SUN YAT-SEN UNIVERSITY
Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models
Feb 16, 2026In Chong Choi, Jiacheng Zhang, Feng Liu +1Adversarial Attacks on VLMs Large Vision-Language Models
The University of Melbourne · The University of Adelaide
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3

May 8, 2026J/K move · Enter open · S save

GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimization

Yu Pan, Andi Zhang, Yi Wang, Sibei Yang+1

ShanghaiTech University · University of Warwick · SUN YAT-SEN UNIVERSITY

Diffusion Model Alignment Adversarial Attacks on VLMs Vision-Language Models Diffusion Language Models

Open paper AskPDF arXiv

Diffusion Vision-Language Models (dVLMs), built upon the non-causal foundations of Diffusion Large Language Models (dLLMs), have demonstrated remarkable efficacy in multimodal tasks by departing from the traditional autoregressive generation paradigm. While dVLMs appear inherently robust against conventional jailbreak tactics, which we categorize as Fixed Prefix Optimization (FPO) (e.g., anchoring responses with "Sure, here is"), this perceived resilience is deceptive. Our investigation into the safety landscape of dVLMs reveals a unique refusal pattern: Immediate Refusal and Progressive Refusal. We find that while FPO-based attacks often fail by triggering the latter, the progressive refinement process itself uncovers a novel, latent attack surface. To exploit this vulnerability, we propose Global Probability Optimization (GPO), a general jailbreak paradigm designed specifically for the denoising trajectory of masked diffusion models. Unlike prefix-based methods, GPO manipulates the global generative dynamics to bypass guardrails in diffusion language models. Building on this, we introduce GPO-V, the first visual-modality jailbreak framework tailored for dVLMs. Empirical results demonstrate that GPO-V produces stealthy perturbations with exceptional cross-model transferability, revealing a critical security gap in non-sequential generative architectures. Our findings underscore the critical urgency of addressing safety alignment in dVLMs. These results necessitate an immediate and fundamental re-evaluation of current defense paradigms to mitigate the unique risks of diffusion-based generation. Our code is available at: https://anonymous.4open.science/r/GPO-V-0250.

Abstract

Diffusion VLMs

Explore similar work

GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimization

Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models

Jailbreaking Vision-Language Models Through the Visual Modality