Multimodal diffusion large language models (dLLMs) generate text through iterative masked denoising with bidirectional attention, offering a compelling alternative to autoregressive vision-language modeling. However, each denoising step requires a full forward pass over long multimodal sequences in which visual tokens dominate the sequence length, and a fixed step budget further compounds this quadratic per-step cost. Existing inference acceleration approaches for multimodal dLLMs characterize attention redundancy implicitly, motivating designs from isolated attention visualizations rather than from measurement. We close this gap with DRAMA, a training-free acceleration framework grounded in a systematic dissection of attention redundancy in multimodal dLLMs. By probing attention tensors along spatial and temporal axes, we reveal two persistent redundancy structures: (i) attention mass consistently concentrates on a sparse subset of visual tokens, and (ii) attention distributions rapidly stabilize across denoising steps after an initial formation phase. These findings motivate two complementary mechanisms: Spatial-Redundancy-Aware Visual Token Pruning (SVTP) and Sufficiency-Guided Adaptive Stopping (SGAS). Together, DRAMA achieves an average 18.67 times inference speedup across six vision-language benchmarks without training, architectural modification, or attention caching, while maintaining competitive accuracy. Our results establish a strong, measurement-grounded acceleration baseline for multimodal dLLM inference.
Figures & tables
Figure 1: Accuracy-latency Pareto frontier . DRAMA variants (diamonds) dominate the Pareto front, achieving higher accuracy at lower latency than all fixed-budget baselines (triangles).
Figure 2: Spatial redundancy analysis . (a) Per-sample attention share of each image patch ranked by dominance. (b) Per-sample attention share distribution of the most-attended patch across the dataset. (c) Cumulative attention coverage by top- k% visual key tokens, stratified by query-key modality pair. (d) Minimum visual tokens needed for 90% attention coverage ( κ(0.9) ) across denoising steps
Figure 3: Temporal redundancy analysis across the denoising trajectory (averaged over six benchmarks). (a) Per-layer inter-step JSD ( Jˉl ) between consecutive denoising steps; peaks highlight steps of greatest attention change. (b) Redundancy heatmap Rl(t) across layers and steps; bright regions indicate high redundancy ( Rl→1 ). (c) Distribution of actual stopping steps of our proposed DRAMA, whose concentration aligns with the plateau onset in (a).
Figure 4: Overview of DRAMA. DRAMA applies Spatial-Redundancy-Aware Visual Token Pruning (SVTP) to select a compact patch subset once before denoising. Sufficiency-Guided Adaptive Stopping (SGAS) terminates the denoising trajectory early.
Baseline
Ablation
DRAMA (Ours)
Metric
64 steps
32 steps
16 steps
w/o SGAS
w/o SVTP FG Prog
w/o SVTP FG Uni
w/o SVTP CK Prog
w/o SVTP CK Uni
FG Prog
FG Uni
CK Prog
CK Uni
AI2D
Acc (%)
73.0
72.7
66.3
72.6
72.7
72.4
73.3
72.6
73.0
73.0
73.0
73.0
Latency (s)
78.2
41.9
19.9
10.4
18.9
9.1
24.2
20.4
5.5
2.6
5.7
6.1
TPS (tok/s)
2.4
4.2
9.2
12.2
9.8
23.4
7.8
9.0
23.9
60.2
23.5
21.7
Speedup
1.00 ×
1.87 ×
3.93 ×
3.76 ×
4.14 ×
8.59 ×
3.23 ×
3.83 ×
14.22 ×
30.08 ×
13.72 ×
12.82 ×
Table 1: Results on six benchmarks. Metrics: Acc (%, ↑ ), Latency (s/sample, ↓ ), TPS (tokens/s, ↑ ), Speedup vs. Baseline 64 steps ( ↑ ). w/o SGAS uses 32 denoising steps; its Speedup is normalized by ÷2 to discount the inherent gain from halving the step count. Best 2nd best (Baselines and DRAMA); Best 2nd best (Ablation). † Evaluated on a subset of the full benchmark.
Figure 5: Distribution of actual denoising steps under SGAS across six benchmarks. Each subplot shows the histogram and KDE for one SGAS variant. FG Uni concentrates most samples at very early steps ( μ≈8 – 12 ), while CK variants cluster around the first eligible checkpoint (step 16). Progressive scheduling shifts the distribution rightward, reflecting its stricter early-step requirements.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Split
#Samples
AI2D [ 22 ]
test (full)
3,088
MMBench [ 33 ]
dev-en (full)
4,329
MME [ 13 ]
full
2,374
MMMU [ 54 ]
validation (full)
1,050
MM-Star [ 8 ]
full
1,500
SEED-Bench [ 26 ]
random subset (seed-fixed)
2,000
Appendix
Table 2: Evaluation splits used in our experiments.
Component
Content
Image token
<image>
Question
Question: {question}
Options
Options: A. {option_a} B. {option_b} ...
Thinking instruction
You should first think about the reasoning process in the mind and then provide the user with the answer. The reasoning process is enclosed within <think> </think> tags, i.e. <think> reasoning process here </think> <answer> answer here: Please select the correct option by responding with only the letter (A, B, C, D, or E) </answer>
Table 5: Lexical coherence thresholds for all SGAS variants ( T=64 ).
Baseline
Ablation
DRAMA (Ours)
Metric
128 steps
w/o SGAS
w/o SVTP FG Prog
w/o SVTP FG Uni
w/o SVTP CK Prog
w/o SVTP CK Uni
FG Prog
FG Uni
CK Prog
CK Uni
AI2D ( n=3088 )
Acc (%)
69.88
69.04
68.36
68.72
68.49
68.75
67.58
67.81
67.81
67.45
Latency (s)
9.98
7.40
7.04
7.07
7.49
7.50
5.30
5.23
5.50
5.54
TPS (tok/s)
12.82
17.29
18.17
18.11
17.08
17.06
24.13
24.49
23.27
23.11
Speedup
1.00 ×
1.35 ×
1.42 ×
1.41 ×
1.33 ×
1.33 ×
1.88 ×
1.91 ×
1.82 ×
1.80 ×
Appendix
Table 6: Benchmark results on LaViDa (128 denoising steps, think mode). Sample counts are given in the benchmark headers. Latency is wall-clock seconds per sample and TPS is computed from the fixed generation length of 128 tokens under the measured latency. Metrics: Acc (%, ↑ ), Latency (s/sample, ↓ ), TPS (tokens/s, ↑ ), Speedup vs. the 128-step baseline ( ↑ ), Acc Retention ( ↑ ).
Benchmark
Method
Setting
Performance
Latency (s)
Speedup
Acc. Retention
AI2D
LLaDA-V (32-step)
500 samples
73.00
49.194
1.00 ×
100.0%
+ RedVTP ( r=0.25 )
500 samples
71.60
38.319
1.28 ×
98.1%
LLaDA-V (32-step)
full set (3,088)
72.73
41.94
1.00 ×
100.0%
+ DRAMA (ours)
full set (3,088)
73.02
2.57
16.32 ×
100.4%
MM-Star
LLaDA-V (32-step)
500 samples
50.00
34.663
1.00 ×
100.0%
+ RedVTP ( r=0.25 )
500 samples
42.40
28.718
1.21 ×
84.8%
Appendix
Table 7: Per-benchmark comparison with RedVTP on LLaDA-V, both reproduced by us. RedVTP runs at its most aggressive operating point ( r=0.25 ) under the same 32-step think-mode regime as DRAMA.
AI2D
MMMU
MM-Star
SEED-Bench
Avg.
Method
Acc (Ret.)
Sp.
Acc (Ret.)
Sp.
Acc (Ret.)
Sp.
Acc (Ret.)
Sp.
Sp.
LaViDa (baseline)
69.88 (1.00)
1.00 ×
42.44 (1.00)
1.00 ×
43.00 (1.00)
1.00 ×
63.50 (1.00)
1.00 ×
1.00 ×
D3ToM-t ( keep=0.25 , L3)
66.80 (0.96)
1.20 ×
39.89 (0.94)
1.15 ×
38.80 (0.81)
1.21 ×
53.00 (0.83)
1.23 ×
1.20 ×
DRAMA CK Uni
67.45 (0.97)
1.80 ×
42.00 (0.99)
1.70 ×
40.00 (0.93)
1.65 ×
61.00 (0.96)
1.77 ×
1.73 ×
DRAMA CK Prog
67.81 (0.97)
1.82 ×
41.44 (0.98)
1.72 ×
39.00 (0.91)
1.63 ×
59.10 (0.93)
1.78 ×
1.74 ×
DRAMA FG Prog
67.58 (0.97)
1.88 ×
41.11 (0.97)
1.80 ×
39.13 (0.91)
1.76 ×
59.10 (0.93)
1.91 ×
1.84 ×
Appendix
Table 9: Comparison with D3ToM on LaViDa. All rows are re-run by us under 128-step think mode, D3ToM included, so that accuracy and speedup are measured under one regime against one baseline; retention is accuracy divided by that baseline. Rows use the full set for AI2D, MME, MMMU, MMstar, and a 1000-sample subset for SEED-Bench. Best , 2nd best .
POPE (F1)
TextVQA (Acc)
Setting
Score
Speedup
Score
Speedup
Baseline (64 steps)
88.30
1.00 ×
64.60
1.00 ×
Baseline (32 steps)
86.37
1.90 ×
60.70
2.00 ×
Baseline (16 steps)
87.24
3.77 ×
–
–
SGAS only (FG Uni)
88.30
3.81 ×
–
–
DRAMA FG Prog
85.35
10.69 ×
51.60
7.91 ×
Appendix
Table 10: Held-out evaluation on POPE and TextVQA, both on LLaDA-V. Neither benchmark was used when setting any threshold. Speedup is wall-clock against the 64-step baseline of the same benchmark. POPE reports F1 on 500 samples; TextVQA reports accuracy on 100 samples. “SGAS only” is DRAMA with SVTP disabled.
Figure 6: Spatial redundancy analysis – AI2D. The diagram-centric nature of AI2D leads to highly concentrated attention on a single informative patch.
Figure 7: Spatial redundancy analysis – MMBench. Despite diverse image content, the top 2–3 patches consistently capture the majority of output → image attention.
Figure 8: Spatial redundancy analysis – MME. Perceptual tasks in MME yield concentrated attention patterns, with κ(0.9) remaining stable across all denoising steps.
Figure 9: Spatial redundancy analysis – MM-Star. Knowledge-intensive queries in MM-Star show similar spatial concentration to MMMU.
Figure 10: Spatial redundancy analysis – SEED-Bench. SEED-Bench exhibits slightly broader spatial distributions due to its diverse scene understanding tasks, yet the dominant patch still captures ≥40% of attention.
Figure 11: Temporal redundancy analysis for all benchmarks. (a) Per-layer inter-step JSD across denoising steps. (b) Layer × step redundancy heatmap.
Figure 12: DRAMA achieves an average 18.67× inference speedup over fixed-budget baselines across six benchmarks (MMMU [ 54 ] , AI2D [ 22 ] , MME [ 13 ] , MM-Star [ 8 ] , SEED-Bench [ 26 ] , and MMBench [ 33 ] ) while maintaining competitive accuracy.
Figure 13: Averaged accuracy–latency Pareto frontier across all benchmarks. Each point represents a method’s mean accuracy and mean latency. DRAMA variants occupy the Pareto front, achieving superior accuracy at substantially lower latency than fixed-budget baselines.
Figure 14: Step distribution – AI2D. FG Uni achieves 85% / 79% step saving (w/o SVTP / DRAMA) owing to the concentrated spatial attention on diagram elements.
Figure 15: Step distribution – MMBench. The diverse visual content of MMBench leads to a slightly broader stopping distribution compared to simpler benchmarks.
Figure 16: Step distribution – MME. Perceptual yes/no tasks converge rapidly, with most samples stopping within the first 15–20 steps.
Figure 17: Step distribution – MMMU. Multi-discipline knowledge questions exhibit moderate step savings, with progressive scheduling providing a useful accuracy safeguard.
Figure 18: Step distribution – MM-Star. Knowledge-intensive queries show stopping patterns similar to MMMU, confirming that task difficulty correlates with stopping behaviour.
Figure 19: Step distribution – SEED-Bench. The diverse scene understanding tasks in SEED-Bench produce a broad stopping distribution, yet SGAS still achieves substantial step savings.