Multimodal diffusion large language models (dLLMs) generate text through iterative masked denoising with bidirectional attention, offering a compelling alternative to autoregressive vision-language modeling. However, each denoising step requires a full forward pass over long multimodal sequences in which visual tokens dominate the sequence length, and a fixed step budget further compounds this quadratic per-step cost. Existing inference acceleration approaches for multimodal dLLMs characterize attention redundancy implicitly, motivating designs from isolated attention visualizations rather than from measurement. We close this gap with DRAMA, a training-free acceleration framework grounded in a systematic dissection of attention redundancy in multimodal dLLMs. By probing attention tensors along spatial and temporal axes, we reveal two persistent redundancy structures: (i) attention mass consistently concentrates on a sparse subset of visual tokens, and (ii) attention distributions rapidly stabilize across denoising steps after an initial formation phase. These findings motivate two complementary mechanisms: Spatial-Redundancy-Aware Visual Token Pruning (SVTP) and Sufficiency-Guided Adaptive Stopping (SGAS). Together, DRAMA achieves an average 18.67 times inference speedup across six vision-language benchmarks without training, architectural modification, or attention caching, while maintaining competitive accuracy. Our results establish a strong, measurement-grounded acceleration baseline for multimodal dLLM inference.
Figures & tables
Figure 1: Accuracy-latency Pareto frontier . DRAMA variants (diamonds) dominate the Pareto front, achieving higher accuracy at lower latency than all fixed-budget baselines (triangles).
Figure 2: Spatial redundancy analysis . (a) Per-sample attention share of each image patch ranked by dominance. (b) Per-sample attention share distribution of the most-attended patch across the dataset. (c) Cumulative attention coverage by top- k% visual key tokens, stratified by query-key modality pair. (d) Minimum visual tokens needed for 90% attention coverage ( κ(0.9) ) across denoising steps
Figure 3: Temporal redundancy analysis across the denoising trajectory (averaged over six benchmarks). (a) Per-layer inter-step JSD ( Jˉl ) between consecutive denoising steps; peaks highlight steps of greatest attention change. (b) Redundancy heatmap Rl(t) across layers and steps; bright regions indicate high redundancy ( Rl→1 ). (c) Distribution of actual stopping steps of our proposed DRAMA, whose concentration aligns with the plateau onset in (a).
Figure 4: Overview of DRAMA. DRAMA applies Spatial-Redundancy-Aware Visual Token Pruning (SVTP) to select a compact patch subset once before denoising. Sufficiency-Guided Adaptive Stopping (SGAS) terminates the denoising trajectory early.
Baseline
Ablation
DRAMA (Ours)
Metric
64 steps
32 steps
16 steps
w/o SGAS
w/o SVTP FG Prog
w/o SVTP FG Uni
w/o SVTP CK Prog
w/o SVTP CK Uni
FG Prog
FG Uni
CK Prog
CK Uni
AI2D
Acc (%)
73.0
72.7
66.3
72.6
72.7
72.4
73.3
72.6
73.0
73.0
73.0
73.0
Latency (s)
78.2
41.9
19.9
10.4
18.9
9.1
24.2
20.4
5.5
2.6
5.7
6.1
TPS (tok/s)
2.4
4.2
9.2
12.2
9.8
23.4
7.8
9.0
23.9
60.2
23.5
21.7
Speedup
1.00 ×
1.87 ×
3.93 ×
3.76 ×
4.14 ×
8.59 ×
3.23 ×
3.83 ×
14.22 ×
30.08 ×
13.72 ×
12.82 ×
Table 1: Results on six benchmarks. Metrics: Acc (%, ↑ ), Latency (s/sample, ↓ ), TPS (tokens/s, ↑ ), Speedup vs. Baseline 64 steps ( ↑ ). w/o SGAS uses 32 denoising steps; its Speedup is normalized by ÷2 to discount the inherent gain from halving the step count. Best 2nd best (Baselines and DRAMA); Best 2nd best (Ablation). † Evaluated on a subset of the full benchmark.
Figure 5: Distribution of actual denoising steps under SGAS across six benchmarks. Each subplot shows the histogram and KDE for one SGAS variant. FG Uni concentrates most samples at very early steps ( μ≈8 – 12 ), while CK variants cluster around the first eligible checkpoint (step 16). Progressive scheduling shifts the distribution rightward, reflecting its stricter early-step requirements.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Split
#Samples
AI2D [ 22 ]
test (full)
3,088
MMBench [ 33 ]
dev-en (full)
4,329
MME [ 13 ]
full
2,374
MMMU [ 54 ]
validation (full)
1,050
MM-Star [ 8 ]
full
1,500
SEED-Bench [ 26 ]
random subset (seed-fixed)
2,000
Appendix
Table 2: Evaluation splits used in our experiments.
Component
Content
Image token
<image>
Question
Question: {question}
Options
Options: A. {option_a} B. {option_b} ...
Thinking instruction
You should first think about the reasoning process in the mind and then provide the user with the answer. The reasoning process is enclosed within <think> </think> tags, i.e. <think> reasoning process here </think> <answer> answer here: Please select the correct option by responding with only the letter (A, B, C, D, or E) </answer>
Table 5: Lexical coherence thresholds for all SGAS variants ( T=64 ).
Baseline
Ablation
DRAMA (Ours)
Metric
128 steps
w/o SGAS
w/o SVTP FG Prog
w/o SVTP FG Uni
w/o SVTP CK Prog
w/o SVTP CK Uni
FG Prog
FG Uni
CK Prog
CK Uni
AI2D ( n=3088 )
Acc (%)
69.88
69.04
68.36
68.72
68.49
68.75
67.58
67.81
67.81
67.45
Latency (s)
9.98
7.40
7.04
7.07
7.49
7.50
5.30
5.23
5.50
5.54
TPS (tok/s)
12.82
17.29
18.17
18.11
17.08
17.06
24.13
24.49
23.27
23.11
Speedup
1.00 ×
1.35 ×
1.42 ×
1.41 ×
1.33 ×
1.33 ×
1.88 ×
1.91 ×
1.82 ×
1.80 ×
Appendix
Table 6: Benchmark results on LaViDa (128 denoising steps, think mode). Sample counts are given in the benchmark headers. Latency is wall-clock seconds per sample and TPS is computed from the fixed generation length of 128 tokens under the measured latency. Metrics: Acc (%, ↑ ), Latency (s/sample, ↓ ), TPS (tokens/s, ↑ ), Speedup vs. the 128-step baseline ( ↑ ), Acc Retention ( ↑ ).
Benchmark
Method
Setting
Performance
Latency (s)
Speedup
Acc. Retention
AI2D
LLaDA-V (32-step)
500 samples
73.00
49.194
1.00 ×
100.0%
+ RedVTP ( r=0.25 )
500 samples
71.60
38.319
1.28 ×
98.1%
LLaDA-V (32-step)
full set (3,088)
72.73
41.94
1.00 ×
100.0%
+ DRAMA (ours)
full set (3,088)
73.02
2.57
16.32 ×
100.4%
MM-Star
LLaDA-V (32-step)
500 samples
50.00
34.663
1.00 ×
100.0%
+ RedVTP ( r=0.25 )
500 samples
42.40
28.718
1.21 ×
84.8%
Appendix
Table 7: Per-benchmark comparison with RedVTP on LLaDA-V, both reproduced by us. RedVTP runs at its most aggressive operating point ( r=0.25 ) under the same 32-step think-mode regime as DRAMA.
AI2D
MMMU
MM-Star
SEED-Bench
Avg.
Method
Acc (Ret.)
Sp.
Acc (Ret.)
Sp.
Acc (Ret.)
Sp.
Acc (Ret.)
Sp.
Sp.
LaViDa (baseline)
69.88 (1.00)
1.00 ×
42.44 (1.00)
1.00 ×
43.00 (1.00)
1.00 ×
63.50 (1.00)
1.00 ×
1.00 ×
D3ToM-t ( keep=0.25 , L3)
66.80 (0.96)
1.20 ×
39.89 (0.94)
1.15 ×
38.80 (0.81)
1.21 ×
53.00 (0.83)
1.23 ×
1.20 ×
DRAMA CK Uni
67.45 (0.97)
1.80 ×
42.00 (0.99)
1.70 ×
40.00 (0.93)
1.65 ×
61.00 (0.96)
1.77 ×
1.73 ×
DRAMA CK Prog
67.81 (0.97)
1.82 ×
41.44 (0.98)
1.72 ×
39.00 (0.91)
1.63 ×
59.10 (0.93)
1.78 ×
1.74 ×
DRAMA FG Prog
67.58 (0.97)
1.88 ×
41.11 (0.97)
1.80 ×
39.13 (0.91)
1.76 ×
59.10 (0.93)
1.91 ×
1.84 ×
Appendix
Table 9: Comparison with D3ToM on LaViDa. All rows are re-run by us under 128-step think mode, D3ToM included, so that accuracy and speedup are measured under one regime against one baseline; retention is accuracy divided by that baseline. Rows use the full set for AI2D, MME, MMMU, MMstar, and a 1000-sample subset for SEED-Bench. Best , 2nd best .
POPE (F1)
TextVQA (Acc)
Setting
Score
Speedup
Score
Speedup
Baseline (64 steps)
88.30
1.00 ×
64.60
1.00 ×
Baseline (32 steps)
86.37
1.90 ×
60.70
2.00 ×
Baseline (16 steps)
87.24
3.77 ×
–
–
SGAS only (FG Uni)
88.30
3.81 ×
–
–
DRAMA FG Prog
85.35
10.69 ×
51.60
7.91 ×
Appendix
Table 10: Held-out evaluation on POPE and TextVQA, both on LLaDA-V. Neither benchmark was used when setting any threshold. Speedup is wall-clock against the 64-step baseline of the same benchmark. POPE reports F1 on 500 samples; TextVQA reports accuracy on 100 samples. “SGAS only” is DRAMA with SVTP disabled.
Figure 6: Spatial redundancy analysis – AI2D. The diagram-centric nature of AI2D leads to highly concentrated attention on a single informative patch.
Figure 7: Spatial redundancy analysis – MMBench. Despite diverse image content, the top 2–3 patches consistently capture the majority of output → image attention.
Figure 8: Spatial redundancy analysis – MME. Perceptual tasks in MME yield concentrated attention patterns, with κ(0.9) remaining stable across all denoising steps.
Figure 9: Spatial redundancy analysis – MM-Star. Knowledge-intensive queries in MM-Star show similar spatial concentration to MMMU.
Figure 10: Spatial redundancy analysis – SEED-Bench. SEED-Bench exhibits slightly broader spatial distributions due to its diverse scene understanding tasks, yet the dominant patch still captures ≥40% of attention.
Figure 11: Temporal redundancy analysis for all benchmarks. (a) Per-layer inter-step JSD across denoising steps. (b) Layer × step redundancy heatmap.
Figure 12: DRAMA achieves an average 18.67× inference speedup over fixed-budget baselines across six benchmarks (MMMU [ 54 ] , AI2D [ 22 ] , MME [ 13 ] , MM-Star [ 8 ] , SEED-Bench [ 26 ] , and MMBench [ 33 ] ) while maintaining competitive accuracy.
Figure 13: Averaged accuracy–latency Pareto frontier across all benchmarks. Each point represents a method’s mean accuracy and mean latency. DRAMA variants occupy the Pareto front, achieving superior accuracy at substantially lower latency than fixed-budget baselines.
Figure 14: Step distribution – AI2D. FG Uni achieves 85% / 79% step saving (w/o SVTP / DRAMA) owing to the concentrated spatial attention on diagram elements.
Figure 15: Step distribution – MMBench. The diverse visual content of MMBench leads to a slightly broader stopping distribution compared to simpler benchmarks.
Figure 16: Step distribution – MME. Perceptual yes/no tasks converge rapidly, with most samples stopping within the first 15–20 steps.
Figure 17: Step distribution – MMMU. Multi-discipline knowledge questions exhibit moderate step savings, with progressive scheduling providing a useful accuracy safeguard.
Figure 18: Step distribution – MM-Star. Knowledge-intensive queries show stopping patterns similar to MMMU, confirming that task difficulty correlates with stopping behaviour.
Figure 19: Step distribution – SEED-Bench. The diverse scene understanding tasks in SEED-Bench produce a broad stopping distribution, yet SGAS still achieves substantial step savings.
Discrete diffusion-based multimodal large language models (dMLLMs) have emerged as a promising alternative to autoregressive MLLMs thanks to their advantages in parallel decoding and bidirectional context modeling, but most existing dMLLMs incur significant computational overhead during inference due to the full-sequence attention computation in each denoising step. Pioneering studies attempt to resolve this issue from a modality-agnostic perspective via key-value cache optimization or efficient sampling but most of them overlook modality-specific visual token redundancy. In this work, we conduct a comprehensive study on how visual token redundancy evolves with different dMLLM architectures and tasks and how visual token pruning affects dMLLM responses and efficiency. Specifically, our study reveals that visual redundancy emerges only in from-scratch dMLLMs while handling long-answer tasks. In addition, we validate that visual token pruning introduces non-negligible information loss in dMLLMs and only from-scratch dMLLMs can recover the lost information progressively during late denoising steps. Furthermore, our study shows that layer-skipping is promising for accelerating AR-to-diffusion dMLLMs, whereas progressive or late-step pruning is more effective for from-scratch dMLLMs. Overall, this work offers a new perspective on efficiency optimization for dMLLMs, greatly advancing their applicability across various multimodal understanding tasks. The code is available at: https://github.com/Yrdal3910/dMLLM-Visual-Token-Redundancy-Analysis.
Duo Li, Zuhao Yang, Xiaoqin Zhang +2
CCDS, NTU, Singapore · CCST, ZJUT, China · Terminus AI Lab, UCAS, China
Vision-language models (VLMs) predominantly rely on autoregressive decoding, which generates tokens one at a time and fundamentally limits inference throughput. This limitation is especially acute in physical AI scenarios such as robotics and autonomous driving, where VLMs are deployed on edge devices at batch size one, making AR decoding memory-bandwidth-bound and leaving hardware parallelism underutilized. While block-wise discrete diffusion has shown promise for parallel text generation, extending it to VLMs remains challenging due to the need to jointly handle continuous visual representations and discrete text tokens while preserving pretrained multimodal capabilities. We present Fast-dVLM, a block-diffusion-based VLM that enables KV-cache-compatible parallel decoding and speculative block decoding for inference acceleration. We systematically compare two AR-to-diffusion conversion strategies: a two-stage approach that first adapts the LLM backbone with text-only diffusion fine-tuning before multimodal training, and a direct approach that converts the full AR VLM in one stage. Under comparable training budgets, direct conversion proves substantially more efficient by leveraging the already multimodally aligned VLM; we therefore adopt it as our recommended recipe. We introduce a suite of multimodal diffusion adaptations, block size annealing, causal context attention, auto-truncation masking, and vision efficient concatenation, that collectively enable effective block diffusion in the VLM setting. Extensive experiments across 11 multimodal benchmarks show Fast-dVLM matches its autoregressive counterpart in generation quality. With SGLang integration and FP8 quantization, Fast-dVLM achieves over 6x end-to-end inference speedup over the AR baseline.
Diffusion Multimodal Large Language Models (DMLLMs) are highly effective for multimodal reasoning, yet their inference efficiency is significantly hindered by fixed-length generation constraints. Since the actual output length is unknown, output sequences are padded to a predefined maximum length, resulting in substantial redundant computation over unnecessary [EOS] tokens. In this work, we discover that DMLLMs implicitly reveal their valid semantic boundary at the very first denoising step through a distinct shift in MLP activation sparsity. Leveraging this observation, we propose Seer, a training-free framework that detects this boundary using a Signal-to-Noise Ratio (SNR)-based criterion and performs one-shot truncation of the redundant suffix for all subsequent computations. To preserve these theoretical gains during batched serving, Seer incorporates a hybrid execution strategy that maximizes throughput while seamlessly accommodating dynamic sequence lengths. Experimental results demonstrate that Seer effectively eliminates padding waste, accelerating throughput by up to ∼31×. Across 9 benchmarks, Seer robustly maintains overall performance and even improves accuracy on complex visual tasks by mitigating noise leakage (e.g., DocVQA score increases from 63.52 to 63.66), offering a highly efficient, plug-and-play solution for DMLLM acceleration.