Speculative decoding accelerates vision-language models(VLMs) while preserving the target model's outputs, but its speedup diminishes under high concurrency serving as the system becomes more compute bounded as the batch size increases. This problem is pronounced for multimodal speculative decoding as the long range of the image tokens in the prompt can significantly increase the computational cost during draft token prediction. The block-diffusion drafters (e.g. DFlash, DSpark) can effectively speed up the draft model efficiency by predicting multiple draft tokens in one forward pass, but their speedup for VLMs under high concurrency is still modest (<9%). In order to understand this gap, we examine the information flow between different modalities from target model to draft model and identify the sparsity pattern of visual attention between the target and draft models: only small set of key-value heads in the draft model need direct visual feature attention in the context and only visual features from top level layers in the target model need to be attended in the context. We therefore propose VisHead-Spec, which selects a small, fixed set of key-value (KV) heads to attend to all image features from top level layers in the target model and restricts the remaining heads to text features. On Qwen3-VL-8B and MMSpec, VisHead-Spec-8 lets 8 of the drafter's 40 KV heads read the image while keeping accepted length within 0.3% of the block-diffusion draft model baseline (i.e. DSpark). At batch size 64, its throughput gain over autoregressive decoding increases to 16.0%, compared with 9.5% for DSpark; at batch size 1, its speedup remains comparable to DSpark.
Figures & tables
Figure 1: Context construction and KV-head routing. Both panels show hidden states cached at layers in the target model S = {1, 9, 17, 25, 33} during normal target computation. I and T denote image- and text-token hidden states. (a) DSpark constructs both image and text rows from all five taps and lets all 40 KV heads in its five draft layers attend to image context. (b) VisHead-Spec-8 constructs image rows from the two highest-index sampled layers, 25 and 33, while text rows still use all five. Four KV heads in each of draft layers 2 and 3 retain image access, shown first in each row, and the other heads attend to text context only. Each cell represents one KV head and its four query heads; all heads retain ordinary block attention.
Figure 2: Target next-token image-contribution retention on six MMSpec tasks, using Eq. ( A1 ). Text positions cannot attend to image positions at or above layer index cutoff ℓ . Dashed lines mark cutoffs 2, 10, 18, 26, and 34, immediately after the sampled layers 1, 9, 17, 25, and 33 in Qwen3-VL-8B.
Figure 3: Reference-token probability ratios after removing all image-token context from the drafter while holding its weights fixed. Each curve shows one task across diffusion block offsets d = 1, …, 5, using 100 ×ρ draft ( d ) in Eq. ( 1 ). Error bars show paired 95% confidence intervals across prompts. The dashed horizontal line marks the full-context reference (100%).
Figure 4: Reference-token probability ratios when image rows are constructed from subsets of the sampled layers in the target model. We use Eq. ( 1 ) with the subset-conditioned probability in place of q 0 , averaged geometrically over the five draft offsets; full context is 100%. Layers are added in a decreasing index. Tick labels list the selected layers; “None” removes image rows, and “All” uses S = {1, 9, 17, 25, 33}. Colored lines show the six tasks and error bars paired 95% confidence intervals across prompts. Figure D1 shows the increasing ordering.
Figure 5: Reference-token probability ratios when selected draft layers’ KV heads may attend to image rows. Layers gain image access in increasing index; tick labels list the enabled draft layers, ending at full context. The gray line is the geometric mean over six tasks; error bars show paired 95% confidence intervals across prompts. The reverse ordering is in Figure D3 .
Method
AL
Batch size 1
Batch size 8
Batch size 16
Batch size 32
Batch size 64
AR
1
1x
1x
1x
1x
1x
EAGLE-3, chain
3.00
1.85x
1.59x
1.36x
1.00x
0.89x
EAGLE-3, tree
4.32
2.34x
1.42x
0.95x
0.64x
0.50x
DFlash
3.01
2.14x
1.76x
1.53x
1.23x
1.03x
DSpark
3.28
2.26x
1.86x
1.62x
1.29x
1.09x
VisHead-Spec-8 (ours)
3.27
2.23x
1.88x
1.65x
1.35x
1.16x
Table 1: Mixed-workload AL and E2E throughput speedup relative to AR for drafters initialized from released checkpoints.
Method
AL
Batch size 1
Batch size 8
Batch size 16
Batch size 32
Batch size 64
DSpark-scratch
2.29
1.58x
1.34x
1.17x
0.96x
0.82x
MSD, chain
1.89
1.17x
1.04x
0.92x
0.68x
0.62x
MSD, tree
2.56
1.61x
0.98x
0.65x
0.44x
0.34x
ViSpec, chain
1.82
1.09x
1.00x
0.88x
0.64x
0.60x
ViSpec, tree
2.44
1.49x
0.92x
0.61x
0.41x
0.32x
VisHead-Spec-8-scratch
2.30
1.58x
1.35x
1.23x
1.01x
0.88x
Table 2: Mixed-workload AL and E2E throughput speedup relative to AR for randomly initialized drafters.
Method
General VQA
Text VQA
Image captioning
Chart understanding
Complex reasoning
Multi-turn conversation
AL
Speedup
AL
Speedup
AL
Speedup
AL
Speedup
AL
Speedup
AL
Speedup
AR
—
1.000
—
1.000
—
1.000
—
1.000
—
1.000
—
1.000
VisHead-8
3.056
1.21x
3.130
1.15x
2.961
1.10x
3.628
1.26x
3.451
1.18x
3.196
1.11x
VisHead-4
3.058
1.21x
3.110
1.15x
2.955
1.11x
3.624
1.26x
3.445
1.18x
3.176
1.12x
DSpark
3.058
1.18x
3.169
1.11x
2.965
1.08x
3.650
1.19x
3.452
1.13x
3.199
1.00x
DFlash
2.807
1.13x
2.903
1.06x
2.709
1.01x
3.377
1.14x
3.182
1.06x
2.933
0.96x
Table 3: Per-task results for drafters initialized from released checkpoints. E2E throughput speedup is measured at batch size 64 relative to AR.
Variant
AL
Batch size 1 speedup
Batch size 8 speedup
Batch size 16 speedup
Batch size 32 speedup
Batch size 64 speedup
VisHead-Spec-0
3.152
2.19x
1.83x
1.62x
1.31x
1.12x
VisHead-Spec-8 (ours)
3.27
2.23x
1.88x
1.65x
1.35x
1.16x
VisHead-Spec-4 (ours)
3.26
2.24x
1.87x
1.66x
1.35x
1.16x
Table 4: Mixed-workload ablation of image-attention budgets after matched fine-tuning. AL is averaged over batch sizes 1, 8, 16, 32, and 64; each batch column is E2E throughput divided by AR throughput at that batch size.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Layers with image attention
AL gain retained (%)
0
0.1
1
3.4
2
29.6
3
41.2
4
10.6
2, 3
81.3
Appendix
Table B1: Layer subsets from the complete 32-subset comparison, including every single-layer candidate and the best three at each larger budget. AL gain retained is Eq. ( B2 ) expressed as a percentage, measured on 100 held-out prompts with the same drafter weights and target context in every condition.
KV heads attending to image context
Head count for Layer 2 + layer 3
Offline AL
Note
16
8 + 8
3.32
All heads in layers 2 and 3
12
8 + 4
3.32
Best at budget 12
10
6 + 4
3.31
Best at budget 10
8
4 + 4
3.29
VisHead-Spec-8
8
6 + 2
3.29
Best at budget 8
6
3 + 3
3.27
Best at budget 6
Appendix
Table B2: Offline AL for selected allocations from the 81-allocation comparison. “Best at budget” identifies the highest measured AL at the given total head count.
Drafter (VisHead-Spec and block-diffusion drafter baseline)
5 Qwen3 decoder layers, 32 query / 8 KV heads, head dimension 128; block size 7; Markov head of rank 256; confidence head; context projection of hidden states from layers 1, 9, 17, 25, 33 in the target model; embedding and LM head shared with the target and frozen
Training warm start
deepseek-ai/dspark_qwen3_8b_block7 (VisHead-Spec and block-diffusion drafter baseline), dflash_qwen3_8b_block7 (DFlash), eagle3_qwen3_8b_ttt7 (EAGLE-3)
Training data
67,304 sequences: 68,000 first-turn prompts from LLaVA-1.5 mix665k (first answer ≥ 5 words; MMSpec images and questions removed; about 15% text-only ShareGPT), re-answered greedily by the target at native resolution ( ≤ 2,048 tokens; looping answers dropped)
Schedule
3 epochs = 8,412 steps; global batch 24; AdamW, learning rate 4 × 10 - 5 , 4% warmup, cosine decay to 0; gradient clipping 1.0; bf16; AMD MI300X; identical sample order for all drafters; final checkpoint
Head router
per-head block mask; image token id of Qwen3-VL; no added parameters
Appendix
Table B3: Data, initialization, objectives, and schedules for drafters initialized from released checkpoints. VisHead-Spec variants differ in their routing masks. The separately initialized MSD and ViSpec comparison follows Table B4 .
Method
Architecture
Training schedule
Optimizer steps / data
DSpark-scratch
5 layers; 32 Q / 8 KV heads per layer; all 40 KV heads read images
AdamW; LR 6 × 10 - 4 ; global batch 512; 4% warmup, cosine decay
10 epochs; 1,310 optimizer steps
VisHead-Spec-8-scratch
Same decoder as DSpark; 8 image-reading KV heads in layers 2 and 3
Same schedule as DSpark-scratch
10 epochs; 1,310 optimizer steps
MSD
1 layer; 32 Q / 8 KV heads; final target hidden state plus token/visual embedding
AdamW; LR 5 × 10 - 5 ; global batch 32; 2,000 warmup steps; norm clipping; paper objective
21,030 steps; text-first curriculum: 63k text draws, 63k draws ramping image fraction 0 → 1, then image data
ViSpec
1 layer; 32 KV heads; image adaptor with 2 queries, one retained image row and one global feature
Stage 1: AdamW 3 × 10 - 5 on text. Stage 2: AdamW 3 × 10 - 6 on image data; global batch 8; MTP 1
Table B4: Architectures and method-specific training schedules for the randomly initialized comparison. All methods use the same 67,304-sequence corpus and a nominal ten-epoch data budget. The target is frozen throughout.
Task
VisHead-0
VisHead-8
VisHead-4
DFlash
EAGLE-3 chain
EAGLE-3 tree
General VQA
− 2.02 ± 0.32
+1.37 ± 0.26
+1.79 ± 0.06
− 4.44 ± 0.44
− 15.07 ± 0.55
− 54.16 ± 0.17
Text VQA
− 3.79 ± 0.17
+3.66 ± 0.63
+3.96 ± 0.21
− 4.26 ± 0.27
− 11.63 ± 0.03
− 45.68 ± 0.24
Image captioning
− 0.31 ± 0.25
+1.02 ± 0.48
+2.04 ± 0.31
− 6.79 ± 0.97
− 17.35 ± 0.23
− 59.39 ± 0.02
Chart understanding
+3.56 ± 0.21
+5.86 ± 0.41
+6.15 ± 0.64
− 4.26 ± 0.09
− 10.94 ± 0.24
− 54.37 ± 0.08
Complex reasoning
+3.46 ± 0.63
+4.36 ± 0.00
+4.57 ± 0.11
− 5.60 ± 0.30
− 17.60 ± 0.72
− 57.96 ± 0.15
Multi-turn conversation
+5.24 ± 0.01
+10.38 ± 0.30
+11.34 ± 0.25
− 4.82 ± 0.03
− 14.21 ± 0.27
− 47.44 ± 0.29
Appendix
Table C1: Per-task E2E throughput change relative to DSpark at batch size 64.
Task
VisHead-0
VisHead-8
VisHead-4
DFlash
EAGLE-3 chain
EAGLE-3 tree
General VQA
− 1.13 ± 0.62
− 0.31 ± 0.93
+0.21 ± 0.08
− 4.73 ± 0.01
− 21.29 ± 2.25
− 50.67 ± 0.11
Text VQA
− 5.05 ± 0.34
+1.88 ± 0.53
+2.09 ± 0.47
− 4.78 ± 0.17
− 16.56 ± 0.57
− 43.89 ± 0.04
Image captioning
− 0.04 ± 0.71
+1.60 ± 0.04
+0.65 ± 0.03
− 7.22 ± 0.37
− 20.19 ± 0.44
− 52.56 ± 0.45
Chart understanding
+1.55 ± 1.03
+3.64 ± 0.71
+3.42 ± 0.44
− 4.16 ± 0.48
− 15.89 ± 0.09
− 49.87 ± 0.31
Complex reasoning
+2.18 ± 0.41
+2.49 ± 0.87
+1.28 ± 0.10
− 6.65 ± 0.02
− 21.25 ± 0.12
− 52.59 ± 0.01
Multi-turn conversation
+4.00 ± 0.69
+7.53 ± 0.41
+6.80 ± 0.32
− 5.53 ± 0.18
− 20.88 ± 0.40
− 44.62 ± 0.86
Appendix
Table C2: Per-task E2E throughput change relative to DSpark at batch size 32.
Task
VisHead-0
VisHead-8
VisHead-4
DFlash
EAGLE-3 chain
EAGLE-3 tree
General VQA
− 1.46 ± 0.09
+0.47 ± 0.54
+0.11 ± 0.11
− 4.05 ± 0.56
− 22.15 ± 0.58
− 42.64 ± 0.22
Text VQA
− 6.30 ± 0.09
+0.73 ± 0.95
+1.82 ± 0.68
− 3.92 ± 0.77
− 16.24 ± 0.56
− 37.48 ± 0.45
Image captioning
− 0.71 ± 1.07
+1.40 ± 0.24
+0.20 ± 1.36
− 6.55 ± 0.99
− 17.80 ± 1.11
− 43.41 ± 0.39
Chart understanding
− 1.04 ± 0.47
+2.00 ± 1.13
+2.17 ± 0.78
− 5.00 ± 0.46
− 16.03 ± 0.02
− 42.56 ± 0.61
Complex reasoning
− 1.69 ± 0.60
+0.19 ± 0.89
− 0.22 ± 0.13
− 7.21 ± 1.22
− 16.06 ± 0.28
− 42.52 ± 0.45
Multi-turn conversation
+0.73 ± 0.88
+5.55 ± 0.58
+5.74 ± 0.83
− 6.18 ± 0.41
− 14.12 ± 1.05
− 35.72 ± 0.84
Appendix
Table C3: Per-task E2E throughput change relative to DSpark at batch size 16.
Task
VisHead-0
VisHead-8
VisHead-4
DFlash
EAGLE-3 chain
EAGLE-3 tree
General VQA
− 1.96 ± 0.37
− 1.18 ± 2.18
− 1.19 ± 1.01
− 4.20 ± 0.38
− 20.22 ± 0.94
− 25.96 ± 0.25
Text VQA
− 7.82 ± 0.21
+0.36 ± 0.04
+0.50 ± 0.37
− 5.43 ± 0.16
− 15.19 ± 0.01
− 22.75 ± 0.30
Image captioning
− 3.10 ± 2.22
− 0.28 ± 0.84
− 0.87 ± 2.42
− 6.94 ± 0.33
− 17.39 ± 0.31
− 22.88 ± 0.77
Chart understanding
− 2.41 ± 0.99
+0.06 ± 0.53
+0.56 ± 0.69
− 5.42 ± 1.03
− 13.90 ± 0.60
− 25.86 ± 0.07
Complex reasoning
− 0.32 ± 1.01
− 0.63 ± 0.09
− 0.45 ± 1.16
− 3.67 ± 0.44
− 13.67 ± 0.00
− 24.58 ± 0.08
Multi-turn conversation
− 0.31 ± 0.33
+5.45 ± 0.39
+3.46 ± 0.49
− 5.62 ± 0.35
− 11.38 ± 0.15
− 19.49 ± 0.12
Appendix
Table C4: Per-task E2E throughput change relative to DSpark at batch size 8.
Task
VisHead-0
VisHead-8
VisHead-4
DFlash
EAGLE-3 chain
EAGLE-3 tree
General VQA
− 1.68 ± 0.03
− 1.72 ± 0.47
− 0.67 ± 0.13
− 5.06 ± 0.54
− 18.32 ± 0.29
+5.10 ± 1.41
Text VQA
− 10.82 ± 0.11
− 1.80 ± 0.20
− 2.55 ± 0.50
− 5.70 ± 0.78
− 19.63 ± 0.35
+2.18 ± 0.46
Image captioning
− 1.60 ± 0.57
− 1.86 ± 0.19
− 1.23 ± 0.47
− 5.52 ± 0.05
− 18.74 ± 0.24
+9.12 ± 0.03
Chart understanding
− 2.98 ± 0.37
− 1.62 ± 0.06
− 1.28 ± 0.27
− 4.87 ± 0.50
− 17.63 ± 0.13
− 0.80 ± 0.59
Complex reasoning
− 1.32 ± 0.35
− 1.39 ± 0.08
− 1.13 ± 0.45
− 5.30 ± 0.02
− 18.38 ± 0.01
+2.03 ± 0.17
Multi-turn conversation
− 3.72 ± 0.44
− 0.07 ± 0.14
− 0.12 ± 0.22
− 5.84 ± 0.04
− 17.41 ± 0.46
+4.16 ± 0.32
Appendix
Table C5: Per-task E2E throughput change relative to DSpark at batch size 1, using service times from the mixed runs.
Figure D1: Reference-token probability ratios when image rows are constructed from subsets of the sampled layers in the target model. We use Eq. ( 1 ) with the subset-conditioned probability in place of q 0 , averaged geometrically over the five draft offsets; full context is 100%. Layers are added in increasing index. Tick labels list the selected layers; “None” removes image rows, and “All” uses S = {1, 9, 17, 25, 33}. Text rows use all five layers throughout. Colored lines show the six tasks and error bars paired 95% confidence intervals across prompts. Figure 4 shows the decreasing ordering.
Figure D2: Served accepted length when image rows use subsets of the sampled layers in the target model, relative to full context. Layers are added (a) in increasing and (b) in decreasing index; tick labels list selected layers. “None” removes image rows; “All” uses S = {1, 9, 17, 25, 33}. Text rows use all five layers throughout.
Figure D3: Reference-token probability ratios when selected draft layers’ KV heads may attend to image rows. Layers gain image access in decreasing index; tick labels list enabled layers, ending at full context. The gray line is the geometric mean over six tasks; error bars show paired 95% confidence intervals across prompts. The increasing ordering is Figure 5 .
Figure D4: Served AL (a) and paired E2E throughput change (b) for DFlash with full, four-head, and eight-head image access at a server batch cap of 64. Full-image DFlash is the throughput reference (0%). Error bars are 95% Student-t confidence intervals over three paired execution orders.
Figure E1: the design of the head router
Method
Top-1 agreement for overall / visual / non-visual
AL
Throughput speedup
Full image access
65.40 / 38.81 / 72.78
3.298
2.389 ×
VisHead-4 (always on)
64.33 / 34.87 / 72.50
3.240
2.271 ×
VisHead-4 + learned per-token router
64.10 / 34.05 / 72.43
3.187
2.219 ×
Appendix
Table E1: Learned routing versus always-on VisHead-4. Top-1 agreement is on a held-out split; AL and throughput speedup are on MMSpec testmini with batch size 1.
Vision-language models (VLMs) achieve strong performance on multimodal tasks but suffer from high inference latency due to large model sizes and long multimodal contexts. Speculative decoding has recently emerged as an effective acceleration technique, yet its behavior in VLMs remains insufficiently understood. We introduce MMSpec, the first benchmark for evaluating speculative decoding in vision-language models. MMSpec contains 600 multimodal samples across six task categories and integrates ten representative speculative decoding algorithms under a unified evaluation framework. Our study reveals three key findings: (1) methods designed for text-only LLMs degrade in multimodal scenarios, (2) vision awareness becomes increasingly important at larger batch sizes, and (3) throughput speedup alone does not reliably reflect latency performance. Motivated by these findings, we propose ViSkip, a plug-and-play speculative decoding method that dynamically adapts speculation to vision tokens and achieves state-of-the-art performance.
Hui Shen, Xin Wang, Ping Zhang +11
University of Michigan · The Ohio State University · Independent +5
Speculative decoding accelerates vision-language model (VLM) inference without altering the output distribution, yet two factors limit its gains. Existing drafters either lack the image evidence required for visually grounded tokens or pay to condition on every visual token, although a short draft block typically depends on only a few image regions. Moreover, the value of verifying additional candidates varies with GPU load, so no fixed verification structure is efficient across serving conditions. We present Uni-LaSpec, a block-parallel speculative decoding framework that addresses both factors. To improve draft quality, Uni-LaSpec reuses the verifier's own attention: the query retained from the preceding verification ranks the cached, int8-quantized visual keys, and a cross-layer vote selects a few visual tokens that the drafter fuses through gated cross-attention. Acceptance-aware training further weights each draft position by its contribution to the accepted prefix. To allocate verification, a marginal-cost criterion selects the structure per request and per round: draft trees exploit idle verifier capacity under light load, whereas a dynamic head truncates linear drafts under heavy load. Across six VLMs from three model families and 17 text and multimodal benchmarks, selecting eight visual tokens retains at least 98.6% of the acceptance obtained from the full image at sub-millisecond selection cost. With greedy decoding at concurrency one, Uni-LaSpec achieves up to 5.11x text and 3.24x multimodal throughput over autoregressive decoding. Across six load levels and three GPU types, it outperforms autoregressive decoding in every setting and matches or exceeds the best fixed verification strategy in 123 of 144 settings; under bursty arrivals, it reduces mean end-to-end latency by 47.6% relative to linear verification.
Hong Nie, Zhangchi Gao, Yuqian Liang +4
University of Science and Technology of China · Unisound AI Technology Co., Ltd.
Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model. Diffusion drafters such as DFlash propose an entire token block in a single forward pass, but mainly focus on text-only tasks, leaving multimodal drafting underexplored. To study this setting, we first introduce a metric of token-level visual dependence based on the KL divergence between next-token distributions with and without visual input. We find that acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest, with this disparity persisting after multimodal training. Motivated by this observation, we propose MMFlash, a simple and effective framework that adaptively upweights visually dependent tokens that the drafter predicts poorly. For long video inputs, MMFlash further employs sparse attention to reduce redundant attention computation and improve drafting quality. Across various multimodal benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58×, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, respectively.