VisHead-Spec: Head-Routed Visual Attention for Multimodal Speculative Decoding
Organizations: Meta Recommendation Systems (MRS)
Abstract
Speculative decoding accelerates vision-language models(VLMs) while preserving the target model's outputs, but its speedup diminishes under high concurrency serving as the system becomes more compute bounded as the batch size increases. This problem is pronounced for multimodal speculative decoding as the long range of the image tokens in the prompt can significantly increase the computational cost during draft token prediction. The block-diffusion drafters (e.g. DFlash, DSpark) can effectively speed up the draft model efficiency by predicting multiple draft tokens in one forward pass, but their speedup for VLMs under high concurrency is still modest (<9%). In order to understand this gap, we examine the information flow between different modalities from target model to draft model and identify the sparsity pattern of visual attention between the target and draft models: only small set of key-value heads in the draft model need direct visual feature attention in the context and only visual features from top level layers in the target model need to be attended in the context. We therefore propose VisHead-Spec, which selects a small, fixed set of key-value (KV) heads to attend to all image features from top level layers in the target model and restricts the remaining heads to text features. On Qwen3-VL-8B and MMSpec, VisHead-Spec-8 lets 8 of the drafter's 40 KV heads read the image while keeping accepted length within 0.3% of the block-diffusion draft model baseline (i.e. DSpark). At batch size 64, its throughput gain over autoregressive decoding increases to 16.0%, compared with 9.5% for DSpark; at batch size 1, its speedup remains comparable to DSpark.
Figures & tables
| Method | AL | Batch size 1 | Batch size 8 | Batch size 16 | Batch size 32 | Batch size 64 |
|---|---|---|---|---|---|---|
| AR | 1 | 1x | 1x | 1x | 1x | 1x |
| EAGLE-3, chain | 3.00 | 1.85x | 1.59x | 1.36x | 1.00x | 0.89x |
| EAGLE-3, tree | 4.32 | 2.34x | 1.42x | 0.95x | 0.64x | 0.50x |
| DFlash | 3.01 | 2.14x | 1.76x | 1.53x | 1.23x | 1.03x |
| DSpark | 3.28 | 2.26x | 1.86x | 1.62x | 1.29x | 1.09x |
| VisHead-Spec-8 (ours) | 3.27 | 2.23x | 1.88x | 1.65x | 1.35x | 1.16x |
| Method | AL | Batch size 1 | Batch size 8 | Batch size 16 | Batch size 32 | Batch size 64 |
|---|---|---|---|---|---|---|
| DSpark-scratch | 2.29 | 1.58x | 1.34x | 1.17x | 0.96x | 0.82x |
| MSD, chain | 1.89 | 1.17x | 1.04x | 0.92x | 0.68x | 0.62x |
| MSD, tree | 2.56 | 1.61x | 0.98x | 0.65x | 0.44x | 0.34x |
| ViSpec, chain | 1.82 | 1.09x | 1.00x | 0.88x | 0.64x | 0.60x |
| ViSpec, tree | 2.44 | 1.49x | 0.92x | 0.61x | 0.41x | 0.32x |
| VisHead-Spec-8-scratch | 2.30 | 1.58x | 1.35x | 1.23x | 1.01x | 0.88x |
| Method | General VQA | Text VQA | Image captioning | Chart understanding | Complex reasoning | Multi-turn conversation | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AL | Speedup | AL | Speedup | AL | Speedup | AL | Speedup | AL | Speedup | AL | Speedup | |
| AR | — | 1.000 | — | 1.000 | — | 1.000 | — | 1.000 | — | 1.000 | — | 1.000 |
| VisHead-8 | 3.056 | 1.21x | 3.130 | 1.15x | 2.961 | 1.10x | 3.628 | 1.26x | 3.451 | 1.18x | 3.196 | 1.11x |
| VisHead-4 | 3.058 | 1.21x | 3.110 | 1.15x | 2.955 | 1.11x | 3.624 | 1.26x | 3.445 | 1.18x | 3.176 | 1.12x |
| DSpark | 3.058 | 1.18x | 3.169 | 1.11x | 2.965 | 1.08x | 3.650 | 1.19x | 3.452 | 1.13x | 3.199 | 1.00x |
| DFlash | 2.807 | 1.13x | 2.903 | 1.06x | 2.709 | 1.01x | 3.377 | 1.14x | 3.182 | 1.06x | 2.933 | 0.96x |
| Variant | AL | Batch size 1 speedup | Batch size 8 speedup | Batch size 16 speedup | Batch size 32 speedup | Batch size 64 speedup |
|---|---|---|---|---|---|---|
| VisHead-Spec-0 | 3.152 | 2.19x | 1.83x | 1.62x | 1.31x | 1.12x |
| VisHead-Spec-8 (ours) | 3.27 | 2.23x | 1.88x | 1.65x | 1.35x | 1.16x |
| VisHead-Spec-4 (ours) | 3.26 | 2.24x | 1.87x | 1.66x | 1.35x | 1.16x |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Layers with image attention | AL gain retained (%) |
|---|---|
| 0 | 0.1 |
| 1 | 3.4 |
| 2 | 29.6 |
| 3 | 41.2 |
| 4 | 10.6 |
| 2, 3 | 81.3 |
| KV heads attending to image context | Head count for Layer 2 + layer 3 | Offline AL | Note |
|---|---|---|---|
| 16 | 8 + 8 | 3.32 | All heads in layers 2 and 3 |
| 12 | 8 + 4 | 3.32 | Best at budget 12 |
| 10 | 6 + 4 | 3.31 | Best at budget 10 |
| 8 | 4 + 4 | 3.29 | VisHead-Spec-8 |
| 8 | 6 + 2 | 3.29 | Best at budget 8 |
| 6 | 3 + 3 | 3.27 | Best at budget 6 |
| Item | Setting |
|---|---|
| Target | Qwen3-VL-8B-Instruct (frozen; 36 layers, hidden size 4096) |
| Drafter (VisHead-Spec and block-diffusion drafter baseline) | 5 Qwen3 decoder layers, 32 query / 8 KV heads, head dimension 128; block size 7; Markov head of rank 256; confidence head; context projection of hidden states from layers 1, 9, 17, 25, 33 in the target model; embedding and LM head shared with the target and frozen |
| Training warm start | deepseek-ai/dspark_qwen3_8b_block7 (VisHead-Spec and block-diffusion drafter baseline), dflash_qwen3_8b_block7 (DFlash), eagle3_qwen3_8b_ttt7 (EAGLE-3) |
| Training data | 67,304 sequences: 68,000 first-turn prompts from LLaVA-1.5 mix665k (first answer 5 words; MMSpec images and questions removed; about 15% text-only ShareGPT), re-answered greedily by the target at native resolution ( 2,048 tokens; looping answers dropped) |
| Schedule | 3 epochs = 8,412 steps; global batch 24; AdamW, learning rate 4 10 - 5 , 4% warmup, cosine decay to 0; gradient clipping 1.0; bf16; AMD MI300X; identical sample order for all drafters; final checkpoint |
| Head router | per-head block mask; image token id of Qwen3-VL; no added parameters |
| Method | Architecture | Training schedule | Optimizer steps / data |
|---|---|---|---|
| DSpark-scratch | 5 layers; 32 Q / 8 KV heads per layer; all 40 KV heads read images | AdamW; LR 6 10 - 4 ; global batch 512; 4% warmup, cosine decay | 10 epochs; 1,310 optimizer steps |
| VisHead-Spec-8-scratch | Same decoder as DSpark; 8 image-reading KV heads in layers 2 and 3 | Same schedule as DSpark-scratch | 10 epochs; 1,310 optimizer steps |
| MSD | 1 layer; 32 Q / 8 KV heads; final target hidden state plus token/visual embedding | AdamW; LR 5 10 - 5 ; global batch 32; 2,000 warmup steps; norm clipping; paper objective | 21,030 steps; text-first curriculum: 63k text draws, 63k draws ramping image fraction 0 1, then image data |
| ViSpec | 1 layer; 32 KV heads; image adaptor with 2 queries, one retained image row and one global feature | Stage 1: AdamW 3 10 - 5 on text. Stage 2: AdamW 3 10 - 6 on image data; global batch 8; MTP 1 | Stage 1: 9,511 text rows 10. Stage 2: 57,793 image rows 10; 72,240 image-stage steps |
| Task | VisHead-0 | VisHead-8 | VisHead-4 | DFlash | EAGLE-3 chain | EAGLE-3 tree |
|---|---|---|---|---|---|---|
| General VQA | 2.02 0.32 | +1.37 0.26 | +1.79 0.06 | 4.44 0.44 | 15.07 0.55 | 54.16 0.17 |
| Text VQA | 3.79 0.17 | +3.66 0.63 | +3.96 0.21 | 4.26 0.27 | 11.63 0.03 | 45.68 0.24 |
| Image captioning | 0.31 0.25 | +1.02 0.48 | +2.04 0.31 | 6.79 0.97 | 17.35 0.23 | 59.39 0.02 |
| Chart understanding | +3.56 0.21 | +5.86 0.41 | +6.15 0.64 | 4.26 0.09 | 10.94 0.24 | 54.37 0.08 |
| Complex reasoning | +3.46 0.63 | +4.36 0.00 | +4.57 0.11 | 5.60 0.30 | 17.60 0.72 | 57.96 0.15 |
| Multi-turn conversation | +5.24 0.01 | +10.38 0.30 | +11.34 0.25 | 4.82 0.03 | 14.21 0.27 | 47.44 0.29 |
| Task | VisHead-0 | VisHead-8 | VisHead-4 | DFlash | EAGLE-3 chain | EAGLE-3 tree |
|---|---|---|---|---|---|---|
| General VQA | 1.13 0.62 | 0.31 0.93 | +0.21 0.08 | 4.73 0.01 | 21.29 2.25 | 50.67 0.11 |
| Text VQA | 5.05 0.34 | +1.88 0.53 | +2.09 0.47 | 4.78 0.17 | 16.56 0.57 | 43.89 0.04 |
| Image captioning | 0.04 0.71 | +1.60 0.04 | +0.65 0.03 | 7.22 0.37 | 20.19 0.44 | 52.56 0.45 |
| Chart understanding | +1.55 1.03 | +3.64 0.71 | +3.42 0.44 | 4.16 0.48 | 15.89 0.09 | 49.87 0.31 |
| Complex reasoning | +2.18 0.41 | +2.49 0.87 | +1.28 0.10 | 6.65 0.02 | 21.25 0.12 | 52.59 0.01 |
| Multi-turn conversation | +4.00 0.69 | +7.53 0.41 | +6.80 0.32 | 5.53 0.18 | 20.88 0.40 | 44.62 0.86 |
| Task | VisHead-0 | VisHead-8 | VisHead-4 | DFlash | EAGLE-3 chain | EAGLE-3 tree |
|---|---|---|---|---|---|---|
| General VQA | 1.46 0.09 | +0.47 0.54 | +0.11 0.11 | 4.05 0.56 | 22.15 0.58 | 42.64 0.22 |
| Text VQA | 6.30 0.09 | +0.73 0.95 | +1.82 0.68 | 3.92 0.77 | 16.24 0.56 | 37.48 0.45 |
| Image captioning | 0.71 1.07 | +1.40 0.24 | +0.20 1.36 | 6.55 0.99 | 17.80 1.11 | 43.41 0.39 |
| Chart understanding | 1.04 0.47 | +2.00 1.13 | +2.17 0.78 | 5.00 0.46 | 16.03 0.02 | 42.56 0.61 |
| Complex reasoning | 1.69 0.60 | +0.19 0.89 | 0.22 0.13 | 7.21 1.22 | 16.06 0.28 | 42.52 0.45 |
| Multi-turn conversation | +0.73 0.88 | +5.55 0.58 | +5.74 0.83 | 6.18 0.41 | 14.12 1.05 | 35.72 0.84 |
| Task | VisHead-0 | VisHead-8 | VisHead-4 | DFlash | EAGLE-3 chain | EAGLE-3 tree |
|---|---|---|---|---|---|---|
| General VQA | 1.96 0.37 | 1.18 2.18 | 1.19 1.01 | 4.20 0.38 | 20.22 0.94 | 25.96 0.25 |
| Text VQA | 7.82 0.21 | +0.36 0.04 | +0.50 0.37 | 5.43 0.16 | 15.19 0.01 | 22.75 0.30 |
| Image captioning | 3.10 2.22 | 0.28 0.84 | 0.87 2.42 | 6.94 0.33 | 17.39 0.31 | 22.88 0.77 |
| Chart understanding | 2.41 0.99 | +0.06 0.53 | +0.56 0.69 | 5.42 1.03 | 13.90 0.60 | 25.86 0.07 |
| Complex reasoning | 0.32 1.01 | 0.63 0.09 | 0.45 1.16 | 3.67 0.44 | 13.67 0.00 | 24.58 0.08 |
| Multi-turn conversation | 0.31 0.33 | +5.45 0.39 | +3.46 0.49 | 5.62 0.35 | 11.38 0.15 | 19.49 0.12 |
| Task | VisHead-0 | VisHead-8 | VisHead-4 | DFlash | EAGLE-3 chain | EAGLE-3 tree |
|---|---|---|---|---|---|---|
| General VQA | 1.68 0.03 | 1.72 0.47 | 0.67 0.13 | 5.06 0.54 | 18.32 0.29 | +5.10 1.41 |
| Text VQA | 10.82 0.11 | 1.80 0.20 | 2.55 0.50 | 5.70 0.78 | 19.63 0.35 | +2.18 0.46 |
| Image captioning | 1.60 0.57 | 1.86 0.19 | 1.23 0.47 | 5.52 0.05 | 18.74 0.24 | +9.12 0.03 |
| Chart understanding | 2.98 0.37 | 1.62 0.06 | 1.28 0.27 | 4.87 0.50 | 17.63 0.13 | 0.80 0.59 |
| Complex reasoning | 1.32 0.35 | 1.39 0.08 | 1.13 0.45 | 5.30 0.02 | 18.38 0.01 | +2.03 0.17 |
| Multi-turn conversation | 3.72 0.44 | 0.07 0.14 | 0.12 0.22 | 5.84 0.04 | 17.41 0.46 | +4.16 0.32 |
| Method | Top-1 agreement for overall / visual / non-visual | AL | Throughput speedup |
|---|---|---|---|
| Full image access | 65.40 / 38.81 / 72.78 | 3.298 | 2.389 |
| VisHead-4 (always on) | 64.33 / 34.87 / 72.50 | 3.240 | 2.271 |
| VisHead-4 + learned per-token router | 64.10 / 34.05 / 72.43 | 3.187 | 2.219 |
Efficient VLM Inference
Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.
See all 530 papers