HAWK: Rethinking Multimodal Drafting for Speculative Decoding

Wenhan Yang; Anirudh Rao; Ashwin Chandra

cs.CVSep 30, 2026

HAWK: Rethinking Multimodal Drafting for Speculative Decoding

Authors: Wenhan Yang, Anirudh Rao, Ashwin Chandra

Organizations: University of California, Los Angeles · Samsung Research America

Abstract

Speculative decoding has achieved substantial lossless speedups for LLMs, but remains less effective for large vision-language models (LVLMs), where lightweight drafters struggle to use rich multimodal information. A second limitation is that standard distillation supervises the drafter only along the original training trajectory, without modeling how target predictions shift after the drafter's own proposals. As drafting moves away from this trajectory, the drafter can increasingly disagree with the target, reducing acceptance in later steps. We propose HAWK to address both limitations. HAWK uses representation similarity to select informative target layers and learns how to combine their hidden states. For visual information, it directly provides the drafter with compressed visual hidden states from the target model instead of raw visual tokens, making the visual information easier for a shallow drafter to use. HAWK also trains the drafter to capture how target predictions change after its own proposals, improving its agreement with the target during multi-step drafting. On SmolVLM-256M across ten multimodal benchmarks, HAWK raises average acceptance length from 3.32 to 4.08 and speedup from 2.19x to 2.60x over EAGLE-3 under greedy decoding, and from 2.89 to 3.41 and 1.92x to 2.19x under sampling.

Figures & tables

Figure 1: Overview of HAWK. (a) Target representations passed to the drafter. Inter-layer CKA divides the frozen target into early, middle, and late regions and selects representative layers. Visual hidden states from those layers are pooled by $4\times$ , then combined within each region by learned weights. The three fused streams are projected into a single-layer drafter, whose proposals the target verifies. (b) Prediction-shift supervision, used only during training. Standard distillation matches the target’s next-token prediction on the original context. The shift loss matches how that prediction changes after the drafter’s own token replaces the original token.

Panel of Figure 2 — Figure 2: Inter-layer CKA and the selected early, middle, and late regions. The boundaries differ across models, so one fixed depth rule does not fit all three.

Figure 3: (a) Probability that the first

k

draft tokens are all accepted, EAGLE-3 vs. Hawk . (b) Relative change of the same quantity over the drafter trained without auxiliary loss, for the supervision variants of Table 3 . The gain from prediction-shift supervision grows from 3% at

k=1

to 23% at

k=5

, against at most 10% for direct distillation.

Setting	SEED-Bench	MM-Vet	MathVista	SQA	GQA	DocVQA	Avg.
Acceptance length $\tau$
No pooling	4.03	3.15	2.85	2.36	3.80	2.08	3.05
$4\times$ pooling	4.03	3.25	2.92	2.45	3.86	2.09	3.10
$16\times$ pooling	4.00	3.20	2.75	2.43	3.82	2.03	3.04
Speedup ratio
No pooling	2.36	2.04	2.18	1.74	2.28	1.79	2.07

Table 2: Ablation of visual-state pooling on SmolVLM-256M. The other components are disabled.

Setting	SEED-Bench	MM-Vet	MathVista	SQA	GQA	DocVQA	Avg.
Acceptance length $\tau$
No auxiliary loss	4.03	3.15	2.85	2.36	3.80	2.08	3.05
Direct distillation	4.00	3.21	2.84	2.56	3.85	2.14	3.10
Shift, $w{=}0.3$	3.94	3.17	2.98	2.60	3.79	2.16	3.11
HAWK	4.01	3.29	2.97	2.68	3.88	2.16	3.17
Speedup ratio

Table 3: Ablation of prediction-shift supervision on SmolVLM-256M. The other components are disabled. Direct distillation matches the target distribution on substituted contexts, while HAWK supervises the induced prediction shift.

Setting	SEED-Bench	MM-Vet	MathVista	SQA	GQA	DocVQA	Avg.
Acceptance length $\tau$
EAGLE-3 layers (reference)	4.03	3.15	2.85	2.36	3.80	2.08	3.05
Uniform layers, learned mix	4.16	3.25	2.86	2.65	3.92	2.19	3.17
CKA layers, uniform mix	4.07	3.29	3.09	2.66	3.93	2.21	3.21
CKA layers, learned mix	4.18	3.34	3.14	2.67	3.90	2.22	3.24
Speedup ratio

Table 4: Ablation of layer selection and fusion on SmolVLM-256M. The other components are disabled. Uniform layers are evenly spaced. CKA layers are selected by representation similarity. Learned mixing trains the region weights; uniform mixing freezes them at equal weights.

Figure 4: Learned fusion weights

\alpha_{r,s}

of the Hawk drafter on SmolVLM-256M. Dotted lines mark uniform weights.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Drafter	Early / Middle / Late	Used in
EAGLE-3	$[1]$ / $[14]$ / $[26]$	Table 1 ; Tables 2 – 3
Hawk	$[1,2,8,10]$ / $[18,20]$ / $[23,26,28]$	Table 1
CKA coverage, $\epsilon_{\mathrm{CKA}}=0.008$	$[1,2,5,9]$ / $[14,18,21]$ / $[23,25,27]$	Table 4
Uniform grid	$[1,4,8,11]$ / $[14,18]$ / $[21,24,28]$	Table 4

Appendix

Table 6: Target layers (0-based decoder blocks) read by each SmolVLM-256M drafter.

Panel of Figure 5 — Figure 5: Region-wise CKA coverage as the number of selected target layers increases for (a) SmolVLM-256M and (b) Qwen2.5-VL-7B. Large markers show where the greedy selection stops at $\epsilon_{\mathrm{CKA}}=0.008$ ; dotted lines show the coverage of the single EAGLE-3 layer in each region.

Figure 6: Visual compression diagnostic. (a) Acceptance length of drafters conditioned on pooled target visual states. (b) Task accuracy of the target when its own visual input is compressed by the same factor.

	$k=1$	$k=2$	$k=3$	$k=4$	$k=5$
EAGLE-3 (20 ep)	0.789	0.603	0.363	0.248	0.161
Hawk (20 ep)	0.840	0.702	0.525	0.365	0.233
No auxiliary loss (4 ep)	0.767	0.585	0.344	0.214	0.136
Direct distillation (4 ep)	0.770	0.595	0.354	0.235	0.148
Prediction shift (4 ep)	0.789	0.612	0.373	0.248	0.168

Appendix

Table 9:

P(\text{accepted length}\geq k)

by draft step

k

, greedy decoding, ten benchmarks. 20-epoch rows use prompts whose output matches autoregressive decoding; 4-epoch rows use all prompts.

Explore similar work

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation
May 30, 2026Zining Liu, Yunhai Hu, Tianhua Xia +4Speculative Decoding Drafts
New York University · Cerebras Systems Inc.
Draft-OPD: On-Policy Distillation for Speculative Draft Models
May 28, 2026Haodi Lei, Yafu Li, Haoran Zhang +8Speculative Decoding Drafts
Shanghai Jiao Tong University · Shanghai AI Laboratory · The Chinese University of Hong Kong +3
Acceptance-Aware Draft Model Training for Speculative Decoding
Sep 21, 2026Tianhua Xia, Mugilan Ganesan, Yifei Feng +3Speculative Decoding Model Training
New York University · University of Waterloo · Independent Researcher

May 30, 2026, cs.LGJ/K move · Enter open · S save

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

Zining Liu, Yunhai Hu, Tianhua Xia, Bo Bao+3

New York University · Cerebras Systems Inc.

Speculative Decoding Drafts Autoregressive Generation Dream Multimodal Generation

Open paper AskPDF arXiv

Speculative decoding (SD) has proven to be an effective technique for accelerating autoregressive generation in large language models (LLMs) however, its application to vision-language models (VLMs) remains relatively unexplored. We propose~\textit{DREAM-S}, a novel SD framework designed specifically for fast and efficient decoding in VLMs. DREAM-S leverages a neural architecture search (NAS) framework with target-aware supernet training to automatically identify both the optimal interaction strategy between the draft and target models, and the most suitable draft model architecture for the underlying hardware implementation platform. DREAM-S additionally incorporates adaptive intermediate feature distillation, guided by attention entropy, to enable efficient draft training. Experiments on a range of well-established VLMs show that DREAM-S achieves up to a

3.85\times

speedup compared to standard decoding approaches and significantly outperforms existing SD baselines. The code is publicly available at: https://github.com/SAI-Lab-NYU/DREAM-S .

Abstract

Explore similar work

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

Draft-OPD: On-Policy Distillation for Speculative Draft Models

Acceptance-Aware Draft Model Training for Speculative Decoding