NarrowSpec: Draft Hidden Width Need Not Match the Target in Speculative Decoding
Organizations: NVIDIA
Abstract
Speculative decoding is widely used to speed up generation in large language models (LLMs), with a small draft model proposing tokens that one forward pass of the target model verifies. Draft models that read the target's hidden states, such as DFlash and EAGLE-3, typically inherit the target's hidden width. We show that the draft's whole residual stream can run at half or a quarter of that width while largely preserving acceptance. For DFlash, which borrows the target's vocabulary projection, the trained output adapter folds into it, giving one narrower projection with logits unchanged up to floating-point rounding. We test two drafter families, DFlash and EAGLE-3, on two targets, Qwen3-8B (NVFP4) and GPT-OSS-20B (MXFP4), in SGLang on a B200 and a DGX Spark (GB10). At batch size 1 and temperature 0, narrowing the draft to half or a quarter width speeds up end-to-end decoding by 1.06-1.12x on B200 and 1.13-1.44x on GB10. The speedup holds at larger batch sizes. From batch 32 to the largest batch each GPU fits, narrowing reaches up to 1.21x on both GPUs, and no narrow draft is slower than full width.
Figures & tables
| DFlash | EAGLE-3 | |
|---|---|---|
| Draft body | 5 / 8 stacked layers | one layer, run times |
| Drafting | block-parallel, one block per anchor | autoregressive chain |
| LM head | target’s, , borrowed frozen | its own , 32K / 64K , trained |
| Head at narrow width | folded: , | already at , no head fold needed |
| Input adapter at serving | runtime GEMM, 4.2–7.2 s | zero — folded into embed_tokens |
| Output adapter | , folded away at load | none |
| Qwen3-8B | GPT-OSS-20B | |
| Body / head precision | NVFP4 / BF16 dense | MXFP4 MoE / BF16 dense |
| Target hidden | 4,096 | 2,880 |
| Vocabulary | 151,936 | 201,088 |
| Corpus samples / tokens per epoch | 799,936 / 587M | 788,544 / 1.04B |
| Generation cap | 512 | 2,048 |
| DFlash draft — borrows the target’s head, folds its output adapter at load | ||
| Target | Draft | (ms) | (ms) | speedup | speedup | ||
|---|---|---|---|---|---|---|---|
| DFlash | |||||||
| Qwen3-8B | full 4,096 | 4.298 | — | 3.585 | 0.834 | 1.00 | 1.00 |
| half 2,048 | 4.334 | 3.313 | 0.764 | 1.08 | 1.09 | ||
| quarter 1,024 | 4.064 | 3.197 | 0.787 | 1.12 | 1.06 | ||
| stock z-lab | 4.304 | 3.584 | 0.833 | 1.00 | 1.00 | ||
| GPT-OSS-20B | full 2,880 | 4.376 | — | 3.342 | 0.765 | 1.00 | 1.00 |
| speedup | |||||
|---|---|---|---|---|---|
| Family | Target | half | quarter | half | quarter |
| DFlash | Qwen3-8B | 1.20 | 1.26 | ||
| DFlash | GPT-OSS-20B | 1.13 | 1.20 | ||
| EAGLE-3 | Qwen3-8B | 1.21 | 1.44 | ||
| EAGLE-3 | GPT-OSS-20B | 1.23 | 1.34 | ||
| Family / target | Draft | (ms) | vs full | (ms) | sum, vs full |
|---|---|---|---|---|---|
| DFlash, Qwen3-8B | full 4,096 | 0.906 | — | 2.578 | 3.484 |
| half 2,048 | 0.634 | −30.0% | 2.576 ( ) | 3.210 (1.09 ) | |
| quarter 1,024 | 0.517 | −43.0% | 2.590 ( ) | 3.106 (1.12 ) | |
| DFlash, GPT-OSS-20B | full 2,880 | 0.956 | — | 2.216 | 3.171 |
| half 1,440 | 0.724 | −24.3% | 2.227 ( ) | 2.951 (1.08 ) | |
| quarter 720 | 0.641 | −32.9% | 2.202 ( ) | 2.842 (1.12 ) |
| Family / target | Draft | draft total | decoder layers | head argmax | rest |
|---|---|---|---|---|---|
| DFlash, Qwen3-8B | full 4,096 | 906.1 | 527.8 | 218.5 | 159.8 |
| half | 634.1 (1.43 ) | 373.6 (1.41 ) | 111.6 (1.96 ) | 148.9 | |
| quarter | 516.6 (1.75 ) | 323.4 (1.63 ) | 65.4 (3.34 ) | 127.8 | |
| DFlash, GPT-OSS-20B | full 2,880 | 955.5 | 613.4 | 179.0 | 163.1 |
| half | 723.5 (1.32 ) | 478.4 (1.28 ) | 99.3 (1.80 ) | 145.8 | |
| quarter | 640.7 (1.49 ) | 432.0 (1.42 ) | 58.5 (3.06 ) | 150.2 |
| Family / target | Draft | attn. proj. | MLP | head | rest of body | outside | |
|---|---|---|---|---|---|---|---|
| DFlash, Qwen3-8B | full 4,096 | 96.3 | 271.4 | 206.2 | 199.4 | 132.9 | 906.1 |
| half | 70.1 (1.37 ) | 148.5 (1.83 ) | 99.4 (2.07 ) | 203.5 (0.98 ) | 112.6 | 634.1 (1.43 ) | |
| quarter | 53.4 (1.80 ) | 103.2 (2.63 ) | 53.3 (3.87 ) | 212.7 (0.94 ) | 94.0 | 516.6 (1.75 ) | |
| DFlash, GPT-OSS-20B | full 2,880 | 113.7 | 211.6 | 166.5 | 325.3 | 138.4 | 955.5 |
| half | 85.9 (1.32 ) | 131.8 (1.61 ) | 86.9 (1.92 ) | 300.7 (1.08 ) | 118.2 | 723.5 (1.32 ) | |
| quarter | 74.8 (1.52 ) | 101.7 (2.08 ) | 46.1 (3.61 ) | 297.8 (1.09 ) | 120.2 | 640.7 (1.49 ) |
| half width vs MLP | quarter width vs MLP | ||||||
|---|---|---|---|---|---|---|---|
| Family | Target | speedup | draft | needed | speedup | draft | needed |
| DFlash | Qwen3-8B | 1.09 | 1.07 | ||||
| DFlash | GPT-OSS-20B | 1.07 | 1.08 | ||||
| EAGLE-3 | Qwen3-8B | 1.08 | 1.14 | ||||
| EAGLE-3 | GPT-OSS-20B | 1.08 | 1.06 | ||||
| Family | Target | Row | ||||||
|---|---|---|---|---|---|---|---|---|
| DFlash | Qwen3-8B | half | 1.09 | 1.12 | 1.14 | 1.16 | 1.13 | 1.13 |
| quarter | 1.13 | 1.17 | 1.16 | 1.21 | 1.18 | 1.19 | ||
| draft share | 25.4% | 26.5% | 28.8% | 29.9% | 30.2% | 31.4% | ||
| GPT-OSS-20B | half | 1.05 | 1.05 | 1.05 | 1.09 | 1.07 | 1.09 | |
| quarter | 1.07 | 1.04 | 1.06 | 1.09 | 1.09 | 1.09 | ||
| draft share | 30.7% | 25.0% | 28.7% | 32.5% | 32.3% | 35.7% |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | Target | Draft | Layers fc | Input side | Output side | Total | vs full | Unfolded total |
|---|---|---|---|---|---|---|---|---|
| DFlash | Qwen3-8B | full 4,096 | 2.097 | — | borrowed | 2.097 | — | 2.097 |
| half 2,048 | 1.049 | 0.017 | 0.622 | 1.688 | 0.80 | 1.082 | ||
| quarter 1,024 | 0.524 | 0.008 | 0.311 | 0.844 | 0.40 | 0.541 | ||
| GPT-OSS-20B | full 2,880 | 1.570 | — | borrowed | 1.570 | — | 1.570 | |
| half 1,440 | 0.785 | 0.008 | 0.579 | 1.372 | 0.87 | 0.801 | ||
| quarter 720 | 0.392 | 0.004 | 0.290 | 0.686 | 0.44 | 0.401 |
| Family | Target | Draft | Learning rate | initializer_range |
|---|---|---|---|---|
| DFlash | both | every width | 6e-4 | default |
| EAGLE-3 | Qwen3-8B | full 4,096 | 2e-4 | 0.005 |
| half 2,048 | 4e-4 | 0.01 | ||
| quarter 1,024 | 8e-4 | 0.02 | ||
| EAGLE-3 | GPT-OSS-20B | full 2,880 | 3e-4 | 0.007 |
| half 1,440 | 6e-4 | 0.02 |
| Family / target | Draft | (ms) | (ms) | speedup | |
|---|---|---|---|---|---|
| DFlash, Qwen3-8B | full 4,096 | 2.818 | 3.724 | 1.326 | 1.00 |
| half | 2.825 | 3.459 | 1.229 | 1.08 | |
| quarter | 2.743 | 3.341 | 1.222 | 1.09 | |
| DFlash, GPT-OSS-20B | full 2,880 | 2.285 | 3.462 | 1.520 | 1.00 |
| half | 2.266 | 3.258 | 1.442 | 1.05 | |
| quarter | 2.219 | 3.144 | 1.421 | 1.07 |
| DFlash | EAGLE-3 | |||||||||||
| Qwen3-8B | GPT-OSS-20B | Qwen3-8B | GPT-OSS-20B | |||||||||
| full | half | qtr | full | half | qtr | full | half | qtr | full | half | qtr | |
| Category | % | % | % | % | % | % | % | % | ||||
| coding | 6.123 | 4.129 | 4.907 | 3.636 | ||||||||
| math | 6.174 | 3.951 | 4.613 | 3.369 | ||||||||
| stem | 5.317 | 4.132 | 4.252 | 3.514 | ||||||||
| DFlash | EAGLE-3 | |||||||
|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | GPT-OSS | Qwen3-8B | GPT-OSS | |||||
| Category | half | qtr | half | qtr | half | qtr | half | qtr |
| coding | 1.09 | 1.02 | 1.05 | 1.05 | 1.07 | 1.12 | 1.07 | 1.08 |
| math | 1.10 | 1.07 | 1.05 | 1.07 | 1.07 | 1.09 | 1.10 | 1.11 |
| stem | 1.09 | 1.06 | 1.05 | 1.07 | 1.09 | 1.14 | 1.08 | 1.10 |
| humanities | 1.09 | 1.07 | 1.06 | 1.09 | 1.08 | 1.13 | 1.11 | 1.12 |
| DFlash | EAGLE-3 | |||||||||||
| Qwen3-8B | GPT-OSS-20B | Qwen3-8B | GPT-OSS-20B | |||||||||
| full | half | qtr | full | half | qtr | full | half | qtr | full | half | qtr | |
| Category | % | % | % | % | % | % | % | % | ||||
| coding | 3.939 | 2.887 | 3.719 | 2.639 | ||||||||
| math | 3.245 | 2.479 | 2.994 | 2.295 | ||||||||
| stem | 2.890 | 2.302 | 2.684 | 2.100 | ||||||||
| DFlash | EAGLE-3 | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | GPT-OSS-20B | Qwen3-8B | GPT-OSS-20B | |||||||||
| Benchmark | full | half | qtr | full | half | qtr | full | half | qtr | full | half | qtr |
| gsm8k | 4.840 | 5.034 | 4.841 | 3.072 | 3.214 | 3.117 | 4.265 | 4.037 | 4.099 | 2.862 | 2.851 | 2.788 |
| math500 | 5.003 | 5.033 | 5.010 | 3.210 | 3.213 | 3.176 | 4.529 | 4.292 | 4.249 | 3.032 | 3.022 | 2.984 |
| aime25 | 3.769 | 3.651 | 3.758 | 2.782 | 2.670 | 2.721 | 3.509 | 3.511 | 3.591 | 2.568 | 2.505 | 2.535 |
| humaneval | 6.000 | 6.018 | 5.811 | 2.961 | 2.991 | 2.940 | 4.984 | 4.799 | 4.820 | 2.811 | 2.755 | 2.753 |
| DFlash | EAGLE-3 | |||||||
| Qwen3-8B | GPT-OSS | Qwen3-8B | GPT-OSS | |||||
| Benchmark | half | qtr | half | qtr | half | qtr | half | qtr |
| Temperature 0 | ||||||||
| gsm8k | 1.10 | 1.11 | 1.06 | 1.06 | 1.05 | 1.11 | 1.06 | 1.10 |
| math500 | 1.10 | 1.12 | 1.06 | 1.06 | 1.05 | 1.10 | 1.07 | 1.09 |
| aime25 | 1.10 | 1.11 | 1.06 | 1.06 | 1.06 | 1.10 | 1.07 | 1.11 |
| Family | Target | Reference | ref. | ours vs ref. | ref. | ours vs ref. |
|---|---|---|---|---|---|---|
| DFlash | Qwen3-8B | z-lab | 4.304 | 2.835 | ||
| DFlash | GPT-OSS-20B | z-lab | 4.309 | 2.332 | −2.0% | |
| EAGLE-3 | Qwen3-8B | RedHatAI | 2.846 | 2.186 | ||
| EAGLE-3 | GPT-OSS-20B | RedHatAI | 3.413 | 2.115 | −1.4% | |
| Family | Target | Row | ||||||
|---|---|---|---|---|---|---|---|---|
| DFlash | Qwen3-8B | full-width (ms) | 3.50 | 7.42 | 12.30 | 22.33 | 42.58 | 83.69 |
| full-width draft | 0.895 | 1.831 | 3.386 | 6.516 | 12.683 | 26.556 | ||
| full-width verify | 2.629 | 5.070 | 8.360 | 15.297 | 29.310 | 57.915 | ||
| draft share | 25.4% | 26.5% | 28.8% | 29.9% | 30.2% | 31.4% | ||
| half draft | 0.643 | 1.320 | 2.213 | 4.113 | 8.414 | 17.905 | ||
| quarter draft | 0.525 | 1.051 | 1.731 | 3.024 | 6.114 | 13.357 |
| 8k 32k | |||||||
|---|---|---|---|---|---|---|---|
| Family | Target | Draft | 8k | 16k | 32k | full | speedup |
| DFlash | Qwen3-8B | full | 3.770 | 3.902 | 4.098 | 8.7% | — |
| half | 3.482 (1.08 ) | 3.620 (1.08 ) | 3.806 (1.08 ) | — | 0.6% | ||
| quarter | 3.340 (1.13 ) | 3.476 (1.12 ) | 3.675 (1.12 ) | — | 1.2% | ||
| GPT-OSS-20B | full | 3.410 | 3.491 | 3.547 | 4.0% | — | |
| half | 3.206 (1.06 ) | 3.303 (1.06 ) | 3.364 (1.05 ) | — | 0.9% | ||
| Qwen3-8B | GPT-OSS-20B | |||||||||
| Operation | full | half | qtr | half save | qtr save | full | half | qtr | half save | qtr save |
| DFlash ( s per iteration) | ||||||||||
| folded LM head | 206.2 | 99.4 | 53.3 | 42.4% | 43.6% | 166.5 | 86.9 | 46.1 | 37.6% | 40.6% |
| FC1 ( gate_up ) | 180.7 | 95.3 | 56.0 | 33.9% | 35.5% | 134.1 | 76.9 | 50.0 | 27.0% | 28.3% |
| FC2 ( down ) | 90.7 | 53.2 | 47.2 | 14.9% | 12.4% | 77.5 | 54.9 | 51.7 | 10.7% | 8.7% |
| QKV projection | 52.3 | 34.2 | 26.1 | 7.2% | 7.5% | 60.5 | 41.7 | 32.2 | 8.9% | 9.5% |
| Setting | B200 | GB10 |
|---|---|---|
| GPU | SM100, HBM3e, 8 TB/s | SM121, unified LPDDR5x, 273 GB/s |
| Target attention, Qwen3-8B | TRT-LLM MHA | FlashInfer |
| Target attention, GPT-OSS-20B | TRT-LLM MHA | FA4 |
| KV page size (follows the target attention backend) | 64 | 1 |
| Static memory fraction | 0.75–0.9 | 0.5 |
| Draft attention | FA4 | FA4 |
| Target | Draft | (ms) | (ms) | speedup | speedup | ||
|---|---|---|---|---|---|---|---|
| DFlash | |||||||
| Qwen3-8B | full 4,096 | 4.156 | — | 41.810 | 10.060 | 1.00 | 1.00 |
| half 2,048 | 4.149 | 34.718 | 8.368 | 1.20 | 1.20 | ||
| quarter 1,024 | 3.899 | 31.025 | 7.956 | 1.35 | 1.26 | ||
| GPT-OSS-20B | full 2,880 | 4.395 | — | 46.232 | 10.520 | 1.00 | 1.00 |
| half 1,440 | 4.306 | 40.142 | 9.323 | 1.15 | 1.13 | ||
| Target | Draft | (ms) | (ms) | speedup | speedup | ||
|---|---|---|---|---|---|---|---|
| DFlash | |||||||
| Qwen3-8B | full 4,096 | 2.848 | — | 42.769 | 15.016 | 1.00 | 1.00 |
| half 2,048 | 2.828 | 35.587 | 12.582 | 1.20 | 1.19 | ||
| quarter 1,024 | 2.738 | 32.004 | 11.688 | 1.34 | 1.28 | ||
| GPT-OSS-20B | full 2,880 | 2.275 | — | 45.786 | 20.128 | 1.00 | 1.00 |
| half 1,440 | 2.249 | 39.828 | 17.713 | 1.15 | 1.14 | ||
| Family / target | Draft | (ms) | vs full | (ms) | sum, vs full |
|---|---|---|---|---|---|
| DFlash, Qwen3-8B | full 4,096 | 15.38 | — | 25.42 | 40.80 |
| half 2,048 | 8.10 | −47.3% | 25.32 ( ) | 33.42 (1.22 ) | |
| quarter 1,024 | 4.30 | −72.0% | 25.42 ( ) | 29.72 (1.37 ) | |
| DFlash, GPT-OSS-20B | full 2,880 | 12.80 | — | 31.45 | 44.25 |
| half 1,440 | 6.67 | −47.9% | 31.72 ( ) | 38.39 (1.15 ) | |
| quarter 720 | 3.72 | −70.9% | 32.01 ( ) | 35.73 (1.24 ) |
| Family / target | Draft | attn. proj. | MLP | head | rest of body | outside | |
|---|---|---|---|---|---|---|---|
| DFlash, Qwen3-8B | full 4,096 | 1.92 | 6.70 | 5.39 | 0.34 | 1.04 | 15.38 |
| half | 0.96 (2.00 ) | 3.39 (1.97 ) | 2.68 (2.01 ) | 0.31 (1.12 ) | 0.76 | 8.10 (1.90 ) | |
| quarter | 0.52 (3.69 ) | 1.71 (3.91 ) | 1.36 (3.95 ) | 0.28 (1.20 ) | 0.43 | 4.30 (3.57 ) | |
| DFlash, GPT-OSS-20B | full 2,880 | 1.93 | 4.73 | 5.02 | 0.43 | 0.69 | 12.80 |
| half | 1.02 (1.90 ) | 2.36 (2.00 ) | 2.53 (1.99 ) | 0.36 (1.20 ) | 0.41 | 6.67 (1.92 ) | |
| quarter | 0.52 (3.70 ) | 1.26 (3.76 ) | 1.38 (3.63 ) | 0.35 (1.23 ) | 0.21 | 3.72 (3.44 ) |
| B200 | GB10 | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| weight | s per kernel | % of peak | s per kernel | % of peak | ||||||||||
| Operation | kernels | MB | full | half | qtr | full | half | qtr | full | half | qtr | full | half | qtr |
| Vocabulary projection | 1 | 1,245 | 206.2 | 99.4 | 53.3 | 75 | 78 | 73 | 5,388 | 2,679 | 1,363 | 85 | 85 | 84 |
| FC1 (gate/up) | 5 | 201 | 36.1 | 19.1 | 11.2 | 70 | 66 | 56 | 882 | 445 | 226 | 84 | 83 | 82 |
| FC2 (down) | 5 | 101 | 18.1 | 10.6 | 9.4 | 69 | 59 | 33 | 457 | 234 | 116 | 81 | 79 | 79 |
| QKV projection | 5 | 50 | 10.5 | 6.8 | 5.2 | 60 | 46 | 30 | 228 | 114 | 58 | 81 | 81 | 80 |
| Family | Target | Row | ||||||
|---|---|---|---|---|---|---|---|---|
| DFlash | Qwen3-8B | full-width (ms) | 45.4 | 110.3 | 187.0 | 383.3 | 804.4 | — |
| half | 1.18 | 1.13 | 1.12 | 1.14 | 1.12 | — | ||
| quarter | 1.29 | 1.17 | 1.16 | 1.19 | 1.16 | — | ||
| draft share | 36.9% | 30.9% | 27.6% | 28.1% | 26.1% | — | ||
| GPT-OSS-20B | full-width (ms) | 45.7 | 96.5 | 126.8 | 202.0 | 382.1 | 709.9 | |
| half | 1.13 | 1.06 | 1.06 | 1.06 | 1.09 | 1.09 |
| Draft | (ms) | (ms) | (ms) | (ms) | speedup | B200 | ||
|---|---|---|---|---|---|---|---|---|
| full 4,096 | 3.886 | — | 6.247 | 1.607 | 1.502 | 4.800 | 1.00 | 1.00 |
| half 2,048 | 3.914 | 5.682 | 1.452 | 0.930 ( ) | 4.807 | 1.11 | 1.09 | |
| quarter 1,024 | 3.715 | 5.395 | 1.452 | 0.633 ( ) | 4.810 | 1.11 | 1.06 |
| Draft | attn. proj. | MLP | head | rest of body | outside | |
|---|---|---|---|---|---|---|
| full 4,096 | 175.7 | 536.8 | 414.8 | 197.8 | 177.1 | 1502.3 |
| half 2,048 | 103.0 (1.71 ) | 285.3 (1.88 ) | 208.0 (1.99 ) | 201.5 (0.98 ) | 132.2 | 929.9 (1.62 ) |
| quarter 1,024 | 65.9 (2.67 ) | 159.4 (3.37 ) | 107.6 (3.86 ) | 193.6 (1.02 ) | 107.0 | 633.4 (2.37 ) |
| Draft | Precision | (ms) | (ms) | speedup | speedup | vs full BF16 | ||
|---|---|---|---|---|---|---|---|---|
| full 4,096 | BF16 | 4.298 | — | 3.585 | 0.834 | — | — | 1.00 |
| half 2,048 | BF16 | 4.334 | — | 3.313 | 0.764 | — | — | 1.09 |
| quarter 1,024 | BF16 | 4.064 | — | 3.197 | 0.787 | — | — | 1.06 |
| full 4,096 | W4A4 | 4.196 | 3.284 | 0.782 | 1.09 | 1.07 | 1.07 | |
| half 2,048 | W4A4 | 4.232 | 3.180 | 0.751 | 1.04 | 1.02 | 1.11 | |
| quarter 1,024 | W4A4 | 3.925 | 3.130 | 0.797 | 1.02 | 0.99 | 1.05 |
Speculative Decoding
Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.
See all 357 papers