cs.DCOct 29, 2026

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

Authors: Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler, Gerhard Wellein

Organizations: Erlangen National High Performance Computing Center, Erlangen, Germany

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language model serving is limited by facility power. Energy per token determines how many tokens a datacenter can serve, while a latency target bounds how quickly each user receives them. During decode, every weight is read from memory for every generated token. Batching amortizes this read across users but makes no individual user faster. Speculative decoding does: a drafter proposes several future tokens, and the target model verifies them in a single forward pass that reuses the same weight read. Because every drafted token must be verified whether or not it is accepted, the energy speculation saves depends on how many drafted tokens are accepted. This saving is hard to pin down. It varies across target models from different attention families and across drafting schemes, and it depends on the text being served. Every drafter we evaluate accepts fewer tokens on conversation than on code, and every trained drafter accepts fewer on Romansh, a low-resource language, than on English. We measure speculative decoding under production serving conditions (SGLang, CUDA graphs enabled) on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. On dense models, verification, not drafting, dominates the energy cost. The saving shrinks as the batch grows, because batching has already shared the weight read that speculation would divide among the accepted tokens. At large batch sizes, block drafters push verification past the balance point between arithmetic and memory traffic into the compute-bound regime, where the saving becomes a loss, and a drafter with low acceptance raises energy per token. Meeting a tighter latency target with speculation then costs additional energy. Larger batches lower energy per token without speculation. Speculation adds to that saving only when acceptance is high, and on the model we tested it leaves little headroom for latency-preserving DVFS.

Figures & tables

familymodelGPUn-gramMTP headEAGLE3DFlashDSpark
GQAQwen3-4BH1004–8 t16 t , 7 t7 t
 Qwen3-8BH1004–8 t16 t , 7 t7 t
 Qwen3-14BH1004–8 t7 t7 t
 Qwen3-Coder-30B-A3B-Instruct ∗H1004––16 t–
GQA+SWAgemma-4-12B-itH10046 v–16 t7 t
 gemma-4-26B-A4B-it ∗H10046 v–16 t–
Table 1. Drafters per model, given as the number of tokens each verifies per request and step and marked by who trained it (c with the model, v by the model’s vendor afterwards, as Google’s assistant for the gemma models, t by a third party), with the GPU each model’s batch sweep runs on, and asterisks marking Mixture-of-Experts models.
familymodelB=1B{=}1B=2B{=}2B=4B{=}4B=8B{=}8B=16B{=}16B=32B{=}32B=64B{=}64
GDN ++ GAQwen3.6-27B+65 ±\pm 5 DF+61 ±\pm 2 DF+54 ±\pm 1 DF+48 ±\pm 1 MTP+45 ±\pm 1 MTP+38 ±\pm 1 MTP–
 Qwen3.6-35B-A3B+38 ±\pm 4 MTP+31 ±\pm 8 MTP+26 ±\pm 2 MTP+25 ±\pm 2 MTP+31 ±\pm 3 MTP+31 ±\pm 1 MTP+25 ±\pm 1 MTP
GQAQwen3-14B+54 ±\pm 1 DS+53 ±\pm 3 DS+49 ±\pm 2 DS+41 ±\pm 1 DS+35 ±\pm 1 DS+34 ±\pm 0 DS+18 ±\pm 1 DS
 Qwen3-4B+56 ±\pm 5 DS+53 ±\pm 4 DS+48 ±\pm 1 DS+41 ±\pm 1 DS+29 ±\pm 2 DS+25 ±\pm 1 DS+19 ±\pm 1 DS
 Qwen3-8B+62 ±\pm 3 DS+55 ±\pm 3 DS+50 ±\pm 2 DS+41 ±\pm 2 DS+37 ±\pm 1 DS+34 ±\pm 0 DS+24 ±\pm 1 DS
 Qwen3-Coder-30B-A3B-Instruct+32 ±\pm 10 DF+35 ±\pm 5 DF+41 ±\pm 10 DF+48 ±\pm 7 DF+49 ±\pm 3 DF+51 ±\pm 1 DF+51 ±\pm 2 DF
Table 2. Largest energy saving among the schemes (%) per model and batch with CUDA graph capture enabled, on the GPU given in Table 1 , as the median over repeated runs ±\pm half their range, the scheme that achieves it (DF DFlash, DS DSpark, MTP an MTP head or gemma’s assistant), and a dash where no speculative run held the batch.
  baseline (J/token)speculative (J/token)saving (%)
modelschemeoffonoffonoffon
B=1B{=}1
gemma-4-31B-itDFlash12.8012.652.962.91+77+77+77\mathbf{+77}
gemma-4-12B-itDFlash6.334.921.501.23+76+76+75\mathbf{+75}
Qwen3.6-27BDFlash11.6010.513.433.04+71+71+71\mathbf{+71}
Qwen3-8BDSpark3.793.101.601.23+57+57+60\mathbf{+60}
Table 3. Energy per output token of the non-speculative baseline and of the scheme with the largest saving with capture enabled, with CUDA graph capture off and on, and the saving each pair yields as the median of per-seed savings over three seeds with every speculative run paired to the baseline of the same seed and capture state, from a separate campaign run on the H100 except for JoyAI-LLM-Flash on the H200, so the Qwen3.6 rows differ from Table 2 .
Figure 2. Energy per output token against batch with CUDA graph capture enabled for 9 of the 11 models, grouped by attention family and labelled where the panel ran on the H200, with the non-speculative baseline in grey, asterisks marking Mixture-of-Experts models, crosses at batches the engine ran below the requested concurrency, and lines that stop short ending where the engine could not start.
Figure 3. Romansh-to-English acceptance ratio and energy saving at a batch of 1, one row per model and drafter labelled with the number of tokens it verifies per step, coloured by how the drafter was trained, and drawn light where the model does not answer consistently in Romansh.
Figure 4. Energy per token against per-user throughput with CUDA graph capture enabled, one panel per attention family, with each trace running from a batch of 1 (hollow marker) to a batch of 64 or to the batch given in its label, shaded over the range of its three seeds.
 baseline (ms)fastest speculative scheme (ms)
modelB=1B{=}1B=32B{=}32B=64B{=}64B=1B{=}1B=32B{=}32B=64B{=}64
GLM-4.7-Flash8.223.428.06.2 MTP18.2 MTP21.5 MTP
JoyAI-LLM-Flash †4.215.420.13.0 MTP10.8 MTP13.6 MTP
Qwen3-14B15.416.917.97.3 DS9.8 DS14.7 DS
Qwen3-4B5.76.87.72.7 DS4.3 DS6.0 DS
Qwen3-8B9.410.211.33.8 DS6.2 DS8.7 DS
Table 4. Mean time between output tokens per user in milliseconds with CUDA graph capture enabled on the code pool, for the non-speculative baseline and for the speculative scheme with the shortest time at each batch, with a dash where the engine did not hold the batch and a dagger where prefill time is not subtracted.
Figure 5. Energy saving against accepted tokens per verification for every model, scheme and batch that has a baseline run at the same batch size, coloured by batch (a), and the share of decode energy spent drafting and verifying at B=1B{=}1 and B=64B{=}64 with capture enabled for every dense model and scheme, measured by removing the draft (b).
Figure 6. Arithmetic intensity of a decode step on Qwen3-4B against batch, with the balance point of each GPU as a horizontal line (a), and the energy saving of the same drafters with capture enabled on each GPU, filled where the step at that batch is past the balance point of that GPU (b), shaded in (a) from the batch at which the DFlash-16 step crosses each balance point and in (b) from the batch at which its saving turns negative on each GPU.
More in this topic

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 314 papers

Explore similar work

CardsList

  1. Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware

    Jul 19, 2026Param ChordiyaOn-Device Language Model InferenceSpeculative Decoding

    University of California, San Diego

  2. The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token

    Oct 20, 2026Chenxu Niu, Batuhan Sencer, Wei Zhang +1Energy-Aware LLM InferenceLLM Inference

    NVIDIA Corporation, USA · Texas Tech University, USA · Texas Advanced Computing Center, USA

  3. LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization

    Aug 9, 2026Zexun Lin, Yuan Feng, Junlin Lv +2Inference-Time OptimizationSpeculative Decoding for Diffusion Language Models

    Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, Jiangsu, China