Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

Bole Ma; Ayesha Afzal; Jan Eitzinger; Harald Koestler; Gerhard Wellein

cs.DCOct 29, 2026

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

Authors: Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler, Gerhard Wellein

Organizations: Erlangen National High Performance Computing Center, Erlangen, Germany

Abstract

Large language model serving is limited by facility power. Energy per token determines how many tokens a datacenter can serve, while a latency target bounds how quickly each user receives them. During decode, every weight is read from memory for every generated token. Batching amortizes this read across users but makes no individual user faster. Speculative decoding does: a drafter proposes several future tokens, and the target model verifies them in a single forward pass that reuses the same weight read. Because every drafted token must be verified whether or not it is accepted, the energy speculation saves depends on how many drafted tokens are accepted. This saving is hard to pin down. It varies across target models from different attention families and across drafting schemes, and it depends on the text being served. Every drafter we evaluate accepts fewer tokens on conversation than on code, and every trained drafter accepts fewer on Romansh, a low-resource language, than on English. We measure speculative decoding under production serving conditions (SGLang, CUDA graphs enabled) on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. On dense models, verification, not drafting, dominates the energy cost. The saving shrinks as the batch grows, because batching has already shared the weight read that speculation would divide among the accepted tokens. At large batch sizes, block drafters push verification past the balance point between arithmetic and memory traffic into the compute-bound regime, where the saving becomes a loss, and a drafter with low acceptance raises energy per token. Meeting a tighter latency target with speculation then costs additional energy. Larger batches lower energy per token without speculation. Speculation adds to that saving only when acceptance is high, and on the model we tested it leaves little headroom for latency-preserving DVFS.

Figures & tables

family	model	GPU	n-gram	MTP head	EAGLE3	DFlash	DSpark
GQA	Qwen3-4B	H100	4	–	8 t	16 t , 7 t	7 t
	Qwen3-8B	H100	4	–	8 t	16 t , 7 t	7 t
	Qwen3-14B	H100	4	–	8 t	7 t	7 t
	Qwen3-Coder-30B-A3B-Instruct ∗	H100	4	–	–	16 t	–
GQA+SWA	gemma-4-12B-it	H100	4	6 v	–	16 t	7 t
	gemma-4-26B-A4B-it ∗	H100	4	6 v	–	16 t	–

Table 1. Drafters per model, given as the number of tokens each verifies per request and step and marked by who trained it (c with the model, v by the model’s vendor afterwards, as Google’s assistant for the gemma models, t by a third party), with the GPU each model’s batch sweep runs on, and asterisks marking Mixture-of-Experts models.

family	model	$B{=}1$	$B{=}2$	$B{=}4$	$B{=}8$	$B{=}16$	$B{=}32$	$B{=}64$
GDN $+$ GA	Qwen3.6-27B	+65 $\pm$ 5 DF	+61 $\pm$ 2 DF	+54 $\pm$ 1 DF	+48 $\pm$ 1 MTP	+45 $\pm$ 1 MTP	+38 $\pm$ 1 MTP	–
	Qwen3.6-35B-A3B	+38 $\pm$ 4 MTP	+31 $\pm$ 8 MTP	+26 $\pm$ 2 MTP	+25 $\pm$ 2 MTP	+31 $\pm$ 3 MTP	+31 $\pm$ 1 MTP	+25 $\pm$ 1 MTP
GQA	Qwen3-14B	+54 $\pm$ 1 DS	+53 $\pm$ 3 DS	+49 $\pm$ 2 DS	+41 $\pm$ 1 DS	+35 $\pm$ 1 DS	+34 $\pm$ 0 DS	+18 $\pm$ 1 DS
	Qwen3-4B	+56 $\pm$ 5 DS	+53 $\pm$ 4 DS	+48 $\pm$ 1 DS	+41 $\pm$ 1 DS	+29 $\pm$ 2 DS	+25 $\pm$ 1 DS	+19 $\pm$ 1 DS
	Qwen3-8B	+62 $\pm$ 3 DS	+55 $\pm$ 3 DS	+50 $\pm$ 2 DS	+41 $\pm$ 2 DS	+37 $\pm$ 1 DS	+34 $\pm$ 0 DS	+24 $\pm$ 1 DS
	Qwen3-Coder-30B-A3B-Instruct	+32 $\pm$ 10 DF	+35 $\pm$ 5 DF	+41 $\pm$ 10 DF	+48 $\pm$ 7 DF	+49 $\pm$ 3 DF	+51 $\pm$ 1 DF	+51 $\pm$ 2 DF

Table 2. Largest energy saving among the schemes (%) per model and batch with CUDA graph capture enabled, on the GPU given in Table 1 , as the median over repeated runs

\pm

half their range, the scheme that achieves it (DF DFlash, DS DSpark, MTP an MTP head or gemma’s assistant), and a dash where no speculative run held the batch.

		baseline (J/token)		speculative (J/token)		saving (%)
model	scheme	off	on	off	on	off	on
$B{=}1$
gemma-4-31B-it	DFlash	12.80	12.65	2.96	2.91	$+77$	$\mathbf{+77}$
gemma-4-12B-it	DFlash	6.33	4.92	1.50	1.23	$+76$	$\mathbf{+75}$
Qwen3.6-27B	DFlash	11.60	10.51	3.43	3.04	$+71$	$\mathbf{+71}$
Qwen3-8B	DSpark	3.79	3.10	1.60	1.23	$+57$	$\mathbf{+60}$

Table 3. Energy per output token of the non-speculative baseline and of the scheme with the largest saving with capture enabled, with CUDA graph capture off and on, and the saving each pair yields as the median of per-seed savings over three seeds with every speculative run paired to the baseline of the same seed and capture state, from a separate campaign run on the H100 except for JoyAI-LLM-Flash on the H200, so the Qwen3.6 rows differ from Table 2 .

Figure 2. Energy per output token against batch with CUDA graph capture enabled for 9 of the 11 models, grouped by attention family and labelled where the panel ran on the H200, with the non-speculative baseline in grey, asterisks marking Mixture-of-Experts models, crosses at batches the engine ran below the requested concurrency, and lines that stop short ending where the engine could not start.

Figure 3. Romansh-to-English acceptance ratio and energy saving at a batch of 1, one row per model and drafter labelled with the number of tokens it verifies per step, coloured by how the drafter was trained, and drawn light where the model does not answer consistently in Romansh.

Figure 4. Energy per token against per-user throughput with CUDA graph capture enabled, one panel per attention family, with each trace running from a batch of 1 (hollow marker) to a batch of 64 or to the batch given in its label, shaded over the range of its three seeds.

	baseline (ms)			fastest speculative scheme (ms)
model	$B{=}1$	$B{=}32$	$B{=}64$	$B{=}1$	$B{=}32$	$B{=}64$
GLM-4.7-Flash	8.2	23.4	28.0	6.2 MTP	18.2 MTP	21.5 MTP
JoyAI-LLM-Flash †	4.2	15.4	20.1	3.0 MTP	10.8 MTP	13.6 MTP
Qwen3-14B	15.4	16.9	17.9	7.3 DS	9.8 DS	14.7 DS
Qwen3-4B	5.7	6.8	7.7	2.7 DS	4.3 DS	6.0 DS
Qwen3-8B	9.4	10.2	11.3	3.8 DS	6.2 DS	8.7 DS

Table 4. Mean time between output tokens per user in milliseconds with CUDA graph capture enabled on the code pool, for the non-speculative baseline and for the speculative scheme with the shortest time at each batch, with a dash where the engine did not hold the batch and a dagger where prefill time is not subtracted.

Figure 5. Energy saving against accepted tokens per verification for every model, scheme and batch that has a baseline run at the same batch size, coloured by batch (a), and the share of decode energy spent drafting and verifying at

B{=}1

and

B{=}64

with capture enabled for every dense model and scheme, measured by removing the draft (b).

Figure 6. Arithmetic intensity of a decode step on Qwen3-4B against batch, with the balance point of each GPU as a horizontal line (a), and the energy saving of the same drafters with capture enabled on each GPU, filled where the step at that batch is past the balance point of that GPU (b), shaded in (a) from the batch at which the DFlash-16 step crosses each balance point and in (b) from the batch at which its saving turns negative on each GPU.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 314 papers

Jul 19, 2026

Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware

Single-stream autoregressive decoding of large language models is bound by memory bandwidth: each generated token requires one full forward pass through the target model, and successive passes cannot be parallelized. Speculative decoding restructures this computation: a small draft model proposes

K

tokens autoregressively, the target model scores all of them in one batched pass, and a rejection-sampling rule provably preserves the target model's output distribution. We present a from-scratch, device-agnostic (CUDA/MPS/CPU) implementation and an empirical study across five draft/target backend configurations on a consumer Apple-silicon laptop. Distribution equivalence is verified at three levels, culminating in a two-sample test over roughly 9,200 real-model tokens per method (

χ^2 = 162.5

, dof

= 200

p = 0.976

) and exact greedy-sequence agreement. The best configuration reaches a measured

1.61\times

wall-clock speedup at

K=6

, on an acceptance profile declining from 69.7% at

K=1

to 37.8% at the optimum, while three of five configurations decelerate, either because the draft fails to out-speed a small target or because the quantized Metal backend executes "parallel" verification serially, an effect we isolate and quantify. The failures are as instructive as the successes: speculative decoding pays off only when verification is genuinely batch-parallel and the draft/target latency gap is real.

Param Chordiya

University of California, San Diego

On-Device Language Model Inference Speculative Decoding LLM Inference Acceleration Parallel Decoding Parallel Speculative Decoding

Ask PDF

Oct 20, 2026

The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token

Energy-efficient deployment of large language models requires understanding how model architecture, hardware, and serving settings shape energy per generated token. We introduce the Token-Energy Law, a compact, interpretable model of GPU-board energy per output token for dense, full-history decoding. It relates token energy to two core dimensions: active bytes and realized decode time. Active bytes describe logical byte demand per output, separating dense weights shared within a decode iteration from private key--value (KV) histories read and updated for each request. Realized decode time is the elapsed decode duration divided by the number of output tokens produced. We combine the two byte coordinates and time in a common affine form, with coefficients calibrated for each model and hardware/runtime configuration. This formulation explains how batching amortizes shared-weight work, longer attended histories increase private KV demand, and throughput determines the time-associated cost per output. Our experiments cover three models under BF16 decoding on NVIDIA Hopper and Blackwell GPUs, including two-GPU tensor parallelism on Hopper. Using fixed coefficients and observed decode time, held-out condition-mean energy estimates differ from measurements by at most 3.73%.

Chenxu Niu, Batuhan Sencer, Wei Zhang +1

NVIDIA Corporation, USA · Texas Tech University, USA · Texas Advanced Computing Center, USA

Energy-Aware LLM Inference LLM Inference

Ask PDF

Aug 9, 2026

LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization

Speculative decoding accelerates large language model inference by drafting multiple tokens for parallel verification, with efficiency critically determined by the speculative length selected at each decoding round. Existing dynamic speculation methods select the speculation length by estimating how many tokens will be accepted, which is reasonable for autoregressive drafters that generates tokens sequentially. The recent wave of diffusion-based drafters, however, generates candidate blocks in parallel at substantially lower drafting cost, shifting the key question from how many tokens to generate to how many generated tokens are worth verifying. We therefore reformulate dynamic speculative-length selection as expected-speedup optimization and derive a marginal criterion that extends the speculative sequence only when its acceptance gain outweighs the additional verification cost. Building on this criterion, we develop LibraSpec, a training-free and plug-and-play algorithm that iteratively determines the speculative length using drafter confidence scores. Theoretically, we prove that LibraSpec monotonically converges toward the optimal speculative length. Experiments across six target models, three diffusion-based speculative decoding methods, and math, coding, and chat benchmarks show consistent improvements under both greedy and sampling settings, achieving a further

0.5\sim1.5\times

improvement over baselines and up to

8.49\times

speedup over autoregressive decoding.

Zexun Lin, Yuan Feng, Junlin Lv +2

Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, Jiangsu, China

Inference-Time Optimization Speculative Decoding for Diffusion Language Models Speculative Decoding LLM Inference Acceleration Diffusion Language Model Decoding

Ask PDF

Abstract

Speculative Decoding

Explore similar work

Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware

The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token

LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization