The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token

Chenxu Niu; Batuhan Sencer; Wei Zhang; Yong Chen

cs.LGOct 20, 2026

The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token

Authors: Chenxu Niu, Batuhan Sencer, Wei Zhang, Yong Chen

Organizations: NVIDIA Corporation, USA · Texas Tech University, USA · Texas Advanced Computing Center, USA

Abstract

Energy-efficient deployment of large language models requires understanding how model architecture, hardware, and serving settings shape energy per generated token. We introduce the Token-Energy Law, a compact, interpretable model of GPU-board energy per output token for dense, full-history decoding. It relates token energy to two core dimensions: active bytes and realized decode time. Active bytes describe logical byte demand per output, separating dense weights shared within a decode iteration from private key--value (KV) histories read and updated for each request. Realized decode time is the elapsed decode duration divided by the number of output tokens produced. We combine the two byte coordinates and time in a common affine form, with coefficients calibrated for each model and hardware/runtime configuration. This formulation explains how batching amortizes shared-weight work, longer attended histories increase private KV demand, and throughput determines the time-associated cost per output. Our experiments cover three models under BF16 decoding on NVIDIA Hopper and Blackwell GPUs, including two-GPU tensor parallelism on Hopper. Using fixed coefficients and observed decode time, held-out condition-mean energy estimates differ from measurements by at most 3.73%.

Figures & tables

Perspective and examples	Main question	Connection to this paper
Configuration measurement ( Samsi et al., 2023 ; Niu et al., 2026 ; Chung et al., 2025 )	Effects of model, workload, and serving choices on energy.	Execution variables that explain configuration-dependent trends.
Analytical work–energy models ( Choi et al., 2013 ; Pizzini Cavagna et al., 2026 ; Vartziotis et al., 2026 )	Energy costs of computation, data movement, and time.	Work–time foundation for shared-weight and KV accounting.
Deployment and operator predictors ( Palladino et al., 2026 ; Song et al., 2026 )	Energy prediction from configurations or operators.	Complements modeling based on realized trace quantities.
Request-window amortization ( Vellaisamy et al., 2026 )	Fixed and incremental energy within a request.	Explicit phase boundaries and useful-output normalization.
This work (Token-Energy Law)	Decode energy from shared weights, private KV history, and realized time.	Trace-derived work coordinates, independent HBM measurements, and frozen-coefficient validation under static and changing request membership.

Table 1. Comparison of inference-energy measurement and modeling approaches.

Figure 1. A common decode measurement interval for a two-GPU replica. Shaded areas under schematic power traces represent

E_{0}

and

E_{1}

over duration

D

;

T

counts useful outputs once for the replica. Loading, prefill, and bootstrap precede metered decode. Two schematic GPU power traces share the same start and synchronization-completed end. Shaded areas represent each board's energy. A single row of output marks represents the replica-wide useful-output count. The right panel shows summed board energy and elapsed duration, each divided by that count.

Panel of Figure 2 — Figure 2. Batch amortization of active bytes. (a,b) Weight demand is shared across outputs; KV histories remain private. (c) Analytical weight-plus-KV demand for Qwen2.5-7B (BF16). Circles mark equal weight and KV byte contributions. Panels a and b compare weight sharing and private KV histories for batches of one and three. Panel c shows logical demand versus batch size for three history lengths.

Panel of Figure 3 — Figure 3. KV active bytes across three decode steps: growing history reads and one input-token KV write per useful output. Three steps read four, five, and six historical records and each writes one input-token KV record. Normalizing by three useful outputs gives $\Kkv(\Lbar+1)$ .

Figure 4. Conditional energy response to decode time at fixed active bytes, output count, and execution stratum (

P_{\tau,\sigma}>0

). Three panels show equal logical work, different durations per output, and their conditional energy predictions. The predicted difference equals the time coefficient multiplied by the duration-per-output difference.

Configuration	Fitting / development	Residual calibration	Held-out evaluation
GH200, Q7	$B\in\{4,8,16,24,32\}$ $H\in\{4,8,16\}$ 75 runs	No independent set	$B\in\{12,20,28\}$ $H\in\{6,10,14\}$ 45 runs
GH200 / GB300 Q14 and M7	$B\in\{4,16\}$ $H\in\{4,10,16\}$ 30 runs each	$B=8$ $H\in\{4,10,16\}$ 15 runs each	$B\in\{6,12\}$ $H\in\{6,12,14\}$ 30 runs each
GH200, Q7 TP=2	$B\in\{4,32\}$ $H\in\{4,10,16\}$ 30 runs	$B=16$ $H\in\{4,10,16\}$ 15 runs	$B\in\{8,24\}$ $H\in\{6,12,14\}$ 30 runs
GB300, Q7	$B\in\{4,32\}$ $H\in\{4,8,16\}$ 30 runs	$B=16$ $H\in\{4,8,16\}$ 15 runs	$B\in\{12,28\}$ $H\in\{6,10,14\}$ 30 runs

Table 2. Workload grids for identification and held-out evaluation.

Figure 5. Qwen2.5-7B HBM reads on GH200. Top: logical weight/KV components, their unscaled sum, and measured means. Bottom: 60 fit residuals,

10^{4}(Q-\widehat{Q})/\widehat{Q}

; diamonds mark five-run means. Whiskers show pointwise 95% Student-

t

confidence intervals. Logical weight and KV demand compared with measured HBM reads, with fit residuals and confidence intervals for cell means.

Model	$\rho_{\mathrm{w}}$	$\rho_{\mathrm{KV}}$	MAPE	Max. APE	Dense-sum gap
Qwen2.5-7B	1.001189	1.002124	0.0460%	0.1757%	0.1714%
Qwen2.5-14B	1.001661	1.000586	0.0324%	0.0822%	0.1494%
Mistral-7B	1.001828	1.000494	0.0304%	0.0787%	0.1490%

Table 3. HBM-read fits on GH200 for

B\in\{4,8,16\}

and

\bar{L}\in\{4096,8192,16384\}

(45 profiles per model). Dense-sum gap is the mean absolute relative deviation of measured reads from logical weight-plus-KV reads, normalized by logical reads.

Metric	GH200	GB300
Profiles	45	27
$\rho_{\mathrm{w}}$	1.001003	1.000784
$\rho_{\mathrm{KV}}$	1.002258	1.004953
MAPE	0.0444%	0.0730%
Max. APE	0.1172%	0.2440%

Table 4. Qwen2.5-7B HBM-read fits on matched grids.

Figure 6. Qwen2.5-7B energy prediction on GH200 GPU0 and unchanged-coefficient transfer to GPU1. Stacks show model terms; diamonds and whiskers show measured means and 95% confidence intervals. Residuals: $100(e-\widehat{e})/e$ . Predicted and measured Qwen2.5-7B energy on GH200 GPU0 and GPU1, with model contributions, confidence intervals for measured means, and signed relative residuals.

Model	Bytes + time	Bytes only	Time only	Total bytes + time
Qwen2.5-7B	2.137%	35.458%	29.395%	5.721%
Qwen2.5-14B	0.971%	28.228%	10.972%	4.381%
Mistral-7B	0.887%	33.564%	14.117%	5.589%

Table 5. Maximum cell-mean absolute percentage error on GB300 holdouts (six cells and 30 runs per model).

Figure 7. Frozen-coefficient predictions under changing request membership on GH200. Stacks show model terms; dots and diamonds denote measured runs and condition means. Model contributions and measured decode energy for Qwen2.5-7B, Mistral-7B, and additional Qwen repetitions under changing request membership.

Coordinates	Q7	M7
Realized	0.404	0.870
$B_{0}$ replaces $B_{\mathrm{eff}}$	9.824	8.258
$\bar{L}_{\mathrm{req}}$ replaces $\bar{L}_{\mathrm{token}}$	3.578	6.395

Table 6. Maximum condition-mean APE (%) after coordinate substitution.

Figure 8. Qwen2.5-7B decoding on two GH200 boards. (a) Measured per-board energy and frozen total-energy predictions. (b) Realized decode time per useful output. (c) Signed errors of condition means. Dots show five runs per condition; whiskers are pointwise 95% Student-

t

confidence intervals for measured means. Measured per-board energy and frozen summed-board predictions for two-GPU decode, alongside realized time per useful output and signed condition-mean errors.

Platform	Model	$\operatorname{SE}(c)$	$\operatorname{SE}(\alpha)$	$\operatorname{SE}(\beta)$	$\operatorname{SE}(P_{\tau})$
		J/token	J/decimal-GB	J/decimal-GB	W
GH200	Q7	0.000861	0.000861	0.001947	0.820
GH200	Q14	0.003848	0.000939	0.001830	0.986
GH200	M7	0.002479	0.000876	0.001865	0.564
GB300	Q7	0.001024	0.000431	0.002242	0.570
GB300	Q14	0.002096	0.000359	0.001181	0.369

Table 7. HC3 standard errors of the fitted coefficients.

Platform	Model	Coefficients	Cells	Runs	Median	Maximum
GH200	Qwen2.5-7B	Local fit	9	45	0.409%	2.22%
GH200	Qwen2.5-14B	Local fit	6	30	0.130%	1.92%
GH200	Mistral-7B	Local fit	6	30	0.390%	1.43%
GH200 TP=2	Qwen2.5-7B	Local fit	6	30	0.245%	0.559%
GB300	Qwen2.5-7B	Local fit	6	30	0.637%	2.137%
GB300	Qwen2.5-14B	Local fit	6	30	0.172%	0.971%

Table 8. Static-workload energy validation. Errors compare predicted and measured cell means; each cell contains five executions. The device-transfer test retains the source-device coefficients.

Figure 9. Held-out energy across models and platforms. Lines connect predictions using observed decode time; markers show measured means. Shading denotes

\pm 1

sample standard deviation across five runs per condition. Two panels compare Token-Energy Law estimates with measured GPU-board energy per useful output on GH200 and GB300. The figure contains 195 executions at 39 conditions. Larger tested batches have lower measured mean energy at matched model, platform, and history settings. Mean energy increases with history in every plotted series. Shading represents repeated-measurement variability.

Energy-Aware LLM Inference

See all 31 papers

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets
Oct 29, 2026Bole Ma, Ayesha Afzal, Jan Eitzinger +2Speculative Decoding Energy-Aware LLM Inference
Erlangen National High Performance Computing Center, Erlangen, Germany
Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms
Aug 28, 2026Prabhu Vellaisamy, Vanessa Lam, Shawn Blanton +1Energy-Aware LLM Inference LLM Inference
Electrical and Computer Engineering Department Carnegie Mellon University Pittsburgh, PA, USA
Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures
Aug 25, 2026Molka Chkir, Syed Muhammad Danish, Jos Höll +1Energy-Aware LLM Inference LLM Inference Efficiency
Algoma University, Brampton, Canada · School of Informatics, Reutlingen University, Reutlingen

Oct 29, 2026J/K move · Enter open · S save

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler+1

Erlangen National High Performance Computing Center, Erlangen, Germany

Speculative Decoding Energy-Aware LLM Inference LLM Inference Acceleration

Open paper AskPDF arXiv

Large language model serving is limited by facility power. Energy per token determines how many tokens a datacenter can serve, while a latency target bounds how quickly each user receives them. During decode, every weight is read from memory for every generated token. Batching amortizes this read across users but makes no individual user faster. Speculative decoding does: a drafter proposes several future tokens, and the target model verifies them in a single forward pass that reuses the same weight read. Because every drafted token must be verified whether or not it is accepted, the energy speculation saves depends on how many drafted tokens are accepted. This saving is hard to pin down. It varies across target models from different attention families and across drafting schemes, and it depends on the text being served. Every drafter we evaluate accepts fewer tokens on conversation than on code, and every trained drafter accepts fewer on Romansh, a low-resource language, than on English. We measure speculative decoding under production serving conditions (SGLang, CUDA graphs enabled) on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. On dense models, verification, not drafting, dominates the energy cost. The saving shrinks as the batch grows, because batching has already shared the weight read that speculation would divide among the accepted tokens. At large batch sizes, block drafters push verification past the balance point between arithmetic and memory traffic into the compute-bound regime, where the saving becomes a loss, and a drafter with low acceptance raises energy per token. Meeting a tighter latency target with speculation then costs additional energy. Larger batches lower energy per token without speculation. Speculation adds to that saving only when acceptance is high, and on the model we tested it leaves little headroom for latency-preserving DVFS.

Abstract

Energy-Aware LLM Inference

Explore similar work

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures