The SIMPLE Law for Energy Cost of LLM Inference

Hiari Pizzini Cavagna; Andrea Proia; Daniele Cesarini; Zeynep Kiziltan; Andrea Bartolini

cs.AINov 2, 2026

The SIMPLE Law for Energy Cost of LLM Inference

Authors: Hiari Pizzini Cavagna, Andrea Proia, Daniele Cesarini, Zeynep Kiziltan, Andrea Bartolini

Organizations: DEI - University of Bologna, Bologna, Italy · DISI - University of Bologna, Bologna, Italy · CINECA, Casalecchio di Reno, Italy

Abstract

Generative AI is billed per token, yet tokens are not equally expensive. Each token enters the KV-cache and raises the cost of those that follow, so energy grows non-linearly along a sequence whose tokens look identical. No analytical model states what a token costs, and the most accurate SOTA model is a fit, not a law: it absorbs architecture into fitted coefficients, requiring per-LLM fitting. We derive closed-form expressions of the computational and memory-access complexity of LLM inference in terms of architectural hyperparameters, covering MHA and GQA, and compare a family of analytical models of increasing granularity. From these, we obtain the SIMPLE Inference Model to Predict LLMs Energy (SIMPLE), a four-parameter law that separates prefill from decode, and decode into KV-cache cost and next-token computation. Over traces of 13 dense LLMs from six families on NVIDIA H100 GPUs, it attains R2 = 0.969 and 11.97% MAPE, 3.1-3.5x below the SOTA, without per-LLM calibration. The SIMPLE law explains as well as predicts: prefill energy grows quadratically with the hidden dimension, so larger LLMs suffer most from long inputs; and more aggressive GQA KV grouping reduces the per-token energy by up to 3x relative to standard MHA. Increasing the reasoning token budget pushes output length beyond the analytically derived energy minimum, driving super-linear energy growth; although accuracy grows with the budget, energy grows faster, so Intelligence per Joule peaks at an intermediate budget and falls by up to 3.14x across a 4.0x reasoning-budget range. Agentic serving raises energy per request by 13.0x relative to non-agentic ones, driven by the growth of both the prefill and the KV-cache terms with input length; while prefix caching removes most of the prefill cost (-93%), the KV-cache term is unaffected, so even a cached agentic request costs 4.8x more.

Figures & tables

Model	$E_{\text{tot}}$ Form	Analytical	LLM-agnostic	Attention-aware
Vellaisamy et al., 2026 [ 38 ]	Linear	$\times$	$\times$	$\times$
Zhan et al., 2026 [ 43 ]	Linear	$\times$	$\times/\checkmark$	$\times$
Delavande et al., 2026 [ 11 ]	Bilinear	$\checkmark$	$\times$	$\times$
Wilkins et al., 2025 [ 40 ]	Bilinear	$\times$	$\times$	$\times$
Oviedo et al., 2026 [ 24 ]	Power-law	$\times$	$\times$	$\times$
SweetSpot, 2026 [ 26 ]	Quadratic	$\checkmark$	$\times$	$\times$

Table 1: Comparison of existing energy models for LLM inference.

Symbol	Dimension	Description	Symbol	Dimension	Description
Scalars			Prefill Phase
$\tau_{\text{in}}$	$\mathbb{N}^{1\times 1}$	Number of input tokens	$\mathbf{X}_{P}$	$\mathbb{R}^{\tau_{\text{in}}\times d}$	Input
$\tau_{\text{out}}$	$\mathbb{N}^{1\times 1}$	Number of output tokens	$\mathbf{Q}_{P},\mathbf{O}_{P}$	$\mathbb{R}^{\tau_{\text{in}}\times d}$	Query, Output projections
$n_{\text{q}}$	$\mathbb{N}^{1\times 1}$	Number of query heads	$\mathbf{K}_{P},\mathbf{V}_{P}$	$\mathbb{R}^{\tau_{\text{in}}\times d_{\text{kv}}}$	Key, Value projections
$n_{\text{kv}}$	$\mathbb{N}^{1\times 1}$	Number of KV heads	$\mathbf{S}_{P}$	$\mathbb{R}^{\tau_{\text{in}}\times\tau_{\text{in}}}$	Attention scores
$L$	$\mathbb{N}^{1\times 1}$	Number of layers	$\mathbf{A}_{P}$	$\mathbb{R}^{\tau_{\text{in}}\times d}$	Attention output

Table 2: Decoder-only LLM inference symbols and tensor dimensions.

Approach	Model	# coeff.	Energy per token, $E_{\text{tok}}$
OPs-centric	Vartziotis et al. [ 36 ]	6	$\dfrac{\alpha_{\text{TC}}C_{\text{TC}}}{\tau_{\text{out}}}+\dfrac{e_{\text{HBM}}\,B\,\eta(N)}{\tau_{\text{out}}}$
	FLOPs-MemOPs	3	$\theta_{0}+\theta_{1}\dfrac{F_{\text{prefill}}+F_{\text{decode}}}{\tau_{\text{out}}}+\theta_{2}\dfrac{M_{\text{prefill}}+M_{\text{decode}}}{\tau_{\text{out}}}$
	Prefill-Decode	5	$\theta_{0}+\theta_{1}\dfrac{F_{\text{prefill}}}{\tau_{\text{out}}}+\theta_{2}\dfrac{F_{\text{decode}}}{\tau_{\text{out}}}+\theta_{3}\dfrac{M_{\text{prefill}}}{\tau_{\text{out}}}+\theta_{4}\dfrac{M_{\text{decode}}}{\tau_{\text{out}}}$
	Fine-Grained	25	$\theta_{0}+\sum_{i=1}^{24}\theta_{i}\dfrac{L\,C_{i}}{\tau_{\text{out}}}\ \hskip 17.00024ptC_{i}\in\left\{F_{Q_{P}},\ldots,\sum_{t=1}^{\tau_{\text{out}}}M_{A_{D}}(t)\right\}$
IO-centric	Zhan et al. [ 43 ]	5	$\theta_{0}+\theta_{1}S\dfrac{1}{\tau_{\text{out}}}+\theta_{2}\dfrac{\tau_{\text{in}}}{\tau_{\text{out}}}+\theta_{3}S\dfrac{\tau_{\text{in}}}{\tau_{\text{out}}}+\theta_{4}S$
	SweetSpot [ 26 ]	6	$\theta_{0}+\theta_{1}\dfrac{\tau_{\text{in}}^{2}}{\tau_{\text{out}}}+\theta_{2}\tau_{\text{in}}+\theta_{3}\dfrac{\tau_{\text{in}}}{\tau_{\text{out}}}+\theta_{4}\tau_{\text{out}}+\dfrac{\theta_{5}}{\tau_{\text{out}}}$

Table 3: Summary of the proposed energy models and selected baselines for predicting the energy per generated token,

E_{\text{tok}}

. SweetSpot is included for direct comparison. For the LLM-agnostic SweetSpot model,

\phi_{0},\ldots,\phi_{5}

denote architecture-dependent analytical coefficients, whose explicit expressions are given in Equation 20 .

Name	Size (B)	Attention	$\mathbf{n_{\text{q}}}$	$\mathbf{R}$	$\mathbf{d}$	$\mathbf{L}$
Llama 3.2	1	GQA	32	1/4	2048	16
OPT	1.3	MHA	32	1	2048	24
Qwen 2	1.5	GQA	12	1/6	1536	28
Gemma 2	2	GQA	8	1/2	2304	26
OPT	2.7	MHA	32	1	2560	32
Llama 3.2	3	GQA	24	1/3	3072	28

Table 4: LLM test set grouped into size-based families (1B–9B parameters), with architectural hyperparameters for each model.

Figure 1: Average energy efficiency across the evaluated LLMs, highlighting the adopted attention architecture.

Figure 2: Aggregated energy efficiency across the dataset as a function of input and output sequence length.

Figure 3: Graphical comparison of the results in Table 5 . Error bars denote standard deviation across folds.

Model	$E_{\text{tot}}$ Form	$E_{\text{tok}}$ Formulation	$\mathbf{R^{2}}$	MAPE (%)
Vellaisamy et al. [ 38 ] Zhan et al. [ 43 ]	Linear	$\theta_{0}+\theta_{1}\dfrac{\tau_{\text{in}}}{\tau_{\text{out}}}$	$0.604$ $(\pm 0.339)$	$38.199$ $(\pm 10.729)$
Delavande et al. [ 11 ] Wilkins et al. [ 40 ]	Bilinear	$\theta_{0}+\theta_{1}\dfrac{\tau_{\text{in}}}{\tau_{\text{out}}}+\theta_{2}\,\tau_{\text{in}}$	$0.886$ $(\pm 0.065)$	$20.477$ $(\pm 4.890)$
Oviedo et al. [ 24 ]	Power-law	$\theta_{0}\,\tau_{\text{in}}^{\theta_{1}}\,\tau_{\text{out}}^{\theta_{2}}$	$0.723$ $(\pm 0.099)$	$63.234$ $(\pm 24.888)$
SweetSpot [ 26 ]	Quadratic	$\theta_{0}+\theta_{1}\dfrac{\tau_{\text{in}}^{2}}{\tau_{\text{out}}}+\theta_{2}\tau_{\text{in}}+\theta_{3}\dfrac{\tau_{\text{in}}}{\tau_{\text{out}}}+\theta_{4}\tau_{\text{out}}+\dfrac{\theta_{5}}{\tau_{\text{out}}}$	$1.000$ $(\pm 0.000)$	$1.841$ $(\pm 0.692)$
SIMPLE	Quadratic	$\theta_{0}+\theta_{1}\,d^{2}L\,\dfrac{\tau_{\text{in}}}{\tau_{\text{out}}}+\theta_{2}\,d\,L\,R\Big(\tau_{\text{in}}+\dfrac{\tau_{\text{out}}}{2}\Big)+\theta_{3}d\,L\$	$\mathbf{0.969}$	$\mathbf{11.974}$

Table 6: Comparison of

R^{2}

and MAPE between SIMPLE and LLM-specific baseline models, all fitted with WLS (

w=1/y^{2}

) on

E_{\text{tok}}

. SIMPLE is fitted once across all LLMs. As LLM-specific models are fitted separately for each LLM, we report their mean and standard deviation across LLMs.

Figure 4: Term-wise decomposition of the predicted

E_{\text{tok}}

under the SIMPLE law, varying hidden dimension

d

(a), number of layers

L

(b), and KV head ratio

R

(c), using Llama 3.1 8B architectural hyperparameters, with

\tau_{\text{in}}=\tau_{\text{out}}=512

. Each band represents the additive contribution of one term of the SIMPLE law.

Figure 5: Term-wise decomposition of the predicted

E_{\text{tok}}

under the SIMPLE law, using Llama 3.2 1B, Qwen2 1.5B, OPT 1.3B, and Llama 3.1 8B, with

\tau_{\text{in}}=\tau_{\text{out}}=512

Panel of Figure 6 — Figure 6: Estimated $E_{\text{tok}}$ contour maps generated by the SIMPLE law of Qwen3-1.7B (a) and DeepSeek-R1-Distill-Llama-8B (b).

Figure 7:

E_{\text{tok}}

additive components of DeepSeek-R1-Distill-Llama-8B predicted by the SIMPLE law, with

\tau_{\text{in}}=64

Reasoning	Acc.	$\boldsymbol{E_{\mathrm{tok}}}$	$\boldsymbol{E_{\mathrm{tot}}}$	IPJ
[tok]	[%]	[mJ/tok]	[J]	[%/kJ]
Qwen3-1.7B
2048 ( $1.0\times$ )	3.0 ( $1.00\times$ )	43.6 ( $1.00\times$ )	89.3 ( $1.00\times$ )	33.7 ( $1.00\times$ )
4063 ( $2.0\times$ )	14.0 ( $4.67\times$ )	73.6 ( $1.69\times$ )	298.9 ( $3.35\times$ )	46.8 ( $1.39\times$ )
6978 ( $3.4\times$ )	27.0 ( $9.00\times$ )	117.0 ( $2.68\times$ )	816.5 ( $9.14\times$ )	33.1 ( $0.98\times$ )
9510 ( $4.6\times$ )	38.0 ( $12.67\times$ )	154.8 ( $3.55\times$ )	1471.9 ( $16.48\times$ )	25.8 ( $0.77\times$ )

Table 7: Accuracy on AIME-24 dataset from [ 30 ] and SIMPLE estimation of

E_{\mathrm{tok}}

and

E_{\mathrm{tot}}

for Qwen3-1.7B and DeepSeek-R1-Distill-Llama-8B as a function of reasoning token budgets. Ratios (in parentheses) are relative to the minimum budget, corresponding to

\tau_{\mathrm{out}}=2048

. Intelligence per Joule (IPJ) is computed as accuracy (%) divided by

E_{\mathrm{tot}}

(J). For both models, the mean

\tau_{\mathrm{in}}

of the dataset is used (

\tau_{\text{in}}=105

Figure 8: Graphical comparison of the accuracy (a),

E_{\text{tot}}

(b) and Intelligence per Joule (c) results reported in Table 7 .

Term [J]	Non-Agentic $\tau_{\mathrm{in}}{=}5$ K, $\tau_{\mathrm{c}}{=}0$ ,	Agentic $\tau_{\mathrm{in}}{=}68$ K, $\tau_{\mathrm{c}}{=}0$	Agentic Cached $\tau_{\mathrm{in}}{=}68$ K, $\tau_{\mathrm{c}}{=}63$ K
Prefill	88.8 ( $1.0\times$ )	1207.8 ( $13.6\times$ )	88.8 ( $1.0\times$ )
KV-cache	43.2 ( $1.0\times$ )	573.8 ( $13.3\times$ )	573.8 ( $13.3\times$ )
Next-token	6.1 ( $1.0\times$ )	6.1 ( $1.0\times$ )	6.1 ( $1.0\times$ )
$E_{\mathrm{tok}}$ [J/tok]	0.56 ( $1.0\times$ )	7.24 ( $12.9\times$ )	2.71 ( $4.8\times$ )
Total $\boldsymbol{E_{\mathrm{tot}}}$	138.1 ( $\mathbf{1.0\times}$ )	1787.7 ( $\mathbf{13.0\times}$ )	668.7 ( $\mathbf{4.8\times}$ )
$\boldsymbol{\Delta E_{\mathrm{tot}}}$ (+1 tok)	0.20 ( $\mathbf{1.0\times}$ )	2.35 ( $\mathbf{11.8\times}$ )	2.35 ( $\mathbf{11.8\times}$ )

Table 8: SIMPLE law estimation of the per-request energy, decomposed by contributions, for Llama-3.1-8B on the median GitHub Copilot coding-agent request from [ 18 ] (

\tau_{\mathrm{out}}=247

). In parentheses, the ratio w.r.t. the non-agentic case.

\Delta E_{\mathrm{tot}}

is the energy cost of one additional token (

\tau_{\mathrm{out}}=248

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Coeff.	Estimate	Std. Err.	$t$	$P{>}\|t\|$
SIMPLE
$\theta_{0}$	$-1.866\times 10^{-3}$	$0.000$	$-4.412$	$<0.001$
$\theta_{1}$	$3.308\times 10^{-11}$	$6.80\times 10^{-13}$	$48.631$	$<0.001$
$\theta_{2}$	$1.041\times 10^{-9}$	$1.24\times 10^{-11}$	$84.004$	$<0.001$
$\theta_{3}$	$2.020\times 10^{-7}$	$8.01\times 10^{-9}$	$25.208$	$<0.001$
FLOPs-MemOPs

Appendix

Table 9: Fitted coefficients of the SIMPLE Law (

R^{2}=0.969

, MAPE

=11.974\%

) and of the OPs-centric FLOPs-MemOPs model (

R^{2}=0.581

, MAPE

=39.384\%

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 304 papers

Aug 28, 2026

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete, since average output-token energy can decrease even when total request energy increases. We characterize this behavior with a decomposed energy model: a fixed one-time prefill with a fixed generation setup cost, while each output-token generation step adds marginal step energy. We evaluate this LLM inference energy model on NVIDIA H100 and H200 GPUs across dense and mixture-of-experts (MoE) models, reporting both request energy and token energy as functions of model type (M), phase (P), batch size (B), context length (C), and output length (N). For Llama-3.2-1B on H200 at batch-16 and context-4K, increasing output length from 10 to 512 tokens reduces token energy from 7.46 to 0.72 J/token while total batched inference-window energy increases from 1.19 to 5.93 kJ. Batching also reduces token energy, but the gain is context-bounded: at 10 output tokens, the batch-16 to batch-1 gain falls from 6.31x at context-512 to 1.17x at context-4K. MoE models amplify this effect: sparse routing and fragmented expert execution increase fixed energy at low concurrency, while batching spreads that energy across more generated tokens and substantially narrows the dense-vs.-MoE token-energy gap. These results show that energy-aware serving should jointly optimize both request energy and token energy, rather than only reducing per-token energy cost.

Prabhu Vellaisamy, Vanessa Lam, Shawn Blanton +1

Electrical and Computer Engineering Department Carnegie Mellon University Pittsburgh, PA, USA

Energy-Aware LLM Inference LLM Inference LLM Inference Efficiency LLM Serving Efficient Language Model Inference

Ask PDF

Jul 29, 2026

From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs

The operational energy consumption of large language model (LLM) inference is becoming an increasingly important component of the environmental footprint of deployed AI systems. However, direct measurement of inference energy often requires hardware telemetry, power instrumentation, or infrastructure-specific monitoring, limiting its applicability in comparative studies, early-stage system design, and sustainability reporting. This report presents an analytically structured, empirically calibrated, GPU-level methodology for estimating LLM inference energy on NVIDIA H100-class accelerators without direct runtime measurement. The proposed estimator combines parameter-scaled transformer FLOP accounting, calibrated memory-traffic factors, and hardware-specific energy coefficients for FP16/BF16 tensor-core computation and high-bandwidth-memory movement. It explicitly separates prompt prefill from autoregressive decoding, enabling energy estimates for input tokens, output tokens, and complete inference requests. The methodology further decomposes total energy into compute, parameter-access, key-value-cache write, and attention-read components, allowing the scaling behavior with model size, context length, and generated-token count to be analyzed. The resulting estimates are not intended to replace physical power measurements; rather, they provide transparent, reproducible, and assumption-explicit approximations suitable for model comparison, green-coding analysis, and design-time evaluation of LLM inference workloads.

Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis +5

1National Technical University of Athens, Patission Complex 42, 10682 Athens, Greece · 2Harvard University, 1350 Massachusetts Avenue, 02138 Cambridge, MA, USA · 3TWT GmbH Science & Innovation, Industriestraße 6, 70565 Stuttgart, DE +4

LLM Inference Efficiency Energy-Efficient ML LLM Inference Energy-Aware LLM Inference

Ask PDF

Mar 16, 2026

This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMs

The energy consumption of Large Language Models (LLMs) is raising growing concerns due to their adverse effects on environmental stability and resource use. Yet, these energy costs remain largely opaque to users, especially when models are accessed through an API -- a black box in which all information depends on what providers choose to disclose. In this work, we investigate inference time measurements as a proxy to approximate the associated energy costs of API-based LLMs. We ground our approach by comparing our estimations with actual energy measurements from locally hosted equivalents. Our results show that time measurements allow us to infer GPU models for API-based LLMs, grounding our energy cost estimations. Our work aims to create means for understanding the associated energy costs of API-based LLMs, especially for end users.

Lars Krupp, Daniel Geißler, Francisco M. Calatrava-Nicolas +3

Embedded Intelligence DFKI and RPTU Kaiserslautern, Germany · Centre for Applied Autonomous Sensor Systems (AASS) Örebro University Örebro, Sweden

Energy-Aware LLM Inference LLM Inference Energy-Efficient ML

Ask PDF

Abstract

LLM Inference

Explore similar work

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs

This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMs