Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

Prabhu Vellaisamy; Vanessa Lam; Shawn Blanton; John Paul Shen

cs.PFAug 28, 2026

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

Authors: Prabhu Vellaisamy, Vanessa Lam, Shawn Blanton, John Paul Shen

Organizations: Electrical and Computer Engineering Department Carnegie Mellon University Pittsburgh, PA, USA

Abstract

Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete, since average output-token energy can decrease even when total request energy increases. We characterize this behavior with a decomposed energy model: a fixed one-time prefill with a fixed generation setup cost, while each output-token generation step adds marginal step energy. We evaluate this LLM inference energy model on NVIDIA H100 and H200 GPUs across dense and mixture-of-experts (MoE) models, reporting both request energy and token energy as functions of model type (M), phase (P), batch size (B), context length (C), and output length (N). For Llama-3.2-1B on H200 at batch-16 and context-4K, increasing output length from 10 to 512 tokens reduces token energy from 7.46 to 0.72 J/token while total batched inference-window energy increases from 1.19 to 5.93 kJ. Batching also reduces token energy, but the gain is context-bounded: at 10 output tokens, the batch-16 to batch-1 gain falls from 6.31x at context-512 to 1.17x at context-4K. MoE models amplify this effect: sparse routing and fragmented expert execution increase fixed energy at low concurrency, while batching spreads that energy across more generated tokens and substantially narrows the dense-vs.-MoE token-energy gap. These results show that energy-aware serving should jointly optimize both request energy and token energy, rather than only reducing per-token energy cost.

Figures & tables

Fig. 1: Request-shape energy abstraction for LLM inference. Prior work reports aggregate energy, power, J/token, tokens/J, serving performance, energy models, or host-side overheads. This work focuses on when lower J/token reflects true energy reduction versus spreading fixed energy across more tokens. We characterize each request shape with five parameters, $(M,P,B,C,N)$ , denoting model, phase, batch size, context length, and output length. Request energy decomposes into prefill and decode energy. Decode energy contains fixed energy paid once for the decode window and step energy added by each decode step. Our proposed energy abstraction explains why batching or longer outputs can reduce token energy while increasing total request energy. Both token energy and request energy must be considered in overall system optimizations.

Fig. 2: Energy abstraction and attribution model: request shape-based energy decomposition by separating absolute request energy from token-normalized request energy.

Work / Group	Focus	Phase	Batch	Ctx.	Out.	HW	MoE	Req./Token	Window
Energy benchmarks [ 13 , 10 , 12 ]	Power/energy measurement and token-normalized inference metrics.	✓	✓	✓	✓	✓	✓	$\times$	$\times$
Serving and optimization [ 11 , 19 ]	Energy impact of frameworks, batching, decoding, hardware, and system control.	✓	✓	✓	✓	✓	✓	$\times$	$\times$
Input/output models [ 17 , 14 , 18 ]	Energy models and operating curves over input/output token counts.	✓	$\times$	✓	✓	✓	✓	$\times$	$\times$
Host-side overhead [ 7 , 8 , 6 ]	Kernel-launch and dispatch decomposition, execution-idle high-power states.	✓	✓	✓	$\times$	✓	$\times$	$\times$	✓
This Work	Abstraction of LLM inference energy costs by differentiating inference request energy and average token energy, and shows how schedulers should account for both for efficient inference service.	✓	✓	✓	✓	✓	✓	✓	✓

TABLE I: Comparison of this work against prior LLM inference energy studies.

Feature	H100 SXM	H200 NVL
Compute generation	Hopper (GH100)	Hopper (GH100)
Peak BF16/FP16 Tensor	1,979 TFLOPS	1,671 TFLOPS
GPU memory	80 GB HBM3	141 GB HBM3e
Memory bandwidth	3.35 TB/s	4.8 TB/s
Maximum configurable TDP	700 W	600 W
Form factor	SXM	PCIe, dual-slot air-cooled

TABLE II: Differences between the evaluated Hopper GPU platforms. Peak Tensor throughput assumes sparsity. Product specifications are from NVIDIA [ 30 , 31 ] . Prior work shows that energy efficiency depends on both GPU architecture and AI workload under power limits [ 32 ] , while memory bandwidth and DVFS can materially affect that relationship [ 33 ] .

Fig. 3: Request energy scaling (mJ/token) across varying output generation lengths. The evaluation contrasts a low-load scenario (left) against a high-load scenario (right) on a single NVIDIA H200 GPU. The y-axis is log-scaled.

Fig. 4: Inference latency scaling (ms) across varying output generation lengths. The evaluation contrasts a low-load scenario (left) against a high-load scenario (right) on a single NVIDIA H200 GPU. The y-axis is log scaled.

Model	$B$	$C$	$E_{\mathrm{fixed}}$	$E_{\mathrm{step}}/B$	Residual (%)
			(J)	(mJ/token)	$N{=}10$	$N{=}128$	$N{=}512$
Llama-3.2-1B	1	512	12.11	2163.56	+38.79	-4.09	+0.26
Llama-3.2-1B	16	2048	276.78	351.06	-5.34	+2.53	-0.18
Llama-3.2-3B	16	512	30.63	437.17	-28.34	+5.95	-0.34
Llama-3.1-8B	16	512	83.73	604.84	-27.81	+7.38	-0.42

TABLE III: Representative H200 affine fits over

N\in\{10,128,512\}

. Residuals are

(E_{\mathrm{fit}}-E_{\mathrm{measured}})/E_{\mathrm{measured}}

Model	$C{=}512$	$C{=}1024$	$C{=}2048$	$C{=}4096$
Llama-3.2-1B	6.31 $\times$	3.19 $\times$	1.85 $\times$	1.17 $\times$
Llama-3.2-3B	4.67 $\times$	2.66 $\times$	1.70 $\times$	1.22 $\times$
Llama-3.1-8B	3.85 $\times$	2.15 $\times$	1.47 $\times$	OOM
OLMoE-1B-7B	12.59 $\times$	11.88 $\times$	6.41 $\times$	3.93 $\times$
Qwen1.5-MoE-A2.7B	6.60 $\times$	4.75 $\times$	3.18 $\times$	1.90 $\times$

TABLE IV: Batching gain for complete H200 requests at

N{=}10

, reported as

E_{\mathrm{token}}(B{=}1)/E_{\mathrm{token}}(B{=}16)

$N$	Llama-3.2-3B	Llama-3.1-8B
128	337 mJ/token	548 mJ/token
512	372 mJ/token	582 mJ/token
1024	455 mJ/token	691 mJ/token
4096	945 mJ/token	1,394 mJ/token
8192	1,600 mJ/token	2,358 mJ/token

TABLE V: Long-output request energy per output token on H200 at

B{=}16

C{=}512

. Values are

E_{\mathrm{request(s)}}(N)/(B\times N)

Fig. 5: Request energy per output token and power scaling on H100 (dashed) vs. H200 (solid) GPUs for output length

N=10

. This figure illustrates the scaling behavior of dense and mixture-of-experts model architectures across varying batch sizes

B

and context lengths

C

$B$	$C$	$N$	$E_{\mathrm{NVML}}$ (mJ)	TKLQT Avg.	TKLQT Sum	$T_{N=1}$ (ms)	Prompt tokens/J
1	512	1	5,988.9	8.05 $\mu$ s	6.76 ms	25.18	85.49
2	512	1	6,810.4	24.38 $\mu$ s	21.23 ms	25.41	150.36
4	512	1	9,295.6	668.23 $\mu$ s	581.89 ms	28.73	220.32
8	512	1	33,905.2	9,507.70 $\mu$ s	8,279.31 ms	45.27	120.81
8	1024	1	71,897.2	40,383.57 $\mu$ s	35,166.01 ms	111.31	113.94
8	2048	1	191,210.2	145,693.76 $\mu$ s	131,561.46 ms	325.74	85.69

TABLE VI: Llama-3.2-1B/H100 TTFT-oriented, prefill-dominated transition from host-bound to compute-heavy request shapes.

T_{N=1}

is the complete

N=1

inference-window time. Prompt tokens/J is computed as

(B\times C)/E_{\mathrm{NVML}}

. The TKLQT [ 7 ] sum is the sum of per-launch launch-to-start delays. Since queue intervals can overlap, the wall-clock time is not additive. Bold text marks configurations where

T_{N=1}

remains nearly constant as batch size increases.

Request Regime	TKLQT	$E_{\mathrm{NVML}}$	Floor	Replay-active
(Configuration)	Avg ( $\mu$ s)	(mJ)	share	share
Host-bound $B{=}1,C{=}512,$ $N{=}1$	8.05	5,988.9	61.1%	13.6%
Compute-heavy $B{=}8,C{=}4096,$ $N{=}1$	540,615.08	686,698.1	27.6%	60.4%

TABLE VII: Llama-3.2-1B/H100 diagnostic floor and isolated-replay estimates for host-bound and compute-heavy TTFT-oriented, prefill-dominated configurations. Floor and replay-active shares are normalized by measured NVML request energy and are reported separately because the diagnostic estimates are not strictly additive.

Fig. 6: TTFT-oriented (

N=1

) and long-output (

N=128

) inference operating points on H200 at

B=4

C=2048

Regime	Observed Energy Behavior	Strategy for Scheduler Actions
Short-context dense decode	Fixed energy dominates; batching strongly lowers token energy.	Use moderate batching when latency allows; tune the target by model scale, architecture, context length, and SLO rather than assuming a universal batch size.
Long-context dense decode	KV-cache traffic raises step energy; batch gain erodes with context length $C$ .	Avoid waiting too long for large batches; optimize latency and request energy jointly.
MoE decode at low concurrency	Low-concurrency MoE behavior is consistent with routing and fragmented dispatch overheads.	Batch MoE requests more aggressively than dense requests at the same context length.
Long-output decode	Token energy first improves as fixed energy is spread across more tokens, then can rise when step energy dominates.	Track both request energy and token energy; do not optimize tokens/J alone.
TTFT-oriented vs. long-output requests	TTFT-oriented windows are prefill-dominated; repeated generation dominates long-output request energy.	Track both TTFT-oriented and long-output request energy within the fixed-plus-marginal model.

TABLE VIII: Request-shape policy summary for energy-aware serving.

Benchmark	Model	Cap	H100 Acc.	H100 Energy (J)	H100 Utility/J	H200 Acc.	H200 Energy (J)	H200 Utility/J
MATH-500	Qwen3-8B	64	26.0	316.8	0.000821	25.4	245.8	0.00103
		256	39.0	1,024.7	0.000381	39.0	796.3	0.000490
		1024	68.8	2,351.2	0.000293	68.2	1,845.8	0.000369
	DS-Qwen-7B	64	9.4	640.7	0.000147	9.2	515.2	0.000179
		256	23.2	1,211.4	0.000192	21.4	977.9	0.000219
		1024	38.0	3,148.8	0.000121	36.0	2,567.4	0.000140

TABLE IX: Utility-aware capped-reasoning results at

B{=}1

. Energy is mean per request and utility per joule is an aggregate ratio. DS denotes DeepSeek-R1-Distill.

Explore similar work

From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs
Jul 29, 2026Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis +5LLM Inference Efficiency Energy-Efficient ML
1National Technical University of Athens, Patission Complex 42, 10682 Athens, Greece · 2Harvard University, 1350 Massachusetts Avenue, 02138 Cambridge, MA, USA · 3TWT GmbH Science & Innovation, Industriestraße 6, 70565 Stuttgart, DE +4
Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
Apr 10, 2026Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-MartínezLLM Inference Efficiency Energy-Efficient ML
Universidad Politécnica de Madrid, C. de los Ciruelos, Boadilla del Monte, 28660, Madrid, Spain · Zurich University of Applied Sciences, Switzerland, Gertrudstrasse 15, Winterthur, 8400, Zurich, Switzerland
EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization
May 14, 2026Zhiye Song, Kyungmi Lee, Eun Kyung Lee +3LLM Inference Efficiency Energy-Efficient ML
Massachusetts Institute of Technology Cambridge, MA 02139 · IBM Research Yorktown Heights, NY 10598

Jul 29, 2026J/K move · Enter open · S save

From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs

Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis, George Dasoulas+4

1National Technical University of Athens, Patission Complex 42, 10682 Athens, Greece · 2Harvard University, 1350 Massachusetts Avenue, 02138 Cambridge, MA, USA · 3TWT GmbH Science & Innovation, Industriestraße 6, 70565 Stuttgart, DE+4

LLM Inference Efficiency Energy-Efficient ML LLM Inference

Open paper AskPDF arXiv

The operational energy consumption of large language model (LLM) inference is becoming an increasingly important component of the environmental footprint of deployed AI systems. However, direct measurement of inference energy often requires hardware telemetry, power instrumentation, or infrastructure-specific monitoring, limiting its applicability in comparative studies, early-stage system design, and sustainability reporting. This report presents an analytically structured, empirically calibrated, GPU-level methodology for estimating LLM inference energy on NVIDIA H100-class accelerators without direct runtime measurement. The proposed estimator combines parameter-scaled transformer FLOP accounting, calibrated memory-traffic factors, and hardware-specific energy coefficients for FP16/BF16 tensor-core computation and high-bandwidth-memory movement. It explicitly separates prompt prefill from autoregressive decoding, enabling energy estimates for input tokens, output tokens, and complete inference requests. The methodology further decomposes total energy into compute, parameter-access, key-value-cache write, and attention-read components, allowing the scaling behavior with model size, context length, and generated-token count to be analyzed. The resulting estimates are not intended to replace physical power measurements; rather, they provide transparent, reproducible, and assumption-explicit approximations suitable for model comparison, green-coding analysis, and design-time evaluation of LLM inference workloads.

Abstract

Explore similar work

From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures

EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization