Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware

Zeyu Yang; Wes Armour

cs.AIOct 9, 2026

Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware

Authors: Zeyu Yang, Wes Armour

Organizations: Department of Engineering Science, University of Oxford

Abstract

Benchmark scores tell us what an LLM can do, but not how much energy is required to deliver that capability. We directly measure this trade-off for 86 open-weight model checkpoints, evaluated on seven benchmarks and six hardware platforms ranging from a consumer GPU to an eight-GPU server. We find large differences in energy efficiency even among models with similar capability. Most configurations lie away from the score--energy Pareto frontier: they use a median of 2.6--4.8 $\times$ more energy than another configuration that scores at least as well, with differences reaching 27 $\times$ . Reasoning can improve capability, but enabling thinking increases energy by about 3.7 $\times$ on average, and the benefit depends strongly on the model family and output budget. Generated-token count is also an imperfect proxy for energy, misordering 18--24% of model pairs. In contrast, FP8 reduces energy by roughly a quarter with little change in score, while changing hardware largely rescales energy without changing the relative ranking of models. These results show that capability alone gives an incomplete picture of efficiency. We argue that benchmark scores should be reported together with measured energy consumption under a specified serving regime and output budget.

Figures & tables

Platform	Class	Arch.	Memory	Model Tier
GeForce RTX 5090	desktop	Blackwell	32 GB GDDR7	32
2 $\times$ RTX A6000	workstation	Ampere	2 $\times$ 48 GB GDDR6	96
GH200	server	Hopper	96 GB HBM3	96
GB10 (DGX Spark)	desktop	Blackwell	128 GB unified LPDDR5X	96
RTX PRO 6000 Blackwell	workstation	Blackwell	96 GB GDDR7	96
8 $\times$ H100 SXM	server	Hopper	8 $\times$ 80 GB HBM3	512

Table 1: Serving platforms. Multi-GPU platforms are NVLink-connected and use tensor parallelism, plus expert parallelism for MoE models on the 8-GPU node.

Benchmark	Ability	Items	$k$	Cap	Headline metric
IFEval ( Zhou et al., 2023 )	instr. following	541	1	8K	prompt-level strict acc.
MMLU-Pro (1k) ( Wang et al., 2024 )	knowledge	1,000	1	16K	exact match
GPQA Diamond ( Rein et al., 2023 )	science	198	2	32K	exact match
MATH-500 ( Lightman et al., 2024 )	math	500	1	16K	exact match
AIME 25/26 ( Zhang and Team, 2026 )	competition math	60	4	32K	exact match
LiveCodeBench ( Jain et al., 2025 )	coding	175	1	32K	pass@1 (executed)

Table 2: The benchmark suite.

k

= seeded attempts. cap = maximum new tokens. † per turn.

Figure 1: Capability–energy frontiers across the three memory tiers. Each panel shows one tier on its reference platform (BF16 for 32/96 GiB; FP8 for 512 GiB).

x

: geometric-mean GPU energy per attempt;

y

: mean suite score. Marker encoding follows the legend; steps show the Pareto fronts. Fig. 10 gives the fully labelled version.

Figure 2: Energy varies through both generation length and energy per token. Each panel shows all complete configurations on a reference platform, across weight formats.

x

: geometric-mean generated tokens per attempt;

y

: GPU energy per generated token, including prefill. Their product gives the suite energy in Fig. 1 . Dashed diagonals show equal energy; dotted lines show platform medians. Marker encoding follows Fig. 1 .

Figure 3: Effect of enabling reasoning with the same weights on the same hardware. Each point compares one hybrid checkpoint with thinking disabled and enabled on its tier’s reference platform (44 pairs), coloured by lineage; boxes show quartiles by platform and pooled. Left: change in suite score, shown in points and as a ratio. Right: change in GPU energy for one pass over the suite, shown in kWh and as a ratio (log scale).

Figure 4: Increasing the output budget can recover capability at higher energy cost. Four thinking configurations are rerun from 8K to 128K output tokens; the main evaluation uses 32K. Left: score against GPU energy per attempt, with marker area proportional to the output cap; energy is normalised to the request count. Right: throughput and energy per generated token by output position. Dashed lines: RTX 5090; solid lines: 8

\times

H100.

Figure 5: Effect of replacing BF16 weights with vendor-released FP8 weights on the same hardware. Each point compares the same base model and reasoning mode in vendor-released BF16 and FP8 variants on its tier’s reference platform (28 matched pairs), coloured by lineage; boxes show quartiles by platform and pooled. Left: change in suite score, in points and as a ratio. Right: change in GPU energy consumption, in kWh and as a ratio.

Figure 6: Energy, performance, and cost across the four 96 GiB platforms. Top: suite GPU energy, accelerator power, runtime, and token throughput. Bottom: accelerator price, hardware cost at matched RTX PRO 6000 throughput, and corresponding annual accelerator electricity cost and power. Secondary-market prices use ten eBay listings;

1,000 is deducted from the GB10 price for its non-GPU resources. Electricity assumes

0.20/kWh and continuous operation. Host and infrastructure power are excluded.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Alibaba Qwen/
Checkpoint	Params	Weights	Precision	Modes	Released
	(B)	(GiB)
Qwen3-4B-Instruct-2507	4	7.5	BF16	I	2025-09-17
Qwen3-4B-Instruct-2507-FP8	4	4.8	FP8	I	2025-09-17
Qwen3-4B-Thinking-2507	4	7.5	BF16	T	2025-08-06
Qwen3-4B-Thinking-2507-FP8	4	4.8	FP8	T	2025-08-06

Appendix

Table 3: Roster of the 32 GiB tier (31 checkpoints, 47 study rows), served on a single NVIDIA GeForce RTX 5090. Params : billions of parameters; MoE checkpoints report total / active parameters (Gemma 4 E-series: total / effective). Weights : size of the safetensors weights loaded by the server. Precision : stored weight format; gpt-oss stores its MoE experts in MXFP4. Modes : I = instruct, T = thinking, NT = non-thinking; hybrid checkpoints listed as T, NT contribute two study rows. L, M, H denote documented low, medium, and high reasoning-effort settings. Released : checkpoint release date.

Alibaba Qwen/
Checkpoint	Params	Weights	Precision	Modes	Released
	(B)	(GiB)
Qwen3-30B-A3B-Instruct-2507	30 / 3	56.9	BF16	I	2025-09-17
Qwen3-30B-A3B-Instruct-2507-FP8	30 / 3	29.0	FP8	I	2025-09-17
Qwen3-30B-A3B-Thinking-2507	30 / 3	56.9	BF16	T	2025-08-17
Qwen3-30B-A3B-Thinking-2507-FP8	30 / 3	29.0	FP8	T	2025-07-30

Appendix

Table 4: Roster of the 96 GiB tier (29 checkpoints, 47 study rows), served on all four 96 GiB platforms: GB10, RTX PRO 6000 Blackwell, GH200, and 2

\times

RTX A6000 (TP=2). Columns are defined in Table 3 .

Checkpoint	Params	Weights	Precision	Modes	Released
	(B)	(GiB)
Alibaba Qwen/
Qwen3-235B-A22B-Instruct-2507	235 / 22	437.9	BF16	I	2025-09-17
Qwen3-235B-A22B-Instruct-2507-FP8	235 / 22	220.2	FP8	I	2025-09-17
Qwen3-235B-A22B-Thinking-2507	235 / 22	437.9	BF16	T	2025-08-17
Qwen3-235B-A22B-Thinking-2507-FP8	235 / 22	220.2	FP8	T	2025-07-30

Appendix

Table 5: Roster of the 512 GiB tier (26 checkpoints, 42 study rows), served on the 8

\times

H100 node with TP=8 and expert parallelism enabled for MoE checkpoints. Columns are defined in Table 3 .

System
	GeForce RTX 5090	2 $\times$ RTX A6000	GH200	GB10 (DGX Spark)	RTX PRO 6000 Blackwell	8 $\times$ H100 SXM
Host CPU	Ryzen 9 7900X	Xeon w7-3455	Grace	10 $\times$ Cortex-X925 + 10 $\times$ Cortex-A725	Core i5-14600K	2 $\times$ Xeon Platinum 8468
Cores/Threads	12C/24T	24C/48T	72C/72T	20C/20T	6P8E/20T	96C/192T
Host RAM	64 GB DDR5	256 GB DDR5	480 GB LPDDR5X	128 GB unified LPDDR5X	128 GB DDR5	1 TB DDR5
OS	Ubuntu 24.04	Ubuntu 24.04	AlmaLinux 9.4	Ubuntu 24.04	Ubuntu 22.04	AlmaLinux 9.4
kernel	6.8	6.8	5.14	6.14	6.8	5.14

Appendix

Table 6: Hardware, software, measurement, and serving details for the six platforms in Table 1 .

Figure 7: Energy outside the primary accelerator measurement on the two superchips. Left: energy over all metered attempts, split by component. For the GB10, the primary measurement is the GPU rail; for the GH200, it is the GPU+HBM reading. Right: the same shares by reasoning mode, weight format, and benchmark, with 95% bootstrap intervals on the primary-counter share and mean power at right. Package/module measurements still exclude power-supply losses, fans, and storage.

Figure 8: Benchmark shares of summed suite-pass energy. For every complete configuration on each reference platform, boxes show the share of total suite-pass energy contributed by each benchmark, separated into non-thinking (orange) and thinking (blue) configurations. Points show individual configurations. The dashed line marks one seventh, the equal weight assigned to each component by the geometric mean.

Figure 9: Each benchmark’s contribution to cross-configuration variation in aggregate energy. For each reference platform, bars show the contribution of each benchmark under the arithmetic sum of raw energies (light) and the geometric mean used in the paper (dark, on log energies). Contributions sum to 100%; error bars are 95% bootstrap intervals over configurations. The dashed line marks an equal one-seventh contribution.

Figure 10: Score–energy fronts with every configuration labelled. Enlarged versions of the three panels in Fig. 1 ; marker encoding is unchanged.

Figure 11: Score–energy fronts under the clean-completion bound. As in Fig. 10 , but benchmark scores are recomputed using only requests that finish within the output cap. Energy is unchanged and the Pareto fronts are recomputed.

Figure 12: Energy per correct answer. All configurations are shown on their tier’s reference platform; AIME 2025 and 2026 are pooled, and BFCL is excluded. Left: energy per correct answer across benchmarks; boxes summarise configurations and points denote reasoning mode, with the extremes labelled. Right: energy per correct AIME answer against AIME accuracy. The dashed step gives the minimum energy per correct answer among configurations achieving at least each accuracy.

Figure 13: Efficiency, capability, and energy over release date. Each complete configuration is placed at its checkpoint release date on its tier’s reference platform, across all weight formats. Rows show energy per score point, suite score, and suite energy. Solid steps show the best value observed up to each date; dashed and dotted lines show log-scale least-squares trends for non-thinking and thinking configurations. Marker encoding follows Fig. 1 .

Figure 14: Batch size 1 against the campaign serving regime on the 8

\times

H100 node. Energy per generated token ( left ) and throughput ( right ) for seven configurations on AIME 2026 and LiveCodeBench. Open markers show one request at a time; filled markers show the corresponding campaign runs, with multiplicative gaps annotated. Squares show geometric means by benchmark, configuration, and across all 14 pairs. Colour denotes lineage; circles are non-thinking and hexagons thinking configurations.

Figure 15: Weights offloaded to Grace memory on the GH200. Minutes per attempt ( left ), throughput ( middle ), and energy per generated token from the GPU+HBM measurement ( right ) against active parameter count for MMLU-Pro, AIME 2026, and LiveCodeBench. Small points show FP8 non-thinking configurations that fit entirely in HBM3; dashed lines and bands show their log-scale least-squares trends and

\pm 1

residual standard deviation. Open markers show Qwen3.5-122B-A10B (57% offloaded) and Qwen3.5-397B-A17B (91%), labelled by their ratio to the fitted trend.

Figure 16: Primary accelerator energy consumed by the completed evaluation campaign. Energy is broken down by platform ( left ) and benchmark ( right ), totalling approximately 1.34 MWh. The accounting includes all metered benchmark attempts, including side experiments and attempts later replaced. Model loading, idle periods, host power, and other infrastructure overheads are excluded.

Energy-Aware LLM Inference

See all 25 papers

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
Apr 10, 2026Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-MartínezEnergy-Aware LLM Inference LLM Inference Efficiency
Universidad Politécnica de Madrid, C. de los Ciruelos, Boadilla del Monte, 28660, Madrid, Spain · Zurich University of Applied Sciences, Switzerland, Gertrudstrasse 15, Winterthur, 8400, Zurich, Switzerland
Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms
Aug 28, 2026Prabhu Vellaisamy, Vanessa Lam, Shawn Blanton +1Energy-Aware LLM Inference LLM Inference
Electrical and Computer Engineering Department Carnegie Mellon University Pittsburgh, PA, USA
EnergyLens: Interpretable Closed-Form Energy Models for Multimodal LLM Inference Serving
May 11, 2026Vittorio Palladino, Gianluca Palermo, Michael E. Papka +1Efficient Multimodal Inference LLM Inference
Politecnico di Milano & UIC · Politecnico di Milano · University of Illinois Chicago

Apr 10, 2026J/K move · Enter open · S save

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures

Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

Universidad Politécnica de Madrid, C. de los Ciruelos, Boadilla del Monte, 28660, Madrid, Spain · Zurich University of Applied Sciences, Switzerland, Gertrudstrasse 15, Winterthur, 8400, Zurich, Switzerland

Energy-Aware LLM Inference LLM Inference Efficiency Energy-Efficient ML Heterogeneous Computing LLM Evaluation

Open paper AskPDF arXiv

While the large energy consumption of Large Language Models (LLMs) is recognized by the community, system operators lack guidance for energy-efficient LLM inference deployments that leverage energy trade-offs of heterogeneous hardware due to a lack of energy-aware benchmarks and data. In this work we address this gap with Watt Counts: the largest open-access dataset of energy consumption of LLMs, with over 5,000 experiments for 50 LLMs across 10 NVIDIA Graphics Processing Units (GPUs) in batch and server scenarios along with a reproducible, open-source benchmark that enables community submissions to expand this dataset. Leveraging this dataset, we conduct a system-level study of LLM inference across heterogeneous GPU architectures and show that GPU selection is crucial for energy efficiency outcomes and that optimal hardware choices vary significantly across models and deployment scenarios, demonstrating the critical importance of hardware-aware deployment in heterogeneous LLM systems. Guided by our data and insights, we show that practitioners can reduce energy consumption by up to 70% in server scenarios with negligible impact on user experience, and by up to 20% in batch scenarios.

Abstract

Energy-Aware LLM Inference

Explore similar work

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

EnergyLens: Interpretable Closed-Form Energy Models for Multimodal LLM Inference Serving