Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures

Mauricio Fadel Argerich; Jonathan Fürst; Marta Patiño-Martínez

cs.DCApr 10, 2026

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures

Authors: Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

Organizations: Universidad Politécnica de Madrid, C. de los Ciruelos, Boadilla del Monte, 28660, Madrid, Spain · Zurich University of Applied Sciences, Switzerland, Gertrudstrasse 15, Winterthur, 8400, Zurich, Switzerland

Abstract

While the large energy consumption of Large Language Models (LLMs) is recognized by the community, system operators lack guidance for energy-efficient LLM inference deployments that leverage energy trade-offs of heterogeneous hardware due to a lack of energy-aware benchmarks and data. In this work we address this gap with Watt Counts: the largest open-access dataset of energy consumption of LLMs, with over 5,000 experiments for 50 LLMs across 10 NVIDIA Graphics Processing Units (GPUs) in batch and server scenarios along with a reproducible, open-source benchmark that enables community submissions to expand this dataset. Leveraging this dataset, we conduct a system-level study of LLM inference across heterogeneous GPU architectures and show that GPU selection is crucial for energy efficiency outcomes and that optimal hardware choices vary significantly across models and deployment scenarios, demonstrating the critical importance of hardware-aware deployment in heterogeneous LLM systems. Guided by our data and insights, we show that practitioners can reduce energy consumption by up to 70% in server scenarios with negligible impact on user experience, and by up to 20% in batch scenarios.

Figures & tables

Dataset	# GPUs	# LLMs	LLM-GPU comb.	Energy Metrics	Workload Type/Scenario	Open Data	Engine
LLM Energy Consumption Dataset [ 13 ]	3	6	18	✓	Batch	✓	ollama
Azure LLM Inf. Traces [ 35 ]	1	6	6	✗ 1	Server	✓	vLLM
AI Energy Score [ 21 ]	1	121	121	✓	Batch	✓	PyTorch, vLLM, Optimum 2
ML.ENERGY [ 6 ]	2	27	54	✓	Batch	✓	vLLM
MLPerf Power [ 38 ]	11 3	3	33	✓ 4	Server, Batch, Single/Multiple Stream	✓	PyTorch, ONNX
J. Fernandez et al. [ 9 ]	3	7	21	✓	Batch	✗	PyTorch, vLLM

Table 1: Comparison of energy-aware LLM inference datasets

Figure 1: Benchmark design overview. The orchestrator launches the System Under Test (SUT) based on a user-provided configuration. The SUT runs in batch mode (vLLM synchronous engine) or server mode (vLLM asynchronous engine with a Poisson-based load generator). The metrics collector aggregates energy, system, and LLM-level performance measurements throughout execution.

GPU	Architecture	Profile	Memory (GB)	TDP (W)	TFLOPS (FP16)	Mem BW (GB/s)	L2 Cache (MB)	TFLOPS per Watt	Year
Tesla V100 SXM2	Volta	Enterprise	32	250	125	898	6	0.5	2018
Tesla T4	Turing	Enterprise	16	70	65	320	4	0.93	2018
A100 SXM4	Ampere	Enterprise	40	400	312	1555	40	0.78	2020
GeForce RTX 3090	Ampere	Consumer	24	350	71	936	6	0.20	2020
A30 PCIe	Ampere	Enterprise	24	165	165	933	24	1.00	2021
GeForce RTX 4090	Ada Lovelace	Consumer	24	450	165	1010	72	0.37	2022

Table 2: NVIDIA GPUs used in the dataset and their main technical characteristics: architecture, profile (consumer or enterprise), memory, TDPs, tensor FP16 TFLOPS without sparsity, memory bandwidth, L2 cache size, TFLOPS (without sparsity) per Watt, and release year.

GPU	1st	2nd	3rd	Total models
A100 SXM4 40 GB	0	0	13	42
A30 PCIe	0	1	13	34
GeForce RTX 4090	0	1	1	34
H100 NVL 94 GB	45	4	1	50
H200 NVL	2	43	2	50
L4	3	1	11	33

Table 3: GPU ranking positions of energy-per-token achieved in the batch scenario. Only GPUs ranked 3rd or above for at least one model are shown.

Figure 2: Mean energy per token for model size categories across GPUs for selected models. Error bars show 95% confidence intervals. Missing bars indicate that the GPU could not run any models in that size category due to memory or compatibility limitations.

Figure 3: Mean throughput for model size categories across GPUs for selected models. Error bars show 95% confidence intervals. Missing bars indicate that the GPU could not run any models in that size category due to memory or compatibility limitations.

Figure 4: The gray area shows the 95% prediction interval of a log–log regression,

\log(E)=\alpha\log(\text{size})+\beta

, fitted over all data points; data points outside this envelope are considered energy outliers.

Figure 5: GPU power draw across model sizes and scenarios. Bars show the minimum, mean, and maximum power draw for different model size categories in order (S/M/L/XL) across GPUs under batch, server low, and server high load scenarios. TDP and measured idle power are reported in the last row for reference.

GPU	1st	2nd	3rd
Server High Load
A100 SXM4 40 GB	6	1	7
A30 PCIe	1	19	14
GeForce RTX 4090	0	0	1
H100 NVL 94 GB	9	6	4
H200 NVL	0	7	5

Table 4: GPU ranking positions based on energy efficiency under server inference.

GPU	TTFT p95 (ms)	Mean Power (W)
Small
A100 SXM4 40 GB	12	67.4
A30 PCIe	33	32.3
GeForce RTX 3090	57	52.3
GeForce RTX 4090	21	31.9
H100 NVL 94 GB	17	90.8

Table 5: Tail latency (TTFT p95) per experiment, averaged per model, then averaged per size category. Scenario: Server Low Load. The GPU with the lowest mean power per category that satisfies the requirement of a TTFT

<70ms

is in bold text.

GPU	TTFT p95 (ms)	Mean Power (W)
Small
A100 SXM4 40 GB	17	112.7
A30 PCIe	31	72.5
GeForce RTX 3090	25	198.0
GeForce RTX 4090	23	121.1
H100 NVL 94 GB	18	119.9

Table 6: Tail latency (TTFT p95) per experiment, averaged per model, then averaged per size category. Scenario: Server high load. The GPU with the lowest mean power per category that satisfies the requirement of a TTFT

<70ms

is in bold text.

Figure 6: Scatter plot showing mean GPU power draw versus the 95th percentile of Time-to-First-Token (TTFT), per model and scenario. Each data point is a (model, GPU, scenario) configuration, averaged over five iterations.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Scenario	Component	Share (%)	Corr. with Total
Batch	GPU	92.5	1.000
	CPU	7.0	0.989
	DRAM	0.5	0.989
Server high	GPU	91.2	1.000
	CPU	7.4	0.995
	DRAM	1.4	0.127

Appendix

Table 7: Mean energy share and Pearson correlation with total system energy consumption per component and deployment scenario in the bare metal instance with RAPL access.

GPU	Model	Params (B)	Mean Energy/Token
Tesla V100 SXM2 32 GB	allenai/OLMoE-1B-7B-0924	1.000000	0.256803
Tesla T4	microsoft/Phi-tiny-MoE-instruct	1.100000	0.135182
Tesla V100 SXM2 32 GB	microsoft/Phi-tiny-MoE-instruct	1.100000	0.140642
L40S	HuggingFaceTB/SmolLM2-1.7B-Instruct	1.700000	0.113926
GeForce RTX 3090	HuggingFaceTB/SmolLM2-1.7B-Instruct	1.700000	0.143895
Tesla T4	HuggingFaceTB/SmolLM2-1.7B-Instruct	1.700000	0.129636

Appendix

Table 8: Log-log energy per token vs model size outliers

Variable	Coef.	Std.Err.	z	P $>\|$ z $\|$	[0.025	0.975]
Intercept	-0.919	0.123	-7.47	0.000	-1.160	-0.678
log_params	0.237	0.014	16.93	0.000	0.210	0.264
C(model_type)[T.gemma2]	-0.495	0.109	-4.54	0.000	-0.709	-0.281
C(model_type)[T.gpt_oss]	-0.219	0.154	-1.42	0.155	-0.521	0.083
C(model_type)[T.granite]	-0.409	0.098	-4.17	0.000	-0.601	-0.217
C(model_type)[T.granitemoe]	-0.443	0.105	-4.22	0.000	-0.649	-0.237

Appendix

Table 9: Mixed-effects linear regression results for log energy per token (Model A: log parameters + model type). GPU is modeled as a random intercept. Coefficients are reported with standard errors,

z

-statistics,

p

-values, and 95% confidence intervals.

Variable	Coef.	Std.Err.	z	P $>\|$ z $\|$	[0.025	0.975]
Intercept	-0.948	0.117	-8.10	0.000	-1.177	-0.719
log_params	0.282	0.013	21.69	0.000	0.257	0.307
hidden_size_resid	-0.098	0.044	-2.23	0.026	-0.184	-0.012
num_layers_resid	0.028	0.015	1.87	0.062	-0.001	0.057
num_attention_heads_resid	0.070	0.024	2.92	0.004	0.023	0.117
num_key_value_heads_resid	0.121	0.014	8.64	0.000	0.094	0.148

Appendix

Table 10: Mixed-effects linear regression results for log energy per token (Model B: residualized architectural features). GPU is modeled as a random intercept. Coefficients are reported with standard errors,

z

-statistics,

p

-values, and 95% confidence intervals.

Model	Family	Arch.	Total Params (B)	Active Params (B)	GPUs
Small models
openai-community/gpt2	gpt2	Dense	0.124	0.124	10
Qwen/Qwen2.5-0.5B-Instruct	qwen2	Dense	0.490	0.490	10
nvidia/AceMath-1.5B-Instruct	qwen2	Dense	1.500	1.500	10
Qwen/Qwen2.5-1.5B-Instruct	qwen2	Dense	1.540	1.540	10
HuggingFaceTB/SmolLM2-1.7B-Instruct	llama	Dense	1.700	1.700	10

Appendix

Table 11: Full list of the 50 LLMs evaluated in this study. Models are grouped by size category (based on minimum GPU memory required) and sorted by total parameter count within each group. MoE models are listed separately as size categories are based on GPU memory fitting of dense models. “GPUs” indicates the number of distinct GPUs each model was evaluated on.

Explore similar work

WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs
Jul 2, 2026Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-MartínezLLM Inference Efficiency Energy-Efficient ML
Universidad Politécnica de Madrid, C. de los Ciruelos, 28660 Boadilla del Monte, Madrid, Spain · Zurich University of Applied Sciences, Gertrudstrasse 15, 8400 Winterthur, Switzerland
From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs
Jul 29, 2026Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis +5LLM Inference Efficiency Energy-Efficient ML
1National Technical University of Athens, Patission Complex 42, 10682 Athens, Greece · 2Harvard University, 1350 Massachusetts Avenue, 02138 Cambridge, MA, USA · 3TWT GmbH Science & Innovation, Industriestraße 6, 70565 Stuttgart, DE +4
EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization
May 14, 2026Zhiye Song, Kyungmi Lee, Eun Kyung Lee +3LLM Inference Efficiency Energy-Efficient ML
Massachusetts Institute of Technology Cambridge, MA 02139 · IBM Research Yorktown Heights, NY 10598

Jul 2, 2026J/K move · Enter open · S save

WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

Universidad Politécnica de Madrid, C. de los Ciruelos, 28660 Boadilla del Monte, Madrid, Spain · Zurich University of Applied Sciences, Gertrudstrasse 15, 8400 Winterthur, Switzerland

LLM Inference Efficiency Energy-Efficient ML LLM Inference

Open paper AskPDF arXiv

Large Language Model (LLM) inference workloads are a rapidly growing contributor to data center energy consumption. Optimizing these deployments requires matching specific LLMs to the most efficient GPUs, but operators currently lack the tools to do so without exhaustively profiling each combination. While some predictive models exist, they still require profiling data and struggle to generalize to hardware unseen during training. To address this, we introduce WattGPU, featuring two predictive models for mean GPU power draw and Inter-Token Latency (ITL). Our approach leverages only publicly available LLM metadata and GPU specifications, eliminating the need for hardware access or profiling while enabling generalization to unseen NVIDIA server-grade GPUs and LLMs. We evaluate our models using rigorous leave-one-GPU-out and leave-one-LLM-out cross-validation on a dataset of 42 open-source LLMs (0.1B--27B parameters) and 8 GPUs under both offline and server scenarios. The mean power draw model achieves a median absolute percentage error of

\leq3.4\%

for offline and

\leq13.5\%

for server scenarios on unseen GPUs, while the latency model achieves

\leq8.5\%

in server mode, both maintaining strong GPU ranking correlations for server scenarios (Kendall

τ\geq0.76

). Compared to standard physically grounded baselines -- Load-Scaled Thermal Design Power (TDP) for power draw and roofline for latency -- our models reduce median absolute percentage error by approximately 4

\times

on unseen LLM-GPU combinations for server scenarios or approximately 2

\times

for completely unseen GPUs. WattGPU's data and code are publicly available at https://github.com/maufadel/wattgpu.

Abstract

Explore similar work

WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs

EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization