The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment

Sai Sathvik Vadari

cs.DCApr 15, 2026

The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment

Authors: Sai Sathvik Vadari

Organizations: 8bit.ai Hyderabad, India

Abstract

The AI inference industry keeps models loaded in GPU memory around the clock to avoid cold-start latency, implicitly treating idle power as a fixed cost of readiness. Yet the structure of this cost has never been empirically decomposed - and never across GPU architectures. We present the first cross-architecture measurement of idle GPU power as a function of VRAM allocation, combining 18 days of production telemetry (335,267 samples, 14 H100 GPUs) with controlled dose-response experiments on three GPU architectures spanning three memory technologies: NVIDIA H100 (HBM3, 80 GB), A100 (HBM2e, 80 GB), and L40S (GDDR6, 48 GB). We observe that idle power is piecewise constant on all three architectures: the CUDA context forces a discrete DVFS transition consuming +26-66 W over bare idle (26-50 W on HBM architectures, 66 W on GDDR6), while the marginal VRAM effect is bounded below measurement relevance ( $|β| < 0.02$ W/GB) on every device tested. The CUDA context accounts for >98% of the parking tax regardless of memory technology. We validate this finding with a real HuggingFace model (Qwen2.5-7B) on all three architectures, confirming <0.5 W difference from empty tensors on every device, and capture cold-start power profiles during model loading. We derive a cold-start breakeven model showing energy-optimal behavior depends on request arrival rate and loading latency - not model size - with breakeven intervals of 1-5 minutes. Our results identify a constraint consistent across all tested architectures: idle-with-context power is determined by DVFS state, not memory occupancy.

Figures & tables

GPU	Memory	TDP	VRAM Range	$n$ / phase
H100 80GB SXM	HBM3	700 W	0–64 GB	40
A100 80GB PCIe	HBM2e	300 W	0–72 GB	40
L40S 48GB	GDDR6	350 W	0–40 GB	40

Table 1. Phase 2 GPU configurations. All experiments used 30 s sampling intervals with 20-minute recording phases.

	H100	A100	L40S
	(HBM3)	(HBM2e)	(GDDR6)
Bare idle (W)	71.8	53.7	35.6
SM clock, bare (MHz)	345	210	210
CUDA ctx power (W)	121.7	80.0	102.1
SM clock, ctx (MHz)	1980	1410	2520
Context overhead (W)	+49.9	+26.3	+66.4

Table 2. Cross-architecture parking tax. Power values are mean

\pm

std.

\beta

is the linear regression slope across CUDA-active phases. All GPUs at 0% utilization.

GPU	Allocation	VRAM (GB)	Idle (W)	$\Delta$
H100	torch.empty 16 GB	16.0	121.51	—
H100	Qwen2.5-7B (fp16)	14.9	121.48	$-$ 0.03 W
A100	Qwen2.5-7B (fp16)	14.8	105.32	—
A100	Post-unload (ctx only)	0.5	105.53	+0.21 W
L40S	Qwen2.5-7B (fp16)	14.8	97.07	—
L40S	Post-unload (ctx only)	0.5	97.54	+0.47 W

Table 3. Real model validation across three architectures. Qwen2.5-7B (fp16,

\sim

14.9 GB) idle power compared against a reference with different VRAM but the same CUDA context. H100: torch.empty 16 GB; A100 and L40S: post-unload residual context (

\sim

0.5 GB). All deltas are within measurement noise.

Figure 1. The Model Parking Tax across three GPU architectures. Large markers at “bare”: bare-idle power (no CUDA context). Connected lines: CUDA-active phases with increasing VRAM. The discrete DVFS step dominates; all three dose-response curves are flat. The result holds across HBM3, HBM2e, and GDDR6 memory technologies.

Figure 2. Parking tax decomposition. Gray: base idle power. Color: CUDA context overhead (98– $>$ 99% of the tax).

Figure 3. VRAM dose-response detail (CUDA-active phases only, zoomed to $\pm$ 2.5 W). Each panel shows one GPU with regression line. H100 and L40S slopes are not significant; A100 slope is significant but negative (thermal drift, not a memory effect). All three are flat within $\pm$ 1 W.

Loading Method	$P_{\text{load}}$ (W)	$t_{\text{load}}$ (s)	$T^{*}$
Qwen2.5-7B (measured) †	124	30	1.2 min
Standard PyTorch (70B)	300	45	4.5 min
ServerlessLLM (70B)	300	8	48 s
Run:ai Streamer (8B)	200	5	20 s

Table 4. Cold-start energy breakeven on H100 (

P_{\text{park}}=49.9

W). † Measured in this work; others estimated from published benchmarks.

Parameter	Low	Base	High
Fleet size	2.0M	3.76M	6.0M
Utilization $\rho$	50%	65%	80%
$\bar{P}_{\text{park}}$ (W)	26.3 (A100)	40	66.4 (L40S)
$E_{\text{park}}$ (GWh/yr)	92	462	1,745

Table 5. Sensitivity of industry impact to key assumptions.

Policy	Energy (Wh)	Savings	Cold starts
Low traffic (5 req/hr, Poisson)
Always-On	2,921	—	1
TTL 5 min	2,407	17.6%	78
Breakeven ( $T^{*}$ =4.5 min)	2,392	18.1%	81
Bursty traffic (2/60 req/hr)
Always-On	2,921	—	1

Table 6. Scheduler simulation results (H100, 24h, standard PyTorch loader). Energy savings vs. always-on baseline (2,921 Wh).

Energy-Efficient ML

Energy-efficient machine learning reduces the energy and environmental costs of training and inference while preserving useful model performance.

See all 257 papers

WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs
Jul 2, 2026Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-MartínezLLM Inference Efficiency Energy-Efficient ML
Universidad Politécnica de Madrid, C. de los Ciruelos, 28660 Boadilla del Monte, Madrid, Spain · Zurich University of Applied Sciences, Gertrudstrasse 15, 8400 Winterthur, Switzerland
Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms
Aug 28, 2026Prabhu Vellaisamy, Vanessa Lam, Shawn Blanton +1Energy-Aware LLM Inference LLM Inference
Electrical and Computer Engineering Department Carnegie Mellon University Pittsburgh, PA, USA
From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs
Jul 29, 2026Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis +5LLM Inference Efficiency Energy-Efficient ML
1National Technical University of Athens, Patission Complex 42, 10682 Athens, Greece · 2Harvard University, 1350 Massachusetts Avenue, 02138 Cambridge, MA, USA · 3TWT GmbH Science & Innovation, Industriestraße 6, 70565 Stuttgart, DE +4

Jul 2, 2026J/K move · Enter open · S save

WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

Universidad Politécnica de Madrid, C. de los Ciruelos, 28660 Boadilla del Monte, Madrid, Spain · Zurich University of Applied Sciences, Gertrudstrasse 15, 8400 Winterthur, Switzerland

LLM Inference Efficiency Energy-Efficient ML LLM Inference

Open paper AskPDF arXiv

Large Language Model (LLM) inference workloads are a rapidly growing contributor to data center energy consumption. Optimizing these deployments requires matching specific LLMs to the most efficient GPUs, but operators currently lack the tools to do so without exhaustively profiling each combination. While some predictive models exist, they still require profiling data and struggle to generalize to hardware unseen during training. To address this, we introduce WattGPU, featuring two predictive models for mean GPU power draw and Inter-Token Latency (ITL). Our approach leverages only publicly available LLM metadata and GPU specifications, eliminating the need for hardware access or profiling while enabling generalization to unseen NVIDIA server-grade GPUs and LLMs. We evaluate our models using rigorous leave-one-GPU-out and leave-one-LLM-out cross-validation on a dataset of 42 open-source LLMs (0.1B--27B parameters) and 8 GPUs under both offline and server scenarios. The mean power draw model achieves a median absolute percentage error of

\leq3.4\%

for offline and

\leq13.5\%

for server scenarios on unseen GPUs, while the latency model achieves

\leq8.5\%

in server mode, both maintaining strong GPU ranking correlations for server scenarios (Kendall

τ\geq0.76

). Compared to standard physically grounded baselines -- Load-Scaled Thermal Design Power (TDP) for power draw and roofline for latency -- our models reduce median absolute percentage error by approximately 4

\times

on unseen LLM-GPU combinations for server scenarios or approximately 2

\times

for completely unseen GPUs. WattGPU's data and code are publicly available at https://github.com/maufadel/wattgpu.

Abstract

Energy-Efficient ML

Explore similar work

WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs