The Real Cost of Context: Predicting Compute Time and Energy Consumption in LLM Prefill and Decode Phases

Lars Krupp; Daniel Geißler; Francisco M. Calatrava-Nicolas; Stefan Gerd Fritsch; Paul Lukowicz; Jakob Karolus

cs.AIOct 29, 2026

The Real Cost of Context: Predicting Compute Time and Energy Consumption in LLM Prefill and Decode Phases

Authors: Lars Krupp, Daniel Geißler, Francisco M. Calatrava-Nicolas, Stefan Gerd Fritsch, Paul Lukowicz, Jakob Karolus

Organizations: RPTU Kaiserslautern-Landau, DFKI · Örebro University · RPTU Kaiserslautern-Landau

Abstract

The integration of Large Language Models (LLMs) into agentic architectures has substantially increased token consumption, raising concerns about their energy footprint. Yet, estimating the energy consumption of proprietary API-based models remains challenging, and existing approaches focus primarily on generated output tokens while overlooking the impact of the large input contexts characteristic of agentic workflows. To address this gap, we systematically profile compute time and energy consumption across a wide range of prefill (input) and decode (output) token lengths. Using generalized linear models, we show that decode compute time and energy scale linearly with output length, whereas prefill compute time and energy exhibit both linear and quadratic scaling with input length. These findings demonstrate that input context becomes a relevant factor for large-context workloads. Furthermore, we show that compute time serves as a reliable proxy for energy consumption when direct energy measurements are unavailable. Our mapping framework enables precise energy estimation and supports more sustainable deployment decisions for agentic AI.

Figures & tables

Figure 1: Compute time in relation to input tokens and output tokens for Mistral-Small-2603 accesssed trough Mistrals API. Added visualizations for the input token region evaluated in related works and how disregarding input tokens affects compute time (dashed line, 100 output tokens).

Predictor	API	H200 2GPU, TP=2	H200 4GPU, TP=4	H100 4GPU, TP=4
$\beta_{0}$ : Intercept	$91.97$	$104.0$	$98.0$	$125.0$
$\beta_{1}$ : $I$	$1.81\times 10^{-2}$	$1.34\times 10^{-2}$	$1.02\times 10^{-2}$	$1.03\times 10^{-2}$
$\beta_{2}$ : $I^{2}$	$2.70\times 10^{-7}$	$2.67\times 10^{-7}$	$1.37\times 10^{-7}$	$1.42\times 10^{-7}$
$\beta_{3}$ : $O$	$6.71$	$4.71$	$4.46$	$4.50$
$\beta_{4}$ : $I\times O$	$1.23\times 10^{-5}$	$2.49\times 10^{-5}$	$2.50\times 10^{-5}$	$2.40\times 10^{-5}$

Table 1: Quasipoisson GLM parameter estimates for Mistral-Small-2603 accessed through Mistrals API and local configurations.

Config	Input	Output	Measured	Predicted	(%)	Measured	Predicted	(%)
	#Tokens		Compute Time (ms)		Diff.	Total Energy (Wh)		Diff.
API	210384	50	15949	16334	+2.40
	210384	1200	27144	27015	-0.50		N/A
	210384	2500	39664	39088	-1.50
H200 2 GPU,TP=2	210384	50	15112	15247	+0.89	5.63	5.64	+0.20
	210384	1200	27135	26701	-1.60	8.03	7.97	-0.75

Table 2: Prediction and Measurements for the compute time in ms and energy consumption in Wh for multiple unseen input–output token configurations with difference in percent.

\text{Diff}=\frac{100\cdot(Predicted-Measured)}{Measured}

Figure 2: Visualization of the relationship between predicted energy (Wh) and predicted compute time (seconds) of the GLMs for different hardware configurations. Linetype indicates the number of generated output tokens, while the lines themselves relate to the number of input tokens. Dots at 1000, 71,000, 140,000, and 210,000 input tokens to allow for easier comparison between different configurations. Input token stepsize = 1000.

Figure 3: Comparison between the estimation of Ecologits (dashed line), the H200 2GPU, TP=2 energy GLM (solid line, adjusted to include PUE) and the locally measured energy consumption (dots, adjusted to include PUE).

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 285 papers

Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures
Aug 25, 2026Molka Chkir, Syed Muhammad Danish, Jos Höll +1Energy-Aware LLM Inference LLM Inference Efficiency
Algoma University, Brampton, Canada · School of Informatics, Reutlingen University, Reutlingen
This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMs
Mar 16, 2026Lars Krupp, Daniel Geißler, Francisco M. Calatrava-Nicolas +3Energy-Aware LLM Inference LLM Inference
Embedded Intelligence DFKI and RPTU Kaiserslautern, Germany · Centre for Applied Autonomous Sensor Systems (AASS) Örebro University Örebro, Sweden
From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs
Jul 29, 2026Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis +5LLM Inference Efficiency Energy-Efficient ML
1National Technical University of Athens, Patission Complex 42, 10682 Athens, Greece · 2Harvard University, 1350 Massachusetts Avenue, 02138 Cambridge, MA, USA · 3TWT GmbH Science & Innovation, Industriestraße 6, 70565 Stuttgart, DE +4

Aug 25, 2026J/K move · Enter open · S save

Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures

Molka Chkir, Syed Muhammad Danish, Jos Höll, Arghavan Asad

Algoma University, Brampton, Canada · School of Informatics, Reutlingen University, Reutlingen

Energy-Aware LLM Inference LLM Inference Efficiency Long-Context Language Model Inference Grouped-Query Attention Sliding-Window Attention

Open paper AskPDF arXiv

The growing adoption of large language models (LLMs) has raised increasing concerns about the energy consumption and environmental impact of inference. This paper presents a systematic empirical study of decode-phase energy consumption across representative open-source LLMs employing Multi-Head Attention (MHA), Grouped Query Attention (GQA), and Grouped Query Attention with Sliding Window Attention (SWA) to characterize how attention architecture influences decode-phase energy consumption under varying inference workloads. We evaluate four models across different context lengths, batch sizes, and generation workloads while measuring GPU energy using NVIDIA hardware counters. We examine the effects of context length, attention mechanism, Key-Value (KV) cache growth, and batching on decode-phase energy consumption. Results show that attention mechanism is the primary factor governing how decode energy scales with context length. MHA models exhibit substantially steeper energy growth than GQA models, whereas GQA with SWA maintains nearly constant energy consumption. We further show that model size primarily determines absolute energy consumption, while batching reduces both energy per generated token and request latency by up to 87%. These findings provide practical guidance for selecting energy-efficient LLM architectures and inference configurations.

Abstract

LLM Inference

Explore similar work

Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures

This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMs

From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs