The growing energy requirements of large language model inference highlight the need for datacenters to support safe grid operation by providing power demand flexibility, while maintaining service for interactive workloads. This work investigates query routing as a mechanism to enable such capability by dynamically allocating queries across different language model instances without deferring requests. We develop a general analytical framework that connects query arrival rates and sequence length distributions to token iteration time, request residence time, serving capacity, and power consumption of model instances. Leveraging this framework, we characterize the feasible query routing allocations and the corresponding power demand flexibilities, enabling the selection of routing strategies to achieve desired power consumption behavior. We validate our framework across four GPU platforms on language models ranging from 1B to 70B parameters, obtaining 2.8% mean absolute percentage error in estimating achievable power flexibility. We evaluate our approach on a real trace-driven simulation of a 10,000 H100 GPUs datacenter, exhibiting flexibility up to 16% of total power consumption, with 90% of the desired power reduction achieved within approximately 10 seconds. These results establish query routing as an effective mechanism for datacenter power demand flexibility and demonstrate the broader applicability of our proposed framework to datacenter level planning and orchestration.
Figures & tables
Figure 1: Power demand flexibility through query routing. Routing queries across large and small language models enables datacenters to adjust power demand for safe grid operation without dropping or deferring inference requests.
Figure 2: Measured and predicted mean token iteration times (TBT) as a function of number of concurrent requests across models and query length distributions. The linear estimator in eq. 1 accurately captures the observed values ( R2>0.99 across all configurations). Larger models exhibit higher TBTs, and workloads with longer requests incur additional attention computation overhead.
Figure 3: Predicted versus measured mean request residence time across configurations. The analytical estimator ( eq. 2 ) captures the increase in request duration with concurrency, achieving an overall 5.5% MAPE, demonstrating reliable estimation of mean request residence time.
Figure 4: Predicted and measured request throughput as a function of request concurrency across configurations. Throughput increases with concurrency but exhibits diminishing returns as the token iteration times grow. The analytical estimator ( eq. 3 ) achieves an overall MAPE of 5.8% , accurately capturing the concurrency-dependent request throughput of model instances.
Figure 5: Estimating request concurrency ( eq. 4 ) from query arrival rate, demonstrating an overall MAPE of 9.2% over feasible operating points. The predicted request concurrency count connects the query arrival rate to the model instance’s inference and power consumption.
Figure 6: Measured and predicted mean GPU board power as a function of request concurrency across configurations. Power consumption increases sharply when transitioning from idle to active inference, followed by a more gradual increase as concurrency grows. The diminishing incremental power at higher concurrency suggests opportunities for reducing aggregate power consumption through request consolidation across model instances.
Figure 7: Capacity-feasible routing decisions (left, eq. 10 ) and corresponding total-power consumption (right, eq. 11 ) across query arrival rates and model–workload configurations. Shaded regions show analytical predictions, bars depict measured ranges.
Figure 8: Total power consumption via query routing for the three Llama-8B and single Llama-1B model instances with queries sampled from ShareGPT. Left: measured and predicted total power ( eq. 11 ) versus the fraction of flexible requests ( f ) assigned to the small model instance, with varying query arrival rates. Right: measured and predicted power flexibility range ( eq. 12 ) over feasible allocations.
Figure 9: Datacenter power demand flexibility over week-long Azure LLM workload trace. Top: Baseline facility power and minimum achievable power under varying fractions of flexible queries. Bottom: Distribution and hourly variation of available power curtailment, highlighting its dependence on query workload.
Figure 10: Datacenter power control during a four-hour curtailment event. Top: Sustained power reductions achieved through flexible query routing. Bottom: Event boundaries showing fast transitions between operating points in the order of seconds.
Figure 11: Impact of GPU availability and resource sharing on power demand flexibility. Top: Achievable power curtailment under reduced GPU availability and the resulting utilization during scheduled sharing. Bottom: Daily power flexibility with GPU sharing, demonstrating resilience to moderate resource constraints.
Domain
Application
Serving design
Model instance pool sizing, deployment comparisons, demand growth headroom.
Power Control
Feasible targets, sustained power curtailments, and transition planning.
Energy coordination
Joint inference/training budgets, storage support, and cost or carbon objectives.
Broader routing
Allocation across execution choices or locations, subject to eligibility constraints.
Table 1: Broader applications of our presented framework.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
λ
Query arrival rate (requests/s)
b
request concurrency
bmax
maximum request concurrency
pˉ,dˉ
Mean prompt and output lengths (tokens)
chunkpre
number of prefill tokens in one chunk
s
approximate token iterations per request
Appendix
Table B.1: Key quantities and their notation as presented in the LLM Inference and Power Consumption model, Section 2 .
Figure B.1: Single model instance predictions across hardware choices. Across query length distributions, language models and serving framework configurations, our predictions closely track the actual measured values, demonstrating accuracy throughout the examined scenarios.
Figure 15
Quantity
H100
RTX 5090
RTX 8000
V100
Power consumption at concurrency b , P(b)
0.9
2.5
0.7
2.5
Power consumption at query arrival rate λ , P(λ)
5.3
6.9
13.3
8.8
Token iteration time, τ(b)
3.3
5.4
5.0
6.3
Request residence time, T(b)
5.5
6.2
11.1
10.5
Request Throughput, μ(b)
5.6
6.7
13.5
9.0
Request Concurrency, b(λ)
10.6
9.8
30.1
31.0
Appendix
Table B.2: Single model instance prediction errors across all evaluated models and workloads on hardware setups. Mean Absolute Percentage Error (MAPE) presented for each case.
Mean-power error (W)
Power std. dev. (W)
Response time t90 (s)
Controller
Median
90th pct.
Median
90th pct.
Median
90th pct.
Analytical model, open loop
34
131
27
136
6.5
16.5
PID feedback
14
85
108
194
12.0
18.6
Appendix
Table B.3: Comparison of open-loop analytical model based power control and PID feedback control across analyzed scenarios. Mean-power error is the absolute difference between average measured power and the target; power standard deviation is the standard deviation of measured power values; response time t90 denotes time to achieve 90% of the desired power reduction. PID controller demonstrates lower mean-power error, but at the cost of higher fluctuations and response times.
Figure B.4: Response timescales under query routing for power control, with respect to grid-service and curtailment requirements. The initial response time (in seconds) for query routing is fast enough to accommodate majority of the demand flexibility requirements, which can be further coupled with approaches like battery/UPS systems for fast response and sustained power control.
Program or Guideline
Response Requirement
Duration and Objectives
Grid services and demand response
ERCOT Fast Frequency Response ( ercot_operating_guide_2026 )
Full response within 15 cycles (0.25 s at 60 Hz).
Sustain for 15 min or until recall, whichever occurs first.
Meet deployment obligations within 10 min of dispatch.
Sustain until recall. Cumulative obligations are 24 h in December–March contract periods and 12 h in other terms.
Appendix
Table B.4: Grid-service requirements and contextual examples accompanying Figure B.4 . Holding capability, market commitments, cumulative deployment obligations, and demonstrated event durations are identified separately. Response speed alone does not establish service qualification or the ability to deliver the required magnitude.
Large language models (LLMs) and agentic AI systems are creating rapidly growing inference energy demands as model sizes grow and reasoning trajectories extend. While in practice, many queries do not require the capabilities of the largest available model, and routinely directing such queries to a high-capability model can introduce unnecessary, considerable computation and energy consumption. In this paper, we investigate whether adaptive routing across a heterogeneous pool of LLMs can reduce this energy burden without substantially compromising task performance. We design a language-model-based router that reads in each query and selects an answer model from a fixed candidate pool. The candidate models are first profiled through an offline tournament that records their correctness, latency, power, and GPU energy for each query. Using these measurements, the router is trained through supervised fine-tuning followed by group relative policy optimization (GRPO) with the tailored paradigms. Results demonstrate that learned routing can selectively allocate expensive model capacity based on query context and improve the accuracy-energy tradeoff in multi-LLM serving. Across seven benchmark tasks, we also observe a sharp accuracy-energy phase transition among routers, providing practical insights into improving energy efficiency while maintaining LLM performance.
Muhammad Abdur Rab Siddiqui, Daniela Rojas, Chen Yang +3
University of Alberta Edmonton, Alberta, Canada · University of California San Diego La Jolla, California, USA · Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China +1
Large language model (LLM) training is among the fastest-growing sources of electricity demand in modern data centers, and power availability is a primary bottleneck to continued AI infrastructure growth. Making the power consumption of these workloads flexible could unlock additional power for AI growth, limit increases in electricity prices, and improve the utilization of existing grid infrastructure. However, to realize this flexibility, we must first understand how the performance of training workloads changes when GPU power is reduced. This paper presents the first systematic characterization of job power elasticity (the sensitivity of throughput to power reductions) in LLM training. To quantify elasticity, we introduce the Power Flexibility Index (PFI), a normalized metric that quantifies the performance cost of power reductions and provides a control primitive for SLA-aware power flexibility. We collect data from 131 LLM training runs on H200 (plus 24 H200 validation runs and 34 matched H100 runs), including both dense and mixture-of-experts models, pretraining and fine-tuning tasks, and up to 32 GPUs. We find that LLM training jobs exhibit substantial but variable power elasticity, and we identify telemetry signals that predict PFI at runtime. Finally, we demonstrate that PFI-aware power allocation maximizes total tokens/second throughput under power constraints. Under a 30% power reduction, PFI-aware power allocation recovers ~1.5k tokens/s per job, 63% of the performance gap between an equal-weight allocation and an oracle with perfect information. Our results establish power elasticity as a measurable property of training jobs and provide a foundation for power-aware, grid-responsive AI infrastructure.
Philip Colangelo, Charles Dawson, Shayan Sengupta +2
Large language model (LLM) inference has become a dominant workload in modern data centers, driving significant GPU utilization and energy consumption. While prior systems optimize throughput and latency by batching, scheduling, and parallelism, they largely treat GPU power as a static constraint rather than a controllable resource. In this paper, we present a power-aware runtime for LLM serving, PALS, that treats GPU power caps as a first-class control knob and jointly optimizes them with software parameters such as batch size. The system combines lightweight offline power-performance models with a feedback-driven controller to select configurations that satisfy throughput targets while maximizing energy efficiency. We implement PALS within an existing LLM serving framework, vLLM, demonstrating that it requires no model retraining or API changes. Across multi-GPU systems and both dense and mixture-of-experts (MoE) models, PALS improves energy efficiency by up to 26.3%, reduces QoS violations by 4x to 7x under power constraints, and tracks dynamic power budgets. These results highlight the potential of integrating power control directly into LLM inference runtimes, enabling energy-proportional and grid-interactive AI systems.
Can Hankendi, Rana Shahout, Minlan Yu +1
Boston University · Electrical and Computer Engineering Department · Boston, MA, USA +3