Creating Power Flexibility in Datacenters via Query Routing

Shivam Patel; Bingqing Chen; Jared Fernandez; Guannan Qu; Gauri Joshi

cs.AINov 3, 2026

Creating Power Flexibility in Datacenters via Query Routing

Authors: Shivam Patel, Bingqing Chen, Jared Fernandez, Guannan Qu, Gauri Joshi

Organizations: Carnegie Mellon University · Bosch Center for AI · Cornell University

Abstract

The growing energy requirements of large language model inference highlight the need for datacenters to support safe grid operation by providing power demand flexibility, while maintaining service for interactive workloads. This work investigates query routing as a mechanism to enable such capability by dynamically allocating queries across different language model instances without deferring requests. We develop a general analytical framework that connects query arrival rates and sequence length distributions to token iteration time, request residence time, serving capacity, and power consumption of model instances. Leveraging this framework, we characterize the feasible query routing allocations and the corresponding power demand flexibilities, enabling the selection of routing strategies to achieve desired power consumption behavior. We validate our framework across four GPU platforms on language models ranging from 1B to 70B parameters, obtaining 2.8% mean absolute percentage error in estimating achievable power flexibility. We evaluate our approach on a real trace-driven simulation of a 10,000 H100 GPUs datacenter, exhibiting flexibility up to 16% of total power consumption, with 90% of the desired power reduction achieved within approximately 10 seconds. These results establish query routing as an effective mechanism for datacenter power demand flexibility and demonstrate the broader applicability of our proposed framework to datacenter level planning and orchestration.

Figures & tables

Figure 1: Power demand flexibility through query routing. Routing queries across large and small language models enables datacenters to adjust power demand for safe grid operation without dropping or deferring inference requests.

Figure 2: Measured and predicted mean token iteration times (TBT) as a function of number of concurrent requests across models and query length distributions. The linear estimator in eq. 1 accurately captures the observed values (

R^{2}>0.99

across all configurations). Larger models exhibit higher TBTs, and workloads with longer requests incur additional attention computation overhead.

Figure 3: Predicted versus measured mean request residence time across configurations. The analytical estimator ( eq. 2 ) captures the increase in request duration with concurrency, achieving an overall

5.5\%

MAPE, demonstrating reliable estimation of mean request residence time.

Figure 4: Predicted and measured request throughput as a function of request concurrency across configurations. Throughput increases with concurrency but exhibits diminishing returns as the token iteration times grow. The analytical estimator ( eq. 3 ) achieves an overall MAPE of

5.8\%

, accurately capturing the concurrency-dependent request throughput of model instances.

Figure 5: Estimating request concurrency ( eq. 4 ) from query arrival rate, demonstrating an overall MAPE of

9.2\%

over feasible operating points. The predicted request concurrency count connects the query arrival rate to the model instance’s inference and power consumption.

Figure 6: Measured and predicted mean GPU board power as a function of request concurrency across configurations. Power consumption increases sharply when transitioning from idle to active inference, followed by a more gradual increase as concurrency grows. The diminishing incremental power at higher concurrency suggests opportunities for reducing aggregate power consumption through request consolidation across model instances.

Figure 7: Capacity-feasible routing decisions (left, eq. 10 ) and corresponding total-power consumption (right, eq. 11 ) across query arrival rates and model–workload configurations. Shaded regions show analytical predictions, bars depict measured ranges.

Figure 8: Total power consumption via query routing for the three Llama-8B and single Llama-1B model instances with queries sampled from ShareGPT. Left: measured and predicted total power ( eq. 11 ) versus the fraction of flexible requests ( $f$ ) assigned to the small model instance, with varying query arrival rates. Right: measured and predicted power flexibility range ( eq. 12 ) over feasible allocations.

Figure 9: Datacenter power demand flexibility over week-long Azure LLM workload trace. Top: Baseline facility power and minimum achievable power under varying fractions of flexible queries. Bottom: Distribution and hourly variation of available power curtailment, highlighting its dependence on query workload.

Figure 10: Datacenter power control during a four-hour curtailment event. Top: Sustained power reductions achieved through flexible query routing. Bottom: Event boundaries showing fast transitions between operating points in the order of seconds.

Figure 11: Impact of GPU availability and resource sharing on power demand flexibility. Top: Achievable power curtailment under reduced GPU availability and the resulting utilization during scheduled sharing. Bottom: Daily power flexibility with GPU sharing, demonstrating resilience to moderate resource constraints.

Domain	Application
Serving design	Model instance pool sizing, deployment comparisons, demand growth headroom.
Power Control	Feasible targets, sustained power curtailments, and transition planning.
Energy coordination	Joint inference/training budgets, storage support, and cost or carbon objectives.
Broader routing	Allocation across execution choices or locations, subject to eligibility constraints.

Table 1: Broader applications of our presented framework.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$\lambda$	Query arrival rate (requests/s)
$b$	request concurrency
$b_{\max}$	maximum request concurrency
$\bar{p},\bar{d}$	Mean prompt and output lengths (tokens)
$\text{chunk}_{\mathrm{pre}}$	number of prefill tokens in one chunk
$s$	approximate token iterations per request

Appendix

Table B.1: Key quantities and their notation as presented in the LLM Inference and Power Consumption model, Section 2 .

Figure B.1: Single model instance predictions across hardware choices. Across query length distributions, language models and serving framework configurations, our predictions closely track the actual measured values, demonstrating accuracy throughout the examined scenarios.

Quantity	H100	RTX 5090	RTX 8000	V100
Power consumption at concurrency $b$ , $P(b)$	0.9	2.5	0.7	2.5
Power consumption at query arrival rate $\lambda$ , $\widehat{P}(\lambda)$	5.3	6.9	13.3	8.8
Token iteration time, $\tau(b)$	3.3	5.4	5.0	6.3
Request residence time, $\widehat{T}(b)$	5.5	6.2	11.1	10.5
Request Throughput, $\widehat{\mu}(b)$	5.6	6.7	13.5	9.0
Request Concurrency, $\widehat{b}(\lambda)$	10.6	9.8	30.1	31.0

Appendix

Table B.2: Single model instance prediction errors across all evaluated models and workloads on hardware setups. Mean Absolute Percentage Error (MAPE) presented for each case.

Controller	Median	90th pct.	Median	90th pct.	Median	90th pct.
	Mean-power error (W)		Power std. dev. (W)		Response time $t_{90}$ (s)
Analytical model, open loop	34	131	27	136	6.5	16.5
PID feedback	14	85	108	194	12.0	18.6

Appendix

Table B.3: Comparison of open-loop analytical model based power control and PID feedback control across analyzed scenarios. Mean-power error is the absolute difference between average measured power and the target; power standard deviation is the standard deviation of measured power values; response time

t_{90}

denotes time to achieve 90% of the desired power reduction. PID controller demonstrates lower mean-power error, but at the cost of higher fluctuations and response times.

Figure B.4: Response timescales under query routing for power control, with respect to grid-service and curtailment requirements. The initial response time (in seconds) for query routing is fast enough to accommodate majority of the demand flexibility requirements, which can be further coupled with approaches like battery/UPS systems for fast response and sustained power control.

Program or Guideline	Response Requirement	Duration and Objectives
Grid services and demand response
ERCOT Fast Frequency Response ( ercot_operating_guide_2026 )	Full response within 15 cycles (0.25 s at 60 Hz).	Sustain for 15 min or until recall, whichever occurs first.
ERCOT RRS, high-set under-frequency relay ( ercot_operating_guide_2026 ; ercot_protocols_2026 )	Interrupt load within 30 cycles (0.5 s).	At least 1 h capability for this resource category following restoration instructions.
ERCOT Regulation ( ercot_protocols_2026 )	Respond to control signals within 5 s; deliver capacity within 5 min.	At least 30 min duration capability with continuous signal tracking.
PJM Regulation ( pjm_manual12_2026 ; pjm_manual11_2026 )	Respond immediately and reach bid capability within 5 min.	Half-hour commitments with continuous tracking and performance scoring.
ERCOT Emergency Response, ERS-10 ( ercot_protocols_2026 )	Meet deployment obligations within 10 min of dispatch.	Sustain until recall. Cumulative obligations are 24 h in December–March contract periods and 12 h in other terms.

Appendix

Table B.4: Grid-service requirements and contextual examples accompanying Figure B.4 . Holding capability, market commitments, cumulative deployment obligations, and demonstrated event durations are identified separately. Response speed alone does not establish service qualification or the ability to deliver the required magnitude.

Data Center Energy Management

See all 22 papers

Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving
Sep 19, 2026Muhammad Abdur Rab Siddiqui, Daniela Rojas, Chen Yang +3Energy-Aware LLM Inference LLM Routing
University of Alberta Edmonton, Alberta, Canada · University of California San Diego La Jolla, California, USA · Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China +1
Characterizing Job Power Elasticity for Power-Flexible AI Training
Sep 10, 2026Philip Colangelo, Charles Dawson, Shayan Sengupta +2Energy-Efficient ML Efficient Language Model Training
Emerald AI
PALS: Power-Aware LLM Serving for Mixture-of-Experts Models
May 20, 2026Can Hankendi, Rana Shahout, Minlan Yu +1LLM Inference Efficiency LLM Serving
Boston University · Electrical and Computer Engineering Department · Boston, MA, USA +3

Sep 19, 2026J/K move · Enter open · S save

Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving

Muhammad Abdur Rab Siddiqui, Daniela Rojas, Chen Yang, Wenqi Cui+2

University of Alberta Edmonton, Alberta, Canada · University of California San Diego La Jolla, California, USA · Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China+1

Energy-Aware LLM Inference LLM Routing LLM Serving Energy-Efficient ML

Open paper AskPDF arXiv

Large language models (LLMs) and agentic AI systems are creating rapidly growing inference energy demands as model sizes grow and reasoning trajectories extend. While in practice, many queries do not require the capabilities of the largest available model, and routinely directing such queries to a high-capability model can introduce unnecessary, considerable computation and energy consumption. In this paper, we investigate whether adaptive routing across a heterogeneous pool of LLMs can reduce this energy burden without substantially compromising task performance. We design a language-model-based router that reads in each query and selects an answer model from a fixed candidate pool. The candidate models are first profiled through an offline tournament that records their correctness, latency, power, and GPU energy for each query. Using these measurements, the router is trained through supervised fine-tuning followed by group relative policy optimization (GRPO) with the tailored paradigms. Results demonstrate that learned routing can selectively allocate expensive model capacity based on query context and improve the accuracy-energy tradeoff in multi-LLM serving. Across seven benchmark tasks, we also observe a sharp accuracy-energy phase transition among routers, providing practical insights into improving energy efficiency while maintaining LLM performance.

Abstract

Data Center Energy Management

Explore similar work

Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving

Characterizing Job Power Elasticity for Power-Flexible AI Training

PALS: Power-Aware LLM Serving for Mixture-of-Experts Models