AI inference clusters are increasingly constrained by instantaneous power, not just energy: grid operators condition new capacity on demand response, imposing time-varying power caps. Existing LLM serving systems optimize a static energy objective or shed fixed priority tiers under load; either way, goodput collapses when the power envelope moves. An LLM pipeline is not a uniform load: compute-bound prefill loses throughput almost linearly with GPU frequency, memory-bound answer decode sustains it down to 0.57× nominal, and reasoning's thinking phase couples KV-cache capacity to scheduling -- so a cap should be steered to where each watt costs the least performance. PowerSlider does so with a new Flex SLO contract that turns bounded user slack into an optimization constraint, prefill--think--answer disaggregation exposing per-stage frequency and KV control, and a Karush--Kuhn--Tucker (KKT) online solver re-solving within 7.7 ms of every cap change, backed by a consolidated fail-safe that power-gates drained instances when DVFS bottoms out on static power. On SGLang with production traces, \sys{} sustains 78.3% online goodput at a 30% cap reduction versus 47.6% for the best of five baselines (1.64×), holds latency-critical tails within 1.3× of nominal (baselines: 2.3--6×, up to 12×), and delivers 92% mean goodput through a replayed CAISO grid-emergency day bottoming at 0.41× (54% at the trough; every baseline below 7%).
Figures & tables
Fig. 1: Demand response is richer than a step cap. (a) California Independent System Operator (CAISO) net load during a grid emergencies; (b) Power caps under 5 min settlement interval in high-demand and typical day [ 18 ] . Black steps: base interruptable program (BIP) events, dispatched 30 min after the alert, deeper on the more severe day [ 55 ] . EEA2/3 are grid emergency levels.
Fig. 2: Load-duration trends show that peak power demand is driven by tail behavior: a small amount of service flexibility (5%) substantially reduces required peak power provisioning. We sweep arrival burstiness (larger coefficient of variation gives more power headroom) and quantify how different load-shedding rates flatten the peak. This burst-driven slack is the raw material PowerSlider converts into demand-response headroom (O1).
Fig. 3: Flexible SLOs enable graceful degradation across power-cap reductions. At full power both paradigms deliver the same total goodput. As the cap tightens, the rigid policy protects LC requests by rejecting BE traffic and still incurs LC violations; the flexible-SLO policy allows controlled LC degradation, preserving BE service and higher overall goodput. Rigid tiers waste the slack of under a cap; Flex tier exists to spend it.
Fig. 4: Measured throughput, power, energy, and latency vs. GPU frequency for five batch sizes (BS) on A100-80GB. The frequency at which throughput saturates shifts with batch size, and BS < 128 (memory-bandwidth-bound) saturates within the DVFS range, above 810 MHz, saturation happens lower for lower BS; from BS>128 (compute-bound) the knee lies beyond the maximum 1,410 MHz, so any frequency decrease costs a proportional throughput drop. PowerSlider exploits this differential: it reduces batch-constrained answer-stage frequency first, shedding power at minimal performance cost. This stage and batch-related asymmetry is the hardware lever PowerSlider ’s solver pulls whenever the cap moves (O2).
Dimension
Examples
Stage
Prefill / Think / Answer
SLO class
LC / Flex / BE
GPU allocation
ki GPUs per stage-class
Frequency
210–1410 MHz per stage-class
KV chunk size
e.g., 512 / 1024 / 2048 tokens
Model / TP setting
Llama-70B, CodeLlama-34B, Qwen-32B; TP-4
TABLE II: The per-cap configuration space PowerSlider must navigate.
Fig. 5: Reasoning workloads have shorter prompts (left) but far longer, heavier-tailed output distributions (right), defeating the output-length prediction that prior resource-allocation and routing schemes depend on. These tails are why PowerSlider admits reasoning work by observation rather than by length prediction (O3), which matters in the near-saturation regime.
Component
Cadence
Crit. path?
Overhead
PSRoute (dispatch/admission)
per request
yes
virtual-queue lookup
PSSched: pool reallocation
5 min + 10 s refine
no
drain-bounded
↪ vote–commit DVFS
50–100 ms
no
one NVML call/GPU
PSOpt (KKT solve)
DR event / hourly
no
under 100 ms
TABLE III: PowerSlider control components
Fig. 6: Architecture overview of PowerSlider . Dark-headed boxes are PowerSlider ’s components; the gray bar is the existing stack (SGLang runtime, GPU driver, NVML DVFS). Three SLO classes flow left-to-right through the Prefill–Think–Answer pipeline; each pool forms its own frequency domain ( fP,fT,fA ), and the KV cache is partitioned per class (colors match the user chips). PSOpt turns the grid cap pmax(t) into per-stage allocations, frequencies, and KV budgets; PSSched actuates them; PSRoute admits and dispatches per request.
Fig. 7: Normalized online and BE goodput vs. power-cap reduction for (top) reasoning and (bottom) non-reasoning workloads. PowerSlider sustains 1.64× the best baseline’s goodput at a 30% reduction (top) and 100% goodput for all classes through a 60% reduction (bottom). Main end-to-end result: PowerSlider widens the demand-response operating envelope, making the 5–20% reductions grid programs request (§ II-A ) essentially free.
Fig. 8: P90 TTLT and TTFAT vs. power-cap reduction for the LC and Flex classes. PowerSlider preserves near-nominal latency up to a 60% power-cap reduction, while the best-performing baseline incurs up to 12× higher LC latency and 18× higher Flex latency at a 50% reduction. The Flex disparity is larger because baselines do not isolate execution stages and prioritize LC at Flex’s expense. Near-nominal tails are what keep demand-response participation compatible with latency SLOs.
Fig. 9: Ablation of PowerSlider components ( -KKT , -PTA , -Flex ) for LC (top) and Flex (bottom) classes; TTFAT (left) and TTLT (right). The full design minimizes P90 latency across all regimes. Annotations indicate % latency increase over PowerSlider at a 30% power-cap reduction. Absorbing a cap needs all three mechanisms; no single knob suffices.
Fig. 10: A real grid-emergency day replayed as the 24 h DR day. The cap schedule (gray) replays the Fig. 1 emergency day (Sep 6; trough 0.41 × ) [ 23 ] . (a) PowerSlider holds ≥98% online goodput all day; every baseline collapses to 0–6.6% at the trough and none recovers in-trace (post-22.5 h windows censored). (b) Per-stage GPU allocation under drain-before-reassign (43 reconfigurations).
Fig. 11: Power tracking (a) and Flex contract validation (b, c) on the compressed 24 h DR day. (a) Delivered power on the replayed day in Fig. 1 : PowerSlider stays under the binding cap throughout; (b) Flex E2E latency CDFs (0–60% reductions; Lc=128.9 s) stay inside the (αc=3,ρc=0.3) envelope at every depth; (c) POLCA breaches it beyond 30% power reduction
Fig. 12: Top: PSOpt solve time scales near-linearly with group count, staying inside one 100 ms DVFS interval. Bottom: per-phase P50 latency, PD vs. PTA: under a 40% cap PD develops decode queueing while PTA keeps it near zero ( 12.5× lower end-to-end latency). Millisecond re-solves track the moving cap; stage isolation keeps it from becoming queueing collapse.
Model
TTF(A)T
TTLT/TBT
Dataset
CodeLlama-34B
72.0s
115.5s
Magpie
QWen-32B
36.0s
121.0s
S1K
DS-R1-Llama-70B
5.0s
0.50s
Azure-Code
QWen-14B
0.30s
0.1s
Azure-Chat
TABLE IV: Workloads in evaluation and latency requirements.
Name
Arch
DVFS Policy
Scheduler
Multi-SLO
B1 Uniform
PD disagg
Uniform
Overlap-KV JSQ
Single class
B2 POLCA
Collocated
Priority-aware
Mixed-JSQ
HP/LP binary
B3 SplitWise+
PD disagg
Stage-aware
Overlap-KV JSQ
+Priority
B4 DynamoLLM+
Collocated
+Uniform
Prediction rtr.
SLO-aware
B5 SLOs-Serve+
Collocated
+Uniform
ProMax
LC/BE priority
PowerSlider
PTA disagg
KKT solver
Multi-SLO JSQ
LC/Flex/BE
TABLE V: Baseline configurations.
Fig. 13: Latency degradation from prediction-based misrouting under increasing per-instance offered load (128-instance cluster). At low load ( ≤ 2 rps per instance) misrouting has negligible impact; near saturation ( ∼ 5 rps per instance), TTFT mean degrades > 100% and P99 exceeds 1,000%, while end-to-end and TTFAT remain buffered until saturation.
Fig. 14: Robustness of the PTA benefit across workload distributions. Panels (a)–(c) increase the T/A ratio from 0.5:1 to 16:1 (average prefill 512); panel (d) uses average prefill 2048. The advantage grows with T/A ratio (49% TBAT reduction at 0.5:1 up to 71% at 16:1 at high load) because longer thinking chains cause more severe batch interference under PD and colocated serving, which PTA eliminates by isolating answer decode onto dedicated workers.
Fig. 15: Per-request response times for LC, Flex, and BE classes as the power cap steps from 90% to 60% of nominal at t=750 s on a 512-GPU cluster. PowerSlider completes the most requests across all classes with stable response times; baselines either shed BE traffic or accumulate unbounded queueing.
Fig. 16: Sensitivity of PowerSlider ’s Flex SLO parameters on LC goodput, normalized to POLCA. Top: varying the relaxation ratio α with ρ=0.2 . Bottom: varying the violation-budget fraction ρ with α=3.0 . At a 60% power-cap reduction, PowerSlider delivers up to 1.85× the goodput of POLCA and remains robust over a wide range of α and ρ . The contract that funds PowerSlider ’s power headroom does not need careful tuning to deliver it.
Fig. 17: Workload-mix sensitivity under a dynamic user mix (measured). (a) The LC/Flex/BE arrival mix drifts from 5% to 80% Flex within a single run (QPS 14, CoV 7.5 bursty arrivals, 64 GPUs); the gray line is the arrival rate. (b) Online goodput per mix segment at a 30% power-cap reduction ( PowerSlider also shown at 50%): PowerSlider holds 95–100% goodput across the entire drift (95.3% at 80% Flex, shedding BE admissions when the cap makes full service infeasible), while POLCA collapses to 10% at 80% Flex. (c) Mix extremes – LC-dominant (90/5/5), default (30/30/40), Flex-dominant (10/80/10) – at 30% (solid) and 50% (faded) reductions.
Fig. 18: Energy–goodput Pareto and economic break-even under power caps (measured). (a) Cluster energy per generated token vs. online goodput, one point per system and cap depth (0–60%; ideal is bottom-right). PowerSlider alone holds the ideal corner at every depth (0.34–0.44 J/token at 97.9–100% goodput; inset), while at a 60% cap the baselines pay ≈ 3 × the energy per useful token at 12–23% goodput. (b) Net benefit per 2 h DR event vs. cap depth; PowerSlider is net-positive at every depth where the baselines turn negative (§ VII-G ).
Program
Resp. req.
PowerSlider path
Feasible
Freq. regulation (AGC)
seconds
DVFS band ( < 1 s)
✓ ( ≤ 20%)
Spinning reserve
10 min
reallocation epoch
✓ (full depth)
Emergency DR
30 min
reallocation epoch
✓ (full depth)
Day-ahead / economic
hours
capacity planning
✓
Stacked (reg. + reserve)
both
both bands
✓
TABLE VI: Grid-program feasibility given PowerSlider ’s actuation paths (7.7 ms solve; < 1 s DVFS band; 5 min reallocation epoch).
Large language model (LLM) inference has become a dominant workload in modern data centers, driving significant GPU utilization and energy consumption. While prior systems optimize throughput and latency by batching, scheduling, and parallelism, they largely treat GPU power as a static constraint rather than a controllable resource. In this paper, we present a power-aware runtime for LLM serving, PALS, that treats GPU power caps as a first-class control knob and jointly optimizes them with software parameters such as batch size. The system combines lightweight offline power-performance models with a feedback-driven controller to select configurations that satisfy throughput targets while maximizing energy efficiency. We implement PALS within an existing LLM serving framework, vLLM, demonstrating that it requires no model retraining or API changes. Across multi-GPU systems and both dense and mixture-of-experts (MoE) models, PALS improves energy efficiency by up to 26.3%, reduces QoS violations by 4x to 7x under power constraints, and tracks dynamic power budgets. These results highlight the potential of integrating power control directly into LLM inference runtimes, enabling energy-proportional and grid-interactive AI systems.
Can Hankendi, Rana Shahout, Minlan Yu +1
Boston University · Electrical and Computer Engineering Department · Boston, MA, USA +3
Large language model (LLM) training is among the fastest-growing sources of electricity demand in modern data centers, and power availability is a primary bottleneck to continued AI infrastructure growth. Making the power consumption of these workloads flexible could unlock additional power for AI growth, limit increases in electricity prices, and improve the utilization of existing grid infrastructure. However, to realize this flexibility, we must first understand how the performance of training workloads changes when GPU power is reduced. This paper presents the first systematic characterization of job power elasticity (the sensitivity of throughput to power reductions) in LLM training. To quantify elasticity, we introduce the Power Flexibility Index (PFI), a normalized metric that quantifies the performance cost of power reductions and provides a control primitive for SLA-aware power flexibility. We collect data from 131 LLM training runs on H200 (plus 24 H200 validation runs and 34 matched H100 runs), including both dense and mixture-of-experts models, pretraining and fine-tuning tasks, and up to 32 GPUs. We find that LLM training jobs exhibit substantial but variable power elasticity, and we identify telemetry signals that predict PFI at runtime. Finally, we demonstrate that PFI-aware power allocation maximizes total tokens/second throughput under power constraints. Under a 30% power reduction, PFI-aware power allocation recovers ~1.5k tokens/s per job, 63% of the performance gap between an equal-weight allocation and an oracle with perfect information. Our results establish power elasticity as a measurable property of training jobs and provide a foundation for power-aware, grid-responsive AI infrastructure.
Philip Colangelo, Charles Dawson, Shayan Sengupta +2
The energy cost of Large Language Model (LLM) inference is rapidly becoming a barrier to sustainable and scalable deployment. Although modern serving architectures expose distinct prefill and decode behaviors, existing systems fail to exploit these phase differences for energy-efficient serving under strict latency SLOs. This paper introduces VoltanaLLM, the first system that explicitly targets and reduces the energy bloat in modern prefill-decode (P/D) disaggregated LLM serving. Guided by a control-theory perspective, VoltanaLLM separates two levers: per-instance operating-point selection (GPU frequency per iteration) and system-level state-space routing of requests. We empirically observe that LLM inference exhibits a U-shaped energy-frequency curve creating "sweet spots" that depend on phase behavior and load. VoltanaLLM exploits this by combining phase-specific, iteration-level frequency selection driven by a lightweight, online-adaptive latency predictor, with a decode state-space guided router that avoids architectural granularity-induced inefficiencies, all while meeting desired SLOs. We implement VoltanaLLM using SGLang and evaluate it across multiple models and real-world workloads. Our results show VoltanaLLM reduces end-to-end energy by up to 36.3% versus a static max-frequency baseline while maintaining high SLO attainment, and generalizes to newer GPUs. These results point to sustainable LLM serving via phase-aware, iteration-level frequency selection coupled with architecture-aware routing. Source code is available in https://github.com/Supercomputing-System-AI-Lab/VoltanaLLM.
Jiahuan Yu, Aryan Taneja, Junfeng Lin +1
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Champaign, USA · Department of Precision Instrument, Tsinghua University, Beijing, China