The rise of Artificial Intelligence and Large Language Models is driving increased GPU usage in data centers for complex training and inference tasks, impacting operational costs, energy demands, and the environmental footprint of large-scale computing infrastructures. This work addresses the online scheduling problem in GPU datacenters, which involves scheduling tasks without knowledge of their future arrivals. We focus on two objectives: minimizing GPU fragmentation and reducing power consumption. GPU fragmentation occurs when partial GPU allocations hinder the efficient use of remaining resources, especially as the datacenter nears full capacity. A recent scheduling policy, Fragmentation Gradient Descent (FGD), leverages a fragmentation metric to address this issue. Reducing power consumption is also crucial due to the significant power demands of GPUs. To this end, we propose PWR, a novel scheduling policy to minimize power usage by selecting power-efficient GPU and CPU combinations. This involves a simplified model for measuring power consumption integrated into a Kubernetes score plugin. Through an extensive experimental evaluation in a simulated cluster, we show how PWR, when combined with FGD, achieves a balanced trade-off between reducing power consumption and minimizing GPU fragmentation.
Figures & tables
GPU Request per Task
0
(0, 1)
1
2
4
8
Task Population (%)
13.3
37.8
48.0
0.2
0.2
0.5
Total GPU Reqs. (%)
0
28.5
64.2
0.5
1.0
5.8
TABLE I : Distribution of tasks in the Default trace.
GPU model
Amount
Power idle (W)
TDP (W)
V100M16
195
30
300
V100M32
204
30
300
P100
265
25
250
T4
842
10
70
A10
2
30
150
G2 (A10)
4,392
30
150
TABLE II : GPU models in the trace data.
Fig. 1 : FGD EOPC (in MW) for workloads from the Default trace, with stacked CPU and GPU components. The dashed line shows the fraction of GPU power (see right-hand y-axis).
Fig. 2 : Power savings (in percentage w.r.t. FGD, top plot ) and GRAR scores ( bottom plot ) measured for PWR and its linear combinations with FGD, workloads from Default trace.
Fig. 3 : Power savings with workloads from the Default trace.
Fig. 4 : Power savings with sharing-GPU workloads – case in which sharing-GPU tasks request 100% of GPU resources.
Fig. 5 : Power savings with multi-GPU workloads.
Fig. 6 : Power savings with constrained-GPU workloads.
Fig. 7 : GRAR with workloads from the Default trace.
Online resource allocation studies how to assign scarce capacity to arriving requests under uncertainty while balancing reward, cost, fairness, and feasibility.
Large language model (LLM) inference has become a dominant workload in modern data centers, driving significant GPU utilization and energy consumption. While prior systems optimize throughput and latency by batching, scheduling, and parallelism, they largely treat GPU power as a static constraint rather than a controllable resource. In this paper, we present a power-aware runtime for LLM serving, PALS, that treats GPU power caps as a first-class control knob and jointly optimizes them with software parameters such as batch size. The system combines lightweight offline power-performance models with a feedback-driven controller to select configurations that satisfy throughput targets while maximizing energy efficiency. We implement PALS within an existing LLM serving framework, vLLM, demonstrating that it requires no model retraining or API changes. Across multi-GPU systems and both dense and mixture-of-experts (MoE) models, PALS improves energy efficiency by up to 26.3%, reduces QoS violations by 4x to 7x under power constraints, and tracks dynamic power budgets. These results highlight the potential of integrating power control directly into LLM inference runtimes, enabling energy-proportional and grid-interactive AI systems.
Can Hankendi, Rana Shahout, Minlan Yu +1
Boston University · Electrical and Computer Engineering Department · Boston, MA, USA +3
The rapid expansion of artificial intelligence (AI) infrastructure is driving unprecedented growth in electricity demand from data centers. Traditional power-system planning treats large computing facilities as inflexible peak loads, leading to costly infrastructure upgrades and long delays in grid interconnection. Recent work has shown that AI clusters can reduce electricity consumption during peak demand through software-based workload orchestration. This article explores how modern GPU-based AI data centers can operate as grid-interactive assets that respond dynamically to power system conditions. We describe an architecture integrating grid signals, workload scheduling, and power telemetry for fine-grained cluster power control. Experimental results from a real-world deployment on a 130 kW GPU cluster demonstrate multiple forms of flexibility, including rapid load reduction, sustained curtailment, and carbon-aware operation while preserving service levels for priority jobs. We further demonstrate performance-aware load shifting across geographically distributed clusters, enabling workloads to migrate toward regions with lower grid stress. Together, these capabilities transform AI infrastructure from static electricity consumers into flexible resources that support grid reliability, accelerate interconnection, and improve computing sustainability.
Chris Williams, Philip Colangelo, Ayse Coskun +15
Emerald AI · Electric Power Research Institute (EPRI) · National Grid +2
Modern AI model training imposes unprecedented computational demands, making it a key contributor to datacenter energy consumption. Yet a significant fraction of the energy consumed during training does not translate to useful computation due to bottlenecks throughout the training pipeline. We present PAFT, a phase-aware, dynamically adaptable GPU frequency tuning system that reduces energy consumption of training workloads with minimal performance overhead. The key insight behind PAFT is that bottlenecks represent an energy optimization opportunity, rather than purely a performance problem: when GPUs are bound to stall, PAFT opportunistically reduces their clock frequencies to match the pace of bottlenecked devices, saving energy without impacting execution time. PAFT achieves this by continuously monitoring pipeline behavior and applying fine-grained frequency adjustments, adapting to workload and system changes. Experiments conducted on twelve widely used models show that PAFT consistently outperforms all baselines, achieving energy savings of up to 46% with an average overhead of 4%.