The rapid growth of generative artificial intelligence (AI) has introduced unprecedented computational demands, driving significant increases in the energy footprint of data centers. However, existing power consumption data is largely proprietary and reported at varying resolutions, creating challenges for estimating whole-facility energy use and planning infrastructure. In this work, we present a methodology that bridges this gap by linking high-resolution workload power measurements to whole-facility energy demand. Using NLR's high-performance computing data center equipped with NVIDIA H100 GPUs, we measure power consumption of AI workloads at 0.1-second resolution for AI training, fine-tuning and inference jobs. Workloads are characterized using MLCommons benchmarks for model training and fine-tuning, and vLLM benchmarks for inference, enabling reproducible and standardized workload profiling. The dataset of power consumption profiles is made publicly available. These power profiles are then scaled to the whole-facility-level using a bottom-up, event-driven, data center energy model. The resulting whole-facility energy profiles capture realistic temporal fluctuations driven by AI workloads and user-behavior, and can be used to inform infrastructure planning for grid connection, on-site energy generation, and distributed microgrids.
Figures & tables
Acronym
Meaning
AI
Artificial Intelligence
CPU
Central Processing Unit
GPU
Graphics Processing Unit
GenAI
Generative Artificial Intelligence
HPC
High-Performance Computing
IT
Information Technology
List of Acronyms
Figure 1: Diagram illustrating the workflow and scaling from data center compute node to facility level 2 2 2 Photos by Agata Bogucka (NLR 104051), Werner Slocum (NLR 63014, 67310), and Joe DelNero (NLR 101334). .
Figure 2: Data Center Infrastructure Planning, Load Optimization for Energy Efficiency (DIPLOEE) model diagram.
Figure 3: DIPLOEE model details for inference data center use case.
Figure 4: Llama-2 70B fine-tuning power profiles. Replicate experiments using the same node count are shown as superimposed transparent curves with different colors. Each power profile aggregates power consumption across all CPU and GPU devices utilized by the workload.
Figure 5: Llama-2 70B fine-tuning energy metrics as a function of number of nodes. Replicate experiments are shown as red dots, vertically distributed with a box-and-whisker plot for each node value. Fitted curves are also presented as dashed lines.
Nodes
Mean Total Runtime (sec)
Mean Epoch Runtime (sec)
Mean Epoch Speedup
Number of Epochs
Number of Evaluations
2
5,152 ± 226
3,930 ± 61
0.00
0.91 ± 0.04
7.20 ± 0.45
4
3,171 ± 403
1,353 ± 178
2.90
1.18 ± 0.14
4.00 ± 0.71
8
2,090 ± 170
795 ± 50
1.70
1.51 ± 0.12
6.17 ± 0.75
16
1,798 ± 15
342 ± 3
2.32
2.64 ± 0.00
5.00 ± 0.00
Table 1: Summary of runtimes from six replicate Llama-2 70B fine-tuning workloads.
Figure 6: Stable Diffusion training power profiles. Replicate experiments using the same node count are shown as superimposed transparent curves with different colors. Each power profile aggregates power consumption across all CPU and GPU devices utilized by the workload.
Figure 7: Stable Diffusion training energy metrics as a function of number of nodes. Replicate experiments are shown as red dots, vertically distributed with a box-and-whisker plot for each node value. Fitted curves are also presented as dashed lines.
Nodes
Mean Total Runtime (sec)
Mean Epoch Runtime (sec)
Mean Epoch Speedup
Number of Epochs
Number of Evaluations
1
24,632 ± 120
62,669 ± 306
0.00
0.39 ± 0.00
5.00 ± 0.00
2
12,480 ± 1,576
31,751 ± 52
1.97
0.39 ± 0.05
5.00 ± 0.63
4
5,784 ± 792
12,262 ± 1,679
2.59
0.47 ± 0.00
3.00 ± 0.00
8
3,202 ± 421
5,091 ± 669
2.41
0.63 ± 0.00
2.00 ± 0.00
16
1,432 ± 149
2,277 ± 238
2.24
0.63 ± 0.00
1.00 ± 0.00
Table 2: Summary of runtimes from six replicate Stable Diffusion training workloads.
Figure 8: Llama-3 70B offline inference power profiles as a function of number of prompts and output token length. Median values are shown as lines, and 10-90 th percentiles as shaded regions.
Figure 9: Llama-3 70B offline inference energy metrics as a function of number of prompts and output tokens. Median values across experiments are shown as dots, and 10-90 th percentiles as a shaded regions.
Figure 10: Power series of Llama-3 70B online inference benchmark as a function of dataset, request rate and output token length, assuming a constant batch size of 1,000 prompts. Median values are shown as lines, and 10-90 th percentiles as shaded regions.
Figure 11: Llama-3 70B offline inference performance metrics as a function of dataset, request-rate (in log-scale) and output token length. Median values are shown as lines, and 10-90 th percentiles as shaded regions.
Figure 12: Llama-3 70B online inference energy metrics as a function of dataset, request-rate (in log-scale) and output token length. Median values are shown as lines, and 10-90 th percentiles as shaded regions.
Figure 13: Power series of Llama-3 70B online inference benchmark as a function of dataset, request rate and output token length, assuming a sustained request rate for 3 minutes.
Average Utilization
20%
40%
60%
80%
Power
Mean (MW)
2.38
3.56
4.74
5.91
Std Dev (MW)
0.85
1.59
1.91
1.78
Median (MW)
2.22
3.27
4.69
7.00
90th Percentile (MW)
3.53
6.40
7.08
7.12
Table 3: Colocation data center aggregate metrics across the whole year for different target utilization levels.
Figure 14: Facility-level power consumption profile distributions for a 10 MW colocation data center at varying utilization levels. 14(a) : by time of the day; 14(b) : by day of the week. Distributions were generated from year-long simulations at 1-minute timesteps. The solid lines represent the median value, and the shaded areas different percentiles.
Average Utilization
20%
40%
60%
80%
Power
Mean (MW)
0.26
0.39
0.53
0.66
Std Dev (MW)
0.06
0.11
0.15
0.13
Median (MW)
0.24
0.36
0.49
0.68
90th Percentile (MW)
0.35
0.58
0.80
0.80
Table 4: Inference data center aggregate metrics across the whole year for different target utilization levels.
Figure 15: Facility-level power consumption profile distributions for a 1 MW inference data center at varying utilization levels. 15(a) : by time of the day; 15(b) : by day of the week. Distributions were generated from year-long simulations at 1-minute timesteps. The solid lines represent the median value, and the shaded areas different percentiles.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 16: GPU power when stressed using gpu-burn [ 51 ] , median value, and 99% confidence bound (z score: 2.58). Mean value of 668.2 W ( ± 1.4 W std) after warm up of 150 seconds.
Figure 17: CPU power for a dense matrix multiplication kernel, median value, and 99% confidence bound (z score: 2.58). Mean value of 338.6 W ( ± 1.0 W std) after warm up of 11 seconds.
Figure 18: CPU (left) and GPU (right) idle power on a Kestrel GPU node. We show the linear regression for the CPU data.
Figure 19: Power consumption of a single node, four device HPL-NVIDIA run with two matrix solves.
The Ohio State University, Columbus, OH 43210, USA. · Meta Platforms Inc., Menlo Park, CA 94025, USA. · Zhejiang A&F University, Hangzhou, Zhejiang 311300, China.