The growing scale and complexity of modern data centers present major challenges in balancing energy efficiency with outage risk. Although Deep Reinforcement Learning (DRL) shows strong potential for intelligent control, its deployment in mission-critical systems is limited by data scarcity and the lack of real-time pre-evaluation mechanisms. This paper introduces the Dual-Loop Control Framework (DLCF), a digital twin-based architecture designed to overcome these challenges. The framework comprises three core entities: the physical system, a digital twin, and a policy reservoir of diverse DRL agents. These components interact through a dual-loop mechanism involving real-time data acquisition, data assimilation, DRL policy training, pre-evaluation, and expert verification. Theoretical analysis shows how DLCF can improve sample efficiency, generalization, safety, and optimality. Leveraging DLCF, we implemented the DCVerse platform and validated it through case studies on a real-world data center cooling system. The evaluation shows that our approach achieves up to 4.09% energy savings over conventional control strategies without violating SLA requirements. Additionally, the framework improves policy interpretability and supports more trustworthy DRL deployment. This work provides a foundation for reliable AI-based control in data centers and points toward future extensions for holistic, system-wide optimization.
Figures & tables
Figure 1: Three key aspects should be considered in algorithm design: system characteristics, optimization objectives, and algorithmic properties.
Figure 2: A representative taxonomy of deep reinforcement learning algorithms.
Figure 3: Dual-loop control framework for the DRL deployment. Dual-loop control framework: (1)-(4). Single-loop control framework: (5)-(6).
DLCF
SLCF
Online model-based
Online model-free
Offline model-based
Vanilla offline
Sample Efficiency
High
Medium
Low
-
-
Generalization
High
-
-
Medium
Low
Optimality
High
Low
High
Low
Low
Safety
High
Low
Low
Low
Low
Interpretability
High
Low
Low
Low
Low
Table 1: Comparison of DLCF and SLCF across key aspects.
Figure 4: Real-world data center cooling layout comprising three loops: condenser water loop, chilled water loop, and air loop.
Figure 5: Digital twin layout. (a) Cooling system. (b) Data hall.
Figure 6: Normalized digital twin cooling system calibration results. (a) Total chilled water pump power. (b) Total chiller power. (c) Total cooling plant power.
Feature
MAPE
CRAH averaged return air temperature
0.97%
Chilled water return temperature
0.50%
Chiller plant total cooling load
1.58%
Total chilled water pump power
2.89%
Total chiller power
1.11%
Total cooling plant power
1.86%
Table 2: Model prediction accuracy for key features.
Figure 7: Comparisons of various control strategies. (a) Normalized total power consumption. (b) SLA: ITE inlet dry-bulb temperature. (c) SLA: ITE inlet relative humidity.
Figure 8: Control actions of various control strategies. (a) CRAH air supply temperature. (b) CRAH air flow rate. (c) Chilled water supply temperature.
Figure 9: Normalized power consumption of cooling system equipment. (a) CRAHs. (b) CHW pumps. (c) Chillers.
Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.
This study proposes a scalable Digital Twin framework for energy optimization in data centers.The framework integrates IoT-based data acquisition, cloud computing, and machine learning techniques to enable real-time monitoring, forecasting, and intelligent energy management. A controlled small-scale data center environment was developed to monitor variables such as power consumption, temperature, and computational workload. Long Short-Term Memory (LSTM) models were employed to predict energy demand and support operational decision-making. Experimental results demonstrated improvements in energy efficiency, including reductions in power consumption and enhancements in Power Usage Effectiveness (PUE). Despite being evaluated in a constrained environment, the proposed framework demonstrates strong potential as a scalable and cost-effective solution for sustainable data center management.
Raphael Hendrigo de Souza Gonçalves, Wendel Marcos dos Santos
Federal University of São João del-Rei (UFSJ) · Federal Institute of Education, Science and Technology of São Paulo (IFSP)
Eco-friendly energy management for artificial intelligence data centers (AIDCs) is crucial because of the significant increase in energy consumption-induced carbon emissions from AIDCs resulting from the rapid expansion of AI applications. This paper proposes a hierarchical carbon-aware multi-agent reinforcement learning (CA-MARL) framework for robust and efficient operations of AIDCs under uncertainties while ensuring low-carbon operation of power distribution systems. The framework comprises a workload manager (WM) agent and multiple local AIDC agents trained using a multi-agent transformer method, corresponding to a global AIDC aggregator and a local AIDC operator, respectively. Leveraging AIDC operation data along with nodal carbon intensity (NCI) calculated from the carbon emission flow-integrated distribution system operator problem, the WM agent spatially allocates AI training and inference jobs among all AIDCs. Based on the jobs allocated from the WM agent and NCI information, each AIDC agent schedules economical and eco-friendly operations of the AIDC by performing the following tasks: i) temporal shifting of training jobs, ii) spatial allocation of training graphics processing unit (GPU) blocks and inference GPUs within the AIDC, and iii) control of the supply air temperature of the cooling system. The effectiveness of the proposed framework was assessed using an IEEE 33-node power distribution system.
Hyunsoo Lee, Panggah Prabawa, Dae-Hyun Choi +1
LG Electronics, 17709, Pyeongtaek, Gyeonggi-do, Republic of Korea · School of Electrical and Electronics Engineering, Chung-Ang University, Seoul 06974, Republic of Korea · Department of Electrical and Computer Engineering, Korea University, Seoul 02841, Republic of Korea
Data centers are increasingly optimized by artificial intelligence and, at the same time, increasingly loaded by it. The literature treats these as two unrelated problems: control studies model workload as an exogenous arrival process, while sustainability studies model infrastructure as a fixed multiplier. We screen roughly 194 papers retrieved through a documented protocol, code 63 of them, and report what the coding shows. Of 28 primary control-oriented studies, 18 are validated in simulation alone and 5 reach physical hardware or a production facility; none account for water withdrawal, and none account for embodied carbon. Reported savings intervals across four technique families overlap almost completely, which means the field cannot presently rank its own methods. Ten recurring gaps are scored for consequence and tractability, and we set out CLEAR-DC, a framework coupling a control-policy branch to a workload-demand branch through an explicit elasticity term, reads out net rather than direct benefit, and emits a schema-conformant record covering energy, carbon, water, embodied share and validation venue. The framework is an architectural and methodological proposal, not a trained system; the contribution we defend empirically is the corpus analysis and the reporting schema derived from it. Coding sheet, derived statistics and all result artifacts: https://github.com/Kimalice/AI-for-Energy-Optimization-in-Data-Centers-Closing-the-Optimizer-Load-Loop
Mohammed Basharath Ullah, Summaiya Unnisa Begum, Mohammed Nadeem Ullah