When does deep reinforcement learning pay off in HVAC supervisory control? Four small Modelica case studies trained directly on FMUs
Authors: Bingnan Zhao, Harshala Gammulle, Tharindu Fernando
Organizations: Signal Processing, Artificial Intelligence and Vision Technologies (SAIVT), Queensland University of Technology, 2 George Street, Brisbane, 4000, Queensland, Australia · International Laboratory for Air Quality and Health, Queensland University of Technology, 2 George Street, Brisbane, 4000, Queensland, Australia
Deep reinforcement learning (DRL) is often proposed for the supervisory control of heating, ventilation and air-conditioning (HVAC) systems. The reported gains are, however, difficult to interpret, because the baselines are often weak and the attainable optimum is seldom known. This paper examines four small, reproducible case studies based on an office building in Brisbane, Australia: pre-cooling under a time-of-use tariff; chiller staging with condenser water reset; chilled water supply temperature reset with a wet cooling coil; and chilled water storage under time-of-use prices and a demand charge. The plants were modelled with the Modelica Buildings library, and in three of the cases a soft actor-critic agent was trained directly on the exported functional mock-up unit (FMU). Each agent was compared with tuned rule-based controllers and, where practicable, with a near-optimal reference from hourly exhaustive search or dynamic programming with perfect foresight. In the two cases without storage the agent reached the near-optimal reference, but the margin above a well-chosen rule was small: 1.3% for condenser water reset and 0.18% for chilled water reset. In the latter case two common reset rules consumed more energy than a fixed set point. With a chilled water storage tank the margin was considerably larger. The agent outperformed both rules (-7.8%, compared with -3.9% and -6.3%) and adopted midday charging during the solar off-peak period, yet it obtained only about 45% of the saving achieved by the perfect-foresight schedule and did little to limit the demand peak. In the pre-cooling case a hand-tuned rule remained superior (-12.0% compared with -5.6%). These results suggest that studies of DRL for HVAC control should state the margin between the best rule and a near-optimal reference. Some practical difficulties encountered in training on co-simulation FMUs are also reported.
781 kW chiller, variable-flow CHW, wet AHU coil, VAV zone with moisture
CHW set point 5.5–11 °C
Weak
4 Storage
Case 2 plant + 400 m 3 stratified tank (3.4 MWh)
Charge/discharge rate
Strong (tank, demand charge)
Table 1: The four case studies.
Case
Controller
Result
Saving
Remark
1
Fixed 24 °C
13.31 AUD
–
Expert pre-cooling rule
11.70 AUD
12.0 %
hand-tuned
DRL (SAC)
12.56 AUD
5.6 %
second seed 6.4 %
2
Fixed 29.4 °C
12,225 kWh
–
Wet bulb + 4 K
12,030 kWh
1.6 %
DRL (SAC)
11,865 kWh
2.9 %
Table 2: Results on the test period (Case 1: three hottest consecutive weekdays; Cases 2–4: one summer week). Savings are relative to the first row of each case; a negative saving means more energy than that row. The headroom is the difference between the best rule and the near-optimal reference (hourly optimum or perfect-foresight schedule), expressed as a percentage of the first row.
Figure 1: Case 1: zone temperature, set point and chiller power over the three test days. Green: off-peak; red: peak.
Figure 2: Case 2: condenser water set point against wet bulb (left) and staging against load (right) for the rule, the agent and the hourly optimum.
Figure 3: Case 3: chilled water set point against zone load (left) and outdoor temperature (right). The agent and the hourly optimum lower the set point as the load rises; the outdoor-air reset does not.
Figure 4: Case 4: building and chiller load, tank heat flow, state of charge and plant power over the test week for the demand-limit rule, the agent and the perfect-foresight schedule.
Heating, ventilation, and air conditioning control optimizes building energy use while maintaining thermal comfort, indoor air quality, and equipment safety.
Efficient control of underfloor heating (UFH) systems coupled with heat pumps (HPs) is challenging due to the high thermal inertia of buildings and the slow dynamic response of UFH systems. Deep Reinforcement Learning (DRL) offers a data-driven control approach capable of simultaneously optimizing energy consumption and thermal comfort. However, the application of DRL-based controllers in occupied buildings under real operating conditions remains limited. This work presents and end-to-end case study (development, evaluation, and real-world deployment) of a DRL-based controller for an integrated UFH-HP system. First, a simulation-based pilot study was conducted to assess different DRL algorithms, control strategies, and configurations. Subsequently, a digital twin of a real residential building was developed and used to train DRL agents under simulated operating conditions. The best-performing agent was then deployed in the physical building without additional online fine-tuning. The results show that the controller successfully maintained indoor temperatures within the prescribed comfort limits while dynamically adjusting the HP supply temperature and UFH valve positions. These findings demonstrate the feasibility of deploying DRL controllers in occupied buildings and highlight their robustness and the practical challenges associated with transferring control policies from digital environments to real heating systems.
Antonio Manjavacas, David Suárez-Caro, Alejandro Campoy-Nieves +3
Department of Computer Science and Artificial Intelligence, Universidad de Granada, Granada, 18071, Spain · Georg Fischer Building Flow Solutions, Avda. Leonardo da Vinci 15-17-19, Getafe, 28906, Madrid, Spain
Optimizing HVAC (Heating, Ventilation and Air Conditioning) can enhance a building's energy efficiency while providing comfort levels for its occupants. Using conventional control systems to maintain HVAC functions is often difficult because of the nonlinear characteristics of a building envelope as it experiences stochastic load variations over time. This paper presents a new approach to optimizing HVAC systems through the use of Deep Reinforcement Learning (DRL) algorithms and the Proximal Policy Optimization (PPO) algorithm implemented in a custom Python performance environment. The DRL system uses a second order resistor-capacitor thermal model and an integrated dynamic mass balance of CO2 to replicate the complex physics associated with buildings. One major innovation of this study is a "Hierarchical Flow Logic," which provides the means to ensure that indoor air quality (IAQ) is maintained by overriding the accepted actions of the agent that cause CO2 to exceed 1000 ppm. In addition, an enthalpy-based economiser is used to create free cooling from the outdoor environment. The experimental data shows that compared to PID controllers tuned by GA or traditional On-Off controls, a PPO agent has better temperature stability and energy efficiency overall. An end-to-end pipeline provides an avenue for robust and generalized solutions to help implement smart building energy management within the context of real hardware implementation.
a Research and Development Department Owtana Tech- Turin, Italy · b Department of Energy Politecnico di Torino- Turin, Italy · c Department of Energy University of Tehran- Tehran, Iran +1
Model-based reinforcement learning (MBRL) offers a promising approach for data-efficient energy management in buildings, combining the strengths of predictive modeling and reinforcement learning. While previous MBRL methods applied to HVAC control have reduced training data requirements, they still require several months of interaction with the building to learn a satisfactory control policy. A key reason is that existing surrogate models attempt to predict the entire state-space, including weather and electricity prices that are unaffected by control actions, or completely ignore these variables. Addressing these issues, we propose Counter-Dyna, a method that enhances the data-efficiency of Dyna, an MBRL method. We create data-efficient counterfactual surrogate models (CSM) by leveraging invariances in the state-space. Using a CSM in Dyna speeds up RL training measured in environment interaction data compared to previous results. In comparison with previous state-of-the-art that used 6-12 months of environment interactions, our method needs only 5 weeks. We evaluate our method in a large simulation study using the literature standard BOPTEST framework and proximal policy algorithm (PPO) as the RL algorithm. Our results show cost-saving potentials of 5.3% to 17.0% in a hypothetical deployment scenario. Our work is a significant step towards making real-world deployment of RL algorithms in HVAC control practically viable.
Jan Marco Ruiz de Vargas, Fabian Raisch, Zoltan Nagy +2
Technical University of Munich Munich, Germany · Technical University of Applied Sciences Rosenheim Rosenheim, Germany · Eindhoven University of Technology Eindhoven, Netherlands +1