When does deep reinforcement learning pay off in HVAC supervisory control? Four small Modelica case studies trained directly on FMUs
Organizations: Signal Processing, Artificial Intelligence and Vision Technologies (SAIVT), Queensland University of Technology, 2 George Street, Brisbane, 4000, Queensland, Australia · International Laboratory for Air Quality and Health, Queensland University of Technology, 2 George Street, Brisbane, 4000, Queensland, Australia
Abstract
Deep reinforcement learning (DRL) is often proposed for the supervisory control of heating, ventilation and air-conditioning (HVAC) systems. The reported gains are, however, difficult to interpret, because the baselines are often weak and the attainable optimum is seldom known. This paper examines four small, reproducible case studies based on an office building in Brisbane, Australia: pre-cooling under a time-of-use tariff; chiller staging with condenser water reset; chilled water supply temperature reset with a wet cooling coil; and chilled water storage under time-of-use prices and a demand charge. The plants were modelled with the Modelica Buildings library, and in three of the cases a soft actor-critic agent was trained directly on the exported functional mock-up unit (FMU). Each agent was compared with tuned rule-based controllers and, where practicable, with a near-optimal reference from hourly exhaustive search or dynamic programming with perfect foresight. In the two cases without storage the agent reached the near-optimal reference, but the margin above a well-chosen rule was small: 1.3% for condenser water reset and 0.18% for chilled water reset. In the latter case two common reset rules consumed more energy than a fixed set point. With a chilled water storage tank the margin was considerably larger. The agent outperformed both rules (-7.8%, compared with -3.9% and -6.3%) and adopted midday charging during the solar off-peak period, yet it obtained only about 45% of the saving achieved by the perfect-foresight schedule and did little to limit the demand peak. In the pre-cooling case a hand-tuned rule remained superior (-12.0% compared with -5.6%). These results suggest that studies of DRL for HVAC control should state the margin between the best rule and a near-optimal reference. Some practical difficulties encountered in training on co-simulation FMUs are also reported.
Figures & tables
| Abbreviations | |||
| AHU | air handling unit | HVAC | heating, ventilation and air-conditioning |
| CHW | chilled water | MPC | model predictive control |
| COP | coefficient of performance | RC | resistance–capacitance (model) |
| DRL | deep reinforcement learning | RL | reinforcement learning |
| FMI | Functional Mock-up Interface | SAC | soft actor-critic |
| FMU | functional mock-up unit | VAV | variable air volume |
| Case | Plant (Modelica) | Decision (hourly) | Coupling in time |
|---|---|---|---|
| 1 Pre-cooling | 100 m 2 RC zone (air + mass), ideal chiller | Zone set point 20–30 °C | Building mass |
| 2 Condenser water | 2 781 kW screw chillers, variable-speed tower, 1.35 MW load | Condenser water set point, staging | Weak |
| 3 Chilled water | 781 kW chiller, variable-flow CHW, wet AHU coil, VAV zone with moisture | CHW set point 5.5–11 °C | Weak |
| 4 Storage | Case 2 plant + 400 m 3 stratified tank (3.4 MWh) | Charge/discharge rate | Strong (tank, demand charge) |
| Case | Controller | Result | Saving | Remark |
| 1 | Fixed 24 °C | 13.31 AUD | – | |
| Expert pre-cooling rule | 11.70 AUD | 12.0 % | hand-tuned | |
| DRL (SAC) | 12.56 AUD | 5.6 % | second seed 6.4 % | |
| 2 | Fixed 29.4 °C | kWh | – | |
| Wet bulb + 4 K | kWh | 1.6 % | ||
| DRL (SAC) | kWh | 2.9 % |
Heating, Ventilation, and Air Conditioning Control
Heating, ventilation, and air conditioning control optimizes building energy use while maintaining thermal comfort, indoor air quality, and equipment safety.
See all 27 papers