cs.LGNov 17, 2026

When does deep reinforcement learning pay off in HVAC supervisory control? Four small Modelica case studies trained directly on FMUs

Authors: Bingnan Zhao, Harshala Gammulle, Tharindu Fernando

Organizations: Signal Processing, Artificial Intelligence and Vision Technologies (SAIVT), Queensland University of Technology, 2 George Street, Brisbane, 4000, Queensland, Australia · International Laboratory for Air Quality and Health, Queensland University of Technology, 2 George Street, Brisbane, 4000, Queensland, Australia

Abstract

Deep reinforcement learning (DRL) is often proposed for the supervisory control of heating, ventilation and air-conditioning (HVAC) systems. The reported gains are, however, difficult to interpret, because the baselines are often weak and the attainable optimum is seldom known. This paper examines four small, reproducible case studies based on an office building in Brisbane, Australia: pre-cooling under a time-of-use tariff; chiller staging with condenser water reset; chilled water supply temperature reset with a wet cooling coil; and chilled water storage under time-of-use prices and a demand charge. The plants were modelled with the Modelica Buildings library, and in three of the cases a soft actor-critic agent was trained directly on the exported functional mock-up unit (FMU). Each agent was compared with tuned rule-based controllers and, where practicable, with a near-optimal reference from hourly exhaustive search or dynamic programming with perfect foresight. In the two cases without storage the agent reached the near-optimal reference, but the margin above a well-chosen rule was small: 1.3% for condenser water reset and 0.18% for chilled water reset. In the latter case two common reset rules consumed more energy than a fixed set point. With a chilled water storage tank the margin was considerably larger. The agent outperformed both rules (-7.8%, compared with -3.9% and -6.3%) and adopted midday charging during the solar off-peak period, yet it obtained only about 45% of the saving achieved by the perfect-foresight schedule and did little to limit the demand peak. In the pre-cooling case a hand-tuned rule remained superior (-12.0% compared with -5.6%). These results suggest that studies of DRL for HVAC control should state the margin between the best rule and a near-optimal reference. Some practical difficulties encountered in training on co-simulation FMUs are also reported.

Figures & tables

Explore similar work

CardsList
  1. Deep Reinforcement Learning for Heat Pump and Underfloor Heating Control: An End-to-End Case Study

    Nov 17, 2026Antonio Manjavacas, David Suárez-Caro, Alejandro Campoy-Nieves +3Heating, Ventilation, and Air Conditioning ControlBuilding Energy Management

  2. A Unified Python Framework for Direct PPO-based Control of AHUs with Economizer Logic and CO2-Constrained Ventilation

    May 23, 2026Erfan Haghighat Damavandi, Davide Papurello, Mahdi Alibeigi +3RL ControlConstrained RL

  3. Counter-Dyna: Data-Efficient RL-Based HVAC Control using Counterfactual Building Models

    May 6, 2026Jan Marco Ruiz de Vargas, Fabian Raisch, Zoltan Nagy +2Neural Surrogate ModelingBuilding Energy Management