A building controller that keeps a zone within comfort bounds at minimum energy can treat the bounds as a soft constraint, by adding the weighted violation to its cost, or as a hard constraint. We analyze the effect of the hard and the soft formulation of the same problem and isolate the effect of the formulation from the effect of the algorithm. To this end, we measure the difference in energy use between the two formulations on the BOPTEST bestest_hydronic_heat_pump case, in model predictive control (MPC) and in reinforcement learning (RL). The soft formulations are a weighted MPC and weighted SAC and DDPG learners, each at twelve weights. The hard formulations are an MPC with an exact penalty and the Lagrangian learners SAC-Lagrangian and primal-dual DDPG. We find that the hard formulations use more energy than the soft formulations at the largest weight, but violate the comfort bounds less. For a large weight, the soft formulation approaches the behavior of the hard formulation. The frontiers of the energy-comfort trade-off of the three weighted controllers lie close to each other, and the weight selects an operating point on this frontier. Under the soft formulation, the three algorithms reach very similar operating points, although at somewhat different weights. For this HVAC problem, the formulation and the weight thus affect the energy use more than the choice of algorithm.
Figures & tables
Fig. 1: The BOPTEST case bestest_hydronic_heat_pump and the control loop. Every Δt=900 s the controller sets the compressor modulation uk from the measurements and the forecasts of the disturbance and of the comfort bounds.
Fig. 2: Visualization of comfort bounds for one week day. The comfort bounds for a weekday are centered around 22.5∘ C and defined by occupied and unoccupied half-widths ho∈[0.5,3] K and hu∈[3,10] K, resulting in lower and upper bounds Lk and Uk . The switching times vary by plus or minus one hour from the default switching times at 07:00 and 20:00.
Fig. 3: The days of the year: the ten episodes of 14 days, each after a warm-up of 7 days, the summer, and the four weeks on which the model of the MPC is identified and validated. Training episodes start on days outside the ten episodes and the summer.
searched
SAC, weighted
DDPG, weighted
SAC-Lagrangian
primal-dual DDPG
learning rate [ 10−3 ]
0.1–3, log
0.18–2.8
0.5–3
2.9
3.0
discount factor
0.97, 0.99, 0.995, 0.999
0.97–0.995
0.97–0.999
0.999
0.99
target update rate [ 10−3 ]
1–20, log
1.4–19
1.2–20
1.1
2.8
batch size
128, 256, 512
128–512
128–512
256
128
updates per transition
1/8, 1/4, 1/2, 1
1/8–1
1/8–1
1
1
action-bound penalty κ
0.1–10, log
0.1–6.8
0.1–9.5
0.11
0.14
TABLE I: Hyperparameters of the learners: the searched range and the chosen value. For the weighted learners, the chosen values are given as the range over the twelve weights of the grid and w=100 .
controller
E
C
baseline PI controller
2.50
59.06
soft
MPC, w=2
2.16
6.47
SAC, w=2
2.11 ± 0.03
13.06 ± 5.86
DDPG, w=2
2.13 ± 0.03
14.29 ± 6.55
SAC, w=100
3.10 ± 0.06
3.80 ± 0.29
DDPG, w=100
2.83 ± 0.10
5.17 ± 0.39
TABLE II: Energy E and constraint cost C , as means over the three test episodes outside the summer. For the learners, mean and standard deviation over five seeds. E in kWh/m 2 , C in K h.
Fig. 4: Energy E against constraint cost C , test episodes outside the summer. Each weighted controller is a curve over its twelve weights. Each constrained controller is a filled point, and each weighted learner trained at w=100 an open point. The inset enlarges the region C≤250 K h, 1.75≤E≤2.2 kWh/m 2 .
Fig. 5: Energy E (solid, linear left axis) and constraint cost C (dashed, right axis) of the weighted controllers against the weight w , as means over the three test episodes outside the summer. The weights are spaced logarithmically, and the axis shows those up to 0.4. Error bars are the standard deviation over five seeds.
Constrained Reinforcement Learning (RL) has emerged as a significant research area within RL, where integrating constraints with rewards is crucial for enhancing safety and performance across diverse control tasks. In the context of heating systems in the buildings, optimizing the energy efficiency while maintaining the residents' thermal comfort can be intuitively formulated as a constrained optimization problem. However, to solve it with RL may require large amount of data. Therefore, an accurate and versatile simulator is favored. In this paper, we propose a novel building simulator I4B which provides interfaces for different usages and apply a model-free constrained RL algorithm named constrained Soft Actor-Critic with Linear Smoothed Log Barrier function (CSAC-LB) to the heating optimization problem. Benchmarking against baseline algorithms demonstrates CSAC-LB's efficiency in data exploration, constraint satisfaction and performance.
Baohe Zhang, Lilli Frison, Thomas Brox +1
Faculty of Computer Science, University of Freiburg, Germany · Systems Control and Optimization Laboratory, Department of Microsystems Engineering, University of Freiburg, Germany
Occupant comfort and grid-aware energy efficiency are competing objectives whose joint optimization depends critically on how reward functions are specified in deep reinforcement learning (DRL) controllers for buildings. Yet reward design remains largely ad hoc: comfort terms are either hand-tuned heuristics or simple temperature-deviation proxies without explicit grounding in thermal-comfort physics. We present PIRS (Physics-Informed Reward Shaping), which replaces these ad-hoc comfort proxies with the ISO 7730 Predicted Mean Vote (PMV) formulation inside a weighted multi-objective reward for Soft Actor-Critic (SAC). By anchoring the comfort signal in the ISO 7730 PMV formulation, PIRS improves reward interpretability and provides a standards-grounded comfort proxy without changing any other component of the learning pipeline. We evaluate PIRS in CityLearn v2.1.2 (challenge 2022 phase 1) with a central SAC agent trained for 50k steps over five random seeds, and compare against a rule-based controller (RBC), a manually engineered reward (E2), an energy-only reward (E3), and a naive temperature-deviation comfort reward (E4). District-level key performance indicators (KPIs), reported as ratios versus RBC, show that PIRS attains cost, carbon, and electricity metrics on par with the manual baseline while substantially outperforming non-physics-grounded designs -- particularly on load ramping (1.78x vs. ~2.4x RBC) and daily peak demand. All DRL policies remain above RBC at this training budget; we interpret this gap honestly and position PIRS as an interpretable, standards-aligned foundation for reward design rather than a claim of dominance over classical control at limited compute.
On--off cycling is the main cause of compressor wear in residential heat pumps, yet reinforcement learning (RL) controllers for buildings typically optimise only energy cost and thermal comfort, ignoring how much the learned policy cycles. We add a levelised compressor-wear term to the control reward and study how the resulting behaviour depends on the RL algorithm. Training Soft Actor---Critic (SAC) and Proximal Policy Optimisation (PPO) on an identical Markov decision process for the BOPTEST bestest hydronic heat pump case, we find that SAC learns a continuous modulation policy that keeps the compressor permanently engaged---the operating principle of an inverter-driven heat pump---achieving zero start-ups per day, whereas PPO collapses to bang-bang control that cycles more than the baseline. On the BOPTEST emulator the SAC policy cuts thermal discomfort by up to 90.7% for an 11.5% cost increase, while eliminating all baseline cycling.
Faizan Ahmed, Aniket Dixit, James Brusey
Centre for Computational Science and Mathematical Modelling, Coventry University, United Kingdom