Uncertainty-Aware Hybrid Offline Reinforcement Learning with Online Action Selection Enabled by Sparse Gaussian-Process Critics
Organizations: Sargent Centre for Process Systems Engineering, Department of Chemical Engineering, Imperial College London
Abstract
Start-ups and grade changes remain largely operator-driven, while developing and maintaining dynamic models for model-based control requires substantial engineering effort. Hybrid Offline Learning and Online Optimisation (HOFLON) uses one-step offline reinforcement learning to select actions from historical transition data. In its original implementation, the online objective combines the predicted return from a tree-ensemble critic with an adversarial-autoencoder (AAE) penalty for unfamiliar state-action pairs. We examine the effect of this penalty and replace the critic and AAE with a sparse variational Gaussian-process (sGP) critic. Actions maximise , an uncertainty-adjusted estimate of the return from one action followed by the historical operating practice. On the HOFLON polymerisation-reactor benchmark, removing the AAE from HOFLON led to 4 failed start-ups in 15 runs over five critics. At its published weight, the variation in the AAE penalty is four orders of magnitude smaller than the variation in the tree-ensemble critic. Nevertheless, it changes actions when the predicted returns of competing candidates are nearly equal, and paired runs with and without the penalty diverged from the first move. With a Matérn-5/2 sGP critic, led to failure in 13 of 15 runs, whereas with none of 15 runs (five critics, three noise realisations) failed. The median and lowest returns were 13,464 and 13,398 (HOFLON: 13,393 and 12,411). A squared-exponential sGP critic with learned inducing inputs needed to remain within the empirical data support in steady state, and then settled with a temperature offset of 2.0 K.
Figures & tables
| Fig. 4 | Support term in the online objective | Failed | Median | Worst | Max. peak | Coolant | Steady-state offset | Off-support [%] | ||
| box | runs | [K] | move [K] | [kg m -3 ] | [K] | 0–20 h | 20 h | |||
| HOFLON [ 3 ] : XGBoost critic , objective ( 3 ) | ||||||||||
| 2 | none | 4/15 | 13,020 | 2,782 | 374.7 | 0.02 | 0.79 | 0.05 | 45 | 0.0 |
| 3 | AAE, (published, HOFLON) | 0/15 | 13,393 | 12,411 | 363.6 | 0.02 | 0.58 | 0.04 | 35 | 0.0 |
| 4 | AAE, | 0/15 | 13,372 | 13,013 | 356.8 | 0.01 | 0.58 | 0.04 | 22 | 0.0 |
| HOFLON-sGP: sparse-GP critic with Matérn-5/2 kernel, objective ( 7 ) | ||||||||||
Offline RL
Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.
See all 327 papers