Heteroscedastic Gaussian Process Q-learning for Sample Efficient Experimentation
Organizations: Imperial College London, UK.
Abstract
Self-driving laboratories must learn to control a process from tens to hundreds of expensive experiments, too few for deep reinforcement learning (RL). We learn the Q-function directly with a Gaussian process (GP), which avoids learning the dynamics, and show that the GP posterior supplies the learning rate of Q-learning. If each Bellman target is treated as a noisy measurement of the Q-function, the learning rate that minimises the mean squared error is the Kalman gain, and applying it to every observed transition is equivalent to GP regression. Heteroscedastic Gaussian Process Q-learning (HGPQ) sets the noise variance of each target with a heteroscedastic GP, which learns how unreliable the targets are across the states and inputs. Targets from noisy operating conditions, or from regions where the value function has large gradients, then move the estimate less than targets from low-noise regions, so HGPQ learns quickly where the data are reliable without being misled where they are not. For kernel value iteration with per-target variances that are assigned when a target is collected, we prove a high-probability bound on the error of the value estimate, uniform over the explored region. Pooled over five simulated benchmarks, which include a chemical reactor and a bioprocess, HGPQ approaches the best return found by any method in fewer episodes than GP, kernel and deep RL baselines.
Figures & tables
| Method | Update | Learning rate |
|---|---|---|
| GPQ-CLR ( Bloor et al., 2025a ) | , GP fit to the | tuned |
| KTD ( Geist and Pietquin, 2010 ) | Kalman gain on fixed features, one noise level | |
| GPTD ( Engel et al., 2005 ) | one GP conditioned on | Kalman gain, one noise level |
| KOVI ( Yang et al., 2020 ) | , | Kalman gain, one noise level |
| HGPQ (ours) | , | Kalman gain, one noise level per target |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Reward | |||
|---|---|---|---|---|
| Non-stationary LMDP | 2 | 1 | 10 | every time step, linear, sign-switching |
| Semi-batch reactor | 5 | 2 | 8 | final time step only (product) |
| Photosynthetic biomass production | 3 | 2 | 8 | final time step only (biomass) |
| Inverted pendulum | 3 | 1 | 20 | final state, small input cost |
| Target tracking | 6 | 2 | 10 | final distance, small input cost |
| A: state-dependent noise | B: step in the value | |||||
|---|---|---|---|---|---|---|
| Target variance | gap | error, | error, | gap | error, | error, |
| Floor | 0.229 | 0.147 | 1.26 | 0.041 | 0.163 | 0.110 |
| One noise level per time step | 0.008 | 0.011 | 0.035 | 0.035 | 0.105 | 0.098 |
| HGPQ | 0.006 | 0.006 | 0.029 | 0.023 | 0.023 | 0.121 |
| Setting | Value |
|---|---|
| Initial episodes | 5 (LMDP), 10 (pendulum, target tracking), 20 (reactor, biomass production) |
| Uniform exploration probability | |
| Thompson draws | pathwise conditioning with 1024 random Fourier features |
| Candidate inputs | 500: the lower corner, the upper corner and the centre of the input box, then Sobol points |
| Local maximisation | L-BFGS-B from the 5 best candidates, at most 50 iterations each |
| Candidate inputs on the LMDP | , since the reward is linear in the input |
| DQN | Ensemble-DQN | DDPG | TD3 | SAC | |
|---|---|---|---|---|---|
| Networks | five critics | actor and critic | actor and critic | actor and critic | |
| Batch size | 64 | 64 | 64 | 256 | 256 |
| Replay capacity | |||||
| Target updates | hard, every 100 steps | hard, every 200 steps | soft, | soft, | soft, |
| Exploration | -greedy, from 1.0 to 0.05, decay 0.99 per episode | upper confidence over the ensemble, | Gaussian noise 0.1 | Gaussian noise 0.1 | entropy, target scale 1.0 |
| Other | gradient norm clipped at 1 | target noise 0.2 clipped at 0.5, policy update every second step, learning after 1000 steps | learning after 200 steps |
| Method | Parameter | LMDP | Reactor | Biomass | Pendulum | Target |
|---|---|---|---|---|---|---|
| HGPQ | ||||||
| (constant) | ||||||
| GPQ-CLR | ||||||
| KOVI | ||||||
| KOVI |
| Homoscedastic | With the next GP’s variance | |||||
| Setting | HGPQ | HGPQ | ||||
| Final return after 150 episodes | ||||||
| Reactor | ||||||
| Biomass | ||||||
| Pendulum | ||||||
| Target | ||||||
| Estimator of | Reactor | Biomass |
|---|---|---|
| Single level, marginal likelihood | ||
| Single level, log-normal prior | ||
| Single level, leave-one-out innovations ( Mehra, 1970 ) | ||
| Single level, leave-one-out predictive probability for every hyperparameter ( Bachoc, 2013 ) | ||
| Single level, with the full covariance of the next GP (Appendix A.3 ) |
RL Control
Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.
See all 298 papers