Simulation-aided Reinforcement Learning with Control Variates
Organizations: Department of Electrical and Computer Engineering, Technion - Israel Institute of Technology
Abstract
Reinforcement Learning (RL) algorithms often rely on data collected from a target environment, which can be expensive and noisy. Although simulators can generate large amounts of additional data, standard simulation mixing approaches, which directly combine simulated and real experience, can degrade performance due to simulator inaccuracies, especially when they bias the solution away from the optimal policy. We propose a conservative simulation-aided learning method based on control variates, where simulated data is used solely to reduce variance in the learning process rather than directly shaping the learning target. We develop our approach for Policy-Evaluation, which is a fundamental building block of RL algorithms. Then we combine it within the actor-critic framework towards policy improvement, as well as extend it towards offline RL scenarios. Experiments on multiple MuJoCo benchmark tasks demonstrate improvements over standard simulation mixing approaches, highlighting control variates as an effective tool for simulation-aided reinforcement learning.
Figures & tables
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Walker2D-v5 | Ant-v5 |
| critic learning rate | 0.0003 | 0.0003 |
| policy learning rate | 0.0003 | 0.0003 |
| total update steps | 1000000 | 1000000 |
| real step frequency | 5 | 5 |
| sim step frequency | 1 | 1 |
| num steps in sim | 100 | 100 |
| Sim Gap | Ant | Walker2D |
|---|---|---|
| Small | ||
| Medium | ||
| Large |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,835 papers