cs.LGNov 9, 2026

Simulation-aided Reinforcement Learning with Control Variates

Authors: Omer Hemo, Ido Sefi, Mirco Mutti, Aviv Tamar, Kfir Y. Levy

Organizations: Department of Electrical and Computer Engineering, Technion - Israel Institute of Technology

Abstract

Reinforcement Learning (RL) algorithms often rely on data collected from a target environment, which can be expensive and noisy. Although simulators can generate large amounts of additional data, standard simulation mixing approaches, which directly combine simulated and real experience, can degrade performance due to simulator inaccuracies, especially when they bias the solution away from the optimal policy. We propose a conservative simulation-aided learning method based on control variates, where simulated data is used solely to reduce variance in the learning process rather than directly shaping the learning target. We develop our approach for Policy-Evaluation, which is a fundamental building block of RL algorithms. Then we combine it within the actor-critic framework towards policy improvement, as well as extend it towards offline RL scenarios. Experiments on multiple MuJoCo benchmark tasks demonstrate improvements over standard simulation mixing approaches, highlighting control variates as an effective tool for simulation-aided reinforcement learning.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards

    Sep 28, 2026Hsiao-Ru Pan, Florent Draye, Bernhard SchölkopfReinforcement LearningActor-Critic Methods

  2. Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning

    May 27, 2026Christoph Dann, Yishay Mansour, Mehryar MohriSim-to-Real TransferActive Learning

  3. Exact Fast Batch Simulation for Tabular Reinforcement Learning

    Oct 3, 2026Haochen Zhang, Lingzhou Xue, Zhong ZhengSimulation-Based RLTabular RL