Reinforcement Learning (RL) algorithms often rely on data collected from a target environment, which can be expensive and noisy. Although simulators can generate large amounts of additional data, standard simulation mixing approaches, which directly combine simulated and real experience, can degrade performance due to simulator inaccuracies, especially when they bias the solution away from the optimal policy. We propose a conservative simulation-aided learning method based on control variates, where simulated data is used solely to reduce variance in the learning process rather than directly shaping the learning target. We develop our approach for Policy-Evaluation, which is a fundamental building block of RL algorithms. Then we combine it within the actor-critic framework towards policy improvement, as well as extend it towards offline RL scenarios. Experiments on multiple MuJoCo benchmark tasks demonstrate improvements over standard simulation mixing approaches, highlighting control variates as an effective tool for simulation-aided reinforcement learning.
Figures & tables
Figure 1: Simulation-aided policy evaluation on the Humanoid task. Simulation parameters (gravity tilt and magnitude) are perturbed relative to the real environment. Left: Normalized Q-value error of each mixing method as a function of the real-data budget. Middle: Effect of λ on CV-Qmix, including the adaptive choice of λ . Right: Effect of α on Reg-Qmix. In all plots, the real-only estimator is shown for reference. Curves show the mean and shaded regions ± 1 std over 10 seeds.
Figure 2: Policy finetuning with biased simulation data mixing. Comparing CV-Qmix (ours) and Reg-Qmix to other off-dynamics methods on Ant and Walker2D tasks across different simulation to real gaps. The policies were pre-trained on the simulation and then transferred to the real environment for fine-tuning with mixed simulation and real data. Each plot shows the mean ± std of 5 seeds.
Figure 3: Policy optimization with mixed simulated data on Hopper and Puck Sliding . Simulator parameters (gravity, friction, and body part lengths) are perturbed relative to the real environment. The plots on the left show online optimization (mean ± std over 10 seeds), while the plots on the right show offline optimization (box plots over 10 seeds).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Policy optimization robustness to simulation bias. Comparison of data-mixing strategies under increasing dynamics mismatch for Ant and Walker2D tasks. Simulator parameters (friction, joint armature, joint damping) are perturbed relative to the real environment. Showing online optimization (mean ± std over 5 seeds).
Table 5Table 6
Hyperparameter
Walker2D-v5
Ant-v5
critic learning rate
0.0003
0.0003
policy learning rate
0.0003
0.0003
total update steps
1000000
1000000
real step frequency
5
5
sim step frequency
1
1
num steps in sim
100
100
Appendix
Table 10: VGDF Hyperparameters
Table 8
Sim Gap
Ant
Walker2D
Small
0.37±0.02
0.45±0.06
Medium
0.21±0.01
0.32±0.06
Large
0.08±0.02
0.25±0.05
Appendix
Table 18: λ^ values of CV-Qmix by simulation gap in the fine-tuning experiment. For each seed, λ^ is averaged over the full training run; values are mean ± std over 5 seeds.
Figure 5: The environment consists of a two-link robotic arm that moves in the horizontal (x,y) plane, along with a puck that can slide on the surface. The objective is for the arm to strike the puck so that it reaches a target location on the plane, which is placed far from the robot. The puck, arms joints, target positions are randomized at each episode.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Recent progress in reinforcement learning with verifiable rewards (RLVR) has highlighted the effectiveness of simple critic-free policy-gradient methods such as Group Relative Policy Optimization (GRPO). In contrast, actor-critic methods rely on learned value functions whose approximation error can introduce bias through commonly used advantage estimators such as temporal-difference error. Motivated by this observation, we revisit trajectory-level control variates through an advantage-value formulation, which we call Advantage-Based Control Variates (ABC). This formulation reveals that the covariance structure is closely related to the return decomposition used in Direct Advantage Estimation (DAE). Finally, we combine ABC with DAE into a single actor-critic algorithm and evaluate it in an offline-to-online RLVR setting, where the critic is first trained on previously collected trajectories and adapted during online learning. On mathematical reasoning tasks, ABC achieves performance competitive with GRPO using substantially fewer online optimization steps.
Model-based reinforcement learning (MBRL) agents typically learn world models by minimizing predictive loss. However, powerful RL optimizers inevitably exploit minor model inaccuracies, leading to simulator exploitation and a reality gap where policies succeed in simulation but fail in the real world. We propose that the objective for learning simulators should be strategic robustness rather than predictive accuracy, and formulate this as a zero-sum minimax game between a model player and an adversarial policy player. We provide a comprehensive theoretical analysis: (1) an online learning guarantee showing the game is learnable with sublinear regret bounds; (2) a tractable critic-based simplification bounding the global policy-value gap by the local critic's loss; and (3) an Error-MDP duality, proving that finding the worst-case policy is formally dual to a standard RL problem where the reward is the one-step critic error. This duality yields a provably convergent active data selection algorithm. Experiments on continuous control tasks demonstrate that our approach reduces prediction error in strategically important regions by 1.5-2.2× and enables policies trained purely in simulation to match near-optimal real-world performance.
Christoph Dann, Yishay Mansour, Mehryar Mohri
Google Research · Tel Aviv University and Google Research · Google Research and Courant Institute of Mathematical Sciences
Simulation is a fundamental computational primitive in reinforcement learning (RL), yet conventional simulation explicitly generates individual trajectories even when downstream procedures use only aggregate statistics. To address this, we develop an exact fast-simulation framework for finite-horizon tabular Markov decision processes. Our framework has two complementary modes. In direct batch simulation, a batch is represented by its aggregate Markov flow. With sufficient parallel simulation resources, this flow can be obtained by trajectory aggregation; when such simulation is unavailable or costly but the initial state and transition distributions are directly accessible, we instead generate an identically distributed flow through forward Markov-flow sampling without materializing individual trajectories. The latter reduces the simulator-side computational dependence on batch size m from O(m) to O(1). In adaptive batch simulation, when batch length is determined by a data-dependent condition, exact multivariate-hypergeometric splitting recursively refines a candidate Markov flow while preserving the conditional law, reducing the cost dependence on m from O(m) to O(logm). Together, these modes accelerate simulation by keeping trajectories aggregated whenever possible and refining flows only when required to locate data-dependent boundaries. The framework applies broadly across simulator-based, offline, and online batch or stage-based RL, as illustrated with representative algorithms from each setting.
Haochen Zhang, Lingzhou Xue, Zhong Zheng
Department of Statistics, The Pennsylvania State University · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame