cs.LGOct 1, 2026
SaveMulti-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning
Organizations: University of Texas at Austin · University of British Columbia
Abstract
Policy gradient methods for on-policy reinforcement learning (RL) can become unstable when expensive, scarce target-domain data yield noisy gradient estimates. We address this challenge by complementing limited high-fidelity (HF) target-domain data with abundant, cheap, but biased low-fidelity (LF) data, e.g., from a simplified simulator. Most existing methods directly optimize biased objectives based on LF data. In contrast, the recently introduced multi-fidelity policy gradient (MFPG) framework uses LF data solely to construct a control variate that reduces variance and improves HF data efficiency without biasing the policy gradient estimator. However, published work on MFPG is limited to REINFORCE on small-scale simulation tasks. We develop MFPG for modern actor-critic learning in GPU-parallel simulation and on a physical robot. Our analysis and experiments show that naive extensions to proximal policy optimization (PPO) can lose cross-fidelity correlation or inflate variance. Our MFPG-PPO addresses these failures by redesigning the sampling, advantage estimation, and control variate construction to preserve cross-fidelity correlation, and by monitoring estimator uncertainty to prevent variance inflation. We also introduce a budget-aware MFPG-PPO to divide a fixed sampling budget among high- and low-fidelity data sources. Across simulated robot locomotion tasks of varying LF-to-HF transfer difficulty and HF data budgets, MFPG-PPO improves upon PPO trained on HF data alone in nearly all settings, and consistently matches the performance of PPO trained with 16x more HF data on the hardest task at the smallest HF budgets. In contrast, most baselines that use LF data perform well only where direct LF-to-HF transfer succeeds. MFPG-PPO enables stable learning on a physical Franka arm using only 4 real-robot episodes per update and no human demonstrations.
Figures & tables
Figure 1: In a simulated humanoid robot locomotion task on rough terrain, MFPG - PPO (orange) scales monotonically with the HF data budget and, where PPO trained on HF data alone (blue) collapses, matches the mean return of PPO trained with more HF data. HF data budgets are set by the number of parallel HF training environments and expressed as fractions of the Isaac Lab default (4096 environments). At each HF budget, MFPG - PPO uses the same amount of HF data as PPO HF -Only, plus abundant LF data from a simplified simulator. PPO LF -Only (gray) trains on abundant LF data alone and is evaluated in the HF environment. Bars show mean HF evaluation returns of final policies across 6 seeds, with seed-bootstrapped confidence intervals. Dots denote individual seeds. The snapshots show the MFPG - PPO policy walking across the terrain, whereas the PPO HF -Only policy remains on its knees despite using more HF training data.
Figure 2: MFPG - PPO collects three batches of transitions. Representative HF , coupled LF , and uncoupled LF rollouts are shown with red circles, filled teal circles, and hollow teal squares. (a) When HF and LF are simulations of different accuracy, each batch contains parallel -step rollouts per iteration. (b) When the HF environment is a robot collecting -step episodes ( ), coupled LF simulation rollouts restart from recorded HF states every steps. In both cases, the LF simulation runs only parallel -step rollouts, while the coupled LF GAE horizon matches the HF horizon, in (a) and in (b). In (b), we stitch the coupled LF rollouts of each episode for GAE .
Figure 3: MFPG - PPO is the only method in our comparison that improves upon PPO HF -Only in nearly every setting, is never distinguishably worse, and scales monotonically with more HF data. Bars denote differences in mean HF evaluation return relative to PPO HF -Only. Error bars denote bootstrapped confidence intervals for the mean of the difference across 6 seeds.
Figure 4: Neither limited real-world data nor abundant simulation data alone yield a good real-robot policy, whereas MFPG - PPO succeeds by combining them. Bars and curves denote means across 3 seeds. Whiskers and shaded bands show their ranges. (a) Final success rate of the deterministic policy over real-robot episodes ( per seed). Horizontal lines denote the shared warm-start checkpoint performance and that of a simulation-trained policy well past convergence ( simulation success), each evaluated over real-robot episodes. (b) Training success rate on the robot (solid lines denote running averages over 40 episodes per seed) and in uncoupled LF simulation (dashed lines). (c) Pearson correlation between the coupled LF and HF paired statistics, computed on episode sums (4 pairs) and per step (240 pairs), averaged over the past 10 iterations.
Figure 5: Curves and bars denote means over 6 seeds. Bands and whiskers denote seed-bootstrapped confidence intervals for the means. (a) HF training return on H1 Rough. (b) Fraction of training iterations whose variance ratio \mathrm{Var}[{\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}}+c\,({\color[rgb]{0,0,0}X^{\mathrm{low}}_{k,t}}-{\color[rgb]{0,0,0}\hat{\mu}^{\mathrm{low}}})]\,/\,\mathrm{Var}[{\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}}] , averaged over minibatches before the monitoring check, exceeds , , and at the HF budget.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Simulation | Real robot |
|---|---|---|
| EMA rate | (FR); (BA) | |
| Bootstrap resamples | — | |
| Evidence-window length | — | |
| Confidence level |
Table 1: Control variate monitoring hyperparameters.
| Method | Candidate values |
|---|---|
| Backbone parameters | |
| PPO | Learning rate ; epochs ; minibatches per epoch |
| REINFORCE | Learning rate ; discount factor |
| SAC | Actor and critic learning rate ; UTD ratio or minibatches per iteration; target-entropy scale |
| Method-specific parameters | |
| MFPG - PPO -FR | EMA rate |
Table 2: Grid values for simulation hyperparameter search.
| PPO backbone hyperparameters inherited from Isaac Lab’s task-default RSL-RL configurations | |
| Parameter | Value |
| Rollout length per environment | 24 steps |
| Optimizer | Adam |
| Discount factor / GAE parameter | / |
| PPO clipping parameter | |
| Value-loss coefficient / value clipping | / enabled |
Table 3: Simulation hyperparameters.
| Parameter | Value |
|---|---|
| Shared | |
| Initial learning rate | |
| Epochs | 4 |
| Minibatches per epoch | 4 |
| HF episodes per update | 4 |
| Optimizer | Adam |
Table 4: Real-robot hyperparameters.
| Method | HF | Coupled LF | Uncoupled LF |
|---|---|---|---|
| PPO HF -Only | — | — | |
| PPO LF -Only | — | — | |
| MFPG - PPO -FR | |||
| MFPG - PPO -BA | |||
| Co-training, DARC, PAR | — | ||
| MFPG -REINFORCE |
Table 5: Simulation rollouts per iteration. Rollouts contain 24 steps, except for MFPG -REINFORCE (100 steps).
Figure 6: Locomotion collision models. (a) Unitree H1 and (b) ANYmal-D. Each panel compares the full HF collision model (left) with the simplified LF collision model (right). Green outlines indicate the collision geometry.
| H1 | ANYmal-D | |||
| Parameter | HF | LF | HF | LF |
| Physics time step (s) | ||||
| Physics steps per action | 10 | 2 | 10 | 8 |
| Position solver iterations | 4 | 2 | 4 | 2 |
| Velocity solver iterations | 4 | 0 | 0 | 0 |
| Self-collisions | Enabled | Disabled | Enabled | Disabled |
Table 6: Locomotion fidelity settings, shared by the flat and rough tasks of each robot.
Figure 7: Franka cube-picking environments. The physical Franka setup (left) provides the HF environment, and ManiSkill simulation (right) provides the LF environment.
| Observation | Dimensions |
|---|---|
| Arm joint positions and velocities | |
| Gripper width | 1 |
| End-effector position and quaternion | |
| Cube position and quaternion | |
| Goal position | 3 |
| End-effector-to-cube and cube-to-goal vectors |
Table 7: State observations for the Franka cube-picking task. Here, is cube yaw and counts elapsed control steps.
Figure 8: Mean final HF evaluation returns for the task-budget settings in . Error bars denote bootstrap confidence intervals across 6 seeds.
Figure 9: Fine-tuning after extensive LF pretraining on H1 Rough. Both methods start from each seed’s final PPO LF -Only checkpoint, whose mean return the gray bar shows. HF budgets are expressed as fractions of 4096 parallel environments. Bars show HF evaluation returns over 2048 episodes per seed, averaged across 6 seeds. Whiskers show seed-bootstrapped confidence intervals for the means.
Figure 10: Percentage of minibatches on which the control variate was applied during Franka cube-picking training in . Each marker aggregates the past 10 iterations. The line is the mean over 3 seeds, the band their range.
Figure 11: Cross-fidelity event timing and correlation between paired statistics. Left. Distribution of grasp-event timing differences between the physical Franka robot and its coupled ManiSkill simulation. The physical robot generally grasps later than its simulated counterpart due to fidelity gaps, causing low correlation between paired per-step statistics. Middle. Distribution of foot-strike timing differences between high- and low-fidelity simulations of the H1 Rough task in Isaac Lab. The timing generally aligns more closely than in the Franka case. Right. Cross-fidelity correlations between paired HF and LF statistics across the tasks in and . The first four x-axis entries are Isaac Lab simulation tasks and the last is the physical robot task. Each marker gives a run’s median correlation over training. Gray triangles denote per-step correlations, orange circles denote episode-sum correlations, and black bars denote medians across runs. A-D denotes ANYmal-D.
Figure 12: Event indicators (top) and paired per-step surrogate statistics ({\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}},{\color[rgb]{0,0,0}X^{\mathrm{low}}_{k,t}}) (bottom). Left. A Franka cube-picking episode. The upper bars indicate detected grasps, and the shaded interval marks the timing mismatch. Right. An H1 Rough rollout. The upper bars indicate left- and right-foot contact. In both panels, the lower curves show the HF and coupled LF surrogate statistics in red and teal, respectively.
Figure 13: Fraction of training iterations whose variance ratio \mathrm{Var}[{\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}}+c\,({\color[rgb]{0,0,0}X^{\mathrm{low}}_{k,t}}-{\color[rgb]{0,0,0}\hat{\mu}^{\mathrm{low}}})]\,/\,\mathrm{Var}[{\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}}] , averaged over minibatches before the monitoring check, exceeds , , and for the HF budget case of .
Figure 14: Training diagnostics for the ablations in at the (left) and (right) HF budgets on H1 Rough. Rows show the variance ratio \mathrm{Var}[{\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}}+c\,({\color[rgb]{0,0,0}X^{\mathrm{low}}_{k,t}}-{\color[rgb]{0,0,0}\hat{\mu}^{\mathrm{low}}})]\,/\,\mathrm{Var}[{\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}}] , the fraction of minibatches failing the monitoring checks, and the estimated correlation between the paired per-step statistics ({\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}},{\color[rgb]{0,0,0}X^{\mathrm{low}}_{k,t}}) . Curves show seed means and bands show bootstrapped confidence intervals for the means.
More in this topic
Policy Gradient Methods
Policy gradient methods optimize stochastic policies by estimating how parameter changes affect expected reward.
See all 164 papers