cs.LGOct 1, 2026

Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning

Authors: Xinjie Liu, Ruihan Zhao, Anirban Chaudhuri, Cyrus Neary, Ufuk Topcu, David Fridovich-Keil

Organizations: University of Texas at Austin · University of British Columbia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Policy gradient methods for on-policy reinforcement learning (RL) can become unstable when expensive, scarce target-domain data yield noisy gradient estimates. We address this challenge by complementing limited high-fidelity (HF) target-domain data with abundant, cheap, but biased low-fidelity (LF) data, e.g., from a simplified simulator. Most existing methods directly optimize biased objectives based on LF data. In contrast, the recently introduced multi-fidelity policy gradient (MFPG) framework uses LF data solely to construct a control variate that reduces variance and improves HF data efficiency without biasing the policy gradient estimator. However, published work on MFPG is limited to REINFORCE on small-scale simulation tasks. We develop MFPG for modern actor-critic learning in GPU-parallel simulation and on a physical robot. Our analysis and experiments show that naive extensions to proximal policy optimization (PPO) can lose cross-fidelity correlation or inflate variance. Our MFPG-PPO addresses these failures by redesigning the sampling, advantage estimation, and control variate construction to preserve cross-fidelity correlation, and by monitoring estimator uncertainty to prevent variance inflation. We also introduce a budget-aware MFPG-PPO to divide a fixed sampling budget among high- and low-fidelity data sources. Across simulated robot locomotion tasks of varying LF-to-HF transfer difficulty and HF data budgets, MFPG-PPO improves upon PPO trained on HF data alone in nearly all settings, and consistently matches the performance of PPO trained with 16x more HF data on the hardest task at the smallest HF budgets. In contrast, most baselines that use LF data perform well only where direct LF-to-HF transfer succeeds. MFPG-PPO enables stable learning on a physical Franka arm using only 4 real-robot episodes per update and no human demonstrations.

Figures & tables

Figure 1: In a simulated humanoid robot locomotion task on rough terrain, MFPG - PPO (orange) scales monotonically with the HF data budget and, where PPO trained on HF data alone (blue) collapses, matches the mean return of PPO trained with 16×16\times more HF data. HF data budgets are set by the number of parallel HF training environments and expressed as fractions of the Isaac Lab default (4096 environments). At each HF budget, MFPG - PPO uses the same amount of HF data as PPO HF -Only, plus abundant LF data from a simplified simulator. PPO LF -Only (gray) trains on abundant LF data alone and is evaluated in the HF environment. Bars show mean HF evaluation returns of final policies across 6 seeds, with 95%95\% seed-bootstrapped confidence intervals. Dots denote individual seeds. The snapshots show the MFPG - PPO policy walking across the terrain, whereas the PPO HF -Only policy remains on its knees despite using 4×4\times more HF training data.
Figure 2: MFPG - PPO collects three batches of transitions. Representative HF , coupled LF , and uncoupled LF rollouts are shown with red circles, filled teal circles, and hollow teal squares. (a) When HF and LF are simulations of different accuracy, each batch contains parallel TT -step rollouts per iteration. (b) When the HF environment is a robot collecting HH -step episodes ( H>TH>T ), coupled LF simulation rollouts restart from recorded HF states every TT steps. In both cases, the LF simulation runs only parallel TT -step rollouts, while the coupled LF GAE horizon matches the HF horizon, TT in (a) and HH in (b). In (b), we stitch the coupled LF rollouts of each episode for GAE .
Figure 3: MFPG - PPO is the only method in our comparison that improves upon PPO HF -Only in nearly every setting, is never distinguishably worse, and scales monotonically with more HF data. Bars denote differences in mean HF evaluation return relative to PPO HF -Only. Error bars denote 95%95\% bootstrapped confidence intervals for the mean of the difference across 6 seeds.
Figure 4: Neither limited real-world data nor abundant simulation data alone yield a good real-robot policy, whereas MFPG - PPO succeeds by combining them. Bars and curves denote means across 3 seeds. Whiskers and shaded bands show their ranges. (a) Final success rate of the deterministic policy over 6060 real-robot episodes ( 2020 per seed). Horizontal lines denote the shared warm-start checkpoint performance and that of a simulation-trained policy well past convergence ( 100%100\% simulation success), each evaluated over 4040 real-robot episodes. (b) Training success rate on the robot (solid lines denote running averages over 40 episodes per seed) and in uncoupled LF simulation (dashed lines). (c) Pearson correlation between the coupled LF and HF paired statistics, computed on episode sums (4 pairs) and per step (240 pairs), averaged over the past 10 iterations.
Figure 5: Curves and bars denote means over 6 seeds. Bands and whiskers denote seed-bootstrapped 95%95\% confidence intervals for the means. (a) HF training return on H1 Rough. (b) Fraction of training iterations whose variance ratio \mathrm{Var}[{\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}}+c\,({\color[rgb]{0,0,0}X^{\mathrm{low}}_{k,t}}-{\color[rgb]{0,0,0}\hat{\mu}^{\mathrm{low}}})]\,/\,\mathrm{Var}[{\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}}] , averaged over minibatches before the monitoring check, exceeds 11 , 1010 , and 10001000 at the 18×\frac{1}{8}\times HF budget.
Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

HyperparameterSimulationReal robot
EMA rate α\alpha0.0020.002 (FR); 0.0250.025 (BA)0.050.05
Bootstrap resamples10310^{3}—
Evidence-window length MM—44
Confidence level 1−δ1-\delta0.90.90.90.9
Appendix
Table 1: Control variate monitoring hyperparameters.
MethodCandidate values
Backbone parameters
PPOLearning rate {5×10−4,10−3,2×10−3}\{5\times 10^{-4},10^{-3},2\times 10^{-3}\} ; epochs {3,5,8}\{3,5,8\} ; minibatches per epoch {2,4,8}\{2,4,8\}
REINFORCELearning rate {7×10−5,2×10−4,7×10−4,2×10−3,7×10−3}\{7\times 10^{-5},2\times 10^{-4},7\times 10^{-4},2\times 10^{-3},7\times 10^{-3}\} ; discount factor {0.95,0.97,0.99}\{0.95,0.97,0.99\}
SACActor and critic learning rate {10−4,2×10−4,5×10−4}\{10^{-4},2\times 10^{-4},5\times 10^{-4}\} ; UTD ratio {8,16,32}\{8,16,32\} or 200200 minibatches per iteration; target-entropy scale {0.05,0.167,0.5}\{0.05,0.167,0.5\}
Method-specific parameters
MFPG - PPO -FREMA rate {0.0005,0.002,0.01}\{0.0005,0.002,0.01\}
Appendix
Table 2: Grid values for simulation hyperparameter search.
PPO backbone hyperparameters inherited from Isaac Lab’s task-default RSL-RL configurations
ParameterValue
Rollout length per environment24 steps
OptimizerAdam
Discount factor / GAE parameter0.990.99 / 0.950.95
PPO clipping parameter0.20.2
Value-loss coefficient / value clipping11 / enabled
Appendix
Table 3: Simulation hyperparameters.
ParameterValue
Shared
Initial learning rate4×10−54\times 10^{-5}
Epochs4
Minibatches per epoch4
HF episodes per update4
OptimizerAdam
Appendix
Table 4: Real-robot hyperparameters.
MethodHFCoupled LFUncoupled LF
PPO HF -OnlyNhighN^{\mathrm{high}}——
PPO LF -Only——40964096
MFPG - PPO -FRNhighN^{\mathrm{high}}NhighN^{\mathrm{high}}40964096
MFPG - PPO -BAmhighm^{\mathrm{high}}mhighm^{\mathrm{high}}mlow−mhighm^{\mathrm{low}}-m^{\mathrm{high}}
Co-training, DARC, PARNhighN^{\mathrm{high}}—4096+Nhigh4096+N^{\mathrm{high}}
MFPG -REINFORCENhighN^{\mathrm{high}}NhighN^{\mathrm{high}}40964096
Appendix
Table 5: Simulation rollouts per iteration. Rollouts contain 24 steps, except for MFPG -REINFORCE (100 steps).
Figure 6: Locomotion collision models. (a) Unitree H1 and (b) ANYmal-D. Each panel compares the full HF collision model (left) with the simplified LF collision model (right). Green outlines indicate the collision geometry.
 H1ANYmal-D
ParameterHFLFHFLF
Physics time step (s)0.0020.0020.010.010.0020.0020.00250.0025
Physics steps per action102108
Position solver iterations4242
Velocity solver iterations4000
Self-collisionsEnabledDisabledEnabledDisabled
Appendix
Table 6: Locomotion fidelity settings, shared by the flat and rough tasks of each robot.
Figure 7: Franka cube-picking environments. The physical Franka setup (left) provides the HF environment, and ManiSkill simulation (right) provides the LF environment.
ObservationDimensions
Arm joint positions and velocities7+77+7
Gripper width1
End-effector position and quaternion3+43+4
Cube position and quaternion3+43+4
Goal position3
End-effector-to-cube and cube-to-goal vectors3+33+3
Appendix
Table 7: State observations for the Franka cube-picking task. Here, φ\varphi is cube yaw and tt counts elapsed control steps.
Figure 8: Mean final HF evaluation returns for the task-budget settings in . Error bars denote 95%95\% bootstrap confidence intervals across 6 seeds.
Figure 9: Fine-tuning after extensive LF pretraining on H1 Rough. Both methods start from each seed’s final PPO LF -Only checkpoint, whose mean return the gray bar shows. HF budgets are expressed as fractions of 4096 parallel environments. Bars show HF evaluation returns over 2048 episodes per seed, averaged across 6 seeds. Whiskers show 95%95\% seed-bootstrapped confidence intervals for the means.
Figure 10: Percentage of minibatches on which the control variate was applied during Franka cube-picking training in . Each marker aggregates the past 10 iterations. The line is the mean over 3 seeds, the band their range.
Figure 11: Cross-fidelity event timing and correlation between paired statistics. Left. Distribution of grasp-event timing differences between the physical Franka robot and its coupled ManiSkill simulation. The physical robot generally grasps later than its simulated counterpart due to fidelity gaps, causing low correlation between paired per-step statistics. Middle. Distribution of foot-strike timing differences between high- and low-fidelity simulations of the H1 Rough task in Isaac Lab. The timing generally aligns more closely than in the Franka case. Right. Cross-fidelity correlations between paired HF and LF statistics across the tasks in and . The first four x-axis entries are Isaac Lab simulation tasks and the last is the physical robot task. Each marker gives a run’s median correlation over training. Gray triangles denote per-step correlations, orange circles denote episode-sum correlations, and black bars denote medians across runs. A-D denotes ANYmal-D.
Figure 12: Event indicators (top) and paired per-step surrogate statistics ({\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}},{\color[rgb]{0,0,0}X^{\mathrm{low}}_{k,t}}) (bottom). Left. A Franka cube-picking episode. The upper bars indicate detected grasps, and the shaded interval marks the timing mismatch. Right. An H1 Rough rollout. The upper bars indicate left- and right-foot contact. In both panels, the lower curves show the HF and coupled LF surrogate statistics in red and teal, respectively.
Figure 13: Fraction of training iterations whose variance ratio \mathrm{Var}[{\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}}+c\,({\color[rgb]{0,0,0}X^{\mathrm{low}}_{k,t}}-{\color[rgb]{0,0,0}\hat{\mu}^{\mathrm{low}}})]\,/\,\mathrm{Var}[{\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}}] , averaged over minibatches before the monitoring check, exceeds 11 , 1010 , and 10001000 for the 132×\frac{1}{32}\times HF budget case of .
Figure 14: Training diagnostics for the ablations in at the 18×\frac{1}{8}\times (left) and 132×\frac{1}{32}\times (right) HF budgets on H1 Rough. Rows show the variance ratio \mathrm{Var}[{\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}}+c\,({\color[rgb]{0,0,0}X^{\mathrm{low}}_{k,t}}-{\color[rgb]{0,0,0}\hat{\mu}^{\mathrm{low}}})]\,/\,\mathrm{Var}[{\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}}] , the fraction of minibatches failing the monitoring checks, and the estimated correlation between the paired per-step statistics ({\color[rgb]{0,0,0}X^{\mathrm{high}}_{k,t}},{\color[rgb]{0,0,0}X^{\mathrm{low}}_{k,t}}) . Curves show seed means and bands show 95%95\% bootstrapped confidence intervals for the means.
More in this topic

Policy Gradient Methods

Policy gradient methods optimize stochastic policies by estimating how parameter changes affect expected reward.

See all 164 papers

Explore similar work

CardsList
Mar 4, 2026

GIPO: Gaussian Importance Sampling Policy Optimization

Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation. However, RL remains limited by poor data efficiency, particularly in settings where interaction data are scarce and quickly become outdated. To address this challenge, GIPO (Gaussian Importance sampling Policy Optimization) is proposed as a policy optimization objective based on truncated importance sampling, replacing hard clipping with a log-ratio-based Gaussian trust weight to softly damp extreme importance ratios while maintaining non-zero gradients. Theoretical analysis shows that GIPO introduces an implicit, tunable constraint on the update magnitude, while concentration bounds guarantee robustness and stability under finite-sample estimation. Experimental results show that GIPO achieves state-of-the-art performance among clipping-based baselines across a wide range of replay buffer sizes, from near on-policy to highly stale data, while exhibiting superior bias--variance trade-off, high training stability and improved sample efficiency. Code is available at https://github.com/distanceLu/GIPO.

Chengxuan Lu, Zhenquan Zhang, Shukuan Wang +3

1IROOTECH TECHNOLOGY · 2Wolf 1069 b Lab, Sany Group

Reinforcement LearningImportance Sampling for RLRL Post-TrainingSample-Efficient RLOff-Policy RL
AskPDF
Jun 19, 2026

ReFPO: Reflow Regularization for Flow Matching Policy Gradients

We present Reflow-regularized Flow Matching Policy Gradients (ReFPO), a simple online RL method that adds explicit Reflow regularization to FPO for efficient flow-based control. We uncover a key structural property: the gradient updates in Flow Matching Policy Gradients (FPO) can be interpreted as an implicit advantage-weighted Reflow process, providing a new geometric perspective on flow-based policy gradients. Building on this insight, ReFPO introduces an explicit geometric regularizer that can be implemented with a single line of code change without incurring additional computational overhead or auxiliary distillation stages. By synergizing advantage-guided updates with path rectification, our method reduces CFM proxy-ratio spikes, stabilizes PPO-style training, and enables high-fidelity one-step inference that often matches or exceeds multi-step performance. We experimentally demonstrate that ReFPO improves average performance and discretization robustness across GridWorld, MuJoCo Playground, and high-dimensional Humanoid Control tasks, providing a scalable and stable approach for generative policies in complex physical simulations.

Ge Wang, Yibo Peng, Fan Feng +10

Ising AI · CUHK-Shenzhen · FNii-Shenzhen

Policy GradientReinforcement LearningRL for Generative ModelsRL for Flow Matching Models
AskPDF
Jul 9, 2026

SafeExplorer: An Unbiased Mixed-Policy Gradient for Reinforcement Learning with Recovery Interventions

Training agents directly on the task they will be deployed on often leads to the best policy performance, but training on physical robots can cause damage through falls, so minimizing training falls is critical. A standard mitigation hands control to a separate recovery policy whenever the agent leaves a designer-specified safe region. The resulting mixed-policy rollouts bias the on-policy update, and importance sampling cannot remove this bias when the recovery policy is deterministic. We propose SafeExplorer, a drop-in modification of proximal policy optimization (PPO) built on an unbiased policy-gradient estimator of the mixed-policy return. The estimator uses the score function only inside the safe region and never evaluates the recovery policy's density, so deterministic recovery policies pose no problem. On final reward, the estimator matches or outperforms truncated importance sampling, which is itself biased for that same gradient. SafeExplorer also speeds up learning near the safe-region boundary with a closed-form critic target at recovery-triggering states when dynamics and recovery are deterministic, and an imitation loss that copies recovery actions only from successful recoveries. On three simulated locomotion environments with ten seeds each, and given an already available recovery controller, SafeExplorer incurs 255×255\times, 43×43\times, and 21×21\times fewer training falls than standard PPO on HalfCheetah, Ant, and Unitree Go1. SafeExplorer also matches or exceeds the final reward of PPO. On Ant, where 8.7% of recovery interventions end in a fall, over 40×40\times the HalfCheetah rate, SafeExplorer reaches the success threshold on every seed and none of the five baselines reaches the threshold on any seed.

Elham Daneshmand, Majid Khadiv, Glen Berseth +1

McGill University & Mila · Technical University of Munich · Université de Montréal & Mila

Robot SafetyProximal Policy OptimizationSafe RLPolicy Learning
AskPDF