cs.ROFeb 2, 2026
SaveFlow Policy Gradients for Robot Control
Organizations: Amazon FAR · UC Berkeley · Stanford · HKU · CMU
Abstract
Likelihood-based policy gradient methods are the dominant approach for training robot control policies from rewards. These methods rely on differentiable action likelihoods, which constrain policy outputs to simple distributions like Gaussians. In this work, we show how flow matching policy gradients, a recent framework that bypasses likelihood computation, can be made effective for training and finetuning more expressive policies in challenging robot control settings. We introduce an improved objective that enables success in legged locomotion, humanoid motion tracking, and manipulation tasks, as well as robust sim-to-real transfer on two humanoid robots. We then present ablations and analysis on training dynamics. Results show how policies can exploit the flow representation for exploration when training from scratch, as well as improved finetuning robustness over baselines.
Figures & tables
Figure 1: Flow policies for robot control. We show how robust control policies for quadrupeds, humanoids, and manipulators can be trained with reinforcement learning.
Figure 2: Improved stability in IsaacLab locomotion environments. We compare episode returns from FPO++ training with FPO; FPO++ curves show the mean over 5 random seeds. For thoroughness and to ensure we are not comparing against a bad set of hyperparameters for the FPO baseline, we include FPO runs for many different hyperparameter choices: all combinations of learning rate , clip parameter , and Monte Carlo samples . FPO++ addresses stability problems that we were unable to solve by tuning FPO hyperparameters.
Figure 3: Sim-to-real transfer. We deploy flow policies for locomotion to a Booster T1 and motion tracking to a Unitree G1. Policies are directly deployed to real robots with reduced sampling step counts, demonstrating stable gaits, tracking for long sequences, and robustness to external forces. The arrows in T1 locomotion (Figure 3a ) indicate velocity commands, while the arrows in the robustness tests (Figure 3c ) highlight external forces.
Figure 4: FPO++ ablations on locomotion. We plot final training and evaluation returns across many runs; each point is a training run with clipping parameter and random seed . (a) compares FPO++ with per-sample ratios (Equation 9 ) against per-action ratios from prior work (Equation 8 ); per-sample ratios produce higher and more consistent returns. (b) compares trust region implementations: standard PPO clipping, SPO [ 33 ] , and our asymmetric ASPO objective.
Figure 5: Manipulation finetuning results. We compare the evaluation success rates for FPO++, FPO, and our two DPPO implementations. We show two rows: the first row contains policy success rates with zero-sampling ( ), while the second row contains policy success rates using standard random sampling ( ). For each task, all algorithms are initialized from the same base policy, which is an image-based flow matching policy trained to predict action chunks.
Figure 6: Correlations before and during training. We sample 10,000 actions conditioned on the same observation, and visualize the correlations between action dimensions before and during FPO++ training for an H1 locomotion policy.
Figure A.1: Legged robots used for experiments. We train policies for the Go2, Spot, H1, and G1 robots in simulation. We deploy policies to physical G1 and Booster T1 robots.
Figure A.2: Manipulation tasks. We compare FPO++ to baseline methods on manipulation tasks from RoboMimic [ 24 ] and DexMimicGen [ 43 ] . We choose tasks that cover diverse embodiments and control regimes: from single-arm to bimanual manipulation, parallel-jaw grippers and dexterous hands, and both short- and long-horizon tasks.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter category | Used value | Sweep range and notes |
|---|---|---|
| FPO++ | ||
| Flow integration steps | ||
| Network output | , denotes data | |
| Samples per action | ||
| Training | ||
| Learning rate | , , | |
Table A.1: FPO++ and training hyperparameters used for locomotion. We report both the final values we use for experiments and the values we performed sweeps over.
| Hyperparameter category | Used value | Sweep range and notes |
|---|---|---|
| FPO++ | ||
| Flow integration steps | ||
| Network output | , denotes data | |
| Samples per action | ||
| Training | ||
| Learning rate | ||
Table A.2: FPO++ and training hyperparameters used for motion tracking. We report both the final values we use for experiments and the values we performed sweeps over.
Figure A.3: Clamping CFM losses improves stability. The x-axis varies the Huber threshold ; “Off” uses squared CFM loss without Huber. Loss-difference clamping is enabled throughout: orange uses naive clamping, and blue preserves gradients through the clamp. Panels (a) and (b) respectively disable and enable clamping of the old- and new-policy CFM losses before subtraction. Bars show mean final training return with standard error over five seeds.
Figure A.4: Motion tracking training curves. Vanilla FPO fails in learning to track the motion, while FPO++ trains stably. Hyperparameters are shared between both runs.
Figure A.5: Sample efficiency for FPO++ and our Gaussian PPO baseline. We compare policy returns between Gaussian PPO and FPO++ with environment counts after 1500 updates for quadrupeds and 2000 updates for humanoids. Minibatch size per policy update increases linearly with the number of parallel environments; other hyperparameters are fixed between runs. Filled regions show standard deviation computed over 5 seeds.
Figure A.6: Different gaits from different algorithms. Rollouts using Gaussian PPO (top) and FPO++ (bottom) for 1500 policy updates. Policies are trained with the same rewards, 4096 parallel environments, and given the same forward velocity command (1 m/s). We found differences reproducible across seeds, clip parameters, and learning rates. We use Viser [ 67 ] for visualization.
Figure A.7: Training curves compared to Gaussian PPO for motion tracking. Blue denotes FPO++ and orange Gaussian PPO. Without entropy regularization or adaptive learning rates, FPO++ achieves higher final returns and longer episodes. With both components enabled, Gaussian PPO attains higher returns, while episode lengths are similar.
Figure A.8: Additional Manipulation Comparison. Finetuning success rates on the RoboMimic Can and Square tasks for flow-based RL methods. All methods start from pretrained behavior-cloning base policies; FPO++ and vanilla FPO share the same base policy, while DPPO and ReinFlow use architectures and implementation from their original papers. Please refer to the base policy setting written in the text for further details. FPO++ learns fastest and attains the highest final success on both tasks, while vanilla FPO, DPPO, and ReinFlow remain stable but underperform relative to FPO++.
Figure A.9: Analysis of base policy quality in RoboMimic Can. This figure illustrates how the initial success rate of the base policy impacts different finetuning methods. The top row shows results for a base policy with 71.35% success under zero-sampling versus 10.00% under random sampling, while the bottom row depicts a higher-quality base policy (96.11% zero-sampling / 64.36% random sampling), demonstrating that DPPO variants are significantly more sensitive to the base policy quality.
Figure A.10: Ablation study in manipulation tasks. Blue denotes per-sample ratios with PPO, light gray per-action ratios with PPO, and dark gray per-sample ratios with ASPO. Results are averaged over three training seeds per configuration, with 200 evaluation episodes per checkpoint per seed. Per-sample PPO achieves the highest final mean success on both Square and Threading. ASPO underperforms per-sample PPO in both tasks, despite its benefits for locomotion.
Figure A.11: ASPO sensitivity to the reward regime in Lift. PPO and ASPO each use their best-tuned configuration for each reward setting, evaluated over ten training seeds. Success uses zero-sampling and the task-completion criterion.
Figure A.12: Algorithmic updates in FPO++ reduce gradient variance. We observe that the per-sample ratio and ASPO trust region result in higher cosine similarity between gradients computed within each policy update, especially during the second half of training. Averages and standard deviations are reported over 5 seeds.
Figure A.13: Effect of ASPO on entropy. We show a flow field during training of H1 locomotion for a single robot joint, with FPO++ using PPO clipping (left) and ASPO clipping (right). The x-axis is integration step .
Figure A.14: FPO++ with PPO clipping at peak performance . Policy flow field density for the Unitree H1 humanoid trained using the standard PPO trust region, captured at the point of maximum average return. The flow is still relatively broad, indicating adequate exploration.
Figure A.15: FPO++ with PPO clipping during policy collapse after convergence. Policy flow field density for the Unitree H1 humanoid trained using the standard PPO trust region, captured as the average return begins to collapse. The action distribution has narrowed significantly for many joints, demonstrating the entropy collapse that leads to instability and performance degradation.
Figure A.16: FPO++ with ASPO objective at intermediate training stage. Policy flow field density for the Unitree H1 humanoid trained using the entropy-preserving ASPO objective, captured at an intermediate stage where the average return matches the peak average return of FPO++ with the standard PPO trust region. This distribution is already wider and more exploratory than the PPO-clipped policy at a similar or later time step, contributing to stable training.
Figure A.17: FPO++ with ASPO objective at converged state. Policy flow field density for the Unitree H1 humanoid trained using the ASPO objective, captured at its final, high-reward converged state. The distribution remains wide and exploratory, confirming that ASPO effectively preserves entropy and prevents the collapse observed with PPO clipping.
| Sampling method | 5 steps | 50 steps |
|---|---|---|
| Random sampling | ||
| Zero-sampling |
Table A.3: Zero-sampling improves motion tracking. Returns are computed over 100 rollouts of a 2.5-minute dancing sequence, with domain randomization and push perturbations. The policy is trained with 50 flow steps; we evaluate using random sampling and zero-sampling with 5 and 50 flow steps.