Likelihood-based policy gradient methods are the dominant approach for training robot control policies from rewards. These methods rely on differentiable action likelihoods, which constrain policy outputs to simple distributions like Gaussians. In this work, we show how flow matching policy gradients, a recent framework that bypasses likelihood computation, can be made effective for training and finetuning more expressive policies in challenging robot control settings. We introduce an improved objective that enables success in legged locomotion, humanoid motion tracking, and manipulation tasks, as well as robust sim-to-real transfer on two humanoid robots. We then present ablations and analysis on training dynamics. Results show how policies can exploit the flow representation for exploration when training from scratch, as well as improved finetuning robustness over baselines.
Figures & tables
Figure 1: Flow policies for robot control. We show how robust control policies for quadrupeds, humanoids, and manipulators can be trained with reinforcement learning.
Figure 2: Improved stability in IsaacLab locomotion environments. We compare episode returns from FPO++ training with FPO; FPO++ curves show the mean over 5 random seeds. For thoroughness and to ensure we are not comparing against a bad set of hyperparameters for the FPO baseline, we include FPO runs for many different hyperparameter choices: all combinations of learning rate ∈{10−5,10−4,3×10−4} , clip parameter ∈{0.04,0.05,0.06} , and Monte Carlo samples ∈{8,16,32} . FPO++ addresses stability problems that we were unable to solve by tuning FPO hyperparameters.
Figure 3: Sim-to-real transfer. We deploy flow policies for locomotion to a Booster T1 and motion tracking to a Unitree G1. Policies are directly deployed to real robots with reduced sampling step counts, demonstrating stable gaits, tracking for long sequences, and robustness to external forces. The arrows in T1 locomotion (Figure 3a ) indicate velocity commands, while the arrows in the robustness tests (Figure 3c ) highlight external forces.
Figure 4: FPO++ ablations on locomotion. We plot final training and evaluation returns across many runs; each point is a training run with clipping parameter ∈{0.04,0.05,0.06} and random seed ∈{0,1,2,3,4} . (a) compares FPO++ with per-sample ratios (Equation 9 ) against per-action ratios from prior work (Equation 8 ); per-sample ratios produce higher and more consistent returns. (b) compares trust region implementations: standard PPO clipping, SPO [ 33 ] , and our asymmetric ASPO objective.
Figure 5: Manipulation finetuning results. We compare the evaluation success rates for FPO++, FPO, and our two DPPO implementations. We show two rows: the first row contains policy success rates with zero-sampling ( ϵ=0 ), while the second row contains policy success rates using standard random sampling ( ϵ∼N(0,I) ). For each task, all algorithms are initialized from the same base policy, which is an image-based flow matching policy trained to predict action chunks.
Figure 6: Correlations before and during training. We sample 10,000 actions conditioned on the same observation, and visualize the correlations between action dimensions before and during FPO++ training for an H1 locomotion policy.
Figure A.1: Legged robots used for experiments. We train policies for the Go2, Spot, H1, and G1 robots in simulation. We deploy policies to physical G1 and Booster T1 robots.
Figure A.2: Manipulation tasks. We compare FPO++ to baseline methods on manipulation tasks from RoboMimic [ 24 ] and DexMimicGen [ 43 ] . We choose tasks that cover diverse embodiments and control regimes: from single-arm to bimanual manipulation, parallel-jaw grippers and dexterous hands, and both short- and long-horizon tasks.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter category
Used value
Sweep range and notes
FPO++
Flow integration steps
64
{8,16,32,64}
Network output
u
{u,x0} , x0 denotes data
Samples per action
16
{8,16,32}
Training
Learning rate
1×10−4
1×10−5 , 1×10−4 , 3×10−4
Appendix
Table A.1: FPO++ and training hyperparameters used for locomotion. We report both the final values we use for experiments and the values we performed sweeps over.
Hyperparameter category
Used value
Sweep range and notes
FPO++
Flow integration steps
50
{10,50}
Network output
u
{u,x0} , x0 denotes data
Samples per action
16
{8,16,32}
Training
Learning rate
3×10−4
Appendix
Table A.2: FPO++ and training hyperparameters used for motion tracking. We report both the final values we use for experiments and the values we performed sweeps over.
Figure A.3: Clamping CFM losses improves stability. The x-axis varies the Huber threshold δ ; “Off” uses squared CFM loss without Huber. Loss-difference clamping is enabled throughout: orange uses naive clamping, and blue preserves gradients through the clamp. Panels (a) and (b) respectively disable and enable clamping of the old- and new-policy CFM losses before subtraction. Bars show mean final training return with standard error over five seeds.
Figure A.4: Motion tracking training curves. Vanilla FPO fails in learning to track the motion, while FPO++ trains stably. Hyperparameters are shared between both runs.
Figure A.5: Sample efficiency for FPO++ and our Gaussian PPO baseline. We compare policy returns between Gaussian PPO and FPO++ with environment counts ∈{28,29,210,211,212} after 1500 updates for quadrupeds and 2000 updates for humanoids. Minibatch size per policy update increases linearly with the number of parallel environments; other hyperparameters are fixed between runs. Filled regions show standard deviation computed over 5 seeds.
Figure A.6: Different gaits from different algorithms. Rollouts using Gaussian PPO (top) and FPO++ (bottom) for 1500 policy updates. Policies are trained with the same rewards, 4096 parallel environments, and given the same forward velocity command (1 m/s). We found differences reproducible across seeds, clip parameters, and learning rates. We use Viser [ 67 ] for visualization.
Figure A.7: Training curves compared to Gaussian PPO for motion tracking. Blue denotes FPO++ and orange Gaussian PPO. Without entropy regularization or adaptive learning rates, FPO++ achieves higher final returns and longer episodes. With both components enabled, Gaussian PPO attains higher returns, while episode lengths are similar.
Figure A.8: Additional Manipulation Comparison. Finetuning success rates on the RoboMimic Can and Square tasks for flow-based RL methods. All methods start from pretrained behavior-cloning base policies; FPO++ and vanilla FPO share the same base policy, while DPPO and ReinFlow use architectures and implementation from their original papers. Please refer to the base policy setting written in the text for further details. FPO++ learns fastest and attains the highest final success on both tasks, while vanilla FPO, DPPO, and ReinFlow remain stable but underperform relative to FPO++.
Figure A.9: Analysis of base policy quality in RoboMimic Can. This figure illustrates how the initial success rate of the base policy impacts different finetuning methods. The top row shows results for a base policy with 71.35% success under zero-sampling versus 10.00% under random sampling, while the bottom row depicts a higher-quality base policy (96.11% zero-sampling / 64.36% random sampling), demonstrating that DPPO variants are significantly more sensitive to the base policy quality.
Figure A.10: Ablation study in manipulation tasks. Blue denotes per-sample ratios with PPO, light gray per-action ratios with PPO, and dark gray per-sample ratios with ASPO. Results are averaged over three training seeds per configuration, with 200 evaluation episodes per checkpoint per seed. Per-sample PPO achieves the highest final mean success on both Square and Threading. ASPO underperforms per-sample PPO in both tasks, despite its benefits for locomotion.
Figure A.11: ASPO sensitivity to the reward regime in Lift. PPO and ASPO each use their best-tuned configuration for each reward setting, evaluated over ten training seeds. Success uses zero-sampling and the task-completion criterion.
Figure A.12: Algorithmic updates in FPO++ reduce gradient variance. We observe that the per-sample ratio and ASPO trust region result in higher cosine similarity between gradients computed within each policy update, especially during the second half of training. Averages and standard deviations are reported over 5 seeds.
Figure A.13: Effect of ASPO on entropy. We show a flow field during training of H1 locomotion for a single robot joint, with FPO++ using PPO clipping (left) and ASPO clipping (right). The x-axis is integration step τ .
Figure A.14: FPO++ with PPO clipping at peak performance . Policy flow field density for the Unitree H1 humanoid trained using the standard PPO trust region, captured at the point of maximum average return. The flow is still relatively broad, indicating adequate exploration.
Figure A.15: FPO++ with PPO clipping during policy collapse after convergence. Policy flow field density for the Unitree H1 humanoid trained using the standard PPO trust region, captured as the average return begins to collapse. The action distribution has narrowed significantly for many joints, demonstrating the entropy collapse that leads to instability and performance degradation.
Figure A.16: FPO++ with ASPO objective at intermediate training stage. Policy flow field density for the Unitree H1 humanoid trained using the entropy-preserving ASPO objective, captured at an intermediate stage where the average return matches the peak average return of FPO++ with the standard PPO trust region. This distribution is already wider and more exploratory than the PPO-clipped policy at a similar or later time step, contributing to stable training.
Figure A.17: FPO++ with ASPO objective at converged state. Policy flow field density for the Unitree H1 humanoid trained using the ASPO objective, captured at its final, high-reward converged state. The distribution remains wide and exploratory, confirming that ASPO effectively preserves entropy and prevents the collapse observed with PPO clipping.
Sampling method
5 steps
50 steps
Random sampling
34.7±55.0
38.4±26.6
Zero-sampling
45.1±27.4
45.5±23.2
Appendix
Table A.3: Zero-sampling improves motion tracking. Returns are computed over 100 rollouts of a 2.5-minute dancing sequence, with domain randomization and push perturbations. The policy is trained with 50 flow steps; we evaluate using random sampling and zero-sampling with 5 and 50 flow steps.
Real-world reinforcement learning for robotic manipulation remains challenging, and this difficulty is amplified for flow matching policies: policy gradients must be backpropagated through time (BPTT) along the multi-step ODE that maps noise to actions, which is computationally prohibitive and numerically fragile. We propose FlowDPG, a DDPG-style method for flow matching policies that bypasses BPTT entirely. FlowDPG evaluates the critic gradient at a one-step estimate of the clean action and distills the resulting correction into the velocity field at training time, leaving multi-step inference unchanged. Intuitively, it combines two complementary vectors: the demonstration-driven velocity that keeps the action feasible, and the critic-driven correction that steers it toward higher value. Our contributions are threefold: (1) a BPTT-free distillation framework for stable DDPG-style improvement of flow matching policies, (2) a formal connection between the FlowDPG update and the vanilla deterministic policy gradient via three explicit approximations, and (3) real-world validation on two long-horizon tasks on different robots. FlowDPG reaches 92% end-to-end success on dual-arm AirPods assembly with Franka arms and an 86% rubric score on scrambled-egg cooking with YAM arms, substantially outperforming recent RL methods. Videos and more results: https://flowdpg.github.io.
Kexin Shi, Junyao Shi, Poorvi Hebbar +5
Skild AI · Carnegie Mellon University · University of Pennsylvania
Flow policies have become a standard policy class for learning robot behaviors from demonstrations, but reinforcement learning is still critical for improving pre-trained flow policies or learning them from scratch through interaction. We introduce QF3 (Fast Flow RL with Filtered Q-Gradients), an online off-policy RL algorithm that trains a flow policy with flow matching plus the critic's action gradient, backpropagated through a one-step prediction of the flow's output. To keep updates where the critic and this prediction are reliable, QF3 applies the critic gradient only to action dimensions that stay near the replay action. To our knowledge, QF3 is the first off-policy flow RL method to train humanoid locomotion policies from scratch and transfer them zero-shot to hardware. Paired with a high-throughput off-policy training recipe, it trains humanoid locomotion and motion-tracking policies with a 10x wall-clock speedup over FPO++, a recent on-policy flow RL method. We further apply QF3 to fine-tune pretrained flow-based manipulation policies on both ABC-Sim and Robomimic tasks. These results suggest that QF3 can both learn robot policies from scratch and refine those acquired from demonstrations. Website: https://qf3-rl.github.io/
Flow-matching policies can represent complex robot behaviors but remain susceptible to local errors under distribution shift at deployment. Many reinforcement learning approaches to policy improvement require reward signals that are difficult to specify or obtain in real-world manipulation. We present PreferenceFlow, a framework for improving a pretrained flow policy at test time without environment rewards or updates to the base policy. Human intervention chunks are paired with robot chunks generated from the same initial conditioning state to train a preference model. During infer- ence, we adopt the QGF sampling update, replacing its value gradient with the preference gradient evaluated at an estimated clean action. A gradient-cap loss penalizes excessive gradients on intervention pairs, while a zero-gradient loss discourages guidance near actions from expert demonstrations. On four real-world precision insertion tasks with a Franka robot, Pref- erenceFlow achieves a mean success rate of 90.5%, compared with 69% for the frozen policy. Ablations support the roles of gradient regularization and correctly ordered preference labels in the evaluated settings. These results demonstrate the utility of human interventions as local preference supervision for guiding frozen generative robot policies.