Outcome-Balanced PPO and Policy-Induced Flow Rectification for Flow-Based VLA Policies
Organizations: Tsinghua University · The Hong Kong University of Science and Technology
Abstract
Reinforcement learning (RL) enables post-training of flow-based vision-language-action (VLA) policies by optimizing expected task return through environment interaction. However, this optimization raises two challenges: policy updates can alter the velocity-field geometry underlying the score approximation used for stochastic transitions, and sparse multi-task rollouts can yield uneven learning signals across tasks and advantage signs. We introduce an RL framework that combines Outcome-Balanced Proximal Policy Optimization (PPO) with policy-induced flow rectification. Outcome-Balanced PPO reweights positive advantages using task success-rate estimates and rescales positive and negative advantages while preserving their signs. Policy-induced rectification periodically fits a linear-bridge flow to ODE endpoint pairs generated by the current policy, refreshing the representation used in subsequent stochastic optimization. We evaluate the framework on simulated manipulation benchmarks and additionally study a component combination in a Real2Sim2Real task, examining task performance, optimization behavior, and flow consistency. The proposed framework improves success rates over the evaluated baselines across the simulated benchmarks.
Figures & tables
| Method | LIBERO | ManiSkill | MetaWorld | CALVIN | Average |
|---|---|---|---|---|---|
| -SFT | 77.1 | 40.1 | 43.8 | 61.3 | 55.6 |
| 97.9 | 90.9 | 70.7 | 87.0 | 86.6 | |
| ReFPO | 92.3 | 86.5 | 69.5 | 85.7 | 83.5 |
| Ours | 98.4 | 95.0 | 73.4 | 89.0 | 88.9 |
| vs. | +0.5 | +4.1 | +2.7 | +2.0 | +2.3 |
| Benchmark | (Baseline) | + DAA | + Paired Adv. Norm. | + Periodic Rectification | vs. (p.p.) |
|---|---|---|---|---|---|
| ManiSkill | 90.9 | 93.2 | 94.5 | 95.0 | +4.1 |
| MetaWorld | 70.7 | 71.4 | 72.8 | 73.4 | +2.7 |
| CALVIN | 87.0 | 87.4 | 88.6 | 89.0 | +2.0 |
| LIBERO-Object | 100.0 | 100.0 | 100.0 | 100.0 | +0.0 |
| LIBERO-Goal | 98.8 | 99.0 | 99.6 | 99.6 | +0.8 |
| LIBERO-Spatial | 99.6 | 99.6 | 99.8 | 99.8 | +0.2 |
| Method | LIBERO | ManiSkill | MetaWorld | CALVIN | Average |
|---|---|---|---|---|---|
| Baseline + Joint Reflow (ReFPO-inspired) | 96.5 | 88.7 | 69.6 | 84.0 | 84.7 |
| Baseline + Periodic Rectification (Ours) | 98.1 | 92.4 | 71.8 | 87.9 | 87.6 |
| Method | LIBERO | ManiSkill | MetaWorld | CALVIN | Average |
|---|---|---|---|---|---|
| DAA + BAPO-inspired Clipping | 96.6 | 89.2 | 70.8 | 86.8 | 85.9 |
| DAA + Paired Norm. (Ours) | 98.3 | 93.6 | 72.9 | 88.5 | 88.3 |
| Threshold | LIBERO | ManiSkill | Mean Interval (PPO Iterations) |
|---|---|---|---|
| 98.2 | 94.9 | 4.6 | |
| 98.4 | 95.0 | 16.6 | |
| 97.4 | 94.0 | 25.9 |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,855 papers