cs.AINov 17, 2026

Outcome-Balanced PPO and Policy-Induced Flow Rectification for Flow-Based VLA Policies

Authors: Yuhan Wang, Ruiyang Wang, Jiaguan Zhu

Organizations: Tsinghua University · The Hong Kong University of Science and Technology

Abstract

Reinforcement learning (RL) enables post-training of flow-based vision-language-action (VLA) policies by optimizing expected task return through environment interaction. However, this optimization raises two challenges: policy updates can alter the velocity-field geometry underlying the score approximation used for stochastic transitions, and sparse multi-task rollouts can yield uneven learning signals across tasks and advantage signs. We introduce an RL framework that combines Outcome-Balanced Proximal Policy Optimization (PPO) with policy-induced flow rectification. Outcome-Balanced PPO reweights positive advantages using task success-rate estimates and rescales positive and negative advantages while preserving their signs. Policy-induced rectification periodically fits a linear-bridge flow to ODE endpoint pairs generated by the current policy, refreshing the representation used in subsequent stochastic optimization. We evaluate the framework on simulated manipulation benchmarks and additionally study a component combination in a Real2Sim2Real task, examining task performance, optimization behavior, and flow consistency. The proposed framework improves success rates over the evaluated baselines across the simulated benchmarks.

Figures & tables

Explore similar work

CardsList
  1. FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization

    Jun 3, 2026Yihao Wu, He Zhang, Junbo Tan +2Offline RLVision-Language-Action Models

  2. RedFlow: Redirect Failure into Action-level Corrections for Flow-matching VLA Policy

    Jul 30, 2026Zhengyang Yan, Junhao Li, Fangqi Zhu +6Robot Policy LearningPolicy Optimization

  3. Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models

    Oct 11, 2025Mingyang Lyu, Yinqian Sun, Erliang Lin +4Flow MatchingRobotic Manipulation