Flow-matching policies adjust inference cost through the number of solver steps, but reinforcement learning at a single fixed solver budget can hide degradation at cheaper deployment budgets and even erase useful one-step behavior present in a pretrained policy. We introduce Action-Ratio-Clip Flow (ArcFlow), a deployment-aware algorithm for training one flow policy across a specified range of solver budgets. ArcFlow combines action-aligned supervision, budget exposure, and worst-budget checkpoint selection. For reconstruction objectives, its bridge aggregates temporal probes into one surrogate ratio and one PPO-style clip per action, with a boundary residual exactly equal to deployed one-step reconstruction error. Cycling exposes the shared field to the deployment range, and selection optimizes its performance floor. For native velocity objectives, multi-budget endpoint supervision applies the same principle without replacing the native update. Both realizations preserve the architecture and deployment-time cost. Across DM Control, manipulation, and locomotion, ArcFlow attains the highest mean one-step return among five flow-policy baselines on 25 of 30 tasks under each method's reported training settings. Component studies identify complementary improvements from boundary supervision, the fixed-grid bridge, budget exposure, and deployment selection. These results establish deployment-budget-aware training as an effective route to strong low-step control with one policy serving multiple inference costs.
Figures & tables
Figure 1: ArcFlow for multi-budget deployment. Left: the solver budget K controls deployment cost. Right: the bridge-based control realization uses fresh noise and the observed action; at the boundary, the residual norm equals one-step reconstruction error. Probes are averaged before one surrogate ratio and PPO clip per action. Rollout budgets cycle, and checkpoint selection maximizes the worst validation return over K∈{1,4,16} .
Figure 2: ArcFlow strengthens one-step deployment across the task census and solver budgets. (a) Within-task normalized one-step return on all 30 retained tasks from the registered selection results; dots mark a one-to-sixteen-step mean-return ratio below 0.5 . The right band shows nine Playground tasks on the display regimes of Table 1 . (b) Mean normalized return versus solver steps over all 30 tasks for every method: 21 held-out DM Control tasks and 9 Playground tasks on their registered display regimes.
Task
FPO
FPO++
DPPO
ReinFlow
PolicyFlow
ArcFlow
K
1
4
16
1
4
16
1
4
16
1
4
16
1
4
16
1
4
16
DM Control
BallInCup
738
856
856
656
852
841
683
647
607
735
850
850
935
937
935
937
949
949
CartpoleBalance
611
695
732
487
793
816
668
659
631
772
999
891
522
496
492
956
959
953
CartpoleSwingupSparse
207
522
495
201
421
423
12.1
12.3
11.2
274
286
263
4.53
4.94
4.67
284
300
299
CheetahRun
409
894
895
156
827
813
469
457
437
410
728
706
672
677
677
849
868
859
Table 1: Cross-domain evaluation at three deployment budgets. Deployed episodic return for ArcFlow and five flow-policy baselines on 21 DM Control, three manipulation, and six locomotion tasks. Bold marks the highest displayed mean for each task and budget.
Figure 3: One-step learning dynamics across 30 tasks. Mean raw episodic return at K=1 , ordered as in Table 1 . Training progress is the percentage of each method’s registered interaction budget. Curves use the registered seeds. Axis ranges are task-specific; compare within panels.
Per-probe exponential ratios; the shared PPO objective replaces the release’s ASPO objective.
DPPO (flow adaptation)
Gaussian-noised Euler transitions and denoising-MDP likelihood ratios; corrected coordinate reduction and denoising discount. Trained from scratch on the shared flow backbone.
ReinFlow
Stochastic-chain exploration model with the release-style noise-ceiling schedule restored; shared PPO clipping and backbone.
PolicyFlow
Velocity-based surrogate with audited Brownian regularization, entropy, and optimizer settings.
Appendix
Table 4: Implementation scope of the shared-backbone baselines.
Figure 4: A fixed energy scale stabilizes the numerical weight across collecting budgets. Clip fraction, log-weight spread, and bridge energy versus collecting K ; medians and quartiles over the selection-pool iterations. The collecting-grid 1/K weight changes with budget; both fixed- 1/4 controls keep the energy scale comparable.
Expressive flow policies can represent multiple high-value actions, but sampling them with few integration steps while retaining tractable action densities remains challenging. We introduce DEFlow, a deep equilibrium flow policy that replaces explicit Euler sampling with implicit Euler integration and solves all intermediate actions jointly as an equilibrium system. We combine implicit differentiation with a discrete change-of-variables formula for the final-action density, enabling off-policy training within soft actor-critic without backpropagating through solver iterations. DEFlow achieves an average relative improvement of approximately 10% in mean successful coverage over the strongest baseline across four Multi-Goal docking variants. It also achieves the highest mean time-averaged return on four of five manipulation tasks, while remaining broadly competitive across five MuJoCo tasks. These results support implicit flow policies for tasks requiring both diversity and precision, at the cost of additional nonlinear and linear solves. Primary
Flow-based policies provide an expressive framework for continuous robot control, but their iterative ODE integration incurs substantial inference cost. Naively reducing the integration budget can severely degrade control, since policies optimized under full-step execution are not explicitly constrained to remain reliable under coarse numerical integration. We refer to this mismatch as the few-step discretization gap. To address this problem, we introduce RFPO, a flow-policy optimization framework for reliable few-step execution. Reward-aware online Reflow rectifies student-induced transport paths during on-policy learning, making the resulting policy more robust to coarse integration. A frozen Gaussian PPO controller supplies complementary action-space supervision at full and intermediate integration budgets, while the deployed policy remains a single flow student executed with one Euler step. Across Unitree Go2, Boston Dynamics Spot, Unitree H1, and Unitree G1, RFPO consistently preserves full-step control performance under one-step execution, with one-step returns remaining within 2.4% of their corresponding 64-step values across both zero and random initialization. On Unitree Go2, one-step execution retains 98.5% of the 64-step reward while reducing onboard mean inference latency from 4.39 ms to 0.08 ms, yielding a 54.9x speedup. Real-robot experiments further validate stable one-step locomotion. Code: https://github.com/AIGeeksGroup/RFPO. Website: https://aigeeksgroup.github.io/RFPO.
Ting Huang, Lisiyu Pan, Haoyu Wang +6
School of Computer Science, Peking University · AI2 Robotics
Flow matching is now the dominant approach for vision-language action (VLA) robot policies, thanks to its expressiveness and simple training objective. However, inference for standard flow matching requires iteratively solving an ODE, limiting the control frequency at deployment. While several works have proposed novel methods to allow for single-shot action generation, we ask whether such additional objectives are actually necessary. We find that across six flow matching VLAs, one-step action generation largely matches multi-step sampling at scale: multi-step integration confers little to no consistent advantage at inference. We empirically find that this holds as long as models are sufficiently trained and have enough capacity. Practically, the implication is that for large VLAs, the single-step policy comes essentially for free from the multi-step model, enabling low-latency deployment without retraining, distillation, or auxiliary objectives.