ArcFlow: Training Flow Policies for Multiple Solver Budgets

Hieu Tran; Hong Yu

cs.LGOct 20, 2026

ArcFlow: Training Flow Policies for Multiple Solver Budgets

Authors: Hieu Tran, Hong Yu

Organizations: University of Massachusetts Amherst · University of Massachusetts Lowell

Abstract

Flow-matching policies adjust inference cost through the number of solver steps, but reinforcement learning at a single fixed solver budget can hide degradation at cheaper deployment budgets and even erase useful one-step behavior present in a pretrained policy. We introduce Action-Ratio-Clip Flow (ArcFlow), a deployment-aware algorithm for training one flow policy across a specified range of solver budgets. ArcFlow combines action-aligned supervision, budget exposure, and worst-budget checkpoint selection. For reconstruction objectives, its bridge aggregates temporal probes into one surrogate ratio and one PPO-style clip per action, with a boundary residual exactly equal to deployed one-step reconstruction error. Cycling exposes the shared field to the deployment range, and selection optimizes its performance floor. For native velocity objectives, multi-budget endpoint supervision applies the same principle without replacing the native update. Both realizations preserve the architecture and deployment-time cost. Across DM Control, manipulation, and locomotion, ArcFlow attains the highest mean one-step return among five flow-policy baselines on 25 of 30 tasks under each method's reported training settings. Component studies identify complementary improvements from boundary supervision, the fixed-grid bridge, budget exposure, and deployment selection. These results establish deployment-budget-aware training as an effective route to strong low-step control with one policy serving multiple inference costs.

Figures & tables

Figure 1: ArcFlow for multi-budget deployment. Left: the solver budget $K$ controls deployment cost. Right: the bridge-based control realization uses fresh noise and the observed action; at the boundary, the residual norm equals one-step reconstruction error. Probes are averaged before one surrogate ratio and PPO clip per action. Rollout budgets cycle, and checkpoint selection maximizes the worst validation return over $K\in\{1,4,16\}$ .

Figure 2: ArcFlow strengthens one-step deployment across the task census and solver budgets. (a) Within-task normalized one-step return on all 30 retained tasks from the registered selection results; dots mark a one-to-sixteen-step mean-return ratio below $0.5$ . The right band shows nine Playground tasks on the display regimes of Table 1 . (b) Mean normalized return versus solver steps over all 30 tasks for every method: 21 held-out DM Control tasks and 9 Playground tasks on their registered display regimes.

DM Control
Task	FPO			FPO++			DPPO			ReinFlow			PolicyFlow			ArcFlow
$K$	1	4	16	1	4	16	1	4	16	1	4	16	1	4	16	1	4	16
BallInCup	738	856	856	656	852	841	683	647	607	735	850	850	935	937	935	937	949	949
CartpoleBalance	611	695	732	487	793	816	668	659	631	772	999	891	522	496	492	956	959	953
CartpoleSwingupSparse	207	522	495	201	421	423	12.1	12.3	11.2	274	286	263	4.53	4.94	4.67	284	300	299
CheetahRun	409	894	895	156	827	813	469	457	437	410	728	706	672	677	677	849	868	859

Table 1: Cross-domain evaluation at three deployment budgets. Deployed episodic return for ArcFlow and five flow-policy baselines on 21 DM Control, three manipulation, and six locomotion tasks. Bold marks the highest displayed mean for each task and budget.

Figure 3: One-step learning dynamics across 30 tasks. Mean raw episodic return at

K=1

, ordered as in Table 1 . Training progress is the percentage of each method’s registered interaction budget. Curves use the registered seeds. Axis ranges are task-specific; compare within panels.

Configuration	$K=1$	Worst $K$	AUC
FPO	$0.350$	$0.334$	$0.587$
$+$ $u$ -supervision	$0.534$	$0.508$	$0.333$
bridge, fixed $K=4$	$0.612$	$0.593$	$\mathbf{0.700}$
bridge + cycle $\{1,4,16\}$	$0.646$	$0.635$	$0.612$
bridge + cycle $\mathcal{S}$	$0.668$	$0.623$	$0.661$
ArcFlow (+ worst- $K$ )	$\mathbf{0.711}$	$\mathbf{0.685}$	$0.667$

Table 5

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Lineage	Adaptation and audited controls
FPO	Noise-reconstruction surrogate; four-step collection.
FPO++	Per-probe exponential ratios; the shared PPO objective replaces the release’s ASPO objective.
DPPO (flow adaptation)	Gaussian-noised Euler transitions and denoising-MDP likelihood ratios; corrected coordinate reduction and denoising discount. Trained from scratch on the shared flow backbone.
ReinFlow	Stochastic-chain exploration model with the release-style noise-ceiling schedule restored; shared PPO clipping and backbone.
PolicyFlow	Velocity-based surrogate with audited Brownian regularization, entropy, and optimizer settings.

Appendix

Table 4: Implementation scope of the shared-backbone baselines.

Figure 4: A fixed energy scale stabilizes the numerical weight across collecting budgets. Clip fraction, log-weight spread, and bridge energy versus collecting

K

; medians and quartiles over the selection-pool iterations. The collecting-grid

1/K

weight changes with budget; both fixed-

1/4

controls keep the energy scale comparable.

Flow-Based Policy Learning

See all 43 papers

DEFlow: Deep Equilibrium Flow Maximum-Entropy Reinforcement Learning
Oct 12, 2026Ruslan Ageev, Huaze Tang, Runqing Miao +2Flow-Based Policy Learning Reinforcement Learning
Tsinghua University · Peng Cheng Laboratory
RFPO: Rectified Flow Policy Optimization for Embodied Control
Oct 7, 2026Ting Huang, Lisiyu Pan, Haoyu Wang +6Robot Policy Learning Flow-Based Policy Learning
School of Computer Science, Peking University · AI2 Robotics
When One Step Is Enough: An Empirical Study of Euler Integration in Flow-Matching Policies
Oct 22, 2026Shivakanth Sujit, Luca Nunziante, Rousslan Fernand Julien Dossa +1Flow Matching Efficient VLA Model Inference
Araya Inc., Tokyo

Oct 12, 2026J/K move · Enter open · S save

DEFlow: Deep Equilibrium Flow Maximum-Entropy Reinforcement Learning

Ruslan Ageev, Huaze Tang, Runqing Miao, Jiarong Li+1

Tsinghua University · Peng Cheng Laboratory

Flow-Based Policy Learning Reinforcement Learning Robot Policy Learning Maximum Entropy RL

Open paper AskPDF arXiv

Expressive flow policies can represent multiple high-value actions, but sampling them with few integration steps while retaining tractable action densities remains challenging. We introduce DEFlow, a deep equilibrium flow policy that replaces explicit Euler sampling with implicit Euler integration and solves all intermediate actions jointly as an equilibrium system. We combine implicit differentiation with a discrete change-of-variables formula for the final-action density, enabling off-policy training within soft actor-critic without backpropagating through solver iterations. DEFlow achieves an average relative improvement of approximately 10% in mean successful coverage over the strongest baseline across four Multi-Goal docking variants. It also achieves the highest mean time-averaged return on four of five manipulation tasks, while remaining broadly competitive across five MuJoCo tasks. These results support implicit flow policies for tasks requiring both diversity and precision, at the cost of additional nonlinear and linear solves. Primary

Abstract

Flow-Based Policy Learning

Explore similar work

DEFlow: Deep Equilibrium Flow Maximum-Entropy Reinforcement Learning

RFPO: Rectified Flow Policy Optimization for Embodied Control

When One Step Is Enough: An Empirical Study of Euler Integration in Flow-Matching Policies