$π$-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs

Siting Wang; Xiaofeng Wang; Zheng Zhu; Minnan Pei; Xinyu Cui; Cheng Deng; Jian Zhao; Guan Huang

cs.ROMar 2, 2026

$π$ -StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs

Authors: Siting Wang, Xiaofeng Wang, Zheng Zhu, Minnan Pei, Xinyu Cui, Cheng Deng, Jian Zhao, Guan Huang, +2 more

Organizations: GigaAI · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tsinghua University · Zhongguancun Academy · The University of Edinburgh · University College London

Abstract

Flow-based vision-language-action (VLA) models excel in embodied control but suffer from intractable likelihoods during multi-step sampling, hindering online reinforcement learning. We propose \textbf{\textit{ $\boldsymbolπ$ -StepNFT}} (Step-wise Negative-aware Fine-Tuning), a critic-and-likelihood-free framework that requires only a single forward pass per optimization step and eliminates auxiliary value networks. We identify that wider exploration spaces necessitate finer-grained, step-wise guidance for alignment. Empirically, $π$ -StepNFT unlocks latent potential on LIBERO with competitive few-shot robustness. Moreover, it achieves superior generalization on ManiSkill, outperforming value-based baselines in OOD scenarios by preventing overfitting to multimodal features. This property offers a scalable solution promising for complex real-world applications.

Figures & tables

Figure 1: Comparison of training paradigms. Left (ODE): Terminal supervision is well-posed for deterministic ODEs but results in a narrow expert manifold. Middle (Naive SDE): Stochastic rollouts introduce a wider exploration space, but coarse terminal supervision fails to correct deviations, leading to misalignment. Right ( $\pi$ -StepNFT): Our method leverages the wider space from SDE but applies finer, step-wise ranking guidance to ensure robust alignment with the expert manifold.

# Full SFT
Model	Spatial	Object	Goal	Long	Avg.	$\Delta$ Avg.
Octo ( Ghosh et al., 2024 )	78.9	85.7	84.6	51.1	75.1	—
OpenVLA ( Kim et al., 2024 )	84.7	88.4	79.2	53.7	76.5	—
$\pi_{\text{fast}}$ ( Pertsch et al., 2025 )	96.4	96.8	88.6	60.2	85.5	—
OpenVLA-OFT ( Kim et al., 2025b )	91.6	95.3	90.6	86.5	91.0	—
$\pi_{0}$	96.8	98.8	95.8	85.2	94.2	—

Table 1: Success rates (%) on LIBERO in the few-shot setting.

Model		IND	OOD
Model		IND	Vision	Semantic	Execution	Avg.
$\pi_{0}$	Full SFT	38.4	32.6	8.4	13.2	18.1
	$\pi_{\texttt{RL}}$ (Flow-SDE + PPO)	78.8	61.1	25.4	31.5	39.3
	$\pi$ -StepNFT	79.2	69.1	49.1	33.1	50.4
$\pi_{0.5}$	Full SFT	40.1	40.2	16.6	22.4	26.4
	$\pi_{\texttt{RL}}$ (Flow-SDE + PPO)	90.9	68.0	34.5	45.4	49.3

Table 2: Success rates (%) on ManiSkill across In-Distribution (IND) and Out-Of-Distribution (OOD) settings.

Panel of Figure 2 — Figure 2: Flow-SDE sampling and step-wise supervision improve on-policy stability.

Panel of Figure 3 — Figure 3: Contrastive ranking enables stable critic-free learning.

Panel of Figure 4 — Figure 4: Hyperparameter sensitivity analysis. Configuration selected for main experiments is highlighted by the bold pink curves.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Sub-category (OOD type)	ManiSkill env IDs
Vision	Unseen Table (background)	PutOnPlateInScene25VisionImage-v1
	Dynamic Textures (foreground, weak)	PutOnPlateInScene25VisionTexture03-v1
	Dynamic Textures (foreground, strong)	PutOnPlateInScene25VisionTexture05-v1
	Dynamic Noise (image-level, weak)	PutOnPlateInScene25VisionWhole03-v1
	Dynamic Noise (image-level, strong)	PutOnPlateInScene25VisionWhole05-v1
Semantics	Unseen Objects	PutOnPlateInScene25Carrot-v1

Appendix

Table 3: OOD task mapping for ManiSkill PutOnPlateInScene25* across Vision, Semantics, and Execution categories.

Parameters	LIBERO								ManiSkill
	$\pi_{0}$				$\pi_{0.5}$				$\pi_{0}$	$\pi_{0.5}$
	Spatial	Object	Goal	Long	Spatial	Object	Goal	Long	Multitask	Multitask
Train epochs	400	400	400	400	400	400	400	400	240	240
Batch size	2048	2048	2048	2048	2048	2048	2048	2048	5120	5120
Update epochs	2	2	4	4	1	1	3	4	5	5
Actor lr	1e-5	1e-5	1e-5	1e-5	1e-5	1e-5	1e-5	1e-5	8e-6	8e-6

Appendix

Table 4: Hyperparameter settings for Libero and ManiSkill.

Figure 5: Step selection ablation. Performance comparison between uniform random solver-step sampling and fixed-step selection strategies.

Explore similar work

Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models
Oct 11, 2025Mingyang Lyu, Yinqian Sun, Erliang Lin +4Flow Matching Robotic Manipulation
Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Beijing Institute of AI Safety and Governance, China. · University of Chinese Academy of Sciences (UCAS), Beijing, China. +3
CF-VLA: Efficient Coarse-to-Fine Action Generation for Vision-Language-Action Policies
Apr 27, 2026Fan Du, Feng Yan, Jianxiong Wu +8Robotic Control Efficient VLA Model Inference
1Southern University of Science and Technology · 2Xi’an Jiaotong University · 3United Nova Technology +1
FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization
Jun 3, 2026Yihao Wu, He Zhang, Junbo Tan +2Offline RL Vision-Language-Action Models
Tencent Robotics X · Futian Laboratory · Tsinghua University

Oct 11, 2025J/K move · Enter open · S save

Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models

Mingyang Lyu, Yinqian Sun, Erliang Lin, Huangrui Li+3

Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Beijing Institute of AI Safety and Governance, China. · University of Chinese Academy of Sciences (UCAS), Beijing, China.+3

Flow Matching Robotic Manipulation Vision-Language-Action Models RL Fine-Tuning RL for VLMs

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models such as OpenVLA, Octo, and

π_0

have shown strong generalization by leveraging large-scale demonstrations, yet their performance is still fundamentally constrained by the quality and coverage of supervised data. Reinforcement learning (RL) provides a promising path for improving and fine-tuning VLAs through online interaction. However, conventional policy gradient methods are computationally infeasible in the context of flow-matching based models due to the intractability of the importance sampling process, which requires explicit computation of policy ratios. To overcome this limitation, we propose Flow Policy Optimization (FPO) algorithm, which reformulates importance sampling by leveraging per-sample changes in the conditional flow-matching objective. Furthermore, FPO achieves stable and scalable online reinforcement fine-tuning of the

π_0

model by integrating structure-aware credit assignment to enhance gradient efficiency, clipped surrogate objectives to stabilize optimization, multi-step latent exploration to encourage diverse policy updates, and a Q-ensemble mechanism to provide robust value estimation. We evaluate FPO on the LIBERO benchmark and the ALOHA simulation task against supervised, preference-aligned, diffusion-based, autoregressive online RL, and

π_0

-FAST baselines, observing consistent improvements over the imitation prior and strong alternatives with stable learning under sparse rewards. In addition, ablation studies and analyses of the latent space dynamics further highlight the contributions of individual components within FPO, validating the effectiveness of the proposed computational modules and the stable convergence of the conditional flow-matching objective during online RL.

Abstract

Explore similar work

Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models

CF-VLA: Efficient Coarse-to-Fine Action Generation for Vision-Language-Action Policies

FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization