PF-RL: Progress Field Reinforcement Learning via Goal-Conditioned Value Geometry for Vision-Language-Action Models

Yunpeng Qing; Yilun Kong; Sixu Lin; Ming Zhou; Yiming Fei; Shuang Luo; Yixiao Chi; Haoming Gu

cs.ROSep 26, 2026

PF-RL: Progress Field Reinforcement Learning via Goal-Conditioned Value Geometry for Vision-Language-Action Models

Authors: Yunpeng Qing, Yilun Kong, Sixu Lin, Ming Zhou, Yiming Fei, Shuang Luo, Yixiao Chi, Haoming Gu,

Organizations: Zhejiang University · Nanyang Technological University · CUHK-Shenzhen · Shanghai AI Laboratory ·

Abstract

Reinforcement Fine-Tuning~(RFT) has emerged as a promising paradigm for improving Vision-Language-Action~(VLA) policies, yet sparse task-level outcomes provide limited credit for intermediate transitions, especially in long-horizon manipulation. A natural approach is to model intermediate task progress and use it as dense feedback for policy improvement. Despite their architectural differences, existing progress-aware methods commonly formulate task progress as an explicit scalar prediction, providing limited structure for modeling how intermediate observations relate to the task goal, which may hinder effective transition-level credit assignment. We introduce Progress Field Reinforcement Learning (PF-RL), which learns a structured goal-conditioned progress representation over pretrained VLA features and converts it into dense credit for policy optimization. A lightweight shared Progress Field head maps current and goal representations into a compact progress space, where geometric distance induces goal-conditioned value, while complementary temporal and goal-structure objectives shape the learned geometry. Transition-level value changes naturally yield dense progress advantages, enabling fine-grained credit assignment for both offline policy improvement and online reinforcement fine-tuning. Extensive experiments on LIBERO, RoboTwin2.0, and real-world bimanual manipulation tasks show that PF-RL consistently improves policy performance over strong supervised fine-tuning, reinforcement fine-tuning, and progress-aware baselines.

Figures & tables

Panel of Figure 1 — Figure 1: Visualization of task progression and learned representations. The representation spaces are projected to three dimensions with PCA. (a) End-effector trajectories of put both moka pots on the stove task, with colors denoting normalized timesteps. (b) Raw VLA representations. (c) Representations from direct scalar progress prediction. (d) Our Progress Field representations.

Figure 2: An illustrative diagram of our proposed Progress Field Reinforcement Learning .

Figure 3

Method	Beat Block Hammer	Pick Dual Bottles	Stack Bowls Two	Place Empty Cup	Avg.
$\pi_{0}$ (SFT) ( Black et al., 2024 )	59	50	53	60	56
RDT (SFT) ( Liu et al., 2024 )	22	18	42	42	31
OpenVLA-OFT ( Kim et al., 2025 )	13	11	82	81	47
RLinf-GRPO † ( Yu et al., 2025 )	82	70	71	94	79
OpenVLA-OFT+PF-RL (offline $\rightarrow$ online)	25 $\rightarrow$ 91	19 $\rightarrow$ 93	53 $\rightarrow$ 86	64 $\rightarrow$ 96	92

Table 2: Success Rate

(\%)

comparison on RoboTwin2.0 Chen et al. (2025b) .

Methods	Fold Clothes				Stack all Plates
Methods	Original	-Obj	-BG	-L	Original	-Obj	-BG	-L
$\pi_{0.5}$ ( Intelligence et al., 2025b )	75	65 (-10%)	70 (-5%)	55 (-20%)	70	65 (-5%)	55 (-15%)	65 (-5%)
Evo-RL ( Contributors, 2026 )	90	70 (-20%)	80 (-10%)	65 (-25%)	90	85 (-5%)	80 (-10%)	85 (-5%)
$\pi_{0.5}$ +PF-RL (offline $\rightarrow$ online)	80 $\rightarrow$ 95	90 (-5%)	90 (-5%)	85 (-10%)	85 $\rightarrow$ 95	95 (-0%)	90 (-5%)	90 (-5%)
Methods	Put all Blocks into Bucket				Fill Vase and Arrange Flowers
Methods	Original	-Obj	-BG	-L	Original	-Obj	-BG	-L

Table 3: Real-world comparison under the original setting and three visual shifts: unseen objects (-Obj), background (-BG), and lighting (-L). All baselines are reproduced under the same protocol and evaluated over 20 rollouts per setting. Detailed settings are provided in Appendix C .

Panel of Figure 4 — Figure 4: Scalar prediction vs. Progress Field. (a) Final performance on four LIBERO suites. (b–c) Online adaptation on held-out tasks, with progress models trained on the remaining nine tasks.

Figure 5: Ablation of Progress Field training objectives on the four LIBERO suites.

Figure 6: Qualitative visualization of goal-conditioned values induced by the learned Progress Field during real-world task execution. The top shows successful cloth folding, while the bottom illustrates putting all blocks into bucket with an intermediate grasp failure followed by successful recovery. More visualizations are provided in Appendix H .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Overview of real-world tasks and visual variations. Columns show the four manipulation tasks: Fold Clothes , Stack all Plates , Put all Blocks into Bucket , and Fill Vase and Arrange Flowers . Rows correspond to the original environment settings and three visual variations involving object (-Obj), background (-BG), and lighting (-L) changes.

Variant	Value Estimation	$\mathcal{L}_{\mathrm{con}}$	Spatial	Object	Goal	Long	Avg.
Direct scalar prediction	$\Phi_{\mathrm{scalar}}(h_{t},h_{g})$	$\times$	97.0	97.4	95.6	95.8	96.5
PF-RL w/o $\mathcal{L}_{\mathrm{con}}$	$-\lVert z_{t}-z_{g}\rVert_{2}$	$\times$	98.4	98.4	98.0	97.6	98.1
MLP readout variant	$\Phi_{\mathrm{scalar}}(z_{t},z_{g})$	$\checkmark$	98.8	98.2	94.4	95.2	96.7
PF-RL	$-\lVert z_{t}-z_{g}\rVert_{2}$	$\checkmark$	99.2	99.5	98.8	99.0	99.1

Appendix

Table 4: Additional comparison with scalar prediction variants on LIBERO.

Panel of Figure 8 — Figure 8: Representation visualization on held-out tasks. The top row shows LIBERO-Goal Task 5, push the plate to the front of the stove , while the bottom row shows LIBERO-Long Task 0, put both the alphabet soup and the tomato sauce in the basket . (a) Expert end-effector trajectories, (b) Raw VLA representations, (c) Intermediate representations of the direct scalar prediction head, and (d) Projected embeddings learned by Progress Field.

Figure 9: Auxiliary subgoal ablation.

Method	Spatial	Object	Goal	Long	Avg.
PF-RL w/o $\mathcal{L}_{\mathrm{td}}$	97.8	98.8	96.6	94.4	96.9
PF-RL w/o $\mathcal{L}_{\mathrm{step}}$	97.4	98.4	96.0	95.2	96.8
PF-RL w/o $\mathcal{L}_{\mathrm{con}}$	98.4	98.4	98.0	97.6	98.1
PF-RL w/o $\mathcal{L}_{\mathrm{step}}$ and $\mathcal{L}_{\mathrm{con}}$ (TD-only)	97.6	99.0	95.4	95.6	96.9
PF-RL	99.2	99.5	98.8	99.0	99.1

Appendix

Table 5: Results of the Progress Field training objective ablation on LIBERO.

Setting	Spatial	Object	Goal	Long	Avg.
GRPO ( $\lambda_{\mathrm{prog}}=0$ )	97.6	98.7	95.2	96.8	97.1
PF-RL ( $\lambda_{\mathrm{prog}}=0.1$ )	98.8	98.8	98.0	97.2	98.2
PF-RL ( $\lambda_{\mathrm{prog}}=0.5$ )	98.6	99.8	97.6	98.2	98.6
PF-RL ( $\lambda_{\mathrm{prog}}=1.0$ )	99.2	99.5	98.8	99.0	99.1
PF-RL ( $\lambda_{\mathrm{prog}}=2.0$ )	99.0	99.4	98.0	99.2	98.9
PF-RL ( $\lambda_{\mathrm{prog}}=5.0$ )	98.8	99.2	96.2	97.2	97.9

Appendix

Table 6: Analysis of the progress advantage on LIBERO.

Panel of Figure 10 — Figure 10: PCA visualization of goal embeddings from successful trajectories, colored by task.

Figure 11: Progress advantage distributions across execution behaviors.

Figure 12: Successful rollout trajectories and corresponding goal-conditioned values.

Figure 13: Rollout trajectories with intermediate failures and corresponding values.

Figure 14: Training time of baselines and PF-RL.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,742 papers

ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation
Mar 29, 2026Hongyu Yan, Qiwei Li, Jiaolong Yang +1Robotic Manipulation Vision-Language-Action Models
Peking University · Microsoft Research Asia (MSRA)
What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models
May 13, 2026Yuanfang Peng, Jingjing Fu, Chuheng Zhang +6VLM Robustness Domain Generalization
1Hong Kong University of Science and Technology · 2Microsoft Research Asia · 3Zhejiang University
Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models
Oct 11, 2025Mingyang Lyu, Yinqian Sun, Erliang Lin +4Flow Matching Robotic Manipulation
Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Beijing Institute of AI Safety and Governance, China. · University of Chinese Academy of Sciences (UCAS), Beijing, China. +3

Mar 29, 2026J/K move · Enter open · S save

ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation

Hongyu Yan, Qiwei Li, Jiaolong Yang, Yadong Mu

Peking University · Microsoft Research Asia (MSRA)

Robotic Manipulation Vision-Language-Action Models Robot Task Progress Estimation Long-Horizon Robotic Manipulation Diffusion Policy

Open paper AskPDF arXiv

Most existing vision-language-action (VLA) models for robotic manipulation lack progress awareness, typically relying on hand-crafted heuristics for task termination. This limitation is particularly severe in long-horizon tasks involving cascaded sub-goals. In this work, we investigate the estimation and integration of task progress, proposing a novel model named {\textbf \vla}. Our technical contributions are twofold: (1) robust progress estimation: We pre-train a progress estimator on large-scale, unsupervised video-text robotic datasets. This estimator achieves a low prediction residual (0.07 on a scale of

[0, 1]

) in simulation and demonstrates zero-shot generalization to unseen real-world samples, and (2) differentiable progress guidance: We introduce an inverse dynamics world model that maps predicted action tokens into future latent visual states. These latents are then processed by the progress estimator; by applying a maximal progress regularization, we establish a differentiable pipeline that provides progress-piloted guidance to refine action tokens. Extensive experiments on the CALVIN and LIBERO benchmarks, alongside real-world robot deployment, consistently demonstrate substantial improvements in success rates and generalization over strong baselines.

Abstract

Reinforcement Learning

Explore similar work

ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation

What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models

Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models