RoboFFT: Finetuning generative robot policy via online reinforcement learning with forward process

Yu Li; Shenghe Hu; Yuhan Wang; Yaoxiang Pu; Haotong Zhang; Yuanpei Chen; Yaodong Yang

cs.ROSep 26, 2026

RoboFFT: Finetuning generative robot policy via online reinforcement learning with forward process

Authors: Yu Li, Shenghe Hu, Yuhan Wang, Yaoxiang Pu, Haotong Zhang, Yuanpei Chen, Yaodong Yang

Organizations: Institute for Artificial Intelligence, Peking University · Zhongguancun Academy · PKU-PsiBot Joint Lab · Nanjing University

Abstract

Generative models, such as diffusion and flow-based models, have shown strong promise for robot policy learning by capturing complex and multimodal action distributions from demonstrations. However, policies trained solely with imitation learning often suffer from imperfect demonstrations and distributional shifts, while further improvement typically requires additional expert data. Reinforcement learning offers a natural solution through environment interaction, but effectively finetuning generative robot policies remains challenging due to the intractability of likelihood estimation. In this work, we propose RoboFFT, a forward-process reinforcement learning framework for finetuning generative robot policies, which applies forward noising to sampled actions and uses the weighted score / flow matching loss to construct a surrogate policy ratio for PPO-style updates. We evaluate RoboFFT with popular generative robot policies on representative simulation benchmarks, including long-horizon planning and sparse reward settings. Extensive experiments and analysis demonstrate that RoboFFT consistently improves performance while achieving better stability and training efficiency. We further integrate RoboFFT into a real world RL framework and demonstrate its effectiveness in real world tasks. Project website: https://student-of-holmes.github.io/RoboFFT/.

Figures & tables

Figure 1: We present RoboFFT , an online reinforcement learning framework for finetuning generative robot policy with forward process. (a) Reverse finetuning formulates the reverse denoising process as a multi-step MDP, while forward finetuning aligns with the same objective as training by following the forward noising process. (b) One framework across popular generative robot policy on simulation benchmarks. (c) Real-world deployment with integration of real world RL. (d) Improved performance in success rate, training stability and efficiency.

Figure 2: Overview of RoboFFT . We first collect the offline data leveraging teleoperation and pretrain the base policy with the score / flow matching loss. The pretrained policy then generates actions using an SDE sampler and receives reward feedback from the environment. Finally, we apply forward noising and reuse the matching loss as a surrogate likelihood for PPO-style policy update.

Panel of Figure 3 — Figure 3: Success rates on Robomimic state- and pixel-input manipulation tasks. Curves show means over three random seeds, with shaded regions indicating one standard deviation across seeds.

Figure 10: Successful real world rollouts on the four selected tasks..

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Task	State obs dim	Pixel prop. dim	Pixel image views	Act dim	$T_{\max}$	Reward
Lift	19	9	$1\times 96\times 96$	7	300	sparse
Can	23	9	$1\times 96\times 96$	7	300	sparse
Square	23	9	$1\times 96\times 96$	7	400	sparse
Transport	59	18	$2\times 96\times 96$	14	800	sparse

Appendix

Table 1: Robomimic task specifications used in our online finetuning experiments. For pixel-input tasks, “Prop. dim” denotes the low-dimensional proprioceptive input dimension paired with image observations, and “Image views” reports the number and spatial resolution of RGB camera views.

Task	Action chunk $(H\times d_{a})$	State-input checkpoint	Pixel-input checkpoint
Lift	$4\times 7$	3000 epochs	2000 epochs
Can	$4\times 7$	3000 epochs	2000 epochs
Square	$4\times 7$	3000 epochs	2000 epochs
Transport	$8\times 14$	3000 epochs	2000 epochs

Appendix

Table 2: Base policy checkpoints used for Robomimic online finetuning. The listed numbers are the behavior-cloning pretraining epochs of the selected checkpoints.

Task	Input	$H_{\mathrm{exec}}$	$n_{\mathrm{env}}$	$n_{\mathrm{step}}$	$n_{\mathrm{itr}}$	Online samples
Lift	state	4	50	300	81	4.86M
Lift	pixel	4	50	300	81	4.86M
Can	state	4	50	300	151	9.06M
Can	pixel	4	50	300	151	9.06M
Square	state	4	50	400	201	16.08M
Square	pixel	4	50	400	201	16.08M

Appendix

Table 3: Online finetuning budgets for Robomimic experiments. The last column equals

H_{\mathrm{exec}}\cdot n_{\mathrm{env}}\cdot n_{\mathrm{step}}\cdot n_{\mathrm{itr}}

, which gives the total number of online interaction samples.

Task	Input	Actor LR	Critic LR	Batch size	$N_{\mathrm{mc}}$	$\alpha_{\mathrm{ratio}}$
Lift	state	$2\times 10^{-6}$	$1\times 10^{-3}$	10000	1	2
Can	state	$2\times 10^{-6}$	$1\times 10^{-3}$	10000	1	2
Square	state	$2\times 10^{-5}$	$1\times 10^{-3}$	10000	2	2
Transport	state	$2\times 10^{-6}$	$1\times 10^{-3}$	10000	1	1
Lift	pixel	$2\times 10^{-5}$	$6.5\times 10^{-4}$	500	1	2
Can	pixel	$2\times 10^{-5}$	$6.5\times 10^{-4}$	500	1	2

Appendix

Table 4: Main RoboFFT-F optimizer hyperparameters for Robomimic. All runs use

\gamma=0.999

, GAE parameter

\lambda=0.95

, 10 update epochs per iteration, target KL

10^{-2}

, and the repeat-samples trick described in Sec. 4.3 .

Task	Input	Denoise steps	FT steps	Eval	Actor LR	Batch	$N_{\mathrm{mc}}$	$\alpha_{\mathrm{ratio}}$	Anchor / $\beta_{\mathrm{neg}}$
Lift	state	20	10	DDPM	$2\times 10^{-5}$	2500	16	1.0	– / 0.125
Can	state	20	10	DDPM	$2\times 10^{-5}$	2500	16	1.0	– / 0.125
Square	state	20	10	DDPM	$2\times 10^{-5}$	2500	16	0.5	0.05 / 0.125
Transport	state	20	10	DDPM	$2\times 10^{-5}$	2500	16	0.5	0.05 / 1.0
Lift	pixel	100	5	DDIM	$2\times 10^{-5}$	500	16	1.0	– / 0.125
Can	pixel	100	5	DDIM	$2\times 10^{-5}$	500	16	1.0	– / 0.125

Appendix

Table 5: Main RoboFFT-D hyperparameters for Robomimic.

N_{\mathrm{mc}}

denotes the number of forward-noising Monte Carlo samples used for estimating the surrogate matching loss. “Anchor” reports the coefficient

\lambda_{\mathrm{anchor}}

; a dash means the anchor loss is disabled.

\beta_{\mathrm{neg}}

is the negative-advantage discount coefficient in Eq. ( 24 ).

Component	RoboFFT-F	RoboFFT-D
Base codebase	ReinFlow-based flow policy implementation	DPPO-based diffusion policy implementation
Prediction target	Velocity field $u_{\theta}=v_{\theta}$	Noise prediction $u_{\theta}=\epsilon_{\theta}$
Forward-process loss	Conditional flow matching loss	Denoising score matching loss
Effective weighting	$\Omega_{F}(k)\equiv 1$	Discrete $\widetilde{\Delta\lambda}$ weighting
Rollout stochasticity	Explicit SDE exploration for deterministic flow sampling	Diffusion sampler stochasticity
MC repetition	Small $N_{\mathrm{mc}}$ , typically $1$ or $2$	Larger $N_{\mathrm{mc}}=16$

Appendix

Table 6: Implementation differences between RoboFFT-F and RoboFFT-D. Both methods use the same high-level RoboFFT objective, but require different stabilization choices due to their different generative parameterizations.

Panel of Figure 11 — Figure 11: Additional-seed evaluation of RoboFFT-F, FPO, and ReinFlow. Curves indicate means and shaded regions indicate one standard deviation across random seeds.

Panel of Figure 12 — Figure 12: Additional ablation studies on Robomimic Square-state. We evaluate the effects of ratio temperature, advantage weighting, and PPO-style clipping. Ratio temperature calibrates the surrogate likelihood-ratio scale, advantage weighting reduces unstable return-weighted updates, and PPO-style clipping stabilizes early finetuning by constraining the surrogate-ratio update.

Panel of Figure 13 — Figure 13: RoboFFT-F ablations on Robomimic Transport-state. Advantage weighting and the PPO-style update prevent the severe degradation observed in the corresponding ablated variants. Ratio temperature, MC repetition, and SDE noise also affect convergence and stability.

Panel of Figure 14 — Figure 14: RoboFFT-D ablations on Robomimic Square-state. Negative-advantage discount, log-SNR weighting and MC repetition substantially affect finetuning performance. Anchor regularization and ratio temperature have comparatively modest effects in the tested configuration.

Figure 15: t-SNE visualization of action chunks sampled from the pretrained and finetuned policies under matched anchor states and latent noises. Each point is colored by the diagnostic advantage $A_{\mathrm{viz}}(s,a)=Q_{\mathrm{viz}}(s,a)-V_{\phi}(s)$ . The finetuned policy forms a denser region of higher-advantage actions, complementing the rollout-timeline analysis.

Parameter	RoboFFT	$\pi_{RL}$
Log probability calculation	score/ flow matching loss	multi step MDP
KL coefficient $\beta$	0.5	0.0
PPO clipping range	0.05 / 0.05	0.20 / 0.20
Samples per action $N_{\mathrm{mc}}$	8	Not used

Appendix

Table 7: Differences in training configurations between RoboFFT and

\pi_{RL}

baseline.

Task	Embodiment	Task Type	Randomization	Key Challenges
Pick Eggplant	Franka Research 3	Pick-and-place	Object position randomized within $3\,\mathrm{cm}$ along $x/y$	Grasping an irregular object under position variation; stable transport and accurate placement.
Peg Insertion	Franka Emika Panda	Precision assembly	Initial end-effector position randomized within $1.5\,\mathrm{cm}$ along $x/y$	Fine-grained pose alignment, tight insertion tolerance, and contact-rich manipulation.
Pouring	Franka Emika Panda	Granular	Object position randomized within $1\,\mathrm{cm}$ along $x/y$	Continuous orientation control, target alignment, and error propagation across sequential stages.
Cabinet Retrieval	Franka Research 3	Long-horizon	Object position randomized within $1\,\mathrm{cm}$ along $x/y$	Long-horizon control, confined-space manipulation, partial occlusion, and sensitivity to intermediate failures.

Appendix

Table 8: Real-world task suite and key challenges.

Task	Observation Space			Action Space
	Visual Obs.	Prop. dim.	Prop. State	Dim.	Action Representation
Pick Eggplant	RGB $(224,224,3)\times 3$	8	7-D joint state + 1-D gripper state	7	6-D target end-effector pose + 1-D gripper command
Peg Insertion	RGB $(224,224,3)\times 3$	7	7-D joint state	6	6-D target end-effector pose; gripper fixed during insertion
Pouring	RGB $(224,224,3)\times 3$	8	7-D joint state + 1-D gripper state	7	6-D target end-effector pose + 1-D gripper command
Cabinet Retrieval	RGB $(224,224,3)\times 3$	8	7-D joint state + 1-D gripper state	7	6-D target end-effector pose + 1-D gripper command

Appendix

Table 9: State and action components of the real-world tasks. We abbreviate dimension as “dim.” and proprioception as “prop.”.

Robot Policy Learning

Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.

See all 372 papers

Steering Generative Reinforcement Learning into Stable Robotic Controller
Jun 15, 2026Yixuan Wang, Shutong Ding, Ke Hu +3Latent Action Learning Robotic RL
1ShanghaiTech University
Temporal Policy: History-Initialized Action Generation for Robotic Learning from Demonstration
Jul 31, 2026Dylan Miller, Martin JagersandFlow Matching Robot Policy Learning
Department of Computing Science, University of Alberta, Canada
Beyond Noise Steering: Dual-Latent Space Reinforcement Learning for Generative Robot Policy
Sep 10, 2026Pengfei Zhang, Teng Sun, Xianchao XiuLatent Action Learning Robot Policy Learning
School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, China

Jun 15, 2026J/K move · Enter open · S save

Steering Generative Reinforcement Learning into Stable Robotic Controller

Yixuan Wang, Shutong Ding, Ke Hu, Tianxiang Gui+2

1ShanghaiTech University

Latent Action Learning Robotic RL RL for Generative Models Robust RL

Open paper AskPDF arXiv

Diffusion and flow-based generative policies provide a powerful policy class for reinforcement learning by inducing rich stochastic exploration through iterative action generation. However, the stochasticity of diffusion policies is not suitable for stable and precise control in high-dimensional robotic systems, where small action variations can accumulate into inconsistent motion and reduced robustness. To address this issue, we propose SteerGenPO, a latent-space reinforcement learning framework that steers a trained generative policy into a robust deterministic robotic controller. The key idea is to replace stochastic latent sampling of the trained generative policy with a learned latent actor that predicts a state-dependent latent input for the generative policies. This separates exploration and control: stochastic generative sampling provides diverse action proposals during policy learning, while deterministic latent steering provides stable and adaptive control at deployment. We evaluate SteerGenPO on six Isaac Lab benchmarks and a Unitree G1 locomotion task. The results show SteerGenPO improves over both classical RL and generative RL baselines, while its deterministic latent steering produces more stable inference-time behaviors and more reliable command responses.

Abstract

Robot Policy Learning

Explore similar work

Steering Generative Reinforcement Learning into Stable Robotic Controller

Temporal Policy: History-Initialized Action Generation for Robotic Learning from Demonstration

Beyond Noise Steering: Dual-Latent Space Reinforcement Learning for Generative Robot Policy