Generative models, such as diffusion and flow-based models, have shown strong promise for robot policy learning by capturing complex and multimodal action distributions from demonstrations. However, policies trained solely with imitation learning often suffer from imperfect demonstrations and distributional shifts, while further improvement typically requires additional expert data. Reinforcement learning offers a natural solution through environment interaction, but effectively finetuning generative robot policies remains challenging due to the intractability of likelihood estimation. In this work, we propose RoboFFT, a forward-process reinforcement learning framework for finetuning generative robot policies, which applies forward noising to sampled actions and uses the weighted score / flow matching loss to construct a surrogate policy ratio for PPO-style updates. We evaluate RoboFFT with popular generative robot policies on representative simulation benchmarks, including long-horizon planning and sparse reward settings. Extensive experiments and analysis demonstrate that RoboFFT consistently improves performance while achieving better stability and training efficiency. We further integrate RoboFFT into a real world RL framework and demonstrate its effectiveness in real world tasks. Project website: https://student-of-holmes.github.io/RoboFFT/.
Figures & tables
Figure 1: We present RoboFFT , an online reinforcement learning framework for finetuning generative robot policy with forward process. (a) Reverse finetuning formulates the reverse denoising process as a multi-step MDP, while forward finetuning aligns with the same objective as training by following the forward noising process. (b) One framework across popular generative robot policy on simulation benchmarks. (c) Real-world deployment with integration of real world RL. (d) Improved performance in success rate, training stability and efficiency.
Figure 2: Overview of RoboFFT . We first collect the offline data leveraging teleoperation and pretrain the base policy with the score / flow matching loss. The pretrained policy then generates actions using an SDE sampler and receives reward feedback from the environment. Finally, we apply forward noising and reuse the matching loss as a surrogate likelihood for PPO-style policy update.
Figure 3: Success rates on Robomimic state- and pixel-input manipulation tasks. Curves show means over three random seeds, with shaded regions indicating one standard deviation across seeds.
Figure 4Figure 5Figure 6
Figure 10: Successful real world rollouts on the four selected tasks..
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Task
State obs dim
Pixel prop. dim
Pixel image views
Act dim
Tmax
Reward
Lift
19
9
1×96×96
7
300
sparse
Can
23
9
1×96×96
7
300
sparse
Square
23
9
1×96×96
7
400
sparse
Transport
59
18
2×96×96
14
800
sparse
Appendix
Table 1: Robomimic task specifications used in our online finetuning experiments. For pixel-input tasks, “Prop. dim” denotes the low-dimensional proprioceptive input dimension paired with image observations, and “Image views” reports the number and spatial resolution of RGB camera views.
Task
Action chunk (H×da)
State-input checkpoint
Pixel-input checkpoint
Lift
4×7
3000 epochs
2000 epochs
Can
4×7
3000 epochs
2000 epochs
Square
4×7
3000 epochs
2000 epochs
Transport
8×14
3000 epochs
2000 epochs
Appendix
Table 2: Base policy checkpoints used for Robomimic online finetuning. The listed numbers are the behavior-cloning pretraining epochs of the selected checkpoints.
Task
Input
Hexec
nenv
nstep
nitr
Online samples
Lift
state
4
50
300
81
4.86M
Lift
pixel
4
50
300
81
4.86M
Can
state
4
50
300
151
9.06M
Can
pixel
4
50
300
151
9.06M
Square
state
4
50
400
201
16.08M
Square
pixel
4
50
400
201
16.08M
Appendix
Table 3: Online finetuning budgets for Robomimic experiments. The last column equals Hexec⋅nenv⋅nstep⋅nitr , which gives the total number of online interaction samples.
Task
Input
Actor LR
Critic LR
Batch size
Nmc
αratio
Lift
state
2×10−6
1×10−3
10000
1
2
Can
state
2×10−6
1×10−3
10000
1
2
Square
state
2×10−5
1×10−3
10000
2
2
Transport
state
2×10−6
1×10−3
10000
1
1
Lift
pixel
2×10−5
6.5×10−4
500
1
2
Can
pixel
2×10−5
6.5×10−4
500
1
2
Appendix
Table 4: Main RoboFFT-F optimizer hyperparameters for Robomimic. All runs use γ=0.999 , GAE parameter λ=0.95 , 10 update epochs per iteration, target KL 10−2 , and the repeat-samples trick described in Sec. 4.3 .
Task
Input
Denoise steps
FT steps
Eval
Actor LR
Batch
Nmc
αratio
Anchor / βneg
Lift
state
20
10
DDPM
2×10−5
2500
16
1.0
– / 0.125
Can
state
20
10
DDPM
2×10−5
2500
16
1.0
– / 0.125
Square
state
20
10
DDPM
2×10−5
2500
16
0.5
0.05 / 0.125
Transport
state
20
10
DDPM
2×10−5
2500
16
0.5
0.05 / 1.0
Lift
pixel
100
5
DDIM
2×10−5
500
16
1.0
– / 0.125
Can
pixel
100
5
DDIM
2×10−5
500
16
1.0
– / 0.125
Appendix
Table 5: Main RoboFFT-D hyperparameters for Robomimic. Nmc denotes the number of forward-noising Monte Carlo samples used for estimating the surrogate matching loss. “Anchor” reports the coefficient λanchor ; a dash means the anchor loss is disabled. βneg is the negative-advantage discount coefficient in Eq. ( 24 ).
Component
RoboFFT-F
RoboFFT-D
Base codebase
ReinFlow-based flow policy implementation
DPPO-based diffusion policy implementation
Prediction target
Velocity field uθ=vθ
Noise prediction uθ=ϵθ
Forward-process loss
Conditional flow matching loss
Denoising score matching loss
Effective weighting
ΩF(k)≡1
Discrete Δλ weighting
Rollout stochasticity
Explicit SDE exploration for deterministic flow sampling
Diffusion sampler stochasticity
MC repetition
Small Nmc , typically 1 or 2
Larger Nmc=16
Appendix
Table 6: Implementation differences between RoboFFT-F and RoboFFT-D. Both methods use the same high-level RoboFFT objective, but require different stabilization choices due to their different generative parameterizations.
Figure 11: Additional-seed evaluation of RoboFFT-F, FPO, and ReinFlow. Curves indicate means and shaded regions indicate one standard deviation across random seeds.
Figure 12: Additional ablation studies on Robomimic Square-state. We evaluate the effects of ratio temperature, advantage weighting, and PPO-style clipping. Ratio temperature calibrates the surrogate likelihood-ratio scale, advantage weighting reduces unstable return-weighted updates, and PPO-style clipping stabilizes early finetuning by constraining the surrogate-ratio update.
Figure 13: RoboFFT-F ablations on Robomimic Transport-state. Advantage weighting and the PPO-style update prevent the severe degradation observed in the corresponding ablated variants. Ratio temperature, MC repetition, and SDE noise also affect convergence and stability.
Figure 14: RoboFFT-D ablations on Robomimic Square-state. Negative-advantage discount, log-SNR weighting and MC repetition substantially affect finetuning performance. Anchor regularization and ratio temperature have comparatively modest effects in the tested configuration.
Figure 15: t-SNE visualization of action chunks sampled from the pretrained and finetuned policies under matched anchor states and latent noises. Each point is colored by the diagnostic advantage Aviz(s,a)=Qviz(s,a)−Vϕ(s) . The finetuned policy forms a denser region of higher-advantage actions, complementing the rollout-timeline analysis.
Parameter
RoboFFT
πRL
Log probability calculation
score/ flow matching loss
multi step MDP
KL coefficient β
0.5
0.0
PPO clipping range
0.05 / 0.05
0.20 / 0.20
Samples per action Nmc
8
Not used
Appendix
Table 7: Differences in training configurations between RoboFFT and πRL baseline.
Figure 16: Real world settings
Task
Embodiment
Task Type
Randomization
Key Challenges
Pick Eggplant
Franka Research 3
Pick-and-place
Object position randomized within 3cm along x/y
Grasping an irregular object under position variation; stable transport and accurate placement.
Peg Insertion
Franka Emika Panda
Precision assembly
Initial end-effector position randomized within 1.5cm along x/y
Fine-grained pose alignment, tight insertion tolerance, and contact-rich manipulation.
Pouring
Franka Emika Panda
Granular
Object position randomized within 1cm along x/y
Continuous orientation control, target alignment, and error propagation across sequential stages.
Cabinet Retrieval
Franka Research 3
Long-horizon
Object position randomized within 1cm along x/y
Long-horizon control, confined-space manipulation, partial occlusion, and sensitivity to intermediate failures.
Appendix
Table 8: Real-world task suite and key challenges.
Diffusion and flow-based generative policies provide a powerful policy class for reinforcement learning by inducing rich stochastic exploration through iterative action generation. However, the stochasticity of diffusion policies is not suitable for stable and precise control in high-dimensional robotic systems, where small action variations can accumulate into inconsistent motion and reduced robustness. To address this issue, we propose SteerGenPO, a latent-space reinforcement learning framework that steers a trained generative policy into a robust deterministic robotic controller. The key idea is to replace stochastic latent sampling of the trained generative policy with a learned latent actor that predicts a state-dependent latent input for the generative policies. This separates exploration and control: stochastic generative sampling provides diverse action proposals during policy learning, while deterministic latent steering provides stable and adaptive control at deployment. We evaluate SteerGenPO on six Isaac Lab benchmarks and a Unitree G1 locomotion task. The results show SteerGenPO improves over both classical RL and generative RL baselines, while its deterministic latent steering produces more stable inference-time behaviors and more reliable command responses.
By relying on independent couplings from uninformative Gaussian priors, standard diffusion and flow matching models are forced to learn complex, high-cost vector fields to reach the physical action space. Generative models excel at capturing multimodal behaviors for robotic Learning from Demonstration (LfD), but often suffer from high inference cost. This paper introduces Temporal Policy, a generative framework based on stochastic interpolants that formulates action generation as a temporally coupled transport problem. By initializing the generative flow at the robot's recent history, we explicitly couple past states to future action sequences. This data-dependent coupling reduces transport cost and produces straight vector fields. We validate Temporal Policy across visuomotor simulation benchmarks and on a physical Barrett WAM 2x 7DoF teleoperation platform. Our approach reduces transport costs by nearly an order of magnitude compared to noise-initialized baselines, achieving a 19.1 ms inference latency on a single NVIDIA RTX 4080. Crucially, these geometric and computational efficiencies are achieved while matching the success rates of state-of-the-art baselines. This simplified transport geometry bypasses the computational bottleneck of independent Gaussian priors, helping enable high-frequency, closed-loop control. The code is publicly available at https://github.com/dmiller12/TemporalPolicy.
Dylan Miller, Martin Jagersand
Department of Computing Science, University of Alberta, Canada
Pretrained generative robot policies learn expressive action priors from demonstrations. However, existing reinforcement learning methods only steer the noisy space but fail to modulate intermediate action representations during the generation process, resulting in performance degradation and inefficiency. To address this limitation, we propose a novel Dual-Latent Space Reinforcement Learning (DLSRL) framework, which complements initial-noise steering with representation-level control inside the frozen generator. Specifically, our actor network predicts two distinct latent variables: an initial-noise latent variable that steers behavior generation, and an action-representation latent variable for intermediate feature modulation. Moreover, this representation latent variable is mapped to adapter features and ingeniously injected into the hidden states of intermediate action tokens via residual connections. Our dual-control design enables direct adjustment of action representations without updating the base policy. Experiments across generative policy architectures and robotic manipulation tasks show that DLSRL effectively accelerates online robot policy adaptation and achieves competitive performance. Our code is available at https://github.com/xianchaoxiu/DLSRL.
Pengfei Zhang, Teng Sun, Xianchao Xiu
School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, China