Authors: Chubin Zhang, Zhenglin Wan, Feng Chen, Fuchao Yang, Lang Feng, Yaxin Zhou, Xingrui Yu, Yang You, Ivor Tsang, Bo An
Organizations: Nanyang Technological University, Singapore · National University of Singapore, Singapore · Carnegie Mellon University, United States · CFAR, Agency for Science, Technology and Research, Singapore · IHPC, Agency for Science, Technology and Research, Singapore
Reinforcement learning (RL) faces a persistent tension: policies that are stable to optimize (e.g., Gaussians) are often too simple to represent the multimodal action distributions required for complex control. Conversely, expressive generative policies---such as diffusion and flow matching---can be difficult to optimize in online RL due to intractable likelihoods and gradients propagating through long sampling chains. We address this tension with a key structural principle: decoupling optimization from generation. Building on this, we introduce GoRL (Generative Online Reinforcement Learning), an algorithm-agnostic framework that trains expressive policies from scratch by confining policy optimization to a tractable latent space while delegating action synthesis to a conditional generative decoder. Using a two-timescale alternating schedule and anchoring decoder refinement to a fixed prior, GoRL enables stable optimization while continuously expanding expressiveness. Empirically, GoRL consistently outperforms unimodal and generative baselines across diverse continuous-control tasks. Notably, GoRL achieves returns exceeding 870 on HopperStand, more than 3× the strongest baseline; on high-dimensional humanoid tasks, it further outperforms the strongest non-GoRL baseline by over an order of magnitude. Code is available at https://github.com/bennidict23/GoRL.
Figures & tables
Figure 1: The Mode-Covering Problem. When the optimal action distribution (blue) is multimodal, a unimodal Gaussian policy (red) must place nontrivial probability mass in the low-reward region between modes. This yields suboptimal actions and brittle performance in environments requiring precise mode selection.
Figure 2: Overview of the GoRL framework. (a) Latent optimization: The decoder gϕ is frozen while the encoder πθ is optimized in latent space using standard RL algorithms (e.g., PPO or SAC). (b) Decoder refinement: The encoder is frozen and the decoder gϕ is updated via supervised generative training on recent rollouts, using a fixed Gaussian prior as refinement inputs. Periodic stage-wise re-initialization of πθ improves stability across stage transitions.
Figure 3: Learning curves across six DMControl tasks. Vertical dashed lines mark decoder-refinement boundaries at 60M, 120M, and 150M steps. Shaded regions denote standard deviation across five seeds. GoRL achieves higher final returns and more stable learning than Gaussian PPO and generative baselines across tasks.
Figure 4: Mechanism Ablations. (a) Refining the decoder on evolving latents leads to performance collapse, whereas anchoring to a fixed prior maintains stability. (b) Stage-wise re-initialization prevents performance drops at stage boundaries compared to the baseline without re-initialization.
Method
CheetahRun
FingerSpin
FingerTurnHard
FishSwim
HopperStand
WalkerWalk
PPO
724.83 ± 155.67
539.03 ± 146.63
738.70 ± 114.45
433.70 ± 73.63
286.09 ± 273.07
825.65 ± 79.70
FPO
599.15 ± 297.45
56.05 ± 124.53
752.08 ± 55.39
204.66 ± 191.49
3.94 ± 1.79
29.00 ± 4.32
DPPO
559.79 ± 99.97
694.06 ± 191.59
633.84 ± 88.21
143.52 ± 26.25
2.14 ± 0.81
345.59 ± 64.45
GoRL-Diff
902.24 ± 2.20
844.74 ± 59.43
884.59 ± 26.95
608.61 ± 22.07
874.63 ± 38.79
908.96 ± 30.45
GoRL-FM
883.40 ± 19.94
903.92 ± 104.08
860.83 ± 14.93
641.01 ± 13.10
733.66 ± 223.76
919.61 ± 60.86
Table 1: Final episodic returns (mean ± std) over five seeds at 180M steps.
Method
HumanoidStand
HumanoidRun
PPO
74.97 ± 4.58
17.39 ± 2.31
FPO
0.00 ± 0.00
0.00 ± 0.00
DPPO
26.57 ± 6.06
6.38 ± 1.74
GoRL-FM
950.27 ± 20.24
337.61 ± 9.06
GoRL-Diff
976.10 ± 2.20
326.30 ± 29.50
Table 2: High-dimensional humanoid evaluation. Final returns are averaged over three seeds; stage-boundary values are mean returns of GoRL-Diff .
Figure 5: Quantifying decoder improvement across stages. We train fresh encoders against decoders frozen at different refinement stages. Each curve estimates the capability ceiling of a fixed decoder. The monotonic lift from Stage 0 to Stage 3 shows that decoder refinement progressively expands the action support available to latent optimization.
Figure 6: Evolution of action distributions on HopperStand . Gaussian KDE plots of policy outputs. Each GoRL checkpoint is sampled using the stage-end encoder and current decoder, before any subsequent decoder refinement and encoder re-initialization. At 60M steps, GoRL is unimodal. By 180M, it develops a clear bimodal structure, whereas Gaussian PPO remains unimodal.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Visual overview of the DMControl tasks. The benchmark suite encompasses a diverse range of dynamics: high-speed planar locomotion ( CheetahRun ), bipedal gait control ( WalkerWalk ), contact-rich manipulation ( FingerSpin , FingerTurnHard ), and fine-grained stabilization tasks ( HopperStand , FishSwim ).
Task
Action dim.
Observation dim.
CheetahRun
6
17
HopperStand
4
15
WalkerWalk
6
24
FingerSpin
2
9
FingerTurnHard
2
9
FishSwim
5
24
Appendix
Table 3: Observation and action dimensions for the selected DMControl benchmarks.
Hyperparameter
Value
Optimizer
Adam
Learning rate
1×10−3
Batch size per update
1024
Rollout horizon
30
Optimization epochs
16
Discount factor γ
0.995
Appendix
Table 4: Shared PPO hyperparameters.
Figure 8: Cost–Benefit Analysis. Training time vs. final return, normalized to Gaussian PPO ( 1.0× ). Results are averaged over CheetahRun , FingerSpin , and HopperStand . While generative policies naturally incur training overhead, only GoRL translates this cost into positive performance gains.
Environment
Hopper-v2
Walker2D-v2
HalfCheetah-v2
Observation Dim
11
17
17
Action Dim
3
6
6
SAC Optimization
Actor LR
3×10−4
3×10−4
3×10−4
Batch Size
256
256
256
Discount ( γ )
0.99
0.99
0.99
Appendix
Table 5: Hyperparameters for GoRL(SAC) experiments on Gym Locomotion.
Figure 9: Off-policy compatibility check (Gym Benchmarks). Learning curves comparing standard Gaussian SAC with GoRL instantiated using an SAC encoder and Diffusion decoder. The results demonstrate that the GoRL factorization can be successfully trained with off-policy algorithms.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
In the field of online reinforcement learning (RL), traditional Gaussian policies and flow-based methods are often constrained by their unimodal expressiveness, complex gradient clipping, or stringent trust-region requirements. Moreover, they all rely on post-hoc penalization of negative samples to correct erroneous actions. This paper introduces Positive-Only Drifting Policy Optimization (PODPO), a likelihood-free and gradient-clipping-free generative approach for online RL. By leveraging the drifting model, PODPO performs policy updates via advantage-weighted local contrastive drifting. Relying solely on positive-advantage samples, it elegantly steers actions toward high-return regions while exploiting the inherent local smoothness of the generative model to enable proactive error prevention. In doing so, PODPO opens a promising new pathway for generative policy learning in online settings.
Many reinforcement learning (RL) tasks have discrete action spaces, but most generative policy methods based on diffusion and flow matching are designed for continuous control. Meanwhile, generative policies usually rely heavily on offline datasets and offline-to online RL is itself challenging, as the policy must improve from new interaction without losing useful behavior learned from static data. To address those challenges, we introduce DRIFT, an online fine-tuning method that updates an offline pretrained continuous time Markov chain (CTMC) policy with an advantage-weighted discrete flow matching loss. To preserve useful pretrained knowledge, we add a path-space penalty that regularizes the full CTMC trajectory distribution, rather than only the final action distribution. For large discrete action spaces, we introduce a candidate-set approximation that updates the actor over a small subset of actions sampled from reference-policy rollouts and uniform exploration. Our theoretical analysis shows that focusing on a small set of high probability actions can reduce computation while keeping the approximation error small, making our method practical for large action spaces. We evaluate on macro-action MinAtar games where offline-pretrained DRIFT outperforms its online-only counterpart across all five games and achieves the highest mean return on three and on oracle free Jericho, DRIFT achieves a mean normalized score of 7.3% with a lightweight GRU encoder, outperforming the evaluated pretrained language-model baselines. Ablations demonstrate the importance of appropriately weighted path-space regularization and reference-guided candidate construction.
Fairoz Nower Khan, Nabuat Zaman Nahim, Peizhong Ju
Department of Computer Science, University of Kentucky, Lexington, Kentucky 40506, USA
Generative control policies (GCPs), such as diffusion- and flow-based control policies, have emerged as effective parameterizations for robot learning. This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient algorithm for finetuning GCPs that maintains off-policy critic networks to maximize data reuse and propagate policy gradients through the full generative process of the policy via a modified PPO objective, using critics as the terminal reward. OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control. To our knowledge, it is also the only method that can fine-tune poorly-initialized behavior cloning policies to near full task-success with no expert data in the online replay buffer, and does so with few task-specific hyperparameter tuning. Through extensive empirical investigations, we demonstrate that OGPO drastically outperforms methods alternatives on policy steering and learning residual corrections, and identify the key mechanisms behind its performance. We further introduce practical stabilization tricks, including success-buffer regularization, two-sided conservative advantages, and Q-variance reduction, to mitigate critic over-exploitation across state- and pixel-based settings. Beyond proposing OGPO, we conduct a systematic empirical study of GCP finetuning, identifying the stabilizing mechanisms and failure modes that govern successful off-policy full-policy improvement.