Authors: Chubin Zhang, Zhenglin Wan, Feng Chen, Fuchao Yang, Lang Feng, Yaxin Zhou, Xingrui Yu, Yang You, Ivor Tsang, Bo An
Organizations: Nanyang Technological University, Singapore · National University of Singapore, Singapore · Carnegie Mellon University, United States · CFAR, Agency for Science, Technology and Research, Singapore · IHPC, Agency for Science, Technology and Research, Singapore
Reinforcement learning (RL) faces a persistent tension: policies that are stable to optimize (e.g., Gaussians) are often too simple to represent the multimodal action distributions required for complex control. Conversely, expressive generative policies---such as diffusion and flow matching---can be difficult to optimize in online RL due to intractable likelihoods and gradients propagating through long sampling chains. We address this tension with a key structural principle: decoupling optimization from generation. Building on this, we introduce GoRL (Generative Online Reinforcement Learning), an algorithm-agnostic framework that trains expressive policies from scratch by confining policy optimization to a tractable latent space while delegating action synthesis to a conditional generative decoder. Using a two-timescale alternating schedule and anchoring decoder refinement to a fixed prior, GoRL enables stable optimization while continuously expanding expressiveness. Empirically, GoRL consistently outperforms unimodal and generative baselines across diverse continuous-control tasks. Notably, GoRL achieves returns exceeding 870 on HopperStand, more than 3× the strongest baseline; on high-dimensional humanoid tasks, it further outperforms the strongest non-GoRL baseline by over an order of magnitude. Code is available at https://github.com/bennidict23/GoRL.
Figures & tables
Figure 1: The Mode-Covering Problem. When the optimal action distribution (blue) is multimodal, a unimodal Gaussian policy (red) must place nontrivial probability mass in the low-reward region between modes. This yields suboptimal actions and brittle performance in environments requiring precise mode selection.
Figure 2: Overview of the GoRL framework. (a) Latent optimization: The decoder gϕ is frozen while the encoder πθ is optimized in latent space using standard RL algorithms (e.g., PPO or SAC). (b) Decoder refinement: The encoder is frozen and the decoder gϕ is updated via supervised generative training on recent rollouts, using a fixed Gaussian prior as refinement inputs. Periodic stage-wise re-initialization of πθ improves stability across stage transitions.
Figure 3: Learning curves across six DMControl tasks. Vertical dashed lines mark decoder-refinement boundaries at 60M, 120M, and 150M steps. Shaded regions denote standard deviation across five seeds. GoRL achieves higher final returns and more stable learning than Gaussian PPO and generative baselines across tasks.
Figure 4: Mechanism Ablations. (a) Refining the decoder on evolving latents leads to performance collapse, whereas anchoring to a fixed prior maintains stability. (b) Stage-wise re-initialization prevents performance drops at stage boundaries compared to the baseline without re-initialization.
Method
CheetahRun
FingerSpin
FingerTurnHard
FishSwim
HopperStand
WalkerWalk
PPO
724.83 ± 155.67
539.03 ± 146.63
738.70 ± 114.45
433.70 ± 73.63
286.09 ± 273.07
825.65 ± 79.70
FPO
599.15 ± 297.45
56.05 ± 124.53
752.08 ± 55.39
204.66 ± 191.49
3.94 ± 1.79
29.00 ± 4.32
DPPO
559.79 ± 99.97
694.06 ± 191.59
633.84 ± 88.21
143.52 ± 26.25
2.14 ± 0.81
345.59 ± 64.45
GoRL-Diff
902.24 ± 2.20
844.74 ± 59.43
884.59 ± 26.95
608.61 ± 22.07
874.63 ± 38.79
908.96 ± 30.45
GoRL-FM
883.40 ± 19.94
903.92 ± 104.08
860.83 ± 14.93
641.01 ± 13.10
733.66 ± 223.76
919.61 ± 60.86
Table 1: Final episodic returns (mean ± std) over five seeds at 180M steps.
Method
HumanoidStand
HumanoidRun
PPO
74.97 ± 4.58
17.39 ± 2.31
FPO
0.00 ± 0.00
0.00 ± 0.00
DPPO
26.57 ± 6.06
6.38 ± 1.74
GoRL-FM
950.27 ± 20.24
337.61 ± 9.06
GoRL-Diff
976.10 ± 2.20
326.30 ± 29.50
Table 2: High-dimensional humanoid evaluation. Final returns are averaged over three seeds; stage-boundary values are mean returns of GoRL-Diff .
Figure 5: Quantifying decoder improvement across stages. We train fresh encoders against decoders frozen at different refinement stages. Each curve estimates the capability ceiling of a fixed decoder. The monotonic lift from Stage 0 to Stage 3 shows that decoder refinement progressively expands the action support available to latent optimization.
Figure 6: Evolution of action distributions on HopperStand . Gaussian KDE plots of policy outputs. Each GoRL checkpoint is sampled using the stage-end encoder and current decoder, before any subsequent decoder refinement and encoder re-initialization. At 60M steps, GoRL is unimodal. By 180M, it develops a clear bimodal structure, whereas Gaussian PPO remains unimodal.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Visual overview of the DMControl tasks. The benchmark suite encompasses a diverse range of dynamics: high-speed planar locomotion ( CheetahRun ), bipedal gait control ( WalkerWalk ), contact-rich manipulation ( FingerSpin , FingerTurnHard ), and fine-grained stabilization tasks ( HopperStand , FishSwim ).
Task
Action dim.
Observation dim.
CheetahRun
6
17
HopperStand
4
15
WalkerWalk
6
24
FingerSpin
2
9
FingerTurnHard
2
9
FishSwim
5
24
Appendix
Table 3: Observation and action dimensions for the selected DMControl benchmarks.
Hyperparameter
Value
Optimizer
Adam
Learning rate
1×10−3
Batch size per update
1024
Rollout horizon
30
Optimization epochs
16
Discount factor γ
0.995
Appendix
Table 4: Shared PPO hyperparameters.
Figure 8: Cost–Benefit Analysis. Training time vs. final return, normalized to Gaussian PPO ( 1.0× ). Results are averaged over CheetahRun , FingerSpin , and HopperStand . While generative policies naturally incur training overhead, only GoRL translates this cost into positive performance gains.
Environment
Hopper-v2
Walker2D-v2
HalfCheetah-v2
Observation Dim
11
17
17
Action Dim
3
6
6
SAC Optimization
Actor LR
3×10−4
3×10−4
3×10−4
Batch Size
256
256
256
Discount ( γ )
0.99
0.99
0.99
Appendix
Table 5: Hyperparameters for GoRL(SAC) experiments on Gym Locomotion.
Figure 9: Off-policy compatibility check (Gym Benchmarks). Learning curves comparing standard Gaussian SAC with GoRL instantiated using an SAC encoder and Diffusion decoder. The results demonstrate that the GoRL factorization can be successfully trained with off-policy algorithms.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.