cs.LGDec 2, 2025

Generative Online Reinforcement Learning

Authors: Chubin Zhang, Zhenglin Wan, Feng Chen, Fuchao Yang, Lang Feng, Yaxin Zhou, Xingrui Yu, Yang You, 

Organizations: Nanyang Technological University, Singapore · National University of Singapore, Singapore · Carnegie Mellon University, United States · CFAR, Agency for Science, Technology and Research, Singapore · 

Abstract

Reinforcement learning (RL) faces a persistent tension: policies that are stable to optimize (e.g., Gaussians) are often too simple to represent the multimodal action distributions required for complex control. Conversely, expressive generative policies---such as diffusion and flow matching---can be difficult to optimize in online RL due to intractable likelihoods and gradients propagating through long sampling chains. We address this tension with a key structural principle: decoupling optimization from generation. Building on this, we introduce GoRL (Generative Online Reinforcement Learning), an algorithm-agnostic framework that trains expressive policies from scratch by confining policy optimization to a tractable latent space while delegating action synthesis to a conditional generative decoder. Using a two-timescale alternating schedule and anchoring decoder refinement to a fixed prior, GoRL enables stable optimization while continuously expanding expressiveness. Empirically, GoRL consistently outperforms unimodal and generative baselines across diverse continuous-control tasks. Notably, GoRL achieves returns exceeding 870 on HopperStand, more than 3×3\times the strongest baseline; on high-dimensional humanoid tasks, it further outperforms the strongest non-GoRL baseline by over an order of magnitude. Code is available at https://github.com/bennidict23/GoRL.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Discrete Flow Matching for Offline-to-Online Reinforcement Learning

    May 12, 2026Fairoz Nower Khan, Nabuat Zaman Nahim, Peizhong JuFlow MatchingReinforcement Learning

  2. OGPO: Sample Efficient Full-Finetuning of Generative Control Policies

    May 4, 2026Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal +14RL for RoboticsRL for Generative Models