cs.CLOct 29, 2026

Li2^{2}SAO: Lifting Light Single-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping

Authors: Zichen Bai, Yanli Wang, Yang Liu, Junhao Shi, Xueqin Liu, Jinguo Sun, Muye Li, Shaoqiang Zheng, 

Organizations: JD Tech Genius Team

Abstract

Asynchronous reinforcement learning (RL) is becoming a mainstream paradigm for post-training large language models (LLMs), as it improves training efficiency by updating the model as rollouts arrive. Recent work argues that group-wise sampling in GRPO does not naturally fit asynchronous RL training because rollout lengths follow a long-tailed distribution, and proposes Single-rollout Asynchronous Optimization (SAO) to improve stability and off-policy behavior, especially on agentic tasks. However, SAO relies on a closed-source value model obtained from scaling value pretraining. In this paper, we analyze GRPO and LiSAO, our lightweight reproduction of SAO without the pretrained value model, on single-turn reasoning and multi-turn agentic tasks. We find that GRPO outperforms LiSAO on reasoning tasks, while the reverse holds on agentic tasks. Since the value model of LiSAO is not yet well trained , LiSAO lags behind GRPO substantially in early training steps. To address this issue, we propose Li2^{2}SAO, which Lifts Light Single-rollout Asynchronous Optimization with group-relative actor-critic bootstrapping: while the critic is cold, a group-relative advantage trains the actor and a group-centered return trains the critic, after which training switches to single-rollout GAE. Experiments show that Li2^{2}SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling. Code is available at https://github.com/jd-cc-coder/Li2SAO.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

    Jul 8, 2026Zhenyu Hou, Yujiang Li, Jie Tang +1Agentic RLAsynchronous RL

  2. SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

    Aug 20, 2026Dayang Liang, Lang Feng, Bo An +1RL for Language ModelsAgentic RL

  3. Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis

    Oct 1, 2026Qijia He, Ruinan Jin, Jun Luo +2Asynchronous RLRL for Language Model Reasoning