Asynchronous reinforcement learning (RL) is becoming a mainstream paradigm for post-training large language models (LLMs), as it improves training efficiency by updating the model as rollouts arrive. Recent work argues that group-wise sampling in GRPO does not naturally fit asynchronous RL training because rollout lengths follow a long-tailed distribution, and proposes Single-rollout Asynchronous Optimization (SAO) to improve stability and off-policy behavior, especially on agentic tasks. However, SAO relies on a closed-source value model obtained from scaling value pretraining. In this paper, we analyze GRPO and LiSAO, our lightweight reproduction of SAO without the pretrained value model, on single-turn reasoning and multi-turn agentic tasks. We find that GRPO outperforms LiSAO on reasoning tasks, while the reverse holds on agentic tasks. Since the value model of LiSAO is not yet well trained , LiSAO lags behind GRPO substantially in early training steps. To address this issue, we propose Li2SAO, which Lifts Light Single-rollout Asynchronous Optimization with group-relative actor-critic bootstrapping: while the critic is cold, a group-relative advantage trains the actor and a group-centered return trains the critic, after which training switches to single-rollout GAE. Experiments show that Li2SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling. Code is available at https://github.com/jd-cc-coder/Li2SAO.
Figures & tables
Figure 1: GRPO vs. LiSAO on single-turn mathematical reasoning (left) and ReTool agentic tool use (right). Each half shows Pass@1 across training steps and Gap (pp) across Pass@ k . Gap (pp) is GRPO minus LiSAO, and a positive value means GRPO is ahead. Bold curves average AIME 2024, 2025, and 2026. Thin curves are the individual years. The complete per-year pass@ k results are in Figure 6 of Appendix E .
Figure 2: Training accuracy of GRPO and LiSAO. Curves are a two-sided EMA (span 10) of per-step 0/1 accuracy. The inset shows the first 200 steps. LiSAO starts at step 10, after critic warmup.
Figure 3: Critic Dynamics of LiSAO over the first 200 steps. Insets magnify steps 5 to 50 and steps 50 to 200 to show the cold-start recovery and the later positive fit. The critic is updated from step 0, including 10 warmup steps before actor updates.
Figure 4: Overview of Li 2 SAO. GRPO waits for a complete prompt group and assigns one group-relative advantage to every action token. LiSAO consumes each rollout on arrival, but a cold critic makes early GAE noisy. Li 2 SAO bootstraps the actor with group-relative advantages and the critic with centered returns, divergence masking, and terminal ranking, then switches to single-rollout GAE when residual explained variance and ranking accuracy exceed the thresholds.
AIME 2024
AIME 2025
AIME 2026
Avg.
Pass@ k
1
2
4
8
16
1
2
4
8
16
1
2
4
8
16
1
2
4
8
16
Single-turn Mathematical Reasoning
Base
21.5
29.3
36.7
45.6
56.7
19.0
24.7
29.5
34.3
40.0
16.0
22.4
30.2
40.0
50.0
18.8
25.5
32.1
40.0
48.9
GRPO
66.7
74.4
79.2
81.5
83.3
61.3
69.9
74.4
78.4
83.3
62.9
69.8
74.6
78.9
83.3
63.6
71.4
76.1
79.6
83.3
LiSAO
69.2
76.7
80.6
81.7
83.3
59.2
69.6
76.9
81.4
83.3
57.3
67.0
74.4
80.9
86.7
61.9
71.1
77.3
81.3
84.4
Li 2 SAO
72.5
80.1
84.2
85.9
90.0
65.0
72.3
76.7
79.8
83.3
63.5
71.3
76.9
81.6
86.7
67.0
74.6
79.3
82.4
86.7
Table 1: Pass@ k (%) at step 500 on AIME 2024, 2025, and 2026. Base and SFT are the shared checkpoints before RL: single-turn reasoning starts from the base model and ReTool from a tool-use supervised checkpoint. All rows use 16 samples per problem at temperature 1.0 and the unbiased estimator of Chen et al. (2021) . Avg. is the mean over the three years. The best results are highlighted in Bold .
Figure 5: Early pass@ k at steps 100 and 200. Each bar averages AIME 2024, 2025, and 2026.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Single-turn
ReTool
Base Model
Qwen3-4B
tool-use SFT
Generation
text, no tools
Python interpreter
Max turns
1
16
Timeout
none
120 s
Evaluation
no tools
same sandbox
Appendix
Table 2: Task-specific settings.
Setting
Value
Policy
Qwen3-4B, thinking off
Training set
DAPO-Math-17K
Context length
32,768
Reward
binary outcome
Overlong penalty
last quarter, factor 1
Rollout steps
500
Appendix
Table 3: Shared training and evaluation configuration.
Setting
GRPO
LiSAO
Samples per prompt
8
1
Prompts per batch
16
128
Advantage
group-relative
GAE
Clip range
0.2 / 0.28
0.3 / 5.0
DIS interval
none
[0.7,6.0]
Critic
none
frozen attention
Appendix
Table 4: Algorithm-specific configuration for the GRPO and LiSAO comparison.
Figure 6: GRPO vs. LiSAO on single-turn mathematical reasoning (left) and ReTool agentic tool use (right). Results for each task are reported separately on AIME 2024, 2025, and 2026. The top row is pass@1 from the shared initialization to step 500. Each lower row is pass@ k at the labeled checkpoint.
Setting
Bootstrap
After switch
Samples per prompt
8
1
Prompts per batch
16
128
Advantage
group-relative
GAE
Critic target
group-centered
absolute
Value clip ϵv
0.2
inactive
Ranking α , m
0.1, 0.1
inactive
Appendix
Table 5: Li 2 SAO training configuration. Shared training choices are in Table 3 . The clip range, DIS interval, critic architecture, critic learning rate, and K match LiSAO in Table 4 .
Figure 7: Li 2 SAO against GRPO and LiSAO. Results are shown separately for single-turn mathematical reasoning (left) and ReTool agentic tool use (right), and for AIME 2024, 2025, and 2026. The top row is pass@1 from the shared initialization to step 500. Each lower row is pass@ k at the labeled checkpoint.
Figure 8: Training accuracy of GRPO, LiSAO, and Li 2 SAO. Curves are a two-sided EMA (span 10) of per-step 0/1 accuracy on the training batches. The inset shows the first 200 steps. Gold stars mark the switch, at step 142 on reasoning and step 137 on ReTool. LiSAO starts at step 10, after critic warmup.
Figure 9: Mean response length of GRPO, LiSAO, and Li 2 SAO. Curves are a two-sided EMA (span 10) of per-step mean response length on the training batches. LiSAO starts at step 10, after critic warmup.
Step 100
Step 200
Step 300
Step 400
Step 500
Variant Pass@ k
1
2
4
8
16
1
2
4
8
16
1
2
4
8
16
1
2
4
8
16
1
2
4
8
16
Single-turn Mathematical Reasoning
Li 2 SAO
47.9
59.3
69.2
77.0
80.0
67.3
74.7
78.5
81.8
83.3
67.7
76.1
80.0
83.2
86.7
71.0
77.9
81.2
84.9
86.7
72.5
80.1
84.2
85.9
90.0
w/o residual target
40.8
50.0
58.8
68.9
80.0
57.1
66.2
71.3
73.2
73.3
61.0
70.5
77.6
83.0
86.7
65.0
72.5
77.3
81.0
83.3
68.1
74.6
77.1
78.3
80.0
w/o ranking loss
46.3
57.4
67.9
75.9
80.0
60.8
70.7
77.6
81.2
83.3
65.0
74.0
79.5
81.6
83.3
68.5
76.8
81.3
83.0
83.3
66.9
76.5
80.9
82.6
83.3
ReTool Agentic Tool Use
Appendix
Table 6: Ablations of Li 2 SAO. Each row keeps the group-relative actor and the readiness switch, and changes one critic target. The three tables report AIME 2024, 2025, and 2026 separately. Pass@ k at steps 100, 200, 300, 400, and 500 uses 16 samples per problem at temperature 1.0 and the unbiased estimator of Chen et al. (2021) .
Method
Train + Rollout
Prompts × Samples
Hours
Single-turn Mathematical Reasoning
GRPO
4 + 4
16 × 8
51.5
GRPO
2 + 6
16 × 8
36.4
LiSAO
4 + 4
128 × 1
31.8
Li 2 SAO
4 + 4
16 × 8 → 128 × 1
28.9
ReTool Agentic Tool Use
Appendix
Table 7: Wall-clock time of 500 training steps for single-turn mathematical reasoning and ReTool agentic tool use. Train + Rollout is the GPU split between the trainer and the rollout workers. In LiSAO and Li 2 SAO, the actor and the critic share the trainer GPUs. Prompts × Samples is the number of prompts in each step times the responses drawn for each prompt.
Figure 10: Mean policy lag of GRPO, LiSAO, and Li 2 SAO. Curves are a two-sided EMA (span 10) of the per-step mean policy lag. GRPO is the training-batch mean. LiSAO and Li 2 SAO are the rollout mean, one policy version lower on runs that log both. LiSAO starts at step 10, after critic warmup.
Figure 11: Off-policy token ratio of GRPO, LiSAO, and Li 2 SAO. Curves are a two-sided EMA (span 10) of the fraction of response tokens generated more than one policy version behind the trainer. A one-step lag is counted as on-policy. Every curve is the training-batch ratio. LiSAO starts at step 10, after critic warmup.
Setting
Value
Agent scaffold
OpenHands
RL backend
modified slime
Weight synchronization
agent-step-level, in flight
Sandbox
instance-specific Docker
Compute
8 nodes ×8 A800 GPUs
Context length
128 K
Appendix
Table 8: Shared configuration for coding agentic RL.
Method
Resolved Rate (%)
Qwen3-30B-A3B-Instruct (thinking off)
Base
21.6
GRPO
24.6
DAPO
24.2
GSPO
25.8
LiSAO
25.4
Appendix
Table 9: SWE-bench Verified resolved rate (%) after coding agentic RL on SWE-Gym.
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).
Zhenyu Hou, Yujiang Li, Jie Tang +1
Tsinghua University · Work done while ZH and YL interned at Z.AI.
Agentic reinforcement learning (RL) has emerged as an important post-training approach for enhancing the capabilities of Large Language Models (LLMs). However, existing methods face a trade-off between policy performance and resource efficiency. Conventional Proximal Policy Optimization (PPO) implementations incur substantial memory overhead from a separate critic, whereas critic-free group-relative methods require multiple rollouts and face potential learning bottlenecks on long-horizon tasks. In this work, we propose Single-rollout Autoregressive Policy Optimization (SAPO), an efficient PPO-style framework that unifies policy optimization and value learning within a single causal language model. SAPO exploits the autoregressive structure of LLMs to sequentially generate action and value estimation at distinct causal boundaries with shared parameters, and then jointly optimizes the PPO objectives and an auxiliary on-policy SARSA objective with turn-level generalized advantage estimation, where the latter is designed to facilitate value learning. Extensive experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B and Qwen3-14B demonstrate that SAPO reduces peak GPU memory usage by 24.8% and per-iteration runtime by 23.1% over strong PPO baseline, while matching or slightly improving task success rate. Our experiments also show that SAPO outperforms Group Relative Policy Optimization (GRPO) and recent cutting-edge variants in both task success and training stability.
Dayang Liang, Lang Feng, Bo An +1
Xiamen University, Xiamen, China · Nanyang Technological University, Singapore
Asynchronous reinforcement learning (RL) improves the efficiency of large language model post-training but introduces stale rollouts generated by earlier policies. Theoretical understanding of how this staleness affects convergence and how to mitigate its impact remains limited. We derive a convergence bound for GRPO-style algorithms that explicitly characterizes the tradeoff between the gradient estimator's second moment and bias. For trajectory-level importance-weighted estimators, our analysis shows that once the second moment is uniformly controlled, delay enters the bound through the bias introduced by clipping or rescaling. Guided by this insight, we propose a novel group mass capping GRPO (GMC-GRPO) method, which minimizes a ratio-based bias bound within a class of weighted estimators sharing a common second-moment guarantee. We establish convergence guarantees for asynchronous GMC-GRPO and show that, compared with TIC-GRPO, it improves the threshold dependence of the fourth-order delay term from O(ε−4) to O(ε−2) as ε→0, where 1+ε is the ratio threshold. Under local policy overlap, the delay-dependent term decreases as G−2/5 after tuning the step size, where G is the group size. For fixed behavior and current policies, the bias introduced by group rescaling also vanishes as G→∞, whereas the bias from trajectory-wise clipping can persist. Experiments across Qwen3 models and reasoning benchmarks demonstrate improved robustness to stale rollouts, with GMC-GRPO achieving the best performance among stable baselines under large rollout delays.
Qijia He, Ruinan Jin, Jun Luo +2
The Ohio State University · Arizona State University