Asynchronous reinforcement learning (RL) is becoming a mainstream paradigm for post-training large language models (LLMs), as it improves training efficiency by updating the model as rollouts arrive. Recent work argues that group-wise sampling in GRPO does not naturally fit asynchronous RL training because rollout lengths follow a long-tailed distribution, and proposes Single-rollout Asynchronous Optimization (SAO) to improve stability and off-policy behavior, especially on agentic tasks. However, SAO relies on a closed-source value model obtained from scaling value pretraining. In this paper, we analyze GRPO and LiSAO, our lightweight reproduction of SAO without the pretrained value model, on single-turn reasoning and multi-turn agentic tasks. We find that GRPO outperforms LiSAO on reasoning tasks, while the reverse holds on agentic tasks. Since the value model of LiSAO is not yet well trained , LiSAO lags behind GRPO substantially in early training steps. To address this issue, we propose Li2SAO, which Lifts Light Single-rollout Asynchronous Optimization with group-relative actor-critic bootstrapping: while the critic is cold, a group-relative advantage trains the actor and a group-centered return trains the critic, after which training switches to single-rollout GAE. Experiments show that Li2SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling. Code is available at https://github.com/jd-cc-coder/Li2SAO.
Figures & tables
Figure 1: GRPO vs. LiSAO on single-turn mathematical reasoning (left) and ReTool agentic tool use (right). Each half shows Pass@1 across training steps and Gap (pp) across Pass@ k . Gap (pp) is GRPO minus LiSAO, and a positive value means GRPO is ahead. Bold curves average AIME 2024, 2025, and 2026. Thin curves are the individual years. The complete per-year pass@ k results are in Figure 6 of Appendix E .
Figure 2: Training accuracy of GRPO and LiSAO. Curves are a two-sided EMA (span 10) of per-step 0/1 accuracy. The inset shows the first 200 steps. LiSAO starts at step 10, after critic warmup.
Figure 3: Critic Dynamics of LiSAO over the first 200 steps. Insets magnify steps 5 to 50 and steps 50 to 200 to show the cold-start recovery and the later positive fit. The critic is updated from step 0, including 10 warmup steps before actor updates.
Figure 4: Overview of Li 2 SAO. GRPO waits for a complete prompt group and assigns one group-relative advantage to every action token. LiSAO consumes each rollout on arrival, but a cold critic makes early GAE noisy. Li 2 SAO bootstraps the actor with group-relative advantages and the critic with centered returns, divergence masking, and terminal ranking, then switches to single-rollout GAE when residual explained variance and ranking accuracy exceed the thresholds.
AIME 2024
AIME 2025
AIME 2026
Avg.
Pass@ k
1
2
4
8
16
1
2
4
8
16
1
2
4
8
16
1
2
4
8
16
Single-turn Mathematical Reasoning
Base
21.5
29.3
36.7
45.6
56.7
19.0
24.7
29.5
34.3
40.0
16.0
22.4
30.2
40.0
50.0
18.8
25.5
32.1
40.0
48.9
GRPO
66.7
74.4
79.2
81.5
83.3
61.3
69.9
74.4
78.4
83.3
62.9
69.8
74.6
78.9
83.3
63.6
71.4
76.1
79.6
83.3
LiSAO
69.2
76.7
80.6
81.7
83.3
59.2
69.6
76.9
81.4
83.3
57.3
67.0
74.4
80.9
86.7
61.9
71.1
77.3
81.3
84.4
Li 2 SAO
72.5
80.1
84.2
85.9
90.0
65.0
72.3
76.7
79.8
83.3
63.5
71.3
76.9
81.6
86.7
67.0
74.6
79.3
82.4
86.7
Table 1: Pass@ k (%) at step 500 on AIME 2024, 2025, and 2026. Base and SFT are the shared checkpoints before RL: single-turn reasoning starts from the base model and ReTool from a tool-use supervised checkpoint. All rows use 16 samples per problem at temperature 1.0 and the unbiased estimator of Chen et al. (2021) . Avg. is the mean over the three years. The best results are highlighted in Bold .
Figure 5: Early pass@ k at steps 100 and 200. Each bar averages AIME 2024, 2025, and 2026.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Single-turn
ReTool
Base Model
Qwen3-4B
tool-use SFT
Generation
text, no tools
Python interpreter
Max turns
1
16
Timeout
none
120 s
Evaluation
no tools
same sandbox
Appendix
Table 2: Task-specific settings.
Setting
Value
Policy
Qwen3-4B, thinking off
Training set
DAPO-Math-17K
Context length
32,768
Reward
binary outcome
Overlong penalty
last quarter, factor 1
Rollout steps
500
Appendix
Table 3: Shared training and evaluation configuration.
Setting
GRPO
LiSAO
Samples per prompt
8
1
Prompts per batch
16
128
Advantage
group-relative
GAE
Clip range
0.2 / 0.28
0.3 / 5.0
DIS interval
none
[0.7,6.0]
Critic
none
frozen attention
Appendix
Table 4: Algorithm-specific configuration for the GRPO and LiSAO comparison.
Figure 6: GRPO vs. LiSAO on single-turn mathematical reasoning (left) and ReTool agentic tool use (right). Results for each task are reported separately on AIME 2024, 2025, and 2026. The top row is pass@1 from the shared initialization to step 500. Each lower row is pass@ k at the labeled checkpoint.
Setting
Bootstrap
After switch
Samples per prompt
8
1
Prompts per batch
16
128
Advantage
group-relative
GAE
Critic target
group-centered
absolute
Value clip ϵv
0.2
inactive
Ranking α , m
0.1, 0.1
inactive
Appendix
Table 5: Li 2 SAO training configuration. Shared training choices are in Table 3 . The clip range, DIS interval, critic architecture, critic learning rate, and K match LiSAO in Table 4 .
Figure 7: Li 2 SAO against GRPO and LiSAO. Results are shown separately for single-turn mathematical reasoning (left) and ReTool agentic tool use (right), and for AIME 2024, 2025, and 2026. The top row is pass@1 from the shared initialization to step 500. Each lower row is pass@ k at the labeled checkpoint.
Figure 8: Training accuracy of GRPO, LiSAO, and Li 2 SAO. Curves are a two-sided EMA (span 10) of per-step 0/1 accuracy on the training batches. The inset shows the first 200 steps. Gold stars mark the switch, at step 142 on reasoning and step 137 on ReTool. LiSAO starts at step 10, after critic warmup.
Figure 9: Mean response length of GRPO, LiSAO, and Li 2 SAO. Curves are a two-sided EMA (span 10) of per-step mean response length on the training batches. LiSAO starts at step 10, after critic warmup.
Step 100
Step 200
Step 300
Step 400
Step 500
Variant Pass@ k
1
2
4
8
16
1
2
4
8
16
1
2
4
8
16
1
2
4
8
16
1
2
4
8
16
Single-turn Mathematical Reasoning
Li 2 SAO
47.9
59.3
69.2
77.0
80.0
67.3
74.7
78.5
81.8
83.3
67.7
76.1
80.0
83.2
86.7
71.0
77.9
81.2
84.9
86.7
72.5
80.1
84.2
85.9
90.0
w/o residual target
40.8
50.0
58.8
68.9
80.0
57.1
66.2
71.3
73.2
73.3
61.0
70.5
77.6
83.0
86.7
65.0
72.5
77.3
81.0
83.3
68.1
74.6
77.1
78.3
80.0
w/o ranking loss
46.3
57.4
67.9
75.9
80.0
60.8
70.7
77.6
81.2
83.3
65.0
74.0
79.5
81.6
83.3
68.5
76.8
81.3
83.0
83.3
66.9
76.5
80.9
82.6
83.3
ReTool Agentic Tool Use
Appendix
Table 6: Ablations of Li 2 SAO. Each row keeps the group-relative actor and the readiness switch, and changes one critic target. The three tables report AIME 2024, 2025, and 2026 separately. Pass@ k at steps 100, 200, 300, 400, and 500 uses 16 samples per problem at temperature 1.0 and the unbiased estimator of Chen et al. (2021) .
Method
Train + Rollout
Prompts × Samples
Hours
Single-turn Mathematical Reasoning
GRPO
4 + 4
16 × 8
51.5
GRPO
2 + 6
16 × 8
36.4
LiSAO
4 + 4
128 × 1
31.8
Li 2 SAO
4 + 4
16 × 8 → 128 × 1
28.9
ReTool Agentic Tool Use
Appendix
Table 7: Wall-clock time of 500 training steps for single-turn mathematical reasoning and ReTool agentic tool use. Train + Rollout is the GPU split between the trainer and the rollout workers. In LiSAO and Li 2 SAO, the actor and the critic share the trainer GPUs. Prompts × Samples is the number of prompts in each step times the responses drawn for each prompt.
Figure 10: Mean policy lag of GRPO, LiSAO, and Li 2 SAO. Curves are a two-sided EMA (span 10) of the per-step mean policy lag. GRPO is the training-batch mean. LiSAO and Li 2 SAO are the rollout mean, one policy version lower on runs that log both. LiSAO starts at step 10, after critic warmup.
Figure 11: Off-policy token ratio of GRPO, LiSAO, and Li 2 SAO. Curves are a two-sided EMA (span 10) of the fraction of response tokens generated more than one policy version behind the trainer. A one-step lag is counted as on-policy. Every curve is the training-batch ratio. LiSAO starts at step 10, after critic warmup.
Setting
Value
Agent scaffold
OpenHands
RL backend
modified slime
Weight synchronization
agent-step-level, in flight
Sandbox
instance-specific Docker
Compute
8 nodes ×8 A800 GPUs
Context length
128 K
Appendix
Table 8: Shared configuration for coding agentic RL.
Method
Resolved Rate (%)
Qwen3-30B-A3B-Instruct (thinking off)
Base
21.6
GRPO
24.6
DAPO
24.2
GSPO
25.8
LiSAO
25.4
Appendix
Table 9: SWE-bench Verified resolved rate (%) after coding agentic RL on SWE-Gym.
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.