Asynchronous reinforcement learning suits long-horizon agents with irregular rollout times, but group-relative updates require repeated rollouts of each prompt. However, group-based sampling limits prompt coverage at a fixed rollout budget and introduces group synchronization barriers. One rollout per prompt avoids these constraints but requires learning from stale, variable-length trajectories without a sibling baseline. We present FlashREINFORCE, to our knowledge the first critic-free single-rollout framework that scales to long asynchronous training and large models. It combines One-Batch REINFORCE, a Sequence Trust Region, and Sample-Mean Optimization, which provide prompt coverage and batch centering while controlling policy drift and length-amplified negative updates. Each component is easy to implement, and the result is their combination: a batch-mean baseline is stable under sample-mean loss, and each common alternative degrades or collapses training. FlashREINFORCE remains stable through 6,000 updates on DeepSeek-R1-Distill-Qwen-1.5B at policy lag four, and through 1,450 on the 30B MoE Qwen3-30B-A3B at lag eight. Under identical loss, data, and lag it matches a strengthened group-relative baseline within 0.5 points, and with half the rollouts exceeds published synchronous baselines on Qwen2.5-Math-1.5B (38.0 versus 36.3 five-benchmark mean). On Qwen2.5-7B-Instruct it maintains tool use where GRPO stops calling tools and achieves 98.3%/96.5% seen/unseen ALFWorld success.
Figures & tables
Method
No
One rollout
Asynchronous
critic
per prompt
training
GRPO ( Shao et al., 2024 )
✓
–
group wait
SAO ( Hou et al., 2026 )
–
✓
✓
C-RF + NTF ( Wu et al., 2026 )
✓
✓
–
FlashREINFORCE
✓
✓
✓
Table 1: Requirements of critic-free and single-rollout methods. FlashREINFORCE is critic-free, uses one rollout per prompt, and trains asynchronously.
Figure 1: Each design choice of FlashREINFORCE (green) and the result of switching it alone to its common alternative (gray; Sec. 4.5 ). Accuracy is the three-benchmark mean unless named; ↓ : lower is better. Token mean is shown 150 updates after the switch: its reward has barely moved, but 45.3% truncation leaves the run untrainable. Token masking collapses near update 1,200. Tool use: Qwen2.5-7B-Instruct; math: Qwen2.5-Math-1.5B.
Figure 2: FlashREINFORCE at a glance: one fresh-batch update, whole-trajectory admission, sample-mean loss.
Figure 3: DeepSeek-R1-Distill-Qwen-1.5B with sequence trust at policy lag ≈4 , through 6,000 cumulative updates. The dashed line marks continuation from the step-5,000 checkpoint ( Tab. 10 ). (a) AIME24/25 mean avg@32 accuracy: dots mark evaluations and curves show a three-checkpoint moving average within each segment. (b,c) Training response length and token entropy, averaged in consecutive 200-update windows.
Method
AMC23
Minerva
AIME25
Mean
Calls
(a) Qwen2.5-7B-Instruct, step 600
GRPO
51.2
32.3
7.5
30.3
0.00
FlashREINFORCE
60.6
28.8
21.7
37.0
3.25
(b) Qwen3-30B-A3B (MoE), step 800
GRPO
93.8
40.4
46.7
60.3
1.19
FlashREINFORCE
95.6
44.9
60.8
67.1
1.94
Table 2: Python-tool avg@32 accuracy (%) at matched steps and rollout budgets. Calls per trajectory are averaged over the 100 updates ending at each checkpoint. In (b), FlashREINFORCE uses δ=10−2 at lag ≈8 and GRPO uses lag ≈1 .
Figure 4: Qwen3-30B-A3B Python-tool accuracy: FlashREINFORCE at lag eight and GRPO at lag one. Markers show avg@32 evaluations, joined by shape-preserving interpolation; the mean equally weights the three benchmarks. R3, the δ=10−3 configuration with routing replay, is shown through 1,450 policy updates.
Batch mean
Group mean
Rollouts
128 prompts × 1
32 prompts × 4
Advantage
Ri minus batch mean
Ri minus prompt mean
Model
DeepSeek-R1-Distill-Qwen-1.5B
Data
The same 1,460 MATH problems
Loss
Token IS, gate ( δ=5×10−3 ), sample mean
Training
Asynchronous, lag ≈4 ; AdamW, peak lr 10−6
Table 3: Setup of the controlled comparison. The batch-mean arm is FlashREINFORCE . Only the first two rows differ.
Figure 5: Batch-mean and group-mean training under the FlashREINFORCE loss, with records through update 3,945 and AIME evaluations through update 3,840. (a,b) avg@32 accuracy at checkpoints; (c,d) consecutive 200-update means and the final 145-update window, joined by lines.
Method
Source
Seen
Unseen
GRPO, G=8
Published
78.6
76.8
C-RF + NTF, G=1
Published
90.5
86.3
FlashREINFORCE , G=1
Ours
98.3
96.5
Table 5: ALFWorld success rate (%) with Qwen2.5-7B-Instruct. Published rows are from Wu et al. (2026) , trained synchronously; FlashREINFORCE is trained asynchronously and evaluated at step 200 after 12.8k training trajectories.
Learning signal
AMC23
Minerva
AIME25
Mean
Calls
Batch-centered
57.5
30.2
20.8
36.2
3.30
Positive-only
12.5
16.9
0.0
9.8
0.00
Table 6: Ablations on Qwen2.5-7B-Instruct tool use. (a) Learning signal at step 700. (b) Loss reduction, continued from the same step-900 checkpoint. Trunc.: share of responses at the token limit.
Admission
Steps
AMC23
Train reward
Outcome
Sequence-level
2,160
55.6 → 53.1
0.23
Stable
Token-level
1,237
51.9 → 26.3
0.18 → 0.01
Collapsed
Table 7: Sequence-level versus token-level admission on Qwen2.5-Math-1.5B with the same drift proxy and threshold ( δ=10−3 ). Arrows show peak to last value.
Setting
Step
Acc. (%)
KL ( ×10−4 )
With sequence trust
4,992
32.2
6.73
Without trust
4,992
30.6
7.38
With sequence trust
5,896
33.7
8.96
Without trust
5,888
30.6
9.45
Table 8: Sequence trust on DeepSeek-R1-Distill-Qwen-1.5B at policy lag 4, with token IS in both runs. Accuracy is the AIME24/25 avg@32 mean; KL is the sampled-action proxy, averaged over updates 4,801–5,000 (first two rows) and 5,801–6,000 (last two).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Data and sampling
Training data
sail/Sanity-Test-R1D-1.5B : 1,460 MATH problems with initial solve rates of 20–80% ( Qi et al., 2026 )
AIME24 and AIME25; 30 problems each; 32 samples per problem; every 128 updates from step 0
Evaluation sampling
Temperature 0.6 ; top- p=0.95 ; 8,192 generated tokens
Update and optimization
Appendix
Table 9: Hyperparameters for the DeepSeek-R1-Distill-Qwen-1.5B trust comparison.
With sequence trust
Without trust (IS only)
Step
AIME24
AIME25
Mean
AIME24
AIME25
Mean
0
22.71
20.63
21.67
22.71
20.42
21.56
1,024
28.13
23.33
25.73
27.60
24.06
25.83
2,048
31.98
26.35
29.17
28.85
25.63
27.24
3,072
32.08
27.92
30.00
33.13
26.46
29.79
3,968
35.42
26.25
30.83
34.58
26.77
30.68
Appendix
Table 10: AIME avg@32 accuracy (%). (a) Trust comparison through 5,000 updates. (b) Continued training within 6,000 cumulative updates; approximate steps align nearby evaluations across arms. Mean equally weights AIME24 and AIME25. Boldface marks the with-trust run’s AIME24 peak.
Setting
Value
Training data
DAPO-style math prompts
Online batch
128 prompts × one rollout
Interaction budget
10 tool turns; 8,192 tokens at comparison
Generation budget
6,144 tokens across turns
Reward
Exact committed answer
Optimization
AdamW; cosine lr 10−6
Appendix
Table 11: Configuration of the Qwen2.5-7B-Instruct Python-tool experiment.
Setting
Value
Model
Qwen3-30B-A3B; thinking disabled
Training data
Deduplicated 7.5k DAPO-Math prompts, formatted for tool use
Up to 10 turns; 8,192 generated tokens per turn; 18,432 total tokens
Rollout sampling
Temperature 1.0 ; top- p=1.0
Appendix
Table 12: Configuration of the Qwen3-30B-A3B Python-tool experiments.
Model and Data
Model
Qwen2.5-Math-1.5B
Prompts
Deduplicated 7.5k DAPO-Math
Token budget
2,048 prompt; 4,096 response
Online batch
128 prompts; one rollout each
Optimization
Update
One full-batch step; no minibatches or reuse
Optimizer
AdamW; lr 10−6 ; wd 0.1
Schedule
Cosine; gradient clip 1.0
Sequence threshold
δ=3×10−3
Collection and Sampling
Appendix
Table 14: Qwen2.5-Math baseline-comparison configuration. Boldface marks the defining one-pass design choices.
Step
Rollouts
MATH-500
AMC23
Minerva
AIME25
OlympiadBench
Mean
1,000
128k
66.4
50.5
21.3
9.0
32.5
35.94
1,500
192k
68.2
51.6
22.8
10.2
33.6
37.29
2,000
256k
69.2
52.5
22.8
10.8
34.5
37.97
Appendix
Table 15: Qwen2.5-Math-1.5B checkpoints through step 2,000 with δ=3×10−3 : avg@32 accuracy (%) and the five-benchmark mean. Boldface marks the checkpoint reported in Tab. 4 .
δ
Step 1,000
Step 2,000
Step 3,000
Step 4,000
1×10−3
36.04
36.89
36.93
37.09
3×10−3
35.94
37.97
37.75
37.58
Appendix
Table 16: Gate-threshold configurations on Qwen2.5-Math-1.5B. Entries are five-benchmark mean avg@32 accuracy (%); the four checkpoints use 128k, 256k, 384k, and 512k training rollouts. Boldface marks the higher mean at each checkpoint.
Setting
Value
Training data
1,024 training games
Online batch
64 games × one rollout
Interaction budget
50 environment actions; 16,384 total tokens
Generation budget
8,192 tokens across turns
Reward
Terminal environment success
Optimization
AdamW; constant lr 10−6
Appendix
Table 17: Configuration of the Qwen2.5-7B-Instruct ALFWorld experiment.
Figure 6: Entropy diagnostic for the optional selector. Left: reward-zero Qwen3.5-4B terminal-task fragments ( Qwen Team, 2026 ) ; boxes mark failure-causing spans. Right: 72% of annotated sites exceed the within-trajectory 80th entropy percentile (versus 20% for length-matched random spans), and 46% exceed the 90th. Top-decile entropy tokens concentrate in reasoning text rather than format-control tokens.
Setting
q
Step 1,000
Step 2,000
Step 3,000
Step 4,000
No filtering
1.0
36.04
36.89
36.93
37.09
Light entropy filtering
0.9
36.03
37.91
37.40
37.01
Aggressive entropy filtering
0.2
37.04
36.80
–
–
Appendix
Table 18: Negative-token filtering on Qwen2.5-Math-1.5B at δ=1×10−3 . Entries are five-benchmark mean avg@32 accuracy (%); q=1 applies no filtering. Boldface marks column bests.
Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).
Zhenyu Hou, Yujiang Li, Jie Tang +1
Tsinghua University · Work done while ZH and YL interned at Z.AI.
Language models are moving beyond generating answers to pursuing long-horizon goals in interactive environments. Post-training these agents requires long, heterogeneous trajectories, and synchronous systems leave learner engines idle until rollout and verification finish. To squeeze out these pipeline bubbles, asynchronous training overlaps rollout and learning across updates, but comes at the cost of policy staleness. We introduce ThunderSyncRL, which starts gradient computation as soon as all required inputs are fixed, without policy staleness. For group relative policy optimization (GRPO), ThunderSyncRL computes each trajectory's score gradient as soon as the reward for that trajectory arrives, without waiting for the group. For on-policy distillation (OPD), it computes gradients for each completed agentic turn's teacher-scored actions while tool calls run in the sandbox. We prove that gradient streaming produces the same GRPO and OPD updates as batch-synchronous training, without changing either objective. On SWE-bench Verified and Terminal Bench 4.0, we train models to the same performance up to 1.9× faster than synchronous training. With zero policy staleness, ThunderSyncRL also outperforms asynchronous training at a fixed budget by up to 2.47 percentage points.
Seil Kang, Hangoo Kang, Tarun Suresh +4
Stanford University · Yonsei University · Korea University +1
Fully asynchronous reinforcement learning (RL) improves resource utilization in large language model post-training by overlapping rollout generation with policy optimization, but it also introduces policy lag as trajectories are generated and queued while the trainer continues to update. We study how this lag accumulates over a trajectory's lifetime and how it can be controlled without sacrificing the wall-clock benefits of asynchronous execution. We decompose trajectory staleness into Generation Staleness, accumulated before rollout completion, and Waiting Staleness, accumulated after a completed trajectory enters the pool. Motivated by this decomposition, we introduce PACE (Pool-Aware Control of Effective Staleness). PACE converts excess pool occupancy into an adaptive rejection budget and ranks completed trajectories using an effective-staleness score that combines Waiting Staleness with prefix-aware Generation Staleness. This avoids penalizing long or interrupted rollouts solely because they span multiple policy versions. In single-turn mathematical reasoning, PACE improves the six-benchmark average validation accuracy by 18.7% over unfiltered asynchronous RL at the same wall-clock budget and matches synchronous RL performance with 47.1% less GPU time. PACE also improves validation performance in multi-turn tool-integrated reasoning, outperforming both synchronous and unfiltered asynchronous RL. Further experiments with the mixture-of-experts model and an alternative RL algorithm support its applicability across model architectures and training algorithms.
Chenliang Li, Neiwen Ling, Zijun Wei +1
ByteDance · Texas A&M University · University of Virginia