FlashREINFORCE: Critic-Free Single-Rollout Asynchronous RL for Agentic Language Models
Organizations: NVIDIA
Abstract
Asynchronous reinforcement learning suits long-horizon agents with irregular rollout times, but group-relative updates require repeated rollouts of each prompt. However, group-based sampling limits prompt coverage at a fixed rollout budget and introduces group synchronization barriers. One rollout per prompt avoids these constraints but requires learning from stale, variable-length trajectories without a sibling baseline. We present FlashREINFORCE, to our knowledge the first critic-free single-rollout framework that scales to long asynchronous training and large models. It combines One-Batch REINFORCE, a Sequence Trust Region, and Sample-Mean Optimization, which provide prompt coverage and batch centering while controlling policy drift and length-amplified negative updates. Each component is easy to implement, and the result is their combination: a batch-mean baseline is stable under sample-mean loss, and each common alternative degrades or collapses training. FlashREINFORCE remains stable through 6,000 updates on DeepSeek-R1-Distill-Qwen-1.5B at policy lag four, and through 1,450 on the 30B MoE Qwen3-30B-A3B at lag eight. Under identical loss, data, and lag it matches a strengthened group-relative baseline within 0.5 points, and with half the rollouts exceeds published synchronous baselines on Qwen2.5-Math-1.5B (38.0 versus 36.3 five-benchmark mean). On Qwen2.5-7B-Instruct it maintains tool use where GRPO stops calling tools and achieves 98.3%/96.5% seen/unseen ALFWorld success.
Figures & tables
| Method | No | One rollout | Asynchronous |
|---|---|---|---|
| critic | per prompt | training | |
| GRPO ( Shao et al., 2024 ) | ✓ | – | group wait |
| SAO ( Hou et al., 2026 ) | – | ✓ | ✓ |
| C-RF + NTF ( Wu et al., 2026 ) | ✓ | ✓ | – |
| FlashREINFORCE | ✓ | ✓ | ✓ |
| Method | AMC23 | Minerva | AIME25 | Mean | Calls |
|---|---|---|---|---|---|
| (a) Qwen2.5-7B-Instruct, step 600 | |||||
| GRPO | 51.2 | 32.3 | 7.5 | 30.3 | 0.00 |
| FlashREINFORCE | 60.6 | 28.8 | 21.7 | 37.0 | 3.25 |
| (b) Qwen3-30B-A3B (MoE), step 800 | |||||
| GRPO | 93.8 | 40.4 | 46.7 | 60.3 | 1.19 |
| FlashREINFORCE | 95.6 | 44.9 | 60.8 | 67.1 | 1.94 |
| Batch mean | Group mean | |
|---|---|---|
| Rollouts | 128 prompts 1 | 32 prompts 4 |
| Advantage | minus batch mean | minus prompt mean |
| Model | DeepSeek-R1-Distill-Qwen-1.5B | |
| Data | The same 1,460 MATH problems | |
| Loss | Token IS, gate ( ), sample mean | |
| Training | Asynchronous, lag ; AdamW, peak lr | |
| Method | Source | Seen | Unseen |
|---|---|---|---|
| GRPO, | Published | 78.6 | 76.8 |
| C-RF + NTF, | Published | 90.5 | 86.3 |
| FlashREINFORCE , | Ours | 98.3 | 96.5 |
| Learning signal | AMC23 | Minerva | AIME25 | Mean | Calls |
|---|---|---|---|---|---|
| Batch-centered | 57.5 | 30.2 | 20.8 | 36.2 | 3.30 |
| Positive-only | 12.5 | 16.9 | 0.0 | 9.8 | 0.00 |
| Admission | Steps | AMC23 | Train reward | Outcome |
|---|---|---|---|---|
| Sequence-level | 2,160 | 55.6 53.1 | 0.23 | Stable |
| Token-level | 1,237 | 51.9 26.3 | 0.18 0.01 | Collapsed |
| Setting | Step | Acc. (%) | KL ( ) |
|---|---|---|---|
| With sequence trust | 4,992 | 32.2 | 6.73 |
| Without trust | 4,992 | 30.6 | 7.38 |
| With sequence trust | 5,896 | 33.7 | 8.96 |
| Without trust | 5,888 | 30.6 | 9.45 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Data and sampling | |
|---|---|
| Training data | sail/Sanity-Test-R1D-1.5B : 1,460 MATH problems with initial solve rates of 20–80% ( Qi et al., 2026 ) |
| Token budget | Prompt: 1,024; response: 8,192; total: 9,216 tokens |
| Rollout sampling | Temperature ; top- |
| Evaluation | AIME24 and AIME25; 30 problems each; 32 samples per problem; every 128 updates from step 0 |
| Evaluation sampling | Temperature ; top- ; 8,192 generated tokens |
| Update and optimization | |
| With sequence trust | Without trust (IS only) | |||||
|---|---|---|---|---|---|---|
| Step | AIME24 | AIME25 | Mean | AIME24 | AIME25 | Mean |
| 0 | 22.71 | 20.63 | 21.67 | 22.71 | 20.42 | 21.56 |
| 1,024 | 28.13 | 23.33 | 25.73 | 27.60 | 24.06 | 25.83 |
| 2,048 | 31.98 | 26.35 | 29.17 | 28.85 | 25.63 | 27.24 |
| 3,072 | 32.08 | 27.92 | 30.00 | 33.13 | 26.46 | 29.79 |
| 3,968 | 35.42 | 26.25 | 30.83 | 34.58 | 26.77 | 30.68 |
| Setting | Value |
|---|---|
| Training data | DAPO-style math prompts |
| Online batch | 128 prompts one rollout |
| Interaction budget | 10 tool turns; 8,192 tokens at comparison |
| Generation budget | 6,144 tokens across turns |
| Reward | Exact committed answer |
| Optimization | AdamW; cosine lr |
| Setting | Value |
|---|---|
| Model | Qwen3-30B-A3B; thinking disabled |
| Training data | Deduplicated 7.5k DAPO-Math prompts, formatted for tool use |
| Rollouts per update | FlashREINFORCE : 128 prompts ; GRPO: 32 prompts |
| Policy lag | FlashREINFORCE : ; GRPO: |
| Interaction budget | Up to 10 turns; 8,192 generated tokens per turn; 18,432 total tokens |
| Rollout sampling | Temperature ; top- |
| Model and Data | |||
| Model | Qwen2.5-Math-1.5B | Prompts | Deduplicated 7.5k DAPO-Math |
| Token budget | 2,048 prompt; 4,096 response | Online batch | 128 prompts; one rollout each |
| Optimization | |||
| Update | One full-batch step; no minibatches or reuse | Optimizer | AdamW; lr ; wd |
| Schedule | Cosine; gradient clip | Sequence threshold | |
| Collection and Sampling | |||
| Step | Rollouts | MATH-500 | AMC23 | Minerva | AIME25 | OlympiadBench | Mean |
|---|---|---|---|---|---|---|---|
| 1,000 | 128k | 66.4 | 50.5 | 21.3 | 9.0 | 32.5 | 35.94 |
| 1,500 | 192k | 68.2 | 51.6 | 22.8 | 10.2 | 33.6 | 37.29 |
| 2,000 | 256k | 69.2 | 52.5 | 22.8 | 10.8 | 34.5 | 37.97 |
| Step 1,000 | Step 2,000 | Step 3,000 | Step 4,000 | |
|---|---|---|---|---|
| 36.04 | 36.89 | 36.93 | 37.09 | |
| 35.94 | 37.97 | 37.75 | 37.58 |
| Setting | Value |
|---|---|
| Training data | 1,024 training games |
| Online batch | 64 games one rollout |
| Interaction budget | 50 environment actions; 16,384 total tokens |
| Generation budget | 8,192 tokens across turns |
| Reward | Terminal environment success |
| Optimization | AdamW; constant lr |
| Setting | Step 1,000 | Step 2,000 | Step 3,000 | Step 4,000 | |
|---|---|---|---|---|---|
| No filtering | 1.0 | 36.04 | 36.89 | 36.93 | 37.09 |
| Light entropy filtering | 0.9 | 36.03 | 37.91 | 37.40 | 37.01 |
| Aggressive entropy filtering | 0.2 | 37.04 | 36.80 | – | – |
RL for LLM Agents
Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.
See all 660 papers