Reinforcement learning with verifiable rewards (RLVR) implemented with Group Relative Policy Optimization (GRPO) has driven substantial progress in training reasoning language models. However, verifiable outcome rewards provide an imperfect measure of reasoning quality: a correct final answer may not be supported by the reasoning trace, whereas an incorrect rollout may contain a nearly correct reasoning trajectory. We present empirical evidence for this mismatch and hypothesize that the resulting reward noise contributes to instability in reinforcement learning for both reasoning accuracy and token efficiency. To address this credit assignment challenge, this paper proposes double-rollout reinforcement learning. For each sampled rollout, we truncate its reasoning trace, sample multiple continuations from the truncated prefix, and replace the original reward with the mean continuation reward, yielding a more robust estimate of reasoning quality. This double-rollout algorithm incurs only a moderate increase in compute cost and yet reaches substantially higher accuracies across math and coding tasks and across Qwen and Nemotron models than baseline algorithms with fewer RL steps. For example, using only DAPO data it trains Nemotron-3-Nano-30B-A3B to reach 95.4% on AIME 2026, versus 91.5% by GRPO and substantially above state of the art for this model size, within 30 RL steps. When combined with length-based rewards, the same algorithm further improves the accuracy-efficiency Pareto frontier over existing efficiency optimizations. These findings suggest that the double-rollout reward estimation is an effective and cost-efficient way to improve both the accuracy and token efficiency of reasoning RL.
Figures & tables
Figure 1: Average correctness of a re-generation continuing from 90% of a correct rollout’s thinking trace. X-axis is the length of the original correct rollout. The prompts are DAPO prompts divided into three subsets by difficulty.
Figure 2: Average correctness of a re-generation continuing from 90% of an incorrect rollout’s thinking trace. X-axis is the length of the original incorrect rollout. The prompts are DAPO prompts divided into three subsets by difficulty.
Start model
Algorithm
RL with DAPO prompts
RL with OCR prompts
AIME 2024
AIME 2025
AIME 2026
LCB v5
LCB v6
Start
90.0%
88.3%
89.6%
72.4%
67.6%
Nemotron-3-Nano
GRPO
91.7%
89.2%
91.5%
72.5%
70.9%
-30B-A3B
GRPO w/ large rollout
90.8%
89.0%
91.9%
70.7%
70.6%
Double-rollout (ours)
92.1%
93.8%
95.4%
74.6%
70.7%
Start
92.5%
83.8%
87.1%
68.3%
66.3%
Table 1: Comparing the double-rollout RL algorithm for max accuracy against RLVR baselines on best achievable accuracies.
Param
RL with DAPO prompts
AIME 2024
AIME 2025
AIME 2026
0-25%
degradation from start model
50%
93.5%
91.7%
94.8%
75%
92.5%
91.5%
93.8%
90%
92.1%
93.8%
95.4%
99%
93.5%
90.0%
90.6%
Table 2: Ablation on truncation hyperparameter: percentage of thinking trace to keep. Double-rollout RL experiments using DAPO data on Nemotron-3-Nano-30B-A3B.
Param
RL with DAPO prompts
AIME 2024
AIME 2025
AIME 2026
1
91.7%
89.2%
91.5%
2
92.7%
91.5%
94.6%
4
92.7%
91.7%
94.0%
8
92.1%
93.8%
95.4%
16
91.5%
92.9%
95.2%
Table 3: Ablation on the number of continuations per original rollout. Double-rollout RL experiments using DAPO data on Nemotron-3-Nano-30B-A3B.
Figure 3: Comparing the accuracy-efficiency Pareto frontier by the double-rollout algorithm against baselines on RL results with DAPO data on Nemotron-3-Nano-30B-A3B.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Reinforcement learning with verifiable rewards (RLVR) has emerged as a highly effective framework for improving LLM reasoning, with methods such as GRPO among its most successful instantiations. However, GRPO relies on repeated generation of long chain-of-thought rollouts. Training time scales with the number of rollouts, a large fraction of which are uninformative. Thus, GRPO is computationally expensive and unstable. To mitigate this, existing approaches either generate a larger pool of rollouts and filter the most informative prompts, or leverage historical signals for filtering at later stages of training. These strategies offer modest performance gains, but slow down the overall process. To address this, we propose VarIance Guided Online Rollout allocation (VIGOR) which instead of allocating a fixed rollout budget per example, begins with a small number of rollouts for all examples in a batch and iteratively allocates additional rollouts to those with the highest group reward variance until a fixed total rollout budget is reached. Theoretically, we show that under RLVR, reward variance controls the gradient magnitude, and derive VIGOR's closed-form speedup ratio over GRPO, which grows with refinement rounds under Pareto-distributed reward variance. Experiments on mathematical reasoning and coding tasks show that VIGOR reaches target accuracy with up to 2.3× fewer rollouts on math, reaches GRPO's final coding full pass rate with 1.49× fewer rollouts, and improves the coding average test pass rate by 3.4 points.
Reinforcement learning is crucial for improving large language models' reasoning and generalization. It relies on massive rollouts whose lengths become increasingly long-tailed as context windows grow. In on-policy training, these long-tail rollouts can result in GPU bubbles, reducing system utilization and limiting RL scalability. Asynchronous or partial-rollout methods improve throughput by relaxing synchronization, but inevitably introduce stale off-policy samples (trajectories) that may hurt final accuracy. Existing approaches mainly mitigate this off-policy issue by reweighting off-policy samples during training, yet they can still leave a performance gap compared to fully on-policy training. In this work, rather than passively reweighting samples during training, we propose RollVerify, a lightweight RL framework built on partial rollout that actively verifies and repairs samples before they enter training. Specifically, it introduces an off-policy shift metric OPS, to quantify the off-policy deviation of partially generated trajectories. Guided by the OPS constraint, RollVerify performs both sequence-level and token-level verification to identify and truncate invalid suffixes of trajectories. This yields high-quality samples that protect the models' accuracy while preserving the efficiency gains of partial rollout. Experiments on mathematical and tool-assisted mathematical reasoning show that RollVerify achieves accuracy comparable to on-policy training while reducing training cost. Additional code-generation results provide preliminary evidence beyond mathematics.
Yongqiang Yao, Jinru Tan, Kaihuan Liang +5
Shanghai Jiao Tong University · Central South University · SenseTime Research +2
Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual reasoning tokens or actions remains difficult. Group-relative methods such as Group Relative Policy Optimization (GRPO) sample several attempts per prompt, but lose their reward-based learning signal when all attempts receive the same outcome. We propose Hindsight-Privileged RLVR (HP-RLVR), a single-rollout method that turns a completed trajectory's own future into token-level update weights. The generating policy snapshot scores each eligible token under its causal history and under a hindsight context containing the realized future and verifier feedback. A normalized Jensen--Shannon divergence between the two views weights the reward relative to a historical baseline, yielding bounded token credit for both successful and failed attempts, with no reference solution or learned critic. In paired Qwen3-8B runs over a shared 100-step schedule, HP-RLVR trains 1.21×, 1.46×, and 2.30× faster than GRPO on mathematics, code generation, and ALFWorld, using one eighth the rollouts per prompt. Accuracy is comparable overall, with HP-RLVR ahead on mathematics at all three sizes and GRPO ahead on code and ALFWorld. In ALFWorld, 43.2% (4B) and 49.0% (8B) of GRPO's gradient-norm records are zero, versus none for HP-RLVR.
Wei Chen, Yubing Wu, Zhou Xu +9
South China University of Technology · Tsinghua University · The Hong Kong University of Science and Technology +3