cs.LGOct 13, 2026

Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards

Authors: Yuante Li, Junwei Quan, Kai Qiu, Kate Zhang, Yicheng Tao

Organizations: Carnegie Mellon University · University of Toronto

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reinforcement learning with verifiable rewards (RLVR) relies on costly model-generated responses, motivating larger rollout batches and square-root learning-rate scaling. Reported to approximately preserve progress per sample with one optimizer step per rollout, this rule is less understood in practical multi-step training. We develop a two-clock framework that separates the samples needed to make progress from those consumed before collapse. Under a local quadratic model, the square-root rule preserves first-order drift per sample but not progress: larger batches take fewer, longer updates and lose more to curvature, yielding a learning-rate-dependent critical batch beyond which sample efficiency degrades. A drift-threshold hypothesis predicts a complementary failure clock whose sample count remains approximately invariant along the square-root rule. Controlled GRPO experiments on Qwen2.5-Math-1.5B across a 16×16\times batch range support both predictions. Ratio clipping delays failure, especially for larger batches, but the progress gap persists without it. Widening the clip trades stability for a transient speed gain. The progress gap reappears on another dataset, a general-purpose checkpoint, and a 7B model. Finally, we use the two-clock framework to derive a simple multi-step RLVR schedule: start below the critical batch and switch to a larger batch before failure. Our code is available at: https://github.com/WinstonLiyt/two-clocks-rlvr.

Figures & tables

Figure 1: The two clocks under the square-root rule with μ=8\mu{=}8 . Color marks the rollout batch, from 512 in blue to 8,192 in dark red. Left, with the clip: samples to reach 15% and 20% accuracy, with the critical-batch curve (gray) between perfect invariance (dotted) and the step-limited extreme (dashed). Right: samples to failure without the clip, against the failure-law band (gray), with an open marker for the run that never failed. Progress slows with the batch, while failure stays on the failure law.
Figure 2: Schematic of the two-clock model, with small (blue) and large (red) batches at the same first-order drift per sample. (A) The large batch covers this drift in fewer, longer steps that lose more progress to curvature, so it needs more samples to reach the target τ\tau . (B) Token log-ratios spread with each step, about 8\sqrt{8} times faster for the large batch, and the clip cuts tokens outside the band. (C) Unclipped, both fail at similar sample counts, and the clip delays failure, most for the large batch. (D) A batch ramp thus starts below B∗B^{*} and switches before failure.
cellPPBrB_{\mathrm{r}}μ\mubbη\eta
S64512864ηS\eta_{S}
M192153681923 ηS\sqrt{3}\,\eta_{S}
L512409685128 ηS\sqrt{8}\,\eta_{S}
XL10248192810244 ηS4\,\eta_{S}
S / L64 / 512512 / 40961512 / 4096ηS\eta_{S} / 8 ηS\sqrt{8}\,\eta_{S}
S645124 / 16128 / 32ηS\eta_{S}
Table 1: Configurations. PP is the number of prompts per rollout, BrB_{\mathrm{r}} the responses per rollout, μ\mu the optimizer steps per rollout, and bb the responses per optimizer step. We name runs by their rollout batch, S, M, L or XL, with μ=8\mu{=}8 unless stated otherwise, and the μ=4\mu{=}4 and μ=16\mu{=}16 runs are trained without the clip.
Figure 3: Learning curves of four rollout batches under the square-root rule with ηS=5×10−6\eta_{S}{=}5{\times}10^{-6} and μ=8\mu{=}8 . Color marks the rollout batch, and bands span two seeds where available. The 1,536 batch closely tracks the 512 batch, while larger batches require progressively more samples.
μ\muηS\eta_{S}JSJ_{S}JLJ_{L}DAUCD_{\mathrm{AUC}}N.15N_{.15} L/Sα\alpha
110−610^{-6}0.1900.1710.0221.430.17
15×10−65{\times}10^{-6}0.2720.2550.0201.230.13
810−610^{-6}0.2510.2400.0161.400.15
85×10−65{\times}10^{-6}0.229/0.048 ¶0.2540.0391.720.28
810−510^{-5}0.199/0.013 ‡0.2280.0222.340.44
8 D5×10−65{\times}10^{-6}0.3050.3470.0292.04 ♭0.37
Table 2: Small versus large batch. JJ is the final accuracy (seed mean), DAUCD_{\mathrm{AUC}} the mean distance between the learning curves (seed noise 0.005–0.009), N.15N_{.15} L/S the ratio of the samples needed to reach 15% accuracy, and α\alpha the pooled sample-cost exponent. Entries marked ¶ and ‡ list both seeds, the second of which fails late. Rows marked D use DeepScaleR, I Qwen2.5-1.5B-Instruct and 7B Qwen2.5-Math-7B with one seed, and values marked ♮ are at 49k samples and ♭ at a model-specific target.
Figure 4: Clip on and off for three rollout batches with ηS=10−5\eta_{S}{=}10^{-5} and μ=8\mu{=}8 . Color marks the rollout batch and line style the clip (solid on, dashed off), with bands over clipped seeds. Left: accuracy with crosses at failure. Right: gradient norm with dotted lines at failure. Unclipped runs fail at similar sample counts despite unequal step counts.
Figure 5: Observed failure points of unclipped runs against the failure law, Eq. ( 7 ). Color marks the rollout batch, marker shape μ\mu , and a black edge the second seed, and open markers with arrows are right-censored. The gray band spans a factor of 1.5 around equality and holds 14 of the 18 failing runs.
Figure 6: Failure point against learning rate for the 512 and 4,096 batches with μ=8\mu{=}8 , with and without the clip. Color marks the rollout batch, filled circles runs without the clip, open squares runs with it, arrows runs that never failed, and lines the failure law, Eq. ( 7 ). Every clipped run fails later than its unclipped counterpart, and the large batch gains most.
Figure 7: Batch ramp with ηS=5×10−6\eta_{S}{=}5{\times}10^{-6} and μ=8\mu{=}8 . Each ramp uses batch 512 for 24.6k samples (dotted line), then switches to 4,096 or 8,192 (two seeds each), and thin lines are fixed batches. Both ramps lie well above the fixed large batches, near the best fixed batch.
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

prompts per rollout PP64 (S), 192 (M), 512 (L), 1024 (XL)
responses per prompt GG8
optimizer steps per rollout μ\mu8 (main), 1, 4 and 16 (controls), mini-batch of P/μP/\mu prompts
learning rateηS∈{10−6,5×10−6,10−5}\eta_{S}\in\{10^{-6},5{\times}10^{-6},10^{-5}\} and the clip-off sweeps of App. D , other batches by the square-root rule
schedulelinear warm-up over 8192 responses, then constant
optimizerAdamW, β=(0.9,0.999)\beta=(0.9,0.999) , weight decay 0.1
gradient-norm clipdisabled ( 10610^{6} ), 1.0 in one control
Appendix
Table 3: Hyperparameters shared by all runs.
Figure 8: Accuracy gap between the 512 and 4,096 batches, J512(N)−J4096(N)J_{512}(N)-J_{4096}(N) , for all five pairs. The two left panels have μ=1\mu{=}1 , where the clip cannot act, and the three right panels μ=8\mu{=}8 . Bands cover all seed pairings, and the gray band marks ±1\pm 1 accuracy point, the reference band that prior one-step scaling results use to judge approximate invariance ( Li et al., 2026 ) . At μ=8\mu{=}8 the conservative gap is an early lead of 2–4 points that decays into the band, the 5×10−65{\times}10^{-6} gap reaches 8 points and stays above the band until the small batch decays late, and the 10−510^{-5} gap peaks at 9 points early and turns negative when the small batch fails.
Figure 9: Samples needed to first reach each accuracy target against the rollout batch, on log–log axes, for every pair. The slope of each line is the sample-cost exponent α\alpha of Table 2 , a missing point means that the run never reaches the target within 98k samples, and the family with ηS=5×10−6\eta_{S}{=}5{\times}10^{-6} , μ=8\mu{=}8 and all four batches is in Fig. 1 (left). The samples needed rise with the batch at every target.
Figure 10: The 512 and 4,096 batches at three learning-rate levels with μ=8\mu{=}8 . Color marks the rollout batch, the larger batch uses 8\sqrt{8} times the smaller batch’s rate ηS\eta_{S} , and bands span two seeds where available. The small batch’s lead grows with the rate, and at the highest rate it closes late, when the small batch starts to fail.
Figure 11: The 512 and 4,096 batches in the conservative regime, ηS=10−6\eta_{S}{=}10^{-6} , with μ=1\mu{=}1 (left, two seeds) and μ=8\mu{=}8 (right, one seed). Color marks the rollout batch, and bands span the two seeds. The larger batch lags in both, although less than 2×10−42{\times}10^{-4} of its tokens are clipped.
Figure 12: Clipped token fraction at each optimizer step kk within a rollout for the 512 and 4,096 batches at three learning-rate levels with μ=8\mu{=}8 , averaged over the first 49k samples, before any of these runs fails. Color marks the rollout batch and line style the learning-rate level, and bands span two seeds where available. Nothing is clipped at k=0k{=}0 , both fractions then rise with kk , and the large batch’s is 1.1–1.6 times the small batch’s at k=1k{=}1 and 2–4 times from k=3k{=}3 on.
free fitβ\betaγ\gammaβ−γ/2\beta-\gamma/2
least squares (18 runs)0.24 [0.00, 0.52]0.64 [0.29, 0.96]−0.08-0.08 [ −0.23-0.23 , 0.09]
Tobit (21 runs, 3 censored)0.47 [0.19, 0.72]0.72 [0.32, 1.04]0.11 [ −0.09-0.09 , 0.32]
fixed formrms factorlog-lik.LR test
b/η\sqrt{b}/\eta (Eq. 7 , C=0.044C{=}0.044 )1.53−15.3-15.3p≈0.29p\approx 0.29
b/η2b/\eta^{2} (random walk)2.32−24.7-24.7p<10−4p<10^{-4}
1/η1/\eta (batch-blind)1.68−23.9-23.9p<10−4p<10^{-4}
Appendix
Table 4: Fits of the failure law Nfail∝bβη−γN_{\mathrm{fail}}\propto b^{\beta}\eta^{-\gamma} . Brackets give 90% bootstrap intervals, and the square-root rule moves the failure point by the factor bβ−γ/2b^{\beta-\gamma/2} . The rms factor is the geometric rms residual of the failing runs, or the fitted noise scale for Tobit, the log-likelihood is censored, and the LR test compares each fixed form with the free Tobit fit.
runcliponset rolloutonset NNacc. halves runcliponset rolloutonset NNacc. halves
S 10−510^{-5} seed 0onnone †–– S 10−510^{-5} ( μ=16\mu{=}16 )offnone–34k
S 10−510^{-5} seed 1on15378k83k M 1.7×10−51.7{\times}10^{-5} seed 1off2031k33k
S 5×10−65{\times}10^{-6} seed 0on18796k– M 10−510^{-5}off4874k81k
S 5×10−65{\times}10^{-6} seed 1on15680k92k M 10−510^{-5} seed 1off2437k48k
S 1.7×10−51.7{\times}10^{-5}onnone–71k L 2.8×10−52.8{\times}10^{-5} s0/s1onnone in 24>> 98k–
S 2×10−52{\times}10^{-5}onnone–36k L 1.4×10−51.4{\times}10^{-5} s0/s1onnone in 24>> 98k–
Appendix
Table 5: Failure points of every run in the active regimes under both parts of the failure criterion (§ 4 ). The columns give the sustained gradient-norm onset in rollouts and in samples, and the sample at which accuracy first halves from its best value, with – for runs that never halve, and clip ε×8\varepsilon{\times}\sqrt{8} is the widened radius. Marks denote a decay from the peak without a blow-up ( † ), a decay from 57k ( ‡ ), a quiet collapse in which accuracy falls while the gradient norm stays near baseline except for one spike ( § ), a second seed rerun on different hardware ( ♮ ), a 40% fall in accuracy from 41k without halving ( ♭ ), a collapse followed by recovery ( †† ), and a KL blow-up with every response at the length cap and the gradient exactly zero afterwards, rather than a gradient-norm excursion ( ♯ ).
Figure 13: Replayed rotation of the update by the clip, measured as one minus the cosine between the clipped and unclipped gradients, at each optimizer step kk within a rollout with μ=8\mu{=}8 (median over checkpoints). Color marks the rollout batch and line style the learning-rate level, the dash-dot line is the 4,096 batch with a widened clip, and the gray line is the clip-off control, which marks the numerical floor. At k=0k{=}0 every ratio is one and the clip cannot act, and with ηS=10−5\eta_{S}{=}10^{-5} the large batch’s rotation is about 0.2 from k=2k{=}2 on, 6 to 50 times the small batch’s.
Figure 14: Widened clip for the 4,096 batch with ηS=10−5\eta_{S}{=}10^{-5} and μ=8\mu{=}8 , against the 512 batch. Color marks the rollout batch and line style the clip radius, solid for the DAPO radius and dashed for the log-space radius multiplied by 8\sqrt{8} , and bands span two seeds. The widened radius gives the large batch the small batch’s shape, with an earlier rise and then decay, and one seed collapses.
Figure 15: Qwen2.5-1.5B-Instruct with one seed per run, comparing the 512 and 4,096 batches with μ=1\mu{=}1 and ηS=5×10−6\eta_{S}{=}5{\times}10^{-6} (left) and with μ=8\mu{=}8 and ηS=2×10−6\eta_{S}{=}2{\times}10^{-6} (right). Color marks the rollout batch. The small batch leads early in both settings.
Figure 16: Switch-point sweep with ηS=5×10−6\eta_{S}{=}5{\times}10^{-6} and μ=8\mu{=}8 , in which one trajectory of the 512 batch (seed 0) switches to the 4,096 batch after 12k, 25k, 49k or 74k samples. Dashed lines are the switched runs, darker for later switches, solid lines the fixed 512, 1,536 and 4,096 batches (seed means), and dotted verticals the switches. Switches at 25k, 49k and 74k perform similarly, and only the earliest, at 12k, gives up part of the small batch’s lead.
More in this topic

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

Explore similar work

CardsList
Jul 28, 2026

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal. Existing remedies either oversample a larger candidate pool and discard saturated prompts (dynamic sampling), paying heavy extra rollouts, or predict prompt difficulty before sampling, which is fragile under a shifting policy. We observe that a group's effectiveness is usually decided early, within the first few of its rollouts, so spending a full group on an already-decided prompt is wasteful. We cast per-step rollout collection as a budget-constrained sequential allocation (optimal stopping) problem and introduce SARA (Sequential Adaptive Rollout Allocation). SARA maintains a Beta posterior over each prompt's success rate, evaluates a closed-form predictor of group effectiveness, and applies a two-threshold, SPRT-style rule that commits effective groups, abandons saturated ones after a short probe, and reallocates the freed budget to fresh prompts, without any extra prediction rollouts. We prove abandonment reliability, expected rollout savings, fixed-budget yield dominance, and a link between effective-group yield and the GRPO gradient norm. On mathematical reasoning and planning with 1.5B/3B models on a single GPU, SARA matches DPS (both below the DS oracle) while using 22% fewer rollouts than DS; composing SARA with DPS yields the best accuracy, slightly above DS, at 67% fewer rollouts (near-uniform cost).

Pixel Nomand, Elena Voss, Marcus Hale +1

University of Wisconsin–Madison · University of Washington

Reinforcement LearningRL for Language Model ReasoningReinforcement Learning with Verifiable RewardsSample-Efficient RL
AskPDF
Oct 16, 2026

Batched Contextual Reinforcement II: Does RLVR Need Long Rollout Windows?

Long rollout windows make reinforcement learning with verifiable rewards (RLVR) expensive, yet simply shortening them can remove the correctness signal: when every single-problem rollout truncates before an answer, group-relative accuracy advantages vanish. As a technical continuation of Batched Contextual Reinforcement (BCR), we study how concurrent rewards relax this dependence on long windows. Concurrent Constraint Reinforcement (CCR) places three independently verifiable problems in one capped training completion. Correct answers completed before truncation retain credit, while a shared budget encourages shorter useful reasoning without an explicit length penalty. We connect this mechanism to conditional reward- and update-variance analyses and study 4B and 8B mathematical reasoning models. Against single-problem GRPO with a 32k window, CCR with an 8,000-token window reduces tokens per training step from 975,464 to 174,957 on 4B and from 793,832 to 165,788 on 8B, saving 82.1% and 79.1%, respectively. An 8B training trace shows increasing accuracy reward alongside shorter completions. Single-problem evaluations provide supporting evidence of retained reasoning quality. These results identify reward-preserving training structure as a practical route to lower per-step cost in short-window RLVR.

Bangji Yang, Hongbo Ma, Ruihan Guo +3

University of Illinois at Urbana-Champaign · Tsinghua University · Rutgers University

Reinforcement LearningRL for Language Model ReasoningReinforcement Learning with Verifiable Rewards
AskPDF
Apr 9, 2026

An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used for post-training Large Language Models, but practical verifiers can make errors. We study how the rate and structure of reward noise affect RLVR in code generation, with a preliminary scientific-reasoning check. In multi-seed Qwen3 8B experiments on MBPP, mean validation reward over two fixed late evaluations is within 1 percentage point of the clean baseline at the tested resampled group-rollout noise rates through 20%, and within about 2 points at 30%. Confidence intervals allow larger losses; these point estimates do not establish a general tolerance threshold. We also examine full-program pass@k, four controlled noise structures, two model-based verifiers, and policy models from three families spanning 4B-9B parameters. We derive conditional advantage distributions for symmetric and asymmetric group noise, including retained format penalties, and show why clipping limits simple gradient-scaling arguments. The analysis identifies information preserved by whole-group corruption and limits on interpreting our asymmetric sweep as a precision-recall comparison. Overall, the results indicate that imperfect verification can support effective RLVR in the tested settings, while aggregate error rates alone do not characterize the learning signal.

Andreas Plesner, Francisco Guzmán, Anish Athalye

Handshake AI · ETH Zurich

RL for Code GenerationReinforcement LearningReinforcement Learning with Verifiable RewardsRL Post-Training
AskPDF