cs.LGOct 13, 2026
SaveTwo Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards
Organizations: Carnegie Mellon University · University of Toronto
Abstract
Reinforcement learning with verifiable rewards (RLVR) relies on costly model-generated responses, motivating larger rollout batches and square-root learning-rate scaling. Reported to approximately preserve progress per sample with one optimizer step per rollout, this rule is less understood in practical multi-step training. We develop a two-clock framework that separates the samples needed to make progress from those consumed before collapse. Under a local quadratic model, the square-root rule preserves first-order drift per sample but not progress: larger batches take fewer, longer updates and lose more to curvature, yielding a learning-rate-dependent critical batch beyond which sample efficiency degrades. A drift-threshold hypothesis predicts a complementary failure clock whose sample count remains approximately invariant along the square-root rule. Controlled GRPO experiments on Qwen2.5-Math-1.5B across a batch range support both predictions. Ratio clipping delays failure, especially for larger batches, but the progress gap persists without it. Widening the clip trades stability for a transient speed gain. The progress gap reappears on another dataset, a general-purpose checkpoint, and a 7B model. Finally, we use the two-clock framework to derive a simple multi-step RLVR schedule: start below the critical batch and switch to a larger batch before failure. Our code is available at: https://github.com/WinstonLiyt/two-clocks-rlvr.
Figures & tables
Figure 1: The two clocks under the square-root rule with . Color marks the rollout batch, from 512 in blue to 8,192 in dark red. Left, with the clip: samples to reach 15% and 20% accuracy, with the critical-batch curve (gray) between perfect invariance (dotted) and the step-limited extreme (dashed). Right: samples to failure without the clip, against the failure-law band (gray), with an open marker for the run that never failed. Progress slows with the batch, while failure stays on the failure law.
Figure 2: Schematic of the two-clock model, with small (blue) and large (red) batches at the same first-order drift per sample. (A) The large batch covers this drift in fewer, longer steps that lose more progress to curvature, so it needs more samples to reach the target . (B) Token log-ratios spread with each step, about times faster for the large batch, and the clip cuts tokens outside the band. (C) Unclipped, both fail at similar sample counts, and the clip delays failure, most for the large batch. (D) A batch ramp thus starts below and switches before failure.
| cell | |||||
| S | 64 | 512 | 8 | 64 | |
| M | 192 | 1536 | 8 | 192 | |
| L | 512 | 4096 | 8 | 512 | |
| XL | 1024 | 8192 | 8 | 1024 | |
| S / L | 64 / 512 | 512 / 4096 | 1 | 512 / 4096 | / |
| S | 64 | 512 | 4 / 16 | 128 / 32 |
Table 1: Configurations. is the number of prompts per rollout, the responses per rollout, the optimizer steps per rollout, and the responses per optimizer step. We name runs by their rollout batch, S, M, L or XL, with unless stated otherwise, and the and runs are trained without the clip.
Figure 3: Learning curves of four rollout batches under the square-root rule with and . Color marks the rollout batch, and bands span two seeds where available. The 1,536 batch closely tracks the 512 batch, while larger batches require progressively more samples.
| L/S | ||||||
| 1 | 0.190 | 0.171 | 0.022 | 1.43 | 0.17 | |
| 1 | 0.272 | 0.255 | 0.020 | 1.23 | 0.13 | |
| 8 | 0.251 | 0.240 | 0.016 | 1.40 | 0.15 | |
| 8 | 0.229/0.048 ¶ | 0.254 | 0.039 | 1.72 | 0.28 | |
| 8 | 0.199/0.013 ‡ | 0.228 | 0.022 | 2.34 | 0.44 | |
| 8 D | 0.305 | 0.347 | 0.029 | 2.04 ♭ | 0.37 |
Table 2: Small versus large batch. is the final accuracy (seed mean), the mean distance between the learning curves (seed noise 0.005–0.009), L/S the ratio of the samples needed to reach 15% accuracy, and the pooled sample-cost exponent. Entries marked ¶ and ‡ list both seeds, the second of which fails late. Rows marked D use DeepScaleR, I Qwen2.5-1.5B-Instruct and 7B Qwen2.5-Math-7B with one seed, and values marked ♮ are at 49k samples and ♭ at a model-specific target.
Figure 4: Clip on and off for three rollout batches with and . Color marks the rollout batch and line style the clip (solid on, dashed off), with bands over clipped seeds. Left: accuracy with crosses at failure. Right: gradient norm with dotted lines at failure. Unclipped runs fail at similar sample counts despite unequal step counts.
Figure 5: Observed failure points of unclipped runs against the failure law, Eq. ( 7 ). Color marks the rollout batch, marker shape , and a black edge the second seed, and open markers with arrows are right-censored. The gray band spans a factor of 1.5 around equality and holds 14 of the 18 failing runs.
Figure 6: Failure point against learning rate for the 512 and 4,096 batches with , with and without the clip. Color marks the rollout batch, filled circles runs without the clip, open squares runs with it, arrows runs that never failed, and lines the failure law, Eq. ( 7 ). Every clipped run fails later than its unclipped counterpart, and the large batch gains most.
Figure 7: Batch ramp with and . Each ramp uses batch 512 for 24.6k samples (dotted line), then switches to 4,096 or 8,192 (two seeds each), and thin lines are fixed batches. Both ramps lie well above the fixed large batches, near the best fixed batch.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| prompts per rollout | 64 (S), 192 (M), 512 (L), 1024 (XL) |
|---|---|
| responses per prompt | 8 |
| optimizer steps per rollout | 8 (main), 1, 4 and 16 (controls), mini-batch of prompts |
| learning rate | and the clip-off sweeps of App. D , other batches by the square-root rule |
| schedule | linear warm-up over 8192 responses, then constant |
| optimizer | AdamW, , weight decay 0.1 |
| gradient-norm clip | disabled ( ), 1.0 in one control |
Table 3: Hyperparameters shared by all runs.
Figure 8: Accuracy gap between the 512 and 4,096 batches, , for all five pairs. The two left panels have , where the clip cannot act, and the three right panels . Bands cover all seed pairings, and the gray band marks accuracy point, the reference band that prior one-step scaling results use to judge approximate invariance ( Li et al., 2026 ) . At the conservative gap is an early lead of 2–4 points that decays into the band, the gap reaches 8 points and stays above the band until the small batch decays late, and the gap peaks at 9 points early and turns negative when the small batch fails.
Figure 9: Samples needed to first reach each accuracy target against the rollout batch, on log–log axes, for every pair. The slope of each line is the sample-cost exponent of Table 2 , a missing point means that the run never reaches the target within 98k samples, and the family with , and all four batches is in Fig. 1 (left). The samples needed rise with the batch at every target.
Figure 10: The 512 and 4,096 batches at three learning-rate levels with . Color marks the rollout batch, the larger batch uses times the smaller batch’s rate , and bands span two seeds where available. The small batch’s lead grows with the rate, and at the highest rate it closes late, when the small batch starts to fail.
Figure 11: The 512 and 4,096 batches in the conservative regime, , with (left, two seeds) and (right, one seed). Color marks the rollout batch, and bands span the two seeds. The larger batch lags in both, although less than of its tokens are clipped.
Figure 12: Clipped token fraction at each optimizer step within a rollout for the 512 and 4,096 batches at three learning-rate levels with , averaged over the first 49k samples, before any of these runs fails. Color marks the rollout batch and line style the learning-rate level, and bands span two seeds where available. Nothing is clipped at , both fractions then rise with , and the large batch’s is 1.1–1.6 times the small batch’s at and 2–4 times from on.
| free fit | |||
|---|---|---|---|
| least squares (18 runs) | 0.24 [0.00, 0.52] | 0.64 [0.29, 0.96] | [ , 0.09] |
| Tobit (21 runs, 3 censored) | 0.47 [0.19, 0.72] | 0.72 [0.32, 1.04] | 0.11 [ , 0.32] |
| fixed form | rms factor | log-lik. | LR test |
| (Eq. 7 , ) | 1.53 | ||
| (random walk) | 2.32 | ||
| (batch-blind) | 1.68 |
Table 4: Fits of the failure law . Brackets give 90% bootstrap intervals, and the square-root rule moves the failure point by the factor . The rms factor is the geometric rms residual of the failing runs, or the fitted noise scale for Tobit, the log-likelihood is censored, and the LR test compares each fixed form with the free Tobit fit.
| run | clip | onset rollout | onset | acc. halves | run | clip | onset rollout | onset | acc. halves | |
|---|---|---|---|---|---|---|---|---|---|---|
| S seed 0 | on | none † | – | – | S ( ) | off | none | – | 34k | |
| S seed 1 | on | 153 | 78k | 83k | M seed 1 | off | 20 | 31k | 33k | |
| S seed 0 | on | 187 | 96k | – | M | off | 48 | 74k | 81k | |
| S seed 1 | on | 156 | 80k | 92k | M seed 1 | off | 24 | 37k | 48k | |
| S | on | none | – | 71k | L s0/s1 | on | none in 24 | 98k | – | |
| S | on | none | – | 36k | L s0/s1 | on | none in 24 | 98k | – |
Table 5: Failure points of every run in the active regimes under both parts of the failure criterion (§ 4 ). The columns give the sustained gradient-norm onset in rollouts and in samples, and the sample at which accuracy first halves from its best value, with – for runs that never halve, and clip is the widened radius. Marks denote a decay from the peak without a blow-up ( † ), a decay from 57k ( ‡ ), a quiet collapse in which accuracy falls while the gradient norm stays near baseline except for one spike ( § ), a second seed rerun on different hardware ( ♮ ), a 40% fall in accuracy from 41k without halving ( ♭ ), a collapse followed by recovery ( †† ), and a KL blow-up with every response at the length cap and the gradient exactly zero afterwards, rather than a gradient-norm excursion ( ♯ ).
Figure 13: Replayed rotation of the update by the clip, measured as one minus the cosine between the clipped and unclipped gradients, at each optimizer step within a rollout with (median over checkpoints). Color marks the rollout batch and line style the learning-rate level, the dash-dot line is the 4,096 batch with a widened clip, and the gray line is the clip-off control, which marks the numerical floor. At every ratio is one and the clip cannot act, and with the large batch’s rotation is about 0.2 from on, 6 to 50 times the small batch’s.
Figure 14: Widened clip for the 4,096 batch with and , against the 512 batch. Color marks the rollout batch and line style the clip radius, solid for the DAPO radius and dashed for the log-space radius multiplied by , and bands span two seeds. The widened radius gives the large batch the small batch’s shape, with an earlier rise and then decay, and one seed collapses.
Figure 15: Qwen2.5-1.5B-Instruct with one seed per run, comparing the 512 and 4,096 batches with and (left) and with and (right). Color marks the rollout batch. The small batch leads early in both settings.
Figure 16: Switch-point sweep with and , in which one trajectory of the 512 batch (seed 0) switches to the 4,096 batch after 12k, 25k, 49k or 74k samples. Dashed lines are the switched runs, darker for later switches, solid lines the fixed 512, 1,536 and 4,096 batches (seed means), and dotted verticals the switches. Switches at 25k, 49k and 74k perform similarly, and only the earliest, at 12k, gives up part of the small batch’s lead.
More in this topic
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,661 papers