Larger batches reduce the variance of stochastic gradients per update and are therefore often expected to accelerate training. Yet whether this statistical benefit translates into lower wall-clock time-to-target remains unclear, because each update consumes more samples and may take longer to execute. We study this tradeoff in reinforcement learning for large language models. We separate its algorithmic and systems effects by comparing learning and execution along their natural axes. At the algorithmic level, we compare configurations at equal cumulative sample counts while retuning batch-dependent hyperparameters. Over a bounded range of batch sizes, this procedure yields an approximately batch-size-invariant family whose members follow similar sample-indexed learning trajectories. At the systems level, we exploit the computational asymmetry between rollout generation and training: autoregressive generation is often memory-bandwidth-bound at low concurrency, whereas training work scales approximately with the number of processed tokens. Combining these two views yields a direct decision rule: a larger-batch configuration reduces time-to-target only when its throughput gain exceeds its samples-to-target penalty. Experiments with GRPO and PPO support both sides of this decomposition. At the algorithmic level, square-root learning-rate scaling with Adam produces approximately batch-size-invariant learning curves over a bounded range of batch sizes. At the systems level, larger batches improve generation throughput by up to 2.29x on fixed hardware. In GRPO, combining higher throughput with learning-rate retuning reduces time-to-target by up to 29%, whereas increasing the batch without retuning is slower despite its higher throughput.
Figures & tables
Figure 1: A data-informed schematic based on selected GRPO operating points from our experiments (Table 1 ). Each trail marker denotes one optimizer update. It highlights three messages: (1) after learning-rate retuning, B , 2B , and 4B are approximately batch-size invariant in cumulative samples; (2) higher realized throughput within this regime improves wall-clock efficiency; and (3) beyond this regime, as at 16B , normalized time-to-target deteriorates, rising above the B baseline.
Figure 2: GRPO scaling at fixed group size G=8 . Left: the gray band is a ±1 -percentage-point visualization band around the mean trajectory fitted at shared checkpoints spaced every 40K training responses using P=64 – 1024 . These runs are approximately batch-size invariant over the measured window, whereas the held-out P=2048 and P=4096 trajectories deviate from the aligned family. Right: points and error bars show the actor-gradient-norm median and interquartile range over the common 40K–160K training-response window. The P=64 – 1024 medians follow the P−1/2 reference closely, while the decline flattens at the largest batches, most clearly from P=2048 to P=4096 .
Figure 3: Learning-rate retuning improves sample-indexed alignment in GRPO. Left: square-root scaling keeps the P=256 trajectory close to the P=128 reference, whereas using the fixed P=128 learning rate slows progress. Right: normalized optimizer updates required to reach the target; the capped intervals show checkpoint resolution. Exact invariance predicts 0.50× when the batch doubles, which is consistent with the scaled-LR run but not the fixed-LR control.
Figure 4: GRPO prompt-batch and group-size comparisons at matched training-response budgets over the common 40.96K–122.88K response window. The pairs (G=8,P=128) versus (G=16,P=64) and (G=8,P=256) versus (G=16,P=128) hold PG fixed at 1024 and 2048 retained responses per update, respectively. Left: validation trajectories approximately align when indexed by cumulative training responses. Right: actor-gradient-norm summaries are also similar within each matched pair.
Figure 5: Training-side batch scaling for PPO. The cumulative-training-response axis excludes the 30×B responses collected during each run’s 30-step critic-only warm-up and is re-zeroed when actor optimization begins. Left: B=256 – 2048 approximately align, whereas the held-out B=4096 trajectory deviates. Right: normalized post-warm-up actor gradient norm decreases with batch, whereas the critic is comparatively batch-flat and noisier.
Figure 6: Fixed-hardware generation scaling for PPO and GRPO. Blue bars show training responses collected per optimization boundary and orange bars show boundary-collection time, each normalized to the smallest batch. Boxed labels report the corresponding generation-stage response throughput.
Figure 7: Schematic normalized time-to-target for a synchronously coupled batch B . The panels show the two possible orderings of Btraincrit and Bgencrit ; B⋆ marks the wall-clock optimum. The curves are schematic rather than fitted to our experiments.
P
η/10−6
J(K⋆)
K⋆
Ntrain⋆ (K)
T (h)
rN
q/q128
64
1/2
78.80
200
102.40
13.56
0.83
0.73
128
1
77.50
120
122.88
11.90
1.00
1.00
256
2
76.72
60
122.88
9.54
1.00
1.25
512
2
78.59
30
122.88
8.77
1.00
1.36
1024
22
77.19
15
122.88
8.42
1.00
1.41
2048
4
79.58
12
196.61
14.68
1.60
1.30
Table 1: GRPO time-to-target accounting at fixed G=8 and J⋆=77% . K⋆ is the evaluated optimizer update used for accounting, and J(K⋆) is its validation value. Ntrain⋆ is reported in thousands of retained training responses, q=Ntrain⋆/T , and all ratios use P=128 as the reference. The final row is the fixed-LR P=256 control; all other rows use square-root LR scaling.
Figure 8: Measured GRPO time-to-target sweep at fixed G=8 and J⋆=77% , using the accounting in Table 1 . Blue shows the square-root-LR configurations, orange shows the fixed-LR control, and the star marks the minimum measured time-to-target. All times are normalized to the P=128 reference.
Algorithm 1 Two-stage batch tuning
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Analytical per-layer stream decomposition under batch scaling for (a) a training proxy and (b) rollout in a representative fixed-hardware setting. Solid and light dashed bars denote B=1 and B=32 , respectively. Assuming ideal overlap between I/O and compute, latency is modeled as the maximum of the two; parenthesized values report ideal aggregate throughput relative to B=1 .
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
While scaling laws guide compute allocation for LLM pre-training, analogous prescriptions for reinforcement learning (RL) post-training of large language models (LLMs) remain poorly understood. We study the compute-optimal allocation of sampling compute for on-policy RL methods in LLMs, framing scaling as a compute-constrained optimization over three resources: parallel rollouts per problem, number of problems per batch, and number of update steps. We find that the compute-optimal number of parallel rollouts per problem increases predictably with compute budget and then saturates. This trend holds across both easy and hard problems, though driven by different mechanisms: solution sharpening on easy problems and coverage expansion on hard problems. We further show that increasing the number of parallel rollouts mitigates interference across problems, while the number of problems per batch primarily affects training stability and can be chosen within a broad range. Validated across base models and data distributions, our results recast RL scaling laws as prescriptive allocation rules and provide practical guidance for compute-efficient LLM RL post-training.
Zhoujun Cheng, Yutao Xie, Yuxiao Qu +12
UC San Diego · MBZUAI-IFM · Carnegie Mellon University
Reinforcement learning with verifiable rewards (RLVR) relies on costly model-generated responses, motivating larger rollout batches and square-root learning-rate scaling. Reported to approximately preserve progress per sample with one optimizer step per rollout, this rule is less understood in practical multi-step training. We develop a two-clock framework that separates the samples needed to make progress from those consumed before collapse. Under a local quadratic model, the square-root rule preserves first-order drift per sample but not progress: larger batches take fewer, longer updates and lose more to curvature, yielding a learning-rate-dependent critical batch beyond which sample efficiency degrades. A drift-threshold hypothesis predicts a complementary failure clock whose sample count remains approximately invariant along the square-root rule. Controlled GRPO experiments on Qwen2.5-Math-1.5B across a 16× batch range support both predictions. Ratio clipping delays failure, especially for larger batches, but the progress gap persists without it. Widening the clip trades stability for a transient speed gain. The progress gap reappears on another dataset, a general-purpose checkpoint, and a 7B model. Finally, we use the two-clock framework to derive a simple multi-step RLVR schedule: start below the critical batch and switch to a larger batch before failure. Our code is available at: https://github.com/WinstonLiyt/two-clocks-rlvr.
Yuante Li, Junwei Quan, Kai Qiu +2
Carnegie Mellon University · University of Toronto
Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for large language models, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose Mu-GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation-optimization stages. This design induces high rollout staleness while greatly reducing rollout-optimization switching overhead. To stabilize learning under stale data, Mu-GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five language models and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds the performance of standard GRPO while achieving around 2x speedup in wall-clock training time, establishing a substantially improved performance-efficiency trade-off for LLM reinforcement learning.