How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies

Bowei He; Yankai Chen; Xiaokun Zhang; Changjiang Han; Ye Yuan; Chong Li; Xue Liu

cs.LGOct 8, 2026

How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies

Authors: Bowei He, Yankai Chen, Xiaokun Zhang, Changjiang Han, Ye Yuan, Chong Li, Xue Liu

Organizations: MBZUAI · McGill University · CityUHK · Columbia University

Abstract

Pretraining scaling laws treat training data as a one-dimensional quantity: a token count. Agentic reinforcement learning (RL) inherits this scalar abstraction, but modern synthesis pipelines now generate tasks, environments, and trajectories independently and at very different costs, turning data into a structured design space. This raises a question pretraining never had to answer: given a synthesis budget, which axis should be scaled? We address this question end-to-end with three tightly coupled contributions. First, we decompose agentic data into three cardinality axes (task, environment, trajectory), plus a reward-density control variable, and fit per-axis scaling relations on different large language models (LLMs) across math+code, SWE, web, and UI/tool application domains. Across all model sizes, model families, domains, and RL algorithms (PPO and GRPO) we measured, the scaling exponents order as environment $>$ task $>$ trajectory. When environment diversity is decomposed into world and environment-structure diversity, its advantage is mainly driven by the former, while the latter is just comparable to the task diversity. Second, we use the fitted laws as a common ruler to price synthesis methods, capturing each method's efficiency as a per-axis discount factor and identifying critical synthetic-to-real ratios at which collapse begins; these ratios differ by an order of magnitude across axes. Third, we cast budget allocation as a constrained optimization under our laws and discounts, deriving a Chinchilla-style compute-optimal recipe and validating it at held-out scale. The empirical results demonstrate that following the recipe yields up to $1.7\times$ compute efficiency over the dominant ``scale-the-trajectories'' practice.

Figures & tables

Figure 1: Same compute, very different outcomes. Two GRPO runs on Qwen3-8B with identical total rollout tokens and gradient FLOPs, differing only in how the data budget is split across the task, environment, and trajectory axes and the reward control. Left: task success rate on AppWorld, ALFWorld, WebShop, and a held-out math suite. Right: the corresponding allocation pies.

Level	What varies	Example	Exponent
L0: content	seed or initial state within the same repository and harness	a new AIME problem; a new issue on the same repository	$0.07\pm 0.02$ (placebo)
L1: structure	tool/API surface, state schema, transition semantics, verifier	stateless vs. persistent REPL; pytest vs. tox output grammar	$\alpha_{\mathrm{V}}=0.28\pm 0.03$ ( $0.41$ on math+code)
L2: world ( $D_{\mathrm{W}}$ )	repository, corpus, app, or site with its own state	a repository or app unseen in training	$\alpha_{\mathrm{W}}=0.44\pm 0.04$

Table 1: Three levels of environment variation (Section 4.4 ). Only L1 and L2 create new instances.

Figure 2: Per-axis scaling laws under controlled sweeps. Each panel sweeps one axis with the other two held at saturation; symbols are mean $\pm$ SE over 3 seeds, lines are weighted log-log fits. Across all three model sizes, environment scaling is the steepest, trajectory scaling the flattest. The exponent ordering $\alpha_{\mathrm{E}}>\alpha_{\mathrm{T}}>\alpha_{\tau}$ holds with non-overlapping 95% CIs at every model size.

Figure 3: Cross-axis interaction. Heatmap of held-out success on Qwen3-8B (math+code) over a $(D_{\mathrm{E}},D_{\mathrm{T}})$ grid at fixed $D_{\tau}=8$ . Iso-cost contours (dashed) reveal a Pareto ridge: at low $D_{\mathrm{E}}$ , scaling $D_{\mathrm{T}}$ has near-zero return. Environment diversity is a prerequisite for task diversity to matter.

Model	Algo.	Domain	$\alpha_{\mathrm{E}}$	$\alpha_{\mathrm{T}}$	$\alpha_{\tau}$
Qwen3-4B	GRPO	Math+Code †	$0.38\pm 0.04$	$0.23\pm 0.03$	$0.10\pm 0.02$
	GRPO	SWE	$0.36\pm 0.05$	$0.21\pm 0.04$	$0.09\pm 0.03$
	GRPO	Web	$0.43\pm 0.04$	$0.26\pm 0.03$	$0.12\pm 0.02$
	PPO	Math+Code †	$0.35\pm 0.04$	$0.21\pm 0.03$	$0.10\pm 0.02$
Qwen3-8B	GRPO	Math+Code †	$0.41\pm 0.03$	$0.26\pm 0.02$	$0.11\pm 0.02$
	GRPO	SWE	$0.39\pm 0.03$	$0.24\pm 0.02$	$0.10\pm 0.02$

Table 2: Fitted exponents across model sizes, algorithms, and domains. Numbers are mean

\pm

bootstrap 95% CI half-width over 3 seeds. Note the ordering

\alpha_{\mathrm{E}}>\alpha_{\mathrm{T}}>\alpha_{\tau}

holds in all 18 settings. † No separable world axis:

\alpha_{\mathrm{E}}

is a structure exponent (Section 4.4 ).

Method	Primary axis	$\kappa$ @ $r{=}1$	$r^{\star}$	Cost/unit
AWM [ 50 ]	$D_{\mathrm{E}}$	$0.71\pm 0.05$	$9.4\pm 0.9$	4.10 USD/env
GenEnv [ 14 ]	$D_{\mathrm{E}}$	$0.66\pm 0.06$	$8.1\pm 1.0$	3.80 USD/env
DreamGym [ 4 ]	$D_{\mathrm{E}}$ $+$ $D_{\tau}$	$0.59\pm 0.05$	$6.2\pm 0.8$	0.04 USD/traj
AutoPlay [ 36 ]	$D_{\mathrm{T}}$	$0.54\pm 0.04$	$3.1\pm 0.4$	0.18 USD/task
SCA [ 59 ]	$D_{\mathrm{T}}$	$0.51\pm 0.05$	$2.8\pm 0.4$	0.21 USD/task
RLVE [ 56 ]	$D_{\mathrm{T}}$	$0.57\pm 0.04$	$3.4\pm 0.4$	0.15 USD/task

Table 3: Synthesis taxonomy along the data axes. Rows: representative synthesis methods, all run through one common harness (Appendix I ). Columns: which axis each primarily scales, the effective discount

\kappa

measured at

r=D_{s}/D_{r}=1

(equal real and synthetic), the axis-specific collapse ratio

r^{\star}

, and rough cost per “unit” (1k tasks, 1 environment, or 1 trajectory) on Qwen3-8B. The estimator of Section 5.2 gives

r^{\star}=9.4

3.4

1.4

for AWM, AutoPlay, SWiRL.

Figure 4: Synthesis collapse, per axis. $x$ -axis: synthetic-to-real ratio $r=D_{s}/D_{r}$ (log scale). $y$ -axis: agent success rate on a held-out evaluation, normalized to the $r=0$ (real-only) value of the same seed group. Vertical dashed lines mark fitted $r^{\star}$ , the ratio at which the curve drops 5% below the real-only baseline. Environment synthesis remains net-positive up to $r{\approx}9$ , while trajectory synthesis collapses by $r{\approx}1.4$ .

Figure 5: Compute-optimal allocation derived from the decomposed law. (a) The Lagrangian-optimal values of the four design variables $(N,D_{\mathrm{E}},D_{\mathrm{T}},D_{\tau})$ scale as power laws in total compute $C$ (Eq. 8 ). Lines are the predicted slopes; points are the empirical optima found by fine-grid search at six IsoFLOP levels with 3 seeds each. (b) Held-out task success rate vs. compute under three allocation strategies, all sharing the same budget $C$ at every point: our recipe (Eq. 8 , solid blue), a Chinchilla-transferred baseline that treats $D{=}D_{\mathrm{E}}{+}D_{\mathrm{T}}{+}D_{\tau}$ as a scalar (dashed orange), and the dominant “naive” practice of pouring budget into trajectory rollouts (dotted grey). Smooth curves are predictions of Eq. 1 under each recipe’s allocation; markers with error bars are 3-seed measurements at the IsoFLOP grid points; the black star is the held-out validation run on Qwen3-14B at $C{=}32C_{0}$ .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Axis	Values	Held-out success (%)	Selected
$D_{\mathrm{E}}$	64 / 128 / 256 / 512	28.1 / 30.4 / 31.2 / 31.5	256
$D_{\mathrm{T}}$	2.6k / 5.1k / 10.2k / 20.5k	27.9 / 29.8 / 30.9 / 31.2	10,240
$D_{\tau}$	2 / 4 / 8 / 16 / 32	25.4 / 28.6 / 30.6 / 31.0 / 31.2	8

Appendix

Table 4: Calibration sweep that fixed the held-axis values (Qwen3-8B, math+code). It was run before any exponent was fitted, under the pre-specified criterion of Appendix A .

Held $(D_{\mathrm{E}},D_{\mathrm{T}},D_{\tau})$	$\alpha_{\mathrm{E}}$	$\alpha_{\mathrm{T}}$	$\alpha_{\tau}$
Default $(256,10{,}240,8)$	$0.41$	$0.26$	$0.11$
Half $(128,5{,}120,4)$	$0.39$	$0.25$	$0.11$
Double $(512,20{,}480,16)$	$0.42$	$0.27$	$0.12$
Quarter $(64,2{,}560,2)$	$0.36$	$0.22$	$0.10$

Appendix

Table 5: Fitted exponents (Qwen3-8B, math+code) when the held axes are set to a quarter, a half, or double the calibrated values.

Domain	$D_{\mathrm{E}}$	Jacc. tools	Jacc. schema	Trans. div.	S3	S2	S1	Raw $\to$ distinct
Math+code	256	0.61	0.54	0.33	94 (37%)	88 (34%)	74 (29%)	2,048 $\to$ 256
SWE	128	0.47	0.70	0.45	43 (34%)	52 (41%)	33 (26%)	1,024 $\to$ 128
Web	96	0.73	0.82	0.57	41 (43%)	34 (35%)	21 (22%)	768 $\to$ 96
UI/tool	224	0.80	0.85	0.60	108 (48%)	71 (32%)	45 (20%)	1,792 $\to$ 224

Appendix

Table 6: Environment pools at the largest grid point of each domain. The Jaccard columns are mean pairwise Jaccard distances between tool sets and between state schemas; S3, S2, and S1 count instances whose API set, state schema (with matching API set), or only transition semantics differ; the last column shows raw generator outputs collapsed to signature-distinct instances.

Pool construction	$\alpha_{\mathrm{E}}$
Default pool (Table 2 )	$0.41\pm 0.03$
Strict signature deduplication, levels S1 to S3	$0.39\pm 0.03$
Levels S2 and S3 only (API set or schema differs)	$0.40\pm 0.04$
Level S3 only (API set differs)	$0.42\pm 0.04$
Placebo: signature fixed, procedural seeds scaled from 8 to 256	$0.07\pm 0.02$
Reference: trajectory exponent $\alpha_{\tau}$	$0.11\pm 0.02$

Appendix

Table 7: Environment exponent under alternative pool constructions (Qwen3-8B, math+code).

Domain	$(1,64)$	$(2,32)$	$(4,16)$	$(8,8)$	$\Delta L$ per doubling of $W$
SWE	$18.0\pm 0.4$	$22.4\pm 0.3$	$25.2\pm 0.4$	$26.1\pm 0.6$	0.219, 0.118, 0.035
UI/tool	$28.1\pm 0.2$	$32.8\pm 0.4$	$35.6\pm 0.3$	$36.5\pm 0.4$	0.158, 0.082, 0.025
Web	$24.0\pm 0.5$	$28.5\pm 0.4$	$31.3\pm 0.5$	n/a	0.172, 0.094

Appendix

Table 8: Held-out-world success (%) as budget moves from structural variants per world (

V

) to worlds (

W

) at fixed

D_{\mathrm{E}}=W\times V

and fixed compute (3 seeds, bootstrap 95% CI half-widths). Column headers give

(W,V)

. Web has five sites, so its grid stops at

W=4

(n/a). The last column gives the reduction

\Delta L

L=-\log p_{\mathrm{success}}

at each successive doubling of

W

$D_{\mathrm{E}}$	Cells	Tasks	Trajectories	Rollout tok.	Gradient tok.	Updates	FLOPs
8	256,000	10,240	2.048M	5.18B	3.19B	1,000	0.98
32	256,000	10,240	2.048M	5.24B	3.23B	1,000	0.99
128	256,000	10,240	2.048M	5.33B	3.28B	1,000	1.01
256	256,000	10,240	2.048M	5.38B	3.31B	1,000	1.02

Appendix

Table 9: Training-signal accounting along the

D_{\mathrm{E}}

sweep (Qwen3-8B, math+code) at four grid points spanning its full range. Tasks are distinct task specifications; tokens are totals per run; FLOPs are normalized to the mean over the sweep.

Convention	$\alpha_{\mathrm{E}}$	$\alpha_{\mathrm{T}}$	$\alpha_{\tau}$
Fixed cells and updates (default)	$0.41\pm 0.03$	$0.26\pm 0.02$	$0.11\pm 0.02$
Exact rollout-token cap (5.2B tokens per run)	$0.40\pm 0.03$	$0.26\pm 0.02$	$0.08\pm 0.02$
Iso-FLOP (1.00 normalized FLOPs per run)	$0.40\pm 0.04$	$0.25\pm 0.03$	$0.07\pm 0.02$

Appendix

Table 10: Fitted exponents (Qwen3-8B, math+code) under three training-budget conventions.

Axis	Range	$\alpha$ (6 points)	$\alpha$ (8 points)	$R^{2}$	Max. leave-one-out shift
$D_{\mathrm{E}}$	4 to 384 ( $96\times$ )	$0.41\pm 0.03$	$0.40\pm 0.02$	0.981	0.015
$D_{\mathrm{T}}$	160 to 20,480 ( $128\times$ )	$0.26\pm 0.02$	$0.26\pm 0.02$	0.984	0.011
$D_{\tau}$	1 to 48 ( $48\times$ )	$0.11\pm 0.02$	$0.12\pm 0.01$	0.986	0.009

Appendix

Table 11: Six-point versus eight-point fits (Qwen3-8B, math+code). The last column is the largest change in

\alpha

when any single grid point is left out.

Segment	1	2	3	4	5	6	7
$D_{\mathrm{E}}$	0.44	0.42	0.41	0.40	0.38	0.37	0.36
$D_{\mathrm{T}}$	0.29	0.28	0.26	0.25	0.24	0.23	0.22
$D_{\tau}$	0.14	0.13	0.12	0.11	0.10	0.09	0.08

Appendix

Table 12: Fit-free local log-log slopes

s_{j}

between adjacent grid points of the 8-point grids (Qwen3-8B, math+code).

Setting	$\alpha_{\mathrm{E}}$	$\alpha_{\mathrm{T}}$	$\alpha_{\tau}$
Qwen3-8B (reference, Table 2 )	$0.41\pm 0.03$	$0.26\pm 0.02$	$0.11\pm 0.02$
Llama-3.1-8B (new family)	$0.36\pm 0.05$	$0.23\pm 0.04$	$0.09\pm 0.03$
Qwen2.5-7B (earlier generation)	$0.38\pm 0.05$	$0.24\pm 0.04$	$0.10\pm 0.03$
Search agent (new domain, Qwen3-8B)	$0.42\pm 0.05$	$0.25\pm 0.04$	$0.12\pm 0.03$

Appendix

Table 13: Fitted exponents on further model families and a new domain (reduced 4-point sweeps).

$\theta$	Effective $D_{\mathrm{T}}$ at $D_{raw}\!=\!10{,}240$	$\alpha_{\mathrm{T}}$	$R^{2}$
0.05	9,820	$0.245\pm 0.02$	$0.974$
0.10	9,310	$0.254\pm 0.02$	$0.978$
0.15 (default)	8,460	$\mathbf{0.261\pm 0.02}$	$\mathbf{0.978}$
0.20	7,210	$0.268\pm 0.02$	$0.972$
0.30	5,380	$0.272\pm 0.03$	$0.968$

Appendix

Table 14: Ablation of the task-deduplication threshold

\theta

. Effective

D_{\mathrm{T}}

is the count of unique tasks remaining after deduplication at each threshold; fitted

\alpha_{\mathrm{T}}

is robust across the range.

Rule for a distinct task	Effective $D_{\mathrm{T}}$	$\alpha_{\mathrm{T}}$
Raw count, no deduplication	10,240	$0.238\pm 0.02$
Embedding distance $>0.15$ , gte-Qwen2-1.5B (default)	8,460	$0.261\pm 0.02$
Embedding distance $>0.15$ , e5-mistral-7b	8,310	$0.257\pm 0.02$
Skill signature (reference tool-call skeleton)	6,120	$0.243\pm 0.03$
Verifier signature (verification-program structure)	6,980	$0.251\pm 0.02$
Human-validated pool	6,540	$0.248\pm 0.03$

Appendix

Table 15: Fitted task exponent under alternative definitions of a distinct task (Qwen3-8B, math+code). Effective

D_{\mathrm{T}}

is the number of distinct tasks at

D_{raw}=10{,}240

Method	Axis	$\kappa$ (Table 3 )	$\kappa$ (common core)	$r^{\star}$ (Table 3 )	$r^{\star}$ (common core)
AWM	$D_{\mathrm{E}}$	0.71	$0.68\pm 0.05$	9.4	$8.7\pm 1.0$
GenEnv	$D_{\mathrm{E}}$	0.66	$0.65\pm 0.06$	8.1	$7.9\pm 1.1$
AutoPlay	$D_{\mathrm{T}}$	0.54	$0.55\pm 0.04$	3.1	$3.2\pm 0.5$
SCA	$D_{\mathrm{T}}$	0.51	$0.52\pm 0.05$	2.8	$2.9\pm 0.5$
RLVE	$D_{\mathrm{T}}$	0.57	$0.56\pm 0.04$	3.4	$3.3\pm 0.5$
SWiRL	$D_{\tau}$	0.34	$0.36\pm 0.06$	1.4	$1.5\pm 0.3$

Appendix

Table 16: Discount and collapse ratio of each single-axis method as reported in Table 3 and after reduction to a common generate, filter, and train core.

Criterion	$r^{\star}_{\mathrm{E}}$ (AWM)	$r^{\star}_{\mathrm{T}}$ (AutoPlay)	$r^{\star}_{\tau}$ (SWiRL)
Crossing of 0.95 by the fitted logistic (default)	9.4	3.4	1.4
Crossing of the real-only baseline (1.00)	8.5	4.0	1.1
Peak of the curve	2.4	1.1	0.4

Appendix

Table 17: Collapse ratio of the three representative synthesis methods (Qwen3-8B, math+code) under alternative definitions of collapse.

Domain	$H$	Rollout tokens/traj	Env latency/call	Verifier/traj	$f$	$\eta$
Math+code	3.4	2.6k	0.9 s	0.4 s	0.07	0.61
SWE	18.2	11.4k	6.3 s	41 s	0.19	0.38
Web	12.7	7.9k	2.4 s	1.8 s	0.14	0.44
UI/tool	9.6	5.2k	1.1 s	0.9 s	0.11	0.52

Appendix

Table 18: Measured cost drivers per domain: average horizon

H

(steps), rollout tokens per trajectory, environment latency per call, verifier time per trajectory, failed-rollout fraction

f

, and accelerator utilization

\eta

Cost model	$D_{\mathrm{E}}$ exp.	$D_{\mathrm{T}}$ exp.	$D_{\tau}$ exp.	$N$ exp.
Linear-additive (Eq. 5 )	$0.34$	$0.27$	$0.13$	$0.26$
$+$ bilinear rollout term $c\cdot N\cdot D_{\tau}$	$0.33$	$0.27$	$0.14$	$0.26$
$+$ tool and environment latency, verifier cost	$0.31$	$0.28$	$0.15$	$0.26$
$+$ failed-rollout overhead $1/(1-f)$	$0.30$	$0.28$	$0.16$	$0.26$
$+$ batching and utilization $\eta$	$0.30$	$0.28$	$0.16$	$0.26$
Full model, SWE coefficients (worst case for $D_{\mathrm{E}}$ )	$0.28$	$0.29$	$0.18$	$0.25$

Appendix

Table 19: Recipe exponents under progressively richer cost models (math+code coefficients unless stated otherwise).

Domain	$D_{\mathrm{E}}$ exp.	$D_{\mathrm{T}}$ exp.	$D_{\tau}$ exp.	$N$ exp.	$\sum$
Math+Code (default)	$0.34$	$0.27$	$0.13$	$0.26$	$1.00$
SWE	$0.31$	$0.28$	$0.15$	$0.27$	$1.01$
Web	$0.36$	$0.27$	$0.12$	$0.25$	$1.00$
UI/Tool	$0.37$	$0.26$	$0.11$	$0.27$	$1.01$
Range across domains	$\pm 0.03$	$\pm 0.01$	$\pm 0.02$	$\pm 0.02$	n/a

Appendix

Table 20: Recipe exponents per domain. The qualitative ordering

D_{\mathrm{E}}^{\star}>D_{\mathrm{T}}^{\star}>D_{\tau}^{\star}

is preserved in every domain, and absolute exponents shift by less than

\pm 0.04

from the math+code reference.

Perturbation	$D_{\mathrm{E}}$ exp.	$D_{\mathrm{T}}$ exp.	$D_{\tau}$ exp.	$N$ exp.
Default	$0.340$	$0.270$	$0.130$	$0.260$
$c_{E}\times 1.5$	$0.305$	$0.291$	$0.142$	$0.262$
$c_{E}\times 0.5$	$0.388$	$0.244$	$0.115$	$0.253$
$c_{T}\times 1.5$	$0.348$	$0.243$	$0.144$	$0.265$
$c_{T}\times 0.5$	$0.328$	$0.305$	$0.114$	$0.253$
$c_{\tau}\times 1.5$	$0.346$	$0.275$	$0.111$	$0.268$

Appendix

Table 21: Recipe-exponent sensitivity to

\pm 50\%

perturbations of cost coefficients (math+code, 8B). All shifts are

<0.05

on every axis.

Agentic RL

Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.

See all 228 papers

Oct 27, 2026

Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis

We explore how to build performant large language model (LLM) tool-use agents that complete complex, multi-step tasks. Specifically, we study how data composition in synthetically generated tool-use environments affects the training of such agents. To do so, we propose the SynthAPI pipeline, which explicitly controls the diversity of generated environments along the axes of tools, tasks, and the databases on which the tools operate. By systematically varying SynthAPI's generation controls in 458,752 tasks across 849 trained models of four different sizes, we derive a scaling law that predicts downstream tool-use agent generalization based on the scale and composition of the synthesized environments. These predictable relationships reveal several lessons for scaling tool-use agent data. As data budgets grow, scaling tool and database diversity becomes more effective than increasing the number of tasks, and increasing data complexity yields larger gains. Scaling according to these findings yields agents that outperform models trained on data from prior work in large-scale tool-use agent and synthetic environment generation across four evaluation benchmarks.

Andrew Szot, Samira Abnar, Omar Attia +5

Apple

Tool-Using Agents LLM Agent Training Synthetic Data Generation for LLM Agents Scaling Laws

Ask PDF

Oct 12, 2026

SLDBench v2: Active Scaling Law Discovery

Scaling laws predict large-model performance from small experiments, but finding a reliable law itself requires costly experiments. SLDBench tests whether language-model agents can discover scaling laws when all experimental results are given upfront. We introduce SLDBench v2 with two extensions. First, we expand the corpus from 8 to 127 scenarios spanning language-model pretraining, downstream performance, and vision-language transfer, and select 30 challenging scenarios on which a reference power-law fit reaches

R^2 \le 0.8

; they represent at least

1.09 \times 10^{23}

FLOPs of source-reported training compute. Second, we introduce active scaling law discovery: an agent chooses which experiments to run, pays each one's recorded cost (e.g., training FLOPs), and revises its law as outcomes arrive, and is scored on the same hidden extrapolation targets as in the passive setting. Across 8 frontier agent systems, median

R^2

drops from 0.72-0.75 (passive) to 0.24-0.57 (active), and prediction error rises 1.5-4.7 times, depending on the system. Agents favor cheap experiments, spending 8-44% of the full-pool cost on average; low-cost acquisition can yield strong predictions, but cheap purchases also leave experimental conditions needed for extrapolation unobserved. Experimental planning thus remains a key open challenge for agents that assist model development in the prevailing AI-for-AI study. Project page: https://lingbei-meng.github.io/SLDBench-v2/.

Lingbei Meng, Rui Huang, Zhirong Shen +5

The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · University of Electronic Science and Technology of China +2

AI Agents for Scientific Discovery Language Model Scaling Laws AI Agent Benchmarks

Ask PDF

Mar 12, 2026

IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL

While scaling laws guide compute allocation for LLM pre-training, analogous prescriptions for reinforcement learning (RL) post-training of large language models (LLMs) remain poorly understood. We study the compute-optimal allocation of sampling compute for on-policy RL methods in LLMs, framing scaling as a compute-constrained optimization over three resources: parallel rollouts per problem, number of problems per batch, and number of update steps. We find that the compute-optimal number of parallel rollouts per problem increases predictably with compute budget and then saturates. This trend holds across both easy and hard problems, though driven by different mechanisms: solution sharpening on easy problems and coverage expansion on hard problems. We further show that increasing the number of parallel rollouts mitigates interference across problems, while the number of problems per batch primarily affects training stability and can be chosen within a broad range. Validated across base models and data distributions, our results recast RL scaling laws as prescriptive allocation rules and provide practical guidance for compute-efficient LLM RL post-training.

Zhoujun Cheng, Yutao Xie, Yuxiao Qu +12

UC San Diego · MBZUAI-IFM · Carnegie Mellon University

Reinforcement Learning RL Post-Training Language Model Scaling Laws

Ask PDF

Abstract

Agentic RL

Explore similar work

Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis

SLDBench v2: Active Scaling Law Discovery

IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL