An Empirical Study of Reinforcement Learning Scaling Laws

Zeyu Zheng; Clare Lyle; Khimya Khetarpal; Will Dabney

cs.LGOct 24, 2026

An Empirical Study of Reinforcement Learning Scaling Laws

Authors: Zeyu Zheng, Clare Lyle, Khimya Khetarpal, Will Dabney

Organizations: Google DeepMind

Abstract

The discovery of scaling laws enables prediction of the performance, the optimal resource allocation strategy, and the best hyperparameter configuration of a large-scale training job based on inexpensive experiments at smaller scales. Guided by these empirical scaling laws, the deep learning community has achieved great success in training large (self-)supervised models on Internet-scale datasets. In contrast, despite its potential and wide application, reinforcement learning (RL) has yet to find a guiding principle for predictable scaling. In this work, we conduct an empirical study of RL scaling laws of an actor-critic algorithm in two procedurally generated environments, Starpilot and Craftax. We discover empirical scaling laws with strong predictive performance in both environments. The discovered scaling laws accurately predict the aforementioned three quantities for larger-scale training jobs up to two orders of magnitude in total training compute. In particular, we employ a variant of Elo scores for the single-agent RL paradigm as an alternative performance metric than the commonly used mean episode return. Our results show that the Elo score is a much more predictable metric than the mean episode return, making it a better choice for fitting scaling laws. By applying our discovered scaling laws to increase training compute, we establish a new state of the art in the challenging Craftax environment. Beneath the surface of the global scaling trend, we also observe a non-monotonic scaling pattern of the compute-optimal model size in Craftax. We connect this observation to a pathology known as Ray interference.

Figures & tables

Environment	$a_{B}$	$d_{B}$	$a_{\alpha}$	$d_{\alpha}$	$a_{M}$	$d_{M}$	$a_{\text{Elo}}$	$d_{\text{Elo}}$
Starpilot	$7.14\text{e}{-4}$	$0.39$	$4.08\text{e}{-3}$	$-0.023$	$5.3$	$0.44$	$-3186$	$240.97$
Craftax	$0.28$	$0.26$	$1.57\text{e}{-2}$	$-0.082$	$0.53$	$0.43$	$-1987$	$223.86$

Table 1: Summary of all fitted coefficients and exponents in the experiments presented in Section 4 .

Panel of Figure 1 — Figure 1: The violin plots show the distributions of the competitive batch sizes and learning rates respectively. The dashed lines visualize the fitted power laws. Red stars mark predicted values.

Panel of Figure 2 — Figure 2: Visualization of performance contour of different hyperparameter configurations at $1\text{e}{17}$ training FLOPs. The number in each cell is the mean episode return on the evaluation set, aggregated by IQM across $5$ seeds. The red stars denote the predicted configurations by the scaling laws.

Environment	predicted FPF	optimal FPF	relative error	Elo of predicted FPF	optimal Elo	win rate error
Starpilot	$4.85\text{e}8$	$3.69\text{e}8$	$+31\%$	$1102.6$	$1110.7$	$-1.17\%$
Craftax	$2.73\text{e}7$	$2.65\text{e}7$	$+3\%$	$2062.8$	$2087.5$	$-3.55\%$

Table 2: Data for evaluating the prediction accuracy of the optimal model size. We compare the predicted FPF and its realized Elo to the empirical optimum.

Panel of Figure 3 — Figure 3: Visualization of data from the isoFLOPs profile experiment in Section 4.2 . The left column visualizes the final performance of each training job and the fitted parabolas. The x-axis denotes the model size measured in FPF. The y-axis denotes the performance measured in Elo. Each parabola corresponds to one training FLOPs. The right column visualizes the derived optimal FPF and the fitted power law. The x-axis is the training FLOPs and the y-axis is the optimal FPF for each training budget. The blue dots are used for fitting the power laws and the yellow dots are used for evaluation.

Environment	predicted @ $1\text{e}{18}$	optimal @ $1\text{e}{18}$	win rate error @ $1\text{e}{18}$	predicted @ $1\text{e}{19}$	actual @ $1\text{e}{19}$	win rate error @ $1\text{e}{19}$
Starpilot	$1151.5$	$1110.7$	$+5.84\%$	$1392.5$	$1312.3$	$+11.34\%$
Craftax	$2043.0$	$2087.5$	$-6.37\%$	$2266.8$	$2185.9$	$+11.44\%$

Table 3: Data for evaluating the accuracy of the Elo predictions. At

1\text{e}{18}

FLOPs, we compare the predicted Elo to the empirically optimal Elo. At

1\text{e}{19}

FLOPs, we compare the predicted Elo to the Elo realized by the predicted optimal configuration.

Panel of Figure 4 — Figure 4: Comparison of different performance metrics. The left column is Elo and the right column is mean episode return. Dots denote empirical optimum. Stars denote the performance of the configuration predicted by the scaling laws. Blue dots are used for fitting the scaling laws. Yellow dots and stars are used for evaluation.

Panel of Figure 5 — Figure 5: Visualization of the inverse scaling pattern in Craftax. The left plot shows the best performance of two different model sizes across different training FLOPs. The right plot shows the success rate of the enemy defeat achievements and the entering dungeon achievement. Error bars show the standard error across $5$ random seeds.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

FLOPs	Model width	Batch size	Learning rate
$1\text{e}{14}$	$\{1,2,3,4\}$	$\{2^{6},2^{7},2^{8},2^{9},2^{10},2^{11},2^{12}\}$	$\{2^{-11},2^{-10.5},2^{-10},2^{-9.5},2^{-9},2^{-8.5},2^{-8}\}$
$3\text{e}{14}$	$\{2,3,4,6\}$	$\{2^{6},2^{7},2^{8},2^{9},2^{10},2^{11},2^{12}\}$	$\{2^{-11},2^{-10.5},2^{-10},2^{-9.5},2^{-9},2^{-8.5},2^{-8}\}$
$1\text{e}{15}$	$\{3,4,6,8\}$	$\{2^{7},2^{8},2^{9},2^{10},2^{11},2^{12},2^{13}\}$	$\{2^{-11},2^{-10.5},2^{-10},2^{-9.5},2^{-9},2^{-8.5},2^{-8}\}$
$3\text{e}{15}$	$\{4,6,8,12\}$	$\{2^{7},2^{8},2^{9},2^{10},2^{11},2^{12},2^{13}\}$	$\{2^{-11},2^{-10.5},2^{-10},2^{-9.5},2^{-9},2^{-8.5},2^{-8}\}$
$1\text{e}{16}$	$\{6,8,12,16\}$	$\{2^{7},2^{8},2^{9},2^{10},2^{11},2^{12},2^{13}\}$	$\{2^{-11},2^{-10.5},2^{-10},2^{-9.5},2^{-9},2^{-8.5},2^{-8}\}$
$3\text{e}{16}$	$\{8,12,16,24\}$	$\{2^{8},2^{9},2^{10},2^{11},2^{12},2^{13},2^{14}\}$	$\{2^{-11},2^{-10.5},2^{-10},2^{-9.5},2^{-9},2^{-8.5},2^{-8}\}$

Appendix

Table 4: Grid search configurations for fitting hyperparameter scaling laws in Starpilot.

FLOPs	Model width	Batch size	Learning rate
$3\text{e}{14}$	$\{3,4,6,8\}$	$\{2^{7},2^{8},2^{9},2^{10},2^{11},2^{12},2^{13}\}$	$\{2^{-12},2^{-11.5},2^{-11},2^{-10.5},2^{-10},2^{-9.5},2^{-9}\}$
$1\text{e}{15}$	$\{4,6,8,12\}$	$\{2^{8},2^{9},2^{10},2^{11},2^{12},2^{13},2^{14}\}$	$\{2^{-12},2^{-11.5},2^{-11},2^{-10.5},2^{-10},2^{-9.5},2^{-9}\}$
$3\text{e}{15}$	$\{6,8,12,16\}$	$\{2^{8},2^{9},2^{10},2^{11},2^{12},2^{13},2^{14}\}$	$\{2^{-12},2^{-11.5},2^{-11},2^{-10.5},2^{-10},2^{-9.5},2^{-9}\}$
$1\text{e}{16}$	$\{6,8,12,16\}$	$\{2^{9},2^{10},2^{11},2^{12},2^{13},2^{14},2^{15}\}$	$\{2^{-12},2^{-11.5},2^{-11},2^{-10.5},2^{-10},2^{-9.5},2^{-9}\}$
$3\text{e}{16}$	$\{8,12,16,24\}$	$\{2^{9},2^{10},2^{11},2^{12},2^{13},2^{14},2^{15}\}$	$\{2^{-12},2^{-11.5},2^{-11},2^{-10.5},2^{-10},2^{-9.5},2^{-9}\}$
$1\text{e}{17}$	$\{12,16,24,32\}$	$\{2^{10},2^{11},2^{12},2^{13},2^{14},2^{15},2^{16}\}$	$\{2^{-12},2^{-11.5},2^{-11},2^{-10.5},2^{-10},2^{-9.5},2^{-9}\}$

Appendix

Table 5: Grid search configurations for fitting hyperparameter scaling laws in Craftax.

FLOPs	Model widths	Batch size	Learning rate
$1\text{e}{15}$	$\{4,5,6,7\}$	$2^{9}$	$2^{-9.12}$
$3\text{e}{15}$	$\{5,6,7,8,9,10\}$	$2^{10}$	$2^{-8.97}$
$1\text{e}{16}$	$\{6,7,8,9,10,11,12,14\}$	$2^{10}$	$2^{-9.35}$
$3\text{e}{16}$	$\{8,10,12,14,16,18,20\}$	$2^{11}$	$2^{-9.19}$
$1\text{e}{17}$	$\{10,12,14,16,18,20,24\}$	$2^{12}$	$2^{-9.07}$
$1\text{e}{18}$	$\{16,20,24,26,28,34,40\}$	$2^{13}$	$2^{-9.30}$

Appendix

Table 6: Configurations for the isoFLOPs profile experiment in Starpilot.

FLOPs	Model widths	Batch size	Learning rate
$1\text{e}{15}$	$\{5,6,7,8,9,10,11,12\}$	$2^{11}$	$2^{-10.17}$
$3\text{e}{15}$	$\{6,7,8,9,10,11,12,14,16\}$	$2^{11}$	$2^{-10.51}$
$1\text{e}{16}$	$\{7,8,9,10,11,12,13,14,16,18,20\}$	$2^{12}$	$2^{-10.38}$
$3\text{e}{16}$	$\{10,12,14,16,18,20,24,28\}$	$2^{12}$	$2^{-10.71}$
$1\text{e}{17}$	$\{16,18,20,22,24,26,28,30,32,34\}$	$2^{13}$	$2^{-10.58}$
$1\text{e}{18}$	$\{24,28,30,32,34,35,36,40,44,48\}$	$2^{14}$	$2^{-10.79}$

Appendix

Table 7: Configurations of the isoFLOPs profile experiment in Craftax.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL
Mar 12, 2026Zhoujun Cheng, Yutao Xie, Yuxiao Qu +12Reinforcement Learning RL Post-Training
UC San Diego · MBZUAI-IFM · Carnegie Mellon University
How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies
Oct 8, 2026Bowei He, Yankai Chen, Xiaokun Zhang +4Agentic RL Synthetic Data Generation for LLM Agents
MBZUAI · McGill University · CityUHK +1
SLDBench v2: Active Scaling Law Discovery
Oct 12, 2026Lingbei Meng, Rui Huang, Zhirong Shen +5AI Agents for Scientific Discovery Language Model Scaling Laws
The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · University of Electronic Science and Technology of China +2

Mar 12, 2026J/K move · Enter open · S save

IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL

Zhoujun Cheng, Yutao Xie, Yuxiao Qu, Amrith Setlur+11

UC San Diego · MBZUAI-IFM · Carnegie Mellon University

Reinforcement Learning RL Post-Training Language Model Scaling Laws

Open paper AskPDF arXiv

While scaling laws guide compute allocation for LLM pre-training, analogous prescriptions for reinforcement learning (RL) post-training of large language models (LLMs) remain poorly understood. We study the compute-optimal allocation of sampling compute for on-policy RL methods in LLMs, framing scaling as a compute-constrained optimization over three resources: parallel rollouts per problem, number of problems per batch, and number of update steps. We find that the compute-optimal number of parallel rollouts per problem increases predictably with compute budget and then saturates. This trend holds across both easy and hard problems, though driven by different mechanisms: solution sharpening on easy problems and coverage expansion on hard problems. We further show that increasing the number of parallel rollouts mitigates interference across problems, while the number of problems per batch primarily affects training stability and can be chosen within a broad range. Validated across base models and data distributions, our results recast RL scaling laws as prescriptive allocation rules and provide practical guidance for compute-efficient LLM RL post-training.

Abstract

Reinforcement Learning

Explore similar work

IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL

How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies

SLDBench v2: Active Scaling Law Discovery