Revisiting scaling laws for reward optimization

Ali Aouad; Aymane El Gadarri; Vivek F. Farias

cs.LGSep 29, 2026

Revisiting scaling laws for reward optimization

Authors: Ali Aouad, Aymane El Gadarri, Vivek F. Farias

Organizations: MIT

Abstract

Scaling laws for optimization against reward models in AI alignment have pinned down how performance depends on optimization effort---measured by a KL-divergence budget relative to a reference policy. Beyond a certain budget, over-optimization (or reward hacking) can arise: because we optimize against a proxy reward model (distinct from true rewards), performance can plateau or degrade. Naturally, the proxy reward's accuracy depends on how much preference data (often in the form of pairwise comparisons) was used to train it. However, existing research does not cleanly identify how performance jointly scales with the amount of training data and the divergence budget. Our main contribution is to provide an empirically accurate and theoretically grounded scaling law in such context. Performance roughly scales as $Θ(\sqrt{\min\{\log(M),K\}})$ , where $M$ is the number of comparisons in training data and $K$ is the policy's divergence budget. We develop an information-theoretic model to establish this upper bound and prove it is tightly achievable through a constructive procedure. Informed by this, we conduct extensive empirical evaluations using a real-world annotation setup, whereby a large 70B gold reward model generates feedback data and proxy reward models are trained from less capable models (0.6B to 4B). Our scaling law provides an excellent fit (R2 from 97% to 99%), outperforms alternative specifications, and remains robust across model sizes, noise, and optimization procedures (best-of- $N$ or policy tilting). Our evidence suggests that reward optimization is analogous to a surprisingly simple selection task: choosing from a sequence of IID Gaussian random variables using noisy preference feedback.

Figures & tables

			$R^{2}$				$p$ -value vs. $\sqrt{\log M}$
Proxy	Noise	Method	$\sqrt{\log M}$	$\log M$	$M$	$M^{1/2}$	$\log M$	$M$	$M^{1/2}$
0.6B	None	BoN	0.992	0.990	0.571	0.803	1.000	$6.1\times 10^{-20}$	$3.4\times 10^{-9}$
0.6B	None	Tilt	0.991	0.988	0.564	0.795	1.000	$2.7\times 10^{-20}$	$3.1\times 10^{-10}$
0.6B	BTL	BoN	0.951	0.954	0.537	0.775	0.392	$1.8\times 10^{-20}$	$7.3\times 10^{-11}$
0.6B	BTL	Tilt	0.947	0.949	0.527	0.767	0.708	$1.3\times 10^{-20}$	$4.7\times 10^{-11}$
1.7B	None	BoN	0.989	0.981	0.520	0.757	0.023	$8.7\times 10^{-25}$	$2.2\times 10^{-14}$

Table 1: Scaling fits across model size, preference noise, and optimization method. Each

R^{2}

fits

a+bf(M)

on all evaluation prompts, bold marks the largest

R^{2}

in each row. The final three columns report Holm-adjusted, two-sided paired

t

-test

p

-values against

\sqrt{\log M}

Figure 1: Gold reward follows square-root-logarithmic comparison-budget scaling. Ten 0.6B models are selected by uniform spacing in $\sqrt{\log M}$ . Each point is the maximum mean gold-reward gain over the evaluated best-of- $N$ grid. Both panels show the same fitted function $a+b\sqrt{\log M}$ : left on a square-root-logarithmic axis, right on a linear $M$ axis. Point labels denote comparison counts.

Figure 2: Comparison of alternative scaling functions. All curves are fitted to the same ten models as Figure 1 . The fits $a+bf(M)$ are shown on a common linear comparison-count axis.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Topic	Training ( $n=1,000$ )	Evaluation ( $n=128$ )
Coding	325 (32.5%)	43 (33.6%)
Mathematics	81 (8.1%)	14 (10.9%)
Creative writing	68 (6.8%)	8 (6.3%)
Topic unspecified	558 (55.8%)	67 (52.3%)

Appendix

Table 2: Training and evaluation prompt topics. Counts and percentages within each set, using original Arena annotations. Tags may overlap. The dataset has no finer topic annotations for prompts marked Topic unspecified .

Figure 3: Gold-reward distributions for four test prompts. Each histogram contains all 8,192 raw gold scores for one prompt. The overlaid Gaussian has the same empirical mean and variance. Panels use the same prompt order as Figure 4 ; prompts are selected by topic and question ID without inspecting rewards.

Figure 4: Normal Q–Q plots for the same four prompts. Ordered gold scores, standardized within each prompt, are plotted against standard normal quantiles at probabilities $(i-1/2)/8{,}192$ . The diagonal is the Gaussian reference. Systematic departures, particularly in the tails, illustrate that empirical gold rewards need not follow a Gaussian distribution.

Hyperparameter	Value
Trainable parameters	Full backbone and scalar reward head
Objective	Pairwise logistic loss
Optimizer	AdamW
Learning rate / schedule	$10^{-5}$ / constant, no warmup
Adam $(\beta_{1},\beta_{2})$ / $\epsilon$	$(0.9,0.999)$ / $10^{-8}$
Weight decay	$10^{-3}$ on non-normalization matrix weights

Appendix

Table 3: Training hyperparameters.

Figure 5: Square-root-logarithmic scaling across all settings in Table 1 . Columns show best-of- $N$ (BoN) and exponential tilting (Tilt); rows vary proxy size and preference noise. Each point is the peak mean gold-reward gain over the evaluated optimization grid. Lines fit $a+b\sqrt{\log M}$ to ten models between $M=600$ and $300{,}000$ .

Figure 6: Best-of- $N$ optimization with noiseless and noisy preferences. Each panel shows four 0.6B checkpoints trained on $M=600$ , $4{,}800$ , $38{,}400$ , and $300{,}000$ comparisons. The $N$ -axis is logarithmic. Solid and dashed lines show mean gold- and proxy-reward gains over the same evaluation prompts, using the unbiased subset-average estimator on 8,192 responses per prompt. Gains are centered at $N=1$ . Gold rewards use the fixed reference standard deviation, and each proxy uses its own standard deviation over the full evaluation pool.

Explore similar work

May 8, 2026cs.LG

Theoretical Limits of Language Model Alignment

Language model (LM) alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are (i) reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and (ii) best-of-

N

alignment, which selects the highest-reward output among

N

independent samples. Despite their widespread use, the fundamental limits of reward improvement under a KL budget remain poorly understood. We characterize the information-theoretic limits of KL-regularized alignment by deriving the maximum achievable expected reward gain for a fixed KL-divergence budget. Our first result provides a closed-form expression for the optimal reward improvement, governed by a Jeffreys divergence term rather than the

\sqrt{\texttt{KL}}

used in prior analyses. We further reformulate this expression as a covariance under the base model, yielding a practical estimator that predicts achievable alignment gains from base model samples alone. We extend our analysis to the proxy reward setting, showing that the gap between ideal and proxy alignment (reward hacking) grows with the magnitude of reward error and when the KL penalty factor decreases. We then prove that reward ensembling mitigates reward hacking, providing a theoretical justification for this technique used in practice. Empirically, we compute the KL-reward Pareto frontier for two tasks for LMs, safety and summarization, and show that best-of-

N

closely approaches the theoretical limit, while PPO and GRPO remain substantially suboptimal. Our theoretical results shed light on several empirically observed phenomena in the alignment literature and suggest that algorithmic improvements are needed to achieve optimal alignment without high inference costs.

Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald +1

Apple

Large Language Model Alignment Fundamental Limits Divergence Pareto Frontier

Ask PDF

Sep 22, 2026cs.AI

Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts

A higher evaluation score does not always mean a better language model system. When optimization exploits an evaluator's mistakes, measured progress can conceal unchanged or deteriorating task performance. This failure can arise through parameter updates, selection among generated outputs, or revisions to persistent prompts. We develop a comparative framework for reward hacking across these three optimization substrates: weights, selection, and text. Building on the Proxy Compression Hypothesis and research on inference-time and in-context reward hacking, we examine how reachable behavior, optimization budgets, and persistent adaptation shape exposure to proxy error. We formalize a distance-dependent upper bound on evaluator disagreement and a capacity ordering for nested policy classes, then show why distance alone cannot establish a universal ranking of vulnerability. An exact finite-output illustration demonstrates how the location of a scoring defect changes the behavior favored by each method. We also map representative defenses across substrates, identifying which mechanisms transfer directly and which offer only functional analogies. Persistent prompts receive particular attention: their contents are inspectable, but the behavior induced by a small textual change may be difficult to anticipate. The formal analysis, numerical illustration, and published evidence together provide a basis for comparing optimization methods and identifying the conditions under which their defenses transfer. The resulting framework connects optimization choices to verification requirements: reliable improvement depends on controlling accessible failure modes and preserving evidence of task quality independent of the score being optimized.

Vansh Wahi

Destination-Only Scoring Scores Prompt Engineering Disagreement Data Compression Methods

Ask PDF

May 2, 2026cs.LG

Prescriptive Scaling Laws for Data Constrained Training

Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law assumes every training token is unique. This limits its ability to guide pretraining decisions in data-constrained regimes. We model the excess loss under repetition with a simple additive overfitting penalty and find that it accurately describes model behavior. Our scaling law yields qualitatively new compute-optimal allocation advice. Beyond a point, further repetition is counterproductive and compute is better spent on model capacity. We show that following our law's recommended configuration improves performance in data-constrained regimes. Finally, because our one-parameter form isolates overfitting in a single coefficient, it enables direct comparison across training configurations. As a case study, we show that strong weight decay (

λ=1.0

) reduces this coefficient by approximately 70%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.

Justin Lovelace, Christian Belardi, Srivatsa Kundurthy +2

Department of Computer Science Cornell University Ithaca, NY

Scaling Laws Weight Decay Capacity

Ask PDF