Responses with headers, lists and greater length receive high scores from reward models (RMs), and RLHF policies learn to produce these features even when true quality does not improve. Practitioners therefore edit their RMs, but they learn whether an edit works only after an RL run. We show that the response of a policy to an RM edit can be forecast before training, from samples of the current policy and the scores of the old and new RM. If we read the RM as a set of prices, the first-order theory of KL-regularized RL implies that each feature moves in proportion to its within-prompt covariance with the change in reward. We implant controlled price changes into retrained RMs under a 2^(4-1) fractional factorial design with a self-distilled placebo, run 54 RL runs from six parent policies, and fix every forecast before the runs. With the covariance predictor, we predict the direction of all 20 changes in supply whose forecast has the same sign at every parent and explain 82% of the standardized change. With counterfactual prices, the classical alternative, we explain 50%. The forecast also holds under GRPO and for real RM upgrades, where we explain 66% out of sample with a coefficient frozen from the factorial runs. For rent, the gap between the proxy reward and a gold reward from stronger judges, we rank RM edits correctly with the same covariance over the first 30 steps (Spearman 0.94). This measures the horizon of the forecast: by step 200, the ranking of the factorial edits reverses under both RLOO and GRPO, while the real upgrades keep their order. When we vary the KL coefficient beta, the initial response does not depend on beta, and the response saturates below the KL equilibrium. In one experiment, the long-run response to a price rise is consistent with an additional implicit regularizer beta_0 of about 0.003 to 0.007.
Figures & tables
Figure 1: Predicted against observed change in supply after 200 steps, relative to the placebo RM, for 6 factorial arms × 6 parents and the three primary features. Units are within-prompt standard deviations at the branch point, and each predictor has one fitted coefficient per feature. Left: covariance predictor CovB(f,Δr) . Right: price predictor ΣBΔp . We computed both predictors at the branch point before any RL step.
SFT samples
Parent responses
Arm
F1a
F1b
F2
F4
F1a
F1b
F2
F4
0000
− 0.69
− 0.73
− 0.45
0.01
− 0.67
− 0.28
− 0.28
− 0.02
1001
0.83
− 0.94
− 0.48
0.37
0.74
− 0.20
− 0.14
− 0.01
0101
− 0.72
0.75
− 0.42
0.39
− 0.87
0.44
− 0.46
0.18
1100
0.65
0.84
− 0.41
0.08
0.65
0.34
− 0.22
0.02
0011
− 0.79
− 0.99
0.74
0.38
− 0.45
− 0.38
0.35
0.39
Table 1: Implanted price changes relative to the placebo RM, per pair, on the common standardized scale. Parent columns average six branch points. The threshold is three times the off-target noise and applies to the primary goods F1a and F1b.
Steps 20–30
Steps 160–200
Predictor
Sign
Share
Sign
Share
Covariance
18/20
0.81
20/20
0.82
Price
17/20
0.58
20/20
0.50
Robustness
Covariance, common κ
0.69
0.82
Price, common κ
0.51
0.45
Table 2: Pre-specified forecasts of supply for 6 arms × 6 parents. Sign: of the 24 changes (6 arms × 4 goods), those whose forecast has the same sign at every parent, and how many of them have the observed sign. Share: explained share of the standardized response of F1a, F1b and length, with one coefficient per feature unless noted. In “+ content price”, the measured change in the price of content enters as a covariate. “Replication” is a second batch on three parents in which each RM has its own reward scale (§ 6 ).
Figure 2: Left: frozen forecast of the change in rent against the observed change over steps 20–30, one point per factorial arm. Right: retrospective forecast for 13 patches of Wang et al. (2026) at step 100.
Figure 3: Prospective test with symmetric shocks: change in formatting after a price rise and a price cut of equal size, divided by the forecast C/(β+β0) , mean over three parents. At β=0.00125 , the policy under the cut reaches the length limit.
β
A
τ (steps)
Initial slope
0.0025
7.4
42
0.150
0.005
6.0
28
0.122
0.01
3.9
17
0.137
Slope on logβ
−0.46
−0.67
−0.07
Equilibrium
−1
−1
0
Table 3: Relaxation of formatting supply after a price shock, mean over three parents. Last two rows: slope of each quantity on logβ , and its value at equilibrium.
Figure 4: Change in formatting supply relative to the placebo after a price shock at three KL coefficients, mean over three parents. Dashed curves: joint fit with an effective coefficient β+β0 .
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Hypothesis
Criterion, fixed before the runs
R1
Spearman ≥0.5 for rent and for gold, steps 160–200
R2
∣ Spearman ∣ of reward spread and change in length at step 200 ≤0.5
R3
Mean change in length of arms 0000, 0101 and 1100 at step 200 at most half of that in the first batch
R4
Price predictor: ≥18/20 signs and share ≥0.5 at the end
U1
≥80% of stable directions correct at the end
U2
Share with frozen κ≥0.5 ; with refitted κ , covariance ≥ price
Appendix
Table 4: Numerical criteria of the pre-specified hypotheses. Values are means over parents unless noted. Stable directions: forecasts with the same sign at every parent.
Analysis
Status
Covariance and price predictors of supply; signs; cross effects
Pre-specified
Feature sets F1a, F1b, L and all five features
Pre-specified
Common and per-feature κ ; standardized units
Pre-specified
Content price as covariate if any arm exceeds 0.3
Pre-specified
Forecasts of rent and gold reward (signs, ranking)
Pre-specified
Arms with more than 10% at the length limit are descriptive
Pre-specified
Appendix
Table 5: Pre-specified and later analyses.
Term
Meaning in this paper
Good, feature
A property of a response that we can count and delete, such as headers or list markers
Supply
Mean count of a feature in the responses of the policy
Price
Mean drop in standardized reward when we delete a feature (Eq. 1 )
Rent
Proxy reward minus gold reward of a response, both standardized
Hedonic price
Coefficient of a feature in a within-prompt regression of reward or rent on all features
Edit
Any change to the reward model, from a label shift to a full replacement
Appendix
Table 6: Terms from economics and their meaning in RLHF.
Table 8: Mean drop in standardized score when goods are deleted, 18,537 parent responses that carry both goods. The interaction row is Δzab−(Δza+Δzb) of Eq. 5 , that is, the sum of the single drops minus the joint drop.
First batch
Replication
P1
P2
P3
P4
P5
P6
P1
P2
P3
Rent, early
0.71
0.94
0.89
0.77
0.94
0.94
0.94
0.83
0.89
Gold, early
0.43
0.89
0.54
0.71
0.83
0.89
0.89
0.83
0.83
Rent, end
−0.71
−0.60
−0.77
−0.49
−0.77
−0.83
−0.71
−0.49
−0.60
Gold, end
−0.77
−0.60
−0.77
−0.60
−0.77
−0.77
−0.77
−0.60
−0.60
Appendix
Table 9: Spearman correlation across six arms between the frozen forecast and the observed change, by parent.
End window, features F1a, F1b, L
6 arms
8 arms
Covariance, per-feature κ
0.82
0.83
Price, per-feature κ
0.50
0.15
Covariance, common κ
0.82
0.79
Price, common κ
0.45
0.08
Appendix
Table 10: Explained share of the standardized response, by arm set.
Figure 5: Change in supply relative to the placebo of the same parent, mean over six parents, first batch. Solid lines: arms with a lower price for headers. Dashed lines: arms with a higher price for headers. Curves are smoothed over three evaluation points.
F1a
F1b
L
Arm
Obs
Cov
Price
Obs
Cov
Price
Obs
Cov
Price
0000
−1.45
−0.54
−1.13
−0.19
−0.63
−1.26
1.28
0.62
−0.47
0101
−1.72
−0.82
−1.25
1.95
1.92
0.94
1.54
1.17
−0.19
1001
1.02
1.24
1.07
−0.30
−0.15
−0.25
1.18
1.54
0.22
1010
0.37
0.71
0.98
−0.52
−0.48
−0.34
0.54
0.72
0.20
1100
1.26
1.33
1.09
2.05
1.66
1.42
2.01
1.67
0.46
Appendix
Table 12: Observed change and forecasts at steps 160–200, standardized units, mean over six parents, first batch. Forecasts use one coefficient per feature fitted across all arms.
Response
F1a
F1b
F2
F4
F1a × F1b
Steps 20–30
F1a
+0.66
−0.12
−0.18
+0.15
+0.07
F1b
+0.27
+0.22
−0.14
+0.04
+0.06
L
+0.51
−0.02
−0.21
+0.18
+0.10
Steps 160–200
F1a
+1.37
−0.01
−0.35
−0.03
+0.10
Appendix
Table 13: Factorial effects in standardized units, 6 arms × 6 parents. F1a × F1b is aliased with F2 × F4.
Step
25
50
100
Covariance, per-feature κ
0.92
0.97
0.97
Price, per-feature κ
0.06
0.15
0.22
Covariance, common κ
0.52
0.64
0.74
Price, common κ
0.05
0.12
0.15
Appendix
Table 14: Calibration runs: explained share of the standardized response of all five features, two arms. With both predictors, 9 of 10 signs are correct.
Hypothesis
Observed
Outcome
R1: Late ranking of rent and gold follows the forecast
Spearman −0.60 (rent), −0.60 (gold)
Not supported
R2: Late length drift no longer tracks reward spread
Spearman 0.03 (first batch: 0.77)
Supported
R3: Length drift of high-spread arms shrinks
drift −49% (134 to 68 tokens)
Not met
R4: Price predictor still forecasts supply
20/20 signs; share 0.59
Supported
Appendix
Table 15: Pre-specified hypotheses of the replication and their outcomes.
Step
25
50
100
200
First batch
0.83
0.66
−0.31
−0.71
Replication
0.71
1.00
0.09
−0.09
Appendix
Table 16: Spearman correlation across arms between rent implied by the observed change in supply at hedonic prices and observed rent.
First batch
Replication
Arm
Extra KL
Δρ
Extra KL
Δρ
0000
0.069
0.109
0.047
0.075
0101
0.091
0.243
0.059
0.161
1100
0.045
0.176
0.025
0.137
1001
0.008
0.103
0.006
0.088
1111
0.006
0.005
0.003
−0.006
Appendix
Table 17: KL accumulated beyond the placebo in steps 160–200 and change in rent at step 200. Arms 0000 and 0101 lower the price of headers.
Figure 6: Change in rent relative to the placebo of the same parent, in the first batch (six parents), the replication and the GRPO batch (three parents each). Shaded: the early and the end window. Solid lines: arms with a lower price for headers.
Figure 7: KL divergence accumulated beyond the placebo in steps 160–200 against the change in rent at step 200, one point per arm. Panel titles give the Spearman correlation.
Forecast
Steps 20–30
Steps 160–200
Arm
ρ
zg
Δρ
Δzg
Δρ
Δzg
1010
0.009
−0.011
0.023
−0.015
0.070
−0.064
1001
0.008
−0.009
0.024
−0.029
0.100
−0.104
1111
0.005
−0.005
−0.001
−0.000
0.004
−0.024
1100
0.001
0.000
−0.006
−0.003
0.160
−0.184
0000
0.000
0.003
−0.008
0.006
0.106
−0.118
Appendix
Table 18: Pre-specified forecasts (covariance at the branch point, mean over six parents) and observed changes relative to the placebo, first batch.
Response
β
A
τ
Slope (0–30)
R2
F1a
0.0025
2.20
34
0.055
0.40
F1a
0.005
1.92
16
0.055
0.45
F1a
0.01
1.32
10
0.060
0.23
F1b
0.0025
5.20
45
0.095
0.80
F1b
0.005
4.22
44
0.067
0.82
F1b
0.01
2.65
28
0.077
0.61
Appendix
Table 19: Relaxation fits by response, mean over three parents.
Response
Model
β0
SSE
BIC
F1
β+β0
0.0065
172
−67.7
F1
β0=0
–
648
249.3
F1
free A , τ
–
165
−61.5
F1b
β+β0
0.0055
57
−334.3
F1b
β0=0
–
241
9.2
F1b
free A , τ
–
55
−329.9
Appendix
Table 20: Joint fits across the three values of β . For headers alone, the estimate of β0 is unstable at one parent.
Arm
β
Slope
τ
Rend
Extra KL
Δzg
ΔL
Rise
0.00125
0.28
20
0.60
−0.042
−0.13
99
Rise
0.005
0.24
23
0.52
0.010
−0.11
100
Rise
0.02
0.18
8
0.35
0.001
−0.02
23
Cut
0.00125 ∗
−0.06
> 400
2.47
0.085
−0.48
261
Cut
0.005
−0.07
48
0.53
0.038
−0.07
150
Cut
0.02
−0.06
46
1.27
0.008
−0.01
−3
Appendix
Table 21: Relaxation with symmetric shocks, mean over three parents ( τ : median). Slope: change in formatting per step over steps 2–20. Rend : change in formatting over steps 360–400 divided by the frozen forecast. Extra KL: KL beyond the placebo over steps 390–400. ΔL in tokens. ∗ All responses reach the length limit.
Hypothesis
Observed
Outcome
S1 (primary): the response overshoots after the cut and falls short after the rise
at 2 of 3 β ; only β=0.02 without capped windows
Partly
S2 (primary): gold falls more after the cut, in step with extra KL
at 1 of 3 β ; Spearman with extra KL −0.35
Not supported
S3: Relaxation is slower after the cut
at 2 of 3 β ; only β=0.02 without capped windows
Partly
S4: The policy travels more KL after the cut
at 3 of 3 β
Supported
S5: Fingerprints of relaxation hold, with an effective coefficient
β0 : 0.0029 (rise), 0 (cut)
Rise only
S6: Initial responses match the forecast ratio
observed / forecast ratio: 0.44, 0.55, 0.61
Not supported
Appendix
Table 22: Pre-specified hypotheses for relaxation with symmetric shocks.
β=0.02
β=0.005
Policy at step 400
Crise
Ccut
Crise
Ccut
Price rise
0.68
0.19
0.64
0.22
Price cut
0.98
2.08
1.04
−2.06
Placebo
0.66
0.73
0.48
0.39
Appendix
Table 23: Covariance of formatting with the reward difference of each treated model, on samples of the policies at step 400, divided by its value at the branch point, mean over three parents. Diagonal of each block: the policy trained on that model.
Hypothesis
Observed
Outcome
G1: Stable supply directions are correct
22/22 correct
Supported
G2: Normalized covariance forecasts supply sizes, at no parent worse than the raw covariance
share 0.75; P3: 0.70 vs. 0.77 (raw)
Partly
G3: Late length drift is independent of reward spread
Spearman +0.77
Not met
G4: Rent forecast holds early; the late ranking reverses, as under RLOO; extra KL orders late rent
early +0.89 ; late −0.83 ; KL vs. rent +0.77
Supported
Appendix
Table 24: Pre-specified hypotheses of the GRPO batch.
Steps 20–30
Steps 160–200
P1
P2
P3
P1
P2
P3
Normalized cov.
0.94
0.88
0.87
0.77
0.81
0.70
Raw cov.
0.90
0.84
0.86
0.74
0.80
0.77
Rent (Spearman)
0.83
0.49
0.83
−0.77
−0.60
−0.83
Gold (Spearman)
0.66
0.26
0.77
−0.77
−0.43
−0.83
Appendix
Table 25: GRPO batch by parent: explained share of F1a, F1b and length with one coefficient per feature, and the Spearman correlation across six arms between the frozen forecast and the observed change in rent and gold reward.
Arm
Extra KL, steps 160–200
Δρ , step 200
0101
0.108
0.263
0000
0.071
0.134
1100
0.040
0.167
1111
0.009
−0.001
1001
0.007
0.103
1010
0.003
0.059
Appendix
Table 26: GRPO batch: KL accumulated beyond the placebo and change in rent at step 200, mean over three parents.
All
Sign
Audits
Length
Demeaned
Rent, step 50
0.91
0.85
0.79
0.80
0.73
Rent, step 100
0.97
1.00
0.93
1.00
0.69
Rent, step 150
0.99
1.00
1.00
1.00
0.79
Gold, step 50
0.28
0.62
− 0.50
1.00
0.37
Gold, step 100
0.40
0.54
− 0.32
1.00
0.47
Gold, step 150
0.51
0.69
− 0.04
1.00
0.56
Appendix
Table 27: Retrospective forecast for 13 patches: Spearman correlation over all patches, share of correct signs, Spearman correlation within audits (7) and within length penalties (4), and after removal of type means.
Upgrade
F1a
F1b
F2
F4
L
ρ
zg
Qwen, 20k
0.51
0.25
0.12
0.07
0.82
−0.051
0.053
Qwen, 80k
0.26
0.10
0.37
0.15
1.07
−0.065
0.062
Llama, 20k
0.22
0.46
0.19
0.24
1.33
−0.033
0.027
Appendix
Table 28: Pre-specified forecasts for real RM upgrades, mean over three parents. Supply in standardized units at the end window with κ frozen from the factorial runs; rent and gold as CovB(⋅,Δr) .
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Reinforcement learning from human feedback (RLHF) is bottlenecked by reward hacking, where the policy exploits errors in a proxy reward model (RM) and produces high RM scores without genuine quality gains. A natural mitigation is pessimism: lowering rewards in regions where the RM is uncertain. However, standard scalar RMs provide no principled notion of uncertainty. We argue that the right object is a distributional reward model p(r∣x,y). Under either a Bayesian inference or a KL-distributionally robust optimization (KL-DRO) lens, the KL-regularized RLHF objective admits a closed-form effective reward r~(x,y)=±βlogEp[e±r/β]. The pessimistic branch unifies the prior heuristics for RM ensemble aggregation: mean aggregation, worst-case optimization (WCO), and uncertainty-weighted optimization (UWO) all emerge as limits or truncations of this single expression. This also clarifies the implicit assumptions of each existing rule.
Ely Hahami, Yoel Zimmermann, Ray Zhou +1
Harvard College, Harvard University, Cambridge, MA, USA · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
Reward model overoptimization in RLHF is usually described as reward hacking. We recast it as rent-seeking: the policy, a firm, earns reward from a regulator, the proxy reward model, that true quality does not back. The lens turns overoptimization into measurable quantities, namely rent, a capture index and deadweight loss, and into testable predictions from the economics of regulation. In a fixed, version-controlled testbed with three model families, accuracy-matched proxy reward models and two gold reward models, RL turns 41% to 66% of its proxy gain into rent while gold reward plateaus. Rent follows each regulator's prices. Neither proxy verifies facts, but one pays for confident specifics and the other for displays of diligence, so RL policies learn to fabricate details about nonexistent entities (97% to 100% of probes) or to deflect factual questions (67% to 98%). Kinship between policy and regulator adds a fixed premium of about 0.1 standard deviations but gives the policy no extra leverage under optimization. At a fixed expected fine, frequent mild audits beat rare severe ones, and random audits are as efficient per audit as targeted ones. Bringing the bunching estimator of tax economics to RL training, we also find a first sign that policies learn to stay just below a targeted audit's threshold.
Zixiao Wang, Xingbo Du, Duzhen Zhang +1
Mohamed bin Zayed University of Artificial Intelligence
Reward models in RLHF are trained to score only the final token of a response - a choice that discards rich signal from every intermediate position and produces models whose token-level outputs are noise. We argue this is a missed opportunity: a well-trained reward model's output at any token should represent the conditional expectation of the final reward given the response so far. We introduce Temporally Coherent Reward Modeling (TCRM), which induces this property via two regularization terms on top of the standard Bradley-Terry loss, with minimizers provably equal to conditional expectations. The regularizers correspond to Monte Carlo and TD value-learning objectives, establishing a direct connection to RL value functions. TCRM requires zero changes to architecture, data, or inference, yet unlocks three capabilities from one principle: interpretable token-level reward trajectories (middle-token pairwise accuracy improved from 50% to 88.9%, final-token accuracy preserved); state-of-the-art PRM performance on ProcessBench (44.9% average F1) among models trained only on outcome data; and unified reward/value modeling in PPO, reducing peak GPU memory by 27% and step time by 19% with matching LLM quality.