Forecasting How RLHF Responds to Reward-Model Edits Before Training
Organizations: Mohamed bin Zayed University of Artificial Intelligence
Abstract
Responses with headers, lists and greater length receive high scores from reward models (RMs), and RLHF policies learn to produce these features even when true quality does not improve. Practitioners therefore edit their RMs, but they learn whether an edit works only after an RL run. We show that the response of a policy to an RM edit can be forecast before training, from samples of the current policy and the scores of the old and new RM. If we read the RM as a set of prices, the first-order theory of KL-regularized RL implies that each feature moves in proportion to its within-prompt covariance with the change in reward. We implant controlled price changes into retrained RMs under a 2^(4-1) fractional factorial design with a self-distilled placebo, run 54 RL runs from six parent policies, and fix every forecast before the runs. With the covariance predictor, we predict the direction of all 20 changes in supply whose forecast has the same sign at every parent and explain 82% of the standardized change. With counterfactual prices, the classical alternative, we explain 50%. The forecast also holds under GRPO and for real RM upgrades, where we explain 66% out of sample with a coefficient frozen from the factorial runs. For rent, the gap between the proxy reward and a gold reward from stronger judges, we rank RM edits correctly with the same covariance over the first 30 steps (Spearman 0.94). This measures the horizon of the forecast: by step 200, the ranking of the factorial edits reverses under both RLOO and GRPO, while the real upgrades keep their order. When we vary the KL coefficient beta, the initial response does not depend on beta, and the response saturates below the KL equilibrium. In one experiment, the long-run response to a price rise is consistent with an additional implicit regularizer beta_0 of about 0.003 to 0.007.
Figures & tables
| SFT samples | Parent responses | |||||||
| Arm | F1a | F1b | F2 | F4 | F1a | F1b | F2 | F4 |
| 0000 | 0.69 | 0.73 | 0.45 | 0.01 | 0.67 | 0.28 | 0.28 | 0.02 |
| 1001 | 0.83 | 0.94 | 0.48 | 0.37 | 0.74 | 0.20 | 0.14 | 0.01 |
| 0101 | 0.72 | 0.75 | 0.42 | 0.39 | 0.87 | 0.44 | 0.46 | 0.18 |
| 1100 | 0.65 | 0.84 | 0.41 | 0.08 | 0.65 | 0.34 | 0.22 | 0.02 |
| 0011 | 0.79 | 0.99 | 0.74 | 0.38 | 0.45 | 0.38 | 0.35 | 0.39 |
| Steps 20–30 | Steps 160–200 | |||
| Predictor | Sign | Share | Sign | Share |
| Covariance | 18/20 | 0.81 | 20/20 | 0.82 |
| Price | 17/20 | 0.58 | 20/20 | 0.50 |
| Robustness | ||||
| Covariance, common | 0.69 | 0.82 | ||
| Price, common | 0.51 | 0.45 | ||
| (steps) | Initial slope | ||
| 0.0025 | 7.4 | 42 | 0.150 |
| 0.005 | 6.0 | 28 | 0.122 |
| 0.01 | 3.9 | 17 | 0.137 |
| Slope on | |||
| Equilibrium |
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Hypothesis | Criterion, fixed before the runs |
|---|---|
| R1 | Spearman for rent and for gold, steps 160–200 |
| R2 | Spearman of reward spread and change in length at step 200 |
| R3 | Mean change in length of arms 0000, 0101 and 1100 at step 200 at most half of that in the first batch |
| R4 | Price predictor: signs and share at the end |
| U1 | of stable directions correct at the end |
| U2 | Share with frozen ; with refitted , covariance price |
| Analysis | Status |
|---|---|
| Covariance and price predictors of supply; signs; cross effects | Pre-specified |
| Feature sets F1a, F1b, and all five features | Pre-specified |
| Common and per-feature ; standardized units | Pre-specified |
| Content price as covariate if any arm exceeds 0.3 | Pre-specified |
| Forecasts of rent and gold reward (signs, ranking) | Pre-specified |
| Arms with more than 10% at the length limit are descriptive | Pre-specified |
| Term | Meaning in this paper |
|---|---|
| Good, feature | A property of a response that we can count and delete, such as headers or list markers |
| Supply | Mean count of a feature in the responses of the policy |
| Price | Mean drop in standardized reward when we delete a feature (Eq. 1 ) |
| Rent | Proxy reward minus gold reward of a response, both standardized |
| Hedonic price | Coefficient of a feature in a within-prompt regression of reward or rent on all features |
| Edit | Any change to the reward model, from a label shift to a full replacement |
| Setting | Value |
|---|---|
| Algorithm | RLOO (one batch: GRPO), KL penalty in the reward |
| KL coefficient | 0.005; stage 4: 0.0025, 0.005, 0.01; symmetric shocks: 0.00125, 0.005, 0.02 |
| Prompts samples | per step |
| Learning rate | |
| Response length | at most 1,024 tokens |
| Steps | 200 (relaxation runs: 400) |
| Deletion | Proxy | Gold | Rent |
| Headers and bold | 0.174 | 0.313 | |
| List markers | 0.028 | 0.044 | |
| Both | 0.485 | 0.216 | |
| Interaction |
| First batch | Replication | ||||||||
| P1 | P2 | P3 | P4 | P5 | P6 | P1 | P2 | P3 | |
| Rent, early | 0.71 | 0.94 | 0.89 | 0.77 | 0.94 | 0.94 | 0.94 | 0.83 | 0.89 |
| Gold, early | 0.43 | 0.89 | 0.54 | 0.71 | 0.83 | 0.89 | 0.89 | 0.83 | 0.83 |
| Rent, end | |||||||||
| Gold, end | |||||||||
| End window, features F1a, F1b, | 6 arms | 8 arms |
|---|---|---|
| Covariance, per-feature | 0.82 | 0.83 |
| Price, per-feature | 0.50 | 0.15 |
| Covariance, common | 0.82 | 0.79 |
| Price, common | 0.45 | 0.08 |
| Steps 20–30 | Steps 160–200 | |||||
|---|---|---|---|---|---|---|
| Predictor | F1a | F1b | F1a | F1b | ||
| Covariance | 0.89 | 0.57 | 0.82 | 0.71 | 0.91 | 0.84 |
| Price | 0.81 | 0.64 | 0.43 | 0.87 | 0.66 | 0.07 |
| F1a | F1b | ||||||||
| Arm | Obs | Cov | Price | Obs | Cov | Price | Obs | Cov | Price |
| 0000 | 1.28 | 0.62 | |||||||
| 0101 | 1.95 | 1.92 | 0.94 | 1.54 | 1.17 | ||||
| 1001 | 1.02 | 1.24 | 1.07 | 1.18 | 1.54 | 0.22 | |||
| 1010 | 0.37 | 0.71 | 0.98 | 0.54 | 0.72 | 0.20 | |||
| 1100 | 1.26 | 1.33 | 1.09 | 2.05 | 1.66 | 1.42 | 2.01 | 1.67 | 0.46 |
| Response | F1a | F1b | F2 | F4 | F1a F1b |
|---|---|---|---|---|---|
| Steps 20–30 | |||||
| F1a | |||||
| F1b | |||||
| Steps 160–200 | |||||
| F1a | |||||
| Step | 25 | 50 | 100 |
|---|---|---|---|
| Covariance, per-feature | 0.92 | 0.97 | 0.97 |
| Price, per-feature | 0.06 | 0.15 | 0.22 |
| Covariance, common | 0.52 | 0.64 | 0.74 |
| Price, common | 0.05 | 0.12 | 0.15 |
| Hypothesis | Observed | Outcome |
|---|---|---|
| R1: Late ranking of rent and gold follows the forecast | Spearman (rent), (gold) | Not supported |
| R2: Late length drift no longer tracks reward spread | Spearman 0.03 (first batch: 0.77) | Supported |
| R3: Length drift of high-spread arms shrinks | drift (134 to 68 tokens) | Not met |
| R4: Price predictor still forecasts supply | 20/20 signs; share 0.59 | Supported |
| Step | 25 | 50 | 100 | 200 |
|---|---|---|---|---|
| First batch | 0.83 | 0.66 | ||
| Replication | 0.71 | 1.00 | 0.09 |
| First batch | Replication | |||
|---|---|---|---|---|
| Arm | Extra KL | Extra KL | ||
| 0000 | 0.069 | 0.109 | 0.047 | 0.075 |
| 0101 | 0.091 | 0.243 | 0.059 | 0.161 |
| 1100 | 0.045 | 0.176 | 0.025 | 0.137 |
| 1001 | 0.008 | 0.103 | 0.006 | 0.088 |
| 1111 | 0.006 | 0.005 | 0.003 | |
| Forecast | Steps 20–30 | Steps 160–200 | ||||
|---|---|---|---|---|---|---|
| Arm | ||||||
| 1010 | 0.009 | 0.023 | 0.070 | |||
| 1001 | 0.008 | 0.024 | 0.100 | |||
| 1111 | 0.005 | 0.004 | ||||
| 1100 | 0.001 | 0.000 | 0.160 | |||
| 0000 | 0.000 | 0.003 | 0.006 | 0.106 | ||
| Response | Slope (0–30) | ||||
|---|---|---|---|---|---|
| F1a | 0.0025 | 2.20 | 34 | 0.055 | 0.40 |
| F1a | 0.005 | 1.92 | 16 | 0.055 | 0.45 |
| F1a | 0.01 | 1.32 | 10 | 0.060 | 0.23 |
| F1b | 0.0025 | 5.20 | 45 | 0.095 | 0.80 |
| F1b | 0.005 | 4.22 | 44 | 0.067 | 0.82 |
| F1b | 0.01 | 2.65 | 28 | 0.077 | 0.61 |
| Response | Model | SSE | BIC | |
|---|---|---|---|---|
| F1 | 0.0065 | 172 | ||
| F1 | – | 648 | 249.3 | |
| F1 | free , | – | 165 | |
| F1b | 0.0055 | 57 | ||
| F1b | – | 241 | 9.2 | |
| F1b | free , | – | 55 |
| Arm | Slope | Extra KL | |||||
|---|---|---|---|---|---|---|---|
| Rise | 0.00125 | 0.28 | 20 | 0.60 | 99 | ||
| Rise | 0.005 | 0.24 | 23 | 0.52 | 0.010 | 100 | |
| Rise | 0.02 | 0.18 | 8 | 0.35 | 0.001 | 23 | |
| Cut | 0.00125 ∗ | 400 | 2.47 | 0.085 | 261 | ||
| Cut | 0.005 | 48 | 0.53 | 0.038 | 150 | ||
| Cut | 0.02 | 46 | 1.27 | 0.008 |
| Hypothesis | Observed | Outcome |
|---|---|---|
| S1 (primary): the response overshoots after the cut and falls short after the rise | at 2 of 3 ; only without capped windows | Partly |
| S2 (primary): gold falls more after the cut, in step with extra KL | at 1 of 3 ; Spearman with extra KL | Not supported |
| S3: Relaxation is slower after the cut | at 2 of 3 ; only without capped windows | Partly |
| S4: The policy travels more KL after the cut | at 3 of 3 | Supported |
| S5: Fingerprints of relaxation hold, with an effective coefficient | : 0.0029 (rise), 0 (cut) | Rise only |
| S6: Initial responses match the forecast ratio | observed / forecast ratio: 0.44, 0.55, 0.61 | Not supported |
| Policy at step 400 | ||||
|---|---|---|---|---|
| Price rise | 0.68 | 0.19 | 0.64 | 0.22 |
| Price cut | 0.98 | 2.08 | 1.04 | |
| Placebo | 0.66 | 0.73 | 0.48 | 0.39 |
| Hypothesis | Observed | Outcome |
|---|---|---|
| G1: Stable supply directions are correct | 22/22 correct | Supported |
| G2: Normalized covariance forecasts supply sizes, at no parent worse than the raw covariance | share 0.75; P3: 0.70 vs. 0.77 (raw) | Partly |
| G3: Late length drift is independent of reward spread | Spearman | Not met |
| G4: Rent forecast holds early; the late ranking reverses, as under RLOO; extra KL orders late rent | early ; late ; KL vs. rent | Supported |
| Steps 20–30 | Steps 160–200 | |||||
| P1 | P2 | P3 | P1 | P2 | P3 | |
| Normalized cov. | 0.94 | 0.88 | 0.87 | 0.77 | 0.81 | 0.70 |
| Raw cov. | 0.90 | 0.84 | 0.86 | 0.74 | 0.80 | 0.77 |
| Rent (Spearman) | 0.83 | 0.49 | 0.83 | |||
| Gold (Spearman) | 0.66 | 0.26 | 0.77 | |||
| Arm | Extra KL, steps 160–200 | , step 200 |
|---|---|---|
| 0101 | 0.108 | 0.263 |
| 0000 | 0.071 | 0.134 |
| 1100 | 0.040 | 0.167 |
| 1111 | 0.009 | |
| 1001 | 0.007 | 0.103 |
| 1010 | 0.003 | 0.059 |
| All | Sign | Audits | Length | Demeaned | |
|---|---|---|---|---|---|
| Rent, step 50 | 0.91 | 0.85 | 0.79 | 0.80 | 0.73 |
| Rent, step 100 | 0.97 | 1.00 | 0.93 | 1.00 | 0.69 |
| Rent, step 150 | 0.99 | 1.00 | 1.00 | 1.00 | 0.79 |
| Gold, step 50 | 0.28 | 0.62 | 0.50 | 1.00 | 0.37 |
| Gold, step 100 | 0.40 | 0.54 | 0.32 | 1.00 | 0.47 |
| Gold, step 150 | 0.51 | 0.69 | 0.04 | 1.00 | 0.56 |
| Upgrade | F1a | F1b | F2 | F4 | |||
|---|---|---|---|---|---|---|---|
| Qwen, 20k | 0.51 | 0.25 | 0.12 | 0.07 | 0.82 | 0.053 | |
| Qwen, 80k | 0.26 | 0.10 | 0.37 | 0.15 | 1.07 | 0.062 | |
| Llama, 20k | 0.22 | 0.46 | 0.19 | 0.24 | 1.33 | 0.027 |
| Hypothesis | Observed | Outcome |
|---|---|---|
| U1: Stable supply directions are correct | 7/7 correct | Supported |
| U2: Frozen coefficients forecast supply sizes; covariance beats price | frozen : 0.66; refitted: 0.77 vs. 0.30 | Supported |
| U3: Early signs and ranking of rent and gold are correct | signs 9/9 (rent), 9/9 (gold); same order | Supported |
| U4a: No late reversal | same order at step 200 | Supported |
| U4b: Rise in headers is close to the forecast | observed / forecast: 1.32, 1.31, 3.49 | Not met (Llama) |
| Forecast | Steps 20–30 | Step 160–200 | ||||
|---|---|---|---|---|---|---|
| Upgrade | ||||||
| Qwen, 20k | 0.053 | 0.053 | 0.211 | |||
| Qwen, 80k | 0.062 | 0.071 | 0.273 | |||
| Llama, 20k | 0.027 | 0.036 | 0.068 | |||
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,855 papers