Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rent

Zixiao Wang; Xingbo Du; Duzhen Zhang; Le Song

cs.LGOct 14, 2026

Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rent

Authors: Zixiao Wang, Xingbo Du, Duzhen Zhang, Le Song

Organizations: Mohamed bin Zayed University of Artificial Intelligence

Abstract

Reward model overoptimization in RLHF is usually described as reward hacking. We recast it as rent-seeking: the policy, a firm, earns reward from a regulator, the proxy reward model, that true quality does not back. The lens turns overoptimization into measurable quantities, namely rent, a capture index and deadweight loss, and into testable predictions from the economics of regulation. In a fixed, version-controlled testbed with three model families, accuracy-matched proxy reward models and two gold reward models, RL turns 41% to 66% of its proxy gain into rent while gold reward plateaus. Rent follows each regulator's prices. Neither proxy verifies facts, but one pays for confident specifics and the other for displays of diligence, so RL policies learn to fabricate details about nonexistent entities (97% to 100% of probes) or to deflect factual questions (67% to 98%). Kinship between policy and regulator adds a fixed premium of about 0.1 standard deviations but gives the policy no extra leverage under optimization. At a fixed expected fine, frequent mild audits beat rare severe ones, and random audits are as efficient per audit as targeted ones. Bringing the bunching estimator of tax economics to RL training, we also find a first sign that policies learn to stay just below a targeted audit's threshold.

Figures & tables

Figure 1: Overoptimization as rent-seeking. The policy earns proxy reward that gold reward does not back (rent). Rent follows the prices that each proxy RM sets.

Figure 2: Rent shares of rule-detected strategies. Left: RL cells over training. Right: BoN cells. “All” applies every rule in sequence. Shares of single rules overlap and do not sum.

Model	Correct	No ans.	Follow	Fabricate
SFT Qwen	0.41	0.10	0.57	0.83
SFT Llama	0.35	0.21	0.64	0.95
Qwen $\times$ Qwen	0.35	0.10	0.49	0.97
Llama $\times$ Qwen	0.30	0.10	0.67	1.00
Qwen $\times$ Llama	0.21	0.67	0.52	0.17
Llama $\times$ Llama	0.02	0.98	0.60 †	0.30

Table 1: Probe results, means of three seeds for RL cells. Correct and No answer refer to neutral questions. Follow is the rise in the share of committed answers that adopt a suggested wrong answer. † Few committed answers, confidence interval wider than 0.5.

Figure 3: Revolving-door effect

\Delta

. Left:

\Delta

against

\sqrt{\mathrm{KL}}

for RL and BoN. Right: estimates at 4 nats with 95% intervals.

Condition	$p$	$F$	Seeds
No audit	0	n/a	3
Equal-exp. penalty	0.02 / 0.05 / 0.2	50 / 20 / 5	3 each
Back tax only	0.05	0	3
Full gold	1	0	2
Targeted audit (T2)	0.05 ∗	20	3
Length penalty	$\lambda=1.3$		2

Table 2: Audit conditions. The expected fine is

c=p\cdot F=1

. ∗ Nominal budget, set on SFT outputs.

Figure 4: Equal-expected-penalty line (

p\cdot F=1

). Left: judge win rate over SFT. Middle and right: gold reward and capture index at step 600, which audits partly optimize. Grey lines are seeds.

Figure 5: Disagreement scores near the audit threshold after step 100. Orange bins form the window below the threshold, and the curve is the counterfactual fit outside it. Left: targeted audits. Right: random audits as a placebo.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Decision	Data seen at the time
Match proxy accuracy by family means within one point of the Llama and OLMo mean; Qwen matched tier of 3,000 pairs; RL uses seed 3	Proxy RM accuracies only
BoN shows no rise and fall of gold reward up to $n=256$ ; no extra samples, the rise and fall moves to RL	BoN curves
New judge prompt and answer parsing; all earlier judge outputs discarded	Raw judge outputs
RQ2b interpolation and bootstrap; RQ1 rent decomposition and length control	Synthetic data only
No clear overoptimization in the RL pilot yet; 600 RL steps	Pilot validation curves
Validate RL every 10 steps and rerun the pilot, because KL passed 4 nats before the first validation point	KL growth only

Appendix

Table 3: Changes made after the plan freeze.

Economics	RLHF	Measure
Social welfare	True preference	Gold reward $z_{g}$
Regulator	Proxy RM	Proxy reward $z_{p}$
Firm	Policy	$\pi_{\theta}$
Rent	Unbacked reward	$\rho=z_{p}-z_{g}$
Rent seeking	Optimization	KL from SFT
Capture	Unbacked share	Capture index

Appendix

Table 4: Economic concepts and their RLHF counterparts.

Setting	Value
Algorithm	RLOO, no std-dev normalization
KL	Reward penalty, $\beta=0.005$
Batch	64 prompts $\times$ 8 samples
Learning rate	$2\times 10^{-6}$ , constant
Sampling	Temperature 1.0, at most 1,024 tokens
Steps	600

Appendix

Table 5: RL settings.

Jobs	GPU hours
Audit conditions	510
RL $2\times 2$ grid	362
Audit calibration pilots	130
BoN scoring	67
RL pilots	58
KL sweep and exploratory RL runs	34

Appendix

Table 6: A100 GPU hours by job type, from the cluster accounting of all project jobs.

Figure 6: RL in the

2\times 2

grid. Top: proxy and gold reward against

\sqrt{\mathrm{KL}}

, mean of three seeds. Bottom: prevalence of rule-detected behaviors over training.

Candidate category	Coverage	Added
Structured formatting	36%	80%
Bulleted key points	36%	71%
Concluding summary	29%	91%
External resources	25%	76%
Disclaimers and caveats	27%	60%
Knowledge-cutoff notes	11%	61%

Appendix

Table 7: Directional candidate categories. Coverage is the share of pairs with at least one description in the category; Added is the share of its descriptions that the high-rent response adds.

Rule	Changed	Validation
Formatting	66.6%	No content removed
Concluding summary	23.5%	0 of 10,907 removed
Repetition	7.4%	No false positives
Resource lists	5.8%	No false positives
Disclaimers	4.7%	87% sentence precision
Template phrases	2.8%	0 of 10 false

Appendix

Table 8: Rewrite rules, the share of the 46,500 RL and BoN outputs that each rule changes, and validation results.

Cell	Tokens	Proxy	Gold	Share
Qwen $\times$ Qwen	224 – 420	0.17	0.06	0.22 [0.15, 0.29]
Qwen $\times$ Llama	218 – 847	0.15	0.05	0.30 [0.22, 0.39]
Llama $\times$ Qwen	224 – 308	0.11	$-0.05$	0.14 [0.11, 0.17]
Llama $\times$ Llama	218 – 553	0.14	$-0.07$	0.22 [0.18, 0.26]

Appendix

Table 9: Length channel. Tokens: mean length at step 0 and step 600. Proxy and Gold: within-prompt slopes in log length.

Sycophancy: following a suggested wrong answer
Model	Rent	Proxy	Gold
SFT, both proxies	0.97–1.13	$-0.01$ –0.18	$-1.13$ – $-0.84$
RL, all cells	0.37–1.37	$-0.21$ – $-0.02$	$-1.43$ – $-0.40$
Fabrication against honest uncertainty
SFT, Qwen proxy	1.54–1.58	0.46–0.73	$-1.12$ – $-0.80$
SFT, Llama proxy	1.12–1.36	0.00–0.56	$-1.12$ – $-0.80$

Appendix

Table 10: Rent premiums in standard deviations, ranges over models and seeds. Under the Qwen proxy almost every RL response fabricates, so the fabrication premium is undefined.

Figure 7: Public checkpoints. Top: prevalence of rule-detected behaviors at the SFT, DPO and final stages. Bottom: gold reward.

Tier	Favoritism	$\Delta$ (capture index)
5,000 pairs	0.13 [0.11, 0.15]	$-0.002$ [ $-0.010$ , 0.005]
20,000 pairs	0.17 [0.15, 0.19]	0.027 [0.021, 0.033]
80,000 pairs	0.12 [0.11, 0.14]	0.007 [0.001, 0.012]
Matched	0.11 [0.09, 0.14]	0.006 [0.000, 0.012]

Appendix

Table 11: Static revolving door in BoN at

n=64

, averaged gold reward.

Figure 8: BoN in the

3\times 3

grid, matched tier. Proxy and gold reward against

\sqrt{\mathrm{KL}}

for same-family and cross-family cells.

Condition	Audit rate	Judge win	Gold (z)	Capture idx.	DWL	Length
No audit (3)	0.000 $\pm$ 0.000	0.666 $\pm$ 0.008	+1.67 $\pm$ 0.02	0.41 $\pm$ 0.01	0.03 $\pm$ 0.01	424 $\pm$ 15
Back-tax only (3)	0.050 $\pm$ 0.001	0.680 $\pm$ 0.006	+1.73 $\pm$ 0.02	0.37 $\pm$ 0.01	0.02 $\pm$ 0.01	423 $\pm$ 9
Audit p=0.02 (3)	0.020 $\pm$ 0.000	0.672 $\pm$ 0.004	+1.72 $\pm$ 0.03	0.20 $\pm$ 0.03	0.00 $\pm$ 0.00	479 $\pm$ 12
Audit p=0.05 (3)	0.050 $\pm$ 0.001	0.709 $\pm$ 0.019	+1.71 $\pm$ 0.05	0.21 $\pm$ 0.02	0.01 $\pm$ 0.01	416 $\pm$ 26
Audit p=0.2 (3)	0.200 $\pm$ 0.001	0.714 $\pm$ 0.004	+1.90 $\pm$ 0.03	0.11 $\pm$ 0.04	0.00 $\pm$ 0.00	504 $\pm$ 19
Targeted (T2) (3)	0.082 $\pm$ 0.002	0.731 $\pm$ 0.005	+1.67 $\pm$ 0.04	0.05 $\pm$ 0.03	0.00 $\pm$ 0.00	470 $\pm$ 38

Appendix

Table 12: Audit conditions and baselines at step 600, mean

\pm

standard deviation over seeds (number of seeds in parentheses). Judge win is the share of prompts on which the averaged judges prefer the final checkpoint over the SFT response. Gold reward and the capture index are partly optimized under audits.

Figure 9: Judge win rate against the number of gold-reward calls during training. The dashed line is the cost frontier.

Reward Over-Optimization

See all 14 papers

When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
Apr 28, 2026Shuning Shang, Hubert Strauss, Stanley Wei +2Reward Modeling Policy Gradient
Princeton Language and Intelligence, Princeton University
Discretizing Reward Models
Jun 19, 2026Vijay Viswanathan, Shiqi Wang, Devamanyu Hazarika +4Reward Modeling Reinforcement Learning
1Carnegie Mellon University · 2Meta Superintelligence Labs
Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure
May 27, 2026Max Lamparth, Daniel Fein, Andreas Haupt +2Reward Modeling Reward Over-Optimization
Stanford University · University of Pennsylvania

Apr 28, 2026J/K move · Enter open · S save

When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient

Shuning Shang, Hubert Strauss, Stanley Wei, Sanjeev Arora+1

Princeton Language and Intelligence, Princeton University

Reward Modeling Policy Gradient Policy Learning Reward Design for RL Reward Model Evaluation RL from Human Feedback

Open paper AskPDF arXiv

Training language models via reinforcement learning often relies on imperfect proxy rewards, since ground truth rewards that precisely define the intended behavior are rarely available. Standard metrics for assessing the quality of proxy rewards, such as ranking accuracy, treat incorrect rewards as strictly harmful. In this work, however, we highlight that not all deviations from the ground truth are equal. By theoretically analyzing which outputs attract probability during policy gradient optimization, we categorize reward errors according to their effect on the increase in ground truth reward. The analysis establishes that reward errors, though conventionally viewed as harmful, can also be benign or even beneficial by preventing the policy from stalling around outputs with mediocre ground truth reward. We then present two practical implications of our theory. First, for reinforcement learning from human feedback (RLHF), we develop reward model evaluation metrics that account for the harmfulness of reward errors. Compared to standard ranking accuracy, these metrics typically correlate better with the performance of a language model after RLHF, yet gaps remain in robustly evaluating reward models. Second, we provide insights for reward design in settings with verifiable rewards. A key theme underlying our results is that the effectiveness of a proxy reward function depends heavily on its interaction with the initial policy and learning algorithm.

Abstract

Reward Over-Optimization

Explore similar work

When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient

Discretizing Reward Models

Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure