Robust Optimization for Mitigating Reward Hacking with Correlated Proxies

Zixuan Liu; Xiaolin Sun; Zizhan Zheng

cs.LGApr 13, 2026

Robust Optimization for Mitigating Reward Hacking with Correlated Proxies

Authors: Zixuan Liu, Xiaolin Sun, Zizhan Zheng

Organizations: Department of Computer Science Tulane University New Orleans, LA 70118, USA

Abstract

Designing robust reinforcement learning (RL) agents in the presence of imperfect reward signals remains a core challenge. In practice, agents are often trained with proxy rewards that only approximate the true objective, leaving them vulnerable to reward hacking, where high proxy returns arise from unintended or exploitative behaviors. Recent work formalizes this issue using r-correlation between proxy and true rewards, but existing methods like occupancy-regularized policy optimization (ORPO) optimize against a fixed proxy and do not provide strong guarantees against broader classes of correlated proxies. In this work, we formulate reward hacking as a robust policy optimization problem over the space of all r-correlated proxy rewards. We derive a tractable max-min formulation, where the agent maximizes performance under the worst-case proxy consistent with the correlation constraint. We further show that when the reward is a linear function of known features, our approach can be adapted to incorporate this prior knowledge, yielding both improved policies and interpretable worst-case rewards. Experiments across several environments show that our algorithms consistently outperform ORPO in worst-case returns, and offer improved robustness and stability across different levels of proxy-true reward correlation. These results show that our approach provides both robustness and transparency in settings where reward design is inherently uncertain. The code is available at https://github.com/ZixuanLiu4869/reward_hacking.

Figures & tables

Env	Traffic
Method	True	Proxy	Worst	Linear Worst ( $\bm{\theta}$ )	Linear Worst* ( $\bm{\theta}$ )	Occ $\downarrow$
ORPO	16.91 $\pm$ 0.12	3.41 $\pm$ 0.13	-1.96e+04 $\pm$ 0.02e+04	-0.69 $\pm$ 0.01 (0.71, 0.21, 0.69)	-0.83 $\pm$ 0.02 (0.63, 0.12, 0.97)	3.82e-04 $\pm$ 0.13e-04
ORPO*	10.26 $\pm$ 0.09	1.35 $\pm$ 0.09	-1.35e+04 $\pm$ 0.02e+04	-0.44 $\pm$ 0.02 (0.46, 0.18, 0.86)	-0.45 $\pm$ 0.01 (0.58, 0.06, 0.81)	1.84e-04 $\pm$ 0.07e-04
Max-Min	12.70 $\pm$ 0.06	3.63 $\pm$ 0.09	-268.31 $\pm$ 4.14	-0.06 $\pm$ 0.01 (0.01, 0.02, 0.96)	-0.06 $\pm$ 0.01 (0.001, 0.02, 0.99)	0.00 $\pm$ 0.00
Linear Max-Min	16.46 $\pm$ 0.10	2.40 $\pm$ 0.11	-1.19e+04 $\pm$ 0.01e+04	0.20 $\pm$ 0.01 (0.64, 0.07, 0.76)	-0.12 $\pm$ 0.01 (0.91, 0.01, 0.67)	0.00 $\pm$ 0.00
Env	Pandemic

Table 1: Evaluation results on Traffic, Pandemic, Glucose, and RLHF environments. All policies are trained using only the proxy reward . In Traffic, the proxy reward is based on vel , accel , headway (1, 1, 0.1), while the true reward uses commute , accel , headway (1, 1, 0.1). In Pandemic, the proxy reward includes infection , lower stage , smooth changes (10, 0.1, 0.01), while the true reward additionally includes political with weight 10 after infection . In Glucose, the proxy uses expected patient cost , and the true reward uses magni_bg . In RLHF, the proxy uses a 70M LLM, and the true reward uses a 8B LLM. We report

\bm{\theta}

in the same order as feature weights. Occ denotes total occupancy over state-action pairs unseen by

\pi_{\text{ref}}

, where discriminator outputs infinity.

Figure 1: Mean reward and standard deviation under sampled $\theta$ and true reward features at different proxy–true reward correlation levels $r$ for the Traffic and Pandemic environments. Our methods ( Max-Min and Linear Max-Min ) yield more stable and higher average performance across all choices of $r$ .

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Discriminator loss across RL iterations in the original ORPO implementation . The loss stays flat and high ( $\sim$ 1.4 for the traffic environment), indicating the discriminator is not adequately trained. This undermines the accuracy of the estimated occupancy ratios.

Panel of Figure 3 — Figure 3: Discriminator loss over training steps within each RL iteration in our implementation. The loss decreases rapidly from its initial value (e.g., 1.4 to values near 0.2 in the Traffic environment), indicating successful training and improved accuracy of occupancy-ratio estimates.

Hyperparameter	Traffic	Glucose	Pandemic
Learning rate	$5\times 10^{-3}$	$1\times 10^{-2}$	$5\times 10^{-4}$
SGD epochs per iteration	200	20	15
Batch size	40000	100000	3860
SGD minibatch size	16384	1024	64

Appendix

Table 2: Hyperparameters used for discriminator network training across different environments.

Hyperparameter	Traffic	Glucose	Pandemic	Tomato
Training iterations	250	500	260	500
Batch size	40000	100000	3860	3000
Optimizer	Adam	Adam	Adam	Adam
Learning rate	$5\times 10^{-5}$	$1\times 10^{-5}$	0.0003	$1\times 10^{-3}$
Gradient clipping	N/A	10	10	0.1
Discount factor ( $\gamma$ )	0.99	0.99	0.99	0.99

Appendix

Table 3: Hyperparameters used for PPO training across different environments.

Algorithm	Traffic	Glucose	Pandemic	Tomato
ORPO	$\approx$ 5h	$\approx$ 7h	$\approx$ 14h	$\approx$ 1h
ORPO* / Max-Min / Linear Max-Min	$\approx$ 10h	$\approx$ 37h	$\approx$ 19h	$\approx$ 1h

Appendix

Table 4: Approximate training time for each algorithm across different environments.

Algorithm	Traffic	Glucose	Pandemic	Tomato
ORPO	$\approx$ 1679MB	$\approx$ 1662MB	$\approx$ 1813MB	$\approx$ 2148MB
ORPO* / Max-Min / Linear Max-Min	$\approx$ 1706MB	$\approx$ 1674MB	$\approx$ 1864MB	$\approx$ 1903MB

Appendix

Table 5: Approximate memory usage for each algorithm across different environments.

Panel of Figure 4 — Figure 4: Evolution of adversarial reward weights $\bm{\theta}$ over training epochs for different environments using the Linear Maxmin method.

Env	Traffic
Method	True	Proxy	Worst	Linear Worst ( $\bm{\theta}$ )	Linear Worst* ( $\bm{\theta}$ )	Occ $\downarrow$
ORPO	16.91 $\pm$ 0.12	3.41 $\pm$ 0.13	-1.96e+04 $\pm$ 0.02e+04	-0.69 $\pm$ 0.01 (0.71, 0.21, 0.69)	-0.83 $\pm$ 0.02 (0.63, 0.12, 0.97)	3.82e-04 $\pm$ 0.13e-04
ORPO*	10.26 $\pm$ 0.09	1.35 $\pm$ 0.09	-1.35e+04 $\pm$ 0.02e+04	-0.44 $\pm$ 0.02 (0.46, 0.18, 0.86)	-0.45 $\pm$ 0.01 (0.58, 0.06, 0.81)	1.84e-04 $\pm$ 0.07e-04
Max-Min	12.70 $\pm$ 0.06	3.63 $\pm$ 0.09	-268.31 $\pm$ 4.14	-0.06 $\pm$ 0.01 (0.01, 0.02, 0.96)	-0.06 $\pm$ 0.01 (0.001, 0.02, 0.99)	0.00 $\pm$ 0.00
Linear Max-Min	16.46 $\pm$ 0.10	2.40 $\pm$ 0.11	-1.19e+04 $\pm$ 0.01e+04	0.20 $\pm$ 0.01 (0.64, 0.07, 0.76)	-0.12 $\pm$ 0.01 (0.91, 0.01, 0.67)	0.00 $\pm$ 0.00
Env	Pandemic

Appendix

Table 6: Evaluation results on Traffic, Pandemic, Glucose, and RLHF environments. All policies are trained using only the proxy reward . In Traffic, the proxy reward is based on vel , accel , headway (1, 1, 0.1), while the true reward uses commute , accel , headway (1, 1, 0.1). In Pandemic, the proxy reward includes infection , lower stage , smooth changes (10, 0.1, 0.01), while the true reward additionally includes political with weight 10 after infection . In Glucose, the proxy uses expected patient cost , and the true reward uses magni_bg . In RLHF, the proxy uses a 70M LLM, and the true reward uses a 8B LLM. We report

\bm{\theta}

in the same order as feature weights. Occ denotes total occupancy over state-action pairs unseen by

\pi_{\text{ref}}

, where discriminator outputs infinity.

Method	Tomato
	True	Proxy	Worst	Occ $\downarrow$	Worst*
ORPO	6.28 $\pm$ 0.22	6.83 $\pm$ 0.28	-1.51 $\pm$ 0.09	2.50e-04 $\pm$ 0.63e-04	-1.51+ $R_{\text{min}}\cdot$ 2.50e-04
ORPO*	4.00 $\pm$ 0.18	3.98 $\pm$ 0.23	-1.09 $\pm$ 0.10	3.09e-05 $\pm$ 0.59e-05	-1.09+ $R_{\text{min}}\cdot$ 3.09e-05
Max-Min	4.56 $\pm$ 0.20	4.68 $\pm$ 0.25	-1.37 $\pm$ 0.06	1.01e-05 $\pm$ 0.43e-05	-1.37+ $R_{\text{min}}\cdot$ 1.01e-05

Appendix

Table 7: Evaluation results on Tomato environments. All policies are trained using only the proxy reward . In Tomato, the proxy includes number of watered tomatoes plus a bonus at a specific state (sprinkler), while the true reward only measures watered tomatoes . Occ in the Tomato environment denotes total occupancy over state-action pairs unseen by

\pi_{\text{ref}}

, based on 1000 sampled trajectories. Worst refers to the expected worst-case reward computed while excluding those unseen state-action pairs. Worst* denotes the actual expected worst-case reward, while

R_{\text{min}}

represents the minimum possible reward of any state-action pair. All rewards are normalized according to the reference policy

\pi_{\text{ref}}

$r$	Occ	Worst ( $r=0.1$ )	Worst ( $r=0.4$ )	Worst ( $r=0.7$ )	Worst ( $r=0.9$ )
0.1	1.36e-03 $\pm$ 0.12e-03	-1.34 $\pm$ 0.05	-1.12 $\pm$ 0.04	-0.74 $\pm$ 0.03	-0.27 $\pm$ 0.02
0.4	1.01e-05 $\pm$ 0.43e-05	-1.66 $\pm$ 0.07	-1.37 $\pm$ 0.06	-0.70 $\pm$ 0.03	-0.05 $\pm$ 0.02
0.7	1.05e-02 $\pm$ 0.10e-02	-2.10 $\pm$ 0.08	-1.82 $\pm$ 0.07	-1.33 $\pm$ 0.06	-0.66 $\pm$ 0.04
0.9	1.29e-05 $\pm$ 0.41e-05	-9.10 $\pm$ 0.20	-8.92 $\pm$ 0.18	-7.60 $\pm$ 0.15	-5.49 $\pm$ 0.12

Appendix

Table 8: Evaluation of robustness in the Tomato environment across different training-time correlation levels

r

. Occ denotes total occupancy over state-action pairs unseen by

\pi_{\text{ref}}

, based on 1000 sampled trajectories. Worst refers to the expected worst-case reward computed while excluding those unseen state-action pairs.

$r$	Occ	Worst ( $r=0.1$ )	Worst ( $r=0.3$ )	Worst ( $r=0.5$ )	Worst ( $r=0.9$ )
0.3	0.00 $\pm$ 0.00	-2794.63 $\pm$ 42.10	-268.31 $\pm$ 4.14	-82.07 $\pm$ 2.04	-22.03 $\pm$ 0.88
0.5	0.00 $\pm$ 0.00	-7.71e+04 $\pm$ 1.20e+03	-1.95e+04 $\pm$ 3.10e+02	-6168.40 $\pm$ 124.75	-1350.22 $\pm$ 27.95
0.9	9.66e-05 $\pm$ 1.84e-05	-3.01e+05 $\pm$ 6.05e+03	-9.51e+04 $\pm$ 1.89e+03	-2.73e+04 $\pm$ 5.45e+02	-9.33e+03 $\pm$ 1.88e+02

Appendix

Table 9: Evaluation of robustness in the Traffic environment across different training-time correlation levels

r

. Occ denotes total occupancy over state-action pairs unseen by

\pi_{\text{ref}}

, based on 200 sampled trajectories. Worst refers to the expected worst-case reward computed while excluding those unseen state-action pairs.

Figure 5: Mean reward and standard deviation under sampled reward functions at different proxy-true reward correlation levels $r$ for the Tomato environment. Our methods ( Max-Min ) yield higher average performance across all choices of $r$ .

Method	Traffic		Pandemic		Glucose		Tomato		RLHF
	True	Proxy	True	Proxy	True ( $\times 10^{3}$ )	Proxy	True	Proxy	True	Proxy
$\pi_{\text{ref}}$	-1004.33 $\pm$ 0.00	1474.30 $\pm$ 0.00	-12.01 $\pm$ 0.00	-12.01 $\pm$ 0.00	-79.7 $\pm$ 0.0	-117.75 $\pm$ 0.00	5.96 $\pm$ 0.00	6.37 $\pm$ 0.00	15.97 $\pm$ 0.00	-0.29 $\pm$ 0.00
ORPO	-666.13 $\pm$ 2.34	1542.57 $\pm$ 2.62	-12.84 $\pm$ 0.17	-10.61 $\pm$ 0.15	-49.7 $\pm$ 0.6	-67.51 $\pm$ 0.27	9.10 $\pm$ 0.11	9.10 $\pm$ 0.11	16.51 $\pm$ 0.07	-0.23 $\pm$ 0.02
ORPO*	-799.02 $\pm$ 1.77	1501.29 $\pm$ 1.83	-11.06 $\pm$ 0.15	-11.06 $\pm$ 0.15	-48.3 $\pm$ 1.0	-59.57 $\pm$ 0.28	7.96 $\pm$ 0.09	7.96 $\pm$ 0.09	N/A	N/A
Max-Min	-750.32 $\pm$ 1.28	1546.86 $\pm$ 1.82	-11.01 $\pm$ 0.14	-11.01 $\pm$ 0.14	-48.2 $\pm$ 0.5	-66.42 $\pm$ 0.29	8.24 $\pm$ 0.10	8.24 $\pm$ 0.10	16.32 $\pm$ 0.06	-0.21 $\pm$ 0.01
Linear Max-Min	-675.12 $\pm$ 2.04	1522.34 $\pm$ 2.19	-9.90 $\pm$ 0.09	-5.93 $\pm$ 0.10	N/A	N/A	N/A	N/A	N/A	N/A

Appendix

Table 10: Unnormalized performance comparison across all environments.

$r$	ORPO				Max–Min
$r$	$\lambda$	True	Worst	Occ	True	Worst	Occ
0.1	0.050	0.46 $\pm$ 0.14	-6.08 $\pm$ 0.07	8.22e-03 $\pm$ 0.27e-03	0.13 $\pm$ 0.15	-1.34 $\pm$ 0.05	1.36e-03 $\pm$ 0.12e-03
0.2	0.049	0.66 $\pm$ 0.03	-6.88 $\pm$ 0.08	2.8e-03 $\pm$ 0.14e-03	7.79 $\pm$ 0.10	-0.72 $\pm$ 0.05	2.18e-03 $\pm$ 0.25e-03
0.3	0.048	0.70 $\pm$ 0.03	-7.55 $\pm$ 0.04	1.87e-03 $\pm$ 0.16e-03	7.68 $\pm$ 0.11	-0.96 $\pm$ 0.04	1.85e-03 $\pm$ 0.16e-03
0.4	0.046	0.16 $\pm$ 0.02	-9.20 $\pm$ 0.06	1.51e-03 $\pm$ 0.13e-03	8.24 $\pm$ 0.10	-1.37 $\pm$ 0.06	1.01e-05 $\pm$ 0.43e-05
0.5	0.043	0.52 $\pm$ 0.08	-6.53 $\pm$ 0.05	0.028 $\pm$ 0.0010	7.38 $\pm$ 0.12	-1.21 $\pm$ 0.04	2.15e-03 $\pm$ 0.37e-03

Appendix

Table 11: Evaluation in the Tomato environment across training-time correlation levels

r

for ORPO and Max–Min.

\lambda

denotes ORPO’s coefficient, with

\lambda=\sigma_{R_{\text{proxy}}}\sqrt{1-r^{2}}

. We use

\sigma_{R_{\text{proxy}}}=0.05

in this environment, consistent with the ORPO setting. Occ denotes the total occupancy over state-action pairs unseen by

\pi_{\text{ref}}

. Worst refers to the expected worst-case reward computed under the training

r

while excluding those unseen state-action pairs.

$r$	ORPO				Max–Min
$r$	$\lambda$	True	Worst	Occ	True	Worst	Occ
0.1	1.99e-4	-1063.75 $\pm$ 1.08	1.20e+03 $\pm$ 0.02e+03	5.53e-03 $\pm$ 0.03e-03	-1428.21 $\pm$ 5.36	-3.43e+04 $\pm$ 0.66e+04	2.29e-04 $\pm$ 0.36e-04
0.2	1.95e-4	-775.79 $\pm$ 1.15	4.62e+04 $\pm$ 0.04e+04	8.71e-04 $\pm$ 0.03e-04	-1312.67 $\pm$ 9.14	-2.83e+04 $\pm$ 0.71e+04	1.49e-04 $\pm$ 0.42e-04
0.3	1.91e-4	-689.31 $\pm$ 1.12	-5.13e+04 $\pm$ 0.04e+04	3.98e-04 $\pm$ 0.02e-04	-750.32 $\pm$ 1.28	-268.31 $\pm$ 4.14	0.00 $\pm$ 0.00
0.4	1.83e-4	-1109.41 $\pm$ 1.59	-1.84e+03 $\pm$ 0.04e+03	4.94e-03 $\pm$ 0.05e-03	-732.86 $\pm$ 1.19	-314.14 $\pm$ 5.37	0.00 $\pm$ 0.00
0.5	1.73e-4	-673.45 $\pm$ 1.36	-1.51e+04 $\pm$ 0.03e+04	6.85e-04 $\pm$ 0.01e-04	-1034.42 $\pm$ 2.32	-6168.40 $\pm$ 124.75	0.00 $\pm$ 0.00

Appendix

Table 12: Evaluation in the Traffic environment across training-time correlation levels

r

for ORPO and Max–Min.

\lambda

denotes ORPO’s coefficient, with

\lambda=\sigma_{R_{\text{proxy}}}\sqrt{1-r^{2}}

. We use

\sigma_{R_{\text{proxy}}}=2e-4

in this environment, consistent with the ORPO setting. Occ denotes the total occupancy over state-action pairs unseen by

\pi_{\text{ref}}

. Worst refers to the expected worst-case reward computed under the training

r

while excluding those unseen state-action pairs.

$r$	ORPO			Max–Min
$r$	$\lambda$	True	Worst	True	Worst
0.1	0.080	-12.22 $\pm$ 0.14	-7.29e+06 $\pm$ 0.05e+06	-18.48 $\pm$ 0.37	-7.44e+04 $\pm$ 0.19e+04
0.2	0.078	-11.77 $\pm$ 0.11	-1.70e+07 $\pm$ 0.10e+07	-16.31 $\pm$ 0.49	-6.03e+04 $\pm$ 0.12e+04
0.3	0.076	-12.49 $\pm$ 0.20	-2.49e+06 $\pm$ 0.05e+06	-19.27 $\pm$ 0.38	-7.04e+04 $\pm$ 0.05e+04
0.4	0.073	-12.17 $\pm$ 0.17	-1.25e+06 $\pm$ 0.05e+06	-19.21 $\pm$ 0.15	-2.03e+03 $\pm$ 0.16e+03
0.5	0.069	-12.26 $\pm$ 0.24	-1.07e+06 $\pm$ 0.04e+06	-13.75 $\pm$ 0.14	-2.58e+03 $\pm$ 0.15e+03

Appendix

Table 13: Evaluation in the Pandemic environment across training-time correlation levels

r

for ORPO and Max–Min.

\lambda

denotes ORPO’s coefficient, with

\lambda=\sigma_{R_{\text{proxy}}}\sqrt{1-r^{2}}

. We use

\sigma_{R_{\text{proxy}}}=0.08

in this environment, consistent with the ORPO setting. Worst refers to the expected worst-case reward computed under the training

r

while excluding those unseen state-action pairs.

$r$	ORPO			Max–Min
$r$	$\lambda$	True( $\times 10^{3}$ )	Worst	True( $\times 10^{3}$ )	Worst
0.1	0.050	-90.2 $\pm$ 0.7	-350.94 $\pm$ 0.37	-169.3 $\pm$ 0.6	-317.97 $\pm$ 0.12
0.2	0.049	-88.1 $\pm$ 0.8	-199.26 $\pm$ 0.59	-150.2 $\pm$ 0.6	-304.15 $\pm$ 0.34
0.3	0.048	-79.1 $\pm$ 0.6	-225.71 $\pm$ 0.45	-118.3 $\pm$ 0.4	-139.47 $\pm$ 0.46
0.4	0.046	-72.2 $\pm$ 0.4	-206.40 $\pm$ 0.27	-113.5 $\pm$ 0.8	-123.11 $\pm$ 0.32
0.5	0.043	-94.4 $\pm$ 0.4	-215.00 $\pm$ 0.27	-125.1 $\pm$ 0.7	-126.41 $\pm$ 0.43

Appendix

Table 14: Evaluation in the Glucose environment across training-time correlation levels

r

for ORPO and Max–Min.

\lambda

denotes ORPO’s coefficient, with

\lambda=\sigma_{R_{\text{proxy}}}\sqrt{1-r^{2}}

. We use

\sigma_{R_{\text{proxy}}}=0.05

in this environment, consistent with the ORPO setting. Worst refers to the expected worst-case reward computed under the training

r

while excluding those unseen state-action pairs.

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 181 papers

Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization
Jun 8, 2026Mohammad Beigi, Ming Jin, Lifu HuangReward Hacking Reinforcement Learning with Verifiable Rewards
1UC Davis · 2Virginia Tech
Modification-Considering Value Learning for Reward Hacking Mitigation in RL
Jun 27, 2026Evgenii Opryshko, Umangi Jain, Igor GilitschenskiReward Hacking Reinforcement Learning
University of Toronto · Vector Institute
Beyond Reward Hacking: Proxy Divergence Across Four Layers of a Staged Humanoid Learning Pipeline
Oct 2, 2026Arunabh BoraRobotic Policy Evaluation Reward Design for RL
London, United Kingdom

Jun 8, 2026J/K move · Enter open · S save

Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization

Mohammad Beigi, Ming Jin, Lifu Huang

1UC Davis · 2Virginia Tech

Reward Hacking Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reward hacking is usually studied after it becomes visible, once a model earns high proxy reward while failing the intended task. We instead study what proxy RL teaches before that failure appears. We introduce Proxy Reward Internalization and Mechanistic Exploitation (PRIME), a learned capability to assess task correctness, predict proxy acceptance, and reason about exploitable proxy--gold gaps. In coding RL environments with exploitable pytest rewards, we measure PRIME through chain-of-thought monitoring, direct probes, and activation-level concept vectors. We find that PRIME emerges in a staged sequence before sustained reward hacking, and that its current direct-probe score forecasts later hack onset and severity even when the visible hack rate is still low. PRIME also adapts when the evaluator changes, retargeting to whichever proxy--gold gap remains rewarded and persisting when gold reward suppresses overt hacking, and ablating its activation directions reduces hacking. Across checkpoints, in-domain PRIME tracks out-of-domain misalignment. Together these results suggest that exploitable proxy RL amplifies a proxy-internalization capability upstream of visible hacking, making PRIME a candidate early-warning signal for broader alignment risk.

Abstract

Reward Hacking

Explore similar work

Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization

Modification-Considering Value Learning for Reward Hacking Mitigation in RL

Beyond Reward Hacking: Proxy Divergence Across Four Layers of a Staged Humanoid Learning Pipeline