Designing robust reinforcement learning (RL) agents in the presence of imperfect reward signals remains a core challenge. In practice, agents are often trained with proxy rewards that only approximate the true objective, leaving them vulnerable to reward hacking, where high proxy returns arise from unintended or exploitative behaviors. Recent work formalizes this issue using r-correlation between proxy and true rewards, but existing methods like occupancy-regularized policy optimization (ORPO) optimize against a fixed proxy and do not provide strong guarantees against broader classes of correlated proxies. In this work, we formulate reward hacking as a robust policy optimization problem over the space of all r-correlated proxy rewards. We derive a tractable max-min formulation, where the agent maximizes performance under the worst-case proxy consistent with the correlation constraint. We further show that when the reward is a linear function of known features, our approach can be adapted to incorporate this prior knowledge, yielding both improved policies and interpretable worst-case rewards. Experiments across several environments show that our algorithms consistently outperform ORPO in worst-case returns, and offer improved robustness and stability across different levels of proxy-true reward correlation. These results show that our approach provides both robustness and transparency in settings where reward design is inherently uncertain. The code is available at https://github.com/ZixuanLiu4869/reward_hacking.
Figures & tables
Env
Traffic
Method
True
Proxy
Worst
Linear Worst ( θ )
Linear Worst* ( θ )
Occ ↓
ORPO
16.91 ± 0.12
3.41 ± 0.13
-1.96e+04 ± 0.02e+04
-0.69 ± 0.01 (0.71, 0.21, 0.69)
-0.83 ± 0.02 (0.63, 0.12, 0.97)
3.82e-04 ± 0.13e-04
ORPO*
10.26 ± 0.09
1.35 ± 0.09
-1.35e+04 ± 0.02e+04
-0.44 ± 0.02 (0.46, 0.18, 0.86)
-0.45 ± 0.01 (0.58, 0.06, 0.81)
1.84e-04 ± 0.07e-04
Max-Min
12.70 ± 0.06
3.63 ± 0.09
-268.31 ± 4.14
-0.06 ± 0.01 (0.01, 0.02, 0.96)
-0.06 ± 0.01 (0.001, 0.02, 0.99)
0.00 ± 0.00
Linear Max-Min
16.46 ± 0.10
2.40 ± 0.11
-1.19e+04 ± 0.01e+04
0.20 ± 0.01 (0.64, 0.07, 0.76)
-0.12 ± 0.01 (0.91, 0.01, 0.67)
0.00 ± 0.00
Env
Pandemic
Table 1: Evaluation results on Traffic, Pandemic, Glucose, and RLHF environments. All policies are trained using only the proxy reward . In Traffic, the proxy reward is based on vel , accel , headway (1, 1, 0.1), while the true reward uses commute , accel , headway (1, 1, 0.1). In Pandemic, the proxy reward includes infection , lower stage , smooth changes (10, 0.1, 0.01), while the true reward additionally includes political with weight 10 after infection . In Glucose, the proxy uses expected patient cost , and the true reward uses magni_bg . In RLHF, the proxy uses a 70M LLM, and the true reward uses a 8B LLM. We report θ in the same order as feature weights. Occ denotes total occupancy over state-action pairs unseen by πref , where discriminator outputs infinity.
Figure 1: Mean reward and standard deviation under sampled θ and true reward features at different proxy–true reward correlation levels r for the Traffic and Pandemic environments. Our methods ( Max-Min and Linear Max-Min ) yield more stable and higher average performance across all choices of r .
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Discriminator loss across RL iterations in the original ORPO implementation . The loss stays flat and high ( ∼ 1.4 for the traffic environment), indicating the discriminator is not adequately trained. This undermines the accuracy of the estimated occupancy ratios.
Figure 3: Discriminator loss over training steps within each RL iteration in our implementation. The loss decreases rapidly from its initial value (e.g., 1.4 to values near 0.2 in the Traffic environment), indicating successful training and improved accuracy of occupancy-ratio estimates.
Hyperparameter
Traffic
Glucose
Pandemic
Learning rate
5×10−3
1×10−2
5×10−4
SGD epochs per iteration
200
20
15
Batch size
40000
100000
3860
SGD minibatch size
16384
1024
64
Appendix
Table 2: Hyperparameters used for discriminator network training across different environments.
Hyperparameter
Traffic
Glucose
Pandemic
Tomato
Training iterations
250
500
260
500
Batch size
40000
100000
3860
3000
Optimizer
Adam
Adam
Adam
Adam
Learning rate
5×10−5
1×10−5
0.0003
1×10−3
Gradient clipping
N/A
10
10
0.1
Discount factor ( γ )
0.99
0.99
0.99
0.99
Appendix
Table 3: Hyperparameters used for PPO training across different environments.
Algorithm
Traffic
Glucose
Pandemic
Tomato
ORPO
≈ 5h
≈ 7h
≈ 14h
≈ 1h
ORPO* / Max-Min / Linear Max-Min
≈ 10h
≈ 37h
≈ 19h
≈ 1h
Appendix
Table 4: Approximate training time for each algorithm across different environments.
Algorithm
Traffic
Glucose
Pandemic
Tomato
ORPO
≈ 1679MB
≈ 1662MB
≈ 1813MB
≈ 2148MB
ORPO* / Max-Min / Linear Max-Min
≈ 1706MB
≈ 1674MB
≈ 1864MB
≈ 1903MB
Appendix
Table 5: Approximate memory usage for each algorithm across different environments.
Figure 4: Evolution of adversarial reward weights θ over training epochs for different environments using the Linear Maxmin method.
Env
Traffic
Method
True
Proxy
Worst
Linear Worst ( θ )
Linear Worst* ( θ )
Occ ↓
ORPO
16.91 ± 0.12
3.41 ± 0.13
-1.96e+04 ± 0.02e+04
-0.69 ± 0.01 (0.71, 0.21, 0.69)
-0.83 ± 0.02 (0.63, 0.12, 0.97)
3.82e-04 ± 0.13e-04
ORPO*
10.26 ± 0.09
1.35 ± 0.09
-1.35e+04 ± 0.02e+04
-0.44 ± 0.02 (0.46, 0.18, 0.86)
-0.45 ± 0.01 (0.58, 0.06, 0.81)
1.84e-04 ± 0.07e-04
Max-Min
12.70 ± 0.06
3.63 ± 0.09
-268.31 ± 4.14
-0.06 ± 0.01 (0.01, 0.02, 0.96)
-0.06 ± 0.01 (0.001, 0.02, 0.99)
0.00 ± 0.00
Linear Max-Min
16.46 ± 0.10
2.40 ± 0.11
-1.19e+04 ± 0.01e+04
0.20 ± 0.01 (0.64, 0.07, 0.76)
-0.12 ± 0.01 (0.91, 0.01, 0.67)
0.00 ± 0.00
Env
Pandemic
Appendix
Table 6: Evaluation results on Traffic, Pandemic, Glucose, and RLHF environments. All policies are trained using only the proxy reward . In Traffic, the proxy reward is based on vel , accel , headway (1, 1, 0.1), while the true reward uses commute , accel , headway (1, 1, 0.1). In Pandemic, the proxy reward includes infection , lower stage , smooth changes (10, 0.1, 0.01), while the true reward additionally includes political with weight 10 after infection . In Glucose, the proxy uses expected patient cost , and the true reward uses magni_bg . In RLHF, the proxy uses a 70M LLM, and the true reward uses a 8B LLM. We report θ in the same order as feature weights. Occ denotes total occupancy over state-action pairs unseen by πref , where discriminator outputs infinity.
Method
Tomato
True
Proxy
Worst
Occ ↓
Worst*
ORPO
6.28 ± 0.22
6.83 ± 0.28
-1.51 ± 0.09
2.50e-04 ± 0.63e-04
-1.51+ Rmin⋅ 2.50e-04
ORPO*
4.00 ± 0.18
3.98 ± 0.23
-1.09 ± 0.10
3.09e-05 ± 0.59e-05
-1.09+ Rmin⋅ 3.09e-05
Max-Min
4.56 ± 0.20
4.68 ± 0.25
-1.37 ± 0.06
1.01e-05 ± 0.43e-05
-1.37+ Rmin⋅ 1.01e-05
Appendix
Table 7: Evaluation results on Tomato environments. All policies are trained using only the proxy reward . In Tomato, the proxy includes number of watered tomatoes plus a bonus at a specific state (sprinkler), while the true reward only measures watered tomatoes . Occ in the Tomato environment denotes total occupancy over state-action pairs unseen by πref , based on 1000 sampled trajectories. Worst refers to the expected worst-case reward computed while excluding those unseen state-action pairs. Worst* denotes the actual expected worst-case reward, while Rmin represents the minimum possible reward of any state-action pair. All rewards are normalized according to the reference policy πref .
r
Occ
Worst ( r=0.1 )
Worst ( r=0.4 )
Worst ( r=0.7 )
Worst ( r=0.9 )
0.1
1.36e-03 ± 0.12e-03
-1.34 ± 0.05
-1.12 ± 0.04
-0.74 ± 0.03
-0.27 ± 0.02
0.4
1.01e-05 ± 0.43e-05
-1.66 ± 0.07
-1.37 ± 0.06
-0.70 ± 0.03
-0.05 ± 0.02
0.7
1.05e-02 ± 0.10e-02
-2.10 ± 0.08
-1.82 ± 0.07
-1.33 ± 0.06
-0.66 ± 0.04
0.9
1.29e-05 ± 0.41e-05
-9.10 ± 0.20
-8.92 ± 0.18
-7.60 ± 0.15
-5.49 ± 0.12
Appendix
Table 8: Evaluation of robustness in the Tomato environment across different training-time correlation levels r . Occ denotes total occupancy over state-action pairs unseen by πref , based on 1000 sampled trajectories. Worst refers to the expected worst-case reward computed while excluding those unseen state-action pairs.
r
Occ
Worst ( r=0.1 )
Worst ( r=0.3 )
Worst ( r=0.5 )
Worst ( r=0.9 )
0.3
0.00 ± 0.00
-2794.63 ± 42.10
-268.31 ± 4.14
-82.07 ± 2.04
-22.03 ± 0.88
0.5
0.00 ± 0.00
-7.71e+04 ± 1.20e+03
-1.95e+04 ± 3.10e+02
-6168.40 ± 124.75
-1350.22 ± 27.95
0.9
9.66e-05 ± 1.84e-05
-3.01e+05 ± 6.05e+03
-9.51e+04 ± 1.89e+03
-2.73e+04 ± 5.45e+02
-9.33e+03 ± 1.88e+02
Appendix
Table 9: Evaluation of robustness in the Traffic environment across different training-time correlation levels r . Occ denotes total occupancy over state-action pairs unseen by πref , based on 200 sampled trajectories. Worst refers to the expected worst-case reward computed while excluding those unseen state-action pairs.
Figure 5: Mean reward and standard deviation under sampled reward functions at different proxy-true reward correlation levels r for the Tomato environment. Our methods ( Max-Min ) yield higher average performance across all choices of r .
Method
Traffic
Pandemic
Glucose
Tomato
RLHF
True
Proxy
True
Proxy
True ( ×103 )
Proxy
True
Proxy
True
Proxy
πref
-1004.33 ± 0.00
1474.30 ± 0.00
-12.01 ± 0.00
-12.01 ± 0.00
-79.7 ± 0.0
-117.75 ± 0.00
5.96 ± 0.00
6.37 ± 0.00
15.97 ± 0.00
-0.29 ± 0.00
ORPO
-666.13 ± 2.34
1542.57 ± 2.62
-12.84 ± 0.17
-10.61 ± 0.15
-49.7 ± 0.6
-67.51 ± 0.27
9.10 ± 0.11
9.10 ± 0.11
16.51 ± 0.07
-0.23 ± 0.02
ORPO*
-799.02 ± 1.77
1501.29 ± 1.83
-11.06 ± 0.15
-11.06 ± 0.15
-48.3 ± 1.0
-59.57 ± 0.28
7.96 ± 0.09
7.96 ± 0.09
N/A
N/A
Max-Min
-750.32 ± 1.28
1546.86 ± 1.82
-11.01 ± 0.14
-11.01 ± 0.14
-48.2 ± 0.5
-66.42 ± 0.29
8.24 ± 0.10
8.24 ± 0.10
16.32 ± 0.06
-0.21 ± 0.01
Linear Max-Min
-675.12 ± 2.04
1522.34 ± 2.19
-9.90 ± 0.09
-5.93 ± 0.10
N/A
N/A
N/A
N/A
N/A
N/A
Appendix
Table 10: Unnormalized performance comparison across all environments.
r
ORPO
Max–Min
λ
True
Worst
Occ
True
Worst
Occ
0.1
0.050
0.46 ± 0.14
-6.08 ± 0.07
8.22e-03 ± 0.27e-03
0.13 ± 0.15
-1.34 ± 0.05
1.36e-03 ± 0.12e-03
0.2
0.049
0.66 ± 0.03
-6.88 ± 0.08
2.8e-03 ± 0.14e-03
7.79 ± 0.10
-0.72 ± 0.05
2.18e-03 ± 0.25e-03
0.3
0.048
0.70 ± 0.03
-7.55 ± 0.04
1.87e-03 ± 0.16e-03
7.68 ± 0.11
-0.96 ± 0.04
1.85e-03 ± 0.16e-03
0.4
0.046
0.16 ± 0.02
-9.20 ± 0.06
1.51e-03 ± 0.13e-03
8.24 ± 0.10
-1.37 ± 0.06
1.01e-05 ± 0.43e-05
0.5
0.043
0.52 ± 0.08
-6.53 ± 0.05
0.028 ± 0.0010
7.38 ± 0.12
-1.21 ± 0.04
2.15e-03 ± 0.37e-03
Appendix
Table 11: Evaluation in the Tomato environment across training-time correlation levels r for ORPO and Max–Min. λ denotes ORPO’s coefficient, with λ=σRproxy1−r2 . We use σRproxy=0.05 in this environment, consistent with the ORPO setting. Occ denotes the total occupancy over state-action pairs unseen by πref . Worst refers to the expected worst-case reward computed under the training r while excluding those unseen state-action pairs.
r
ORPO
Max–Min
λ
True
Worst
Occ
True
Worst
Occ
0.1
1.99e-4
-1063.75 ± 1.08
1.20e+03 ± 0.02e+03
5.53e-03 ± 0.03e-03
-1428.21 ± 5.36
-3.43e+04 ± 0.66e+04
2.29e-04 ± 0.36e-04
0.2
1.95e-4
-775.79 ± 1.15
4.62e+04 ± 0.04e+04
8.71e-04 ± 0.03e-04
-1312.67 ± 9.14
-2.83e+04 ± 0.71e+04
1.49e-04 ± 0.42e-04
0.3
1.91e-4
-689.31 ± 1.12
-5.13e+04 ± 0.04e+04
3.98e-04 ± 0.02e-04
-750.32 ± 1.28
-268.31 ± 4.14
0.00 ± 0.00
0.4
1.83e-4
-1109.41 ± 1.59
-1.84e+03 ± 0.04e+03
4.94e-03 ± 0.05e-03
-732.86 ± 1.19
-314.14 ± 5.37
0.00 ± 0.00
0.5
1.73e-4
-673.45 ± 1.36
-1.51e+04 ± 0.03e+04
6.85e-04 ± 0.01e-04
-1034.42 ± 2.32
-6168.40 ± 124.75
0.00 ± 0.00
Appendix
Table 12: Evaluation in the Traffic environment across training-time correlation levels r for ORPO and Max–Min. λ denotes ORPO’s coefficient, with λ=σRproxy1−r2 . We use σRproxy=2e−4 in this environment, consistent with the ORPO setting. Occ denotes the total occupancy over state-action pairs unseen by πref . Worst refers to the expected worst-case reward computed under the training r while excluding those unseen state-action pairs.
r
ORPO
Max–Min
λ
True
Worst
True
Worst
0.1
0.080
-12.22 ± 0.14
-7.29e+06 ± 0.05e+06
-18.48 ± 0.37
-7.44e+04 ± 0.19e+04
0.2
0.078
-11.77 ± 0.11
-1.70e+07 ± 0.10e+07
-16.31 ± 0.49
-6.03e+04 ± 0.12e+04
0.3
0.076
-12.49 ± 0.20
-2.49e+06 ± 0.05e+06
-19.27 ± 0.38
-7.04e+04 ± 0.05e+04
0.4
0.073
-12.17 ± 0.17
-1.25e+06 ± 0.05e+06
-19.21 ± 0.15
-2.03e+03 ± 0.16e+03
0.5
0.069
-12.26 ± 0.24
-1.07e+06 ± 0.04e+06
-13.75 ± 0.14
-2.58e+03 ± 0.15e+03
Appendix
Table 13: Evaluation in the Pandemic environment across training-time correlation levels r for ORPO and Max–Min. λ denotes ORPO’s coefficient, with λ=σRproxy1−r2 . We use σRproxy=0.08 in this environment, consistent with the ORPO setting. Worst refers to the expected worst-case reward computed under the training r while excluding those unseen state-action pairs.
r
ORPO
Max–Min
λ
True( ×103 )
Worst
True( ×103 )
Worst
0.1
0.050
-90.2 ± 0.7
-350.94 ± 0.37
-169.3 ± 0.6
-317.97 ± 0.12
0.2
0.049
-88.1 ± 0.8
-199.26 ± 0.59
-150.2 ± 0.6
-304.15 ± 0.34
0.3
0.048
-79.1 ± 0.6
-225.71 ± 0.45
-118.3 ± 0.4
-139.47 ± 0.46
0.4
0.046
-72.2 ± 0.4
-206.40 ± 0.27
-113.5 ± 0.8
-123.11 ± 0.32
0.5
0.043
-94.4 ± 0.4
-215.00 ± 0.27
-125.1 ± 0.7
-126.41 ± 0.43
Appendix
Table 14: Evaluation in the Glucose environment across training-time correlation levels r for ORPO and Max–Min. λ denotes ORPO’s coefficient, with λ=σRproxy1−r2 . We use σRproxy=0.05 in this environment, consistent with the ORPO setting. Worst refers to the expected worst-case reward computed under the training r while excluding those unseen state-action pairs.
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
Reward hacking is usually studied after it becomes visible, once a model earns high proxy reward while failing the intended task. We instead study what proxy RL teaches before that failure appears. We introduce Proxy Reward Internalization and Mechanistic Exploitation (PRIME), a learned capability to assess task correctness, predict proxy acceptance, and reason about exploitable proxy--gold gaps. In coding RL environments with exploitable pytest rewards, we measure PRIME through chain-of-thought monitoring, direct probes, and activation-level concept vectors. We find that PRIME emerges in a staged sequence before sustained reward hacking, and that its current direct-probe score forecasts later hack onset and severity even when the visible hack rate is still low. PRIME also adapts when the evaluator changes, retargeting to whichever proxy--gold gap remains rewarded and persisting when gold reward suppresses overt hacking, and ablating its activation directions reduces hacking. Across checkpoints, in-domain PRIME tracks out-of-domain misalignment. Together these results suggest that exploitable proxy RL amplifies a proxy-internalization capability upstream of visible hacking, making PRIME a candidate early-warning signal for broader alignment risk.
Reinforcement learning agents can exploit misspecified reward signals to achieve high apparent returns while failing on the intended objective, a failure mode known as reward hacking. Existing practical defenses typically constrain policy updates to stay near a known safe reference, creating a tension between suppressing hacking and permitting legitimate improvement. We propose Modification-Considering Value Learning (MCVL), which operationalizes the theoretical idea of current utility optimization for standard value-based RL. MCVL wraps an off-policy learner and treats each incoming transition as a candidate modification: it forecasts two training paths, one that includes the transition and one that does not, and scores both with a frozen bootstrapped-return estimator derived from a learned reward model and value function. The transition is admitted only if inclusion does not decrease the score. We formalize conditions under which this filtering is both safe and permissive, and instantiate MCVL with DDQN and TD3. Across four safety-relevant gridworlds and three modified MuJoCo continuous-control tasks with diverse hacking mechanisms, MCVL mitigates reward hacking while continuing to improve the intended objective. Project website: ktolnos.github.io/mcvl/.
A reinforcement-learning (RL) pipeline for a legged robot is assembled from proxies. A reward stands in for intended behaviour, a curriculum gate stands in for competence, an evaluation statistic stands in for robustness, and a reference motion stands in for an achievable skill. The traditional view treats only the first of these as optimised against, and so locates specification failure (reward hacking) in the reward alone. I argue that all four are proxies in the same formal sense, that each has a characteristic divergence mechanism, and that each admits a reformulation that closes it. For every layer I state the traditional formulation, derive the condition under which it diverges from its target, and give the alternative: first-order (L1) costs where quadratic kernels are flat, peak and outcome statistics where curriculum gates average, gate reachability and information checks, deterministic and phase-desynchronised evaluation, curriculum state treated as part of the model, feasibility-first reference design with residual feed-forward, and function-preserving input widening that lets one policy grow instead of being retrained. The arguments are illustrated by measurements from one continuous lineage of a PPO policy for a simulated 1.91 m humanoid, grown over four stages and 13,500 iterations on a single laptop GPU. Among them, a curriculum gate built on averaged error advanced at its rate limit on every check while the skill it gated was absent, and a batched push test whose synchronised resets aliased the gait phase ranked a 0.5 m/s push as more dangerous than a 2.0 m/s one.