Optimistic Verifiers Get Hacked: Reward Hacking Under an Exact Process Oracle
Organizations: University College of Engineering, BIT Campus, Anna University, Tiruchirappalli, India · Central University of Karnataka, Kalaburagi, India · National Institute of Technology Tiruchirappalli, India · Jawaharlal Nehru University, New Delhi, India
Abstract
Reinforcement learning from verifiable rewards inherits whatever its verifier gets wrong. Outside mathematics and code the verifier is a proxy, a rubric, a learned judge or a hand-written checker, and its distance from the true objective is never measured, so a rising reward cannot be told apart from a hack. We build a setting where the true reward is exactly computable at the process level: executable authorization policies over 124,416 contexts admit an oracle that lists every minimal sufficient reason for every decision. We compile these policies into exact process rewards, degrade the verifier along seven realistic axes (outcome-only, sufficiency, recall-only, extraction noise, coarse taxonomy, stale policy, LLM judge), and train against each while the exact verifier scores every rollout. Four phenotypes appear. Optimistic verifiers are hacked by padding: the sufficiency proxy ends at 0.940 while true faithfulness falls from 0.731 to 0.667 on 96 held-out episodes. Outcome-only verifiers saturate and stop teaching, noise dilutes, and ranking-preserving gaps do no harm. A model-free audit of the verifier, its selection error and gap sign, together with a short saturation pilot, anticipates which phenotype a run will produce. Across nine configurations spanning two frameworks, two model vendors, two adapter sizes, three precisions, two initialisations and two seeds, the phenotypes recur, sufficiency damage is positive in all 10 replication cells and beyond two standard errors in 7, and optimistic gaps remain the steeper branch. The damage magnitude does not replicate, and five pre-registered attempts to explain it are reported as they fell. We release the testbed, the training instrumentation, and 14 real public policies (Cedar, OPA, Gatekeeper) ported to executable oracles.
Figures & tables
| Verifier | Gap type | Reward gap | Selection error | Decision disagr. |
|---|---|---|---|---|
| Exact (control) | n/a | 0.000 | 0.000 | n/a |
| Outcome-only | metric | 0.629 | 0.504 | n/a |
| Sufficiency proxy | metric | 0.239 | 0.208 | n/a |
| Recall-only proxy | metric | 0.090 | 0.132 | n/a |
| Extraction noise | extraction | 0.051 | 0.043 | n/a |
| Extraction noise | extraction | 0.156 | 0.143 | n/a |
| Training verifier | True 0 | True 150 ( ) | Proxy | Gap | Starved | Cited | Words |
|---|---|---|---|---|---|---|---|
| Coarse taxonomy (payload) | 0.731 | 0.844 0.017 | 0.961 | 0.017 | 9/150 | 6.2 4.2 | 61 45 |
| Coarse taxonomy (s12) | 0.731 | 0.806 0.022 | 0.975 | 0.000 | 12/150 | 4.0 5.4 | 64 47 |
| Exact (control, s11) | 0.731 | 0.838 0.019 | 0.932 | 0.000 | 11/150 | 5.3 3.9 | 65 54 |
| Exact (control, s12) | 0.731 | 0.905 0.016 | 0.887 | 0.000 | 24/150 | 4.0 3.8 | 67 50 |
| Stale policy (s11) | 0.731 | 0.875 0.016 | 0.619 | -0.314 | 16/150 | 5.0 3.3 | 62 41 |
| Stale policy (s12) | 0.731 | 0.840 0.021 | 0.458 | -0.375 | 4/150 | 4.1 4.5 | 66 37 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Verifier | Gap type | Reward gap | Selection error | Decision disagr. |
|---|---|---|---|---|
| Exact (control) | n/a | 0.000 | 0.000 | n/a |
| Outcome-only | metric | 0.629 | 0.504 | n/a |
| Sufficiency proxy | metric | 0.239 | 0.208 | n/a |
| Recall-only proxy | metric | 0.090 | 0.132 | n/a |
| Extraction noise | extraction | 0.051 | 0.043 | n/a |
| Extraction noise | extraction | 0.156 | 0.143 | n/a |
| Policy | Source | Features | Rules | Block rate | Split | |
|---|---|---|---|---|---|---|
| cedar-photoapp | cedar-examples | 8 | 384 | 5 | 0.43 | train |
| cedar-streaming-service | cedar-examples | 10 | 3,456 | 7 | 0.73 | heldout |
| cedar-tags-n-roles | cedar-examples | 5 | 162 | 2 | 0.80 | train |
| cedar-tax-preparer | cedar-examples | 7 | 128 | 4 | 0.81 | heldout |
| gk-block-endpoint-edit | gatekeeper-library | 5 | 112 | 2 | 0.11 | train |
| gk-container-limits | gatekeeper-library | 5 | 192 | 8 | 0.48 | heldout |
| Hyperparameter | Final recipe | Initial recipe |
|---|---|---|
| Base model | Qwen3-4B-4bit (MLX) | same |
| LoRA rank / scale / dropout | 8 / 20 / 0.0 | same |
| Layers adapted | 16 | same |
| Optimizer | Adam | Adam |
| Learning rate | ||
| Gradient clipping (global norm) | 1.0 | none |
| Component | Volume | Wall-clock | haiku calls |
|---|---|---|---|
| Definitive grid (final recipe, excl. judge cell) | 14 runs 150 steps | 17.4 h | n/a |
| LLM-judge cell (final recipe) | 1 run 150 steps | 16.2 h (API-bound) | 1,496 |
| Coverage-null control (single gate) | 1 run 150 steps | 1.2 h | n/a |
| Pilot grid (recipe selection) | 6 runs 50 steps | 2.9 h | n/a |
| Initial-recipe runs (Appendix E ) | 8 runs 150 steps | 6.9 h | n/a |
| Transfer evaluation (§ 8 ) | 6 variants 80 episodes | h | 7,580 |
| Cell | Phenotype | Sel. err. | Signed gap | Eff. upd. | Damage | True 150 | |
|---|---|---|---|---|---|---|---|
| Sufficiency proxy (s11) | hacked | 0.209 | +0.185 | 0.547 | 0.114 | +0.204 | 0.667 |
| Sufficiency proxy (s12) | hacked | 0.209 | +0.185 | 0.660 | 0.138 | +0.232 | 0.639 |
| Outcome-only | starved | 0.453 | +0.163 | 0.087 | 0.039 | +0.158 | 0.714 |
| Noise | diluted | 0.080 | -0.027 | 1.000 | 0.080 | +0.051 | 0.821 |
| Noise | diluted | 0.192 | -0.060 | 1.000 | 0.192 | +0.193 | 0.678 |
| Noise | diluted | 0.351 | -0.117 | 1.000 | 0.351 | +0.230 | 0.641 |
| Held-out policy | Base | SFT | PG-exact (s11) | PG-exact (s12) | PG-suff. (hacked) | PG-single-gate | |
|---|---|---|---|---|---|---|---|
| cedar-streaming-service | 16 | 0.755 | 0.796 | 0.916 | 0.687 | 0.748 | 0.723 |
| cedar-tax-preparer | 16 | 0.823 | 0.640 | 0.531 | 0.740 | 0.768 | 0.710 |
| gk-container-limits | 16 | 0.623 | 0.663 | 0.690 | 0.644 | 0.767 | 0.734 |
| gk-psp-seccomp | 16 | 0.829 | 0.775 | 0.731 | 0.717 | 0.817 | 0.671 |
| opa-loadbalancer | 16 | 0.610 | 0.856 | 0.833 | 0.938 | 0.794 | 0.913 |
| All episodes | 80 | 0.728 | 0.746 | 0.740 | 0.745 | 0.779 | 0.750 |
| Model | Precision [95% CI] | Recall [95% CI] | [95% CI] |
|---|---|---|---|
| Base (no training) | 0.717 [0.63, 0.79] | 0.800 [0.72, 0.88] | 0.728 [0.65, 0.80] |
| SFT | 0.701 [0.63, 0.76] | 0.880 [0.82, 0.93] | 0.746 [0.69, 0.80] |
| PG-exact (s11) | 0.754 [0.68, 0.82] | 0.829 [0.77, 0.89] | 0.740 [0.68, 0.79] |
| PG-exact (s12) | 0.750 [0.67, 0.82] | 0.804 [0.73, 0.87] | 0.745 [0.68, 0.81] |
| PG-sufficiency (hacked) | 0.702 [0.64, 0.76] | 0.965 [0.94, 0.99] | 0.779 [0.73, 0.82] |
| PG-exact, single gate | 0.786 [0.71, 0.85] | 0.780 [0.71, 0.84] | 0.750 [0.69, 0.81] |
| ID | What it required | What happened | Verdict |
| The first pre-registration | |||
| P5 | Sufficiency damage above stale-policy damage at every initialisation | Ordering holds in every configuration | held |
| unnamed | Diluted damage monotone in | +0.024, +0.018, +0.071; the first two differ by less than their combined error | not met strictly, met within noise |
| Headroom | |||
| P2 | Damage rises with policy competence as a dose-response | +0.078 over two seeds, inside the standard-SFT band | refuted |
| P3 | More optimisation from a weak start produces the hack | -0.011 0.037 at 300 steps, though inflation grew to +1.38 | refuted |
| Setup | Adapter | Control | Sufficiency damage | Cite infl. | Opt. gap |
|---|---|---|---|---|---|
| MLX, Qwen3-4B, 4-bit (2 seeds) | 7.3M | 0.871 | +0.218 0.018 | +4.29 | +0.310 |
| PyTorch fp16, Qwen3-4B | 1.3M | 0.758 | +0.084 0.029 | +0.84 | +0.174 |
| PyTorch fp16, Qwen3-4B, long SFT, seed 11 | 1.3M | 0.791 | +0.061 0.024 | +1.45 | +0.205 |
| PyTorch fp16, Qwen3-4B, long SFT, seed 12 | 1.3M | 0.819 | +0.096 0.024 | +1.73 | +0.199 |
| PyTorch fp16, Qwen3-4B, matched adapter, seed 11 | 7.3M | 0.797 | +0.083 0.028 | +2.15 | +0.193 |
| PyTorch fp16, Qwen3-4B, matched adapter, seed 12 | 7.3M | 0.812 | +0.093 0.027 | +2.01 | +0.192 |
| Cell | Audit sign | Skipped | Predicted | Observed |
|---|---|---|---|---|
| Sufficiency proxy | optimistic | 27/150 | +0.311 | +0.084 0.029 |
| Outcome-only | optimistic | 79/150 | +0.389 | +0.033 0.028 |
| Stale policy | pessimistic | 0/150 | +0.051 | -0.004 0.028 |
| Coarse taxonomy | optimistic | 0/150 | +0.065 | +0.040 0.031 |
| Cell | Damage, seed 11 | Damage, seed 12 | Within | ||
|---|---|---|---|---|---|
| Suff | +0.083 | +0.093 | 0.011 | 0.078 | yes |
| Outcome | +0.019 | +0.094 | 0.075 | 0.082 | yes |
| Fam | -0.006 | -0.013 | 0.008 | 0.077 | yes |
| Coarse | +0.010 | +0.019 | 0.009 | 0.073 | yes |
| NoiseLo | +0.024 | +0.001 | 0.023 | 0.079 | yes |
| NoiseMid | +0.018 | +0.012 | 0.007 | 0.074 | yes |
| Training verifier | True 150 | Proxy 150 | Damage | Cite infl. |
|---|---|---|---|---|
| Sufficiency proxy | 0.752 | 0.929 | +0.033 0.043 | +0.21 |
| Outcome-only | 0.759 | 1.000 | +0.026 0.044 | -0.04 |
| Extraction noise | 0.777 | 0.706 | +0.008 0.041 | +0.05 |
| Stale policy | 0.768 | 0.611 | +0.017 0.043 | -0.06 |
Reward Hacking
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
See all 200 papers