cs.LGNov 16, 2026

Optimistic Verifiers Get Hacked: Reward Hacking Under an Exact Process Oracle

Authors: Arunkumar V, Manoranjan Gandhudi, Gangadharan G. R., Arun Prakash, S. Senthilkumar

Organizations: University College of Engineering, BIT Campus, Anna University, Tiruchirappalli, India · Central University of Karnataka, Kalaburagi, India · National Institute of Technology Tiruchirappalli, India · Jawaharlal Nehru University, New Delhi, India

Abstract

Reinforcement learning from verifiable rewards inherits whatever its verifier gets wrong. Outside mathematics and code the verifier is a proxy, a rubric, a learned judge or a hand-written checker, and its distance from the true objective is never measured, so a rising reward cannot be told apart from a hack. We build a setting where the true reward is exactly computable at the process level: executable authorization policies over 124,416 contexts admit an oracle that lists every minimal sufficient reason for every decision. We compile these policies into exact process rewards, degrade the verifier along seven realistic axes (outcome-only, sufficiency, recall-only, extraction noise, coarse taxonomy, stale policy, LLM judge), and train against each while the exact verifier scores every rollout. Four phenotypes appear. Optimistic verifiers are hacked by padding: the sufficiency proxy ends at 0.940 while true faithfulness falls from 0.731 to 0.667 on 96 held-out episodes. Outcome-only verifiers saturate and stop teaching, noise dilutes, and ranking-preserving gaps do no harm. A model-free audit of the verifier, its selection error and gap sign, together with a short saturation pilot, anticipates which phenotype a run will produce. Across nine configurations spanning two frameworks, two model vendors, two adapter sizes, three precisions, two initialisations and two seeds, the phenotypes recur, sufficiency damage is positive in all 10 replication cells and beyond two standard errors in 7, and optimistic gaps remain the steeper branch. The damage magnitude does not replicate, and five pre-registered attempts to explain it are reported as they fell. We release the testbed, the training instrumentation, and 14 real public policies (Cedar, OPA, Gatekeeper) ported to executable oracles.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Reward Hacking in Rubric-Based Reinforcement Learning

    May 12, 2026Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang +3Reward HackingReinforcement Learning

  2. Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control

    Sep 28, 2026Christian Moya, Elliott Thornley, Guang LinReward HackingReinforcement Learning

  3. Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO

    Oct 26, 2026Shurun Li, Michael Hua Wang, Haibo Zeng +1Reward HackingReinforcement Learning with Verifiable Rewards