We demonstrate that RL finetuning an LLM using debate, a two-player adversarial game between a generator and a critic adjudicated by a weaker LLM judge, reduces reward hacking compared to a reinforcement learning from AI feedback (RLAIF) baseline. Reward hacking is a central obstacle in RLAIF: as training progresses, the policy learns to exploit systematic errors in its AI judge, degrading task performance, a problem that worsens precisely when the judge is weaker than the policy, the setting most relevant to overseeing increasingly capable AI systems. We study mathematics tasks, where final-answer correctness is verifiable, allowing us to measure reward hacking dynamics. We train a Gemini2.5 Flash-class policy with a frozen, weaker Gemini2.5 Flash Lite judge, comparing a single-player RLAIF baseline against debate. While the baseline quickly hacks the judge, debate maintains judge performance throughout training, leading to a higher peak validation accuracy (45% performance gap recovered) that persists through many RL steps. Additional experiments show that: 1) further weakening the judge leads to faster hacking, but this can be compensated by adding an additional debate round; 2) debate incentives override prompted misalignment; 3) RL using an LLM judge has a smaller train/validation reward gap than RL from verifiable rewards; 4) learning to critique to convince the judge using ground truth labels is possible but slow. Taken together, our results are a positive update on the feasibility of debate, while highlighting that balancing multi-agent training is critical: without player constraints, adversarial training risks defaulting to critic judge-hacking. We show that critique word limits (effective up to 150 words) successfully balance the game and avoid judge hacking, though this introduces a trade-off by restricting critic expressive clarity.
Figures & tables
Figure 1: Debate reduces reward hacking. The single-player generator baseline, RLAIF- A , exhibits increasing reward (left), but decreasing judge correlation to ground-truth labels (Matthews correlation coefficient, MCC, centre) and decreasing generator solution final answer accuracy (right). Our method, Debate- AB (generator with an adversarial critic), moderates reward increase and maintains judge MCC, leading to a higher peak accuracy that persists through many RL steps. RLVR shown as roofline. All metrics reported on held-out validation data.
Figure 2: Protocol variants. Alice (trainable) Bob (trainable) Frozen judge Ground-truth verifier - -> Reward. Solid borders denote trainable nodes; dotted borders denote frozen or non-trainable nodes. Alice’s reward arrows (left, r ) and Bob’s (right, 1−r ) reflect the constant-sum structure.
Figure 3: Reward, Judge MCC and generator accuracy, as in Figure 1 , but including both validation (solid) and train (dashed) datasets.
Group
P(Best)
Peak Acc
95% CI LB
Obs Mean
Debate- AB
0.9987
0.7474
0.7369
0.7475
RLAIF- A
0.0013
0.7263
0.7174
0.7263
RLVR
-
0.7730
0.7585
0.7731
Table 1: Validation peak comparison from amalgamated experiment groups
Figure 4: Cross-play win-rate comparing earlier and later checkpoints. Heatmap with colour indicating Alice’s win-rate where we vary Alice (columns) and Bob (row) RL step.
Figure 5: Weaker Judge: reward (left), judge MCC (centre), and accuracy (right) comparing A , AB , and ABA with a weakened judge.
Figure 6: Prompted Misalignment of players: reward (left), judge MCC (centre), and accuracy (right). Misalignment prompt (solid) compared with standard prompt (dotted). Top row: standard judge. Bottom row: weakened judge. Hat notation in legend indicates a misaligned prompt given to the player.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Smaller dataset: reward (left), judge MCC (centre), and accuracy (right). Training on 20% of the data.
Figure 8: Pairwise preference reward hacks faster than a judgment-matched baseline.
Figure 9: Step by Step: reward (left), judge MCC (centre), and accuracy (right). Step-by-step formatting initially has worse accuracy improvement and eventually hacks more severely than the baseline.
Figure 10: Ground Truth: Bob reward when trained with ground truth labels. B-judge: rewards from acting as a judge to classify the Alice solution as correct/not. B-convince: rewards from convincing the judge to classify correctly.
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems. We introduce the Reward Hacking Benchmark (RHB), a suite of multi-step tasks requiring sequential tool operations with naturalistic shortcut opportunities such as skipping verification steps, inferring answers from task-adjacent metadata, or tampering with evaluation-relevant functions. RHB supports independent and chained task regimes, where chain length acts as a proxy for longer-horizon agent behavior. We evaluate 13 frontier models from OpenAI, Anthropic, Google, and DeepSeek. Exploit rates range from 0% (Claude Sonnet 4.5) to 13.9% (DeepSeek-R1-Zero), varying sharply by post-training style. A controlled sibling comparison (DeepSeek-V3 vs. DeepSeek-R1-Zero) shows RL post-training is associated with substantially higher reward hacking (0.6% vs. 13.9%), with consistent gaps across all four task families. We identify six exploit categories and find that 72% of reward hacking episodes include explicit chain-of-thought rationale, suggesting models often frame exploits as legitimate problem-solving. Simple environmental hardening reduces exploit rates by 5.7 percentage points (87.7% relative) without degrading task success. Models with near-zero exploit rates on standard tasks show elevated rates on harder variants, suggesting that production-aligned post-training appears to suppress reward hacking only below a complexity threshold where honest solutions remain tractable.
Reinforcement Learning from Human Feedback (RLHF) remains vulnerable to reward hacking, where models exploit spurious correlations in learned reward models to achieve high scores while violating human intent. Existing mitigations rely on static defenses that cannot adapt to novel exploitation strategies. We propose Adversarial Reward Auditing (ARA), a framework that reconceptualizes reward hacking as a dynamic, competitive game. ARA operates in two stages: first, a Hacker policy discovers reward model vulnerabilities while an Auditor learns to detect exploitation from latent representations; second, Auditor-Guided RLHF (AG-RLHF) gates reward signals to penalize detected hacking, transforming reward hacking from an unobservable failure into a measurable, controllable signal. Experiments across three hacking scenarios demonstrate that ARA achieves the best alignment-utility tradeoff among all baselines: reducing sycophancy to near-SFT levels while improving helpfulness, decreasing verbosity while achieving the highest ROUGE-L, and suppressing code gaming while improving Pass@1. Beyond single-domain evaluation, we show that reward hacking, detection, and mitigation all generalize across domains -- a Hacker trained on code gaming exhibits increased sycophancy despite no reward for this behavior, and an Auditor trained on one domain effectively suppresses exploitation in others, enabling efficient multi-domain defense with a single model.
Mohammad Beigi, Ming Jin, Junshan Zhang +2
Department of Computer Science, University of California, Davis, USA · Virginia Tech, Blacksburg, USA · Meta AI Research.
Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training.