Reward Hacking

Reward hacking occurs when a system optimizes a proxy—such as a reward model, judge, rubric, or benchmark verifier—without achieving the intended goal. Research studies this in settings including reinforcement learning from human feedback, verifiable-reward language-model training, tool-using agents, and image or robot generation. Common approaches include building controlled tests and adversarial audits, hardening verifiers, using uncertainty-aware or multiple reward signals, and monitoring behavior through outputs or internal activations. A central challenge is that optimization can expose new weaknesses or adapt around defenses, so high scores and low detector readings do not necessarily establish genuine task success or safety.