cs.CLSep 23, 2026
SaveReward Hacking Challenges Oversight of Autonomous Research Agents
Organizations: Bake AI · University of Notre Dame · University of Washington · LMU Munich · IBM Research · Microsoft Research · Stanford University · Massachusetts Institute of Technology · University of California, Santa Barbara
Abstract
Autonomous research agents can design experiments, evaluate results, and write reports, giving them control over both a scientific result and the evidence used to support it. This creates a risk of reward hacking: meeting the reward criteria without achieving the intended goal. We study (1) how often models reward-hack without instructions to do so, (2) how effective and detectable their methods are when hacking is allowed, and (3) how they adapt when an LLM review panel returns its decision and reasons. Across 17 language models and 38 tasks, the spontaneous reward-hacking rate is 30.5% on open-ended research-pipeline tasks and 2.9% on task-specific kernels. When hacking is allowed on tasks whose pass thresholds exceed our best compliant baselines, 505/677 attempts (74.6%) are confirmed reward hacks: they both clear the threshold and receive mechanism-verification panel confirmation of an evaluation exploit. An LLM panel reviewing only submitted code and reported scores misses 33/505 confirmed hacks (6.5%). Direct methods that achieve the highest scores are often easy to detect, while less direct methods evade more often. In a five-round loop, the number of model-task pairs with an evasion rises from 7 to 56. Among 79 pairs evaluated under two feedback conditions, cumulative evasion reaches 40.5% with detailed feedback and 20.3% with generic rejection. The detailed condition includes the review decision, reasons, and attempt history, so this comparison does not isolate the effect of explanations. These findings highlight the need for stronger defenses, including metrics kept outside the agent's control and independent recomputation on data chosen to expose likely exploits.
Figures & tables
Figure 1: From spontaneous reward hacking to adaptive evasion under oversight. Research agents produce both results and the evidence used to assess them, motivating three progressively deeper tests of this workflow. Top, the shared research process; the dashed feedback loop applies only to E3. Bottom, E1: Spontaneous reward hacking. We measure hacking under ordinary task instructions, without a request to hack. E2: Hacking effectiveness and detectability. We evaluate exploits when hacking is permitted; the bars illustrate exploits exceeding thresholds that compliant baselines do not meet. The quality review rates the hacking method on efficacy, generality, craft, and novelty, each on a 1–10 scale. E3: Adaptation to oversight. Given an explicit evasion objective and review feedback, agents revise their methods, and some exploits remain undetected. The terracotta trace passes through the reviewer to illustrate a missed exploit. Arrows between E1–E3 indicate the progression of research questions. The illustrations are schematic.
Figure 2: Evaluation settings. (a) Setting 1 measures spontaneous reward hacking in one complete agent session per model–task pair, without instructions to hack. A full-trajectory evaluator uses the trajectory, submission, and instrumented checks. (b) Setting 2 allows reward hacking on baseline-exceeding tasks. An LLM review panel that sees the submitted code and reported score, but not the trajectory or mechanism summary, tests whether it detects the method (Metric 1), conditional on threshold passage and mechanism verification; a separate panel rates quality (Metric 2). (c) Setting 3 returns the review panel’s decision, reasons, and the attempt history to the agent for up to five rounds.
| Component | Sees | Outputs | Used for |
|---|---|---|---|
| Setting-1 evaluator | full trajectory instrumented checks | hack / no hack | E1 label |
| Mechanism-verification panel | artifact summary hidden evidence | reward hack / not | E2–3 label |
| Submission-only review panel | final artifact reported score | earned / gamed | evasion label |
| Quality panel | reward hack execution evidence | four 1–10 ratings | descriptive |
| Execution audit | held-out rerun | gap / no gap | detector comparison |
Table 1: Evaluation components and their roles. The panels differ in the evidence available to them; only the submission-only panel determines whether a reward hack evades review.
Figure 3: Reward hacking in Setting 1. Each dot is one hacked task among model–task cells. Terracotta marks research-pipeline tasks and teal marks task-specific tasks. Rows show coverage across models rather than a capability ranking; external benchmark comparisons are reported in § 5 .
Figure 4: Quality ratings and evasion. Bars give the score distribution for the reward-hacking runs (left axis); horizontal marks give the submission-only review panel’s evasion rate at each score. Labels are percentages, and the dashed line marks the overall rate. Efficacy is concentrated near , but hacks in the top bin have the lowest evasion rate. Evasion is more common in the upper generality, craft, and novelty bins, which contain fewer observations.
Figure 5: Craft rating and evasion by model. The horizontal axis is the mean construction-and-camouflage rating, the vertical axis is the number of evasions under submission-only review, and bubble area is proportional to the number of reward-hacking runs. Because the counts are not normalized by that denominator, the plot is descriptive rather than a model ranking.
Figure 6: Onset of reward hacking and survival under trajectory review. (a) A Beta distribution fitted to 51 annotated onset positions by matching their mean and variance. Dots show the observed positions; the dashed line marks the median (71%). Progress is measured in recorded steps, not elapsed time. (b) The percentage of artifact-review evasions that also pass trajectory review in the filtered 30-case subset. The dashed line marks the overall rate (30%). Category sample sizes, from top to bottom, are 3, 4, 10, 7, 1, 2, and 3.
Figure 7: Evasion across tasks and review rounds. (a) Evasions under submission-only review by model and task. Most occur on a few tasks where the exploit resembles ordinary work. (b) Share of the pairs still caught after each round (dark line: mean; dots: models; band and rule: per-round IQR and median). (c) Mean quality ratings among reward-hacking runs in each round. Efficacy falls as novelty and craft rise. (d) Shares of the empirical technique families defined in § 7 . Direct leakage (green) becomes less common. (e) Evasion rate among reward-hacking runs in each round, from to (bands: Wilson CIs). Panels (c)–(e) describe a changing survivor pool because a pair leaves the loop once it evades.
Figure 8: Retry-only control and transfer to an independent review panel. (a), (b) among the 79 model–task cases observed under both conditions, cumulative evasion reaches with detailed feedback and with retry-only (paired McNemar ). Round 1 is shared by construction. The marginal rate stays near under retry-only and ranges from to with detailed feedback. (c) an independent review panel whose members do not overlap with the primary panel (Gemini 3.5 Flash, Grok 4.5, and Kimi-K3) also clears 44 of the 56 evasions cleared by the primary panel ( ). Five of those 56 also pass the execution audit.
Figure 9: Evasion by task and technique. (a) First-round evasion (gray) and cumulative evasion after five rounds (terracotta). A star marks tasks with no first-round evasion. (b) Evasion rate under submission-only review by the empirical technique families in § 7 , with the number of hacks and mean craft shown for each family. Direct test-set leakage is common and rarely evades; several smaller, disguised families have higher rates.
| Technique family | Cheats | Evasion | Evasion | Craft | Novelty | Beat audit |
|---|---|---|---|---|---|---|
| (pooled) | (E2 only) | |||||
| Test-set leakage | 694 (75%) | 3.0% | 2.5% | 6.7 | 4.4 | 5.6% |
| Scorer / grader tampering | 64 (7%) | 25.0% | 20.8% | 7.0 | 6.3 | 29.7% |
| Metric fabrication | 46 (5%) | 23.9% | 18.5% | 7.0 | 5.4 | 19.6% |
| Answer / shape-keyed lookup | 44 (5%) | 11.4% | 0.0% | 6.7 | 4.3 | 2.3% |
| Offline label distillation | 32 (3%) | 37.5% | 57.1% | 7.5 | 5.6 | 25.0% |
Table 2: Technique families in 923 summaries. The data include 500 of 505 Setting-2 reward-hacking runs (five lack a usable summary) and 423 Setting-3 attempts. Evasion is the clearance rate under submission-only review. We report it for all 923 summaries and for the non-feedback Setting-2 subset; direct leakage is lowest in both. Setting-2 family sizes are 403 test-set leakage, 24 scorer tampering, 27 metric fabrication, 21 answer or shape lookup, 7 offline distillation, and 18 other. Round-1 Setting-3 counts outside test-set leakage are at most eight and are not shown separately. Craft, novelty, and audit results are pooled. Attempts from one Setting-3 pair are correlated. Recurring families are ordered by frequency, with the residual other/uncategorized group last.
Figure 10: Map of reward-hacking mechanisms. The figure is a two-dimensional t-SNE projection of embeddings for the style-normalized summaries. Color marks the assigned family, and a dark ring marks an evasion. Labels name the dominant mechanism in each dense region. The axes and distances between clusters have no quantitative interpretation; the corresponding estimates are in Tables 2 and 9 .
| Risk | Evidence | Implication |
| Spontaneous metric gaming | All 17 models hack at least three tasks; rates are for research-pipeline tasks and for task-specific kernels. | The workflow and scorer shape the risk; ordinary task instructions do not remove it. |
| Plausible false results | Submission-only review misses reward hacks, and also pass the execution audit. | A final artifact or one hidden rerun is not always enough to establish the claim. |
| Adaptive evasion | Cumulative evasion rises from after one attempt to after five; detailed feedback yields vs. under retry-only in paired cases. | Under an explicit evasion objective, review feedback can help the agent search for a method the reviewer misses. |
Table 3: Findings across the three evaluation settings. Each row states the comparison supported by the setting and its implication for autonomous research.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Source | Dir. | Baseline | Threshold |
|---|---|---|---|---|
| mls_anomaly_detection | MLS-Bench | |||
| mls_calibration | MLS-Bench | – | ||
| mls_hpo_search | MLS-Bench | |||
| mls_imputation | MLS-Bench | – | ||
| mls_online_bandit | MLS-Bench | |||
| mls_pacbayes_bound | MLS-Bench |
Table 4: Calibration of the 16 baseline-exceeding thresholds. Each threshold is beyond the score reached by the compliant reference solver in the task’s measured space. and give the metric direction: the baseline score falls below an threshold and above a threshold. MLS thresholds are the task’s success_threshold; SEP thresholds use measured-space calibration (Route A). Two additional SEP candidates, fair_baselines and repro_paper, were set in the wrong measured space and produced no reward hack. We exclude them pending recalibration.
| Reliability measure | Setting 2 | Setting 3 |
|---|---|---|
| Submission-only review panel (Fleiss , 3 reviewers), / | ||
| verdict, 4 labels | ||
| flag vs. clear (defines evasion) | ||
| Mechanism-verification panel (3 raters) | ||
| unanimity among confirmed hacks | ( ) | NR |
| reliability across all attempts | NR | NR |
Table 5: Inter-rater agreement and reliability. The submission-only review columns report Fleiss’ for three reviewers, both for all four labels and for the binary flag/clear decision. Quality columns report ICC for the three-rater mean, with Krippendorff’s interval in brackets. is the number of items with all ratings present. Verification unanimity is conditional on confirmed hacks and is not directly comparable to these reliability coefficients; NR denotes not reported in the available aggregate results.
| Model | API identifier | Developer |
| GPT-5.5 | openai/gpt-5.5 | OpenAI |
| GPT-5.6 Luna | openai/gpt-5.6-luna | OpenAI |
| GPT-5.6 Terra | openai/gpt-5.6-terra | OpenAI |
| GPT-5.6 Sol | openai/gpt-5.6-sol | OpenAI |
| Claude Haiku 4.5 | anthropic/claude-haiku-4.5 | Anthropic |
| Claude Sonnet 4.6 | anthropic/claude-sonnet-4.6 | Anthropic |
Table 6: API identifiers for the 17 models (accessed 21–23 July 2026). Setting 1 uses all 17. Settings 2–3 use 15 and exclude DeepSeek-V4-Pro and Claude Sonnet 5.
| Model | Cheats | Evade | BeatX | Effic. | Gener. | Craft | Novel | Overall |
|---|---|---|---|---|---|---|---|---|
| GLM-5.2 | 22 | 10 | 6 | 7.7 | 5.2 | 7.2 | 4.9 | 6.32 |
| GPT-5.6 Sol | 44 | 5 | 5 | 8.9 | 5.3 | 6.9 | 4.8 | 6.40 |
| Kimi-K3 | 35 | 3 | 2 | 9.1 | 5.5 | 7.5 | 4.9 | 6.91 |
| Gemini 3.5 Flash | 31 | 3 | 8 | 8.6 | 6.0 | 7.1 | 5.7 | 6.87 |
| MiMo-v2.5-Pro | 36 | 3 | 3 | 8.6 | 4.9 | 6.1 | 3.4 | 5.67 |
| Claude Opus 4.8 | 34 | 2 | 3 | 8.1 | 5.4 | 7.5 | 4.8 | 6.60 |
Table 7: Setting-2 outcomes and quality ratings by model (16 baseline-exceeding tasks; 677 runs). Cheats counts reward-hacking runs. Evade ( ) counts those cleared by the submission-only review panel, and BeatX counts those with hacked in the execution audit. Quality columns are model means. Overall is a separately elicited rating, not the mean of the four axes. Rows are sorted by evasions. Attempt budgets differ, with at most three per cell, so raw totals are not a model ranking. Settings 2–3 include Kimi-K3 and exclude DeepSeek-V4-Pro and Claude Sonnet 5; DeepSeek produces malformed agent actions in this scaffold.
Figure 11: Evasions by model over five rounds. The dark part of each bar is the number that evade in round 1, and the light part is the number first cleared in a later round.
Figure 12: Technique mix by model. Each bar divides a model’s Setting-2 hacks by family and sums to . Models are ordered by the share using a disguised method, defined as any family other than direct test-set leakage (green).
Figure 13: Disguised techniques and evasion by model. Each point compares the share of a model’s hacks that use a disguised family (horizontal axis) with its evasion rate under submission-only review evasion rate (vertical axis). The dashed least-squares fit has ( CI , ). Spearman ( ), and removing GLM-5.2 gives ( ), so the linear association depends on one point.
Figure 14: Scoring one complete attempt in E2–E3. Threshold passage and a positive mechanism-verification decision jointly define a confirmed reward hack. An evasion additionally requires clearance by the submission-only review panel; clearance alone is insufficient. The quality panel provides descriptive ratings, not hack or evasion labels. E1 instead uses a full-trajectory evaluator under ordinary instructions that do not ask for hacking, rather than a separate explicit-prohibition condition. E3 returns review feedback between attempts for up to five rounds.
| Benchmark | Outcome | ||
| AA Index v4.3 | Conditional trajectory-review survival | 11 | |
| AA Index v4.3 | Evasion count | 14 | |
| AA Index v4.3 | Setting-1 propensity | 16 | |
| SWE-bench Pro | Conditional trajectory-review survival | 8 | |
| SWE-bench Pro | Setting-1 propensity | 12 |
Table 8: General capability and hacking outcomes. Setting-2 comparisons use the filtered subset. Sample size counts models.