Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models
Abstract
Benchmark contamination is not new, but recent model releases show it in a sharper form. We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing, which optimizes toward a benchmark without seeing its test set. We document two escalations over the past month. First, in post-training, the traces have become explicit: recent audits show an agent stating hidden test requirements it was never given in 23% of its attempts, where its predecessor did so in none, and continuing to follow them after the task is changed to contradict them (Schaeffer and Räuker, 2026; Lo and Lo, 2026). Second, the problem has reached pre-training. Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input. For the recently released ERNIE-5.5-Flash-Base, 96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base. To quantify what a reported score owes to such examples, we define an Anti-Hack score: the benchmark score recomputed with every leaked example marked wrong and the number of examples unchanged. Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base. Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.
Figures & tables
| Benchmarking | Benchmaxxing | Benchleaking | |
|---|---|---|---|
| Relation to the benchmark | Evaluation on held-out examples | Benchmark-targeted optimization | Test examples enter training |
| Test examples in training | No | No; same-distribution data | Yes: questions, reference answers, or hidden tests |
| Diagnostic evidence | Scores hold on fresh or perturbed variants | High benchmark scores that do not transfer to perturbed or freshly written variants | Exact reproduction of test examples absent from the input |
| Model | Original Tasks DeepSWE (89 tasks) | Modified Tasks MaxxBench (89 tasks) | Drop (pp) |
|---|---|---|---|
| DeepSeek-V4.1-Flash | 75.8% | 62.5% | 13.3 |
| Kimi-K3.1-Pro | 83.8% | 68.0% | 15.8 |
| Setting | Kimi-K3.1-Pro | Kimi-K3 |
|---|---|---|
| While solving | 14.3% (9/63) | 1.6% (1/63) |
| Prefilled reasoning | 51.5% (34/66) | 9.1% (6/66) |
| Benchmark | DeepSeek-V4-Flash Base | Hy3-preview Base | MiMo-V2.5 Base | ERNIE-5.5-Flash Base |
|---|---|---|---|---|
| TriviaQA | 0.3% | 0.3% | ||
| HellaSwag | 0.0% | 0.0% | 0.0% | 0.0% |
| ARC-Challenge | 0.0% | 0.0% | 0.0% | 2.7% |
| ARC-Easy | 0.0% | 0.0% | 0.0% | |
| OpenBookQA | 0.0% | 0.0% | 0.0% | 85.4% |
| PIQA | 0.0% | 0.0% | 0.0% | 0.0% |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Question (L1) | Additional fields (L2) | Answer (L3) |
|---|---|---|---|
| TriviaQA | Full question | None | Exact answer or stored alias |
| HellaSwag | Full context | Activity label; all endings | Gold ending |
| ARC-Challenge | Full question | All choice texts | Gold choice text |
| ARC-Easy | Full question | All choice texts | Gold choice text |
| OpenBookQA | Full question stem | All choice texts | Gold choice text |
| PIQA | Full goal | Both solutions | Gold solution text |
| Question | Additional fields | Answer | |
| L1 | |||
| L2 | |||
| L3 |
| Benchmark | DeepSeek-V4-Flash Base | Hy3-preview Base | MiMo-V2.5 Base | ERNIE-5.5-Flash Base | ||||
|---|---|---|---|---|---|---|---|---|
| Anti-Hack | Original | Anti-Hack | Original | Anti-Hack | Original | Anti-Hack | Original | |
| TriviaQA | 85.72 | 85.97 | 85.51 | 85.59 | 83.98 | 84.30 | 88.90 | 88.92 |
| HellaSwag | 93.16 | 94.97 | 93.39 | 95.92 | 91.67 | 94.16 | 94.27 | 97.49 |
| ARC-Challenge | 96.16 | 96.16 | 96.42 | 96.50 | 95.65 | 95.65 | 92.66 | 98.04 |
| ARC-Easy | 99.12 | 99.12 | 98.99 | 99.03 | 98.53 | 98.61 | 99.07 | 99.12 |
| OpenBookQA | 95.80 | 95.80 | 94.00 | 94.80 | 88.20 | 88.40 | 11.00 | 98.60 |
| Benchmark | DeepSeek-V4-Flash Base | Hy3-preview Base | MiMo-V2.5 Base | ERNIE-5.5-Flash Base | ||||
|---|---|---|---|---|---|---|---|---|
| Anti-Hack | Original | Anti-Hack | Original | Anti-Hack | Original | Anti-Hack | Original | |
| TriviaQA | 85.72 | 85.97 | 85.51 | 85.59 | 83.98 | 84.30 | 88.90 | 88.92 |
| HellaSwag | 94.97 | 94.97 | 95.92 | 95.92 | 94.16 | 94.16 | 97.49 | 97.49 |
| ARC-Challenge | 96.16 | 96.16 | 96.50 | 96.50 | 95.65 | 95.65 | 95.31 | 98.04 |
| ARC-Easy | 99.12 | 99.12 | 99.03 | 99.03 | 98.61 | 98.61 | 99.07 | 99.12 |
| OpenBookQA | 95.80 | 95.80 | 94.80 | 94.80 | 88.40 | 88.40 | 14.20 | 98.60 |
| Benchmark | DeepSeek-V4-Flash Base | Hy3-preview Base | MiMo-V2.5 Base | ERNIE-5.5-Flash Base | ||||
|---|---|---|---|---|---|---|---|---|
| Anti-Hack | Original | Anti-Hack | Original | Anti-Hack | Original | Anti-Hack | Original | |
| TriviaQA | 85.72 | 85.97 | 85.51 | 85.59 | 84.00 | 84.30 | 88.90 | 88.92 |
| HellaSwag | 94.97 | 94.97 | 95.92 | 95.92 | 94.16 | 94.16 | 97.49 | 97.49 |
| ARC-Challenge | 96.16 | 96.16 | 96.50 | 96.50 | 95.65 | 95.65 | 95.31 | 98.04 |
| ARC-Easy | 99.12 | 99.12 | 99.03 | 99.03 | 98.61 | 98.61 | 99.07 | 99.12 |
| OpenBookQA | 95.80 | 95.80 | 94.80 | 94.80 | 88.40 | 88.40 | 14.20 | 98.60 |
| Benchmark | Dataset | Task | Items |
| Commonsense | |||
| TriviaQA | mandarjoshi/trivia_qa (rc.nocontext) | Factual short-answer questions without evidence context | 17,944 |
| HellaSwag | Rowan/hellaswag | Selecting plausible continuations of everyday events | 10,042 |
| ARC-Challenge | allenai/ai2_arc | School-level science questions requiring reasoning | 1,172 |
| ARC-Easy | allenai/ai2_arc | School-level science questions, easy split | 2,376 |
| OpenBookQA | allenai/openbookqa | Combining basic science facts with commonsense | 500 |
| Benchmark | Prompt form | Shots | Budget | Answer form in the demonstrations |
|---|---|---|---|---|
| TriviaQA | Direct answer | 5 | 1,024 | Answer: followed by the answer string |
| HellaSwag | Multiple choice | 5 | 1,024 | the answer is (X) |
| ARC-Challenge | Multiple choice | 5 | 1,024 | the answer is (X) |
| ARC-Easy | Multiple choice | 5 | 1,024 | the answer is (X) |
| OpenBookQA | Multiple choice | 5 | 1,024 | the answer is (X) |
| PIQA | Multiple choice | 5 | 1,024 | the answer is (X) |
| Benchmark | Shots | Metric | Decoding | Samples | Budget |
|---|---|---|---|---|---|
| TriviaQA | 5 | Exact match (any alias) | Greedy | 1 | 256 |
| HellaSwag | 5 | Exact match | Greedy | 1 | 2,048 |
| ARC-Challenge | 5 | Exact match | Greedy | 1 | 2,048 |
| ARC-Easy | 5 | Exact match | Greedy | 1 | 2,048 |
| OpenBookQA | 5 | Exact match | Greedy | 1 | 2,048 |
| PIQA | 5 | Exact match | Greedy | 1 | 2,048 |
Benchmark Contamination
Benchmark contamination is the presence of evaluation data or cues in training or development, which can inflate scores and weaken claims of generalization.
See all 58 papers