Benchmark contamination is not new, but recent model releases show it in a sharper form. We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing, which optimizes toward a benchmark without seeing its test set. We document two escalations over the past month. First, in post-training, the traces have become explicit: recent audits show an agent stating hidden test requirements it was never given in 23% of its attempts, where its predecessor did so in none, and continuing to follow them after the task is changed to contradict them (Schaeffer and Räuker, 2026; Lo and Lo, 2026). Second, the problem has reached pre-training. Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input. For the recently released ERNIE-5.5-Flash-Base, 96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base. To quantify what a reported score owes to such examples, we define an Anti-Hack score: the benchmark score recomputed with every leaked example marked wrong and the number of examples unchanged. Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base. Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.
Figures & tables
Figure 1: Left: given five solved examples and an MMLU abstract-algebra test question, ERNIE-5.5-Flash-Base answers it, then writes out the next test questions in order, with options and correct answers (highlighted); none appears in the input. IDs show zero-based positions in the subject’s test set. The headline percentages refer to ERNIE-5.5-Flash-Base. Right: benchmark averages before and after leaked examples receive zero credit (Anti-Hack). DeepSeek-V4-Flash-Base and Hy3-preview-Base lose 1.5 and 1.3 points; MiMo-V2.5-Base and ERNIE-5.5-Flash-Base lose 5.0 and 16.6.
Benchmarking
Benchmaxxing
Benchleaking
Relation to the benchmark
Evaluation on held-out examples
Benchmark-targeted optimization
Test examples enter training
Test examples in training
No
No; same-distribution data
Yes: questions, reference answers, or hidden tests
Diagnostic evidence
Scores hold on fresh or perturbed variants
High benchmark scores that do not transfer to perturbed or freshly written variants
Exact reproduction of test examples absent from the input
Table 1: Benchmarking, benchmaxxing, and benchleaking, ordered by increasing exposure to test examples.
Model
Original Tasks DeepSWE (89 tasks)
Modified Tasks MaxxBench (89 tasks)
Drop (pp)
DeepSeek-V4.1-Flash
75.8%
62.5%
13.3
Kimi-K3.1-Pro
83.8%
68.0%
15.8
Table 2: Pooled pass@1 on 89 shared tasks under the same local harness. Only trials with a native verifier score are counted. DeepSWE uses four trials per task and MaxxBench two. Drops are in percentage points.
Setting
Kimi-K3.1-Pro
Kimi-K3
While solving
14.3% (9/63)
1.6% (1/63)
Prefilled reasoning
51.5% (34/66)
9.1% (6/66)
Table 3: Tasks with hidden-requirement. Counts come from an automatic semantic judge. Normal solving covers 63 CPU tasks and prefilled reasoning covers 66 tasks.
Benchmark
DeepSeek-V4-Flash Base
Hy3-preview Base
MiMo-V2.5 Base
ERNIE-5.5-Flash Base
TriviaQA
0.3%
<0.1%
0.3%
<0.1%
HellaSwag
0.0%
0.0%
0.0%
0.0%
ARC-Challenge
0.0%
0.0%
0.0%
2.7%
ARC-Easy
0.0%
0.0%
0.0%
<0.1%
OpenBookQA
0.0%
0.0%
0.0%
85.4%
PIQA
0.0%
0.0%
0.0%
0.0%
Table 4: How much of each test set leaks? Percentage of distinct leaked test examples. One to five symbols indicate increasing leakage severity, corresponding to coverage of [5,20)% , [20,40)% , [40,60)% , [60,80)% , and [80,100]% , respectively. Coverage below 5% is left unmarked. † WinoGrande matches can arise from its twin-sentence construction; they are shown without leakage markers (Appendix A ).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Question (L1)
Additional fields (L2)
Answer (L3)
TriviaQA
Full question
None
Exact answer or stored alias
HellaSwag
Full context
Activity label; all endings
Gold ending
ARC-Challenge
Full question
All choice texts
Gold choice text
ARC-Easy
Full question
All choice texts
Gold choice text
OpenBookQA
Full question stem
All choice texts
Gold choice text
PIQA
Full goal
Both solutions
Gold solution text
Appendix
Table 5: Matching requirements for all 20 benchmarks. Every field must match after case and whitespace normalization. L2 includes L1; L3 includes L1 and L2.
Question
Additional fields
Answer
L1
✓
L2
✓
✓
L3
✓
✓
✓
Appendix
Table 6: Percentage of distinct test examples leaked under the L1, L2, and L3 criteria; a tick marks a field the model must reproduce at that level. Each example counts once. Increasingly darker shading indicates coverage of [5,20)% , [20,40)% , [40,60)% , [60,80)% , and [80,100]% . † WinoGrande matches can arise from its twin-sentence construction and are not evidence of leakage (Appendix A ).
Benchmark
DeepSeek-V4-Flash Base
Hy3-preview Base
MiMo-V2.5 Base
ERNIE-5.5-Flash Base
Anti-Hack
Original
Anti-Hack
Original
Anti-Hack
Original
Anti-Hack
Original
TriviaQA
85.72
85.97
85.51
85.59
83.98
84.30
88.90
88.92
HellaSwag
93.16
94.97
93.39
95.92
91.67
94.16
94.27
97.49
ARC-Challenge
96.16
96.16
96.42
96.50
95.65
95.65
92.66
98.04
ARC-Easy
99.12
99.12
98.99
99.03
98.53
98.61
99.07
99.12
OpenBookQA
95.80
95.80
94.00
94.80
88.20
88.40
11.00
98.60
Appendix
Table 7: Scores (%) under L1 matching. Each model has Anti-Hack on the left and Original on the right. Matched targets receive zero credit with denominators unchanged. An Anti-Hack cell is shaded when it falls at least 5 points below Original, in five steps at 5, 20, 40, 60 and 80 points lost, so the darkest cells are the scores that rest most on leaked examples. Code maxima are recomputed after adjustment; the overall average uses the 20 representative entries, not every variant row.
Benchmark
DeepSeek-V4-Flash Base
Hy3-preview Base
MiMo-V2.5 Base
ERNIE-5.5-Flash Base
Anti-Hack
Original
Anti-Hack
Original
Anti-Hack
Original
Anti-Hack
Original
TriviaQA
85.72
85.97
85.51
85.59
83.98
84.30
88.90
88.92
HellaSwag
94.97
94.97
95.92
95.92
94.16
94.16
97.49
97.49
ARC-Challenge
96.16
96.16
96.50
96.50
95.65
95.65
95.31
98.04
ARC-Easy
99.12
99.12
99.03
99.03
98.61
98.61
99.07
99.12
OpenBookQA
95.80
95.80
94.80
94.80
88.40
88.40
14.20
98.60
Appendix
Table 8: Scores (%) under L2 matching. Each model has Anti-Hack on the left and Original on the right. Matched targets receive zero credit with denominators unchanged. An Anti-Hack cell is shaded when it falls at least 5 points below Original, in five steps at 5, 20, 40, 60 and 80 points lost, so the darkest cells are the scores that rest most on leaked examples. Code maxima are recomputed after adjustment; the overall average uses the 20 representative entries, not every variant row.
Benchmark
DeepSeek-V4-Flash Base
Hy3-preview Base
MiMo-V2.5 Base
ERNIE-5.5-Flash Base
Anti-Hack
Original
Anti-Hack
Original
Anti-Hack
Original
Anti-Hack
Original
TriviaQA
85.72
85.97
85.51
85.59
84.00
84.30
88.90
88.92
HellaSwag
94.97
94.97
95.92
95.92
94.16
94.16
97.49
97.49
ARC-Challenge
96.16
96.16
96.50
96.50
95.65
95.65
95.31
98.04
ARC-Easy
99.12
99.12
99.03
99.03
98.61
98.61
99.07
99.12
OpenBookQA
95.80
95.80
94.80
94.80
88.40
88.40
14.20
98.60
Appendix
Table 9: Scores (%) under L3 matching. Each model has Anti-Hack on the left and Original on the right. Matched targets receive zero credit with denominators unchanged. An Anti-Hack cell is shaded when it falls at least 5 points below Original, in five steps at 5, 20, 40, 60 and 80 points lost, so the darkest cells are the scores that rest most on leaked examples. Code maxima are recomputed after adjustment; the overall average uses the 20 representative entries, not every variant row.
Benchmark
Dataset
Task
Items
Commonsense
TriviaQA
mandarjoshi/trivia_qa (rc.nocontext)
Factual short-answer questions without evidence context
17,944
HellaSwag
Rowan/hellaswag
Selecting plausible continuations of everyday events
Table 10: The 20 benchmarks. Items are the test examples evaluated. All 448 GPQA-Main examples are used. Dataset identifiers name the versions loaded by our evaluation harness.
Benchmark
Prompt form
Shots
Budget
Answer form in the demonstrations
TriviaQA
Direct answer
5
1,024
Answer: followed by the answer string
HellaSwag
Multiple choice
5
1,024
the answer is (X)
ARC-Challenge
Multiple choice
5
1,024
the answer is (X)
ARC-Easy
Multiple choice
5
1,024
the answer is (X)
OpenBookQA
Multiple choice
5
1,024
the answer is (X)
PIQA
Multiple choice
5
1,024
the answer is (X)
Appendix
Table 11: Continuation-run settings. Budget is max_gen_toks , the only thing that ends generation once stopping is removed. Decoding is greedy at temperature 0 with one sample per question, except HumanEval and HumanEval+, which draw four samples at temperature 0.2 and top- p0.95 .
Benchmark
Shots
Metric
Decoding
Samples
Budget
TriviaQA
5
Exact match (any alias)
Greedy
1
256
HellaSwag
5
Exact match
Greedy
1
2,048
ARC-Challenge
5
Exact match
Greedy
1
2,048
ARC-Easy
5
Exact match
Greedy
1
2,048
OpenBookQA
5
Exact match
Greedy
1
2,048
PIQA
5
Exact match
Greedy
1
2,048
Appendix
Table 12: Standard-run settings. Greedy decoding is temperature 0 . Budget is max_gen_toks . Avg@4 averages the test-passing outcome over the four sampled completions of each problem and then over problems. MATH-500 is scored as correct if any of exact match, symbolic equivalence, or a normalized string comparison accepts the extracted answer.
Benchmark contamination is the presence of evaluation data or cues in training or development, which can inflate scores and weaken claims of generalization.
Khoury College of Computer Sciences Northeastern University San Jose, CA, USA · School of Engineering and Applied Science The George Washington University Bellevue, WA, USA · Khoury College of Computer Sciences Northeastern University Boston, MA, USA +1