Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation

Johanna Angulo; Víctor Yeste; Hector Espinos-Morato

cs.CRAug 29, 2026

Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation

Authors: Johanna Angulo, Víctor Yeste, Hector Espinos-Morato

Organizations: Universidad Europea de Madrid Madrid, Spain · Universidad Europea de Valencia Valencia, Spain

Abstract

A benchmark score is a joint property of the model, the evaluation harness, the elicitation budget, the sampled population, and contamination status. Leaderboards publish the model and the score, so capability and leakage stay observationally equivalent. Existing taxonomies classify contamination for automated detection, not the question a reporter faces at publication: given the mitigations already applied, which validity threats remain open? We introduce a taxonomy organized by the mitigation each type defeats -- direct, derivative, temporal, distributional, and acquired -- spanning training-time and evaluation-time leakage. Holding out a private test set closes the first alone. The fifth is acquired during the evaluation itself; because it is a property of one run, it must be recorded with the reported score rather than with the benchmark release. We operationalize it as a four-field disclosure protocol in which "unknown" is a valid entry, released under CC BY 4.0 with a JSON Schema, a validator, and worked examples. Two coders external to the design team applied a pre-registered instrument to 41 documents. Per-variable linear-weighted $κ$ runs from 0.00 to 0.35 (median 0.21) over 29 main-pass documents against a single-coder test-retest ceiling of 0.84, collapsing under the class skew the registration anticipated; pooling raises it to 0.46 through chance correction rather than better agreement. Two variables fall below the prevalence-robust threshold registered in advance: strata reporting and the acquired type introduced here. Disagreement concentrates on when a variable applies rather than on what a document states. Elicitation budgets are reported in 13% of documents, and no document addresses all five types. The contribution is the taxonomy, the score-side artifact that follows from it, and a pre-registered measurement of instrument reliability and current disclosure.

Figures & tables

Artifact	Documents	Contamination treated as	El.	St.	Rg.	M/R	T5
Model cards, datasheets ( Mitchell et al., 2019 ; Gebru et al., 2021 )	a model or dataset	out of scope	–	$\circ$	–	–	–
Contamination Transparency Card ( Nourbakhsh et al., 2026 )	a benchmark at release	four tiers of overlap form	–	–	$\circ$	–	–
STREAM ( McCaslin et al., 2025 )	a score at report	out of scope	$\bullet$	–	–	$\circ$	–
Evaluation Cards ( Ghosh et al., 2026 )	a score at report	one schema line, before execution	$\circ$	$\bullet$	–	$\bullet$	–
This form (§ 4 )	a score at report	five types by defeated mitigation	$\bullet$ *	$\bullet$	$\bullet$	$\bullet$	$\bullet$

Table 1: The nearest artifacts against the four fields: elicitation, strata, regeneration, machine-readability, Type 5.

\bullet

covered,

\circ

partial, – absent. * by adopting STREAM rather than restating it. The last column is not a coverage gap: Type 5 is a property of a run (§ 3.2.1 ), which no release-side artifact can close.

Panel of Figure 1 — Figure 1: The five types and the paths they travel; the arrow direction is the distinction that matters. At training time four paths run into the system. At evaluation time one runs outward , across the isolation boundary, to an answer key already in the environment (5a), reachable on the web (5b), or reachable only once the boundary is defeated (5c).

Type	Unit	Mitigation it defeats	Documented as, and where
1 Direct	The item	— (the one holding out fixes)	exact / verbatim / label-level contamination ( Magar and Schwartz, 2022 ; Xu et al., 2024 ; Deng et al., 2024 ; Palavalli et al., 2024 )
2 Derivative	The source document	Held-out private test sets	indirect, semantic or rephrase contamination ( Yang et al., 2023 ; Balloccu et al., 2024 ; Fu et al., 2024 )
3 Temporal	The resolved outcome	Overlap checks of every kind	task contamination; temporal analysis ( Li and Flanigan, 2024 ; Zhang et al., 2025 )
4 Distributional	The pattern	All overlap-based methods	in-distribution or soft contamination ( Tu et al., 2024 ; Dekoninck et al., 2024 ; Spiesberger et al., 2026 )
5 Acquired	The evaluation run	At 5a/5b, held-out sets, decontamination and cutoff reasoning alike, once the answer is reachable; at 5c, network isolation itself	search-time; solution contamination ( Han et al., 2025 ; NIST Center for AI Standards and Innovation, 2025 ; Wang et al., 2026b ; Marchand et al., 2026 )

Table 2: The five types, organised by the mitigation each defeats, with the prior source for each. The last column is the paper’s own novelty audit: every type is documented somewhere, across three literatures that largely do not cite one another, and row 5’s sources are disjoint from rows 1–4’s.

Panel of Figure 2 — Figure 2: An abridged disclosure record adapted from worked examples in the formal specification ( Angulo, 2026a ) (scores omitted). Declaring unknown and not_controlled shows what the instrument is for: recording a limitation costs the reporter little and tells the reader more than an omitted field. The validator flags this record, because a controlled claim for Type 5 without boundary monitoring is an assumption rather than a control.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Variable	Coded 2 when the document states …
F1 Strata	per-stratum scores for a named stratification of the focal evaluation
F2 Elicitation	an identified harness—named, pinned to a public version, or fully specified as a scaffold—plus at least two of version, budget, attempts, attempt resolution
F3 t1 Direct	overlap or decontamination checking, canary strings, a genuinely held-out set
F3 t2 Derivative	attention to whether the source material is public; provenance tracking
F3 t3 Temporal	a cutoff related to item dates ; temporal splitting; post-cutoff construction
F3 t4 Distributional	perturbation or paraphrase robustness; score distributions across variants

Appendix

Table 3: The eight coded variables. Full definitions, worked positive and negative examples, and the search terms used before any 0 is recorded are in the released manual.

Panel of Figure 3 — Figure 3: The instrument end to end: frame, focal-evaluation rule, coding design, and the statistics score.py computes from the two sheets.

	Agreement (two coders, main pass)					Disclosure (adjudicated)
Variable	$n$	Raw	$\kappa_{w}$	95% CI	AC2	2	1 + 2	$k$
F1 Strata	29	45%	0.18	[-0.12, 0.47]	0.43	39%	58%	26
F2 Elicitation	29	48%	0.10	[-0.10, 0.32]	0.63	13%	92%	26
F3 t1 Direct	29	69%	0.33	[ - 0.02, 0.63]	0.68	5%	21%	26
F3 t2 Derivative	29	79%	0.00	[ - 0.00, 0.00]	0.89	0%	11%	26
F3 t3 Temporal	29	83%	0.31	[-0.06, 0.76]	0.88	8%	16%	26

Appendix

Table 4: Agreement and disclosure rates per coded variable.

\kappa_{w}

is linear-weighted Kappa with a 95% bootstrap interval; AC2 is Gwet’s prevalence-robust companion under the same weight matrix, reported because

\kappa

collapses under high class skew. Two variables—F1 and t5—fall below the registered

0.6

AC2 threshold. 2 is the rate of full reporting, 1 + 2 of any partial or full reporting;

k

is the number of contributing clusters. The agreement columns exclude the pilot documents (§ 5 ); pilot-inclusive agreement figures are in the supplementary materials. The disclosure columns include them: the denominator is all

38

included documents, eight of which are pilot rows still coded at v1.4. † Not pre-registered, reported for completeness. The pooled row exceeds every per-variable

\kappa_{w}

through chance correction, not higher agreement: observed weighted agreement is

0.759

whether pooled or averaged over the eight variables, while expected agreement falls from

0.693

0.554

once variables with different skews are mixed. The per-variable column is the study’s agreement figure.

Disclosure, 2 fully reported ( 1 + 2 mentioned at all)
	A System cards	B NeurIPS D&B	C Third-party
	( $n$ =12)	( $n$ =19)	( $n$ =7)
F1 Strata	0% (17%)	63% (89%)	43% (43%)
F2 Elicitation	0% (100%)	21% (95%)	14% (71%)
F3 t1 Direct	0% (0%)	11% (37%)	0% (14%)
F3 t2 Derivative	0% (0%)	0% (21%)	0% (0%)

Appendix

Table 5: Disclosure and agreement by stratum. The two axes are independent: system cards are the easiest documents to code and the least forthcoming, third-party reports the hardest to code without being the most forthcoming, which benchmark papers are. The sub-element means rest on the

12

19

and

5

documents whose f2_notes record parses. Rates are adjudicated; NA is excluded per field. Only t5 has NA cells, so its row rests on

9

5

and

7

documents rather than on the column

n

; every other row uses the full column.

Scale	Raw	$\kappa_{w}$	AC2
Registered three-level ( 2 / 1 / 0 )	0.65	0.46	0.66
Collapsed: any mention ( 1 / 2 ) vs absent	0.73	0.56	0.79
Collapsed: fully reported ( 2 ) vs not	0.78	0.49	0.85
NA folded into 0	0.68	0.36	0.72

Appendix

Table 6: Scale-collapse ablation, main pass, 232 cells. Exploratory; not pre-registered. Merging 1 with 2 —dropping the partial-versus-full distinction—raises

\kappa_{w}

from 0.46 to

0.56

, while merging 0 with 1 raises it only to

0.49

. Residual disagreement therefore sits at the partial-versus-full boundary—how much is enough—which the manual states in prose and should state mechanically. F2 already records five sub-elements separately so its threshold is recomputable; extending that design to the remaining variables is the next version of the instrument.

Benchmark Contamination

Benchmark contamination is the presence of evaluation data or cues in training or development, which can inflate scores and weaken claims of generalization.

See all 49 papers

Aug 27, 2026

CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds

Public benchmark scores may reflect skill, prior exposure to the questions, or both, and for most models the training data are unknown. We present CleanScore, a black-box audit using scored outputs only. Each benchmark question becomes a parent item with one public form and two independently written fresh forms preserving its numbers, facts and answer. The audit reports an interval for the public-form advantage rather than a verdict, and a private negative-control bank with an explicit transport radius separates exposure from ordinary form mismatch. A registered controlled-exposure experiment detects planted exposure and stays quiet under fresh-form exposure. A registered audit of five open models on 200 GSM8K and 200 ARC-Challenge items finds no exposure-consistent advantage, bounding surface-form inflation below five points. Registered positive controls then bound what such a null can mean. Leaking an item raises accuracy on paraphrases the model never saw almost as much as on the leaked wording, leaving 52% to 110% of the effect invisible to a paraphrase audit. On ARC a planted 49-point advantage shows an observable gap of -0.020, and about 20 points survive rewriting stem and options, across four training seeds. A surface-form null bounds far less than the phrase contamination audit implies.

Jeffery Opoku, David Banahene

School of Mathematical and Statistical Sciences The University of Texas Rio Grande Valley, Edinburg, Texas, USA · Department of Biostatistics, Robert Stempel College of Public Health and Social Work Florida International University, Miami, Florida, USA

Benchmark Auditing Black-Box Model Auditing Data Contamination in Language Models Benchmark Contamination Benchmark Validity

Ask PDF

Oct 5, 2026

Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation

Agentic leaderboards increasingly evaluate systems on public benchmarks whose task statements and solution-bearing artifacts can remain accessible. We propose a measurement-first audit framework that distinguishes contamination claims according to the evidence required to support them. It separates three channels that require different evidence: training-time exposure, evaluation-time retrieval, and pipeline/scaffold leakage. Each channel is coded as open, partial, closed, or unknown under a fail-closed rule. Across nine Holistic Agent Leaderboard (HAL) configurations, none of the 27 channel assessments was coded closed, but incidents were confirmed in four configurations. We then apply the behavioral component of the framework to a reported file-localization gap on SWE-bench Verified, using an outcome-blind, same-repository matched-control design with symmetric prompt-leakage screening, paired and repository-aware uncertainty analyses, and scorer validation, evaluated on GPT-4.1 and DeepSeek-V4-Flash. Among the 100 pairs retained after symmetric screening and the pair-integrity exclusion, GPT-4.1 showed a

+10.0

-point pair-weighted Top-3 benchmark-associated gap, but the 95% intervals from both the prespecified paired-bootstrap procedure and the post-hoc repository-balanced analysis included zero, leaving the benchmark-associated gap inconclusive. The reproduction scorer did not pass its validation gate: against consensus human labels, sufficient scorer sensitivity could not be established for either model, and both DeepSeek-V4-Flash firings on correct-gold comparisons were false positives. Without provenance evidence, appropriate controls, symmetric leakage screening, and validated scorers, stronger contamination claims are not warranted. The results do not establish training-data membership, contamination prevalence, or benchmark-induced score inflation.

Dishu Yang, Qi Su, Hongbo Qin +1

Khoury College of Computer Sciences Northeastern University San Jose, CA, USA · School of Engineering and Applied Science The George Washington University Bellevue, WA, USA · Khoury College of Computer Sciences Northeastern University Boston, MA, USA +1

Benchmark Auditing AI Agent Auditing Benchmark Contamination Agent Evaluation

Ask PDF

Aug 8, 2026

When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits

Behavioral contamination detectors can return "no evidence" either because a benchmark is clean or because the audit has little power. We formalize this distinction for a benchmark in which an unknown fraction alpha of items was seen during training. With matched clean and seen controls, the behavioral channel is the sparse mixture Q_alpha = (1 - alpha) P_0 + alpha P_1, and an exact second-moment argument shows that detectability is governed by alpha * rho * sqrt(m), where rho^2 = chi^2(P_1 || P_0) measures behavioral separability. Any scalar detector reduces to its efficacy, ef = |E_1 f - E_0 f| / sqrt(Var_0(f)) <= rho, which can be estimated from controls before the audit is run. A separate sample-split certificate lower-bounds alpha distribution-free, without requiring an orientation assumption. Our empirical finding is two-sided. Frozen calibration efficacy predicts held-out power curves, with R^2 = 0.83-0.98 across six exact-permutation channels, but the efficacy-only Gaussian budget is miscalibrated at the small sample sizes it prescribes, failing in 9/9 gate-passing channels even though efficacy itself transports. The failure is in the inversion, not the calibration. A predeclared two-stage planner that simulates the deployed test repairs the budgets, is uniformly conservative, and abstains when its probe does not transport. The certificate is valid but vacuous at audit scale, and a five-seed paired injection study recovers the mechanism ordering verbatim > paraphrase > surface, in which the apparent answer-only signal is explained by baseline drift. We report the audit contract and its failures together: a non-rejection is interpretable only alongside the efficacy, budget, and validity gates that produced it.

Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

Department of Computer Science, Iowa State University · Department of Civil, Construction & Environmental Engineering, Iowa State University

Statistical Power Analysis Benchmark Auditing Benchmark Contamination

Ask PDF

Abstract

Benchmark Contamination

Explore similar work

CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds

Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation

When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits