Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation

Johanna Angulo; Víctor Yeste; Hector Espinos-Morato

cs.CRAug 29, 2026

Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation

Authors: Johanna Angulo, Víctor Yeste, Hector Espinos-Morato

Organizations: Universidad Europea de Madrid Madrid, Spain · Universidad Europea de Valencia Valencia, Spain

Abstract

A benchmark score is a joint property of the model, the evaluation harness, the elicitation budget, the sampled population, and contamination status. Leaderboards publish the model and the score, so capability and leakage stay observationally equivalent. Existing taxonomies classify contamination for automated detection, not the question a reporter faces at publication: given the mitigations already applied, which validity threats remain open? We introduce a taxonomy organized by the mitigation each type defeats -- direct, derivative, temporal, distributional, and acquired -- spanning training-time and evaluation-time leakage. Holding out a private test set closes the first alone. The fifth is acquired during the evaluation itself; because it is a property of one run, it must be recorded with the reported score rather than with the benchmark release. We operationalize it as a four-field disclosure protocol in which "unknown" is a valid entry, released under CC BY 4.0 with a JSON Schema, a validator, and worked examples. Two coders external to the design team applied a pre-registered instrument to 41 documents. Per-variable linear-weighted $κ$ runs from 0.00 to 0.35 (median 0.21) over 29 main-pass documents against a single-coder test-retest ceiling of 0.84, collapsing under the class skew the registration anticipated; pooling raises it to 0.46 through chance correction rather than better agreement. Two variables fall below the prevalence-robust threshold registered in advance: strata reporting and the acquired type introduced here. Disagreement concentrates on when a variable applies rather than on what a document states. Elicitation budgets are reported in 13% of documents, and no document addresses all five types. The contribution is the taxonomy, the score-side artifact that follows from it, and a pre-registered measurement of instrument reliability and current disclosure.

Figures & tables

Artifact	Documents	Contamination treated as	El.	St.	Rg.	M/R	T5
Model cards, datasheets ( Mitchell et al., 2019 ; Gebru et al., 2021 )	a model or dataset	out of scope	–	$\circ$	–	–	–
Contamination Transparency Card ( Nourbakhsh et al., 2026 )	a benchmark at release	four tiers of overlap form	–	–	$\circ$	–	–
STREAM ( McCaslin et al., 2025 )	a score at report	out of scope	$\bullet$	–	–	$\circ$	–
Evaluation Cards ( Ghosh et al., 2026 )	a score at report	one schema line, before execution	$\circ$	$\bullet$	–	$\bullet$	–
This form (§ 4 )	a score at report	five types by defeated mitigation	$\bullet$ *	$\bullet$	$\bullet$	$\bullet$	$\bullet$

Table 1: The nearest artifacts against the four fields: elicitation, strata, regeneration, machine-readability, Type 5.

\bullet

covered,

\circ

partial, – absent. * by adopting STREAM rather than restating it. The last column is not a coverage gap: Type 5 is a property of a run (§ 3.2.1 ), which no release-side artifact can close.

Panel of Figure 1 — Figure 1: The five types and the paths they travel; the arrow direction is the distinction that matters. At training time four paths run into the system. At evaluation time one runs outward , across the isolation boundary, to an answer key already in the environment (5a), reachable on the web (5b), or reachable only once the boundary is defeated (5c).

Type	Unit	Mitigation it defeats	Documented as, and where
1 Direct	The item	— (the one holding out fixes)	exact / verbatim / label-level contamination ( Magar and Schwartz, 2022 ; Xu et al., 2024 ; Deng et al., 2024 ; Palavalli et al., 2024 )
2 Derivative	The source document	Held-out private test sets	indirect, semantic or rephrase contamination ( Yang et al., 2023 ; Balloccu et al., 2024 ; Fu et al., 2024 )
3 Temporal	The resolved outcome	Overlap checks of every kind	task contamination; temporal analysis ( Li and Flanigan, 2024 ; Zhang et al., 2025 )
4 Distributional	The pattern	All overlap-based methods	in-distribution or soft contamination ( Tu et al., 2024 ; Dekoninck et al., 2024 ; Spiesberger et al., 2026 )
5 Acquired	The evaluation run	At 5a/5b, held-out sets, decontamination and cutoff reasoning alike, once the answer is reachable; at 5c, network isolation itself	search-time; solution contamination ( Han et al., 2025 ; NIST Center for AI Standards and Innovation, 2025 ; Wang et al., 2026b ; Marchand et al., 2026 )

Table 2: The five types, organised by the mitigation each defeats, with the prior source for each. The last column is the paper’s own novelty audit: every type is documented somewhere, across three literatures that largely do not cite one another, and row 5’s sources are disjoint from rows 1–4’s.

Panel of Figure 2 — Figure 2: An abridged disclosure record adapted from worked examples in the formal specification ( Angulo, 2026a ) (scores omitted). Declaring unknown and not_controlled shows what the instrument is for: recording a limitation costs the reporter little and tells the reader more than an omitted field. The validator flags this record, because a controlled claim for Type 5 without boundary monitoring is an assumption rather than a control.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Variable	Coded 2 when the document states …
F1 Strata	per-stratum scores for a named stratification of the focal evaluation
F2 Elicitation	an identified harness—named, pinned to a public version, or fully specified as a scaffold—plus at least two of version, budget, attempts, attempt resolution
F3 t1 Direct	overlap or decontamination checking, canary strings, a genuinely held-out set
F3 t2 Derivative	attention to whether the source material is public; provenance tracking
F3 t3 Temporal	a cutoff related to item dates ; temporal splitting; post-cutoff construction
F3 t4 Distributional	perturbation or paraphrase robustness; score distributions across variants

Appendix

Table 3: The eight coded variables. Full definitions, worked positive and negative examples, and the search terms used before any 0 is recorded are in the released manual.

Panel of Figure 3 — Figure 3: The instrument end to end: frame, focal-evaluation rule, coding design, and the statistics score.py computes from the two sheets.

	Agreement (two coders, main pass)					Disclosure (adjudicated)
Variable	$n$	Raw	$\kappa_{w}$	95% CI	AC2	2	1 + 2	$k$
F1 Strata	29	45%	0.18	[-0.12, 0.47]	0.43	39%	58%	26
F2 Elicitation	29	48%	0.10	[-0.10, 0.32]	0.63	13%	92%	26
F3 t1 Direct	29	69%	0.33	[ - 0.02, 0.63]	0.68	5%	21%	26
F3 t2 Derivative	29	79%	0.00	[ - 0.00, 0.00]	0.89	0%	11%	26
F3 t3 Temporal	29	83%	0.31	[-0.06, 0.76]	0.88	8%	16%	26

Appendix

Table 4: Agreement and disclosure rates per coded variable.

\kappa_{w}

is linear-weighted Kappa with a 95% bootstrap interval; AC2 is Gwet’s prevalence-robust companion under the same weight matrix, reported because

\kappa

collapses under high class skew. Two variables—F1 and t5—fall below the registered

0.6

AC2 threshold. 2 is the rate of full reporting, 1 + 2 of any partial or full reporting;

k

is the number of contributing clusters. The agreement columns exclude the pilot documents (§ 5 ); pilot-inclusive agreement figures are in the supplementary materials. The disclosure columns include them: the denominator is all

38

included documents, eight of which are pilot rows still coded at v1.4. † Not pre-registered, reported for completeness. The pooled row exceeds every per-variable

\kappa_{w}

through chance correction, not higher agreement: observed weighted agreement is

0.759

whether pooled or averaged over the eight variables, while expected agreement falls from

0.693

0.554

once variables with different skews are mixed. The per-variable column is the study’s agreement figure.

Disclosure, 2 fully reported ( 1 + 2 mentioned at all)
	A System cards	B NeurIPS D&B	C Third-party
	( $n$ =12)	( $n$ =19)	( $n$ =7)
F1 Strata	0% (17%)	63% (89%)	43% (43%)
F2 Elicitation	0% (100%)	21% (95%)	14% (71%)
F3 t1 Direct	0% (0%)	11% (37%)	0% (14%)
F3 t2 Derivative	0% (0%)	0% (21%)	0% (0%)

Appendix

Table 5: Disclosure and agreement by stratum. The two axes are independent: system cards are the easiest documents to code and the least forthcoming, third-party reports the hardest to code without being the most forthcoming, which benchmark papers are. The sub-element means rest on the

12

19

and

5

documents whose f2_notes record parses. Rates are adjudicated; NA is excluded per field. Only t5 has NA cells, so its row rests on

9

5

and

7

documents rather than on the column

n

; every other row uses the full column.

Scale	Raw	$\kappa_{w}$	AC2
Registered three-level ( 2 / 1 / 0 )	0.65	0.46	0.66
Collapsed: any mention ( 1 / 2 ) vs absent	0.73	0.56	0.79
Collapsed: fully reported ( 2 ) vs not	0.78	0.49	0.85
NA folded into 0	0.68	0.36	0.72

Appendix

Table 6: Scale-collapse ablation, main pass, 232 cells. Exploratory; not pre-registered. Merging 1 with 2 —dropping the partial-versus-full distinction—raises

\kappa_{w}

from 0.46 to

0.56

, while merging 0 with 1 raises it only to

0.49

. Residual disagreement therefore sits at the partial-versus-full boundary—how much is enough—which the manual states in prose and should state mechanically. F2 already records five sub-elements separately so its threshold is recomputable; extending that design to the remaining variables is the next version of the instrument.

Benchmark Contamination

Benchmark contamination is the presence of evaluation data or cues in training or development, which can inflate scores and weaken claims of generalization.

See all 49 papers

CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds
Aug 27, 2026Jeffery Opoku, David BanaheneBenchmark Auditing Black-Box Model Auditing
School of Mathematical and Statistical Sciences The University of Texas Rio Grande Valley, Edinburg, Texas, USA · Department of Biostatistics, Robert Stempel College of Public Health and Social Work Florida International University, Miami, Florida, USA
Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation
Oct 5, 2026Dishu Yang, Qi Su, Hongbo Qin +1Benchmark Auditing AI Agent Auditing
Khoury College of Computer Sciences Northeastern University San Jose, CA, USA · School of Engineering and Applied Science The George Washington University Bellevue, WA, USA · Khoury College of Computer Sciences Northeastern University Boston, MA, USA +1
When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits
Aug 8, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaStatistical Power Analysis Benchmark Auditing
Department of Computer Science, Iowa State University · Department of Civil, Construction & Environmental Engineering, Iowa State University

Aug 27, 2026J/K move · Enter open · S save

CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds

Jeffery Opoku, David Banahene

School of Mathematical and Statistical Sciences The University of Texas Rio Grande Valley, Edinburg, Texas, USA · Department of Biostatistics, Robert Stempel College of Public Health and Social Work Florida International University, Miami, Florida, USA

Benchmark Auditing Black-Box Model Auditing Data Contamination in Language Models Benchmark Contamination Benchmark Validity

Open paper AskPDF arXiv

Public benchmark scores may reflect skill, prior exposure to the questions, or both, and for most models the training data are unknown. We present CleanScore, a black-box audit using scored outputs only. Each benchmark question becomes a parent item with one public form and two independently written fresh forms preserving its numbers, facts and answer. The audit reports an interval for the public-form advantage rather than a verdict, and a private negative-control bank with an explicit transport radius separates exposure from ordinary form mismatch. A registered controlled-exposure experiment detects planted exposure and stays quiet under fresh-form exposure. A registered audit of five open models on 200 GSM8K and 200 ARC-Challenge items finds no exposure-consistent advantage, bounding surface-form inflation below five points. Registered positive controls then bound what such a null can mean. Leaking an item raises accuracy on paraphrases the model never saw almost as much as on the leaked wording, leaving 52% to 110% of the effect invisible to a paraphrase audit. On ARC a planted 49-point advantage shows an observable gap of -0.020, and about 20 points survive rewriting stem and options, across four training seeds. A surface-form null bounds far less than the phrase contamination audit implies.

Abstract

Benchmark Contamination

Explore similar work

CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds

Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation

When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits