CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds

Jeffery Opoku; David Banahene

cs.LGAug 27, 2026

CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds

Authors: Jeffery Opoku, David Banahene

Organizations: School of Mathematical and Statistical Sciences The University of Texas Rio Grande Valley, Edinburg, Texas, USA · Department of Biostatistics, Robert Stempel College of Public Health and Social Work Florida International University, Miami, Florida, USA

Abstract

Public benchmark scores may reflect skill, prior exposure to the questions, or both, and for most models the training data are unknown. We present CleanScore, a black-box audit using scored outputs only. Each benchmark question becomes a parent item with one public form and two independently written fresh forms preserving its numbers, facts and answer. The audit reports an interval for the public-form advantage rather than a verdict, and a private negative-control bank with an explicit transport radius separates exposure from ordinary form mismatch. A registered controlled-exposure experiment detects planted exposure and stays quiet under fresh-form exposure. A registered audit of five open models on 200 GSM8K and 200 ARC-Challenge items finds no exposure-consistent advantage, bounding surface-form inflation below five points. Registered positive controls then bound what such a null can mean. Leaking an item raises accuracy on paraphrases the model never saw almost as much as on the leaked wording, leaving 52% to 110% of the effect invisible to a paraphrase audit. On ARC a planted 49-point advantage shows an observable gap of -0.020, and about 20 points survive rewriting stem and options, across four training seeds. A surface-form null bounds far less than the phrase contamination audit implies.

Figures & tables

Figure 1: The finding of Section 7.11, stated once at the start because it bounds how much any result in this paper can mean. Leaking an item’s wording raises accuracy on that wording, and most of that gain carries over to paraphrases of the same item, where an audit that compares an original against a rewrite cannot see it. The estimated split is 77% invisible, with an interval running from 52% to 110%; at the top of that interval nothing is visible at all.

Figure 2: The matched-form audit. Each sampled parent item is put to the model in its public form and in two fresh forms that keep the same numbers, facts, and answer; the public-form advantage is the difference in mean correctness between them. A never-public control bank of items no model can have memorised is scored the same way and supplies the form-mismatch term. The registered claim rule requires both intervals to exclude zero, so ordinary form mismatch cannot on its own produce a contamination claim.

Family	Contrast	Mean	95% CI	Seeds positive	p (two-sided)
SmolLM2-360M	Exact exposure minus related-task	+0.020	[+0.006, +0.034]	9/10	0.010
SmolLM2-360M	Paraphrase exposure minus related-task	-0.002	[-0.009, +0.005]	3/10	0.61
Qwen2.5-0.5B	Exact exposure minus related-task	+0.043	[-0.018, +0.104]	7/10	0.14
Qwen2.5-0.5B	Paraphrase exposure minus related-task	-0.015	[-0.045, +0.016]	3/10	0.31

Table 1: Registered seed-paired contrasts in the controlled-exposure validation (ten seeds per family).

Figure 3: Registered seed-paired contrasts, ten seeds per family. Exact exposure to public forms shifts the public-form advantage upward in both families, decisively in SmolLM2-360M; exposure to fresh forms of the same items does not. The wide Qwen2.5-0.5B interval reflects that model’s larger run-to-run variation, not a weaker effect.

Design	Sample size	Hoeffding coverage	Hoeffding half-width	Wald coverage	Mean Wald half-width
Simple random sample	120	1.0000	0.2480	0.9476	0.0625
Simple random sample	400	1.0000	0.1358	0.9506	0.0321
Simple random sample	800	1.0000	0.0960	0.9522	0.0203
Simple random sample	1,200	1.0000	0.0784	0.9468	0.0144
Stratified sample	400	1.0000	0.1358	0.9488	0.0329

Table 2: Coverage and width of the two intervals in the finite-population calibration, 5,000 replications per design.

Figure 4: Interval precision and empirical coverage in the finite-population simulation. The conservative bound is valid but wide in these settings. The finite-population Wald interval is much sharper and stays close to the nominal 95% level.

Figure 5: Sensitivity to the transport radius when the true control-to-benchmark mismatch is 0.05. Understating the radius can cause severe undercoverage. A radius that covers the true mismatch restores protection but widens the interval.

Figure 6: Wald coverage for the 5,000-item benchmark settings. Results are generally close to 95%, but the wider stress grid shows that this approximation is not uniformly reliable. Small sparse audits are the important failure case.

Model	Benchmark	Public score	Fresh score	$\hat{\Delta}$	Half-width	Nonzero gaps	$b_{C}$ (control)
OLMo-2-7B	GSM8K	0.720	0.705	+0.015	0.045	58	+0.017 $\pm$ 0.081
OLMo-2-7B	ARC	0.625	0.608	+0.018	0.040	40	+0.013 $\pm$ 0.102
SmolLM2-1.7B	GSM8K	0.475	0.430	+0.045	0.058	83	0.000 $\pm$ 0.119
SmolLM2-1.7B	ARC	0.485	0.470	+0.015	0.192 (H)	26	-0.088 $\pm$ 0.110
Phi-3-mini	GSM8K	0.820	0.803	+0.018	0.035	35	+0.008 $\pm$ 0.049
Phi-3-mini	ARC	0.595	0.598	-0.003	0.192 (H)	27	0.000 $\pm$ 0.105

Table 3: Registered real-benchmark audit, all five models.

\hat{\Delta}

is the public-form advantage over 200 sampled parents. The half-width is the primary interval: the finite-population Wald interval, or the Hoeffding bound (marked H) where the nonzero-gap guardrail applies.

b_{C}

is the control-bank form gap with its Wald half-width. No model-benchmark pair meets the registered claim rule.

Figure 7: Public-form advantage with its primary interval for every model-benchmark pair, with the control-bank gap $b_{C}$ marked separately. Dotted intervals are the two pairs where the registered nonzero-gap guardrail makes the conservative bound primary. Every interval covers zero.

Model	Setting	Public score	Fresh score	$\hat{\Delta}$	Half-width	Truncated
Qwen2.5-7B	Registered (320 tokens)	0.710	0.715	-0.005	0.047	29.8%
Qwen2.5-7B	Amendment 1 (768 tokens)	0.930	0.938	-0.008	0.192 (H)	0.0%
Qwen2.5-1.5B	Registered (320 tokens)	0.475	0.500	-0.025	0.055	40.7%
Qwen2.5-1.5B	Amendment 1 (768 tokens)	0.695	0.735	-0.040	0.054	0.0%

Table 4: Amendment 1: the two Qwen models on GSM8K with a 768-token budget and a boxed-answer parser. Truncation falls to zero. ARC is unaffected by both changes and was reproduced bit-for-bit.

Reading of the control bank	Control half-width	Mean width of the $\tau$ interval	Pairs excluding zero
Wald (registered)	0.057 to 0.136	0.335	0 of 10
Hoeffding (own guardrail)	0.382 to 0.468	0.982	0 of 10
Fixed bank, no interval	0	0.131	1 of 10

Table 5: The sensitivity interval under three readings of the control bank, at the registered radius. Wald is the registered construction. Hoeffding applies the paper’s own small-sample guardrail to the control bank as well. Fixed treats the bank as a fully measured object, so

b_{C}

is a constant and the transport radius carries everything.

Interval	Coverage range	Settings below 0.95	Width vs Hoeffding at n = 200
Hoeffding	0.9945 to 1.0000	0	1.00 by definition
Empirical Bernstein	0.9998 to 1.0000	0	0.77 to 1.62, depending on the shape
Finite-population Wald	0.9390 to 0.9563	24	not distribution-free

Table 6: Coverage of the three intervals across the 36 stress settings (4,000 audits each), and the width of the Bernstein bound relative to the Hoeffding bound at the audit’s own sample size, n = 200 in a 1,000-item benchmark.

Model	Benchmark	Public score	Fresh score	Confidence, public	Confidence, fresh	PaCoST p	PaCoST flags	CleanScore claim
OLMo-2-7B	GSM8K	0.720	0.705	-0.098	-0.096	0.28	no	no
OLMo-2-7B	ARC	0.625	0.608	-3.19	-3.17	0.71	no	no
SmolLM2-1.7B	GSM8K	0.475	0.430	-0.156	-0.156	0.83	no	no
SmolLM2-1.7B	ARC	0.485	0.470	-3.74	-3.67	0.18	no	no
Phi-3-mini	GSM8K	0.820	0.803	-0.119	-0.119	0.94	no	no
Phi-3-mini	ARC	0.595	0.598	-3.15	-3.20	0.017	yes	no

Table 7: Baseline detectors on the audited items. PaCoST flags a pair when the paired confidence test rejects at 0.05 with higher confidence on the public form.

Benchmark	Pattern	Observed	Null mean	p
GSM8K	rewrites harder	4	3.37	0.45
GSM8K	rewrites easier	2	3.29	0.88
ARC	rewrites harder	3	0.70	0.02
ARC	rewrites easier	0	0.30	1.00

Table 8: Parents on which at least two of the five models show the pattern, against a permutation null with 20,000 draws.

Model	$\hat{\Delta}$ , all 200	$\hat{\Delta}$ , deep rewrites only	Shift	r (similarity, gap)	p
OLMo-2-7B	+0.015	+0.028	+0.013	+0.029	0.69
Phi-3-mini	+0.018	+0.017	-0.001	+0.023	0.75
Qwen2.5-1.5B	-0.025	-0.020	+0.005	-0.035	0.62
Qwen2.5-7B	-0.005	+0.003	+0.008	-0.072	0.31
SmolLM2-1.7B	+0.045	+0.050	+0.005	-0.004	0.96

Table 9: Sensitivity of the GSM8K audit to rewrite depth. The third column drops the 21 parents whose closest fresh form meets or exceeds the ARC dissimilarity threshold. The last column is the correlation, across all 200 parents, between a parent’s rewrite similarity and its measured gap.

Benchmark	Audited items	Mean half-width	Exact order	Pairs correct
GSM8K	25	0.142	0.39	0.92
GSM8K	50	0.101	0.50	0.95
GSM8K	100	0.070	0.62	0.97
GSM8K	150	0.056	0.73	0.98
ARC	25	0.093	0.15	0.78
ARC	50	0.067	0.25	0.86

Table 10: Budgeted-audit curves (mean over models and 2,000 subsamples). "Exact order" is recovery of the full five-model ordering; "pairs correct" is the share of the ten model pairs ordered as the full sample orders them.

Figure 8: Budgeted audits. Precision improves smoothly with audit size, and pairwise ordering is recovered cheaply, but exact recovery of the five-model ordering lags well behind because two model pairs differ by less than the audit’s own half-width.

Benchmark	Audited items	delta = 0.02	delta = 0.05	delta = 0.10	delta = 0.20
GSM8K	50	0.00	0.00	0.00	0.00
GSM8K	100	0.00	0.00	0.00	0.04
GSM8K	200	0.22	0.56	0.95	1.00
GSM8K	400	0.43	0.89	1.00	1.00
GSM8K	800	0.82	1.00	1.00	1.00
ARC	50	0.00	0.00	0.00	0.00

Table 11: Detection rate for a planted public-form advantage, 4,000 resampled audits per cell, pooling the five audited models within each benchmark.

Figure 9: Left: the primary half-width, which is the smallest advantage the audit would declare, for each model and benchmark at n = 200; open markers are the two pairs where the guardrail applies. Right: detection rate against audit size for planted advantages of two, five, ten, and twenty points. The flat zero region below 200 items is the guardrail.

Benchmark	Audited items	delta	Interval	Exact test
GSM8K	50	0.10	0.00	0.43
GSM8K	50	0.20	0.00	0.92
GSM8K	100	0.05	0.00	0.32
GSM8K	100	0.10	0.00	0.77
GSM8K	200	0.05	0.56	0.61
GSM8K	200	0.10	0.95	0.97

Table 12: Detection rate of the exact randomization test against the interval of Section 7.9, same planted advantages, same resampled banks, 2,000 audits per cell. The test’s simulated type-I error across all sizes and both benchmarks runs from 0.029 to 0.052 against a nominal 0.05.

Figure 10: Left and centre: detection rate against audit size for a five-point and a ten-point planted advantage, exact test against interval. Right: simulated type-I error of the exact test, obtained by relabelling the real correctness vectors so the null holds by construction; the test sits at or below the nominal level at every size.

Form gap in the data	Rejection rate at the 0.05 level
0.000 (exchangeable)	0.039
0.020	0.107
0.050	0.315
0.088 (largest measured)	0.689

Table 13: What ordinary form mismatch does to the exact test. No contamination is present in any row; every rejection is caused by the fresh forms being slightly harder than the public one. The largest form gap is the largest measured on any control bank in this audit.

Parents	Form	Before	After	Change
Exposed (n = 100)	public, trained on	0.460	0.610	+0.150
Exposed (n = 100)	fresh, never trained on	0.485	0.570	+0.085
Exposed (n = 100)	measurable gap	-0.025	+0.040	+0.065
Unexposed (n = 100)	public	0.490	0.400	-0.090
Unexposed (n = 100)	fresh	0.515	0.425	-0.090
Unexposed (n = 100)	measurable gap	-0.025	-0.025	+0.000

Table 14: Where the contamination effect went. Accuracy before and after exposure, on the hundred leaked parents and the hundred untouched ones. The measurable gap is the difference between the public and fresh columns, which is what any paraphrase audit observes.

Quantity	Exposed	Unexposed	Difference in differences	95% CI
Change in public-form accuracy	+0.150	-0.090	+0.240	[+0.080, +0.400]
Change in fresh-form accuracy	+0.085	-0.090	+0.175	[+0.035, +0.315]
Change in the measurable gap	+0.065	+0.000	+0.065	—

Table 15: Difference-in-differences estimates for the effect of leaking an item’s public wording, with bootstrap intervals over parents.

Model	Family	DiD public	DiD fresh	Visible gap
Qwen2.5-1.5B	Qwen	+0.240 [+0.080, +0.400]	+0.175 [+0.035, +0.315]	+0.065
SmolLM2-1.7B	SmolLM2	+0.100 [-0.050, +0.250]	+0.040 [-0.060, +0.140]	+0.060
OLMo-2-1B	OLMo	+0.500 [+0.340, +0.660]	+0.350 [+0.230, +0.470]	+0.150
StableLM-2-1.6B	StableLM	+0.200 [+0.050, +0.350]	+0.230 [+0.115, +0.340]	-0.030
Pooled		+0.260 [+0.183, +0.335]	+0.199 [+0.140, +0.258]	+0.061

Table 16: Four contaminated models from four families. Difference-in-differences against the unexposed control parents of the same model, with bootstrap intervals. The visible gap is what a paraphrase audit observes.

Figure 11: Left: for each model, the effect of leaking an item on the wording that was trained on and on paraphrases the model never saw, as differences in differences against uncontaminated control items, with bootstrap intervals. Right: the visible gap, the difference between the two, which is all a paraphrase audit observes. For StableLM-2-1.6B it is negative.

Figure 12: The positive control. Left: accuracy before and after exposure on the leaked parents, for the trained public wording and for paraphrases never shown to the model; the shaded band is the portion a paraphrase audit can see. Right: the same for the untouched parents, where fine-tuning moved both forms by the same amount and the measurable gap did not move.

Parents	Form	Before	After	Difference in differences	95% CI
Exposed (n = 100)	public, trained on	0.460	0.280	-0.220	[-0.350, -0.090]
Exposed (n = 100)	fresh, never trained on	0.420	0.375	-0.090	[-0.205, +0.025]
Exposed (n = 93)	deep paraphrase	0.387	0.301	-0.131	[-0.231, -0.032]
Unexposed (n = 100)	public	0.400	0.440	—	—
Unexposed (n = 100)	fresh	0.400	0.445	—	—

Table 17: The failed manipulation. Accuracy on the ARC items that were leaked, beside the ones that were not, before and after fine-tuning. The difference in differences on the trained wording is the manipulation check, and the registration required it to be at least +0.05.

Figure 13: The failed manipulation and its cause. Left: training on an item lowered accuracy on that item while untouched items rose, which is the opposite of contamination. Right: how often the scorer’s preferred option is the one listed first, on the exposed items only. Before fine-tuning the model sits at chance on every form. After it, the pull toward the first option is strongest on the trained wording, nearly as strong on the released fresh forms, which keep the same option strings, and almost gone on the deep paraphrases, which reword them.

Parents	Form	Before	After	Difference in differences	95% CI
Exposed (n = 100)	public, trained on	0.460	1.000	+0.490	[+0.360, +0.620]
Exposed (n = 100)	fresh, never trained on	0.420	0.980	+0.510	[+0.395, +0.625]
Exposed (n = 93)	deep paraphrase	0.376	0.602	+0.176	[+0.039, +0.315]
Unexposed (n = 100)	public	0.400	0.450	—	—
Unexposed (n = 100)	fresh	0.400	0.450	—	—

Table 18: The corrected leak. Accuracy on the ARC items that were leaked, beside the hundred that were not, before and after fine-tuning. The rightmost column is the difference in differences against the untouched parents.

Figure 14: Study 4 and Study 5 side by side. Left: the difference in differences on the trained wording under the two exposure formats, from the same model, the same items and the same seed. The only difference is whether the leaked line ended in a letter or in the correct option’s text. Right: accuracy on the exposed parents before and after the corrected leak, for the trained wording, the released fresh forms that keep the options verbatim, and the deep paraphrases that reword them. The gap between the first two is what an audit sees.

Parents	Form	Before	After	Difference in differences	95% CI
Exposed (n = 100)	public, trained on	0.460	1.000	+0.490	[+0.360, +0.620]
Exposed (n = 100)	fresh, options kept	0.420	0.980	+0.510	[+0.395, +0.625]
Exposed (n = 93)	deep paraphrase	0.376	0.602	+0.238	[+0.089, +0.387]
Unexposed (n = 100)	public	0.400	0.450	—	—
Unexposed (n = 100)	fresh	0.400	0.450	—	—
Unexposed (n = 83)	deep paraphrase	0.410	0.398	—	—

Table 19: The complete deep-paraphrase bank. Accuracy on the leaked parents beside the untouched ones, for all three form types, before and after the same fine-tuning as Study 5. The deep-paraphrase row is now a direct comparison rather than a substituted one.

Figure 15: The three arms of Study 6 with bootstrap intervals over parents. The public and fresh-form effects are censored by a ceiling at 1.000 and are shown hollow for that reason. The deep-paraphrase effect is not censored, rests on a direct comparison against control parents that carry the same form type, and excludes zero. The bracket marks the quantity a paraphrase audit of this benchmark actually reports.

Form family	Accuracy on leaked-half parents	Difference in differences, no leak	95% CI	Study 6, with the leak
Public form	0.460 to 0.460	-0.010	[-0.120, +0.100]	+0.490
Released fresh	0.420 to 0.435	+0.020	[-0.060, +0.100]	+0.510
Deep paraphrase	0.376 to 0.344	-0.008	[-0.147, +0.128]	+0.238

Table 20: The same statistic with the leak removed. No audited item’s wording is in the training mixture, so every entry in the last column is an empirical null for the whole procedure, and the planted effects from Study 6 are shown beside them for scale.

Training seed	Public form	Released fresh	Deep paraphrase
20260824 (the registered run, Section 7.14)	+0.490	+0.510	+0.238
20260828	+0.460	+0.510	+0.179
20260829	+0.500	+0.495	+0.204
20260830	+0.480	+0.505	+0.195
Mean	+0.483	+0.505	+0.204
Spread	0.040	0.015	0.059

Table 21: All four training seeds, including Study 6’s. Each row is a complete rerun of Section 7.14 differing only in the trainer seed and the order of the training records.

Benchmark Auditing

Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.

See all 97 papers

Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation
Aug 29, 2026Johanna Angulo, Víctor Yeste, Hector Espinos-MoratoBenchmark Contamination Benchmark Validity
Universidad Europea de Madrid Madrid, Spain · Universidad Europea de Valencia Valencia, Spain
LeakScale: Estimating the Causal Effect of Benchmark Exposure
Sep 23, 2026Divyansh SinghCausal Effect Estimation Benchmark Design
University of Florida
Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakage
Sep 11, 2026Foad Namjoo, Remy Ogasawara, Amirali Abdullah +3Data Leakage LLM Evaluation
University of Utah · Thoughtworks · University of Massachusetts Amherst +1

Aug 29, 2026J/K move · Enter open · S save

Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation

Johanna Angulo, Víctor Yeste, Hector Espinos-Morato

Universidad Europea de Madrid Madrid, Spain · Universidad Europea de Valencia Valencia, Spain

Benchmark Contamination Benchmark Validity Benchmark Auditing

Open paper AskPDF arXiv

A benchmark score is a joint property of the model, the evaluation harness, the elicitation budget, the sampled population, and contamination status. Leaderboards publish the model and the score, so capability and leakage stay observationally equivalent. Existing taxonomies classify contamination for automated detection, not the question a reporter faces at publication: given the mitigations already applied, which validity threats remain open? We introduce a taxonomy organized by the mitigation each type defeats -- direct, derivative, temporal, distributional, and acquired -- spanning training-time and evaluation-time leakage. Holding out a private test set closes the first alone. The fifth is acquired during the evaluation itself; because it is a property of one run, it must be recorded with the reported score rather than with the benchmark release. We operationalize it as a four-field disclosure protocol in which "unknown" is a valid entry, released under CC BY 4.0 with a JSON Schema, a validator, and worked examples. Two coders external to the design team applied a pre-registered instrument to 41 documents. Per-variable linear-weighted

κ

runs from 0.00 to 0.35 (median 0.21) over 29 main-pass documents against a single-coder test-retest ceiling of 0.84, collapsing under the class skew the registration anticipated; pooling raises it to 0.46 through chance correction rather than better agreement. Two variables fall below the prevalence-robust threshold registered in advance: strata reporting and the acquired type introduced here. Disagreement concentrates on when a variable applies rather than on what a document states. Elicitation budgets are reported in 13% of documents, and no document addresses all five types. The contribution is the taxonomy, the score-side artifact that follows from it, and a pre-registered measurement of instrument reliability and current disclosure.

Abstract

Benchmark Auditing

Explore similar work

Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation

LeakScale: Estimating the Causal Effect of Benchmark Exposure

Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakage