CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds
Authors: Jeffery Opoku, David Banahene
Organizations: School of Mathematical and Statistical Sciences The University of Texas Rio Grande Valley, Edinburg, Texas, USA · Department of Biostatistics, Robert Stempel College of Public Health and Social Work Florida International University, Miami, Florida, USA
Public benchmark scores may reflect skill, prior exposure to the questions, or both, and for most models the training data are unknown. We present CleanScore, a black-box audit using scored outputs only. Each benchmark question becomes a parent item with one public form and two independently written fresh forms preserving its numbers, facts and answer. The audit reports an interval for the public-form advantage rather than a verdict, and a private negative-control bank with an explicit transport radius separates exposure from ordinary form mismatch. A registered controlled-exposure experiment detects planted exposure and stays quiet under fresh-form exposure. A registered audit of five open models on 200 GSM8K and 200 ARC-Challenge items finds no exposure-consistent advantage, bounding surface-form inflation below five points. Registered positive controls then bound what such a null can mean. Leaking an item raises accuracy on paraphrases the model never saw almost as much as on the leaked wording, leaving 52% to 110% of the effect invisible to a paraphrase audit. On ARC a planted 49-point advantage shows an observable gap of -0.020, and about 20 points survive rewriting stem and options, across four training seeds. A surface-form null bounds far less than the phrase contamination audit implies.
Figures & tables
Figure 1: The finding of Section 7.11, stated once at the start because it bounds how much any result in this paper can mean. Leaking an item’s wording raises accuracy on that wording, and most of that gain carries over to paraphrases of the same item, where an audit that compares an original against a rewrite cannot see it. The estimated split is 77% invisible, with an interval running from 52% to 110%; at the top of that interval nothing is visible at all.
Figure 2: The matched-form audit. Each sampled parent item is put to the model in its public form and in two fresh forms that keep the same numbers, facts, and answer; the public-form advantage is the difference in mean correctness between them. A never-public control bank of items no model can have memorised is scored the same way and supplies the form-mismatch term. The registered claim rule requires both intervals to exclude zero, so ordinary form mismatch cannot on its own produce a contamination claim.
Family
Contrast
Mean
95% CI
Seeds positive
p (two-sided)
SmolLM2-360M
Exact exposure minus related-task
+0.020
[+0.006, +0.034]
9/10
0.010
SmolLM2-360M
Paraphrase exposure minus related-task
-0.002
[-0.009, +0.005]
3/10
0.61
Qwen2.5-0.5B
Exact exposure minus related-task
+0.043
[-0.018, +0.104]
7/10
0.14
Qwen2.5-0.5B
Paraphrase exposure minus related-task
-0.015
[-0.045, +0.016]
3/10
0.31
Table 1: Registered seed-paired contrasts in the controlled-exposure validation (ten seeds per family).
Figure 3: Registered seed-paired contrasts, ten seeds per family. Exact exposure to public forms shifts the public-form advantage upward in both families, decisively in SmolLM2-360M; exposure to fresh forms of the same items does not. The wide Qwen2.5-0.5B interval reflects that model’s larger run-to-run variation, not a weaker effect.
Design
Sample size
Hoeffding coverage
Hoeffding half-width
Wald coverage
Mean Wald half-width
Simple random sample
120
1.0000
0.2480
0.9476
0.0625
Simple random sample
400
1.0000
0.1358
0.9506
0.0321
Simple random sample
800
1.0000
0.0960
0.9522
0.0203
Simple random sample
1,200
1.0000
0.0784
0.9468
0.0144
Stratified sample
400
1.0000
0.1358
0.9488
0.0329
Table 2: Coverage and width of the two intervals in the finite-population calibration, 5,000 replications per design.
Figure 4: Interval precision and empirical coverage in the finite-population simulation. The conservative bound is valid but wide in these settings. The finite-population Wald interval is much sharper and stays close to the nominal 95% level.
Figure 5: Sensitivity to the transport radius when the true control-to-benchmark mismatch is 0.05. Understating the radius can cause severe undercoverage. A radius that covers the true mismatch restores protection but widens the interval.
Figure 6: Wald coverage for the 5,000-item benchmark settings. Results are generally close to 95%, but the wider stress grid shows that this approximation is not uniformly reliable. Small sparse audits are the important failure case.
Model
Benchmark
Public score
Fresh score
Δ^
Half-width
Nonzero gaps
bC (control)
OLMo-2-7B
GSM8K
0.720
0.705
+0.015
0.045
58
+0.017 ± 0.081
OLMo-2-7B
ARC
0.625
0.608
+0.018
0.040
40
+0.013 ± 0.102
SmolLM2-1.7B
GSM8K
0.475
0.430
+0.045
0.058
83
0.000 ± 0.119
SmolLM2-1.7B
ARC
0.485
0.470
+0.015
0.192 (H)
26
-0.088 ± 0.110
Phi-3-mini
GSM8K
0.820
0.803
+0.018
0.035
35
+0.008 ± 0.049
Phi-3-mini
ARC
0.595
0.598
-0.003
0.192 (H)
27
0.000 ± 0.105
Table 3: Registered real-benchmark audit, all five models. Δ^ is the public-form advantage over 200 sampled parents. The half-width is the primary interval: the finite-population Wald interval, or the Hoeffding bound (marked H) where the nonzero-gap guardrail applies. bC is the control-bank form gap with its Wald half-width. No model-benchmark pair meets the registered claim rule.
Figure 7: Public-form advantage with its primary interval for every model-benchmark pair, with the control-bank gap bC marked separately. Dotted intervals are the two pairs where the registered nonzero-gap guardrail makes the conservative bound primary. Every interval covers zero.
Model
Setting
Public score
Fresh score
Δ^
Half-width
Truncated
Qwen2.5-7B
Registered (320 tokens)
0.710
0.715
-0.005
0.047
29.8%
Qwen2.5-7B
Amendment 1 (768 tokens)
0.930
0.938
-0.008
0.192 (H)
0.0%
Qwen2.5-1.5B
Registered (320 tokens)
0.475
0.500
-0.025
0.055
40.7%
Qwen2.5-1.5B
Amendment 1 (768 tokens)
0.695
0.735
-0.040
0.054
0.0%
Table 4: Amendment 1: the two Qwen models on GSM8K with a 768-token budget and a boxed-answer parser. Truncation falls to zero. ARC is unaffected by both changes and was reproduced bit-for-bit.
Reading of the control bank
Control half-width
Mean width of the τ interval
Pairs excluding zero
Wald (registered)
0.057 to 0.136
0.335
0 of 10
Hoeffding (own guardrail)
0.382 to 0.468
0.982
0 of 10
Fixed bank, no interval
0
0.131
1 of 10
Table 5: The sensitivity interval under three readings of the control bank, at the registered radius. Wald is the registered construction. Hoeffding applies the paper’s own small-sample guardrail to the control bank as well. Fixed treats the bank as a fully measured object, so bC is a constant and the transport radius carries everything.
Interval
Coverage range
Settings below 0.95
Width vs Hoeffding at n = 200
Hoeffding
0.9945 to 1.0000
0
1.00 by definition
Empirical Bernstein
0.9998 to 1.0000
0
0.77 to 1.62, depending on the shape
Finite-population Wald
0.9390 to 0.9563
24
not distribution-free
Table 6: Coverage of the three intervals across the 36 stress settings (4,000 audits each), and the width of the Bernstein bound relative to the Hoeffding bound at the audit’s own sample size, n = 200 in a 1,000-item benchmark.
Model
Benchmark
Public score
Fresh score
Confidence, public
Confidence, fresh
PaCoST p
PaCoST flags
CleanScore claim
OLMo-2-7B
GSM8K
0.720
0.705
-0.098
-0.096
0.28
no
no
OLMo-2-7B
ARC
0.625
0.608
-3.19
-3.17
0.71
no
no
SmolLM2-1.7B
GSM8K
0.475
0.430
-0.156
-0.156
0.83
no
no
SmolLM2-1.7B
ARC
0.485
0.470
-3.74
-3.67
0.18
no
no
Phi-3-mini
GSM8K
0.820
0.803
-0.119
-0.119
0.94
no
no
Phi-3-mini
ARC
0.595
0.598
-3.15
-3.20
0.017
yes
no
Table 7: Baseline detectors on the audited items. PaCoST flags a pair when the paired confidence test rejects at 0.05 with higher confidence on the public form.
Benchmark
Pattern
Observed
Null mean
p
GSM8K
rewrites harder
4
3.37
0.45
GSM8K
rewrites easier
2
3.29
0.88
ARC
rewrites harder
3
0.70
0.02
ARC
rewrites easier
0
0.30
1.00
Table 8: Parents on which at least two of the five models show the pattern, against a permutation null with 20,000 draws.
Model
Δ^ , all 200
Δ^ , deep rewrites only
Shift
r (similarity, gap)
p
OLMo-2-7B
+0.015
+0.028
+0.013
+0.029
0.69
Phi-3-mini
+0.018
+0.017
-0.001
+0.023
0.75
Qwen2.5-1.5B
-0.025
-0.020
+0.005
-0.035
0.62
Qwen2.5-7B
-0.005
+0.003
+0.008
-0.072
0.31
SmolLM2-1.7B
+0.045
+0.050
+0.005
-0.004
0.96
Table 9: Sensitivity of the GSM8K audit to rewrite depth. The third column drops the 21 parents whose closest fresh form meets or exceeds the ARC dissimilarity threshold. The last column is the correlation, across all 200 parents, between a parent’s rewrite similarity and its measured gap.
Benchmark
Audited items
Mean half-width
Exact order
Pairs correct
GSM8K
25
0.142
0.39
0.92
GSM8K
50
0.101
0.50
0.95
GSM8K
100
0.070
0.62
0.97
GSM8K
150
0.056
0.73
0.98
ARC
25
0.093
0.15
0.78
ARC
50
0.067
0.25
0.86
Table 10: Budgeted-audit curves (mean over models and 2,000 subsamples). "Exact order" is recovery of the full five-model ordering; "pairs correct" is the share of the ten model pairs ordered as the full sample orders them.
Figure 8: Budgeted audits. Precision improves smoothly with audit size, and pairwise ordering is recovered cheaply, but exact recovery of the five-model ordering lags well behind because two model pairs differ by less than the audit’s own half-width.
Benchmark
Audited items
delta = 0.02
delta = 0.05
delta = 0.10
delta = 0.20
GSM8K
50
0.00
0.00
0.00
0.00
GSM8K
100
0.00
0.00
0.00
0.04
GSM8K
200
0.22
0.56
0.95
1.00
GSM8K
400
0.43
0.89
1.00
1.00
GSM8K
800
0.82
1.00
1.00
1.00
ARC
50
0.00
0.00
0.00
0.00
Table 11: Detection rate for a planted public-form advantage, 4,000 resampled audits per cell, pooling the five audited models within each benchmark.
Figure 9: Left: the primary half-width, which is the smallest advantage the audit would declare, for each model and benchmark at n = 200; open markers are the two pairs where the guardrail applies. Right: detection rate against audit size for planted advantages of two, five, ten, and twenty points. The flat zero region below 200 items is the guardrail.
Benchmark
Audited items
delta
Interval
Exact test
GSM8K
50
0.10
0.00
0.43
GSM8K
50
0.20
0.00
0.92
GSM8K
100
0.05
0.00
0.32
GSM8K
100
0.10
0.00
0.77
GSM8K
200
0.05
0.56
0.61
GSM8K
200
0.10
0.95
0.97
Table 12: Detection rate of the exact randomization test against the interval of Section 7.9, same planted advantages, same resampled banks, 2,000 audits per cell. The test’s simulated type-I error across all sizes and both benchmarks runs from 0.029 to 0.052 against a nominal 0.05.
Figure 10: Left and centre: detection rate against audit size for a five-point and a ten-point planted advantage, exact test against interval. Right: simulated type-I error of the exact test, obtained by relabelling the real correctness vectors so the null holds by construction; the test sits at or below the nominal level at every size.
Form gap in the data
Rejection rate at the 0.05 level
0.000 (exchangeable)
0.039
0.020
0.107
0.050
0.315
0.088 (largest measured)
0.689
Table 13: What ordinary form mismatch does to the exact test. No contamination is present in any row; every rejection is caused by the fresh forms being slightly harder than the public one. The largest form gap is the largest measured on any control bank in this audit.
Parents
Form
Before
After
Change
Exposed (n = 100)
public, trained on
0.460
0.610
+0.150
Exposed (n = 100)
fresh, never trained on
0.485
0.570
+0.085
Exposed (n = 100)
measurable gap
-0.025
+0.040
+0.065
Unexposed (n = 100)
public
0.490
0.400
-0.090
Unexposed (n = 100)
fresh
0.515
0.425
-0.090
Unexposed (n = 100)
measurable gap
-0.025
-0.025
+0.000
Table 14: Where the contamination effect went. Accuracy before and after exposure, on the hundred leaked parents and the hundred untouched ones. The measurable gap is the difference between the public and fresh columns, which is what any paraphrase audit observes.
Quantity
Exposed
Unexposed
Difference in differences
95% CI
Change in public-form accuracy
+0.150
-0.090
+0.240
[+0.080, +0.400]
Change in fresh-form accuracy
+0.085
-0.090
+0.175
[+0.035, +0.315]
Change in the measurable gap
+0.065
+0.000
+0.065
—
Table 15: Difference-in-differences estimates for the effect of leaking an item’s public wording, with bootstrap intervals over parents.
Model
Family
DiD public
DiD fresh
Visible gap
Qwen2.5-1.5B
Qwen
+0.240 [+0.080, +0.400]
+0.175 [+0.035, +0.315]
+0.065
SmolLM2-1.7B
SmolLM2
+0.100 [-0.050, +0.250]
+0.040 [-0.060, +0.140]
+0.060
OLMo-2-1B
OLMo
+0.500 [+0.340, +0.660]
+0.350 [+0.230, +0.470]
+0.150
StableLM-2-1.6B
StableLM
+0.200 [+0.050, +0.350]
+0.230 [+0.115, +0.340]
-0.030
Pooled
+0.260 [+0.183, +0.335]
+0.199 [+0.140, +0.258]
+0.061
Table 16: Four contaminated models from four families. Difference-in-differences against the unexposed control parents of the same model, with bootstrap intervals. The visible gap is what a paraphrase audit observes.
Figure 11: Left: for each model, the effect of leaking an item on the wording that was trained on and on paraphrases the model never saw, as differences in differences against uncontaminated control items, with bootstrap intervals. Right: the visible gap, the difference between the two, which is all a paraphrase audit observes. For StableLM-2-1.6B it is negative.
Figure 12: The positive control. Left: accuracy before and after exposure on the leaked parents, for the trained public wording and for paraphrases never shown to the model; the shaded band is the portion a paraphrase audit can see. Right: the same for the untouched parents, where fine-tuning moved both forms by the same amount and the measurable gap did not move.
Parents
Form
Before
After
Difference in differences
95% CI
Exposed (n = 100)
public, trained on
0.460
0.280
-0.220
[-0.350, -0.090]
Exposed (n = 100)
fresh, never trained on
0.420
0.375
-0.090
[-0.205, +0.025]
Exposed (n = 93)
deep paraphrase
0.387
0.301
-0.131
[-0.231, -0.032]
Unexposed (n = 100)
public
0.400
0.440
—
—
Unexposed (n = 100)
fresh
0.400
0.445
—
—
Table 17: The failed manipulation. Accuracy on the ARC items that were leaked, beside the ones that were not, before and after fine-tuning. The difference in differences on the trained wording is the manipulation check, and the registration required it to be at least +0.05.
Figure 13: The failed manipulation and its cause. Left: training on an item lowered accuracy on that item while untouched items rose, which is the opposite of contamination. Right: how often the scorer’s preferred option is the one listed first, on the exposed items only. Before fine-tuning the model sits at chance on every form. After it, the pull toward the first option is strongest on the trained wording, nearly as strong on the released fresh forms, which keep the same option strings, and almost gone on the deep paraphrases, which reword them.
Parents
Form
Before
After
Difference in differences
95% CI
Exposed (n = 100)
public, trained on
0.460
1.000
+0.490
[+0.360, +0.620]
Exposed (n = 100)
fresh, never trained on
0.420
0.980
+0.510
[+0.395, +0.625]
Exposed (n = 93)
deep paraphrase
0.376
0.602
+0.176
[+0.039, +0.315]
Unexposed (n = 100)
public
0.400
0.450
—
—
Unexposed (n = 100)
fresh
0.400
0.450
—
—
Table 18: The corrected leak. Accuracy on the ARC items that were leaked, beside the hundred that were not, before and after fine-tuning. The rightmost column is the difference in differences against the untouched parents.
Figure 14: Study 4 and Study 5 side by side. Left: the difference in differences on the trained wording under the two exposure formats, from the same model, the same items and the same seed. The only difference is whether the leaked line ended in a letter or in the correct option’s text. Right: accuracy on the exposed parents before and after the corrected leak, for the trained wording, the released fresh forms that keep the options verbatim, and the deep paraphrases that reword them. The gap between the first two is what an audit sees.
Parents
Form
Before
After
Difference in differences
95% CI
Exposed (n = 100)
public, trained on
0.460
1.000
+0.490
[+0.360, +0.620]
Exposed (n = 100)
fresh, options kept
0.420
0.980
+0.510
[+0.395, +0.625]
Exposed (n = 93)
deep paraphrase
0.376
0.602
+0.238
[+0.089, +0.387]
Unexposed (n = 100)
public
0.400
0.450
—
—
Unexposed (n = 100)
fresh
0.400
0.450
—
—
Unexposed (n = 83)
deep paraphrase
0.410
0.398
—
—
Table 19: The complete deep-paraphrase bank. Accuracy on the leaked parents beside the untouched ones, for all three form types, before and after the same fine-tuning as Study 5. The deep-paraphrase row is now a direct comparison rather than a substituted one.
Figure 15: The three arms of Study 6 with bootstrap intervals over parents. The public and fresh-form effects are censored by a ceiling at 1.000 and are shown hollow for that reason. The deep-paraphrase effect is not censored, rests on a direct comparison against control parents that carry the same form type, and excludes zero. The bracket marks the quantity a paraphrase audit of this benchmark actually reports.
Form family
Accuracy on leaked-half parents
Difference in differences, no leak
95% CI
Study 6, with the leak
Public form
0.460 to 0.460
-0.010
[-0.120, +0.100]
+0.490
Released fresh
0.420 to 0.435
+0.020
[-0.060, +0.100]
+0.510
Deep paraphrase
0.376 to 0.344
-0.008
[-0.147, +0.128]
+0.238
Table 20: The same statistic with the leak removed. No audited item’s wording is in the training mixture, so every entry in the last column is an empirical null for the whole procedure, and the planted effects from Study 6 are shown beside them for scale.
Training seed
Public form
Released fresh
Deep paraphrase
20260824 (the registered run, Section 7.14)
+0.490
+0.510
+0.238
20260828
+0.460
+0.510
+0.179
20260829
+0.500
+0.495
+0.204
20260830
+0.480
+0.505
+0.195
Mean
+0.483
+0.505
+0.204
Spread
0.040
0.015
0.059
Table 21: All four training seeds, including Study 6’s. Each row is a complete rerun of Section 7.14 differing only in the trainer seed and the order of the training records.
A benchmark score is a joint property of the model, the evaluation harness, the elicitation budget, the sampled population, and contamination status. Leaderboards publish the model and the score, so capability and leakage stay observationally equivalent. Existing taxonomies classify contamination for automated detection, not the question a reporter faces at publication: given the mitigations already applied, which validity threats remain open? We introduce a taxonomy organized by the mitigation each type defeats -- direct, derivative, temporal, distributional, and acquired -- spanning training-time and evaluation-time leakage. Holding out a private test set closes the first alone. The fifth is acquired during the evaluation itself; because it is a property of one run, it must be recorded with the reported score rather than with the benchmark release. We operationalize it as a four-field disclosure protocol in which "unknown" is a valid entry, released under CC BY 4.0 with a JSON Schema, a validator, and worked examples. Two coders external to the design team applied a pre-registered instrument to 41 documents. Per-variable linear-weighted κ runs from 0.00 to 0.35 (median 0.21) over 29 main-pass documents against a single-coder test-retest ceiling of 0.84, collapsing under the class skew the registration anticipated; pooling raises it to 0.46 through chance correction rather than better agreement. Two variables fall below the prevalence-robust threshold registered in advance: strata reporting and the acquired type introduced here. Disagreement concentrates on when a variable applies rather than on what a document states. Elicitation budgets are reported in 13% of documents, and no document addresses all five types. The contribution is the taxonomy, the score-side artifact that follows from it, and a pre-registered measurement of instrument reliability and current disclosure.
Johanna Angulo, Víctor Yeste, Hector Espinos-Morato
Universidad Europea de Madrid Madrid, Spain · Universidad Europea de Valencia Valencia, Spain
Benchmark contamination is usually treated as a provenance question: did evaluation material appear in training? Provenance cannot supply the quantity needed to interpret a benchmark score: how much exposure changed it. We introduce LeakScale, a causal measurement framework that makes benchmark-specific exposure manipulable and measures its effect. LeakScale constructs fresh executable tasks whose correct behavior depends on private, non-derivable information, counterfactually certifies every candidate answer, and trains complementary adapters so that every family is exposed in exactly one adapter and unexposed in the other. Across 2,048 families, two model families, two executable domains, three task formulations, and 589,824 generations, exposure increases exact-task accuracy in every model-by-domain cell. Primary effects range from +3.22 to +10.66 percentage points; every family-clustered 95% interval excludes zero, and all four primary tests survive Holm correction. The effect also remains positive in all eight surface-paraphrase and isomorphic-transfer measurements. These results show that benchmark exposure can causally and reproducibly alter reported performance, including after the evaluation surface changes. LeakScale shifts contamination analysis from detecting contact to measuring how strongly a score depends on it.
Binary-choice truth benchmarks ask models to choose between a correct and an incorrect answer, but if the two answers differ systematically in surface-level features, models can exceed chance without performing the intended reasoning. We show that this failure mode is detectable and can be exploited by downstream classifiers. In TruthfulQA, a simple six-feature logistic classifier achieves substantial accuracy in separating correct from incorrect answers. We further show that similar surface-level artifacts are present in additional benchmarks. To counteract this, we developed a general mechanism to clean them by removing the most leakage-reinforcing pairs. We release a version of TruthfulQA with surface-feature leakage reduced close to chance and provide a mechanism, Audit-Prune, so that the datasets can be cleaned before release.
Foad Namjoo, Remy Ogasawara, Amirali Abdullah +3
University of Utah · Thoughtworks · University of Massachusetts Amherst +1