Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation

Dishu Yang; Qi Su; Hongbo Qin; Hansong Zhang

cs.AIOct 5, 2026

Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation

Authors: Dishu Yang, Qi Su, Hongbo Qin, Hansong Zhang

Organizations: Khoury College of Computer Sciences Northeastern University San Jose, CA, USA · School of Engineering and Applied Science The George Washington University Bellevue, WA, USA · Khoury College of Computer Sciences Northeastern University Boston, MA, USA · Department of Computer Science University of California, Berkeley Berkeley, CA, USA

Abstract

Agentic leaderboards increasingly evaluate systems on public benchmarks whose task statements and solution-bearing artifacts can remain accessible. We propose a measurement-first audit framework that distinguishes contamination claims according to the evidence required to support them. It separates three channels that require different evidence: training-time exposure, evaluation-time retrieval, and pipeline/scaffold leakage. Each channel is coded as open, partial, closed, or unknown under a fail-closed rule. Across nine Holistic Agent Leaderboard (HAL) configurations, none of the 27 channel assessments was coded closed, but incidents were confirmed in four configurations. We then apply the behavioral component of the framework to a reported file-localization gap on SWE-bench Verified, using an outcome-blind, same-repository matched-control design with symmetric prompt-leakage screening, paired and repository-aware uncertainty analyses, and scorer validation, evaluated on GPT-4.1 and DeepSeek-V4-Flash. Among the 100 pairs retained after symmetric screening and the pair-integrity exclusion, GPT-4.1 showed a $+10.0$ -point pair-weighted Top-3 benchmark-associated gap, but the 95% intervals from both the prespecified paired-bootstrap procedure and the post-hoc repository-balanced analysis included zero, leaving the benchmark-associated gap inconclusive. The reproduction scorer did not pass its validation gate: against consensus human labels, sufficient scorer sensitivity could not be established for either model, and both DeepSeek-V4-Flash firings on correct-gold comparisons were false positives. Without provenance evidence, appropriate controls, symmetric leakage screening, and validated scorers, stronger contamination claims are not warranted. The results do not establish training-data membership, contamination prevalence, or benchmark-induced score inflation.

Figures & tables

	Susceptibility			Realized incident
Benchmark	T	R	L	Confirmed	Revision
Online Mind2Web (BrowserUse / SeeAct)	P [Y/N/N/?]	O	?	Not established	Unknown
AssistantBench (Browser agent)	P [Y/Y/Y/?]	O	O	L	Current
GAIA (Open Deep Research)	P [Y/Y/Y/?]	O	?	Not established	Unknown
CORE-Bench Hard (CORE-Agent / Generalist)	P [Y/Y/Y/?]	O	P	Not established	Unknown
ScienceAgentBench (SAB Self-Debug)	P [Y/Y/Y/?]	P	P	None	—

Table 1: HAL-wide susceptibility and incident card. No channel was coded closed; confirmed incidents are narrower than susceptibility.

Model	Metric	Pair-weighted gap [95% CI]	Repository-balanced gap [95% CI]
DeepSeek-V4-Flash	Top-3 (primary)	$+0.070$ $[-0.070,+0.210]$	$+0.081$ $[-0.079,+0.240]$
	Top-1	$+0.100$ $[-0.030,+0.230]$	$+0.086$ $[-0.081,+0.252]$
	MRR@3	$+0.088$ $[-0.042,+0.217]$	$+0.081$ $[-0.072,+0.233]$
	Path F1	$+0.084$ $[-0.040,+0.205]$	$+0.078$ $[-0.057,+0.213]$
GPT-4.1	Top-3 (primary)	$+0.100$ $[-0.030,+0.230]$	$+0.109$ $[-0.116,+0.334]$
	Top-1	$+0.160$ $[+0.020,+0.300]$	$+0.138$ $[-0.051,+0.328]$

Table 2: Probe 1 benchmark-associated gaps (benchmark minus control), 100-pair subset. All gaps are positive; no Top-3 interval excludes zero.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Stage II cohort construction and probe-specific attrition. All post-matching counts refer to matched pairs. Leakage screening excluded both members when either arm met the exclusion criterion. The same-issue exclusion removed one pair whose two arms addressed the same underlying issue. For Probe 2, the final arrow combines sequential exclusions for persistent truncation (7 pairs) and absence of an eligible strict signature in at least one arm (131 pairs).

Stage	Change	Pairs retained
Initial matched sample	—	199
Pair-level leakage exclusion	$-18$	181
Cross-arm underlying-issue collision	$-1$	180
Persistent truncation	$-7$	173
No eligible strict signature in benchmark arm	$-89$	84
No eligible strict signature in control arm	$-42$	42

Appendix

Table A1: Probe 2 sample construction and descriptive automatic results.

Model	Human-positive responses	Detected by strict scorer	Weighted sensitivity	Exact 95% CI, unweighted sensitivity	Precision
DeepSeek-V4-Flash	2/41	0/2	0.00	$[0.00,0.84]$	0/2
GPT-4.1	3/38	1/3	0.19	$[0.01,0.91]$	1/1

Appendix

Table A2: Strict-scorer validation against consensus human labels. The scorer did not pass the validation gate: sensitivity could not be established for either model, and both DeepSeek-V4-Flash firings were false positives.

Model	Sampling stratum	Available arms $N_{h}$	Reviewed arms $n_{h}$	Inclusion probability $\pi_{h}$	Weight $1/\pi_{h}$
DeepSeek-V4-Flash	Exact hit	2	2	1.000	1.00
	Fuzzy-only	6	6	1.000	1.00
	High overlap	15	15	1.000	1.00
	Random tail	61	20	0.328	3.05
GPT-4.1	Exact hit	1	1	1.000	1.00
	Fuzzy-only	4	4	1.000	1.00

Appendix

Table A3: Sampling design for the Probe 2 manual scorer validation.

Model	N pairs	Pair-weighted gap [95% CI]	Repository-balanced gap [95% CI]	Hierarchical 95% CI
DeepSeek-V4-Flash	64	$+0.063$ $[-0.109,+0.234]$	$+0.035$ $[-0.206,+0.276]$	$[-0.244,+0.301]$
GPT-4.1	64	$+0.109$ $[-0.063,+0.281]$	$+0.128$ $[-0.160,+0.416]$	$[-0.197,+0.436]$

Appendix

Table A4: Probe 1 sensitivity analysis restricted to task-type-concordant pairs.

Agreement unit	Agreements / total	Agreement
Field-level judgments	90/117	76.9%
Derived T labels	9/9	100.0%
Derived R labels	6/9	66.7%
Derived L labels	6/9	66.7%
All derived T/R/L labels	21/27	77.8%
Incident channel/status	4/9	44.4%

Appendix

Table B1: Pre-adjudication agreement for the Stage I susceptibility and incident card.

Hypothesis	Disposition	Evidence
H1: Benchmark tasks exhibit higher file-path accuracy.	Inconclusive; point estimates were positive, but the primary 95% intervals included zero.	All gaps were positive; no Top-3 primary interval excluded zero (Section 4.3 ).
H2: Benchmark tasks exhibit higher patch-specific line recovery.	Unresolved: measurement validity insufficient.	The scorer did not pass the overall validation gate: sufficient sensitivity could not be established for either model, and both DeepSeek-V4-Flash firings on correct-gold comparisons were human-negative. The post-hoc strict-score contrasts therefore remain exploratory.
H3: Reproduction is rarer than localization and, conditional on adequate scorer sensitivity, has an equal or larger benchmark-control gap.	Unresolved: required sensitivity condition unmet.	The sensitivity condition could not be established, so neither the reproduction-frequency comparison nor the conditional benchmark-control gap comparison could be adjudicated.

Appendix

Table E1: Disposition of the prespecified hypotheses.

Explore similar work

Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation
Aug 7, 2026Sahil Pardasani, Madhusudan SinghReal-Tool Benchmarks Deepseek
Blockchain Data Intelligence Lab The Pennsylvania State University, 201 Old Main, University Park, 16802, PA, United States of America.
Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes
Sep 28, 2026Weijun Luo, Kelvin Luu, Xinyi Liu +6Pass-Rate Evaluation Agentic Benchmarks
Scale AI
Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
Jul 24, 2026Jiaqi Shao, Hanck Chen, Wei Zhang +2Agentic Benchmarks Exploitation
1Hunyuan Team, Tencent · 2The Hong Kong University of Science and Technology · 3Duke Kunshan University

Aug 7, 2026, cs.AIJ/K move · Enter open · S save

Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation

Sahil Pardasani, Madhusudan Singh

Blockchain Data Intelligence Lab The Pennsylvania State University, 201 Old Main, University Park, 16802, PA, United States of America.

Real-Tool Benchmarks Deepseek Cryptographic Merkle Tree Commitments

Open paper AskPDF arXiv

LLM benchmarks can build an organization's reputation and attract customers, but only when results are transparent and verifiable. Unverified claims that DeepSeek R1 outperformed OpenAI's o1 contributed to market panic on January 27, 2025, when Nvidia lost USD589 billion in market value. Yet vendor benchmarks often depend on an honor system. Academic reassessments and independent leaderboards have found undisclosed changes to proprietary models, contaminated training data, and selective reporting. LLM-as-a-judge methods scale evaluation by reducing human review. Studies, however, suggest that judges may show identity-aware bias, scoring an answer according to its source model rather than its quality. This bias has not been fully measured or corrected across politically sensitive, reasoning-intensive, and preference-based tasks. We examine this problem using seven verifier models: GPT-OSS 120B, Llama 3.3 70B, GLM 5.1, Qwen3 32B, DeepSeek V4 Pro, Mistral Large3, and Sarvam M. They score anonymous and identity-disclosed responses from three primary models on 58 factual, reasoning, political, and preference-based questions. Identity disclosure slightly raises scores for factual questions, moderately affects stress-reasoning tasks, and causes large changes for geopolitically sensitive topics. Notable results include GLM5.1 (+7.00 points, p = 0.0249) and Llama 3.3 70B (+1.56 points, p = 0.00). We also introduce a blockchain-based commit-reveal protocol using Autonomous Economic Agents on an Ethereum-compatible ledger. In Phase 1, each judge records a one-way hash of its score and a secret salt before candidate identities are revealed. In Phase 2, the identity and raw score are disclosed and verified on-chain. This creates a tamper-evident audit trail that separates blind evaluation from post-hoc claims and reduces the verification burden on independent researchers and leaderboard operators.

Abstract

Explore similar work

Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation

Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes

Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI