What Does an Evaluation License? A Commit-Bound Census of Claim Replay in Inspect Evals

Xi Qin; Jizhou Tong

cs.SEAug 18, 2026

What Does an Evaluation License? A Commit-Bound Census of Claim Replay in Inspect Evals

Authors: Xi Qin, Jizhou Tong

Organizations: Wuhan University

Abstract

Benchmarks can run without determining what their results license. We freeze a large evaluation collection and attempt to replay its historical claims. Most units stop because the evidence required for replay is not bound. Where replay is possible, different claims remain stable at different resolutions. We make this otherwise implicit inference step explicit and executable.

Figures & tables

Panel of Figure 1 — Figure 1: Three questions, three outputs. Missing evidence, multiple evaluator meanings, and disagreement across claim levels are not one generic robustness outcome.

Primary (2A): one winner; one full order; 3/3 pairs agree
Reading	Family	Claude 3.5	Gemini 2.0	GPT-4o-mini
Published micro	Primary	1.113	20.827	27.186
All-run micro	Primary	1.113	14.120	27.186
Common-suite macro	Review only	1.473	26.731	30.665
Utility-conditional	Review only	1.316	32.000	19.745
Worst suite	Review only	3.810	66.667	57.143

Table 1: AgentDojo targeted attack success rate (lower is better) over the same outcomes. The primary analysis uses the first two readings; the review family also asks what happens under the three disputed readings.

Evidence stream	Design	What it licenses
124-unit census	Complete mechanically eligible frame at one commit	Within-frame accounting: 14 complete, 110 explicit stop records
Frozen random ten	Two per stratum; case selection frozen before terminal audit outcomes; outcome exposure recorded separately	Completion under that frozen balanced-strata draw: 3/10
Mechanism portfolio	Three purposive deep audits plus eight labeled outcome-exposed discoveries	Examples of how claims agree or disagree, not a frequency

Table 2: Three evidence streams and the statements they license.

Native object	Already supports	What the historical claim still needs
task/scorer	Forward task and scoring definition	Historical instance and replay target
EvalLog , when retained	Inputs, outputs, targets, scores, metadata, reductions	Support decisions and admitted evaluator family
report	Metrics and reproducibility metadata	The claim being asked and the answers that agree or disagree

Table 3: What native evaluation objects provide, and what a historical claim still needs.

Stream / case	Evaluator choices	P/E winner or direction	P/E order / stable pairs
Random (3 complete)	credit, aggregation, claim target	all: non-id / non-id	all: non-id / non-id ; 0/1, 0/1, 0/3 in both
Purposive: AgentDojo	eligibility, denominator, aggregation	winner id / id	P: id , 3/3; E: non-id , 2/3
Purposive: AutoML	missingness, task/rank aggregation	winner id / id	P: id , 10/10; E: non-id , 9/10
Purposive: BEIR (5 views)	metric, cutoff, relevance grading	winner id / id	non-id / non-id ; 2/3 in both
Outcome-exposed exploratory (8)	source/evaluator readings and audit proposals	Case-specific landscape; no pooled P/E claim

Table 4: What stays fixed across evaluator meanings, separated by evidence stream. P/E denote the primary/review families

\mathcal{F}_{\mathrm{primary}}/\mathcal{F}_{\mathrm{review}}

. Detailed random and exploratory rows are in the appendix.

Question	Available-case sensitivity summary	Claim-replay result
Can the claim be replayed?	Conditions on cases with usable observations or leaves missing cases outside the numerical analysis	Retains all 124 records; 110 receive an explicit stop with the first missing item and minimum unblock
Which choices define the claim?	May pool all executable variants into one choice set	Separates the primary family from the wider review family; AgentDojo and AutoML have one primary order but multiple review-family orders
What survives variation?	Often reports a scalar range or any flip	Reports claim-specific answer sets, witnesses, and stable relations; the BEIR sensitivity shows that 2/3 stable pairs over five views can become 0/3 over an expanded evaluator grid

Table 5: What an available-case sensitivity summary can lose, and what claim replay adds.

Panel of Figure 2 — Figure 2: What the added interface changes. It preserves cases that cannot be replayed, separates primary from disputed meanings, and states what remains fixed alongside what changes.

Panel of Figure 3 — Figure 3: One rule changes the conclusion while predictions stay fixed. Left: replacing the original one-sided terminal rule with either symmetric rule moves the adoption effect from approximately $-40$ pp to zero. Right: the exclude-both family still permits five weak orders, yet Mini is the unique winner.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Four coordinates that a scalar sensitivity range can conflate. The present contract enumerates semantic variation on fixed evidence, keeps sampling uncertainty separate, and removes purely reversible representation changes before claim comparison.

Layer	What is checked	Licensed conclusion
Mechanical closure	Hashes and schema; terminal accounting; deterministic source-table recomputation; family projection; claim images, witnesses, and stable pairs	The bound bytes produce the reported arithmetic and identified sets through an independently checked code path
Human-auditable semantic provenance	Source locators and quotations; normalized authorization; same-target/support rationale; A/D/X judgments; evidence horizon	An independent reader can inspect, dispute, and correct the semantic ledger; the automated verifier does not prove construct validity or provenance correctness
Outside both layers	Unreleased sample-level facts; unenumerated semantics; future releases or interventions	No family-completeness, ecosystem-prevalence, independent-replication, benchmark-validity, causal-utility, or future-stop-reduction claim

Appendix

Table 6: Two validation layers and their distinct licenses.

View	Complete	stop	Licensed interpretation
All 124	14	110	Terminal auditability inside the complete eligible frame
Outcome-blind census-review path (114)	11	103	Source review, admission, and fixed-observation execution before outcome inspection
Frozen random ten	3	7	Completion under the balanced two-per-stratum draw; stopped units retained

Appendix

Table 7: Nested accounting for the commit-bound frame. The rows are accounting views, not pooled result samples.

Panel of Figure 5 — Figure 5: Two accounting axes. The top line exhausts the 124-unit frame. The bottom line names evidence panels with distinct selection and exposure licenses; arithmetic equality does not make them one cohort.

Panel of Figure 6 — Figure 6: The preservation chain is claim-relative. A retained scorer is only the first layer; later claims can require historical outputs, mediator decisions, state, support joins, and an admitted semantic registry. The preservation layers are conceptual dependencies; the verifier precedence used for first-stop accounting is G1–G8 below.

Layer	Required object	Enables	Typed stop when absent
Source	task, scorer, registry revision, dependencies	Ground candidate formulas	source/evaluator binding
Observation	sample-keyed outputs and system/prompt binding	Apply alternatives to the same responses	comparative observations
Mediator/state	judge decisions or sufficient terminal state	Replay mediated or stateful endpoints	judge trace or execution state
Support	keys, DNF/missingness masks, denominators	Compare without silent eligibility change	support or missingness
Semantic registry	endpoint type, origin/admission, same target/support, units/ties	Form $(D,\mathcal{F},q)$	family or claim review

Appendix

Table 8: Preservation layers required by claim replay.

Reader label	Count	First unavailable requirement
Evidence binding	47	Source or observation bytes could not be jointly bound
Comparative observations	35	Fixed outputs for a common system/support set were absent
Judge trace	9	Responses, judge binding, and decisions were not jointly frozen
Endpoint type	4	Available outputs lacked a shared typed endpoint
Semantic target	4	Candidate metrics addressed different targets
Missingness / DNF	2	Missing outcomes lacked a declared non-ignorable rule

Appendix

Table 9: Primary stopping codes for the 103 stops on the outcome-blind 114-case path.

Panel of Figure 7 — Figure 7: First unavailable requirement among the 103 stops on the outcome-blind 114-case path. The distribution identifies release-interface bottlenecks; it is not a benchmark-quality score and not evidence of intervention utility.

Panel of Figure 8 — Figure 8: Claim resolution across the six complete audits. P/E denote primary/review scopes. The last column reports stable pair counts. AgentDojo and AutoML make the admission effect visible: their primary orders are identified even though their review-envelope orders are not. The BEIR row is scoped to its declared five-view family; the separate 25-view coverage sensitivity is reported below.

Panel of Figure 9 — Figure 9: Stable-backbone retention across declared complete-audit families. Bar length is the fraction of system pairs whose relation is invariant; labels give exact stable/total counts. The BEIR bar is 2/3 over the declared five views, not an evaluator-wide claim: its 25-view sensitivity yields 0/3.

Case	Audit intervention	Claim effect	Licensed lesson
AutoML	Reclassify average task rank from AP/D to SM/A after binding the canonical missing-value, fold, task-rank, and average-rank procedure	Ledger changes from 3A+2D to 4A+1D; winner, order image, 10/10 primary pairs, and 9/10 envelope pairs are unchanged	Provenance can be corrected without destabilizing inference
BEIR SciFact	Retain the declared five views, then separately expand to all 25 evaluator-exposed metric–cutoff cells executable from the top-100 runs	Winner ID $\to$ NID; order remains NID but expands to five weak orders; backbone 2/3 $\to$ 0/3	Mechanically correct arithmetic does not license transport beyond the named family

Appendix

Table 10: Provenance correction and family expansion are different perturbations.

Case	Perturbation	Winner after perturbation	Complete-order consequence
AgentDojo	common-key suite-macro ASR, D $\to$ A alone	Claude 3.5 Sonnet	ID; primary order unchanged
AgentDojo	utility-conditional ASR, D $\to$ A alone	Claude 3.5 Sonnet	NID; GPT-4o-mini/Gemini pair reverses
AgentDojo	worst-suite ASR, D $\to$ A alone	Claude 3.5 Sonnet	NID; GPT-4o-mini/Gemini pair reverses
AutoML	worst-task AUC, D $\to$ A alone	H2OAutoML	NID; AutoWEKA/TPOT pair reverses
BEIR SciFact	No D candidate; five $\to$ 25 is a coverage, not admission, perturbation	Non-identified	NID; all three pairs become unstable

Appendix

Table 11: One-at-a-time boundary perturbations propagated through the claim calculation.

Packet	Intact	Source	Claim	Endpoint	Observations
ComputeEval (021)	COMPLETE	G1 STOP	G2 STOP	G3 STOP	G4 STOP
MLRC-Bench (043)	COMPLETE	G1 STOP	G2 STOP	G3 STOP	G4 STOP
TAC (086)	COMPLETE	G1 STOP	G2 STOP	G3 STOP	G4 STOP
UCCB (101)	COMPLETE	G1 STOP	G2 STOP	G3 STOP	G4 STOP
Pre-Flight (105)	COMPLETE	G1 STOP	G2 STOP	G3 STOP	G4 STOP

Appendix

Table 12: Controlled evidence-ablation conformance. Cells report the replay terminal after the named single-class deletion.

Case	Design	Winner P/E	Order P/E	Pairs P/E
TheAgentCompany	random complete	NID/NID	NID/NID	0/1 $\to$ 0/1
BBEH	random complete	NID/NID	NID/NID	0/1 $\to$ 0/1
Personality	random complete	NID/NID	NID/NID	0/3 $\to$ 0/3
AgentDojo	purposive	ID/ID	ID/NID	3/3 $\to$ 2/3
AutoML	purposive	ID/ID	ID/NID	10/10 $\to$ 9/10
BEIR SciFact	purposive	ID/ID	NID/NID	2/3 $\to$ 2/3

Appendix

Table 13: Claim-relative family stress. P/E denote primary/review; pair entries are stable/total.

Cohort	Object	Controlling audit outcome
Random 10	GDM InterCode CTF	stopped after semantic candidates: comparative observation absent
Random 10	TheAgentCompany	complete finite-family audit
Random 10	FrontierScience	stopped after semantic candidates: model judge trace absent
Random 10	BBEH	complete finite-family audit
Random 10	Agent Bench	stopped after semantic candidates: stateful execution artifact absent
Random 10	CORE-Bench	stopped after semantic candidates: stateful execution artifact absent

Appendix

Table 23

System/artifact	Specification output	Exact reconstructed value
inspect_haiku45	any_progress	0.6176470588235294
inspect_haiku45	mean_checkpoint	0.36264705882352943
inspect_haiku45	strict_completion	0.11764705882352941
original_sonnet4	any_progress	0.47058823529411764
original_sonnet4	mean_checkpoint	0.2164705882352941
original_sonnet4	strict_completion	0.14705882352941177

Appendix

Table 24

System/artifact	Specification output	Exact reconstructed value
inspect_reproduction	harmonic_no_pseudocount	0.0425272071389958
inspect_reproduction	harmonic_plus_0_01	0.07898405712442381
inspect_reproduction	sample_micro	0.2150442477876106
inspect_reproduction	task_macro_arithmetic	0.22086956521739132
original_paper	harmonic_no_pseudocount	0.0
original_paper	harmonic_plus_0_01	0.060462587043807435

Appendix

Table 25

System/artifact	Specification output	Exact reconstructed value
claude3_opus	negative_mae_all8	-4.825000000000001
claude3_opus	negative_mae_big5	-7.300000000000002
claude3_opus	negative_rmse_all8	-6.252199612936236
claude3_opus	pairwise_rank_agreement_all8	1.0
claude3_opus	pearson_all8	0.9977220344447256
gpt35_turbo	negative_mae_all8	-4.04125

Appendix

Table 26

Case	Compared readings	Candidate status	Claim resolution	Boundary
VQA-RAD	overall micro vs equal closed/open macro	second aggregation audit-proposed	exact non-id ; winner/order id ; 1/1 stable	source does not declare equal answer-type overall
MORU	sample-first overall vs equal 16-dimension macro	second aggregation audit-proposed	exact non-id ; winner id ; order non-id ; 9/10 stable	dimension vector exposed; second overall not declared
InstrumentalEval	prompt-global vs equal six-type macro	second aggregation audit-proposed	single-system exact non-id	per-type metrics do not declare a second overall
ANIMA	historical overall vs dimension-normalised average	both source-declared	exact non-id ; winner id ; order non-id ; 5/6 stable	table-level reconstruction at displayed precision
AIR Bench	item-micro vs equal-category macro delta	second aggregation audit-proposed	exact delta non-id ; direction id	category rows do not declare a macro headline
LiveBench	subtask–category vs item–category macro	both source/evaluator-exposed	exact non-id ; winner/order id ; 1/1 stable	historical aggregates and displayed precision only

Appendix

Table 18: Claim-resolution profile for eight exploratory cases.

Panel of Figure 10 — Figure 10: Mechanism map for the eight exploratory cases. Position is a qualitative routing summary, not a statistical embedding. It separates harmless representation changes, exact-value sensitivity, lower-order changes, and the BFCL support-limited winner candidate.

System	Sample-first	Equal 16-dimension macro
GPT-5.2	(421/500)	(12593/16000)
GPT-5-mini	(102/125)	(3117/4000)
Claude Haiku 4.5	(37/50)	(11413/16000)
Gemini 2.5 Flash Lite	(713/1000)	(2157/3200)
Grok 4.1 Fast	(711/1000)	(5407/8000)

Appendix

Table 19: MORU exact rational reconstruction; the equal-dimension macro is audit-proposed.

System	Historical overall	Dimension-normalised
Claude Haiku 4.5	(639/1000)	(8009/13000)
Gemini 2.5 Flash Lite	(161/250)	(3971/6500)
Grok 4 Fast	(17/25)	(8821/13000)
GPT-5-mini	(391/500)	(9681/13000)

Appendix

Table 20: ANIMA table-level reconstruction; both columns are source-declared.

System	Count-weighted micro	Equal ten-group macro
Claude Haiku 4.5	(16701/20600)	(5767/7500)
GPT-4.1-mini	(225821/283250)	(11639/15000)

Appendix

Table 21: BFCL calculation on the 3,399-item published-category subset.

Panel of Figure 11 — Figure 11: Independent recomputation localizes each order change. MORU and ANIMA preserve the winner and change one lower pair; BFCL changes the only pair but remains a support-limited exploratory result.

Group	Required content
Claim and substrate	case/query identifiers; endpoint type, orientation, unit, tie rule; systems and common observation/support keys; scorer, judge, or state binding
Candidate ledger	formula or implementation; variation layer; origin; reliability result; A/D/X admission and rationale; same-target/support decision
Aggregation and support	denominator and aggregation tree; missingness/DNF rule; weights; unit transform; declared support changes
Exposure and freeze	evidence horizon; pre-outcome ledger digest; exposure state; append-only amendments
Output or stop	primary/review values, top sets, weak orders, pair relations, witnesses; otherwise stop stage, missing object, no-imputation rationale, and minimum unblock

Appendix

Table 22: Minimal field groups for an executable claim-identification record.

Panel of Figure 12 — Figure 12: Executable contract. Human judgment fixes type and admission before this state machine. Thereafter both family projection and identified-set enumeration are deterministic; a missing critical binding exits through a typed stop.

Panel of Figure 13 — Figure 13: The prospective question is deliberately separate. The present paper supplies the intervention targets and measurement contract; it does not claim the three future endpoints.

Figure 14: Controlled diagnostic specifications. Object contract, disposition policy, and terminal semantics index the downstream endpoint. Sampling precision inside one box does not integrate over an omitted semantic coordinate.

Disposition	Original asymmetric	Retain both	Exclude both	Aggregate symmetric minus original
Empty/empty credit	$-43.03[-50.82,-35.14]$	$0[0,0]$	$0[0,0]$	$+43.03[+35.14,+50.82]$
Empty no credit	$-41.30[-48.91,-33.70]$	$0[0,0]$	$0[0,0]$	$+41.30[+33.70,+48.91]$
Conditional nonempty	$-40.76[-48.61,-32.98]$	$0[0,0]$	$0[0,0]$	$+40.76[+32.98,+48.61]$

Appendix

Table 23: Task-disjoint adoption effects and frozen 95% task-cluster percentile intervals, in percentage points.

Figure 15: Controlled result: effect typing and ranking structure separate. Panel A shows that symmetric terminal handling collapses the contrast-comparable effect to zero. Panel B shows that complete rankings can remain non-identified while a unique winner or stable relations survive.

Treatment	PageRank–Earliest intervals	Weak orders	Reversal pairs	Kendall $\tau_{b}$	Possible winners
Retain both	$B/C/N:[43.84,55.43]\,pp$	3	1	$[0.5534,1]$	5
Exclude both	$B:[-0.72,13.41];C:[2.54,14.13];N:[2.54,14.66]\,pp$	5	6	$[0.4286,1]$	1

Appendix

Table 24: Task-disjoint ranking summaries under contrast-comparable terminal treatments.

Policy	Average effect (95% CI)	Interpretation
Empty/empty credit	$-7.72[-12.10,-3.92]\,pp$	Fixed 217-row total-score estimand; cellwise crossing
Empty no credit	$-3.46[-5.99,-1.15]\,pp$	Same rows and denominator; crossing removed
Conditional nonempty	$-0.89[-4.96,+2.87]\,pp$	Narrower support and different endpoint

Appendix

Table 25: Empty-family semantics on fixed or explicitly narrowed support.

Benchmark Auditing

Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.

See all 97 papers

ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations
Sep 2, 2026Peiying Zhu, Sidi ChangClaim Verification Data Provenance
Blossom AI Blossom AI Labs San Francisco, CA, USA
Are We Measuring Anticipation? Auditing Privileged Information in Procedural Video Evaluation
Oct 2, 2026Mahsa Mohammadi, Sareh RowlandsBenchmark Auditing Temporal Video Understanding
University of Exeter Exeter, United Kingdom
ReplayLens: Auditing Agents' Use of Outcomes
Sep 28, 2026Dong Xu, Zhangfan Yang, Jiantao Wu +5AI Agent Auditing LLM Agent Memory
School of Artificial Intelligence, Shenzhen University · EasternDawn · School of Computer Science, University of Nottingham Ningbo

Sep 2, 2026J/K move · Enter open · S save

ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations

Peiying Zhu, Sidi Chang

Blossom AI Blossom AI Labs San Francisco, CA, USA

Claim Verification Data Provenance Agent Evaluation

Open paper AskPDF arXiv

Agent evaluations face two distinct evidentiary questions: whether a reported claim is recomputable from retained evidence (sufficiency), and whether the retained records cover the committed experiment set (coverage). Generic logs and hash-linked transcripts answer neither reliably. We introduce ClaimReceipt, a claim-relative receipt specification and selective verifier that binds typed transaction evidence to a signed experiment manifest and returns PASS, INVALID, or INCONCLUSIVE per claim. We freeze the specification before implementation (SHA-256 18d109...b81). On 1,392 historical buyer--seller records, a CR-2 verifier reproduces all five manually labeled audit verdicts, exactly replays 600 deterministic and 792 post-generation records, makes every one of 13 declared field groups non-redundant under tested ablations, and returns the expected result on 11/11 semantic faults with 0/8 false positives. We then run a separate prospective CR-3 epoch: 30 assignments are committed before inference, terminal receipts are signed and chained, and private evidence is encrypted for an auditor. Complete evidence yields coverage and accounting PASS; withholding one terminal receipt returns INCONCLUSIVE_COVERAGE, while withholding all private openings preserves coverage and protocol verification but makes economic claims inconclusive, exactly matching a preregistered prediction. Receipt instrumentation adds 0.021% of model-inference time and 9.9 KB per transaction. A specification-legibility probe indicates that our own frozen specification is not yet unambiguous to an independent reader. Claim verification therefore requires both claim-sufficient evidence and a committed universe against which omissions become visible.

Abstract

Benchmark Auditing

Explore similar work

ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations

Are We Measuring Anticipation? Auditing Privileged Information in Procedural Video Evaluation

ReplayLens: Auditing Agents' Use of Outcomes