Verify Claims, Not Scores: Evidence-Based Verification of Modular Agents

Ali Atiah Alzahrani

cs.AIOct 1, 2026

Verify Claims, Not Scores: Evidence-Based Verification of Modular Agents

Authors: Ali Atiah Alzahrani

Abstract

When developers change one component of an agent, such as its controller, a learned model or its verifier, they usually judge the change by an aggregate task score. That score cannot tell whether improvement was attainable, which component lost value, or what the agent's own checks certify. We introduce a claim-specific verification audit for modular agents that plan, act, check and refine. Instead of scoring the agent, the audit scores the evidence: each conclusion is recorded with the evidence behind it, one of four verdicts (supported, unsupported, unresolved or not evaluated) and the boundary within which it holds. Three tools supply that evidence. Oracle policies measure attainable improvement under an explicitly stated action set, so that a low value can be traced to the evaluation rather than to the environment. Replacing one component at a time with a perfect counterpart locates lost value, with null results read as unresolved whenever a downstream component could mask them. A separate test asks whether the verifier's score identifies the quantity it is read as bounding. Applied to a constrained portfolio-allocation agent in a synthetic market with known hidden regimes, the audit shows that the value of perfect regime information depends on the action set used to measure it, that the scenario generator discards most of the regime signal while better local fidelity does not improve decisions, and that the runtime verifier can be bypassed with no visible change in outcomes. The contribution is the protocol and the evidential distinctions it enforces; the empirical findings are specific to the agent and environment studied.

Figures & tables

Practice	Question it answers	Null traced to harness	Masking-aware localisation	Quality / admiss. / provenance kept apart	Verifier score identification
Aggregate benchmarking	How well does the agent do?	–	–	–	–
Harness auditing	Is the grader correct?	partly	–	–	–
Component ablation	How much does removal hurt?	–	–	–	–
Runtime verification	Does a run meet a spec?	–	–	partly	–
Decision-focused learning	Does fidelity serve decisions?	–	partly	–	–
Claim-specific audit	Which claims does the evidence support?	✓	✓	✓	✓

Table 1: What each practice is typically designed to establish. Entries describe the usual focus of a family of methods, not the limits of every work within it. ✓: addressed by design; “partly”: addressed for a specific case; –: not the focus.

Figure 1: ARC-Agent , the agent under audit, and its three verification layers. Arrows trace one rebalancing decision at date

t

. (A) Information and model layer : a regime HMM turns past returns into regime posteriors

\pi_{t}

(walk-forward), and a regime-conditioned diffusion generator, whose base and crisis experts are mixed by a gate

g_{t}

and trained with a tail-weighted loss

\mathcal{L}_{\mathrm{tail}}

, draws return scenarios from which the risk quantities (

\mathrm{VaR}_{\alpha}

\mathrm{CVaR}_{\alpha}

) and moments

(\hat{\mu}_{t},\hat{\Sigma}_{t})

are computed. The red dashed box marks where the audit finds decision-relevant signal lost: the generator retains

29\%

of the true bull–crisis mean gap (a bootstrap retains

101\%

) while local checks pass (§ 5.2 ). (B) Epistemic evidence layer : the operational evidence filter scores each scenario set with calibration checks (KS, energy and variogram scores, unconditional coverage, CVaR error) and either accepts it or requests refinement, at most

B

times; it certifies no

\mathrm{CVaR}_{\alpha}

bound (§ 4.4 ). (C) Action-containment layer : the CVaR allocator solves a convex QP under the budget, box and turnover constraints. Its KKT and active-set record is optimality evidence for the solver output

w^{\mathrm{sol}}

, whereas the committed weights

w_{t}

receive only a post-hoc feasibility audit, which supports admissibility for the reported runs.

Figure 2: Three failure modes exposed by the audit (Sharpe; seeds and intervals in Table 14 ). (a) Representational underreach : regime-conditional weights (L2) sit

+0.38

above the static frontier at matched drawdown, but switching between two portfolios (L1) only

+0.18

, a paired gain over L0 of

+0.07\pm 0.08

. (b) Interventional diagnosis : the true-regime replacement recovers nothing and is unresolved because the generator can mask it; removing the turnover cap, adding scenarios and shrinking the mean also fail; only the full generator bypass recovers the gap (

+0.68

). (c) Joint-repair structure (exploratory; Sharpe only): the mean-plus-scenario arm improves over baseline; the

55\%

interaction share is a point estimate whose six-seed interval includes zero.

Verification claim	Evidence	Verdict	Boundary
Decision-relevant information is present and expressible	L2 information–action hierarchy	supported	U3, specified action set
ARC-Agent captures it	matched-risk frontier	unsupported	current architecture
Generator is an observed bottleneck	full generator bypass	supported	current trained generator
Improving perception would not improve downstream value	true-regime replacement	unresolved	masked by generator
Joint mean $+$ scenario repair improves on the all-generator baseline	$2^{3}$ coalition table	supported	U3; 6 seeds, exploratory; Sharpe only
Mean $\times$ scenario superadditivity	Shapley–Taylor interval	unresolved	interval includes zero at 6 seeds

Table 2: Claim ledger for ARC-Agent (boundaries expanded in App. L ). Unresolved is not a weaker form of supported ; it marks a claim the evidence cannot settle either way.

A5 validator compromise: specified but not executed; no robustness claim.
Attack	Claim surface	Effect on decision-quality evidence	Feas. audit	Solver prov.
A3 gate bypass	quality-verification layer	gate bypassed; metrics benign	passed	retained
A1 corrupted controller	untrusted configuration	degraded, within noise	passed	retained
A2 poisoned memory	untrusted memory	degraded, within noise	passed	retained
A4 adversarial controller	untrusted configuration	moved along frontier	passed	retained

Table 3: Attack surfaces and what survives each executed attack. Decision quality, admissibility and provenance are reported separately because they are different claims. Exploratory:

5

seeds, unpaired; levels in Table 11 . The feasibility audit is a post-hoc check, not a construction guarantee; solver provenance is a record for

w^{\mathrm{sol}}

, not KKT-optimality of

w_{t}

. A5 was not executed , so no claim covers compromise of the trusted path.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	Verification question	Evidence	Permitted claim
Q1	Was improvement attainable under an action set able to express it?	L0–L3 oracle Sharpe at fixed information, widening the action set (§ 5.1 , Table 10 ); 24 seeds, paired, confirmatory	Measured headroom depends on the action representation, not on the information alone
Q2	Which component replacement recovers downstream value?	Stage-wise oracle replacement with the true regime given to every arm (§ 5.2 , Table 5 ); 10–16 seeds, paired, confirmatory	A positive replacement identifies an observed bottleneck; a null one does not identify irrelevance
Q3	Does improved local fidelity verify downstream improvement?	Retention, directional cosine and vector error against paired Sharpe (§ 5.3 , Table 7 ); 6 seeds, exploratory	Improved local scores do not establish downstream improvement
Q4	Do joint repairs reveal structure that one-at-a-time checks miss?	Complete $2^{3}$ coalition table with Shapley and Shapley–Taylor summaries (§ 5.4 , Tables 6 , 15 ); 6 seeds, exploratory	The joint arm improves over the all-generator baseline; its conditional contribution and the interaction magnitude remain unresolved

Appendix

Table 4: Verification questions posed of ARC-Agent , the evidence used, and the claim that evidence permits. Each row concerns a different construct. Sharpe is an operational proxy for risk-adjusted decision value, not a general measure of agent quality. Verdicts are collected in Table 2 .

Intervention	Sharpe	Turn.%	% of gap recovered
I0 baseline (scenario moments, cap $0.20$ )	$0.465\pm 0.090$	$19.7$	n/a
I1 $+$ clean regime-conditional scenarios and moments (full generator bypass)	$\mathbf{1.141\pm 0.127}$	$18.8$	$\mathbf{+121\%}$
I2 $+$ no turnover cap	$0.377\pm 0.084$	$60.2$	$-16\%$
L2 regime-weight reference policy	$1.023\pm 0.111$	$8.8$	n/a

Appendix

Table 5: Oracle-replacement interventions (

10

seeds, paired, common random numbers; true regime given to every row). Only a full generator bypass recovers the measured gap; no other intervention recovers a detectable share of it. The factorial of Table 6 subsequently separates that single bypass into the conditional-mean, covariance and scenario tail/dependence channels; the two experiments use different baselines and seed counts and their levels are not directly comparable (§ 5.4 ). “% of gap recovered” is

(\text{Sharpe}-\text{Sharpe}_{\mathrm{I0}})/(\text{Sharpe}_{\mathrm{L2}}-\text{Sharpe}_{\mathrm{I0}})

\pm

values are across-seed standard errors over

10

seeds under common random numbers. L2 is a reference policy, not an upper bound : it optimises mean–variance per regime, so a CVaR allocator given the clean scenario law can exceed it.

Channels replaced with clean regime-conditional history	Sharpe	$\Delta$ Sharpe	$95\%$ CI
none (all generator; baseline)	$0.635$	n/a	n/a
covariance $\hat{\Sigma}$ only	$0.634$	$-0.001$	$\pm 0.001$
conditional mean $\hat{\mu}$ only	$0.655$	$+0.020$	$\pm 0.206$
scenarios only (tail $+$ dependence)	$0.713$	$+0.079$	$\pm 0.066$
mean $+$ covariance	$0.654$	$+0.020$	$\pm 0.212$
scenarios $+$ covariance	$0.714$	$+0.079$	$\pm 0.066$

Appendix

Table 6: The complete

2^{3}

factorial: all eight cells. Paired

\Delta

Sharpe against the all-generator baseline (

6

seeds, common random numbers,

95\%

CI over seeds). Reading down the single-channel rows, covariance is worth nothing; reading across the pairs, adding covariance to any other repair changes nothing (mean

+0.020

versus mean

+

covariance

+0.020

; scenarios

+0.079

versus scenarios

+

covariance

+0.079

), while mean

+

scenarios recovers essentially the whole effect and adding covariance on top of it does not improve matters (

+0.221

vs.

+0.218

). At six seeds the design is underpowered to distinguish superadditivity from additivity . Descriptively the joint-arm estimate exceeds either single-arm estimate; because the paired joint-versus- scenario-only contrast cannot be recovered from the retained summaries, neither superadditivity nor a nonzero conditional mean contribution is established. The interaction estimate and its seed-bootstrap interval are reported in Table 15 ; at six seeds that interval includes zero. These cells report Sharpe only. We did not record MaxDD or turnover per cell, so the risk-axis check we insist on elsewhere (§ 5.1 , § 6 ) has not been applied to the factorial: a channel repair that moved a cell along the risk frontier rather than off it would not be visible here.

Quantity	Definition	Reported values	Arm
Mean-gap retention	$\\|\Delta\hat{\mu}\\|_{1}/\\|\Delta\mu^{\mathrm{true}}\\|_{1}$	$29\%$ ( $90$ / $313$ bps)	deployed pipeline
same, step sweep	as above	$19$ / $30$ / $76\%$ ( $S{=}20$ / $50$ / $100$ )	step-count sweep
same, repair arm	as above	$47\%\to 186\%$	repair arm
Directional similarity	$\cos(\Delta\hat{\mu},\Delta\mu^{\mathrm{true}})$	$0.59\to 0.97$	repair arm
Size ratio	$\\|\Delta\hat{\mu}\\|_{2}/\\|\Delta\mu^{\mathrm{true}}\\|_{2}$	$208\%$ (overshoot)	repair arm
Relative vector error	$\\|\Delta\hat{\mu}-\Delta\mu^{\mathrm{true}}\\|_{2}/\\|\Delta\mu^{\mathrm{true}}\\|_{2}$	$0.82\to 1.18$	repair arm

Appendix

Table 7: Retention metrics.

\Delta\mu^{\mathrm{true}}

is the true bull-minus-crisis conditional mean vector and

\Delta\hat{\mu}

the generator’s. The first row is the headline

29\%

; the middle rows are the same scalar computed in other arms; the last two rows are the directional and vector diagnostics that show why the scalar is insufficient.

Method	Sharpe	MaxDD%	Turn.%	Frontier	Excess
Equal weight	$0.74\pm 0.46$	$18.5$	0.0	0.74	$+0.00$
Minimum variance	$0.37\pm 0.32$	$\mathbf{8.5}$	2.8	0.38	$-0.01$
Gaussian $\to$ QP	$0.41\pm 0.27$	$12.1$	19.7	0.60	$-0.19$
Block bootstrap $\to$ QP	$0.44\pm 0.28$	$10.3$	19.6	0.52	$-0.08$
Regime-cond. bootstrap $\to$ QP	$\mathbf{0.85\pm 0.10}$	$10.9$	19.6	0.53	$\mathbf{+0.32}$
ARC-Agent (full)	$0.56\pm 0.29$	$10.5$	18.1	0.60	$-0.04$

Appendix

Table 8: Does the complete system deliver genuine improvement at matched risk? ( exploratory , 5 seeds;

\pm

is the across-seed standard deviation, not a paired contrast; those are in Tables 5 – 6 .) The static frontier is the reference: “Excess” is Sharpe relative to the frontier at the same realised MaxDD, the comparator being the best static blend within the restricted EW–MinVar family. ARC-Agent does not outperform the frontier, whereas the regime-conditional bootstrap (ten lines, no generative model) does, which is what shows the environment contains usable regime information. The

-

CVaR ablation lies beyond the frontier’s risk endpoint (off-supp.), as does mean–variance (Table 9 ).

Method	Sharpe	MaxDD%	Turn.%
Risk parity	$0.52\pm 0.30$	$11.4$	$1.5$
Mean–variance	$0.71\pm 0.50$	$19.2$	$12.0$
Hierarchical risk parity	$0.36\pm 0.29$	$8.7$	$3.2$
Historical CVaR	$0.45\pm 0.29$	$10.1$	$19.6$
DRO-CVaR	$0.34\pm 0.27$	$9.7$	$19.9$

Appendix

Table 9: The five remaining implemented baselines ( exploratory ,

5

seeds; mean

\pm

across-seed standard deviation). None reaches the regime-conditional bootstrap’s

0.85

. Mean–variance attains the highest Sharpe of the five but at

19.2\%

drawdown, beyond the static frontier’s risk endpoint, the same risk-axis confound the frontier test exists to expose, and the reason it is not evidence against our conclusion.

Figure 3: Oracles versus the static EW/MinVar frontier (

24

seeds; bars are

\pm 1

s.e.; confirmatory, paired under common random numbers; Sharpe at

rf{=}1.6\%

). Construct: attainable decision value of perfect latent-state information under each action representation. The wider L2 and L3 action representations exhibit positive matched-risk oracle headroom in U3. ARC-Agent lies close to the evaluated static frontier under the reported matched-risk comparison. Unresolved: whether this headroom transfers beyond U3 or beyond the four action sets shown.

	Oracle over…	Sharpe	MaxDD%	Excess
L0	(no regime knowledge)	$0.78$	$12.4$	$0.00$
L1	two-point action set (EW $\leftrightarrow$ MinVar)	$0.84$	$16.0$	$+0.18$
L2	regime-conditional weights $w^{\star}(k)$	$\mathbf{1.06}$	$17.2$	$\mathbf{+0.38}$
L3	ex-post optimal weight path	$1.28$	$14.7$	$+0.62$
	ARC-Agent	$0.56$	$10.5$	$\mathbf{-0.04}$

Appendix

Table 10: Oracle hierarchy (

24

seeds, paired). All Sharpe levels in this table are computed at the preregistered

rf{=}1.6\%

, not at the

rf{=}0

used for the primary U3 metrics (App. Q ); the paired L2

-

L0 contrast is

+0.28\pm 0.08

here and

+0.248

rf{=}0

(§ 5.1 ). “Excess” is Sharpe over the static EW/MinVar frontier at the same drawdown , a different quantity from that paired contrast. The action set decides the answer: L1 is near-tautological, while L2, the oracle appropriate to a regime-conditional allocator, shows positive paired headroom under the wider action representation.

	Attack	Sharpe	MaxDD%	Infeas.	Solver record
ARC-Agent	none (base)	$0.56\pm 0.29$	$10.5$	0	100%
	A1 corrupted controller	$0.50\pm 0.41$	$12.4$	0	100%
	A2 poisoned memory	$0.52\pm 0.28$	$10.4$	0	100%
	A3 conformal-gate bypass	$0.55\pm 0.29$	$9.9$	0	100%
	A4 adversarial controller (injection proxy)	$0.83\pm 0.38$	$16.7$	0	100%
weights-emitting agent (no solver)		$0.50\pm 0.42$	$25.7$	345	0%

Appendix

Table 11: Red-team results, measured (

5

seeds;

\pm

is the across-seed standard deviation, not a paired contrast). Feasibility and provenance survive every executed attack (A1–A4), but so does feasibility for an agent that merely projects, which is why the surviving claim concerns auditability. “Infeas.” is a post-hoc audit, not a construction guarantee : every committed weight vector is checked against the budget (

10^{-4}

), box (

10^{-6}

) and turnover (

10^{-4}

) tolerances, and the column counts the rebalances that fail (App. O ). “Solver record” reports whether the underlying convex solve produced a complete active-set and KKT record; it does not imply that the post-processed committed vector is itself KKT-optimal. A5 (validator injection) was not executed and has no row. The two comparison rows are rule-based weights-emitting controls; no LLM was executed.

Benchmark
Setting	Value
Regimes $K$	$3$ (bull, stagnation, crisis); $K\in\{2,3,4\}$ searched
Instruments $d$	$10$ , of which $3$ defensive (negative crisis beta)
Transition matrix	Eq. ( 2 ); stationary $(0.669,0.283,0.048)$
Regime persistence	$(0.992,\,0.980,\,0.978)$ ; mean crisis episode $\approx 45$ days
Factor drift (annualised)	$(0.20,\,0.01,\,-0.55)$

Appendix

Table 12: Reproducibility summary for the measured (U3) experiments. All values are taken from the frozen preregistration and the implementation; the data-generating process, architecture and verifier are given in full in App. O .

Figure 4: The U3 benchmark, drawn on an illustrative

4{,}536

-day path from the same generator (panel (d) uses

2{,}600

-day paths per seed); this is for visualising the data-generating process and is not the evaluated horizon, which is

2{,}200

days per seed (

756

warm-up,

1{,}444

evaluated). (a) ground-truth regimes; (b) crisis returns are fat-tailed and left-skewed; (c) correlations and lower-tail dependence both spike in crisis, so diversification fails exactly when it is needed; (d) the detectability dial: at regime_separation

=0

balanced accuracy equals chance (

1/3

) by construction.

Method	Family	CVaR-QP	Reported in
Equal weight	baseline (classical)	n/a	Tab. 8
Minimum variance	baseline (classical)	n/a	Tab. 8
Risk parity	baseline (classical)	n/a	Tab. 9
Mean–variance	baseline (classical)	n/a	Tab. 9
Hierarchical risk parity	baseline (classical)	n/a	Tab. 9
Historical CVaR	baseline (scenario)	✓	Tab. 9

Appendix

Table 13: Method registry. Every method in the study, its family, whether it is fed by the shared CVaR-QP, and where it is reported. “Not reported” means implemented but not executed for this version, so no outcome has been observed; nothing here was run and then withheld.

Where	Quantity	Seeds	Paired	$\pm$ denotes	Status
Tab. 8	Sharpe by method	5	no	across-seed s.d.	exploratory
Tab. 5	Sharpe by intervention	10	yes	standard error	confirmatory
Tab. 6	$\Delta$ Sharpe, channels	6	yes	$95\%$ CI	exploratory
Tab. 10	oracle Sharpe	24	yes	point estimates	confirmatory
Tab. 11	Sharpe under attack	5	no	across-seed s.d.	exploratory
Tab. 9	remaining baselines	5	no	across-seed s.d.	exploratory

Appendix

Table 14: Seed and uncertainty registry. “Paired” means the arms share return paths under common random numbers, so the reported interval is on the contrast rather than on the level. Confidence intervals on paired contrasts are

95\%

percentile intervals from a bootstrap over seeds (

R=10{,}000

replications); because time is paired under common random numbers, resampling is over seeds, not over time. The stationary block bootstrap (block length

21

days) is a separate procedure applied to time-series Sharpe and CVaR differences (App. Q ), and the two should not be conflated. The last column of the table states which experiments are confirmatory and which are exploratory.

A. Component credit (Shapley values; sum to $v(N)$ )
	$\phi$	$95\%$ CI	share of $v(N)$
conditional mean $\hat{\mu}$	$+0.080$	$\pm 0.178$	$36.8\%$
scenario tail/dependence	$+0.139$	$\pm 0.086$	$63.8\%$
covariance $\hat{\Sigma}$	$-0.001$	$\pm 0.002$	$-0.6\%$
B. Interaction diagnostics (Shapley–Taylor indices; not added to A)
mean $\times$ scenarios	$+0.120$	$\pm 0.184$	$55.3\%$

Appendix

Table 15: Interventional attribution, measured. Shapley values and Shapley–Taylor interaction indices computed from the complete

2^{3}

coalition table (Table 6 ;

6

seeds, paired,

95\%

CI by seed bootstrap). The two blocks are alternative attribution summaries, not additive pieces of one decomposition : block A already sums to

v(N)=+0.218

exactly, as efficiency requires, and block B is a further diagnostic of non-additivity that must not be added on top of it. Covariance is inert with tight intervals throughout. The mean

\times

scenario interaction is

55\%

v(N)

as a point estimate and the total pairwise interaction share is

54\%

; both are underpowered at six seeds. These are descriptive attribution summaries: neither the interaction magnitude nor the conditional contribution of the mean channel is established by them.

Explore similar work

May 11, 2026cs.AI

Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation

Interactive agent benchmarks map an agent run to a binary outcome through outcome checks. When these checks rely on surface level signals or fail to capture the agent's actual action path, they cannot reliably determine whether the run succeeded. For example, a benchmark task may ask whether Alice's shipping address was changed, while the outcome check only verifies that the agent clicked "Save." This does not guarantee that the intended state change occurred, since the agent may have modified the wrong record. Treating such a run as successful therefore makes the reported score misleading. Benchmark quality thus depends not only on task design, but also on the reliability of outcome detection. We address this problem by introducing an outcome evidence reporting layer for existing benchmarks, without modifying their tasks, agents, or evaluators. The layer performs three functions. First, before scoring, it specifies which stored artifacts are required to verify the claimed outcome for each case. Second, it applies a locked checklist to each completed run and assigns one of three evidence labels: Evidence Pass, Evidence Fail, or Unknown. Third, it reports evidence supported score bounds that quantify uncertainty arising from Unknown cases. Rather than silently counting, discarding, or hiding uncertain cases inside a single aggregate success rate, the framework keeps them explicitly visible. We evaluate the outcome evidence layer on five public benchmarks: ANDROIDWORLD, AGENTDOJO, APPWORLD, tau3 bench retail, and MINIWOB. The resulting reports separate several empirically distinct failure modes.

Shanshan Gao, Liyi Zhou

The University of Sydney

Agentic Benchmarks Device Upper Bounds

Ask PDF

Sep 2, 2026cs.AI

ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations

Agent evaluations face two distinct evidentiary questions: whether a reported claim is recomputable from retained evidence (sufficiency), and whether the retained records cover the committed experiment set (coverage). Generic logs and hash-linked transcripts answer neither reliably. We introduce ClaimReceipt, a claim-relative receipt specification and selective verifier that binds typed transaction evidence to a signed experiment manifest and returns PASS, INVALID, or INCONCLUSIVE per claim. We freeze the specification before implementation (SHA-256 18d109...b81). On 1,392 historical buyer--seller records, a CR-2 verifier reproduces all five manually labeled audit verdicts, exactly replays 600 deterministic and 792 post-generation records, makes every one of 13 declared field groups non-redundant under tested ablations, and returns the expected result on 11/11 semantic faults with 0/8 false positives. We then run a separate prospective CR-3 epoch: 30 assignments are committed before inference, terminal receipts are signed and chained, and private evidence is encrypted for an auditor. Complete evidence yields coverage and accounting PASS; withholding one terminal receipt returns INCONCLUSIVE_COVERAGE, while withholding all private openings preserves coverage and protocol verification but makes economic claims inconclusive, exactly matching a preregistered prediction. Receipt instrumentation adds 0.021% of model-inference time and 9.9 KB per transaction. A specification-legibility probe indicates that our own frozen specification is not yet unambiguous to an independent reader. Claim verification therefore requires both claim-sufficient evidence and a committed universe against which omissions become visible.

Peiying Zhu, Sidi Chang

Blossom AI Blossom AI Labs San Francisco, CA, USA

Category-Aware Atomic Claims Transaction Evidence

Ask PDF

Jul 27, 2026cs.AI

Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation

A correct answer can conceal why an agent succeeded. Once agents change their information state during evaluation, correctness no longer distinguishes intended reasoning from answer acquisition. Outcome evidence and exposure detection do not establish whether success depended on an acquired target; we call this missing evaluation object success provenance. AcquaBench audits it through matched CLEAN, GOLD, and SHAM value substitution on four standardized surfaces with joint qid-clustered analysis. CLEAN retains benchmark-authorized information. GOLD makes the correct target available. SHAM preserves source structure and exposure opportunity but substitutes a matched incorrect value. GOLD minus CLEAN measures the total score response to correct-target availability; GOLD minus SHAM tests whether that response tracks target correctness beyond matched source exposure. In D0, GOLD exceeds SHAM by 19.1 to 25.9 percentage points, showing that success follows the correct value. In D2, GOLD still exceeds SHAM under distributed sufficiency while coloc no longer transfers as a high-score marker, with AUROC 0.376 and 0.142. Behavioral dependence can thus persist beyond this probe's intended observation unit. In model comparison, a supported 5.0-point CLEAN score gap compresses to a raw GOLD difference of -0.6 points without establishing rank inversion. Agent benchmarks should report success together with whether the evaluated information state supported it.

Jingkun Luo, Da-Tian Peng

Northwestern Polytechnical University

Agentic Evaluations Generation Provenance Provenance Silver Bullet Explainable AI Methods

Ask PDF