When Can You Trust an Agent's Own Verifier? Stress-Testing Counterfactual Rationale Verification Under Search Pressure and Oracle Access

Jonghyun Lee; Moonsu Kim; Sunghwan Park

cs.AIOct 16, 2026

When Can You Trust an Agent's Own Verifier? Stress-Testing Counterfactual Rationale Verification Under Search Pressure and Oracle Access

Authors: Jonghyun Lee, Moonsu Kim, Sunghwan Park

Organizations: Yonsei University

Abstract

Automated agent design increasingly trusts a single score to decide whether a change is an improvement -- but a score optimized aggressively enough ceases to measure what it was meant to. When an agent also explains its design, that explanation is a second signal we can check without human labels: negate each stated claim, re-run, and observe whether performance drops. We ask whether such a verifier stays reliable once an agent optimizes against it. Our contribution is not a new verifier but a way to stress-test one: a pressure ratio $ρ$ measuring how much the verifier steers the agent's search, and four falsifiable probes covering four failure modes -- giving no signal, being too weak to matter, being gamed by stating fewer claims, and being queried before the agent commits. We apply the test to our own verifier, the Rationale Consistency Score (RCS), inside an LLM agent that designs machine-learning pipelines, with one generator model, six tasks, and three seeds. First, in our prior configuration the verifier barely affects the search objective ( $ρ\ll 1$ ), partly because of an accuracy gate, so its robustness had not been tested; it still shapes the reported artifact through a final-selection tie-break that $ρ$ does not register. Second, a capability control shows the agent shortens its claim lists when instructed, yet search does not, even at $ρ$ over 100 times the prior level; since feedback omits claim volume and horizons are short, this null result is largely structural rather than evidence of robustness. Third, when the agent can probe the verifier before committing, it keeps claims that pass while held-out faithfulness falls -- a decline invisible to in-loop metrics, though partly confounded by how RCS scores small pipelines. We offer these as preliminary evidence and propose a reporting standard: audit a verifier with $ρ$ and these probes before trusting it.

Figures & tables

Panel of Figure 1 — Figure 1: Multi-agent architecture of the stress-test loop. The Design Agent proposes a pipeline and rationale in a single call, and through the tool interface only in the P2 arms (Section 6.3 ); the Verifier Agent parses claims, builds counterfactuals, and returns $\widehat{\mathrm{RCS}}$ ; the Search Agent forms $R=\hat{Q}+\lambda_{t}\widehat{\mathrm{RCS}}$ and updates Thompson Sampling. Cross-agent edges carry the artifact, oracle probes (F4, oracle arm only), task/verifier scores, and the next-iteration recommendation. Tags F1–F4 mark where each protocol probe attaches.

Dataset	Modality	$n$	$d$	Notes
OpenML Credit-G	tabular	1002	20	credit scoring
UCI Breast Cancer	tabular	288	9	binary recurrence
Madelon	tabular	2002	500	synthetic, high- $d$
Ecommerce Text	text	50427	1200	TF-IDF features
Butterfly Image	image	6499	26	75-class species ID
Electricity	time-series	45312	23	ELEC2, up/down

Table 1: Design tasks and modalities; sources and feature construction are in Appendix B .

n

counts the rows actually ingested, which for four tasks exceeds the canonical size by two.

Dataset	Objective	Test acc.	Held-out RCS	Claims/iter	$\rho$
Credit-G	RCS included	0.723 [0.682, 0.751]	0.037 [0.010, 0.062]	11.6	0.012
Credit-G	RCS excluded	0.697 [0.682, 0.706]	0.016 [0.002, 0.029]	11.3	0
Breast	RCS included	0.724 [0.690, 0.741]	0.011 [0.004, 0.015]	9.3	0.010
Breast	RCS excluded	0.718 [0.707, 0.741]	0.014 [0.010, 0.020]	10.1	0
Madelon	RCS included	0.741 [0.713, 0.771]	0.109 [0.005, 0.185]	20.4	0.089
Madelon	RCS excluded	0.759 [0.738, 0.771]	0.146 [0.101, 0.173]	17.0	0

Table 2: Main configuration with dynamic

\lambda

, three seeds per dataset. Brackets: bootstrap 95% CIs. Held-out RCS is an uncapped raw measure.

Figure 2: Dose–response across six datasets (three seeds each); shading marks deployment. (a) Over $>2$ orders above it, claims show no significant downward trend (Spearman $\in[-0.12,+0.38]$ , all $p>0.15$ ); crosses mark lower levels reached under instructed minimization on four responsive datasets. (b) Uncapped held-out RCS shows no consistent increase with $\rho$ .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	A	B	$\Delta$	A $\to$ B	ratio	$\|\Delta\|$
	Claims/iter			Held-out RCS		Test acc.
Credit-G	10.6	11.5	$+7.7\%$	0.0222 $\to$ 0.0235	$\times$ 1.06	0.010
Breast	10.2	9.9	$-2.7\%$	0.0099 $\to$ 0.0061	$\times$ 0.61	0.011
Madelon	20.3	18.9	$-6.7\%$	0.0783 $\to$ 0.0766	$\times$ 0.98	0.022
Ecommerce	9.5	10.5	$+11.4\%$	0.1493 $\to$ 0.1691	$\times$ 1.13	0.048
Butterfly	13.9	14.5	$+4.1\%$	0.0554 $\to$ 0.0792	$\times$ 1.43	0.017

Appendix

Table 3: Identical-configuration replicate. Run A is the

\lambda_{\mathrm{nom}}=5

arm of the dose–response sweep (Table 7 ); Run B is the default arm of the capability control (Table 5 ). Both fix static

\lambda_{\mathrm{nom}}=5

and seeds

42/43/44

and apply no policy prompt, so they differ only in being invoked twice. Three-seed means; held-out RCS is uncapped.

Arm	Runs	Flip rate	Acc. conceded (mean/max)
deployed (dynamic $\lambda$ )	18	5/18 (28%)	0.000 / 0.000
$\lambda_{\mathrm{nom}}=0$	18	0/18 (0%)	0.000 / 0.000
$\lambda_{\mathrm{nom}}=0.12$	18	6/18 (33%)	0.003 / 0.009
$\lambda_{\mathrm{nom}}=1$	18	10/18 (56%)	0.011 / 0.044
$\lambda_{\mathrm{nom}}=5$	18	14/18 (78%)	0.034 / 0.209
$\lambda_{\mathrm{nom}}=20$	18	13/18 (72%)	0.036 / 0.192

Appendix

Table 4: Selection flip rate: how often the verifier term decides which pipeline a run reports. For every run we re-select over the iterations it actually executed, comparing

\arg\max_{t}\,[\hat{Q}(P_{t})+\lambda_{t}\widehat{\mathrm{RCS}}(P_{t})]

with

\arg\max_{t}\,\hat{Q}(P_{t})

. Acc. conceded is the validation accuracy the flipped choice gives up, so

0.000

means the term broke a tie. Post-hoc re-selection, not a re-run: removing the term would also change the trajectory.

Dataset	Policy	Claims/iter	Verified rate	Held-out RCS
Credit-G	default	11.5	0.52	0.023 [0.008, 0.033]
Credit-G	minimize	9.4	0.45	0.014 [0.009, 0.022]
Breast	default	9.9	0.36	0.006 [0.004, 0.010]
Breast	minimize	9.8	0.41	0.010 [0.001, 0.020]
Madelon	default	18.9	0.57	0.077 [0.006, 0.123]
Madelon	minimize	13.9	0.60	0.136 [0.101, 0.162]

Appendix

Table 5: Capability control at nominal

\lambda_{\mathrm{nom}}=5

: default/minimize claim policies, three seeds.

Figure 3: Capability control at $\lambda_{\mathrm{nom}}=5$ : claim volume under the default vs. minimize policy (bars, bootstrap 95% CIs). The dashed line is the mean claim volume reached by the nominal- $\lambda$ sweep under search; instruction drives volume below it on the four datasets where search never did. Percent labels give the minimize reduction relative to default.

Dataset	Arm	Probes/iter	Claims/iter	Verified rate	Test acc.	Held-out RCS
Credit-G	tools	0.0	12.4	0.40	0.718 [0.682, 0.756]	0.034 [0.025, 0.051]
Credit-G	tools+probe	3.1	11.7	0.50	0.725 [0.701, 0.741]	0.016 [0.010, 0.020]
Breast	tools	0.0	10.5	0.38	0.690 [0.672, 0.724]	0.008 [0.004, 0.011]
Breast	tools+probe	3.6	10.3	0.41	0.701 [0.655, 0.724]	0.009 [0.004, 0.014]
Madelon	tools	0.0	25.6	0.50	0.792 [0.771, 0.810]	0.188 [0.142, 0.250]
Madelon	tools+probe	3.4	14.8	0.51	0.787 [0.758, 0.808]	0.095 [0.005, 0.274]

Appendix

Table 6: Probe access to the verifier, three seeds. Held-out RCS is uncapped.

Dataset	$\lambda_{\rm nom}$	$\rho$	Claims/iter	Verified rate	Chars	Held-out RCS
Credit-G	0	0.00	10.8	0.51	282	0.053 [0.038, 0.066]
	0.12	0.07	11.3	0.56	333	0.047 [0.034, 0.068]
	1	1.05	12.2	0.52	297	0.019 [0.012, 0.033]
	5	4.42	10.6	0.48	320	0.022 [0.005, 0.051]
	20	11.58	11.0	0.53	285	0.044 [0.030, 0.070]
Breast	0	0.00	9.9	0.42	698	0.009 [0.005, 0.012]

Appendix

Table 7: Dose–response to nominal static verifier weight, three seeds; dataset scale

s_{d}

remains active. Held-out RCS is uncapped and

\rho

measures pressure.

Figure 4: Oracle access signature (three seeds averaged). Each arrow runs from the tools arm (circle) to the tools+probe arm (triangle); marker area is proportional to uncapped held-out RCS. The shaded upper-left quadrant is the selection regime — fewer claims at a higher verified rate. Arrows point into it on four datasets; held-out fidelity (marker area) drops with non-overlapping CIs only on Credit-G and is suggestive but noisy across seeds elsewhere.

Dataset	$\lambda_{\rm nom}$	$\rho$	Claims/iter	Verified rate	Chars	Held-out RCS
Credit-G	0	0.00	10.6	0.52	290	0.045 [0.039, 0.051]
	0.12	0.12	10.7	0.53	275	0.028 [0.015, 0.043]
	1	0.55	11.2	0.54	300	0.021 [0.005, 0.039]
	5	6.46	10.1	0.46	281	0.029 [0.024, 0.034]
	20	9.12	10.7	0.55	278	0.035 [0.025, 0.047]
Breast	0	0.00	10.0	0.38	634	0.010 [0.001, 0.015]

Appendix

Table 8: Dose–response under deterministic UCB1 with the base cap doubled to 30 before applying the dataset-specific iteration-budget multiplier (typically 34–38 iterations; two

\lambda_{\rm nom}=0

runs used 18 and 26). The nominal static-weight sweep matches Table 7 ; held-out RCS is uncapped and

\rho

measures pressure. Claim-count curves are nonmonotonic in

\lambda_{\rm nom}

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 142 papers

Oct 1, 2026

Verify Claims, Not Scores: Evidence-Based Verification of Modular Agents

When developers change one component of an agent, such as its controller, a learned model or its verifier, they usually judge the change by an aggregate task score. That score cannot tell whether improvement was attainable, which component lost value, or what the agent's own checks certify. We introduce a claim-specific verification audit for modular agents that plan, act, check and refine. Instead of scoring the agent, the audit scores the evidence: each conclusion is recorded with the evidence behind it, one of four verdicts (supported, unsupported, unresolved or not evaluated) and the boundary within which it holds. Three tools supply that evidence. Oracle policies measure attainable improvement under an explicitly stated action set, so that a low value can be traced to the evaluation rather than to the environment. Replacing one component at a time with a perfect counterpart locates lost value, with null results read as unresolved whenever a downstream component could mask them. A separate test asks whether the verifier's score identifies the quantity it is read as bounding. Applied to a constrained portfolio-allocation agent in a synthetic market with known hidden regimes, the audit shows that the value of perfect regime information depends on the action set used to measure it, that the scenario generator discards most of the regime signal while better local fidelity does not improve decisions, and that the runtime verifier can be bypassed with no visible change in outcomes. The contribution is the protocol and the evidential distinctions it enforces; the empirical findings are specific to the agent and environment studied.

Ali Atiah Alzahrani

AI Agent Auditing Agent Evaluation

Ask PDF

Oct 19, 2026

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.

Jianing Yin, Ziyi Chen, Hao Wang +4

University of Pennsylvania · University of Illinois at Urbana-Champaign · University of California, Berkeley +2

LLM Agent Verification AI Agent Benchmarks Automated Agent Evaluation

Ask PDF

Oct 8, 2026

TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation

Verifier scores now serve as both benchmark metrics and training rewards for large language model (LLM) agents, and a change in score is routinely read as a change in capability. It may instead reflect a change in the evaluation. We introduce TRACE, a protocol that turns a score change from a verdict into a testable diagnosis: it applies a targeted change to one part of an evaluation, compares paired runs, checks whether the agent's behavior changed, and rescores unchanged trajectories to test whether the scoring rule is responsible. In a controlled suite of 25 synthetic tasks, renaming tools lowers a scripted agent's score by 0.250 even though it performs exactly the same operations; restoring the original names at scoring time closes the entire gap, while the same mutation exposes a genuine behavioral failure in a second agent. On public

τ^2

-bench tasks with four LLM agents, an initial 30-task study finds mixed reward changes whose one clear effect does not replicate. In a larger follow-up on 88 new tasks with repeated runs per condition, renaming tools or reformatting tool outputs leaves reward unchanged to within

\pm

0.10 for seven of eight agent-change pairs, whereas tool names that deliberately mislead lower every agent's reward by 0.20-0.44, showing that the setup can detect real effects. Identical reruns flip 15-36% of task outcomes, so single-run comparisons cannot separate presentation effects from run-to-run variation. Two frontier LLM judges give consistent verdicts when a fixed trajectory is presented differently, yet disagree with each other on 57% of the same records, largely because one grades procedure rather than outcome. TRACE thus separates what a score change says about the agent from what it says about the measurement.

Radhika Gaonkar

Prime Intellect

LLM Agent Evaluation AI Agent Evaluation AI Agent Benchmarks

Ask PDF

Abstract

LLM Agent Verification

Explore similar work

Verify Claims, Not Scores: Evidence-Based Verification of Modular Agents

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation