Not Self-Decidable: LLMs Cannot Draw the Boundary of What an Agent Verifier Can Check

Anthony Rhodes

cs.AIOct 3, 2026

Not Self-Decidable: LLMs Cannot Draw the Boundary of What an Agent Verifier Can Check

Authors: Anthony Rhodes

Organizations: Confidential Core AI

Abstract

A verifier for an agent faces rules of two kinds: the ones a fixed check can settle and the ones that require a judge. A team that derives its own checks fixes that split up front. Where the requirements come from outside, as in finance, healthcare and law, the agent enforces rules it did not write, so the split falls to runtime, recurring for every predicate of every rule on every action at a rate no reviewer can audit. Every escalation scheme assumes a model can make that decision itself, that it is self-decidable. Across six corpora, including the EU AI Act, FINRA guidance and a deployed credit agent, we collect roughly 22,000 labels from four models built by three labs. They agree almost perfectly where the answer is obvious and collapse on regulatory text; their errors run in opposite directions, so no model can be trusted as the conservative choice; and on the deployed agent's own rule-set they err together, over-claiming that a fixed check will do, the direction that never gets escalated. We introduce CoVer (corroborate-then-verify), which treats unanimity as a nomination, admitting a predicate only when the check synthesized for it survives intervention, reading fields the agent cannot write and holding under deterministic rewording. That gate rejects most of what corroboration wrongly admits, at a cost in coverage we report rather than tune away. The obvious alternative, agreement with a reference judge, certifies nothing: it climbs from 30% to 77% across calibration bands while the genuinely decidable share does not move, because a judge drawn from the population under indictment ratifies the blind spot it shares. Self-decidability is not a capability to elicit from a model but a boundary the verifier must construct.

Figures & tables

Panel of Figure 1 — Figure 1: The geometry of the failure . A Admitting a fixed procedure is a property of the criterion (schematic): the calibration endpoints sit far from the line, and predicates decomposed from regulatory text pile up on it. B Measured, not schematic. On the 231 regulatory-text predicates GPT-4o calls 41 decidable against Llama’s 145, its set lying inside Llama’s on all but 2; on the 73 agent rules the containment reverses, Llama’s 48 inside GPT-4o’s 62 on all but 1. The contested zone this leaves is 61% of our regulatory-text predicates, while the endpoints stay unanimous ( $\kappa$ 0.92–0.99).

Panel of Figure 2 — Figure 2: There is no conservative model to pick. On regulatory text (left) GPT-4o is the panel’s most cautious model, calling 1% of judgment predicates decidable; on an operational agent rule-set (right) the same model is the most reckless, at 71%. Every model shifts toward over-claiming on agent rules, the unsafe direction, and that is the setting in which a runtime verifier acts on the label without escalation. Denominators differ: over-claim over the true-judgment predicates (78 regulatory text, 24 agent rules), under-claim over the true-decidable ones (153 and 49).

Panel of Figure 3 — Figure 3: CoVer end to end (Algorithm 1 ). Corroboration only nominates : a predicate the panel does not unanimously call decidable leaves at once. A nominated predicate is admitted only if the check synthesized for it clears all four gates. Both ways the router can err lead out along the same dashed rail, so the gate fails closed, and no model is consulted after nomination.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Router	Admitted	Prec.	Cov.	Held
Majority ( $\geq$ 3 of 4)	56	92.9%	44.4%	–
Unanimous (4)	24	95.8%	19.7%	–
Graduated ( $\tau=1$ )	43	79.1%	29.3%	76.7%
Unanimous and graduated	7	100.0%	6.0%	76.2%

Appendix

Table 2: The agreement gate on the EU AI Act corpus, against the label routers. Label rows use all 175 labelled predicates, graduation rows the 173 the gate ran on. Held is agreement with the reference judge on held-out cases, which a label router cannot report because a label is not executable.

Corpus	$n$	R-base	R-proc	R-spec
EU AI Act	175	65%	59%	34%
FINRA	56	52%	55%	32%
Agent rule-set	73	33%	40%	33%
Synthetic	328	23%	23%	11%
Endpoints	120	0%	0%	0%

Appendix

Table 3: Contested rate (no unanimous label across four models) by corpus and rubric. R-base is the published rubric.

Corpus	3-model	4-model
EU AI Act (regulatory text)	54%	65%
FINRA (regulatory text)	43%	52%
Synthetic (designed, gold labels)	16%	23%
Agent rule-set (operational, agentic)	22%	33%
Pooled (policy corpora)	31%	39%

Appendix

Table 4: Contested rate (no unanimous decidable/judgment label) by corpus and panel size. Measured ground-truth-free as cross-model disagreement.

If a feature (humility)	What we observe
Models would hedge or abstain on contested items	Rubric forbids hedging; models comply confidently and disagree oppositely , confident contradiction, not shared doubt
No per-model directional bias (everyone equally unsure)	Directional, model-specific bias, one model leans over-claim, another under-claim. Idiosyncratic priors, not calibrated uncertainty
Degradation everywhere (the task is just hard)	$\kappa$ 0.99 on clear endpoints, collapse only on the operationally critical middle, localized exactly where the routing decision is consequential
Each model internally consistent (a stable line)	(§ E.2 , direct test) a model that flips its own label on meaning-preserving paraphrases is not tracking a property of the predicate

Appendix

Table 5: Predictions of the feature (humility) reading versus what we observe.

Model	Paraphrase	Template
Opus	28% / 6% (z=3.15, p=0.002 )	33% / 5%
GPT-4o	15% / 8% (n.s.)	44% / 7%
Sonnet	25% / 8% (z=2.41, p=0.016 )	13% / 10%
Llama-3.1	16% / 10%	20% / 27%

Appendix

Table 6: Within-model flip rates (borderline / control) under two perturbation instruments. p-values vs. clean-case baseline.

Panel of Figure 4 — Figure 4: Every pair of the four models, across three corpora. Where the answer is clear the panel is near-unanimous ( $\kappa$ 0.92–0.99). On regulatory text it collapses to 0.20–0.50: the six lines fan out, and the worst pair agrees barely above chance. Agreement partially recovers on agent rules, but that recovery is not reassurance. It is where the models over-claim together (Fig. 2 ), and correspondingly where the precision of a unanimous panel falls from 97–99% to 83% (§ 8 ).

Panel of Figure 5 — Figure 5: The dual cost. Rows are what a criterion is; columns are where the verifier sent it. The two errors lie on the anti-diagonal and are not mirror images: under-claiming is expensive and auditable, over-claiming is cheap and silent. No choice of model minimises both, since the bias runs in opposite directions in different models (§ 6 ).

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 147 papers

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
Oct 19, 2026Jianing Yin, Ziyi Chen, Hao Wang +4LLM Agent Verification AI Agent Benchmarks
University of Pennsylvania · University of Illinois at Urbana-Champaign · University of California, Berkeley +2
Certified Selective Automation of LLM Agent Evaluation
Sep 28, 2026Chengguang Gan, Yunhao Liang, Qinghao Zhang +1LLM-as-a-Judge Selective Prediction
Independent Researcher · University of Chinese Academy of Sciences · Pusan National University +1
Look Before You Leap: Pre-Action Verification for LLM Agents
Sep 14, 2026Asaad AlthoubiLLM Agent Verification AI Agent Benchmarks
Oklahoma State University Stillwater, OK USA

Oct 19, 2026J/K move · Enter open · S save

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Jianing Yin, Ziyi Chen, Hao Wang, Yiyou Sun+3

University of Pennsylvania · University of Illinois at Urbana-Champaign · University of California, Berkeley+2

LLM Agent Verification AI Agent Benchmarks Automated Agent Evaluation

Open paper AskPDF arXiv

Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.

Abstract

LLM Agent Verification

Explore similar work

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Certified Selective Automation of LLM Agent Evaluation

Look Before You Leap: Pre-Action Verification for LLM Agents