cs.AIOct 3, 2026

Not Self-Decidable: LLMs Cannot Draw the Boundary of What an Agent Verifier Can Check

Authors: Anthony Rhodes

Organizations: Confidential Core AI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A verifier for an agent faces rules of two kinds: the ones a fixed check can settle and the ones that require a judge. A team that derives its own checks fixes that split up front. Where the requirements come from outside, as in finance, healthcare and law, the agent enforces rules it did not write, so the split falls to runtime, recurring for every predicate of every rule on every action at a rate no reviewer can audit. Every escalation scheme assumes a model can make that decision itself, that it is self-decidable. Across six corpora, including the EU AI Act, FINRA guidance and a deployed credit agent, we collect roughly 22,000 labels from four models built by three labs. They agree almost perfectly where the answer is obvious and collapse on regulatory text; their errors run in opposite directions, so no model can be trusted as the conservative choice; and on the deployed agent's own rule-set they err together, over-claiming that a fixed check will do, the direction that never gets escalated. We introduce CoVer (corroborate-then-verify), which treats unanimity as a nomination, admitting a predicate only when the check synthesized for it survives intervention, reading fields the agent cannot write and holding under deterministic rewording. That gate rejects most of what corroboration wrongly admits, at a cost in coverage we report rather than tune away. The obvious alternative, agreement with a reference judge, certifies nothing: it climbs from 30% to 77% across calibration bands while the genuinely decidable share does not move, because a judge drawn from the population under indictment ratifies the blind spot it shares. Self-decidability is not a capability to elicit from a model but a boundary the verifier must construct.

Figures & tables

Figure 1: The geometry of the failure . A Admitting a fixed procedure is a property of the criterion (schematic): the calibration endpoints sit far from the line, and predicates decomposed from regulatory text pile up on it. B Measured, not schematic. On the 231 regulatory-text predicates GPT-4o calls 41 decidable against Llama’s 145, its set lying inside Llama’s on all but 2; on the 73 agent rules the containment reverses, Llama’s 48 inside GPT-4o’s 62 on all but 1. The contested zone this leaves is 61% of our regulatory-text predicates, while the endpoints stay unanimous ( κ\kappa 0.92–0.99).
Figure 2: There is no conservative model to pick. On regulatory text (left) GPT-4o is the panel’s most cautious model, calling 1% of judgment predicates decidable; on an operational agent rule-set (right) the same model is the most reckless, at 71%. Every model shifts toward over-claiming on agent rules, the unsafe direction, and that is the setting in which a runtime verifier acts on the label without escalation. Denominators differ: over-claim over the true-judgment predicates (78 regulatory text, 24 agent rules), under-claim over the true-decidable ones (153 and 49).
Figure 3: CoVer end to end (Algorithm 1 ). Corroboration only nominates : a predicate the panel does not unanimously call decidable leaves at once. A nominated predicate is admitted only if the check synthesized for it clears all four gates. Both ways the router can err lead out along the same dashed rail, so the gate fails closed, and no model is consulted after nomination.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

RouterAdmittedPrec.Cov.Held
Majority ( ≥\geq 3 of 4)5692.9%44.4%–
Unanimous (4)2495.8%19.7%–
Graduated ( τ=1\tau=1 )4379.1%29.3%76.7%
Unanimous and graduated7100.0%6.0%76.2%
Appendix
Table 2: The agreement gate on the EU AI Act corpus, against the label routers. Label rows use all 175 labelled predicates, graduation rows the 173 the gate ran on. Held is agreement with the reference judge on held-out cases, which a label router cannot report because a label is not executable.
CorpusnnR-baseR-procR-spec
EU AI Act17565%59%34%
FINRA5652%55%32%
Agent rule-set7333%40%33%
Synthetic32823%23%11%
Endpoints1200%0%0%
Appendix
Table 3: Contested rate (no unanimous label across four models) by corpus and rubric. R-base is the published rubric.
Corpus3-model4-model
EU AI Act (regulatory text)54%65%
FINRA (regulatory text)43%52%
Synthetic (designed, gold labels)16%23%
Agent rule-set (operational, agentic)22%33%
Pooled (policy corpora)31%39%
Appendix
Table 4: Contested rate (no unanimous decidable/judgment label) by corpus and panel size. Measured ground-truth-free as cross-model disagreement.
If a feature (humility)What we observe
Models would hedge or abstain on contested itemsRubric forbids hedging; models comply confidently and disagree oppositely , confident contradiction, not shared doubt
No per-model directional bias (everyone equally unsure)Directional, model-specific bias, one model leans over-claim, another under-claim. Idiosyncratic priors, not calibrated uncertainty
Degradation everywhere (the task is just hard)κ\kappa 0.99 on clear endpoints, collapse only on the operationally critical middle, localized exactly where the routing decision is consequential
Each model internally consistent (a stable line)(§ E.2 , direct test) a model that flips its own label on meaning-preserving paraphrases is not tracking a property of the predicate
Appendix
Table 5: Predictions of the feature (humility) reading versus what we observe.
ModelParaphraseTemplate
Opus28% / 6% (z=3.15, p=0.002 )33% / 5%
GPT-4o15% / 8% (n.s.)44% / 7%
Sonnet25% / 8% (z=2.41, p=0.016 )13% / 10%
Llama-3.116% / 10%20% / 27%
Appendix
Table 6: Within-model flip rates (borderline / control) under two perturbation instruments. p-values vs. clean-case baseline.
Figure 4: Every pair of the four models, across three corpora. Where the answer is clear the panel is near-unanimous ( κ\kappa 0.92–0.99). On regulatory text it collapses to 0.20–0.50: the six lines fan out, and the worst pair agrees barely above chance. Agreement partially recovers on agent rules, but that recovery is not reassurance. It is where the models over-claim together (Fig. 2 ), and correspondingly where the precision of a unanimous panel falls from 97–99% to 83% (§ 8 ).
Figure 5: The dual cost. Rows are what a criterion is; columns are where the verifier sent it. The two errors lie on the anti-diagonal and are not mirror images: under-claiming is expensive and auditable, over-claiming is cheap and silent. No choice of model minimises both, since the bias runs in opposite directions in different models (§ 6 ).
More in this topic

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 147 papers

Explore similar work

CardsList
Oct 19, 2026

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.

Jianing Yin, Ziyi Chen, Hao Wang +4

University of Pennsylvania · University of Illinois at Urbana-Champaign · University of California, Berkeley +2

LLM Agent VerificationAI Agent BenchmarksAutomated Agent Evaluation
AskPDF
Sep 28, 2026

Certified Selective Automation of LLM Agent Evaluation

Evaluating LLM agents still ends with a human reading trajectories, because automatic judges carry no guarantee on how often they are wrong. We ask the operational question: what fraction of agent evaluation can a judge take over, with a certificate that the error rate among auto-decided trajectories stays below a budget alpha? Agent corpora resist the standard answer: many agents attempt the same tasks, so trajectories arrive in correlated clusters, and the i.i.d. certificates of existing selective-judging methods can overstate what is safe: a naive certificate can claim 98% automation while its realized error exceeds the budget in 17.5% of task resamples. We introduce a task-level bootstrap certificate that is valid in every regime we test while matching the naive certificate's coverage; finite-sample cluster-valid alternatives certify nothing at realistic task counts. Under this certificate, a 4B logprob judge trained with SFT and reject-weighted GRPO certifies 0.30-0.59 of evaluation on tool-use and web corpora at alpha=0.1, the only judge, among strongly elicited frontier models, certifying on both headline corpora. Certified coverage is predictable before training from base rate and discrimination alone (leave-one-corpus-out R^2=0.96). Finally, the certificate doubles as a self-training filter: pseudo-labels harvested inside certified regions have contamination bounded by alpha by construction (realized 0.000-0.041 across six harvests), letting a judge enter an unseen domain at in-domain strength with zero target training labels.

Chengguang Gan, Yunhao Liang, Qinghao Zhang +1

Independent Researcher · University of Chinese Academy of Sciences · Pusan National University +1

LLM-as-a-JudgeSelective PredictionLLM Agent Evaluation
AskPDF
Sep 14, 2026

Look Before You Leap: Pre-Action Verification for LLM Agents

An LLM agent acts on the world by emitting actions: shell commands to run, edits to apply. A wrong action does not always fail loudly; it can fail silently, producing a plausible but incorrect effect that raises no error. We argue that a cheap deterministic check, run before an action takes effect, is an effective and underused form of agent oversight, and we study it across two action modalities in one framework. The idea is to fix an action's correct effect by construction, before any executor runs, so that silent failure is measured directly and the verifier may abstain rather than guess. For shell commands, a static verifier over 9930 commands and 482 tools catches 95.8% of invalid commands at a 10.0% false-positive rate. Its syntax and binary checks are oracle-exact, giving zero false positives while catching half of all errors; the flag check is bounded only by help-text coverage and accounts for every false positive. For code edits, a benchmark of 640 edits over 224 files isolating the apply step exposes a sharp split. Content-anchored formats such as search/replace and diff fail cleanly, whereas location-anchored formats fail silently: line numbers corrupt 99.1% of files under a one-line shift, and function-name edits hit the wrong function 12.7% of the time. In both settings a refuse-when-unsure policy turns silent failures into recoverable ones at a tunable cost in applicability: selective grounding reaches 0.958 recall at 7.0% false positives, and an anchor-and-verify applier records one silent misapplication in 8320 trials (0.01%). We release both benchmarks, the verifiers, and the guards.

Asaad Althoubi

Oklahoma State University Stillwater, OK USA

LLM Agent VerificationAI Agent Benchmarks
AskPDF