VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Jianing Yin; Ziyi Chen; Hao Wang; Yiyou Sun; Zijian Jin; Dawn Song; Daoan Zhang

cs.SEOct 19, 2026

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Authors: Jianing Yin, Ziyi Chen, Hao Wang, Yiyou Sun, Zijian Jin, Dawn Song, Daoan Zhang

Organizations: University of Pennsylvania · University of Illinois at Urbana-Champaign · University of California, Berkeley · New York University ·

Abstract

Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.

Figures & tables

Figure 1: VerifierBench overview. Source tasks and source verifiers are combined with constructed test suites; V4V screens both, and the retained source verifiers become reference verifiers. The agent receives the instruction and environment used for ordinary task solving and generates one verifier, which is fixed before it is scored on multiple positive and negative test cases.

Task: credit-card retention. Submit the requested card application and record the closure reason. Permitted tools include the read-only query get_user_dispute_history_7291 .
Generated check: The agent creates a tool allowlist that omits this query and rejects unlisted operations.
Self-tests during generation	Hidden positive test cases
The intended-positive candidate omits the dispute-history query.	Valid candidates include the permitted dispute-history query.
The positive self-test returns FAIL : replay is unavailable after budget exhaustion.	The verifier rejects the query as “Unrelated or unsupported mutation.”
The agent acknowledges that positive validation is incomplete.	All 9 positive cases receive FAIL without environment execution.
An incomplete allowlist excludes a legal read; self-testing never confirms acceptance of a complete valid solution.

Figure 2: Overlooked task details lead to incorrect verification. The agent’s tool allowlist omits a permitted read-only query, causing valid solutions to be rejected. Its self-tests omit the query and never confirm acceptance. GPT-6-astra with Claude Code on

\tau^{2}

-bench banking task 044 rejects all 9 positive cases.

Task type	Source benchmarks	Tasks	PASS	FAIL
Tool-call	BFCL v4 ( Patil et al., 2025 ) , ACEBench ( Chen et al., 2025 )	95	734	2214
State	$\tau^{2}$ -bench ( Barres et al., 2025 ) , AppWorld ( Trivedi et al., 2024 ) , AgentDojo ( Debenedetti et al., 2024 )	80	642	769
Code	SWE-bench Verified ( Jimenez et al., 2024 )	47	190	444
Total		222	1566	3427

Table 1: Benchmark composition after V4V screening. Top: tasks and reference labels by task type. Bottom: the four test-case groups. Appendix Tables 6 and 10 give counts by source and by construction category.

Model	BA norm	TPR	TNR	FPR h
GPT-6-astra	62.1	67.4	94.6	7.4
Claude Opus 4.8	54.0	67.7	86.3	13.5
Kimi K3	53.4	70.2	83.2	13.4
GPT-5.6-sol	52.9	56.8	96.0	0.8
GPT-5.6-terra	52.9	56.5	96.3	3.6
GLM-5.2	48.8	68.1	80.8	12.1

Table 2: Main results with the common Pi harness , on the same 222 tasks and ordered by BA

{}_{\rm norm}=\mathrm{TPR}+\mathrm{TNR}-100

. Rates are percentages, rounded independently; bold marks the best value per column (lowest for FPR h ). Constant baselines return valid decisions throughout, and random scores are expectations. Table 3 and Appendix Table 12 cover the 17 alternative configurations.

Figure 3: Acceptance versus rejection. TPR and TNR for the 19 models with the Pi harness, ordered by descending BA norm . Every model accepts positive cases (TPR) less reliably than it rejects negative ones (TNR).

Model	Native	Pi	Native	$\Delta$
Claude Opus 4.8	Claude Code	54.0	66.2	$+12.2$
GPT-5.5	Codex	46.6	30.3	$-16.4$
GPT-5.6-sol	Codex	52.9	26.2	$-26.6$
GPT-5.6-terra	Codex	52.9	31.0	$-21.8$
GPT-6-astra	Codex	62.1	58.0	$-4.1$
Gemini 3.5 Flash	Gemini CLI	35.7	50.7	$+15.0$

Table 3: Native harness comparison. BA norm on the same 222 tasks with Pi and with each family’s native harness. Bold marks the higher score in each pair;

\Delta

is native minus Pi, rounded independently.

Self-tests used only allowed solutions
Unjustified check	Tasks	Verifiers	Positive FAIL
Overbroad source-file protection	2	3	12
Rejects permitted representations	3	5	35
Subtotal	5	8	47
No positive self-test passed
Rejects authorized user-tool calls	3	3	27

Table 4: Unjustified checks in selected Astra verifiers. Lower block: no positive self-test returned a complete PASS during generation. Upper block: the remaining verifiers, whose self-tests used only solutions the check allows. Each verifier is a task–condition pair; the 119 positive FAIL decisions cover 90 distinct test cases. The review is targeted and does not estimate prevalence.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Element	Content
Model input	Task instruction and environment, identical to the solving setting; the reference implementation and hidden test cases are withheld.
Model output	One executable verifier $\widehat{V}_{t}$ for task $t$ .
Test suite	Positive cases (the reference solution and valid variants) and negative cases (ordinary errors and attacks), each carrying a reference label.
Reference	A fixed reference verifier $V_{t}^{\star}$ retained after V4V screening; its decision on every test case equals the reference label.
Scoring	For each hidden case, compare $\widehat{V}_{t}$ ’s PASS / FAIL decision against the reference label.
Metrics	Task-weighted TPR and TNR, summarized by BA norm .

Appendix

Table 5: Task contract for verifier generation and evaluation.

Source	Type	Tasks	PASS	FAIL	Variants	Attack
BFCL v4	Tool-call	45	390	1247	345	270
ACEBench	Tool-call	50	344	967	294	200
Tool-call subtotal		95	734	2214	639	470
$\tau^{2}$ -bench	State	32	287	303	255	96
AppWorld	State	28	216	278	188	56
AgentDojo injection	State	10	60	86	50	42

Appendix

Table 6: Task sources. Inventory of the 222 tasks. PASS and FAIL are reference labels. Variants counts valid variants (cosmetic changes and alternative solutions); Attack combines hack and process_violation . These are subsets of PASS and FAIL , respectively.

Source	Task ID	Reason
BFCL	parallel_65	D
SWE	astropy__astropy-7671	S
TB	feal-differential-cryptanalysis	P
TB	catch-me-if-you-can	P
TB	amuse-install	P
TB	feal-linear-cryptanalysis	P

Appendix

Table 7: All 45 excluded tasks. BFCL denotes BFCL v4, SWE denotes SWE-bench Verified, and TB denotes Terminal-Bench. Reason codes refer to the exclusion categories described in this section.

Check	Recorded passes	Evidence and coverage
C1: Replay	222/222	Reference decisions on all 4,993 cases agree with the labels assigned during construction.
C2: Repeatability	222/222	Repeated executions under the same initial conditions return stable decisions that agree with the labels.
C3: Provenance	222/222	All 4,993 case files and 222 task specifications match their frozen snapshots.
C4: Input-label consistency	222/222	A scan of all 4,993 cases finds no opposite-label groups with identical public inputs within a task.
C5: Trivial baselines	222/222	Every task has passing records for all three comparisons: always-accept, always-reject, and exact-match.
C6: Mutation testing	222/222	Each task’s suite detects every mutation in its operator set (1,410 configurations total); Section B.3 reports the criterion and representative detection evidence.

Appendix

Table 8: V4V outcomes and evidence. The second column reports frozen screening statuses. Checks follow the C1–C6 order in Section 2.3 .

Source	Tasks	Operators/task	Configurations	Checks modified
BFCL	45	9	405	Required and extra arguments, types, values, function calls, and containers.
ACEBench	50	11	550	Total calls, function sets, per-function call counts, and arguments.
$\tau^{2}$ -bench	32	3	96	Communication, database, and action reward checks.
AgentDojo	20	2	40	Text and final-state checks.
AppWorld	28	3	84	Answer comparison, state assertions, and failure handling.
SWE-bench	47	5	235	Newly passing and previously passing tests, missing tests, expected failures, and empty reports.

Appendix

Table 9: Mutation configurations. Operators alter reference checks; the counts are task–operator pairs and are not numbers of independent executions.

Label	Class	Count	Construction and intended distinction
PASS	gold	222	Reference solution; tests acceptance of the designated reference solution.
PASS	cosmetic	840	Changes formatting or other irrelevant representation details; tests acceptance of equivalent presentations.
PASS	overstrict_trap	504	Uses an alternative implementation or execution path; tests acceptance of valid alternatives.
FAIL	base	222	Uses an initial, unrepaired, or otherwise unfinished outcome; tests recognition of basic task failure.
FAIL	partial	999	Removes required arguments, actions, or changes; tests whether all required parts are checked.
FAIL	near_miss	907	Alters a consequential value or local behavior; tests detection of subtle errors.

Appendix

Table 10: Test-case categories. Construction and reference labels for the final 4,993 test cases. All final leaderboard configurations use the same task set. Test cases are withheld during verifier generation.

Model	Alternative	BA norm Pi harness	BA norm Alternative	$\Delta$ BA norm	Submitted/ $N$ Pi harness	Submitted/ $N$ Alternative
Claude Opus 4.8	Claude Code	54.0	66.2	$+12.2$	218/222	220/222
GPT-6-astra	Claude Code	62.1	61.1	$-0.9$	222/222	222/222
GPT-6-astra	Codex	62.1	58.0	$-4.1$	222/222	219/222
GPT-6-astra	mini-swe	62.1	54.9	$-7.2$	222/222	222/222
GPT-6-astra	OpenClaw	62.1	51.1	$-10.9$	222/222	216/222
Gemini 3.5 Flash	Gemini CLI	35.7	50.7	$+15.0$	222/222	221/222

Appendix

Table 11: Harness comparison. Configurations ordered by the alternative configuration’s normalized BA.

\Delta

BA norm is the alternative harness score minus the Pi harness score, in normalized-BA points. Full metrics appear in Table 2 for the Pi harness and in Table 12 for the alternatives.

Model	Harness	BA norm	TPR	TNR	FPR h
Claude Opus 4.8	Claude Code	66.2	75.8	90.4	7.8
GPT-6-astra	Claude Code	61.1	68.0	93.2	9.6
GPT-6-astra	Codex	58.0	66.2	91.8	9.2
GPT-6-astra	mini-swe	54.9	61.9	93.0	9.5
GPT-6-astra	OpenClaw	51.1	63.2	87.9	9.1
Gemini 3.5 Flash	Gemini CLI	50.7	59.2	91.5	7.7

Appendix

Table 12: Full results for alternative harness configurations. The 17 configurations outside the common Pi harness, ordered by BA norm and scored on the same 222 tasks as Table 2 . Differences across harnesses describe configurations rather than isolated model capability.

Table 13: Reasoning-effort ablation for GPT-6-astra. Each condition uses the same 222 tasks and stated generation budgets. Differences are computed from unrounded scores and may differ from subtraction of displayed values. These four conditions form the Astra trace batch, which is separate from the main release in Table 2 .

Rank	Model	$\mathrm{BA}_{\mathrm{norm}}\uparrow$	TPR $\uparrow$	TNR $\uparrow$	$\mathrm{FPR}_{h}\downarrow$
1	GPT-5.6-sol	85.2	86.5	98.7	1.8
2	GPT-6-astra	83.0	92.8	90.2	17.1
3	Kimi K2.6	79.6	87.1	92.5	10.4
4	GLM-5.2	75.9	93.3	82.6	25.7
5	DeepSeek V4 Flash	74.7	87.8	86.9	18.2
6	GPT-5.6-terra	74.6	77.6	96.9	6.1

Appendix

Table 16: Tool-call verification ranking. All 19 models use the Pi harness, evaluated on the same 95 tasks. Rates are percentages;

\mathrm{BA}_{\mathrm{norm}}

is reported in points. Ranks use unrounded scores.

Rank	Model	$\mathrm{BA}_{\mathrm{norm}}\uparrow$	TPR $\uparrow$	TNR $\uparrow$	$\mathrm{FPR}_{h}\downarrow$
1	Claude Opus 4.8	34.6	41.2	93.3	0.5
2	Qwen3.7 Plus	30.7	49.0	81.7	10.5
3	Doubao Seed 2.1 Pro	30.0	36.1	93.9	3.6
4	GPT-5.6-terra	30.0	35.0	95.0	2.8
5	GPT-6-astra	29.1	31.0	98.1	0.2
6	Gemini 3.5 Flash	27.6	33.6	94.0	0.5

Appendix

Table 17: State-based verification ranking. All 19 models use the Pi harness, evaluated on the same 80 tasks. Rates are percentages;

\mathrm{BA}_{\mathrm{norm}}

is reported in points. Ranks use unrounded scores.

Rank	Model	$\mathrm{BA}_{\mathrm{norm}}\uparrow$	TPR $\uparrow$	TNR $\uparrow$
1	Kimi K3	76.9	79.5	97.5
2	GPT-6-astra	75.8	78.3	97.5
3	DeepSeek V4 Pro	66.8	73.6	93.2
4	Claude Opus 4.8	59.0	63.4	95.6
5	GLM-5.2	58.1	65.1	93.0
6	GPT-5.5	55.4	59.1	96.2

Appendix

Table 18: Code-based verification ranking. All 19 models use the Pi harness, evaluated on the same 47 tasks. Rates are percentages;

\mathrm{BA}_{\mathrm{norm}}

is reported in points. Ranks use unrounded scores.

(a) Positive test cases
Model	Gold	Cosmetic	Valid alternative
GPT-6-astra	154/67/1	573/263/4	285/215/4
Claude Opus 4.8	152/65/5	578/243/19	323/172/9
Kimi K3	156/49/17	590/186/64	323/119/62
GPT-5.6-sol	131/85/6	498/319/23	257/231/16
GPT-5.6-terra	131/89/2	500/332/8	240/259/5

Appendix

Table 19: Decisions on positive test cases. Grouped counts on positive test cases for all 19 configurations using the Pi harness. Each cell is

P/F/U

: valid PASS , valid FAIL , and no valid decision. The cell denominator is their sum,

N=P+F+U

(b) Negative test cases
Model	Ordinary errors	Hack	Process violation
GPT-6-astra	144/2477/8	83/711/2	1/1/0
Claude Opus 4.8	390/2197/42	156/628/12	2/0/0
Kimi K3	311/2220/98	156/593/47	2/0/0
GPT-5.6-sol	35/2548/46	9/775/12	0/2/0
GPT-5.6-terra	64/2541/24	38/754/4	1/1/0

Appendix

Table 20: Decisions on negative test cases. Grouped counts on negative test cases for all 19 configurations using the Pi harness. Each cell is

P/F/U

: valid PASS , valid FAIL , and no valid decision. The cell denominator is their sum,

N=P+F+U

. Ordinary errors combine base, partial, near-miss, and regression test cases.

Figure 4: Test-case-level agreement and missing or invalid outputs. Rows follow the order of the configurations using the Pi harness in Table 2 . Agreement is

P/N

for the reference solution (Gold), cosmetic variants (Cosm.), and alternative implementations (Alt.), and

F/N

for ordinary errors (Error) and hacks (Hack); missing or invalid output is

U/N

. All 19 models are included. Each model uses the same group denominators from Tables 19 and 20 . The two process-violation test cases are omitted here and retained in Table 20 . Both color scales span 0–100%. These test-case-weighted diagnostics differ from the task-averaged leaderboard metrics.

Model	$N$	$N^{+}$	$P$	$F$	$U$	Rejection share (%)	Valid outputs (%)
GPT-6-astra	4993	1566	1012	545	9	98.4	99.6
Claude Opus 4.8	4993	1566	1053	480	33	93.6	98.3
Kimi K3	4993	1566	1069	354	143	71.2	94.2
GPT-5.6-sol	4993	1566	886	635	45	93.4	97.9
GPT-5.6-terra	4993	1566	871	680	15	97.8	99.1
GLM-5.2	4993	1566	1055	336	175	65.8	92.1

Appendix

Table 21: False rejection and valid outputs. Outcomes on positive test cases and valid-output rates for all 19 configurations using the Pi harness.

N=4{,}993

is the test-case total across all six groups;

N^{+}=1{,}566

is the positive-test denominator for every model. Rejection share is

100F/(F+U)

, the fraction of unaccepted positives receiving an explicit FAIL . The valid-output rate includes all six groups and pools their test cases.

Model	Positive PASS	Negative FAIL	No valid decision
Tool-call: update a user profile (ACE preference 17)
GPT-5.6-sol	0/6	18/18	0
Claude Opus 4.8	6/6	6/18	0
Kimi K3	6/6	16/18	0
State: change an order address ( $\tau^{2}$ retail/34)
GPT-5.6-sol	0/9	8/8	0

Appendix

Table 24: Model comparisons on identical test cases. All three models use the Pi harness. Fractions count correct decisions over all cases of the corresponding class. The six missing decisions for Kimi K3 on Django-11848 are timeouts on one positive and five negatives, which remain in the denominators.

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 142 papers

AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputs
Apr 24, 2026Pouya Pezeshkpour, Estevam HruschkaLLM Evaluation LLM Answer Verification
Megagon Labs
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
Apr 20, 2026Wentao Shi, Yu Wang, Yuyang Zhao +8LLM-as-a-Judge LLM Agent Evaluation
University of Science and Technology of China · National University of Singapore · Meituan
VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Oct 1, 2026Caiqi Zhang, Rujun Han, Zifeng Wang +4LLM Agent Verification Long-Horizon LLM Agents
University of Cambridge · Google Cloud AI Research

Apr 24, 2026J/K move · Enter open · S save

AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputs

Pouya Pezeshkpour, Estevam Hruschka

Megagon Labs

LLM Evaluation LLM Answer Verification LLM-Based Program Synthesis LLM Reliability

Open paper AskPDF arXiv

Verification is becoming central to both reinforcement-learning-based training and inference-time control of large language models (LLMs). Yet current verifiers face a fundamental trade-off: LLM-based verifiers are expressive but hard to control and prone to error, while deterministic executable verifiers are reliable and interpretable but often limited in capability. We study the following question: given a development set of LLM outputs and labels for a target objective, such as correctness, can we automatically induce a minimal set of Python verifiers whose joint satisfaction closely matches that objective? We propose AutoPyVerifier, a framework that uses an LLM to synthesize candidate verifier functions and then refines them through search over a directed acyclic graph (DAG). By navigating the DAG, AutoPyVerifier systematically explores the space of deterministic executable verifiers and selects a compact verifier set whose joint satisfaction best approximates the target objective. Across mathematical reasoning, coding, function calling, and instruction-following benchmarks for several state-of-the-art LLMs, AutoPyVerifier improves target-objective prediction by up to 55.0 F1 points over the initial LLM-generated verifier sets. Additional analyses show that the most useful verification targets vary by benchmark and model, and that the DAG-based search shifts the learned verifier sets toward more structural and semantically grounded checks. We further show that exposing the discovered verifier set to an LLM as an external tool improves downstream accuracy by up to 17.0 points. We release our code

Abstract

LLM Agent Verification

Explore similar work

AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputs

AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks