The effectiveness of a self-improving system relies heavily on the quality of its verifier, which judges the outcome of each improvement round. However, the verifiers of most tasks are written by hand, which is expensive and does not scale to tasks that come without one. We propose Veritas, a harness that generates verifiers automatically. Unlike general-purpose agent harnesses, Veritas adds three mechanisms for verifier generation: an evidence header stating what the verifier will observe, a library of domain rules that tasks leave unstated, and authoring principles, partly refined by a self-improvement loop. We evaluate on six benchmarks: four scored by execution, whose official verifiers we first certify, and two that compare pairs of responses. With the model fixed, the verifiers Veritas generates agree better with the official verdicts than those of seven other harnesses on all four execution-scored benchmarks. On the two pairwise benchmarks they are at or near the best. They also agree better than those of the vendor's own harness for two of three other model families. Its mechanisms also improve 9 of 12 harness-benchmark pairs scored by execution across three harnesses.
Figures & tables
Figure 1: Overview. κ∗ is Cohen’s κ against the official labels, with a missing verdict counted as an error. Plain harnesses lack E, R and H; icons in (b), in reading order: Codex CLI, OpenCode, pi, Terminus 2, Hermes, Claude Code, Gemini CLI. All harnesses use GLM-5.2; values from Table 2 .
Benchmark
Harness
V
Concretely
τ2 -bench
T,E0
τ,E0,Ef
messages, DB before, DB after
SWE-bench Verified
T,E0
Ef
patched repository; V runs its tests inside it
Terminal-Bench 4.0
T,E0
Ef
final-state filesystem
TB-Science 0.1
T,E0
Ef
final-state filesystem
LLMBar, JudgeBench
T,τA,τB
T,τA,τB
the two responses
Table 1: Inputs per benchmark: to the harness when it writes V , and to V when it judges. Where τ is not recorded, V judges only what the trajectory left behind. On the two pairwise benchmarks there is no environment, E0=Ef=∅ , a candidate is a response, the harness sees both responses (candidate-visible), and V returns a preference rather than PASS or FAIL.
plain harnesses
Benchmark ( n tasks)
Codex
OpenCode
pi
Terminus 2
Hermes
Claude Code
Gemini CLI
Veritas
τ2 -bench (58)
0.688
0.563
0.238
0.185
0.287
0.726
0.164
0.787
SWE-bench Verified (138)
0.599
0.678
0.693
0.518
0.503
0.557
0.396
0.720
Terminal-Bench (33)
0.387
0.336
0.048
−0.043
0.087
0.238
0.022
0.465
TB-Science (39)
0.351
0.474
−0.201
0.165
0.023
0.144
0.084
0.548
LLMBar (75 pairs; acc.)
0.920
0.760
0.893
0.933
0.893
0.827
0.840
0.920
Table 2: Agreement with the official verdicts with the model fixed (GLM-5.2) and the harness varied: κ∗ on the four benchmarks scored by execution, two-order-consistent accuracy on the two pairwise benchmarks. Best per row in bold.
Codex
Claude Code
plain
+ ERH
Δ
plain
+ ERH
Δ
τ2 -bench
0.688
0.751
+0.063
0.726
0.688
−0.038
SWE
0.599
0.640
+0.041
0.557
0.524
−0.033
TB
0.387
0.337
−0.050
0.238
0.586
+0.348
TB-Sci
0.351
0.533
+0.182
0.144
0.465
+0.321
LLMBar
0.920
0.893
−0.027
0.827
0.880
+0.053
Table 3: κ∗ of two plain harnesses with E, R and H added unchanged (+ ERH), GLM-5.2; two-order-consistent accuracy on LLMBar and JudgeBench. TB: Terminal-Bench.
Full
− E
− R
− H
Baseline
τ2 -bench
0.787
0.726
0.472
0.250
0.563
SWE
0.741
0.579
0.741
0.709
0.678
LLMBar
0.920
0.880
0.920
0.893
0.760
JudgeBench
0.980
0.961
0.961
0.980
0.980
Table 4: κ∗ with one mechanism removed at a time, on OpenCode, and two-order-consistent accuracy on LLMBar and JudgeBench. Baseline is plain OpenCode. On SWE-bench R is never activated, because its evidence does not include the trajectory, so − R is the same configuration as Full and has the same value. On SWE-bench, Full is the run of the ablation series, matched to the other variants, and Table 2 reports another run of Veritas ( 0.720 ) on the same substrate.
Harness
τ2
SWE
TB
TB-Sci
LLMB.
JB
GPT-5.6-sol
Veritas
—
0.692
0.277
0.423
0.933
0.980
Codex CLI
—
0.720
0.311
0.540
0.933
0.980
Qwen3.7-Plus
Veritas
0.546
0.554
0.680
0.712
0.893
0.941
Qwen Code
0.376
0.542
0.508
0.514
0.800
0.961
Table 5: Agreement with the model varied: Veritas against the harness the model’s vendor provides. κ∗ on the four benchmarks scored by execution, two-order-consistent accuracy on LLMBar (LLMB.) and JudgeBench (JB). τ2 : τ2 -bench. TB: Terminal-Bench. Better of each pair in bold. GPT-5.6-sol delivered too few verifiers on τ2 -bench to report (see text). † : on 74 of the 75 LLMBar pairs.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Phase
Rounds
Proposed by
What was edited
Accepted rounds
1
1–11
model
add a principle to H retry or repeat generation change timeouts or concurrency
3 and 4 (two principles in H)
2
12–25
researchers
change the instructions to the agent add a delivery safeguard change the instructions and delivery together change scoring on DeepSWE
15 (MCP-Atlas only) 25 (DeepSWE only)
3
26–32
researchers ∗
change a principle in H test the verifier on synthetic wrong candidates declare the evidence each check reads split the verifier into separate assertions
none
Appendix
Table 6: The 32 rounds of the self-improvement loop by phase. The acceptance rule of each phase is given in the text. ∗ Round 26 was proposed by a model.
oracle
no-op
trunc
linedrop
C1
C2
C3
C4
C5
C6
all derived
Terminal-Bench
33/33
0/33
9/33
15/32
4/12
4/13
3/7
6/7
3/29
—
44/133
TB-Science
39/39
0/39
20/39
13/28
5/19
0/20
0/2
5/8
0/11
3/19
46/146
Appendix
Table 7: Candidates the official tests accept, by kind, on the 33 Terminal-Bench and 39 TB-Science tasks. C1–C6 are the corruption operators listed above. Many accepted trunc and linedrop candidates are identical to the oracle in effect. Seven TB-Science tasks have no corrupted candidates: in five the corruption stage crashed, in one the corrupted candidate’s image was missing when the verifiers ran, and one produced none.
Benchmark
Official verifier
Tasks
Confirmed by execution
Agents’ Last Exam [ 56 ]
deterministic evaluator per task
82
27 tasks accept a wrong candidate or reject a correct one
τ2 -bench [ 7 , 55 ]
state comparison + expected actions
146
57 tasks lack full power: 33 reject the no-op but accept a near-miss; 24 more pass the no-op on the database comparison
DeepSWE [ 24 ]
hand-written checks per task
106
36 tasks give full marks to the gold patch minus one hunk
MCP-Atlas [ 6 ]
one shared model judge
500
72 tasks change verdict on identical input; 47 reject a correct paraphrase
Terminal-Bench 4.0 [ 36 ]
the task’s tests, run on the final state
66
6 tasks accept a corrupted output that adjudication found defective
TB-Science 0.1 [ 60 ]
the task’s tests, run on the final state
70
3 tasks accept a corrupted output that adjudication found defective
Appendix
Table 8: Confirmed defects in official verifiers. Every count is backed by a candidate executed against the real verifier. Findings that survived review but were not executed are counted separately in the text. Task counts are the tasks audited (DeepSWE: after G5), not the benchmark size.
plain harnesses
+ ERH
Codex
OpenCode
pi
Terminus 2
Hermes
Claude Code
Gemini CLI
Veritas
Codex
Claude Code
oracle TPR
0.36
0.36
0.39
0.30
0.30
0.42
0.30
0.45
0.36
0.52
no-op TNR
0.94
0.94
0.64
0.73
0.82
0.82
0.70
0.97
0.94
1.00
cut or corrupted TNR
0.91
0.89
0.63
0.70
0.73
0.79
0.65
0.89
0.92
0.94
Appendix
Table 9: Verdicts by candidate kind on Terminal-Bench (33 tasks), with a missing verdict counted as wrong. TPR is the share of candidates the official tests accept that the generated verifier also accepts; TNR is the share of candidates the official tests reject that it also rejects. The last row covers the 89 candidates derived from the oracle (Appendix B.2 ) that the official tests reject.
Candidate
Official
Plain OpenCode
Veritas
agent run
PASS
PASS
PASS
agent run without the verification call
FAIL
PASS
FAIL
empty
FAIL
FAIL
FAIL
Appendix
Table 10: Verdicts on the certified candidates of τ2 -bench banking task 23. Bold: the verdict that disagrees with the official label.
Candidate
Official
Plain OpenCode
Veritas
oracle
PASS
PASS
PASS
reference script cut to three quarters
PASS
PASS
PASS
reference script without its last line
PASS
PASS
PASS
one comparison flipped in the solution
PASS
PASS
PASS
no-op (placeholder left in place)
FAIL
PASS
FAIL
solution file emptied
FAIL
FAIL
FAIL
Appendix
Table 11: Verdicts on the certified candidates of the TB-Science task guided-wave-localization. Bold: the verdict that disagrees with the official label.
Candidate
Official
Plain Claude Code
Claude Code + ERH
agent run (one free bag)
PASS
FAIL
PASS
agent run without the booking call
FAIL
FAIL
FAIL
empty
FAIL
FAIL
FAIL
Appendix
Table 12: Verdicts on the certified candidates of τ2 -bench airline task 24. Bold: the verdict that disagrees with the official label.
Qwen3.7-Plus
Gemini 3.5 Flash
Candidate
Official
Qwen Code
Veritas
Gemini CLI
Veritas
oracle
PASS
FAIL
PASS
FAIL
PASS
no-op
FAIL
FAIL
FAIL
FAIL
FAIL
reference script cut to three quarters
FAIL
FAIL
FAIL
FAIL
FAIL
reference script without its last line
FAIL
FAIL
FAIL
FAIL
FAIL
one number scaled by 1.1
FAIL
FAIL
FAIL
FAIL
FAIL
Appendix
Table 13: Verdicts on the certified candidates of the Terminal-Bench task protein-autointerp-disulfide, for two model families. Bold: the verdict that disagrees with the official label.
Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.
Jianing Yin, Ziyi Chen, Hao Wang +4
University of Pennsylvania · University of Illinois at Urbana-Champaign · University of California, Berkeley +2
Agent harnesses shape how language-model agents use instructions, tools, and runtime components, but adapting these harnesses requires costly verification. Existing propose-and-verify methods typically score every candidate on a fixed task set, wasting rollouts on unrelated behaviors and allowing aggregate scores to obscure specific regressions. We introduce HarnessLens, a budget-aware framework for automated harness evolution. HarnessLens jointly explores the task space and user-configurable components, derives candidate modifications from execution trajectories, and selectively verifies each candidate on behavior-relevant tasks using an attributable-evidence gate. Across three agent harnesses and four benchmarks, HarnessLens improves average held-out performance by 7.6-13.6% while consuming substantially less evaluation budget than competing baselines. These results demonstrate that behavior-aware verification with explicit attribution enables more reliable and sample-efficient harness evolution under constrained interaction budgets. Our code is available at https://github.com/jhxu5214/HarnessLens.
Jinghan Xu, Yikai Zhang, Aili Chen +3
School of Data Science, Fudan University · Shanghai Key Laboratory of Data Science · College of Computer Science and Artificial Intelligence, Fudan University
As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance, bringing gains over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. We further show that verification skills can self-improve from failure feedback, demonstrating VeriHarness as a novel and critical approach for scaling long-horizon agentic verification. We release the full pool of approximately 26,000 rollouts across all five benchmarks and both models, produced at a cost of over $100,000, to support future research on agentic verification.
Caiqi Zhang, Rujun Han, Zifeng Wang +4
University of Cambridge · Google Cloud AI Research