VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

Caiqi Zhang; Rujun Han; Zifeng Wang; Zoey CuiZhu; Nigel Collier; Tomas Pfister; Chen-Yu Lee

cs.AIOct 1, 2026

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

Authors: Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu, Nigel Collier, Tomas Pfister, Chen-Yu Lee

Organizations: University of Cambridge · Google Cloud AI Research

Abstract

As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance, bringing gains over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. We further show that verification skills can self-improve from failure feedback, demonstrating VeriHarness as a novel and critical approach for scaling long-horizon agentic verification. We release the full pool of approximately 26,000 rollouts across all five benchmarks and both models, produced at a cost of over $100,000, to support future research on agentic verification.

Figures & tables

Panel of Figure 1 — Figure 1: VeriHarness and its results. Left: the verifier is the generator’s own model inside a harness. A disagreement resolver tests the claims on which the $N$ rollouts differ, and a consensus challenger seeks evidence against the claims they share. Adjudication combines their findings to guide artifact selection and revision. Right: with Gemini 3.5 Flash as both generator and verifier, VeriHarness with revision improves over the single-rollout baseline on all five benchmarks, including gains of 6.7 points on APEX-Agents and 8.9 points on Workspace-Bench Lite (WSB).

Figure 2: Consensus can preserve errors; disagreement can contain correct alternatives that voting discards. APEX-Agents claims from ten-rollout pools of Claude Opus 4.8; percentages are within each group, correctness uses the benchmark’s rubric grades, and

H

is the claim entropy of Section 2 .

Figure 3: VeriHarness on a worked example. The task asks for FY2025 revenue from the final financial report. (1) Two rollouts cite the draft and report 100m; a third cites the final report and gives 120m, and all three label the figure USD. (2a) The resolver checks version history and establishes that the final report supersedes the draft, supporting 120m. (2b) In a separate context, the challenger checks source metadata and finds that the currency every rollout uses should be EUR. (3) Adjudication in a fresh context combines the evidence into a revision plan, and delivery produces the revised report with its verification record.

Benchmark	Tasks	Domain	Artifact	Grader
APEX-Agents (v1.0)	480	banking, law, consulting	memos, models, answers	expert rubric
Workspace-Bench Lite	100	file-heavy workspaces	documents and files	rubric judge
WorkBuddy Bench	200	code, office, web	patches and office files	composite verifier
SpreadsheetBench 2	321	business spreadsheets	workbooks	recompute, compare
JobBench	65	35 white-collar occupations	multi-file artifacts	rubric judge

Table 1: Benchmarks. Tasks is the number of tasks in our pools; every task has

N=10

rollouts per model.

Gemini 3.5 Flash
Method	Avg	APEX	WSB	WorkBuddy	SB-2	JobBench
Single rollout	47.2	48.1	56.5	73.2	27.3	30.9
Majority voting	47.5	47.3 $\,\pm 0.4$	57.0 $\,\pm 1.8$	74.0 $\,\pm 1.0$	27.0 $\,\pm 1.3$	32.4 $\,\pm 0.5$
Best-of- $N$ with judge	47.3	47.2 $\,\pm 0.4$	57.0 $\,\pm 2.1$	73.6 $\,\pm 1.2$	27.7 $\,\pm 0.3$	30.9 $\,\pm 0.9$
Pairwise tournament	49.1	49.2 $\,\pm 1.4$	59.6 $\,\pm 1.0$	74.8 $\,\pm 0.8$	30.4 $\,\pm 0.7$	31.4 $\,\pm 1.5$
LLM-as-a-Verifier	49.5	49.8 $\,\pm 0.5$	60.4 $\,\pm 0.2$	75.3 $\,\pm 0.5$	30.2 $\,\pm 0.5$	31.9 $\,\pm 1.2$

Table 2: Main results. Three-seed mean score on each benchmark’s own scale, with the cross-seed standard deviation in gray; Avg is the unweighted mean over the five benchmarks, and gains are over the single rollout. Within each model, bold marks the best score in the primary selection comparison and, separately, among the methods with revision; CLI variants are reported separately, followed by methods that produce new artifacts. The single rollout is a reference baseline. The selection oracle takes the highest-scoring rollout per task under the benchmark grader and bounds selection from the fixed pool; it does not bound revision. SB-2: SpreadsheetBench 2; WSB: Workspace-Bench Lite.

Figure 4: Evolved libraries improve held-out scores. (a) Held-out scores of the four libraries. A: empty; B: human-authored; C: evolved from A; D: evolved from B. (b) Held-out scores recorded each round for C and D, with A and B as dashed lines; end labels give the final-library scores of (a). Filled markers after round 0 indicate a new library retained using development scores; hollow markers indicate that the current library is unchanged.

Figure 5: Selection gains where the pool disagrees. Selection score minus pool mean on APEX-Agents, Claude Opus 4.8, by the claim entropy of the pool.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Source	How it helps verification
Exposed alternatives	Multiple rollouts make competing values and interpretations explicit. The verifier can investigate a concrete discrepancy that an individual generation attempt never encountered.
Focused checking	Given completed artifacts, the verifier can devote its attention to testing particular claims. Separate resolver and challenger contexts reserve attention for both disputed and shared claims.
Environmental evidence	A targeted source lookup or recomputation can reveal evidence omitted from the rollouts. The generator has the same tools; verification directs their use toward a specific claim whose validity is now in question.
Accumulated skills	Human-authored checks and procedures learned from development failures suggest what to examine, including requirements that every candidate omitted. This experience can guide later verification without changing model weights.

Appendix

Table 3: Sources of advantage in same-model verification.

Figure 6: Selection oracle against pool size. Expected score of the best rollout among

N

drawn without replacement from the pool of ten, on each benchmark’s own scale;

N

=1 is the single rollout and

N

=10 is the selection oracle of Table 2 . The curves are computed from the grader scores in the released rollout pool.

Gemini 3.5 Flash
Method	Avg	APEX	WSB	WorkBuddy	SB-2	JobBench
Single rollout	47.2	48.1	56.5	73.2	27.3	30.9
Resolver only	51.5	53.1	63.2	77.0	30.0	34.0
Challenger only	50.0	50.9	60.3	75.2	29.7	33.7
VeriHarness without skills	51.3	50.9	63.5	77.1	30.6	34.4
VeriHarness in a single context	52.6	53.9	64.5	77.9	31.3	35.3

Appendix

Table 4: Score of the full harness with one part disabled. Three-seed mean on each benchmark’s own scale after applying the revision plan; Avg is the unweighted mean over the five benchmarks.

Part	What it provides	Examples
Workspace and tools $(\mathcal{W},\mathcal{T})$	Access to the objects under check	Read a file, recompute a table, trace a source
Verification protocol $\Pi$	Division of checking labor	Resolver, challenger, adjudication
Verification skills $\mathcal{S}$	Reusable knowledge of what to check	Period, unit, source version, required items

Appendix

Table 5: Components of the verification harness.

Figure 7: Where each skill was used. For each of the eighteen skills, the number of invocations per task in the runs of Table 2 in our implementation, split by benchmark with both models pooled. We report rates rather than counts so that a large benchmark does not weigh more. An invocation is the skill being loaded into a context, its file being read by the model, or one of its scripts being run.

Skill	Deliverable	What it supplies	Scripts
evidence-xlsx	any	formulas beside cached values, workbook diffs, rendered sheets	xlsx_dump , xlsx_diff , xlsx_gaps , xlsx_forks , xlsx_render
evidence-pdf	any	tables with their columns, words with coordinates, rendered pages	pdf_render , pdf_tables , pdf_words
evidence-docx	any	text in reading order, tables, tracked changes and comments	docx_changes , docx_text
evidence-pptx	any	shapes in slide order, tables, charts, notes, rendered slides	pptx_render , pptx_text
evidence-bundle	file bundle	inventory of what each candidate delivered and states	bundle_inventory
evidence-patch	patch or page	build and test of every candidate patch; headless-browser probe of a delivered page	patchlab , pageprobe

Appendix

Table 6: The skill library. The eighteen skills of the human-authored library, the deliverable each is written for, what it supplies, and the scripts it ships.

Relation between the tested claim and the graded item	Share
The tested claim is an input or intermediate quantity; the graded quantity lies downstream	70%
Same quantity, different definition, period, or source version	14%
The challenger confirmed the wrong value of the graded quantity	12%
Omission or mapping noise	4%

Appendix

Table 7: Why shared errors survive the challenger. APEX-Agents, Claude Opus 4.8; share of the unanimous errors that the challenger upheld.

Figure 8: Cost against gain. Compute per task at list price with cached input at the cache-read rate, on a log scale with cost decreasing to the right, against the gain over the single rollout, the five-benchmark mean of Table 2 . Hollow markers with an asterisk are estimates.

Explore similar work

LLM-as-a-Verifier: A General-Purpose Verification Framework
Jul 6, 2026Jacky Kwok, Shulu Li, Pranav Atreya +6Large Language Models(Llms Feedback
Stanford University · UC Berkeley · NVIDIA Research
FineVerify: Scaling Test-Time Compute with Fine-Grained Self-Verification for Agentic Search
May 30, 2026James Xu Zhao, Hui Chen, Bryan Hooi +1Agentic Search Agentic Benchmarks
National University of Singapore
LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks
Aug 3, 2026Ziyu Ma, Hailang Huang, Shun Zou +5Long-Horizon Agents Long-Horizon Task Planning
DreamX Team, Alibaba Group

Jul 6, 2026, cs.AIJ/K move · Enter open · S save

LLM-as-a-Verifier: A General-Purpose Verification Framework

Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu+5

Stanford University · UC Berkeley · NVIDIA Research

Large Language Models(Llms Feedback Mathematical Reasoning Benchmarks

Open paper AskPDF arXiv

Scaling pre-training, post-training, and test-time compute have become the central paradigms for improving the capabilities of LLMs. In this work, we identify verification, the ability to determine the correctness of a solution, as a new scaling axis. To unlock this and demonstrate its effectiveness, we introduce LLM-as-a-Verifier, a general-purpose verification framework that provides fine-grained feedback for agentic tasks without requiring additional training. Unlike standard LM judges that prompt LLMs to produce discrete scores for candidate solutions, LLM-as-a-Verifier computes the expectation over the distribution of scoring token logits to generate continuous scores. This probabilistic formulation enables verification to scale along multiple dimensions: (1) score granularity, (2) repeated evaluation, and (3) criteria decomposition. In particular, we show that scaling the scoring granularity leads to better separation between positive and negative solutions, resulting in more calibrated comparisons. Moreover, scaling repeated evaluation and criteria decomposition consistently lead to additional gains in verification accuracy through variance and complexity reduction. We further introduce a cost-efficient ranking algorithm for selecting the best solution among candidates using the verifier's continuous scores. LLM-as-a-Verifier achieves state-of-the-art performance on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%). Beyond verification, the fine-grained signals from LLM-as-a-Verifier can also serve as a proxy for estimating task progress. We build an extension for Claude Code, enabling developers to monitor and improve their own agentic systems. Finally, we show that LLM-as-a-Verifier can provide dense feedback for RL, improving the sample efficiency of SAC and GRPO on robotics and mathematical reasoning benchmarks.

Abstract

Explore similar work

LLM-as-a-Verifier: A General-Purpose Verification Framework

FineVerify: Scaling Test-Time Compute with Fine-Grained Self-Verification for Agentic Search

LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks