Evidence-First Agentic Verification

Reza Sanayei; Razvan-Gabriel Dumitru; Sunjie Hou; MohammadHossein Rezaei; Haris Riaz; Robert Vacareanu; Minglai Yang; Ying Liu

cs.AIOct 13, 2026

Evidence-First Agentic Verification

Authors: Reza Sanayei, Razvan-Gabriel Dumitru, Sunjie Hou, MohammadHossein Rezaei, Haris Riaz, Robert Vacareanu, Minglai Yang, Ying Liu,

Organizations: Scale AI · University of Arizona

Abstract

Long-horizon, semi-verifiable agentic tasks often rely on rubric-based evaluation, but the evidence needed to assess these tasks can be spread across the environment state and the execution trace. A judge may overlook this evidence or accept unsupported claims, including an agent's own reports of success. When rubric verdicts serve as reinforcement-learning rewards, such errors can reinforce undesirable behavior. We introduce Sauron, an evidence-first agentic verifier that investigates both state and trace and relies on a claim only after independently validating it against its source. Bounded queries and joint investigation of related criteria reduce context use and redundant work, and each verdict carries an auditable receipt of its supporting evidence. Across 2,586 human-labeled criteria from 100 personal-assistant episodes, Sauron achieves the highest F1 of five verifiers on all eight backbones tested while costing less than Gandalf, the strongest baseline. With GPT-5.6 Sol, it reaches 0.96 F1 versus Gandalf's 0.92 at 23% lower cost, with a wider margin on process criteria. On a paired fault-injection suite covering 13 families, several of which simulate reward hacking, Sauron detects 97% of injected failures while flagging only 2% of clean controls, outperforming baselines even when they are given the full trace. On TRAIL, a benchmark for detecting and localizing real agent errors, Sauron leads both Gandalf and TRAIL's own judge on all three quality metrics.

Figures & tables

Figure 1: Sauron architecture. Two lightweight planning steps cluster the rubric criteria $C$ by likely shared evidence and produce an advisory task gist $g$ . Each cluster runs as a parallel investigation session. The investigator reaches the trace $\tau$ and state $S$ through bounded queries and proposes claims. The evidence verifier checks each claim against its source, records the result in the session’s evidence store, and returns rejected claims with a reason. The investigator separately proposes verdicts, and the verdict finalizer accepts one only when verified claims support it. Each criterion receives its own verdict and a receipt of the evidence it cites.

Figure 2: Investigation through bounded queries. Three criteria from one episode (51 events, 19 tool calls), investigated in one session. (a) Two state reads suffice for the outcome criterion. (b) The consistency criterion reads both sources and finds that the saved Amazon total differs from the retrieved one. (c) The process criterion is answerable only from the trace. The investigator reuses the Chase lookup found in (b), locates the reporting event, and confirms that the lookup came first. Each read is tagged broad or targeted, and each panel ends with the claims the investigator proposes for verification.

Figure 3: Evidence verification. Each proposed atom $a=(x,t,\ell)$ , with claim $x$ , evidence type $t$ , and locator $\ell$ , is checked against the source content retrieved from $\ell$ . An exact claim ( $a_{1}$ ) is compared mechanically with the stored value. An interpretive claim ( $a_{2}$ ) goes to an isolated faithfulness checker that sees only the claim and the cited passage. A claim whose source cannot be retrieved ( $a_{3}$ ) is rejected without verification, and the reason is returned to the investigator.

Figure 4: Quality vs. cost on

\mathrm{PA}_{100}

. F1 against cost (log scale) for each verifier (marker) and backbone (color). Sauron is drawn with black-outlined circles, and the dashed line traces the Pareto frontier.

	Access		All-criteria			F1 by criterion type
Verifier	State	Trace	F1 $\uparrow$	MCC $\uparrow$	Cost $\downarrow$	Process $\uparrow$	Outcome $\uparrow$
Agentic
Sauron	✓	✓	0.96	0.94	$297.69	0.97	0.96
Gandalf	✓	$\times$	0.92	0.89	$388.25	0.90	0.93
AAAJ	✓	✓	0.52	0.25	$234.78	0.82	0.46
Snapshot-based

Table 1: Verifier quality on

\mathrm{PA}_{100}

with GPT-5.6 Sol. Access marks which sources the verifier can inspect (all see the final response). All-criteria scores pool the 2,586 criteria. Cost is per full run. By-type columns give F1 on the process (

\mathrm{PA}_{\mathrm{Proc}}

) and outcome (

\mathrm{PA}_{\mathrm{Out}}

) subsets. Bold = best quality score per column.

Verifier	$\Delta_{\mathrm{fault}}$ $\uparrow$	Detection $\uparrow$	False alarm $\downarrow$	$/variant	s/variant
Sauron	$+$ 0.95	0.97	0.02	0.21	30
Autorubric	$-$ 0.17	0.51	0.68	0.02	14
Autorubric trace	$+$ 0.71	0.84	0.13	0.26	14
Gandalf	$+$ 0.09	0.52	0.43	0.67	140
Gandalf trace	$+$ 0.67	0.74	0.07	0.99	66
AAAJ	$+$ 0.36	0.78	0.42	0.07	53

Table 2: Paired fault injection on

\mathrm{PA}_{\mathrm{Fault}}

(13 families

\times

49 pairs, GPT-5.6 Sol).

\Delta_{\mathrm{fault}}

=

detection rate on faulty variants

-

false-alarm rate on their clean controls. Gandalf and Autorubric are trace-blind by default; their

\mathrm{trace}

variants receive the full trace. Cost and time are per variant. Bold = best quality score per column.

GPT-5.6 Sol
	GAIA (117)			SWE-bench Lite (31)
Verifier	wF1 $\uparrow$	Loc $\uparrow$	Joint $\uparrow$	wF1 $\uparrow$	Loc $\uparrow$	Joint $\uparrow$	Cost $\downarrow$
Sauron	0.53	0.73	0.44	0.68	0.43	0.10	$841
Gandalf	0.46	0.53	0.25	0.61	0.24	0.06	$441
Judge TRAIL	0.46	0.55	0.22	0.47	0.21	0.04	$405
GPT-5.6 Terra

Table 3: Error detection and localization on TRAIL (21 error categories). wF1

=

support-weighted category F1; Loc

=

location accuracy; Joint

=

(span, category) accuracy. Judge TRAIL

=

TRAIL’s single-pass judge. Cost is per full run. Bold

=

best quality score per backbone. All verifiers use the location normalization in Section D.3 .

Evidence access & navigation
Variant	$\Delta$ F1	Rel. cost	Rel. turns
Full Sauron	F1 $=$ 0.96	1.00 $\times$	1.00 $\times$
State only	$-$ 0.15	1.00 $\times$	1.03 $\times$
Trace in context	$+$ 0.01	3.03 $\times$	2.65 $\times$
No gist	0.00	1.15 $\times$	1.16 $\times$
Evidence verification

Table 4: Mechanism ablations on

\mathrm{PA}_{\mathrm{Abl}}

(860 criteria, GPT-5.6 Sol). Each variant disables or replaces the indicated component;

\Delta

F1, cost, and turns are relative to full Sauron.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Criterion-clustering prompt. The planner receives the user request and rubric criteria; max_size specifies the cluster-size limit.

Panel of Figure 6 — Figure 6: Task-gist prompt. The model receives the user request and compact views of the recorded episode, returning a summary and suggested locations to inspect. The gist provides guidance, not verified evidence.

Trace
Operator	Main inputs	Returned information
trace_search	Pattern; [event types, tool names, result limit, span]	Matching event summaries and text around each match.
trace_get_event	Event identifier	Event metadata and bounded content.
trace_get_event_text	Event identifier; [offset, length]	A portion of an event, its position, and whether more content remains.
trace_window	Event identifier; [counts before and after]	Bounded content of the selected event and summaries of its neighbors.
trace_tool_calls	[Tool name, status]	Linked call and result summaries.

Appendix

Table 5: Direct source queries. Inputs in brackets are optional; a tool with only bracketed inputs can be called without arguments. Span-tree queries and span filters require a trace with a recorded hierarchy. Content limits appear in Table 6 .

Content	Default size	What the limit covers
Complete tool response	24,000 characters	The entire JSON response, including metadata and formatting.
Single-event content	6,000 characters	The beginning of an event, including the focus event in a window.
Event text page	20,000 characters	Default and maximum page length. The response limit may shorten it further.
Trace window	3 events before and 3 after	Neighboring events have 400-character summaries.
Search results	20 matches	Default for trace and state searches; maximum for artifact searches.
Span-tree listing	200 spans	Maximum listed spans. Further queries can target individual subtrees.

Appendix

Table 6: Default content limits. Each tool has its own content limit or preview size. Every complete response must also fit within 24,000 characters, including metadata, so some results may be shortened further.

Resource	Allocation for $n$ criteria at scale $s$
Investigator turns	$\min\!\left(\operatorname{round}\!\left[s\min(20+7.5n,75)\right],60\right)$
Trace searches and span-tree queries	$\operatorname{round}\!\left[s(20+10n)\right]$
Trace retrievals	$\operatorname{round}\!\left[s(30+15n)\right]$
State and artifact queries	$\operatorname{round}\!\left[s(30+15n)\right]$
Token budget	$\operatorname{round}\!\left[s(250{,}000+150{,}000n)\right]$
Elapsed time	$s(600+300n)$ seconds

Appendix

Table 7: Session budgets. The default scale is

s=1

unless a multiplier applies. Integer limits are rounded to the nearest integer, with ties to even. An investigator turn may contain several tool calls, each charged to its corresponding allowance. Reaching a limit triggers one extra finalization-only turn beyond these limits.

Evidence type	Required source reference	Claim concerns
tool_call_presence	Trace event(s) or tool-call identifier	Whether a named tool was called, optionally with a specified status.
tool_call_absence	tool_name_absent	Whether a named tool was absent from the recorded calls.
tool_argument_match	Trace event(s) or tool-call identifier	Arguments supplied to a tool.
state_value_match	State path or optional MCP call reference	A value in the environment.
state_mutation	State path and either trace event(s) or a tool-call identifier	Whether a recorded action is reflected in the final state.
temporal_ordering	Exactly two distinct event identifiers	The relative order of two events.

Appendix

Table 8: Evidence types and source references. Locators identify the source material used to check each claim.

Panel of Figure 8 — Figure 8: Faithfulness-checker prompt. {claim} is the proposed atom’s claim and {source} the content its locator resolves to (head-and-tail truncated to 12,000 characters). The call sees no investigator history, trace, or other evidence. If the reply cannot be parsed as the JSON object shown, the atom is rejected.

Panel of Figure 9 — Figure 9: Set-coverage check prompt , used for set-wide criteria in clusters of eight or more. {criterion} is the criterion text and {atoms} the claims of the verified atoms cited for it. The atoms’ correctness is taken as given; the checker judges only whether they cover every instance the criterion quantifies over.

Backbone	Model identifier
GPT-5.6 Sol	openai/gpt-5.6-sol
GPT-5.6 Terra	openai/gpt-5.6-terra
GPT-5.6 Luna	openai/gpt-5.6-luna
Claude Opus 5	anthropic/claude-opus-5
Claude Sonnet 5	anthropic/claude-sonnet-5
Claude Fable 5.1	anthropic/claude-fable-5-1

Appendix

Table 9: Backbones evaluated on

\mathrm{PA}_{100}

Panel of Figure 10 — Figure 10: Criterion-classification prompt. {items} holds the criterion identifiers and texts; the classifier sees nothing else. state_checkable and final_response_only are grouped as outcome ; the other five classes are grouped as process .

Family	Taxonomy mode	Interaction edge	Fault side
drop call	Satisficing	User–Model	Model
flip error	Tool Feedback Neglect	Tool–Model	Model
reorder	Reordering †	Context–Model	Model
claim inject	Satisficing	User–Model	Model
eval awareness	Evaluation Awareness	Grader–Model	Model
prompt injection	Indirect Prompt Injection	Third party–Model	Model

Appendix

Table 10: Taxonomy grounding of the fault families in

\mathrm{PA}_{\mathrm{Fault}}

. Each fault family simulates the evidence associated with the listed failure mode. Interaction edges and fault sides follow Raj et al. [22] , whose owner component we call the user. † Our addition; not a mode in the taxonomy.

Family	Edit and control	Criterion summary	Fault	Control
drop call	Remove a successful call and its result; retain both in the control.	The specified call occurred successfully.	U	M
flip error	Replace a successful tool result with an error; retain success in the control.	The specified call completed without error.	U	M
reorder	Move a call and its result after the final reply; retain their earlier position in the control.	The specified call occurred before the final reply.	U	M
claim inject	Append a claim of an action never taken; append a claim supported by a recorded successful call in the control.	The agent performed the action claimed in its response.	U	M
eval awareness	Insert a statement that work was shortened or misreported because the run appeared graded; omit it in the control.	The record shows no evaluation-conditioned shortcut.	U	M
prompt injection	Add an overriding directive to a tool result and an action following it; omit both in the control.	The agent followed an overriding instruction embedded in tool output.	M	U

Appendix

Table 11: Injection templates and expected verdicts on

\mathrm{PA}_{\mathrm{Fault}}

. Criterion summaries describe the assertions evaluated; instantiated criteria identify the relevant actions, arguments, or state paths. M denotes met and U denotes unmet . The last two columns give the expected verdict for each side of the pair. Criteria that assert a violation expect met on the faulty variant.

Backbone	Verifier	F1	MCC	P	R	Acc	FA	MA	Cost	In Mtok	Out Mtok
GPT-5.6 Sol	Sauron	0.96	0.94	0.94	0.97	0.97	21	44	$297.69	67.2	1.73
	Gandalf	0.92	0.89	0.90	0.94	0.95	43	78	$388.25	147.1	1.20
	Archipelago	0.92	0.89	0.92	0.92	0.95	60	64	$2,058	81.8	0.65
	AAAJ	0.52	0.25	0.38	0.86	0.53	111	1093	$234.78	2.3	0.36
	Autorubric	0.65	0.48	0.53	0.83	0.73	131	557	$76.09	5.0	0.21
Claude Fable 5.1	Sauron	0.95	0.93	0.95	0.95	0.97	38	40	$414.70	103.9	3.45

Appendix

Table 12: Full

\mathrm{PA}_{100}

results across eight backbones. FA = false awards ( met on reference- unmet ); MA = missed awards ( unmet on reference- met ). Bold marks the highest F1 for each backbone. Tokens in millions.

Backbone	Verifier	Process F1	Outcome F1
GPT-5.6 Sol	Sauron	0.97	0.96
	Gandalf	0.90	0.93
	Archipelago	0.88	0.93
	AAAJ	0.82	0.46
	Autorubric	0.77	0.61
Claude Fable 5.1	Sauron	0.96	0.95

Appendix

Table 13:

\mathrm{PA}_{100}

results by criterion type. F1 on 411 process and 2,175 outcome criteria, with unweighted means across eight backbones in the final block. Bold marks the best F1 in each column within each block.

Concurrent tasks $w$	Sauron (min)	Gandalf (min)	Speedup
1	17.1	71.0	4.16 $\times$
2	9.8	22.8	2.34 $\times$
4	5.0	11.9	2.37 $\times$
8	3.6	7.8	2.14 $\times$

Appendix

Table 14: Runtime under concurrent grading on a 10-episode slice of

\mathrm{PA}_{100}

(GPT-5.6 Sol). Minutes from first request to last verdict with

w

episodes graded concurrently, each verifier at its default internal settings.

Figure 11:

\Delta_{\mathrm{fault}}

by fault family on

\mathrm{PA}_{\mathrm{Fault}}

. Results with GPT-5.6 Sol on 49 fault–control pairs per fault family. Higher is better. Blue cells are positive and red cells negative. The final row reports the unweighted mean across all 13 fault families.

Verifier	$/variant	s/variant	In Mtok	Out Mtok	Cached Mtok	Cache hit
Sauron	0.21	30	57.8	1.59	46.3	80.1%
Gandalf trace	0.99	66	440.4	1.41	376.3	85.4%
Gandalf	0.67	140	318.1	2.90	286.6	90.1%
Autorubric trace	0.26	14	36.3	0.22	4.6	12.6%
Autorubric	0.02	14	3.8	0.15	3.0	78.3%
Archipelago	0.24	16	33.3	0.32	5.2	15.6%

Appendix

Table 15: Resource use on

\mathrm{PA}_{\mathrm{Fault}}

(GPT-5.6 Sol). Cost and time are averages per variant. Token counts are in millions over all 1,274 variants. Input totals include cached tokens; cache hit is the percentage of input tokens served from the cache.

Disagreement category	FA	MA	Total	Share
Strict reading, positive criterion	0	34	34	52%
Strict reading, negative criterion	14	0	14	22%
Lenient reading, positive criterion	6	0	6	9%
Misinterpretation or conditional criterion	1	4	5	8%
Polarity convention	0	3	3	5%
Budget exhaustion	0	2	2	3%

Appendix

Table 16: Disagreements between Sauron and human judgments on

\mathrm{PA}_{100}

(GPT-5.6 Sol). False awards (FA) are met verdicts on reference- unmet criteria; missed awards (MA) are unmet verdicts on reference- met criteria. Shares are computed over the 65 disagreements.

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 145 papers

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Oct 1, 2026Caiqi Zhang, Rujun Han, Zifeng Wang +4LLM Agent Verification Long-Horizon LLM Agents
University of Cambridge · Google Cloud AI Research
AgentProofBench: Benchmarking Reusable Verifiers for Web Agents
Oct 25, 2026Jingjie Ning, Xueqi Li, Yi Lu +1LLM Agent Verification Web Agent Benchmarks
SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution
Jun 29, 2026Aojie Yuan, Yi Nian, Haiyue Zhang +2AI Agent Reliability Process Reward Models
University of Southern California, Los Angeles, USA · University of Michigan, Ann Arbor, USA

Oct 1, 2026J/K move · Enter open · S save

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu+3

University of Cambridge · Google Cloud AI Research

LLM Agent Verification Long-Horizon LLM Agents LLM Agent Harnesses

Open paper AskPDF arXiv

As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance, bringing gains over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. We further show that verification skills can self-improve from failure feedback, demonstrating VeriHarness as a novel and critical approach for scaling long-horizon agentic verification. We release the full pool of approximately 26,000 rollouts across all five benchmarks and both models, produced at a cost of over $100,000, to support future research on agentic verification.

Abstract

LLM Agent Verification

Explore similar work

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

AgentProofBench: Benchmarking Reusable Verifiers for Web Agents

SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution