Long-horizon, semi-verifiable agentic tasks often rely on rubric-based evaluation, but the evidence needed to assess these tasks can be spread across the environment state and the execution trace. A judge may overlook this evidence or accept unsupported claims, including an agent's own reports of success. When rubric verdicts serve as reinforcement-learning rewards, such errors can reinforce undesirable behavior. We introduce Sauron, an evidence-first agentic verifier that investigates both state and trace and relies on a claim only after independently validating it against its source. Bounded queries and joint investigation of related criteria reduce context use and redundant work, and each verdict carries an auditable receipt of its supporting evidence. Across 2,586 human-labeled criteria from 100 personal-assistant episodes, Sauron achieves the highest F1 of five verifiers on all eight backbones tested while costing less than Gandalf, the strongest baseline. With GPT-5.6 Sol, it reaches 0.96 F1 versus Gandalf's 0.92 at 23% lower cost, with a wider margin on process criteria. On a paired fault-injection suite covering 13 families, several of which simulate reward hacking, Sauron detects 97% of injected failures while flagging only 2% of clean controls, outperforming baselines even when they are given the full trace. On TRAIL, a benchmark for detecting and localizing real agent errors, Sauron leads both Gandalf and TRAIL's own judge on all three quality metrics.
Figures & tables
Figure 1: Sauron architecture. Two lightweight planning steps cluster the rubric criteria C by likely shared evidence and produce an advisory task gist g . Each cluster runs as a parallel investigation session. The investigator reaches the trace τ and state S through bounded queries and proposes claims. The evidence verifier checks each claim against its source, records the result in the session’s evidence store, and returns rejected claims with a reason. The investigator separately proposes verdicts, and the verdict finalizer accepts one only when verified claims support it. Each criterion receives its own verdict and a receipt of the evidence it cites.
Figure 2: Investigation through bounded queries. Three criteria from one episode (51 events, 19 tool calls), investigated in one session. (a) Two state reads suffice for the outcome criterion. (b) The consistency criterion reads both sources and finds that the saved Amazon total differs from the retrieved one. (c) The process criterion is answerable only from the trace. The investigator reuses the Chase lookup found in (b), locates the reporting event, and confirms that the lookup came first. Each read is tagged broad or targeted, and each panel ends with the claims the investigator proposes for verification.
Figure 3: Evidence verification. Each proposed atom a=(x,t,ℓ) , with claim x , evidence type t , and locator ℓ , is checked against the source content retrieved from ℓ . An exact claim ( a1 ) is compared mechanically with the stored value. An interpretive claim ( a2 ) goes to an isolated faithfulness checker that sees only the claim and the cited passage. A claim whose source cannot be retrieved ( a3 ) is rejected without verification, and the reason is returned to the investigator.
Figure 4: Quality vs. cost on PA100 . F1 against cost (log scale) for each verifier (marker) and backbone (color). Sauron is drawn with black-outlined circles, and the dashed line traces the Pareto frontier.
Access
All-criteria
F1 by criterion type
Verifier
State
Trace
F1 ↑
MCC ↑
Cost ↓
Process ↑
Outcome ↑
Agentic
Sauron
✓
✓
0.96
0.94
$297.69
0.97
0.96
Gandalf
✓
×
0.92
0.89
$388.25
0.90
0.93
AAAJ
✓
✓
0.52
0.25
$234.78
0.82
0.46
Snapshot-based
Table 1: Verifier quality on PA100 with GPT-5.6 Sol. Access marks which sources the verifier can inspect (all see the final response). All-criteria scores pool the 2,586 criteria. Cost is per full run. By-type columns give F1 on the process ( PAProc ) and outcome ( PAOut ) subsets. Bold = best quality score per column.
Verifier
Δfault↑
Detection ↑
False alarm ↓
$/variant
s/variant
Sauron
+ 0.95
0.97
0.02
0.21
30
Autorubric
− 0.17
0.51
0.68
0.02
14
Autorubric trace
+ 0.71
0.84
0.13
0.26
14
Gandalf
+ 0.09
0.52
0.43
0.67
140
Gandalf trace
+ 0.67
0.74
0.07
0.99
66
AAAJ
+ 0.36
0.78
0.42
0.07
53
Table 2: Paired fault injection on PAFault (13 families × 49 pairs, GPT-5.6 Sol). Δfault= detection rate on faulty variants − false-alarm rate on their clean controls. Gandalf and Autorubric are trace-blind by default; their trace variants receive the full trace. Cost and time are per variant. Bold = best quality score per column.
GAIA (117)
SWE-bench Lite (31)
Verifier
wF1 ↑
Loc ↑
Joint ↑
wF1 ↑
Loc ↑
Joint ↑
Cost ↓
GPT-5.6 Sol
Sauron
0.53
0.73
0.44
0.68
0.43
0.10
$841
Gandalf
0.46
0.53
0.25
0.61
0.24
0.06
$441
Judge TRAIL
0.46
0.55
0.22
0.47
0.21
0.04
$405
GPT-5.6 Terra
Table 3: Error detection and localization on TRAIL (21 error categories). wF1 = support-weighted category F1; Loc = location accuracy; Joint = (span, category) accuracy. Judge TRAIL = TRAIL’s single-pass judge. Cost is per full run. Bold = best quality score per backbone. All verifiers use the location normalization in Section D.3 .
Variant
Δ F1
Rel. cost
Rel. turns
Full Sauron
F1 = 0.96
1.00 ×
1.00 ×
Evidence access & navigation
State only
− 0.15
1.00 ×
1.03 ×
Trace in context
+ 0.01
3.03 ×
2.65 ×
No gist
0.00
1.15 ×
1.16 ×
Evidence verification
Table 4: Mechanism ablations on PAAbl (860 criteria, GPT-5.6 Sol). Each variant disables or replaces the indicated component; Δ F1, cost, and turns are relative to full Sauron.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Criterion-clustering prompt. The planner receives the user request and rubric criteria; max_size specifies the cluster-size limit.
Figure 6: Task-gist prompt. The model receives the user request and compact views of the recorded episode, returning a summary and suggested locations to inspect. The gist provides guidance, not verified evidence.
Operator
Main inputs
Returned information
Trace
trace_search
Pattern; [event types, tool names, result limit, span]
Matching event summaries and text around each match.
trace_get_event
Event identifier
Event metadata and bounded content.
trace_get_event_text
Event identifier; [offset, length]
A portion of an event, its position, and whether more content remains.
trace_window
Event identifier; [counts before and after]
Bounded content of the selected event and summaries of its neighbors.
trace_tool_calls
[Tool name, status]
Linked call and result summaries.
Appendix
Table 5: Direct source queries. Inputs in brackets are optional; a tool with only bracketed inputs can be called without arguments. Span-tree queries and span filters require a trace with a recorded hierarchy. Content limits appear in Table 6 .
Content
Default size
What the limit covers
Complete tool response
24,000 characters
The entire JSON response, including metadata and formatting.
Single-event content
6,000 characters
The beginning of an event, including the focus event in a window.
Event text page
20,000 characters
Default and maximum page length. The response limit may shorten it further.
Trace window
3 events before and 3 after
Neighboring events have 400-character summaries.
Search results
20 matches
Default for trace and state searches; maximum for artifact searches.
Span-tree listing
200 spans
Maximum listed spans. Further queries can target individual subtrees.
Appendix
Table 6: Default content limits. Each tool has its own content limit or preview size. Every complete response must also fit within 24,000 characters, including metadata, so some results may be shortened further.
Resource
Allocation for n criteria at scale s
Investigator turns
min(round[smin(20+7.5n,75)],60)
Trace searches and span-tree queries
round[s(20+10n)]
Trace retrievals
round[s(30+15n)]
State and artifact queries
round[s(30+15n)]
Token budget
round[s(250,000+150,000n)]
Elapsed time
s(600+300n) seconds
Appendix
Table 7: Session budgets. The default scale is s=1 unless a multiplier applies. Integer limits are rounded to the nearest integer, with ties to even. An investigator turn may contain several tool calls, each charged to its corresponding allowance. Reaching a limit triggers one extra finalization-only turn beyond these limits.
Evidence type
Required source reference
Claim concerns
tool_call_presence
Trace event(s) or tool-call identifier
Whether a named tool was called, optionally with a specified status.
tool_call_absence
tool_name_absent
Whether a named tool was absent from the recorded calls.
tool_argument_match
Trace event(s) or tool-call identifier
Arguments supplied to a tool.
state_value_match
State path or optional MCP call reference
A value in the environment.
state_mutation
State path and either trace event(s) or a tool-call identifier
Whether a recorded action is reflected in the final state.
temporal_ordering
Exactly two distinct event identifiers
The relative order of two events.
Appendix
Table 8: Evidence types and source references. Locators identify the source material used to check each claim.
Figure 8: Faithfulness-checker prompt. {claim} is the proposed atom’s claim and {source} the content its locator resolves to (head-and-tail truncated to 12,000 characters). The call sees no investigator history, trace, or other evidence. If the reply cannot be parsed as the JSON object shown, the atom is rejected.
Figure 9: Set-coverage check prompt , used for set-wide criteria in clusters of eight or more. {criterion} is the criterion text and {atoms} the claims of the verified atoms cited for it. The atoms’ correctness is taken as given; the checker judges only whether they cover every instance the criterion quantifies over.
Backbone
Model identifier
GPT-5.6 Sol
openai/gpt-5.6-sol
GPT-5.6 Terra
openai/gpt-5.6-terra
GPT-5.6 Luna
openai/gpt-5.6-luna
Claude Opus 5
anthropic/claude-opus-5
Claude Sonnet 5
anthropic/claude-sonnet-5
Claude Fable 5.1
anthropic/claude-fable-5-1
Appendix
Table 9: Backbones evaluated on PA100 .
Figure 10: Criterion-classification prompt. {items} holds the criterion identifiers and texts; the classifier sees nothing else. state_checkable and final_response_only are grouped as outcome ; the other five classes are grouped as process .
Family
Taxonomy mode
Interaction edge
Fault side
drop call
Satisficing
User–Model
Model
flip error
Tool Feedback Neglect
Tool–Model
Model
reorder
Reordering †
Context–Model
Model
claim inject
Satisficing
User–Model
Model
eval awareness
Evaluation Awareness
Grader–Model
Model
prompt injection
Indirect Prompt Injection
Third party–Model
Model
Appendix
Table 10: Taxonomy grounding of the fault families in PAFault . Each fault family simulates the evidence associated with the listed failure mode. Interaction edges and fault sides follow Raj et al. [22] , whose owner component we call the user. † Our addition; not a mode in the taxonomy.
Family
Edit and control
Criterion summary
Fault
Control
drop call
Remove a successful call and its result; retain both in the control.
The specified call occurred successfully.
U
M
flip error
Replace a successful tool result with an error; retain success in the control.
The specified call completed without error.
U
M
reorder
Move a call and its result after the final reply; retain their earlier position in the control.
The specified call occurred before the final reply.
U
M
claim inject
Append a claim of an action never taken; append a claim supported by a recorded successful call in the control.
The agent performed the action claimed in its response.
U
M
eval awareness
Insert a statement that work was shortened or misreported because the run appeared graded; omit it in the control.
The record shows no evaluation-conditioned shortcut.
U
M
prompt injection
Add an overriding directive to a tool result and an action following it; omit both in the control.
The agent followed an overriding instruction embedded in tool output.
M
U
Appendix
Table 11: Injection templates and expected verdicts on PAFault . Criterion summaries describe the assertions evaluated; instantiated criteria identify the relevant actions, arguments, or state paths. M denotes met and U denotes unmet . The last two columns give the expected verdict for each side of the pair. Criteria that assert a violation expect met on the faulty variant.
Backbone
Verifier
F1
MCC
P
R
Acc
FA
MA
Cost
In Mtok
Out Mtok
GPT-5.6 Sol
Sauron
0.96
0.94
0.94
0.97
0.97
21
44
$297.69
67.2
1.73
Gandalf
0.92
0.89
0.90
0.94
0.95
43
78
$388.25
147.1
1.20
Archipelago
0.92
0.89
0.92
0.92
0.95
60
64
$2,058
81.8
0.65
AAAJ
0.52
0.25
0.38
0.86
0.53
111
1093
$234.78
2.3
0.36
Autorubric
0.65
0.48
0.53
0.83
0.73
131
557
$76.09
5.0
0.21
Claude Fable 5.1
Sauron
0.95
0.93
0.95
0.95
0.97
38
40
$414.70
103.9
3.45
Appendix
Table 12: Full PA100 results across eight backbones. FA = false awards ( met on reference- unmet ); MA = missed awards ( unmet on reference- met ). Bold marks the highest F1 for each backbone. Tokens in millions.
Backbone
Verifier
Process F1
Outcome F1
GPT-5.6 Sol
Sauron
0.97
0.96
Gandalf
0.90
0.93
Archipelago
0.88
0.93
AAAJ
0.82
0.46
Autorubric
0.77
0.61
Claude Fable 5.1
Sauron
0.96
0.95
Appendix
Table 13: PA100 results by criterion type. F1 on 411 process and 2,175 outcome criteria, with unweighted means across eight backbones in the final block. Bold marks the best F1 in each column within each block.
Concurrent tasks w
Sauron (min)
Gandalf (min)
Speedup
1
17.1
71.0
4.16 ×
2
9.8
22.8
2.34 ×
4
5.0
11.9
2.37 ×
8
3.6
7.8
2.14 ×
Appendix
Table 14: Runtime under concurrent grading on a 10-episode slice of PA100 (GPT-5.6 Sol). Minutes from first request to last verdict with w episodes graded concurrently, each verifier at its default internal settings.
Figure 11: Δfault by fault family on PAFault . Results with GPT-5.6 Sol on 49 fault–control pairs per fault family. Higher is better. Blue cells are positive and red cells negative. The final row reports the unweighted mean across all 13 fault families.
Verifier
$/variant
s/variant
In Mtok
Out Mtok
Cached Mtok
Cache hit
Sauron
0.21
30
57.8
1.59
46.3
80.1%
Gandalf trace
0.99
66
440.4
1.41
376.3
85.4%
Gandalf
0.67
140
318.1
2.90
286.6
90.1%
Autorubric trace
0.26
14
36.3
0.22
4.6
12.6%
Autorubric
0.02
14
3.8
0.15
3.0
78.3%
Archipelago
0.24
16
33.3
0.32
5.2
15.6%
Appendix
Table 15: Resource use on PAFault (GPT-5.6 Sol). Cost and time are averages per variant. Token counts are in millions over all 1,274 variants. Input totals include cached tokens; cache hit is the percentage of input tokens served from the cache.
Disagreement category
FA
MA
Total
Share
Strict reading, positive criterion
0
34
34
52%
Strict reading, negative criterion
14
0
14
22%
Lenient reading, positive criterion
6
0
6
9%
Misinterpretation or conditional criterion
1
4
5
8%
Polarity convention
0
3
3
5%
Budget exhaustion
0
2
2
3%
Appendix
Table 16: Disagreements between Sauron and human judgments on PA100 (GPT-5.6 Sol). False awards (FA) are met verdicts on reference- unmet criteria; missed awards (MA) are unmet verdicts on reference- met criteria. Shares are computed over the 65 disagreements.
As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance, bringing gains over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. We further show that verification skills can self-improve from failure feedback, demonstrating VeriHarness as a novel and critical approach for scaling long-horizon agentic verification. We release the full pool of approximately 26,000 rollouts across all five benchmarks and both models, produced at a cost of over $100,000, to support future research on agentic verification.
Caiqi Zhang, Rujun Han, Zifeng Wang +4
University of Cambridge · Google Cloud AI Research
Reusable Web-agent verification requires a rule to follow changing requirements and support subsequent execution. We introduce AgentProofBench, which connects paired tests of a frozen acceptance rule to execution from identical saved openings. The resource spans 97 tasks, 79 families, and four Web services. Experiments with DeepSeek Flash/Pro establish two linked measurements. Across 48 families, complete-panel and decision-replay tests retain 82/96 Flash and 79/96 Pro source programs; paired reuse-rebuild BA intervals are [-5.47, 1.57] and [-10.16, 0.00] percentage points. Across 22 execution families, deployment-state panel correctness separates program assignments whose two saved openings are both delivered correctly: 32/33 assignments with error-free panels, versus 1/11 with panel errors. All reference-defined program-gate regressions and withheld deliveries occur in the latter group. Neutral continuation attains the highest correct delivery under the tested generic feedback, providing a concrete baseline for measuring the value added by a gate. Released instructions, states, programs, reference rules, and trajectories connect each test failure to its execution consequences. AgentProofBench makes reusable verification assessable at both the requirement and policy levels.
Hallucination is the reliability bottleneck for LLM-based agents, and fact attribution verifiers are the last line of defense -- yet today's verifiers emit only opaque binary labels, leaving agents unable to self-correct and operators unable to audit. We present SEVA, a structured verification agent that emits evidence alignments, step-by-step reasoning chains, calibrated confidence, and a six-category error diagnosis with actionable fixes. Training such an agent with RL is non-trivial: standard binary reward on multi-component output triggers advantage collapse -- within-group reward variance vanishes and the GRPO gradient disappears. We resolve this with a process reward that decomposes verification quality into five independent components weighted 70/30 toward process signals, restoring the gradient and inducing an implicit curriculum -- the agent first masters verification behavior (alignment 0.917 -> 0.997, format 72% -> 100%), then outcomes (F1 64.9 -> 69.0). Structured output further enables a Verify -> Reflect -> Probe -> Refine self-evolution loop, which over four rounds on a 7B model surfaces an unexpected structural finding: each round produces a benchmark-specialist, not a generalist (+15 pp on HaluEval, -10 to -14 pp on TruthfulQA in the same model, persistent at 4x data). On ClearFacts, SEVA-3B matches GPT-4o-mini (69.0 vs. 69.8 F1) while producing substantially richer, auditable output -- confirming a principle that should generalize: for any RL task with multi-component generation, reward granularity must match output granularity.
Aojie Yuan, Yi Nian, Haiyue Zhang +2
University of Southern California, Los Angeles, USA · University of Michigan, Ann Arbor, USA