Can you trust a finance agent you did not build? We take the reference agent shipped with the Vals Finance Agent Benchmark v2, leave it untuned, and ask four questions a leaderboard score does not answer. Is the score stable? Seven identical runs on 18 questions pass between 105 and 123 of 149 rubric criteria, and the bill varies by up to 17%. When the agent fails, who owns the fix? We classify 20 failed trajectories, from three runs of nine held-out questions, by the layer whose owner would remedy them, under a taxonomy frozen in advance: 13 belong to the agent's instructions, 3 to its tools, 4 to the model, none to its data. Can that diagnosis be trusted? Four blind readers from three model families, given only the traces, name the same fault as our LLM analyzer in 94% of cases, the rate at which they agree with one another (93%). Can a change be trusted? An optimization loop reads the traces, makes one targeted change per round and keeps it only if it clears a fixed threshold. Under a cost objective its final configuration costs 0.34x the baseline on the held-out questions (95% CI [0.22, 0.58]) at accuracy within noise. Under an accuracy objective it gains 14 points on the development questions and 1.5 on the held-out ones (CI [-5.3, +7.2]). Judged against the measured run-to-run noise, under a rule written before the test was opened, the first is a real saving and the second is not. This is a case study of one agent and nine test questions, not a general result.
Figures & tables
Figure 1. The system and the protocol. Agent, judge, analyzer and loop form the optimization cycle of § 6 , run on the dev split only. Shaded boxes are frozen before use: the test split and its decision rule before any test number existed (§ 6.3 ), the taxonomy before the test trajectories were analysed (§ 4.1 ).
idx
category
issuers
1
General qualitative analysis
MAR, WH
4
General quantitative analysis
CZR
8
Market analysis
HUBS, NOW, TOST
9
Comparables
AER, FTAI, WLFC
14
Precedents
PFE, SGEN
17
Adjustments
ASPI
Table 1. The nine test questions (Vals FABv2 public split).
run
cells/149
pass rate
agent $
1
119
0.799
34.42
2
116
0.779
28.58
3
120
0.805
32.48
4
123
0.826
32.28
5
105
0.705
31.79
6
116
0.779
32.98
Table 2. Seven identical runs of the shipped configuration on 18 dev questions, 149 rubric criteria. Twins are consecutive baselines; a flip is a criterion passed in one run and not the other.
Figure 2. The four measurements. Top left on the dev split, the other three on the frozen test split. (a) Cells passed by the shipped configuration in seven identical dev runs. (b) Primary layer of the 20 failed test trajectories per baseline replica. (c, d) Both loop winners on the frozen test, three replicas each, paired bootstrap over the nine questions: the cost winner’s cost interval lies entirely below the drift band; every accuracy interval includes zero.
fix_level
harness_component
layer
prompt
any but none
policy
prompt
none
policy or model, by Rule M
skill
any
policy
tool description
any
tool
orchestration
action, observation, state
tool
orchestration
context
data
Table 3. The projection, frozen before the test trajectories were analysed. fix_level is the primary key; harness_component decides only where the key is ambiguous.
replica
failed
policy
tool
data
model (A)
model (B)
r1
7
3
1
0
3
4
r2
7
6
1
0
0
0
r3
6
4
1
0
1
1
total
20
13
3
0
4 (20%)
5 (25%)
Table 4. Primary layer per failed trajectory, three test baseline replicas. Reading A is Rule M as written; reading B additionally treats calculator arguments as model-authored and is a sensitivity only.
question
r1
r2
r3
ASPI
policy
tool
policy
CRM
model
policy
policy
CZR
tool
policy
tool
FTAI
policy
policy
model
MAR/WH
model
policy
–
PFE (disclosure)
policy
policy
policy
Table 5. Primary layer of each failed test question across the three replicas (reading A). “–” = passed.
pair
same fault
within ± 5
analyzer vs Gemini Pro
17/20
10/20
analyzer vs Gemini Flash
19/20
11/20
analyzer vs GPT-5.5
20/20
10/20
analyzer vs Opus
19/20
14/20
analyzer vs readers
75/80
45/80
readers vs readers (6 pairs)
112/120
84/120
Table 6. Blind panel, 20 failed test trajectories. Same fault: judged on unlabelled justification pairs; under the pre-registered rule, which also counts decisive events within ± 5, the two totals are 75/80 and 113/120. Within ± 5: decisive event within five trace events.
round
lever
cells
$ (ratio)
verdict
base
as shipped
105
31.79
1
retrieval model → Haiku 4.5
120
13.90 (0.437)
promoted
2
tool desc.: batch per document
103
13.85 (0.996)
rejected
3
system prompt: batch per document
117
11.73 (0.843)
promoted
4
system prompt: chain arithmetic
117
11.24 (0.958)
rejected
5
system prompt: pivot after empty EDGAR
104
10.53 (0.898)
promoted
Table 7. Cost arm, dev split: five rounds, gate = cost ratio against the round’s parent.
round
lever
cells ( Δ )
$
verdict
base
as shipped
109
29.42
1
sys. prompt: scope qualifiers
116 (+7)
33.31
promoted
2
sys. prompt: five ambiguity rules
130 (+14)
38.09
promoted
3
sys. prompt: re-read item by item
113 ( − 17)
36.98
rejected
4
sys. prompt: reconcile multi-year
117 ( − 13)
36.81
rejected
Table 8. Accuracy arm, dev split: four rounds then plateau, gate = cells against the round’s parent.
configuration
cells/270
pass rate
Δ pts
95% CI
must-pass ok/27
total $
cost ratio
95% CI
verdict (accuracy / cost)
baseline (shipped)
171
0.633
9
71.17
cost winner (round 5)
161
0.596
− 3.7
[ − 8.9, +1.1]
11
24.27
0.341
[0.216, 0.580]
within noise / real saving
cost arm, round 3 (secondary)
168
0.622
− 1.1
[ − 7.4, +5.4]
12
28.26
0.397
[0.325, 0.495]
within noise / real saving
accuracy winner (round 2)
175
0.648
+1.5
[ − 5.3, +7.2]
11
76.98
1.082
[0.844, 1.505]
within noise / within noise
Table 9. Both loop winners on the frozen test split, 9 questions × 3 replicas, pooled. Intervals are paired bootstraps over the nine questions (10,000 resamples). The noise band for cost is [0.83, 1/0.83], the widest dev twin ratio; the accuracy rule is whether the interval excludes zero. Verdicts are those the pre-registered rule returns.
question
base cells
winner cells
base $
winner $
MAR/WH
17/21
17
6.63
1.60
CZR
0/24
0
13.01
2.35
HUBS/NOW/TOST
27/27
27
3.81
1.74
AER/FTAI/WLFC
22/39
20
3.81
1.73
PFE/SGEN
27/27
27
4.70
0.74
ASPI
4/27
6
9.26
5.33
Table 10. Cost arm, per test question, summed over three replicas.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
replica
question
decisive event
primary category
fix × comp
layer A
layer B
r1
ASPI
assistant #20
EDGE_CASE
prompt × none
policy
r1
CZR
tool_result #18
INTEGRATION_ERROR
orchestration × observation
tool
r1
CRM
submit #143
INSTRUCTION_VIOLATION
prompt × none
model
r1
FTAI
calculator #53
REASONING_ERROR
prompt × none
policy
model
r1
PFE
assistant #49
KNOWLEDGE_GAP
prompt × context
policy
r1
MAR/WH
submit #33
REASONING_ERROR
prompt × none
model
Appendix
Table 11. Per-trajectory verdicts, three test baseline replicas. “fix × comp” is the analyzer’s fix_level × harness_component of the primary category.
Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.
Financial agents are becoming an important class of real-world AI systems. Financial tasks vary widely in type and difficulty, and many are open-ended with no ground-truth answers. However, existing benchmarks neither capture this breadth and open-endedness nor separate grounding from correctness. To bridge this gap, we introduce AgenticFinBench, a benchmark of 230 real-world financial tasks spanning 8 categories, 2 languages, and 4 complexity levels. We also propose a reference-free evaluation framework that combines question-specific rubrics with a two-stage claim audit that detects hallucinations, defined as claims absent from the agent's trace and refuted by external evidence. This framework better matches how financial work is assessed in practice. Experiments across 18 agent configurations reveal three key findings: (1) Ignoring hallucinations can overstate agent performance: hallucination auditing lowers the baseline leader's score by 7.58 points and moves it to fourth place. (2) Harness differences explain more variation in evidence grounding than model differences (28.6% vs 8.8%). (3) Quality scores fall from 89.44 on direct retrieval to 79.18 on the most complex tasks. Case studies show that agents struggle to explain how events affect company profits or stock prices, compare financial data on a consistent basis, and justify their conclusions. By releasing AgenticFinBench, we aim to advance financial reasoning, factual verification, and agent design for more reliable responses to diverse real-world financial requests.
Evaluating finance research agents requires rubrics that reflect expert standards and fix the values correct as of an information cutoff. Expert-reviewed finance benchmarks rely on fixed, per-item rubrics, which are costly to extend and cannot encode each institution's own standard. In FinAutoRubric, experts specify reusable evaluation guidance, while agents and code carry out query-specific rubric generation, review, and validation. This expert guidance governs every agent, as prompts and as rules that code enforces, and a Task Bank of reusable criteria carries it across tasks. In long-horizon loops that follow the expert guidance, a writer agent researches every expected value and a reviewer agent verifies it, and failures escalate to a human. On three expert-authored finance benchmarks, its rubrics track expert scoring as closely as the strongest evaluated generator while stating the expert rubric's expected value for more criteria, their scores agree with human grading, and in-house analysts prefer them in a blind review. The released 100-query FinAutoRubric Benchmark, built from in-house analysts' key questions across 78 tasks and eight asset classes, shows that rubrics from an earlier model generation still leave headroom for a later one.
Financial-research answers are decision-relevant only when another analyst can audit how they were produced: which source was chosen, which period and accounting definition were used, which assumptions were made, and how the calculation was performed. Existing finance benchmarks largely evaluate isolated subskills or final answers, leaving the auditable derivation itself under-measured. We introduce BigFinanceBench, a 928-item expert-authored benchmark of open-ended financial-research tasks in which each item pairs a ground-truth reference answer with a point-weighted rubric that decomposes the derivation into independently checkable steps. BigFinanceBench is workflow-grounded in that it evaluates the full derivation rather than only the final output. Across 36,241 rubric points, the benchmark supports partial-credit evaluation and localization of failures across the analyst workflow. Evaluating ten current frontier and open-weight agents, we find substantial headroom: the best system reaches only 58.8% rubric score, final-answer accuracy is a useful but lossy proxy for derivation quality, and model capability varies non-uniformly across financial workflows.