Organizations: The Fin AI · Stevens Institute of Technology · Independent Researcher · University of Manchester · MBZUAI · DeepKin · University of Montreal
Language agents increasingly work for users on the open Web, and finance is where trusting them is hardest. The data must come from the Web, the strategy lives in the user's head, and correctness depends on time, so an agent can fail silently: a strategy the user never asked for, unverified data, or a backtest that uses the future yields a report as convincing as a correct one. Existing benchmarks close the world somewhere, fixing the task, the data, or the timeline, and thereby remove these failures instead of measuring them. We introduce FINCOSMOS, an open-world evaluation paradigm for agents doing financial work. A simulated user holds a hidden strategy sampled from a compositional grammar and reveals it only when asked. The agent receives no data, so it must acquire data from the Web, build its own backtest under a data cutoff, and deliver verifiable results. Grading needs no reference answer. Programs verify point-in-time correctness and recompute every reported metric, the delivered strategy is checked against the hidden specification, and a run is accepted only when the user is satisfied and none of 15 veto checks fails. Across four coding agents and a model without tools in 150 sessions, no session is accepted: every session that the simulated user signed off fails a program check, and agents raise only 54--64% of the details they were not told. Human experts attribute 88.33% of failures to the agents. With FINCOSMOS feedback, an agent with persistent memory learned all ten tasks.
Figures & tables
Figure 1: A silent failure. The user is satisfied, yet the delivery gets the intent and the data wrong; only time is right.
Open real world
Tasks and evaluation
Benchmark
Hidden intent
Self-src. data
Point-in- time
Gen. tasks
Exec. deliv.
Ref.- free
FinBen, MultiFinBen [ 57 , 39 ]
FinDeepResearch [ 70 ]
AlphaForgeBench [ 65 ]
CLQT [ 43 ]
InvestorBench [ 27 ]
Table 1: Comparison of F i n C o s m o s with related benchmarks.
Figure 2: Overview of F i n C o s m o s . Left: its three components hide a strategy b from the agent and grade what it delivers. Right: the agent works in the open real world, finds its own data, and must stay before the cutoff. Red badges mark where each silent failure arises and the check that catches it.
None, stop-loss with position caps, drawdown gate, volatility targeting, VaR/CVaR limit, tail hedge.
5. Execution
Single fill, TWAP, VWAP, POV, implementation shortfall.
6. Transaction cost
None, fixed basis points.
Table 2: Options for the six strategy components.
Outcome
Quality (0–100)
Process
Cost per session
Agent
Sat.
AC
pass^2
Quality
Data
Strategy
Backtest
Assess.
UX
Vetoes
Cov.
Cut
Tokens
USD
(%) ↑
(%) ↑
↑
[95% CI] ↑
↑
↑
↑
↑
↑
(#) ↓
(%) ↑
(%) ↓
(M) ↓
↓
Claude Code
30
0
0
37.1 [27.8, 45.0]
44.7
38.2
26.9
23.2
52.2
7.4
54
65
1.13
2.24
Codex
33
0
0
37.2 [30.6, 43.3]
45.8
33.4
31.9
17.2
57.6
7.2
64
69
3.20
0.80
Gemini CLI
33
0
0
22.2 [16.9, 27.4]
25.3
30.8
15.6
7.0
32.4
9.3
57
75
3.51
0.80
Kimi Code
20
0
0
22.3 [14.3, 31.6]
26.8
23.4
16.5
6.5
38.2
10.1
57
74
0.89
0.83
Table 3: Main results on the ten frozen tasks in the realistic condition, with three sessions per task (30 per agent). Bold marks the best of the four agents in each column; arrows give the better direction. Quality is the pre-veto quality with a 95% bootstrap interval over tasks, and category scores use the same 0–100 scale. Cut: agent turns stopped at the 300-second limit. Cost is API-equivalent; Codex and Kimi Code ran on subscription plans. The bare model is Claude Opus 5.5 without tools.
Agent
Signed off
+ LLM-judged vetoes
All vetoes
Claude Code
9
7
0
Codex
10
8
0
Gemini CLI
10
0
0
Kimi Code
6
2
0
All 150 sessions
35 (23%)
17 (11%)
0
Table 4: Sessions counted as successes under three graders, of 30 per agent. LLM-judged vetoes: 2.7, 4.1, 5.1, and 5.6. All vetoes: the acceptance rule of F i n C o s m o s .
Step
Claude
Codex
Gemini
Kimi
Bare
Agents
In seq.
(1) User’s strategy
10
9
2
4
9
25
25
(2) Data in cutoff
11
24
11
9
0
55
15
(3) No look-ahead
21
23
9
10
0
63
14
(4) Reproducible
0
0
0
0
0
0
0
Table 5: Sessions passing each step of the workflow, of 30 per agent. (1) The delivered strategy is the user’s (2.1–2.7). (2) Data stay within the cutoff (1.1). (3) Fills follow their signals (3.1). (4) The delivered trades reproduce the reported returns (3.2 ≥ 2). Agents: the four agents together, of 120. In sequence: agent sessions that pass this step and every earlier one.
Failed vetoes per session
Diff.
Sess.
1st
First 5
Last 5
Satisfied
Last to clear
7
12
3
2.8
4.6
73%
3.2 Independent recomputation
10
132
14
9.4
1.6
31%
2.7 Strategy ambiguity
11
23
10
6.4
1.4
59%
2.7 Strategy ambiguity
11
26
4
6.4
2.2
28%
2.7 Strategy ambiguity
12
113
9
9.0
1.4
6%
2.7 Strategy ambiguity
Table 6: Learning from feedback on the ten tasks by difficulty. Sessions: up to the first acceptance. Failed vetoes: in the first session, and averaged over the first and last five sessions before acceptance. Satisfied: share of those sessions the user signed off. Last to clear: the veto that failed last.
Figure 3: Failed vetoes per session along practice for each of the ten tasks (gray) and their median (blue): in the first session, averaged over the first and the last five sessions before acceptance, and in the accepted session.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
ID
Dimension
Judge
Veto
Data
1.1
Point-in-time strictness
P
<4
1.2
Source traceability
L
1.3
Source diversity
L
1.4
Delivered data quality
P
1.5
Evidence grade match
L
Appendix
Table 7: Rubric: the 45 dimensions, their judges, and veto thresholds.
Financial agents are becoming an important class of real-world AI systems. Financial tasks vary widely in type and difficulty, and many are open-ended with no ground-truth answers. However, existing benchmarks neither capture this breadth and open-endedness nor separate grounding from correctness. To bridge this gap, we introduce AgenticFinBench, a benchmark of 230 real-world financial tasks spanning 8 categories, 2 languages, and 4 complexity levels. We also propose a reference-free evaluation framework that combines question-specific rubrics with a two-stage claim audit that detects hallucinations, defined as claims absent from the agent's trace and refuted by external evidence. This framework better matches how financial work is assessed in practice. Experiments across 18 agent configurations reveal three key findings: (1) Ignoring hallucinations can overstate agent performance: hallucination auditing lowers the baseline leader's score by 7.58 points and moves it to fourth place. (2) Harness differences explain more variation in evidence grounding than model differences (28.6% vs 8.8%). (3) Quality scores fall from 89.44 on direct retrieval to 79.18 on the most complex tasks. Case studies show that agents struggle to explain how events affect company profits or stock prices, compare financial data on a consistent basis, and justify their conclusions. By releasing AgenticFinBench, we aim to advance financial reasoning, factual verification, and agent design for more reliable responses to diverse real-world financial requests.
Can you trust a finance agent you did not build? We take the reference agent shipped with the Vals Finance Agent Benchmark v2, leave it untuned, and ask four questions a leaderboard score does not answer. Is the score stable? Seven identical runs on 18 questions pass between 105 and 123 of 149 rubric criteria, and the bill varies by up to 17%. When the agent fails, who owns the fix? We classify 20 failed trajectories, from three runs of nine held-out questions, by the layer whose owner would remedy them, under a taxonomy frozen in advance: 13 belong to the agent's instructions, 3 to its tools, 4 to the model, none to its data. Can that diagnosis be trusted? Four blind readers from three model families, given only the traces, name the same fault as our LLM analyzer in 94% of cases, the rate at which they agree with one another (93%). Can a change be trusted? An optimization loop reads the traces, makes one targeted change per round and keeps it only if it clears a fixed threshold. Under a cost objective its final configuration costs 0.34x the baseline on the held-out questions (95% CI [0.22, 0.58]) at accuracy within noise. Under an accuracy objective it gains 14 points on the development questions and 1.5 on the held-out ones (CI [-5.3, +7.2]). Judged against the measured run-to-run noise, under a rule written before the test was opened, the first is a real saving and the second is not. This is a case study of one agent and nine test questions, not a general result.
Financial agents produce numerical claims that can be wrong even when their calculations execute correctly, due to mismatched reporting periods, unsupported sources, incompatible units, or incorrect financial definitions. We introduce FinVerify, an evidence-grounded framework for verifying financial agents' numerical claims. Based on an analysis of 190 incorrect trajectories from the Agent Finance Reasoning dataset, we define five error categories--temporal integrity, source fidelity, comparability, accounting integrity, and financial-model fidelity--and operationalize them as verification protocols. An LLM structures claims and supporting evidence, while deterministic validators check financial requirements; when authoritative formulas are available, Z3 checks consistency with grounded inputs. FinVerify accepts only if all applicable checks pass, rejects proven violations, and abstains when evidence is insufficient. We evaluate the framework against a Direct baseline and an LLM judge across five fixed-candidate benchmarks. On Agent Finance Reasoning, accepted-claim precision increases from 56.4% for the LLM judge to 76.7% at 81.2% coverage. On XBRLFiling, FinVerify achieves 99.8% accepted-claim precision and 97.0% decision accuracy; on FinanceBench-67, it reaches 87.9% accepted-claim precision versus 69.7% for the judge. Improvements are smaller on FinQA and TAT-QA, where authoritative evidence is limited. These results suggest that explicit, evidence-backed verification can improve financial-agent reliability while exposing trade-offs between correctness, false rejection, and abstention.
Mao Guan, Qian Chen, Ge Meng +1
Independent Researcher · New York University · Columbia University