Organizations: The Fin AI · Stevens Institute of Technology · Independent Researcher · University of Manchester · MBZUAI · DeepKin · University of Montreal
Language agents increasingly work for users on the open Web, and finance is where trusting them is hardest. The data must come from the Web, the strategy lives in the user's head, and correctness depends on time, so an agent can fail silently: a strategy the user never asked for, unverified data, or a backtest that uses the future yields a report as convincing as a correct one. Existing benchmarks close the world somewhere, fixing the task, the data, or the timeline, and thereby remove these failures instead of measuring them. We introduce FINCOSMOS, an open-world evaluation paradigm for agents doing financial work. A simulated user holds a hidden strategy sampled from a compositional grammar and reveals it only when asked. The agent receives no data, so it must acquire data from the Web, build its own backtest under a data cutoff, and deliver verifiable results. Grading needs no reference answer. Programs verify point-in-time correctness and recompute every reported metric, the delivered strategy is checked against the hidden specification, and a run is accepted only when the user is satisfied and none of 15 veto checks fails. Across four coding agents and a model without tools in 150 sessions, no session is accepted: every session that the simulated user signed off fails a program check, and agents raise only 54--64% of the details they were not told. Human experts attribute 88.33% of failures to the agents. With FINCOSMOS feedback, an agent with persistent memory learned all ten tasks.
Figures & tables
Figure 1: A silent failure. The user is satisfied, yet the delivery gets the intent and the data wrong; only time is right.
Open real world
Tasks and evaluation
Benchmark
Hidden intent
Self-src. data
Point-in- time
Gen. tasks
Exec. deliv.
Ref.- free
FinBen, MultiFinBen [ 57 , 39 ]
FinDeepResearch [ 70 ]
AlphaForgeBench [ 65 ]
CLQT [ 43 ]
InvestorBench [ 27 ]
Table 1: Comparison of F i n C o s m o s with related benchmarks.
Figure 2: Overview of F i n C o s m o s . Left: its three components hide a strategy b from the agent and grade what it delivers. Right: the agent works in the open real world, finds its own data, and must stay before the cutoff. Red badges mark where each silent failure arises and the check that catches it.
None, stop-loss with position caps, drawdown gate, volatility targeting, VaR/CVaR limit, tail hedge.
5. Execution
Single fill, TWAP, VWAP, POV, implementation shortfall.
6. Transaction cost
None, fixed basis points.
Table 2: Options for the six strategy components.
Outcome
Quality (0–100)
Process
Cost per session
Agent
Sat.
AC
pass^2
Quality
Data
Strategy
Backtest
Assess.
UX
Vetoes
Cov.
Cut
Tokens
USD
(%) ↑
(%) ↑
↑
[95% CI] ↑
↑
↑
↑
↑
↑
(#) ↓
(%) ↑
(%) ↓
(M) ↓
↓
Claude Code
30
0
0
37.1 [27.8, 45.0]
44.7
38.2
26.9
23.2
52.2
7.4
54
65
1.13
2.24
Codex
33
0
0
37.2 [30.6, 43.3]
45.8
33.4
31.9
17.2
57.6
7.2
64
69
3.20
0.80
Gemini CLI
33
0
0
22.2 [16.9, 27.4]
25.3
30.8
15.6
7.0
32.4
9.3
57
75
3.51
0.80
Kimi Code
20
0
0
22.3 [14.3, 31.6]
26.8
23.4
16.5
6.5
38.2
10.1
57
74
0.89
0.83
Table 3: Main results on the ten frozen tasks in the realistic condition, with three sessions per task (30 per agent). Bold marks the best of the four agents in each column; arrows give the better direction. Quality is the pre-veto quality with a 95% bootstrap interval over tasks, and category scores use the same 0–100 scale. Cut: agent turns stopped at the 300-second limit. Cost is API-equivalent; Codex and Kimi Code ran on subscription plans. The bare model is Claude Opus 5.5 without tools.
Agent
Signed off
+ LLM-judged vetoes
All vetoes
Claude Code
9
7
0
Codex
10
8
0
Gemini CLI
10
0
0
Kimi Code
6
2
0
All 150 sessions
35 (23%)
17 (11%)
0
Table 4: Sessions counted as successes under three graders, of 30 per agent. LLM-judged vetoes: 2.7, 4.1, 5.1, and 5.6. All vetoes: the acceptance rule of F i n C o s m o s .
Step
Claude
Codex
Gemini
Kimi
Bare
Agents
In seq.
(1) User’s strategy
10
9
2
4
9
25
25
(2) Data in cutoff
11
24
11
9
0
55
15
(3) No look-ahead
21
23
9
10
0
63
14
(4) Reproducible
0
0
0
0
0
0
0
Table 5: Sessions passing each step of the workflow, of 30 per agent. (1) The delivered strategy is the user’s (2.1–2.7). (2) Data stay within the cutoff (1.1). (3) Fills follow their signals (3.1). (4) The delivered trades reproduce the reported returns (3.2 ≥ 2). Agents: the four agents together, of 120. In sequence: agent sessions that pass this step and every earlier one.
Failed vetoes per session
Diff.
Sess.
1st
First 5
Last 5
Satisfied
Last to clear
7
12
3
2.8
4.6
73%
3.2 Independent recomputation
10
132
14
9.4
1.6
31%
2.7 Strategy ambiguity
11
23
10
6.4
1.4
59%
2.7 Strategy ambiguity
11
26
4
6.4
2.2
28%
2.7 Strategy ambiguity
12
113
9
9.0
1.4
6%
2.7 Strategy ambiguity
Table 6: Learning from feedback on the ten tasks by difficulty. Sessions: up to the first acceptance. Failed vetoes: in the first session, and averaged over the first and last five sessions before acceptance. Satisfied: share of those sessions the user signed off. Last to clear: the veto that failed last.
Figure 3: Failed vetoes per session along practice for each of the ten tasks (gray) and their median (blue): in the first session, averaged over the first and the last five sessions before acceptance, and in the accepted session.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
ID
Dimension
Judge
Veto
Data
1.1
Point-in-time strictness
P
<4
1.2
Source traceability
L
1.3
Source diversity
L
1.4
Delivered data quality
P
1.5
Evidence grade match
L
Appendix
Table 7: Rubric: the 45 dimensions, their judges, and veto thresholds.