Professional work involves under-specified assignments, where relevant information is spread across workplace applications and may conflict or change over time. To evaluate how well frontier agents can perform economically valuable work in such settings, we introduce ATLAS-Finance, a benchmark of 100 end-to-end investment-banking tasks across 13 simulated firms. As in real banking workflows, agents must search email, chat, calendars, notes, and files to identify requirements, gather information, reconcile conflicts/updates, and then build complex financial models. Tasks were developed by experienced bankers based on their real deal work and take them 15-30 hours to complete. Agent deliverables are evaluated with expert-authored rubrics (median: 124.5 criteria) covering financial rationale and results, spreadsheet construction, completeness, and formatting. Despite completing substantial portions of the requested work, current agents rarely produce correct deliverables. Across today's frontier models, the best ones (Claude Opus and Fable) pass fewer than 11% of tasks.
Figures & tables
Figure 1: Walkthrough of an example task called Project Helios. The figure shows a shortened version of the initial prompt and relevant environment information. The agent must find and reconcile additional instructions and source materials, then build and submit a workbook implementing a formula-driven project-finance model. The agent’s work is subsequently graded against an expert rubric by an agentic verifier provided as part of the benchmark.
Workflow
Share
Included workflows
M&A, capital markets, and strategic transactions
32%
M&A and divestitures, strategic alternatives and board decisions, capital markets, minority investments and exits
LBO and sponsor underwriting
19%
LBO and take-private underwriting, carve-out, buy-and-build and turnaround LBOs, portfolio re-underwriting and dividend recapitalization
Operating, public-equity, and valuation models
17%
Public-company and coverage valuation, operating plans, pricing and growth models, asset and portfolio valuation
Project and infrastructure finance
15%
Renewable energy and energy-transition finance, infrastructure, network and utility finance, asset and capital-program finance, export finance
Credit, structured finance, and restructuring
13%
Distressed credit and restructuring, asset-based lending (ABL), refinancing and lender monitoring, structured credit and securitization
Operating-model rebuilds
4%
Detailed operating-model rebuilds across multiple entities or locations
Table 1: Distribution of ATLAS-Finance tasks across six workflow families.
Figure 2: Project Argos correspondence defining the transaction scope, modeling rules, source updates, and tax guidance.
Component
Mean (range)
Tasks
7.7 tasks (6–10)
Outlook
1,360 unique emails (996–1,912), 834 threads (324–1,163), and 7,075 mailbox records (999–11,381)
Slack
27 conversations (11–49) and 412 messages (3–1,148)
Google Drive
176 files (45–284) and 7.8 folders (0–22)
Google Calendar
14.5 accounts (4–18) and 228 events (25–390)
OneNote
5.7 notebooks (1–11), 18.5 sections (6–36), and 19.8 pages (5–56)
Table 2: Mean counts across 13 environments, with ranges in parentheses. Runtime updates are counted across all tasks within each environment.
Figure 3: Distribution of rubric size across the 100 ATLAS-Finance tasks. The median is 124.5 criteria.
Figure 4: Pass rates on ATLAS-Finance. Error bars show 95% confidence intervals computed via bootstrap.
Model
Mean score (%)
Pass@3 (%)
Claude Opus 5
71.7 ±
0.6
15.0
Claude Fable 5.1
71.5 ±
0.4
13.0
GPT-6 Astra
70.0 ±
0.4
14.0
Kimi K3
64.4 ±
0.6
5.0
Grok 4.6
64.3 ±
0.7
6.0
GLM 5.3
61.9 ±
0.7
7.0
Table 3: Mean rubric scores and Pass@3 on ATLAS-Finance. Error ranges show 95% confidence intervals computed via bootstrap.
Model
M&A / Cap. Mkts.
LBO
Operating / Valuation
Project Finance
Credit / Structured
Operating Rebuilds
Overall
Claude Opus 5
74.3
66.7
73.9
75.5
64.5
74.7
71.7
Claude Fable 5.1
71.4
68.2
75.4
76.8
64.6
73.1
71.5
GPT-6 Astra
73.4
64.6
75.3
69.0
61.8
76.9
70.0
Kimi K3
68.3
57.6
70.2
62.3
57.9
71.3
64.4
Grok 4.6
66.9
58.0
66.5
69.8
55.1
74.1
64.3
GLM 5.3
64.5
58.0
65.9
61.2
54.8
69.1
61.9
Table 4: Mean rubric scores (%) achieved for the tasks within each workflow family.
Model
Upfront (%)
Workplace (%)
Δ (pp) 95% CI
GPT-6 Astra
80.2
70.0
10.2 [7.3, 13.4]
Claude Opus 5
79.2
71.7
7.5 [3.3, 11.7]
Grok 4.6
76.1
64.3
11.8 [7.5, 16.3]
Table 5: Mean rubric scores with all information upfront versus the full workplace. Intervals are 95% environment-clustered bootstrap confidence intervals for paired differences.
Model
Financial logic
Finance integration
Spreadsheet engineering
Data handling
Scope / completeness
Presentation / delivery
GPT-6 Astra
58.3
0.5
8.3
0.0
25.2
7.6
GPT-5.6 Sol
54.1
1.4
7.7
0.3
26.5
10.1
Fable 5.1
70.9
0.8
6.1
0.0
17.1
5.1
Opus 5
64.0
0.6
6.8
0.0
22.0
6.6
Gemini 3.8 Flash
49.8
2.2
11.1
0.0
27.3
9.6
Gemini 3.7 Flash
44.9
1.7
14.4
0.0
25.8
13.3
Table 6: Distribution of root causes by failure category. Percentages are calculated within each model and describe error composition, not overall performance. All values are percentages. Rows may not sum to 100 due to rounding.
Category
Definition
Financial logic
Incorrect financial methodology, accounting treatment, economic assumption, or calculation basis.
Finance integration
Required components exist but are not propagated into the calculation or control they are intended to govern.
Spreadsheet engineering
Formula wiring, reference, circularity, hardcoding, or workbook-construction defect whose first repair is mechanical.
Data handling
Wrong source, incomplete provenance, missing as-of information, or incorrect ingestion of source data.
Scope/completeness
Required schedule, scenario, output, or workstream is absent and no more specific upstream error is identified.
Presentation/delivery
Formatting, filename, recipient, subject, attachment, thread, or other submission-contract violation.
Table 7: Failure-category definitions used for root-cause attribution.
Figure 5: An unsupported assumption survives validation. The agent introduces an existing-goodwill deduction in its Python shadow model and carries the same treatment into the Excel workbook. Because subsequent checks validate the workbook against that shadow model, the internal checks pass even though the tax treatment is incorrect.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Requirements that determine passing across the 100 tasks: critical criteria per task (left), critical criteria as a share of positive criteria (center), and penalty criteria per task (right). Dashed lines mark medians.
Judge
Agreement with experts
Cohen’s κ
Cost (USD per rollout)
Gemini 3.8 Flash
93.6% (1,082/1,156)
0.872
$3.88
Claude Opus 4.8
92.0% (1,063/1,156)
0.840
$6.45
Appendix
Table A1: Comparing our verifier against expert grading of two agent models’ outputs across 1,156 grading decisions.
Figure A2 : Share of tasks with at least one failed section gate or triggered penalty, by model. A task counts as affected when the event occurs in any attempt.
Figure A3 : Mean rubric score versus cost per rollout (left) and output tokens per rollout (right).
Scheme
Kendall τ
Max ∣Δrank∣
Max ∣Δscore∣ (pp)
Original weights
1.00
0
0.0
Equal weights
0.96
1
3.2
Square-root weights
0.96
1
1.6
Squared weights
1.00
0
3.4
Half penalties
1.00
0
0.4
Double penalties
1.00
0
0.7
Appendix
Table A2 : Sensitivity of model rankings and mean rubric scores to weight and penalty changes.
Figure A4 : Survey results ( n=43 ): Willingness to pay per user seat and intended deployment under hypothetical system-performance scenarios.
Existing AI benchmarks lack the fidelity to assess economically meaningful progress on professional workflows. To evaluate frontier AI agents in a high-value, labor-intensive profession, we introduce BankerToolBench (BTB): an open-source benchmark of end-to-end analytical workflows routinely performed by junior investment bankers. To develop an ecologically valid benchmark grounded in representative work environments, we collaborated with 502 investment bankers from leading firms. BTB requires agents to execute senior banker requests by navigating data rooms, using industry tools (market data platform, SEC filings database), and generating multi-file deliverables--including Excel financial models, PowerPoint pitch decks, and PDF/Word reports. Completing a BTB task takes bankers up to 21 hours, underscoring the economic stakes of successfully delegating this work to AI. BTB enables automated evaluation of any LLM or agent, scoring deliverables against 100+ rubric criteria defined by veteran investment bankers to capture stakeholder utility. Testing 9 frontier models, we find that even the best-performing model (GPT-5.4) fails nearly half of the rubric criteria and bankers rate 0% of its outputs as client-ready. Our failure analysis reveals key obstacles (such as breakdowns in cross-artifact consistency) and improvement directions for agentic AI in high-stakes professional workflows.
AI agents are increasingly deployed for professional investment research, yet no benchmark captures the complexity of the full investor workflow. Existing benchmarks mainly target financial data extraction, a narrow slice that current models have largely saturated, while reference-based metrics and generic LLM-as-a-judge scoring fall short on the open-ended, long-form answers that real analyst queries demand. We introduce FrontierFinance, a fully open benchmark of 220 expert-crafted queries and 11,543 source-attributed rubrics spanning six crucial use cases across the full investor workflow. FrontierFinance is both broader and harder than existing public finance benchmarks. Evaluating frontier models and agent systems under a common harness restricted to publicly available data, we find that the tool harness, not the model alone, strongly shapes quality and efficiency; that Samaya's in-house system leads at 56.0%, ahead of the strongest frontier model (Claude Fable 5, 49.2%) at roughly 2.2x lower cost; and that the best open-weight model (Kimi K3, 46.4%) nearly matches the best proprietary model at 4.5x lower cost. Screening & Discovery and Sector, Industry & Macro remain the hardest use cases across all systems, where even the best systems reach only 33% and 39%. We make the dataset and grading code publicly available.
Yuhao Zhang, O. Ozan Koyluoglu, Thejas Venkatesh +4
As AI agents advance, the central question is no longer whether they can solve isolated financial tasks, but whether their capabilities generalize across complete professional workflows. Existing benchmarks cannot answer this question because they evaluate either static competencies or agents within a single interaction structure. We introduce Herculean, a benchmark of end-to-end agentic financial intelligence that crosses agent architecture, backbone model, and four structurally distinct workflows: Trading, Hedging, Market Insights, and Auditing. Each workflow is implemented as an MCP skill that preserves its native state, actions, constraints, and outcome criteria, allowing intact agent systems to be evaluated under matched external conditions. Across 30 architecture--backbone configurations, we find that agent capability is not globally ordered. Workflow rankings are weakly aligned, changing the architecture frequently reverses performance under the same backbone, and each architecture concentrates failure in different forms of workflow control. These results establish that agent performance is an interaction between architecture and workflow demand, rather than an intrinsic property of the underlying model. Consequently, success on a single financial workflow provides little evidence of dependable, general financial intelligence.