Organisations keep decades of financial records as scanned PDFs on shared drives, and AI agents can now reach those drives through Model Context Protocol (MCP) servers. Existing benchmarks do not mirror this production setting because they simplify the task. In some, the relevant documents are handed to the agent, so it never has to search. In others, the agent searches through documents with pre-extracted text, so it never has to read scanned images. We introduce TreasuryQA Bench, which requires both: 70 questions over 1,435 scanned U.S. Treasury publications, held in Veris's high-fidelity Dropbox twin. In each task the agent searches for the relevant file through Dropbox's MCP server, reads the pages it needs with Reducto's hosted OCR server or its own tools, and replies with one exact figure. A Python grader checks each answer against an accepted range around the correct value. In this paper we present results from ten agents. The best, Claude Opus 5.5 in Claude Code, solves 64.8% of attempts and the worst, GPT-5.6 Terra in Codex, solves 16.7%, so the benchmark is far from saturated. We also test how scan quality affects performance: every agent solves fewer tasks on the blurriest scans. Most failed attempts reply with a wrong number rather than no number, and the most common cause is a digit misread from the scan. Three worked examples are included with their answer keys.
Figures & tables
Benchmark
What the agent reads
Scans
Retrieval
Live tools
Grading
Tasks
DocVQA [ 12 ]
one page image
✓
✗
✗
ANLS similarity
50,000
TAT-DQA [ 25 ]
1–3 report pages
∘
✗
✗
exact match, F1
16,558
GAIA [ 14 ]
web pages, files
✗
✓
∘
quasi-exact match
466
BrowseComp [ 19 ]
open web
✗
✓
∘
LLM judge
1,266
OfficeQA Pro [ 16 ]
Treasury Bulletins
∘
✓
∘
exact match
133
Finance Agent [ 3 ]
SEC filings, web
✗
✓
✓
rubric, LLM judge
537
Table 1: TreasuryQA Bench and the closest benchmarks. Scans : page images without machine-readable text. Retrieval : the agent must locate the source documents in a larger collection. Live tools : tool servers or stateful services. ∘ : partly (part of the corpus scanned; open web or a local copy only).
Figure 1: One TreasuryQA Bench attempt. The agent finds the volume in a fresh copy of the Veris Dropbox twin through Dropbox’s MCP server and hands a download link to Reducto’s MCP server, which returns the pages it asks for. The attempt is solved if its ANSWER: number lies in the task’s range.
Figure 2: An example task. The answer is one cell of one table. The agent must find the volume, pick the Subtotal row and settle a faint digit with the page’s own totals.
Agent
Harness
pass@1 [95% CI]
pass@3
pass 3
$/att.
$ total
Min.
Calls
Claude Opus 5.5
Claude Code
64.8 [55.7, 73.3]
81.4
45.7
1.56
328
5.3
24
Gemini 4 Argon
Gemini CLI
60.0 [50.0, 70.0]
71.4
45.7
2.45
514
16.8
46
GPT-6.1 Sol
Codex
51.4 [41.0, 61.4]
64.3
38.6
0.62
130
11.2
33
Gemini 3.8 Flash
Gemini CLI
51.0 [40.5, 61.4]
65.7
37.1
1.74
366
10.3
86
Claude Sonnet 5.5
Claude Code
41.9 [31.9, 51.9]
58.6
27.1
1.19
251
3.0
25
DeepSeek V4.1 Flash
Pi
37.1 [28.1, 46.2]
57.1
17.1
0.25
53
8.8
67
Table 2: Main results on 70 tasks, three attempts per task: pass@1 with its 95% task-bootstrap interval, pass@3 and pass 3 (Section 3.5 ); agent-side cost per attempt and in total at list price, with cache reads billed at the cached rate; and median minutes and tool calls per attempt. Best value per column in bold.
Figure 3: Very blurry scans separate the agents. pass@1 (%) by task tag over all three attempts; brackets give the number of tasks, and Sources the documents a task draws on.
Figure 4: Tokens per attempt. Mean tokens by kind for each agent, labelled with the total and the agent-side cost per attempt at list price.
Per attempt
Reducto (%)
Agent
Shell
Reducto
Dropbox
Images
Upload
First read
Errors
OCR
Claude Opus 5.5
4.9
13.0
5.7
2.8
90.1
96.1
12.3
0
Gemini 4 Argon
7.5
36.8
10.9
2.2
87.4
76.2
15.1
0
GPT-6.1 Sol
6.6
25.0
11.5
0.0
87.3
97.1
7.5
2
Gemini 3.8 Flash
42.2
19.0
16.1
16.8
98.8
74.8
6.7
2
Claude Sonnet 5.5
2.5
12.2
10.0
1.1
38.3
32.0
26.6
0
Table 3: Tool use from the agents’ traces, which undercount DeepSeek’s, Kimi’s and GLM’s calls and omit Claude Code’s sub-agents. Per attempt : mean calls and page images viewed. Reducto : share of reads that name an uploaded volume by its reducto:// identifier ( Upload ), of attempts whose first parse_document call succeeds ( First read ), and of calls that return an error. OCR : attempts in which OCR software run in the shell returned text.
Figure 5: Misread digits are the most common failure for seven of the ten agents. Bars split each agent’s failed attempts ( n ; 1,269 in all) by primary cause; segments of 10% or more are labelled.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
What the question asks
Tasks
Very blurry
Multi- document
Typical trap
Pass@1
Best agent
Read one printed figure
7
43%
0%
Neighbouring row or total (3)
37.1%
Opus (76.2%)
Add up lines of one table
10
40%
0%
Misprinted or misread figure (10)
36.3%
Argon (86.7%)
Difference of two items in one year
12
42%
8%
Sign or direction (6)
25.6%
Argon (38.9%)
Change between two periods
12
33%
75%
Sign or direction (8)
40.3%
Opus (77.8%)
Share of a total
13
23%
8%
Wrong base (9)
44.1%
Opus (69.2%)
Percentage change or excess
5
20%
60%
Wrong base (2)
63.3%
Sonnet (86.7%)
Appendix
Table 4: Tasks by what the question asks the agent to do. Very blurry and multi-document are shares of the bucket’s tasks. Of the 70 tasks, 60 list a misprinted or misread figure among their known wrong answers. The trap column names the most common other kind of known wrong answer in the bucket, with the number of its tasks that carry it, or the misprinted or misread figure when no other kind reaches a quarter of the bucket. Pass@1 is over all attempts by the ten agents on the bucket’s tasks; the best agent has the highest pass@1 on them.
Agent
Model identifier
Harness
Provider
Effort
Claude Opus 5.5
us.anthropic.claude-opus-5-5
Claude Code 2.1.284
Amazon Bedrock
high
Claude Sonnet 5.5
us.anthropic.claude-sonnet-5-5
Claude Code 2.1.284
Amazon Bedrock
high
GPT-6.1 Sol
openai.gpt-6.1-sol
Codex 0.153.3
Amazon Bedrock
high
GPT-5.6 Terra
openai.gpt-5.6-terra
Codex 0.153.3
Amazon Bedrock
high
Gemini 3.8 Flash
gemini-3.8-flash
Gemini CLI 0.61.0
Gemini API
high
Gemini 4 Argon
gemini-4-argon
Gemini CLI 0.61.0
Gemini API
high
Appendix
Table 5: Agents. Each agent is a model, called by the identifier shown through the provider shown, in a command-line harness pinned to one version; effort is the reasoning effort set through the harness. Codex runs with its own sandbox off inside the container.
Input
Output (k)
Cost ($)
Agent
Hours
Uncached (k)
Cache read (M)
Cache write (M)
All
Reasoning
Mean
Total
Claude Opus 5.5
24.9
3.0
2.31
0.15
16.1
5.1
1.56
327.91
Gemini 4 Argon
64.7
910.6
3.32
0.00
34.4
24.3
2.45
514.01
GPT-6.1 Sol
44.4
4.1
2.69
0.11
6.0
2.0
0.62
129.51
Gemini 3.8 Flash
42.5
867.7
12.57
0.00
46.6
33.2
1.74
366.11
Claude Sonnet 5.5
17.5
0.1
2.81
0.18
17.1
5.4
1.19
250.51
Appendix
Table 6: Cost and tokens per agent. Hours : total agent time; tokens and mean cost are per attempt, with uncached input, cache reads and cache writes making up the input, and reasoning counted within output; cost is agent-side at list price; All gives means over all attempts.