cs.AIMar 9, 2026
SaveOfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning
Organizations: Databricks
Abstract
We introduce OfficeQA Pro, a benchmark for evaluating AI agents on grounded, multi-document reasoning over a large and heterogeneous document corpus. The corpus consists of U.S. Treasury Bulletins spanning nearly 100 years, comprising 89,000 pages and over 26 million numerical values. OfficeQA Pro consists of 133 questions that require precise document parsing, retrieval, and analytical reasoning across both unstructured text and tabular data. Frontier LLMs including Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview achieve less than 5% accuracy on OfficeQA Pro when relying on parametric knowledge, and less than 12% with additional access to the web. When provided directly with the document corpus, frontier agents still struggle on over half of questions, scoring 34.1% on average. We find that providing agents with a structured document representation produced by Databricks' ai_parse_document yields a 16.1% average relative performance gain across agents. We conduct additional ablations to study the effects of model selection, table representation, retrieval strategy, and test-time scaling on performance. Despite these improvements, significant headroom remains before agents can be considered reliable at enterprise-grade grounded reasoning.
Figures & tables
Figure 1: Agent performance on OfficeQA Pro. Each agent framework—Google Agent (Gemini CLI), OpenAI Agent (Codex CLI), and Anthropic Agent (Claude Agent SDK)—is evaluated with the provider’s latest flagship model, along with the flagship model available at the time OfficeQA was created. Agent performance with latest frontier models is 34.1% on average and none of the agents surpass 50% accuracy.
Figure 2: Representative pages from the OfficeQA Pro corpus, illustrating how Treasury Bulletins vary in formatting and layout across decades. Left: March 1941 customs duties and taxable imports by tariff schedule with CY1939–1940 aggregates and monthly values (thousands of dollars). Right: March 1980 Treasury Survey of Ownership of interest-bearing marketable public debt securities by type, maturity, and issue across investor classes (millions of dollars).
Figure 3: Sample questions from OfficeQA Pro. Question UID0013 (left) requires information from 1 bulletin, and linear regression analysis. Question UID0005 (right) requires information from 2 separate bulletins, multi-step math, and web-search to retrieve the correct CPI value.
Figure 4: Overview of the benchmark creation process. In Phase 1, the question is created and initially verified by a new annotator. In Phase 2, two passes of end-to-end quality assurance are completed using AI agents to produce alternative answers, which are used to inform human verification of both the question and the ground truth answer.
Figure 5: Performance of frontier LLMs on OfficeQA Pro across three configurations: Prompt Only, Web Search Enabled, and Oracle Page(s) Provided + Web Search. Results are reported as correctness (%) across allowable absolute relative error thresholds of 5.0%, 1.0%, 0.1%, and 0.0% for the three models. The Oracle configuration denotes the original documents as “PDF” while “Databricks Parsed” refers to parsing via Databricks’ ai_parse_document . Leveraging the Databricks’ parsed pages provides a relative gain of +50.2% on average at 0.0% allowable absolute relative error.
| Agent | Corpus | Format | Correctness (%) | Latency (min) | Tool Calls | Cost (\times$ the cost per question. |
| Claude Opus 4.6 | Full | 48.12 | 31.2 | 82.4 | 4.55 | |
| DBX Parsed | 54.14 | 5.3 | 68.5 | 5.93 | ||
| Oracle | 60.90 | 7.3 | 26.5 | 1.19 | ||
| DBX Parsed | 66.92 | 3.5 | 24.5 | 2.76 | ||
| GPT-5.4 | Full | 36.09 | 13.1 | 57.0 | 1.79 | |
| DBX Parsed | 56.39 | 3.6 | 34.5 | 1.26 |
Table 1: Agent performance across evaluated configurations including the Full Corpus vs. Oracle Pages (Full vs. Oracle); and PDFs vs. documents parsed with Databricks’ ai_parse_document (PDF vs. DBX Parsed). We report correctness (%) at 0.0% allowable absolute relative error, average latency, average tool calls, and cost per sample.
Figure 6: Agent correctness on OfficeQA Pro (left: full corpus; right: oracle pages) across allowable absolute relative error thresholds (5.0%, 1.0%, 0.1%, 0.0%). Solid lines (“Databricks Parse”) use documents parsed via Databricks’ ai_parse_document ; dashed lines (“PDF”) use the raw PDF corpus.
Figure 7: Custom agent correctness (%) at 0.0% error threshold vs. average agent cost per sample on the full OfficeQA Pro corpus using file search, across 3 models and 3 parsers. Databricks’ ai_parse_document is highlighted in bold.
Figure 8: Top: "Liabilities to Foreigners" values from the September 2013 Treasury Bulletin (March, May, June 2013). Bottom: revised values for the three months in the December 2013 issue.
Figure 9: Top: original February 1950 Treasury Bulletin table on budget expenditures from CY January 1940 - December 1943. Bottom: parsed output with two errors highlighted (May 1942, July 1941).
Figure 10: Models struggle with visual reasoning tasks such as determining the total local maxima from this line plot.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Question difficulty distribution and capability breakdown of OfficeQA-Full and Pro. Left shows the difficulty distribution of questions (113 easy / 133 hard). Right details sample breakdown by the capabilities required to answer the question, including data analysis, retrieval from >3 distinct pages, external knowledge search, and visual understanding.
Figure 12: Sample questions from the OfficeQA benchmark, including the Full and Pro split. Questions typically require multi-step analytical reasoning such as statistical modeling across multiple documents and datapoints.
| Corpus | Config | Format | Correctness (%) | Latency (min) |
| Oracle Pages | Human (N=3) | 51.1 | 19.2 | |
| Agent (N=3) | 71.1 | 5.3 | ||
| Agent (N=3) | Parsed | 72.2 | 2.1 | |
| Full Corpus | Human (N=3) | 34.6 | 31.4 | |
| Agent (N=3) | 50.0 | 15.4 | ||
| Agent (N=3) | Parsed | 56.7 | 3.5 |
Table 2: Human vs. agent performance on a N=30 subset of the OfficeQA-Full benchmark. Human results span 3 annotators; agent results average Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview (N=3).
Figure 13: Performance of three human annotators and one AI agent (Claude Opus 4.6, PDF) on a 30-sample subset of OfficeQA-Full across all error thresholds (5.0% to 0.0%). Left: oracle pages provided; right: full corpus.
| Tool | Action | Example Equivalent Shell Command |
| fs_search | Directory listing | bash ls |
| List files by pattern | bash ls treasury_bulletins/2013.txt | |
| Search directory recursively | bash grep -rn -i -B 2 -A 2 "national saving" treasury_bulletins/ | |
| Search for a pattern ‘in specific files | bash grep -rn -i "Individual income taxes" treasury_bulletins/treasury_bulletin_2013*.txt | |
| fs_read | Read first N lines | bash head -n 20 foo.txt |
| Read a slice of lines | bash sed -n ’101,150p’ foo.txt |
Table 3: Mapping of fs_search and fs_read tool capabilities to equivalent shell commands.
Figure 14: Correctness (%) vs. average cost and latency per sample for the custom agent with Databricks’ ai_parse_document and file search on the full Treasury Bulletin corpus, evaluated at 0.0% error threshold. Pareto-optimal models are shown in red.
| Model | Reasoning | Correctness (%) | Latency (min) | Tool Calls | Cost ($) |
| GPT-5.4 | High | 51.10 | 10.87 | 104.8 | $6.13 |
| GPT-5.1 | High | 45.10 | 20.6 | 60.1 | $4.06 |
| GPT-5 Mini | High | 30.80 | 48.1 | 120.4 | $2.80 |
| Claude Opus 4.6 | Default | 57.10 | 5.3 | 53.3 | $8.61 |
| Claude Opus 4.5 | Default | 46.60 | 2.8 | 27.5 | $5.34 |
| Claude Sonnet 4.6 | Default | 51.90 | 5.4 | 69.2 | $6.45 |
Table 4: Model correctness (%) at 0.0% error threshold, latency (minutes), average tool calls, and cost for the custom agent with Databricks’ ai_parse_document and file search on OfficeQA Pro.
Figure 15: A representative table from the Treasury Bulletin dataset, with nested headers and rows.
Figure 16: Directional comparison of HTML vs. Markdown table representations across correctness (at 0.0% error threshold), latency, and cost, using the custom agent configured with file search over the full corpus of documents parsed with Databricks’ ai_parse_document . Arrows point from HTML to Markdown; filled markers indicate the better format.
| Model | Table Format | Correctness (%) | Latency (min) | Tool Calls | Cost ($) |
| GPT-5.4 | HTML | 51.10 | 10.87 | 104.8 | $6.13 |
| MD | 50.40 | 11.64 | 114.3 | $6.03 | |
| GPT-5.1 | HTML | 45.10 | 20.6 | 60.1 | $4.06 |
| MD | 45.10 | 28.2 | 69.1 | $4.25 | |
| GPT-5 Mini | HTML | 30.80 | 48.1 | 120.4 | $2.80 |
| MD | 29.30 | 47.2 | 122.7 | $2.69 |
Table 5: Full performance comparison across HTML and Markdown (MD) table formats using the custom agent with Databricks’ ai_parse_document and file search on the full OfficeQA Pro corpus, at 0.0% error threshold.
| Model | Search Tool | Correctness (%) | Latency (min) | Tool Calls | Cost ($) |
| GPT-5.4 | VS | 39.10 | 20.9 | 220.4 | $12.46 |
| C-VS | 49.60 | 12.28 | 123.9 | $7.28 | |
| FS + C-VS | 51.90 | 8.93 | 86.4 | $5.31 | |
| Claude Opus 4.6 | VS | 45.90 | 9.04 | 75.0 | $19.60 |
| C-VS | 49.60 | 5.16 | 41.3 | $9.34 | |
| FS + C-VS | 53.40 | 5.16 | 47.4 | $8.10 |
Table 6: Performance comparison at 0.0% error threshold by search tool using the custom agent with access to the full corpus of documents parsed with Databricks’ ai_parse_document . VS: vector search with standard embeddings; C-VS: contextual embeddings; FS + C-VS: combined.
Figure 17: Test-time scaling on OfficeQA Pro using the custom agent with FS + C-VS search over the full corpus parsed with Databricks’ ai_parse_document . Correctness is reported at 0.0% error threshold for N=1 to 4 rollouts. Plurality vote and oracle reward strategies improve over single-run average as rollouts increase.
| Model | Config | Correctness (%) |
| claude-4.6-opus | llm-only | 2.26 |
| llm web-search-only | 3.01 | |
| llm-oracle PDF | 37.59 | |
| llm-oracle-Databricks parse | 50.38 | |
| llm-oracle PDF-web-search | 36.09 | |
| llm-oracle-Databricks parse-web-search | 57.14 |
Table 7: LLM baseline correctness across evaluated configurations.
| Model | Provider | Input ($/MTok) | Output ($/MTok) | Cached Input ($/MTok) |
| GPT-5.4 | OpenAI | 2.50 | 15.00 | 0.250 |
| GPT-5.1 | OpenAI | 1.25 | 10.00 | 0.125 |
| GPT-5 Mini | OpenAI | 0.25 | 2.00 | 0.025 |
| Claude Opus 4.6 | Anthropic | 5.00 | 25.00 | 0.50 |
| Claude Opus 4.5 | Anthropic | 5.00 | 25.00 | 0.50 |
| Claude Sonnet 4.6 | Anthropic | 3.00 | 15.00 | 0.30 |
Table 8: Token pricing assumptions used for cost calculations (USD per 1 million tokens). Sourced from provider pricing pages as of March 2026.