cs.AIMar 7, 2026
SaveFinSheet-Bench: From Simple Lookups to Complex Reasoning, Where LLMs Break on Financial Spreadsheets
Organizations: Qubera AG, Gartenstrasse 33, 8002 Zürich · University of Zurich, Department of Informatics, Binzmühlestrasse 14, 8050 Zürich
Abstract
While Large Language Models (LLMs) can accelerate text-heavy tasks in alternative investment due diligence, a gap remains in their ability to accurately extract and reason over structured tabular data from complex financial spreadsheets. Progress is held back by the lack of real industry fund portfolio datasets for benchmarking, as private equity data rooms are confidential. To address this, we introduce FinSheet-Bench, a benchmark of synthetic financial portfolio data modeled on real private equity fund structures, designed to evaluate LLM performance on text-serialized spreadsheet question answering and numeric reasoning tasks. Our evaluation of ten model configurations from OpenAI, Google, and Anthropic on financial spreadsheets, including complex layouts, fund dividers, and multi-line column names, reveals that no standalone model achieves error rates low enough for unsupervised use in professional finance applications. The best-performing model, Gemini 3.1 Pro, achieves 82.4% accuracy across twenty-four evaluation files of varying complexity and structural layout (approximately 1 error per 6 questions), followed by GPT-5.2 with reasoning at 80.4%, Claude Opus 4.6 with thinking at 80.2%, and Gemini 3 Pro at 80.2%. Performance degrades substantially on larger, more complex spreadsheets: the largest spreadsheet (152 companies, 8 funds) yields an average accuracy of just 48.6% across all models, compared to 86.2% on the easiest evaluation file. These difficulty patterns are consistent across all ten models, indicating that they reflect LLM limitations rather than idiosyncratic model weaknesses. Reliable financial spreadsheet extraction will likely require architectural approaches that separate document understanding from deterministic computation.
Figures & tables
| Model | Provider | Release Date |
| GPT-3.5-Turbo | OpenAI | Jan 2024 |
| GPT-4o | OpenAI | May 2024 |
| GPT-5.2 (no reasoning) | OpenAI | Dec 2025 |
| GPT-5.2 (reasoning) | OpenAI | Dec 2025 |
| Gemini 2.0 Flash | Feb 2025 | |
| Gemini 2.5 Pro | Jun 2025 |
Table 1: Models evaluated in this study, with approximate public release dates. No provider publicly discloses parameter counts; we therefore report only the model name, provider, and release date.
| Q# | Question (abbreviated) | Category | Complexity |
|---|---|---|---|
| 1 | How many funds are there? | Simple Lookup | Low |
| 2 | How many companies are in each fund? | Counting | Medium |
| 3 | Which fund is the latest? | Simple Lookup | Low |
| 4 | List all companies in the newest fund | List Extraction | Medium |
| 5 | List all companies sorted by entry EBITDA | Sorting | High |
| 6 | Highest entry EBITDA company per fund? | Aggregation | High |
Table 2: Question classification by type and complexity. Each of the 16 question templates is mapped to one of the seven categories defined above.
| Answer Type | Tier 1 (Exact) | Tier 2 (Fuzzy) | Tier 3 (LLM) |
|---|---|---|---|
| Numeric/Dict (Q1–2, Q6, Q8, Q10–12, Q14–16) | 15% | 30% | 51% |
| String/Date/Bool (Q3, Q9, Q13) | 60% | 10% | 26% |
| List (Q4, Q5, Q7) | 4% | 19% | 73% |
| Overall (all types) | 25% | 25% | 48% |
Table 3: Verification tier distribution by answer type. Values are approximate percentages of answers resolved at each tier, aggregated across all model and question evaluations ( total). The high proportion of LLM adjudication reflects the difficulty of rule-based parsing of free-text responses, particularly for multi-field (dict) and list answers.
Figure 1: LLM accuracy on financial spreadsheet tasks plotted against model release date. Bubble size indicates average response time per question. At 82.4%, even the best model produces approximately 1 error per 6 questions. Newer models and models with reasoning generally achieve higher accuracy. See Table 4 for detailed results.
| Model | Cov. | Acc. | Eff. Acc. | Time (s) | Total Tokens |
|---|---|---|---|---|---|
| Gemini 3.1 Pro | 100% | 82.4% | 82.4% | 27.60 | 13,455,183 |
| GPT-5.2 (reasoning) | 100% | 80.4% | 80.4% | 16.37 | 8,861,018 |
| Claude Opus 4.6 (thinking) | 100% | 80.2% | 80.2% | 23.14 | 9,759,287 |
| Gemini 3 Pro | 100% | 80.2% | 80.2% | 21.62 | 13,575,897 |
| Gemini 2.5 Pro | 100% | 78.8% | 78.8% | 11.17 | 13,122,140 |
| Claude Opus 4.6 | 100% | 66.7% | 66.7% | 7.11 | 9,145,661 |
Table 4: Overall model performance on FinSheet-Bench. All models except GPT-3.5-Turbo achieve 100% coverage (fraction of questions the model could process). Accuracy (Processed) is the primary comparison metric: accuracy computed on the subset of questions each model could process. Effective Accuracy = Accuracy (Processed) Coverage; this composite metric is reported only for GPT-3.5-Turbo to capture its coverage penalty and should not be compared directly against other models’ accuracy scores. For all models with 100% coverage, Effective Accuracy equals Accuracy (Processed).
| Category | Complexity | Example | Attempts | All Models | Top 3 |
|---|---|---|---|---|---|
| Simple Lookup | Low | How many funds are there? | 1,870 | 89.1% | 93.6% |
| List Extraction | Medium | List all companies in the newest fund | 200 | 85.5% | 93.3% |
| Filtering | Medium | Which funds have unrealized investments? | 230 | 70.9% | 84.1% |
| Aggregation | High | What is the average entry EV per fund? | 1,780 | 53.7% | 76.2% |
| Counting | Medium | How many unrealized investments per fund? | 460 | 41.7% | 66.7% |
| Sorting | High | List all companies sorted by entry EBITDA | 240 | 37.5% | 79.2% |
Table 5: Pooled accuracy by question category. “All Models” is the pooled accuracy across all ten model configurations; “Top 3” is the pooled accuracy of the three best-performing models (Gemini 3.1 Pro, GPT-5.2 reasoning, Claude Opus 4.6 thinking).
Figure 2: LLM accuracy by question category (stacked bars). The dark base shows pooled accuracy across all ten models; the middle segment adds the gain when restricting to the top 3 models; the lightest segment adds the further gain of the single best-performing model per category (labeled above each bar). Simple lookups achieve high accuracy, but performance drops for tasks requiring calculation or multi-step reasoning. Sorting (37.5% all models, 83% best) is particularly challenging given that list extraction achieves 85.5%; models can identify what is in a list but struggle with ordering by magnitude . See text for a detailed discussion of tokenization artifacts vs. reasoning failures.
Figure 3: Accuracy vs. task complexity level (stacked bars). The dark base shows pooled accuracy across all ten models; the middle segment adds the gain when restricting to the top 3 models; the lightest segment adds the further gain of the single best-performing model per complexity level (labeled above each bar). Low-complexity tasks achieve 89–95% accuracy, but performance drops sharply at High and Very High complexity, where tasks require multi-step calculations, aggregations, or sorting. Even the best individual model drops to 47.8% on Very High complexity.
Figure 4: Model performance comparison by question type across all ten model configurations (OpenAI, Google, Anthropic).
Figure 5: Heatmap of accuracy by model and question type across all ten model configurations (4 OpenAI, 4 Google, 2 Anthropic). Green indicates high accuracy; red indicates failure. Results span twenty-four evaluation files (approximately 500 questions per model; some files contribute fewer questions due to missing columns; see Section 4.2 for per-file question counts).
Figure 6: Impact of extended reasoning/thinking on model performance across three metrics. Left: Accuracy comparison, where GPT-5.2 gains +22.8pp and Claude Opus 4.6 gains +13.6pp when reasoning/thinking is enabled. Center: Response time increases with reasoning enabled. Right: Output token usage per question increases substantially with reasoning (GPT-5.2: 11 , Claude: 5 ).
| Provider | Model | Coverage | Effective Acc. | vs Previous |
|---|---|---|---|---|
| OpenAI | GPT-3.5-Turbo ∗ | 73.7% | 24.8% | — |
| OpenAI | GPT-4o | 100% | 54.1% | +29.3pp |
| OpenAI | GPT-5.2 (no reasoning) | 100% | 57.7% | +3.6pp |
| OpenAI | GPT-5.2 (reasoning) | 100% | 80.4% | +22.8pp |
| Gemini 2.0 Flash | 100% | 50.1% | — | |
| Gemini 2.5 Pro | 100% | 78.8% | +28.7pp |
Table 6: Accuracy progression across model generations. “vs Previous” compares against the preceding row within the same provider, except for GPT-5.2 (reasoning), which compares against its non-reasoning variant. See Table 4 for GPT-3.5-Turbo’s coverage breakdown.
Figure 7: Average input and output token usage per question by model, ordered by accuracy. Input tokens (top) and output tokens (bottom) are shown on separate scales so that the much smaller output token counts remain visible. Reasoning-enabled models consume significantly more output tokens. Claude Opus 4.6 with thinking uses 5 more output tokens than default mode.
| Model | Input | Output | Total | Accuracy |
| Gemini 2.0 Flash | 24,916 | 101 | 25,017 | 50.1% |
| Gemini 2.5 Pro | 24,917 | 1,274 | 26,191 | 78.8% |
| Gemini 3 Pro | 24,917 | 2,180 | 27,097 | 80.2% |
| Gemini 3.1 Pro | 24,722 | 2,133 | 26,855 | 82.4% |
| GPT-3.5-Turbo ∗ | 8,737 | 158 | 8,895 | 24.8% |
| GPT-5.2 (no reasoning) | 16,567 | 106 | 16,673 | 57.7% |
Table 7: Average token consumption per question by model
| File | Companies / Funds | Accuracy | Questions |
|---|---|---|---|
| synthetic1_A.csv | 45 / 4 | 74.1% | 22 |
| synthetic1_B.txt | 30 / 4 | 76.8% | 22 |
| synthetic1_C.txt | 30 / 4 | 83.6% | 22 |
| synthetic2_A.csv | 89 / 5 | 69.5% | 21 |
| synthetic2_B.txt | 59 / 5 | 55.2% | 21 |
| synthetic2_C.txt | 59 / 5 | 86.2% | 21 |
Table 8: Accuracy by data file (all models, averaged). Version B and C files are structural variants with approximately one-third fewer rows and modified column layouts. For files where GPT-3.5-Turbo’s evaluation returned no verification result (6 of 24 files), that model is excluded from the average.
Figure 8: Heatmap of accuracy by model and data file. Each cell shows the percentage of questions answered correctly by a given model on a given file. Files are ordered by average difficulty (hardest at top). The number of questions per file ranges from 10 (synthetic3_B) to 22 (most files), with three parameterized templates sampled at sample_size=3 . synthetic4_A is consistently the hardest file across all models; synthetic2_C is among the easiest. GPT-3.5-Turbo shows no result (context overflow) on 6 of 24 files exceeding its context limit, with failures distributed across A, B, and C variants.
Figure 9: Average accuracy of the top three models (Gemini 3.1 Pro, GPT-5.2 reasoning, Claude Opus 4.6 thinking) vs. file length in characters. Each point represents one evaluation file; the solid black line shows the average trend. The negative slope ( pp per 1K characters, ) indicates a moderate relationship, though file-specific structural complexity remains an important confounding factor.
More in this topic
Table QA
Table question answering studies how AI systems retrieve, interpret, and reason over structured data to answer natural-language questions.
See all 66 papers