Spreadsheets are among the most widely used tools for computations, and formulas are their core building blocks. Besides agentic spreadsheet tools, proactive formula suggestions enable significant productivity gains and are now available in popular spreadsheet software (Microsoft Excel and Google Sheets). Such suggestions pose interesting challenges: the right context must be collected and represented for a machine learning model, which must then be able to write a correct formula, in a way that is fast and cheap enough to trigger on every = (or subsequent keystrokes for auto-completion) without taking users out of their editing flow. Surprisingly, there is no public benchmark for formula suggestions on real spreadsheets. In this paper, we therefore (1) introduce fxBench as a manually curated benchmark of 503 formula suggestion tasks based on Sheetpedia, (2) break down the formula suggestion problem into four capabilities (gather context, understand context, understand intent, write formula) that a formula suggestion system requires, and (3) evaluate existing and new approaches to understand bottlenecks in these capabilities that pose interesting research directions. Specifically, we annotate each benchmark with the cells that describe intent to determine the theoretical performance limit of a given context selection strategy, we evaluate performance of small and large models on different representations of context, including a dense format with high theoretical coverage at low token counts, and we fine-tune smaller language models to show that we can learn to understand such dense formats. Additionally, we perform quantitative and qualitative analysis on failure modes to understand bottlenecks in the formula writing capability; the best configuration only achieved 58.8% correct suggestions.
Figures & tables
Figure 1: The formula suggestion workflow. Given a spreadsheet and a target cell (left), a pre-processing step extracts and serializes the relevant context (middle), which is passed to an LLM that predicts the formula (right).
Figure 2: The fx Bench curation pipeline.
Figure 3
Figure 4: Minisheet representation. Blob detection finds contiguous non-empty regions; edge extraction keeps boundary cells; semantic grouping emits a compact view of values, numeric clusters, formulas and tables sections.
Context
Intent printed
Intent mentioned
Ref. printed
Ref. mentioned
Upper
Tok (Avg)
Tok (Median)
Pipe-delimited
100.0%
100.0%
100.0%
100.0%
100.0%
19,984
3,128
Viewport ( 20×10 )
82.1%
82.1%
70.8%
70.8%
69.0%
2,603
1,913
Viewport ( 30×20 )
94.6%
94.6%
85.7%
85.7%
85.1%
3,806
2,574
Viewport ( 40×30 )
98.2%
98.2%
90.1%
90.1%
89.9%
4,533
2,881
SpreadsheetCoder
74.8%
74.8%
61.8%
61.8%
59.0%
1,806
1,364
SheetCompressor
90.5%
100.0%
43.5%
100.0%
94.6%
5,156
1,372
Table 2: Required-cell exposure coverage and token cost by context strategy on fx Bench.
Class
Representation
GPT 5.2
Sonnet 4.6
Qwen3-8B
acc
acc †
ovl
acc
acc †
ovl
acc
acc †
ovl
F
HTML
58.3
58.3
67.6
60.0
60.0
71.0
20.7
20.7
36.1
CSV
49.7
49.7
63.5
53.3
53.3
64.9
9.3
9.3
21.6
JSON
56.9
56.9
67.1
60.4
60.4
69.7
14.5
14.5
30.6
Markdown
53.3
53.3
66.1
54.7
54.7
67.5
11.3
11.3
25.2
Pipe-delimited
58.8
58.8
66.3
59.8
59.8
69.9
12.1
12.1
27.4
Table 3: Content strategy baselines on fx Bench.
Figure 7
Comparison
Model
Training data
Records
pass@1
pass@3
best@8
Base context
Qwen3-4B
–
–
10.8%
12.6%
13.9%
Base context
Qwen3-8B
–
–
9.7%
12.2%
14.5%
Base context
Qwen3-14B
–
–
10.0%
13.4%
15.5%
Model size
Qwen3-4B
Curated
25,614
16.5%
25.7%
32.8%
Model size / anchor
Qwen3-8B
Curated
25,614
21.2%
30.4%
38.0%
Model size
Qwen3-14B
Curated
25,614
19.5%
30.3%
40.2%
Table 5: Combined fine-tuning results on fx Bench, comparing model size on Curated training data and dataset choice at fixed Qwen3-8B scale using eight sampled completions per problem.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Target cell location categories on a single worksheet. Dashed blue marks a structured Excel Table (Ctrl+T); dashed purple marks a blob. Each formula is color-coded by its location class: =SUM(E3:E6) is inside_ctrl_t , =SUM(I3:I5) is inside_blob , =E71.1 is adjacent_ctrl_t , and =A9B10+3 is unstructured .
Figure 7: Temporal snapshot pipeline
Figure 9: Temporal snapshot: before.
Figure 10: Temporal snapshot: after.
Figure 11: Sheet Size Distribution
Statistical (36 functions, 139 records)
AVERAGE (17)
COUNTIFS (16)
COUNTIF (15)
MIN (13)
MAX (12)
LARGE (10)
COUNTA (7)
COUNT (4)
INTERCEPT (4)
SMALL (4)
AVERAGEIF (3)
AVERAGEIFS (3)
FORECAST (3)
MINIFS (3)
LINEST (2)
STDEV.P (2)
MAXA (2)
AVERAGEA (1)
CORREL (1)
COUNTBLANK (1)
FORECAST.LINEAR (1)
MAXIFS (1)
MINA (1)
MODE.MULT (1)
Appendix
Table 6: Exhaustive per-category function inventory of fx Bench. Every distinct function is listed under its Excel category with its record count in parentheses.
Failure label
Description
incorrect_formula_type
The prediction uses a different root function or a substantially different nested-function strategy from the target.
right_type_incorrect_args
The root and nested function structure are broadly correct, but non-range arguments such as criteria, constants, operators, argument order, or wrapper arguments are wrong.
right_type_incorrect_range
The root and nested function structure are broadly correct, but one or more referenced cells or ranges are wrong.
partial_correct
The prediction shares the outer formula strategy and some nested structure, but an important nested subexpression is missing, added, or replaced.
abstain_wrong
The model makes an abstention mistake: it abstains when a formula is expected, or produces a formula when abstention is expected.
syntax_error
The predicted formula cannot be parsed or accepted as a valid Excel formula.
Appendix
Table 7: Deterministic failure labels used in the oracle failure analysis. Labels are assigned only to actual failures (FP/FN/Error); correct predictions and correct abstentions are excluded.
Figure 13: Sheet view for Case 1 ( Max Moving Average!B2 ). The formula bar shows the trailing-window construction; C2 holds the window length (15).
Figure 14: Sheet view for Case 2 ( Sheet2!B6 ). The minimum price 50.00 ties between Sup2 (quantity 50) and Sup4 (quantity 70); the expected formula breaks the tie by selecting the larger quantity.
Figure 16: Sheet view for Case 4 ( restruct!Q4 ). Interest is quoted as an annual rate (11.0000%) in Q3 but the output column is Months , so NPER needs the monthly rate Q3/12 .
Figure 17: Sheet view for Case 5 ( Data_Recruitment Act!AC6 ). The expected formula bar shows a trailing window anchored to the last populated row of column S , with length AD4 =3.
Figure 18: Left: share of records (%) covered by the to three and top ten primary function types. Right: distribution of function nesting depth across dataset variants.
Context Builder
Model
TP
FP
TN
FN
Acc%
Overlap%
Syntax%
Sketch%
Full-sheet strategies
HTML
GPT-5.2
281
204
12
6
58.3
67.6
98.0
27.4
Sonnet-4-6
288
199
14
2
60.0
71.0
91.3
27.4
Qwen3-8B
90
352
14
47
20.7
36.1
81.9
3.2
CSV
GPT-5.2
236
243
14
10
49.7
63.5
96.8
25.6
Sonnet-4-6
251
230
17
5
53.3
64.9
91.8
24.9
Appendix
Table 8: Content strategy baselines on fx Bench: outcome counts and auxiliary metrics across GPT-5.2, Sonnet-4-6, and Qwen3-8B.
Spreadsheets are widely used for business analysis, financial modeling, reporting, and decision-making. However, most existing spreadsheet benchmarks evaluate isolated operations such as single-formula generation or local cell edits, and therefore fail to capture end-to-end workflows in realistic business settings. We introduce SpreadsheetBench 2, a workflow-level benchmark for spreadsheet agents that covers three task categories: generation, debugging, and visualization. The benchmark is constructed from authentic business data, including financial reports and corporate filings, and is annotated and validated by domain experts. The benchmark contains 321 tasks; each instance averages 11.8 worksheets and requires 593.5 cell modifications, reflecting large multi-sheet workbooks with cross-sheet dependencies. We evaluate eight frontier large language models under a unified multi-turn agent scaffold, and additionally include several LLM-based spreadsheet products as complementary baselines. Results show that current systems remain far from reliable on real-world workflows: the best model achieves 34.89% overall task accuracy, and debugging accuracy is as low as 12.00%. Trajectory analysis and a failure taxonomy further indicate that insufficient spreadsheet inspection and incorrect target-cell selection are the dominant bottlenecks. Together, these findings position SpreadsheetBench 2 as a challenging testbed for advancing reliable spreadsheet automation. Project page: https://spreadsheetbench.github.io/
Jian Zhu, Yuzheng Zhang, Zeyao Ma +11
School of Information, Renmin University of China · Aptura AI · Shortcut AI +1
Spreadsheets are the lingua franca of modern finance, driving valuation, forecast, audit, and many other core workflows of the industry. As LLM agents advance to automate knowledge work, spreadsheet construction is a natural and pertinent testbed. However, most existing spreadsheet benchmarks evaluate agents on fill-in-the-blank style tasks, where an agent performs minor edits in otherwise complete spreadsheets. This protocol fails to capture the most central and time-consuming part of the spreadsheet workflow: the modeling decisions, format design, and sheet layout regular knowledge workers need to toil through to build a workbook from scratch. We introduce MBABench-2.0, a benchmark of 101 original spreadsheet construction tasks, the hardest of which is estimated to take 33 hours for a modeler with 3+ years of experience. Unlike coding or math, where a unit test or a final answer can verify an agent's output, the quality of a financial spreadsheet that concerns readability, maintainability, or auditability, is not so simply verifiable. To holistically evaluate spreadsheet construction, we introduce 129 criteria spanning 12 categories that operationalize quality beyond the accuracy of the spreadsheet. Evaluating over 20 agents across 8 models and 4 harness types, we find that frontier agents rarely construct spreadsheets that are both accurate and meet our quality criteria: the top agent achieves a pass rate of only 16% on the benchmark. The benchmark demonstrates a substantial gap in spreadsheet building capability of current agents.
Julian Poeltl, Thomson Yen, Patrick Shea +5
ESB Reutlingen · Columbia Business School · TensorBlock
We introduce the workbook time machine, a pipeline that automatically creates benchmarks evaluating the ability of language models to create derived objects in spreadsheets (formulas, charts, pivot tables, and conditional formatting). Applied to public workbook corpora, it produces wtmcorpus--a collection of (input workbook, output workbook, query) triples spanning four artifact types and varying complexity. From this corpus we curate wtmbench, a 150-task evaluation benchmark with queries at three levels of specificity. We evaluate existing spreadsheet manipulation agents and baselines on wtmbench across artifact types, step complexity, and instruction granularity. Our evaluations show that query specificity, agent orchestration, and interface API used to control spreadsheets play a big role in LLM performance on Excel tasks.