fxBench: Evaluating and Understanding Formula Suggestions in Spreadsheets

Sanket Mhatre; Sumit Gulwani; Vu Le; Yasharth Bajpai; Gust Verbruggen

cs.LGOct 17, 2026

fxBench: Evaluating and Understanding Formula Suggestions in Spreadsheets

Authors: Sanket Mhatre, Sumit Gulwani, Vu Le, Yasharth Bajpai, Gust Verbruggen

Organizations: Proximal · Microsoft, USA · Microsoft, India · Microsoft, Belgium

Abstract

Spreadsheets are among the most widely used tools for computations, and formulas are their core building blocks. Besides agentic spreadsheet tools, proactive formula suggestions enable significant productivity gains and are now available in popular spreadsheet software (Microsoft Excel and Google Sheets). Such suggestions pose interesting challenges: the right context must be collected and represented for a machine learning model, which must then be able to write a correct formula, in a way that is fast and cheap enough to trigger on every = (or subsequent keystrokes for auto-completion) without taking users out of their editing flow. Surprisingly, there is no public benchmark for formula suggestions on real spreadsheets. In this paper, we therefore (1) introduce fxBench as a manually curated benchmark of 503 formula suggestion tasks based on Sheetpedia, (2) break down the formula suggestion problem into four capabilities (gather context, understand context, understand intent, write formula) that a formula suggestion system requires, and (3) evaluate existing and new approaches to understand bottlenecks in these capabilities that pose interesting research directions. Specifically, we annotate each benchmark with the cells that describe intent to determine the theoretical performance limit of a given context selection strategy, we evaluate performance of small and large models on different representations of context, including a dense format with high theoretical coverage at low token counts, and we fine-tune smaller language models to show that we can learn to understand such dense formats. Additionally, we perform quantitative and qualitative analysis on failure modes to understand bottlenecks in the formula writing capability; the best configuration only achieved 58.8% correct suggestions.

Figures & tables

Figure 1: The formula suggestion workflow. Given a spreadsheet and a target cell (left), a pre-processing step extracts and serializes the relevant context (middle), which is passed to an LLM that predicts the formula (right).

Figure 2: The fx Bench curation pipeline.

Figure 4: Minisheet representation. Blob detection finds contiguous non-empty regions; edge extraction keeps boundary cells; semantic grouping emits a compact view of values, numeric clusters, formulas and tables sections.

Context	Intent printed	Intent mentioned	Ref. printed	Ref. mentioned	Upper	Tok (Avg)	Tok (Median)
Pipe-delimited	100.0%	100.0%	100.0%	100.0%	100.0%	19,984	3,128
Viewport ( $20\times 10$ )	82.1%	82.1%	70.8%	70.8%	69.0%	2,603	1,913
Viewport ( $30\times 20$ )	94.6%	94.6%	85.7%	85.7%	85.1%	3,806	2,574
Viewport ( $40\times 30$ )	98.2%	98.2%	90.1%	90.1%	89.9%	4,533	2,881
SpreadsheetCoder	74.8%	74.8%	61.8%	61.8%	59.0%	1,806	1,364
SheetCompressor	90.5%	100.0%	43.5%	100.0%	94.6%	5,156	1,372

Table 2: Required-cell exposure coverage and token cost by context strategy on fx Bench.

Class	Representation	GPT 5.2			Sonnet 4.6			Qwen3-8B
		acc	acc †	ovl	acc	acc †	ovl	acc	acc †	ovl
F	HTML	58.3	58.3	67.6	60.0	60.0	71.0	20.7	20.7	36.1
	CSV	49.7	49.7	63.5	53.3	53.3	64.9	9.3	9.3	21.6
	JSON	56.9	56.9	67.1	60.4	60.4	69.7	14.5	14.5	30.6
	Markdown	53.3	53.3	66.1	54.7	54.7	67.5	11.3	11.3	25.2
	Pipe-delimited	58.8	58.8	66.3	59.8	59.8	69.9	12.1	12.1	27.4

Table 3: Content strategy baselines on fx Bench.

Comparison	Model	Training data	Records	pass@1	pass@3	best@8
Base context	Qwen3-4B	–	–	10.8%	12.6%	13.9%
Base context	Qwen3-8B	–	–	9.7%	12.2%	14.5%
Base context	Qwen3-14B	–	–	10.0%	13.4%	15.5%
Model size	Qwen3-4B	Curated	25,614	16.5%	25.7%	32.8%
Model size / anchor	Qwen3-8B	Curated	25,614	21.2%	30.4%	38.0%
Model size	Qwen3-14B	Curated	25,614	19.5%	30.3%	40.2%

Table 5: Combined fine-tuning results on fx Bench, comparing model size on Curated training data and dataset choice at fixed Qwen3-8B scale using eight sampled completions per problem.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Target cell location categories on a single worksheet. Dashed blue marks a structured Excel Table (Ctrl+T); dashed purple marks a blob. Each formula is color-coded by its location class: =SUM(E3:E6) is inside_ctrl_t , =SUM(I3:I5) is inside_blob , =E7*1.1 is adjacent_ctrl_t , and =A9*B10+3 is unstructured . — Figure 6: Target cell location categories on a single worksheet. Dashed blue marks a structured Excel Table (Ctrl+T); dashed purple marks a blob. Each formula is color-coded by its location class: =SUM(E3:E6) is inside_ctrl_t , =SUM(I3:I5) is inside_blob , =E71.1 is adjacent_ctrl_t , and =A9B10+3 is unstructured .

Figure 7: Temporal snapshot pipeline

Figure 11: Sheet Size Distribution

Statistical (36 functions, 139 records)
AVERAGE (17)	COUNTIFS (16)	COUNTIF (15)	MIN (13)
MAX (12)	LARGE (10)	COUNTA (7)	COUNT (4)
INTERCEPT (4)	SMALL (4)	AVERAGEIF (3)	AVERAGEIFS (3)
FORECAST (3)	MINIFS (3)	LINEST (2)	STDEV.P (2)
MAXA (2)	AVERAGEA (1)	CORREL (1)	COUNTBLANK (1)
FORECAST.LINEAR (1)	MAXIFS (1)	MINA (1)	MODE.MULT (1)

Appendix

Table 6: Exhaustive per-category function inventory of fx Bench. Every distinct function is listed under its Excel category with its record count in parentheses.

Failure label	Description
incorrect_formula_type	The prediction uses a different root function or a substantially different nested-function strategy from the target.
right_type_incorrect_args	The root and nested function structure are broadly correct, but non-range arguments such as criteria, constants, operators, argument order, or wrapper arguments are wrong.
right_type_incorrect_range	The root and nested function structure are broadly correct, but one or more referenced cells or ranges are wrong.
partial_correct	The prediction shares the outer formula strategy and some nested structure, but an important nested subexpression is missing, added, or replaced.
abstain_wrong	The model makes an abstention mistake: it abstains when a formula is expected, or produces a formula when abstention is expected.
syntax_error	The predicted formula cannot be parsed or accepted as a valid Excel formula.

Appendix

Table 7: Deterministic failure labels used in the oracle failure analysis. Labels are assigned only to actual failures (FP/FN/Error); correct predictions and correct abstentions are excluded.

Figure 13: Sheet view for Case 1 ( Max Moving Average!B2 ). The formula bar shows the trailing-window construction; C2 holds the window length (15).

Figure 14: Sheet view for Case 2 ( Sheet2!B6 ). The minimum price 50.00 ties between Sup2 (quantity 50) and Sup4 (quantity 70); the expected formula breaks the tie by selecting the larger quantity.

Figure 16: Sheet view for Case 4 ( restruct!Q4 ). Interest is quoted as an annual rate (11.0000%) in Q3 but the output column is Months , so NPER needs the monthly rate Q3/12 .

Figure 17: Sheet view for Case 5 ( Data_Recruitment Act!AC6 ). The expected formula bar shows a trailing window anchored to the last populated row of column S , with length $AD$4 =3. — Figure 17: Sheet view for Case 5 ( Data_Recruitment Act!AC6 ). The expected formula bar shows a trailing window anchored to the last populated row of column S , with length $AD$ 4 =3.

Panel of Figure 18 — Figure 18: Left: share of records (%) covered by the to three and top ten primary function types. Right: distribution of function nesting depth across dataset variants.

Context Builder	Model	TP	FP	TN	FN	Acc%	Overlap%	Syntax%	Sketch%
Full-sheet strategies
HTML	GPT-5.2	281	204	12	6	58.3	67.6	98.0	27.4
	Sonnet-4-6	288	199	14	2	60.0	71.0	91.3	27.4
	Qwen3-8B	90	352	14	47	20.7	36.1	81.9	3.2
CSV	GPT-5.2	236	243	14	10	49.7	63.5	96.8	25.6
	Sonnet-4-6	251	230	17	5	53.3	64.9	91.8	24.9

Appendix

Table 8: Content strategy baselines on fx Bench: outcome counts and auxiliary metrics across GPT-5.2, Sonnet-4-6, and Qwen3-8B.

Figure 19: Token Usage

Spreadsheet Automation

Spreadsheet automation uses AI to understand, create, edit, and validate workbooks while preserving their formulas, structure, and intended use.

See all 20 papers

SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows
Jun 29, 2026Jian Zhu, Yuzheng Zhang, Zeyao Ma +11Benchmark Design Spreadsheet Automation
School of Information, Renmin University of China · Aptura AI · Shortcut AI +1
MBABench-2.0: Evaluating AI Agents on Constructing Financial Spreadsheets Beyond Simple Verifiability
Oct 22, 2026Julian Poeltl, Thomson Yen, Patrick Shea +5Spreadsheet Automation AI Agent Benchmarks
ESB Reutlingen · Columbia Business School · TensorBlock
Back to the Future: A workbook time machine for spread sheet creation benchmarks
Aug 8, 2026Mansi Uniyal, Agamdeep Singh, Ananya Singha +5LLM Agent Evaluation Synthetic Benchmark Generation
Microsoft

Jun 29, 2026J/K move · Enter open · S save

SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows

Jian Zhu, Yuzheng Zhang, Zeyao Ma, Bohan Zhang+10

School of Information, Renmin University of China · Aptura AI · Shortcut AI+1

Benchmark Design Spreadsheet Automation Agent Benchmarks

Open paper AskPDF arXiv

Spreadsheets are widely used for business analysis, financial modeling, reporting, and decision-making. However, most existing spreadsheet benchmarks evaluate isolated operations such as single-formula generation or local cell edits, and therefore fail to capture end-to-end workflows in realistic business settings. We introduce SpreadsheetBench 2, a workflow-level benchmark for spreadsheet agents that covers three task categories: generation, debugging, and visualization. The benchmark is constructed from authentic business data, including financial reports and corporate filings, and is annotated and validated by domain experts. The benchmark contains 321 tasks; each instance averages 11.8 worksheets and requires 593.5 cell modifications, reflecting large multi-sheet workbooks with cross-sheet dependencies. We evaluate eight frontier large language models under a unified multi-turn agent scaffold, and additionally include several LLM-based spreadsheet products as complementary baselines. Results show that current systems remain far from reliable on real-world workflows: the best model achieves 34.89% overall task accuracy, and debugging accuracy is as low as 12.00%. Trajectory analysis and a failure taxonomy further indicate that insufficient spreadsheet inspection and incorrect target-cell selection are the dominant bottlenecks. Together, these findings position SpreadsheetBench 2 as a challenging testbed for advancing reliable spreadsheet automation. Project page: https://spreadsheetbench.github.io/

Abstract

Spreadsheet Automation

Explore similar work

SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows

MBABench-2.0: Evaluating AI Agents on Constructing Financial Spreadsheets Beyond Simple Verifiability

Back to the Future: A workbook time machine for spread sheet creation benchmarks