From Benchmarks to Production: A Text-to-SQL System for Complex Financial Data

Arijit Sehanobish; Bruno Gomes Coelho; Guillaume Michel; Sophia Zhi; Valerie Faucon-Morin; Kristen Howell

cs.AIOct 2, 2026

From Benchmarks to Production: A Text-to-SQL System for Complex Financial Data

Authors: Arijit Sehanobish, Bruno Gomes Coelho, Guillaume Michel, Sophia Zhi, Valerie Faucon-Morin, Kristen Howell

Organizations: Kensho Technologies

Abstract

General-purpose Text-to-SQL systems achieve strong performance on academic benchmarks like Spider and BIRD, where schemas are relatively shallow and column values are often human readable. In production financial databases, where concepts are stored as opaque integer keys rather than human-readable strings, these methods fall below 50%, as even simple queries require multiple joins and filter predicates reference opaque IDs. We present Financial LINking Text-to-SQL (FLINT), a domain-specialized Text-to-SQL system that closes this gap through three key components: (1) a lookup agent that dynamically resolves natural-language concepts to question-specific reference table constraints, (2) embedding-based retrieval of structurally similar query templates from a compact, expert-authored bank, and (3) schema linking that prunes a large table schema to the relevant subset by traversing foreign-key chains, rather than relying on name similarity alone. We evaluate on two datasets totaling 359 questions over production financial schemas. FLINT outperforms various state-of-the-art baselines using the same LLM. The system is deployed in production as part of a financial data retrieval service.

Figures & tables

Panel of Figure 1 — Figure 1: FLINT pipeline. Stages arranged horizontally execute in parallel. Dashed arrows show the error-driven reflection loop.

Property	Financials	Transactions
Total tables	24	18
Reference / lookup	16	12
Core data	8	6
Avg. joins per query	6.0	5.2
Max joins (single query)	13	13
Queries requiring CTEs	40	17

Table 1: Schema and benchmark statistics for both datasets. Reference tables store lookup values (metrics, period types, currencies); core tables store financial data and transactions.

Panel of Figure 2 — Figure 2: Our multi-stage evaluation pipeline: The generated queries that execute successfully are first matched via AST trees with the gold query. For the unmatched queries we pass them through heuristics and eventually to LLM-as-a-judge.

Multi-agent / multi-candidate
		Fin.	Trans.
Method	Approach	Accuracy (%)
FLINT	Domain-specialized agents	68.8	65.2
CHESS Talaei et al. (2024)	Self-consistency ( $k{=}1$ )	27.1	42.0
CHESS Talaei et al. (2024)	Self-consistency ( $k{=}10$ )	35.6	49.6
Chase-SQL Pourreza et al. (2025)	Multi-path + tournament	16.3	21.0

Table 2: Comparison with existing T2S methods on normalized financial schemas. All methods use Claude 4.5 Opus and share base infrastructure (App. D.1 ); DAIL-SQL, DIN-SQL, and FlexSQL additionally receive shared pipeline components (App. D.10 ), making their comparison conservative; execution success is

\geq

99% for all.

Figure 3: Accuracy vs. generation time on ScienceBenchmark (CORDIS, OncoMX, SDSS), under one harness and LLM. Top-left is best—faster and more accurate. FLINT is the most accurate on every database while generating SQL 5

\times

faster than the multi-agent systems: CHESS issues

k{=}10

candidate generations per question yet trails FLINT’s single pass, and ReFoRCE’s iterative refinement cannot recover semantically wrong filters that still execute. Generation time excludes database execution.

	Financials		Transactions
Configuration	Acc.	$\Delta$	Acc.	$\Delta$
FLINT (full system)	68.8	—	65.2	—
w/o Lookup Agent	54.8	$-$ 14.0	33.3	$-$ 31.9
w/o Few-Shot Retrieval	57.9	$-$ 10.9	40.6	$-$ 24.6
w/o Schema Linking	65.2	$-$ 3.6	50.7	$-$ 14.5
w/o Reflection	67.9	$-$ 0.9	55.1	$-$ 10.1

Table 3: Ablation study (strict accuracy, %). Each row disables one component while keeping the rest of the pipeline intact. Rows sorted by average degradation.

Error Category	Fin.	Trans.	Tot.
Correct tables & columns
Wrong filter / aggregation	15	15	30
Correct tables, wrong columns
Partial match ( $\geq$ 50%)	35	15	50
Low match ( $<$ 50%)	11	10	21
Wrong table selection

Table 4: Error taxonomy for FLINT ’s 117 incorrect predictions. All failures produce executable SQL.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	Financials		Transactions
Difficulty	n	%	n	%
Simple (1–3 joins)	22	10	38	28
Moderate (4–6 joins)	142	64	66	48
Complex (7+ joins / CTE)	57	26	34	25

Appendix

Table 5: Question difficulty distribution by structural complexity of the gold SQL.

Panel of Figure 4 — Figure 4: Opaque identifiers in Financials. The schema exposes only integer foreign keys. As a result, resolving “quarterly EBITDA” to concrete filter values requires querying reference tables, a step the Lookup Agent performs automatically.

Property	Spider	BIRD	Ours
Avg. joins per query	1.2	1.8	5.6
Max joins (single query)	4	6	13
Reference/lookup tables	0	0–2	12–16
Opaque integer IDs	×	×	✓
Currency conversion	×	×	✓
Self-describing values	✓	✓	×

Appendix

Table 6: Comparison with Spider and BIRD. Key differentiators: deep normalization with opaque IDs, mandatory multi-table joins for even simple queries, and production-deployed schemas.

Component	FLINT	CHESS	DAIL/DIN	MAC-SQL	Chase-SQL	FlexSQL	ReFoRCE
Entity linking	✓	× ∗	✓	× †	×	✓	×
Lookup agent	✓	×	✓	× †	×	×	×
Schema linking	✓	✓	✓	LLM	×	✓	×
Similar query retrieval	✓	×	✓	×	×	✓	✓
Reflection / retry	✓	×	DIN	Refiner	Fixer	✓	✓
DB exploration	×	×	×	✓	×	×	✓

Appendix

Table 7: Infrastructure and architectural components per method. ∗ CHESS uses its own schema-value matching. † MAC-SQL achieves equivalent resolution through interactive DB queries.

	Financials		Transactions
Method	ID Acc.	Fail $\to$ ID	ID Acc.	Fail $\to$ ID
FLINT (full)	94%	13%	75%	28%
w/o Lookup Agent	10%	93%	7%	94%
CHESS ( $k{=}10$ )	46%	64%	58%	61%
MAC-SQL	30%	76%	39%	62%
DAIL-SQL	21%	83%	28%	79%

Appendix

Table 8: Filter predicate analysis. ID Acc. : fraction of queries where all required numeric IDs are correctly resolved. Fail

\to

ID : among incorrect predictions, the fraction where

\geq

1 required ID is wrong. On Financials, 64–96% of all failures are attributable to incorrect filter values.

Difficulty	n / N	Acc.	n / N	Acc.
	Financials		Transactions
Simple (1–3 joins)	14/22	64	25/38	66
Moderate (4–6 joins)	107/142	75	47/66	71
Complex (7+ / CTE)	31/57	54	18/34	53
Overall	152/221	69	90/138	65

Appendix

Table 9: FLINT accuracy (%) by query difficulty.

Figure 5: Median end-to-end query latency (log scale). FLINT achieves 10–20s median, almost 2

\times

faster than the next-fastest baseline on Financials and almost 100

\times

faster than CHESS.

Dataset	Median	$\geq$ 90% match	Closest (min TED)
Financials	0.78	10.6%	0.023
Transactions	0.62	0 0.0%	0.111

Appendix

Table 10: Nearest-neighbor content-aware overlap (

1-\text{TED}

) from each evaluation query to the query bank.

Panel of Figure 6 — Figure 6: Accuracy on ScienceBenchmark Zhang et al. (2023) : CORDIS (EU funding, 19 tables), OncoMX (cancer biomarkers, 25 tables), SDSS (astronomy, 6 tables).

	SM3-Text-to-Query
Method	All (n=2000)	Hard (n=631)
Zero-shot	77.8	58.2
Few-shot ( $k{=}3$ )	92.3	85.3
FLINT	95.7	96.5

Appendix

Table 11: SM3-Text-to-Query Sivasubramaniam et al. (2024) results. Hard subset: multi-table joins and unanswerable questions. FLINT ’s gain concentrates on the hard subset (+11pp over few-shot).

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 169 papers

BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations
Aug 29, 2026Yunfan Zhou, Qiming Shi, Yizhou Yang +2Text-to-SQL Text-to-SQL Evaluation
State Key Lab of CAD&CG, Zhejiang University · School of Software Technology, Zhejiang University
TAHOE: Text-to-SQL with Automated Hint Optimization from Experience
Jun 10, 2026Zhiyi Chen, Jie Song, Peng LiLLM Prompting Text-to-SQL
ByteDance Inc. · Georgia Institute of Technology · Work done during an internship at the ByteBrain team, ByteDance
MDB-Link: Hierarchical Schema Linking for Multi-Database Text-to-SQL
Aug 10, 2026Beiyu Xu, Zhenyu Wu, Jiaoyan Chen +1Schema Linking Text-to-SQL
University of Manchester Faculty of Science and Engineering Department of Computer Science

Aug 29, 2026J/K move · Enter open · S save

BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations

Yunfan Zhou, Qiming Shi, Yizhou Yang, Di Weng+1

State Key Lab of CAD&CG, Zhejiang University · School of Software Technology, Zhejiang University

Text-to-SQL Text-to-SQL Evaluation LLM Grounding

Open paper AskPDF arXiv

While recent Large Language Model (LLM)-based text-to-SQL systems achieve impressive performance on standard benchmarks, they struggle when user queries implicitly rely on domain-specific knowledge, such as business logic, data conventions, and analytical practices, that is neither captured by the schema nor explicitly stated in the natural language question. Historical SQL query logs offer a valuable source of such knowledge, yet existing benchmarks do not adequately support evaluation of history-driven approaches. To address this gap, we introduce BIRD-History, a benchmark consisting of 1,393 tasks across 11 databases, designed to evaluate text-to-SQL systems' ability to ground underspecified natural language questions using historical SQL scripts. Each task is annotated with ground-truth labels specifying which historical queries contain relevant knowledge and which SQL clauses encode it, enabling systematic evaluation of both retrieval effectiveness and knowledge utilization. Alongside the benchmark, we propose a plug-in retriever that extracts five types of external knowledge from historical SQL scripts, then retrieves and reranks relevant fragments for query generation. The retriever integrates seamlessly into existing few-shot text-to-SQL pipelines without requiring prompt modifications. Experiments demonstrate consistent improvements across four text-to-SQL systems, highlighting the value of leveraging historical query logs for handling underspecified queries. Dataset and code are open-sourced on https://github.com/zjuidg/BIRD-History.

Abstract

Text-to-SQL

Explore similar work

BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations

TAHOE: Text-to-SQL with Automated Hint Optimization from Experience

MDB-Link: Hierarchical Schema Linking for Multi-Database Text-to-SQL