Both Ends Count! Just How Good are LLM Agents at "Text-to-Big SQL"?

Germán T. Eizaguirre; Lars Tissen; Marc Sánchez-Artigas

cs.DBFeb 25, 2026

Both Ends Count! Just How Good are LLM Agents at "Text-to-Big SQL"?

Authors: Germán T. Eizaguirre, Lars Tissen, Marc Sánchez-Artigas

Organizations: Universitat Rovira i Virgili Tarragona, Spain · RWTH Aachen University Aachen, Germany

Abstract

Text-to-SQL and Big Data are both extensively benchmarked fields, yet there is limited research that evaluates them jointly. In the real world, Text-to-SQL systems are often embedded with Big Data workflows, such as large-scale data processing or interactive data analytics. We refer to this as ``Text-to-Big SQL''. However, existing text-to-SQL benchmarks remain narrowly scoped and overlook the cost and performance implications that arise at scale. For instance, translation errors that are minor on small datasets lead to substantial cost and latency overheads as data scales, a relevant issue completely ignored by text-to-SQL metrics. In this paper, we overcome this overlooked challenge by introducing novel and representative metrics for evaluating Text-to-Big SQL. Our study focuses on production-level LLM agents, a database-agnostic system adaptable to diverse user needs. Via an extensive evaluation of frontier models, we show that text-to-SQL metrics are insufficient for Big Data. In contrast, our proposed text-to-Big SQL metrics accurately reflect execution efficiency, cost, and the impact of data scale. For example, GPT-4o compensates for roughly 7% lower accuracy than the top-performing later-generation models with up to a 12.16x speedup, while GPT-5.2 is more than twice as cost-effective as Gemini 3 Pro at large input scales.

Figures & tables

Figure 1. Architecture and typical execution flow of the evaluated LLM agent.

Model	EX	E2E (s)		% of E2E Time
		Mean	$\sigma$	list	schema	check	run
GPT-4o	0.93	6.55	2.08	9.22	13.15	62.08	13.64
Gemini 3 Flash	1.00	8.37	2.61	9.74	10.15	66.76	11.71
GPT-5.2	0.69	8.44	2.74	13.18	21.36	48.88	14.88
Gemini 2.5 Flash	0.95	9.18	3.21	11.70	10.53	66.07	10.11
Claude Opus 4.5	1.00	11.40	2.82	16.90	18.57	42.32	20.93

Table 1. EX, total execution time and the breakdown of execution time per "Observation-Thought-Action" stage for each LLM, on selected BIRD queries. Results are displayed in ascending order of total execution time.

Model	VES (norm)	VES* (norm)	Time Variation (x)
			list	schema	check	run
GPT-4o	1.00	1.00	1.00x	1.00x	1.00x	1.00x
Gemini 3 Flash	1.06	0.81	1.35x	0.99x	1.37x	1.10x
Gemini 2.5 Flash	1.00	0.78	1.78x	1.12x	1.49x	1.04x
Claude Opus 4.5	1.09	0.57	3.19x	2.46x	1.19x	2.67x
Claude Opus 4.6	1.09	0.51	3.58x	2.65x	1.32x	2.95x

Table 2. VES and VES* for the selected BIRD queries. Both metrics are normalized to the best scoring LLM. LLMs are displayed in descending order of VES*.

Model	VCES norm. ( $\$ ^{-1}$ )	CVQ ($)	Cost Variation (x)
			list	schema	check	run
Gemini 3 Flash	1.00	0.0044	1.00x	1.00x	1.00x	1.00x
Gemini 2.5 Flash	0.85	0.0053	1.36x	1.18x	1.12x	0.98x
GPT-4o	0.55	0.0107	2.14x	2.93x	2.10x	2.64x
Kimi K2.5	0.53	0.0047	1.07x	1.48x	0.99x	1.05x
GPT-5.2	0.25	0.0124	2.62x	4.07x	1.42x	2.46x

Table 3. VCES and CVQ for the selected BIRD queries. The VCES values are normalized to the best-performing LLM, and models are shown in descending order of VCES.

Figure 2. Breakdown of agent execution time (a) and cost (b) across different scale factors (SF) for TPC-H Query 21. Each bar series represents a specific model: Gemini 3 Pro (G), Claude Opus 4.5 (A), and GPT-5.2 (O).

Figure 3. Execution accuracy across three models for TPC-H queries 1, 17, 18, and 21. Accuracy is computed as the average value of Equation 2 over 10 zero-shot translations of each corresponding natural language query.

Panel of Figure 4 — Figure 4. Text-to-SQL (a) and text-to-Big SQL (b) metrics across three models and scale factors, averaged for TPC-H queries 1, 17, 18, and 21.

Figure 5. Challenges of Text-to-Big SQL include, but are not limited to, those of Text-to-SQL.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

QID	Model	Acc.	VES	VES*	VCES	CVQ
61	GPT-4o	1.00	1.08	0.011	1.11	0.010
	GPT-5.2	1.00	0.99	0.006	0.69	0.009
	Claude 4.6	1.00	1.06	0.006	0.20	0.030
	Gemini 3 Flash	1.00	1.06	0.006	0.91	0.006
	Claude 4.5	1.00	1.07	0.006	0.14	0.040
	Kimi K2.5	1.00	1.08	0.005	1.11	0.005

Appendix

Table 4. Per-query BIRD metrics by model (mean

\pm

std across runs). VES, VES*, and VCES follow the paper definitions; CVQ reports the expected cost per valid query.

Figure 6. Correct and incorrect SQL examples for query 886.

Model	Incorrect Translations
DeepSeek Chat	214
GPT-5.2	190
GPT-5	104
GPT-4o	78
Gemini 2.5 Flash	70
Kimi K2.5	57

Appendix

Table 5. Total number of incorrect translations (Eq. 2

==0

) per model. Each model was evaluated on all 10 BIRD queries with 50 translations per query (500 iterations per model).

Figure 7. Distribution of identified text-to-SQL translation errors across all 930 incorrect BIRD query translations, categorized by the taxonomy proposed in ( Shen et al., 2025 )

Panel of Figure 8 — Figure 8. Taxonomy ( Shen et al., 2025 ) of SQL translation errors from our LLM agent. Numbers in parentheses denote error counts. Categories: A=Syntax, B=Schema, C=Logic, D=Convention, E=Semantic, F=Not an Error. Total: 1730 errors across 5 categories.

QID	Model	Acc.	VES	VES*	VCES	CVQ
1	GPT 5.2	0.60 $\pm$ 0.49	0.6507 $\pm$ 0.5318	0.2608 $\pm$ 0.2134	23.1794 $\pm$ 19.0188	0.0188 $\pm$ 0.0000
	Opus 4.5	1.00 $\pm$ 0.00	0.9567 $\pm$ 0.1057	0.2513 $\pm$ 0.0138	3.6030 $\pm$ 0.2671	0.0697 $\pm$ 0.0015
	Gemini 3 Pro (T)	0.00 $\pm$ 0.00	0.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	–
17	Opus 4.5	0.20 $\pm$ 0.40	0.2028 $\pm$ 0.4235	0.0323 $\pm$ 0.0647	0.3865 $\pm$ 0.6072	0.4182 $\pm$ 0.0025
	Gemini 3 Pro (T)	0.10 $\pm$ 0.30	0.1294 $\pm$ 0.3883	0.0140 $\pm$ 0.0420	0.3525 $\pm$ 0.7373	0.3973 $\pm$ 0.0000
	GPT-5.2	0.40 $\pm$ 0.49	0.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	0.0168 $\pm$ 0.0004

Appendix

Table 6. Per-query metrics in TPC-H (SF1) by model (mean

\pm

std across runs). VES, VES*, and VCES follow the paper definitions; CVQ reports the expected cost per valid query.

Query	SF	Model	VES	VES*	VCES	CVQ
Q1	10	Gemini 3 Pro	0.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	–
		Opus 4.5	0.5668 $\pm$ 0.0000	0.2133 $\pm$ 0.0064	1.8351 $\pm$ 0.0783	0.1163 $\pm$ 0.0015
		GPT-5.2	0.3235 $\pm$ 0.2641	0.1857 $\pm$ 0.1517	3.0724 $\pm$ 2.5113	0.1007 $\pm$ 0.0000
	100	Gemini 3 Pro	0.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	–
		Opus 4.5	0.2654 $\pm$ 0.0000	0.1494 $\pm$ 0.0031	0.8809 $\pm$ 0.0259	0.1697 $\pm$ 0.0015
		GPT-5.2	0.1599 $\pm$ 0.1306	0.1170 $\pm$ 0.0955	1.0526 $\pm$ 0.8598	0.1852 $\pm$ 0.0000

Appendix

Table 7. TPC-H metrics by query, model, and scale factor (mean

\pm

std across runs). VES, VES*, and VCES follow the paper definitions; CVQ reports the expected cost per valid query.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 173 papers

PExA: Parallel Exploration Agent for Complex Text-to-SQL
Apr 24, 2026Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang +3LLM Inference Efficiency Text-to-SQL
University of California Los Angeles · Bloomberg
Reliable Text-to-SQL in Resource-Constrained Settings with Small LLMs via Iterative Retry Workflows
Oct 12, 2026Bilel Abderrahmane Benziane, Xavier ScheplerText-to-SQL Small Language Models
Data Lab, Generix Group, Paris, France
LIMIT: Less Is More for Instruction Tuning in Text-to-SQL
Sep 21, 2026Haoyuan Ma, Hengwei Liu, Linjuan Wu +2Text-to-SQL Instruction-Tuning Data Selection
Zhejiang University

Apr 24, 2026J/K move · Enter open · S save

PExA: Parallel Exploration Agent for Complex Text-to-SQL

Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang, Sachith Sri Ram Kothur+2

University of California Los Angeles · Bloomberg

LLM Inference Efficiency Text-to-SQL LLM Agents

Open paper AskPDF arXiv

LLM-based agents for text-to-SQL often struggle with latency-performance trade-off, where performance improvements come at the cost of latency or vice versa. We reformulate text-to-SQL generation within the lens of software test coverage where the original query is prepared with a suite of test cases with simpler, atomic SQLs that are executed in parallel and together ensure semantic coverage of the original query. After iterating on test case coverage, the final SQL is generated only when enough information is gathered, leveraging the explored test case SQLs to ground the final generation. We validated our framework on a state-of-the-art benchmark for text-to-SQL, Spider 2.0, achieving a new state-of-the-art with 70.2% execution accuracy.

Abstract

Text-to-SQL

Explore similar work

PExA: Parallel Exploration Agent for Complex Text-to-SQL

Reliable Text-to-SQL in Resource-Constrained Settings with Small LLMs via Iterative Retry Workflows

LIMIT: Less Is More for Instruction Tuning in Text-to-SQL