Agentic-SQL Revisited: Autonomy-Based Taxonomy and Empirical Benchmark Analysis for LLM Text-to-SQL

Yiyun Su; Zujun Peng; Yu Tian; Yuting Liu; Changruo Zhao; Huiying Zhu; Luyan Zhang; Heming Zeng

cs.AIAug 15, 2026

Agentic-SQL Revisited: Autonomy-Based Taxonomy and Empirical Benchmark Analysis for LLM Text-to-SQL

Authors: Yiyun Su, Zujun Peng, Yu Tian, Yuting Liu, Changruo Zhao, Huiying Zhu, Luyan Zhang, Heming Zeng

Abstract

LLM-based Text-to-SQL progress is reported across heterogeneous benchmarks, backbones, and inference protocols, making cross-system comparison fragile. We reframe the field as a leaderboard aggregation: we collect the metrics authors themselves report and organize them along an inference-autonomy axis spanning constrained, in-context, iterative, agentic, and reasoning-internalized generation, with traceable provenance for every cell. To anchor the aggregation empirically, we run a focused case study on Spider, comparing 8B open-source backbones with and without chain-of-thought (CoT) supervision against few-shot DeepSeek~~V3 and GLM-4 baselines. Four patterns emerge: Spider gains transfer unevenly to BIRD and Spider~~2.0; autonomy buys robustness at non-trivial cost; reasoning internalization sits between answer-only decoding and externally orchestrated agents; and CoT gains concentrate on Hard and Extra-Hard queries. We release a Python harness mirroring the autonomy axis so that future methods can be added directly to the leaderboard.

Figures & tables

Aspect	Spider	BIRD	Spider 2.0
Primary stress	Cross-domain composition	Realistic grounding	Workflow realism
Question style	Single-turn NL $\rightarrow$ SQL	NL $\rightarrow$ SQL + evidence $K$	Multi-step, debug-aware
Schema scale	Multi-table, medium	Wide, large	Very large, evolving
Value realism	Moderate	High	High
External knowledge	Not required	Often required	Frequently required
SQL dialect	SQLite mostly	SQLite mostly	BigQuery / Snowflake / etc.

Table 1: Datasets that anchor the leaderboard. The shift is from compositional-generalization stress to grounding-and-realism stress to enterprise-workflow stress; the three benchmarks are not interchangeable.

System	Autonomy	Training	Backbone class
PICARD [ 6 ]	L0	SFT + constrained dec.	T5-3B
DAIL-SQL [ 7 ]	L1	ICL	GPT-4 / Code-LLaMA
DIN-SQL [ 8 ]	L2	ICL	GPT-4
DART-SQL [ 9 ]	L2	ICL + rewriting	GPT-4 / open
DTS-SQL [ 10 ]	L2	SFT	DeepSeek 7B / open
TS-SQL [ 11 ]	L2	ICL + test-driven	GPT-4 / open

Table 2: Method catalog. ICL = in-context learning; SFT = supervised fine-tuning; STR = self-taught reasoning. The autonomy column maps each system to a level in Section 3.1. The four CoT-SFT and 3-shot rows at the bottom are run under the protocol of Section 5.

System	Backbone	Autonomy	EM	EX
PICARD [ 6 ]	T5-3B	L0	71.9	75.1
DAIL-SQL [ 7 ]	GPT-4	L1	–	–
DIN-SQL+GPT-4 [ 8 ]	GPT-4	L2	60.0	85.3
DART-SQL [ 9 ]	GPT-4	L2	–	–
DTS-SQL [ 10 ]	DeepSeek 7B	L2	–	–
TS-SQL [ 11 ]	GPT-4 / open	L2	–	–

Table 3: Spider leaderboard, v1. Numbers are taken from the source paper of each system, cited in the System column. The bottom block is produced by our case study (Section 5) on the Spider holdout test set under a uniform protocol. Dashes indicate that the original paper did not report the metric in a directly-comparable form.

System	Backbone	Autonomy	EX	VES
DIN-SQL+GPT-4 [ 8 ]	GPT-4	L2	55.90	59.44
DAIL-SQL [ 7 ]	GPT-4	L1	–	–
DART-SQL [ 9 ]	GPT-4	L2	–	–
DTS-SQL [ 10 ]	DeepSeek 7B	L2	–	–
TS-SQL [ 11 ]	GPT-4 / open	L2	–	–
MAC-SQL [ 12 ]	GPT-4	L3	–	–

Table 4: BIRD leaderboard, v1. Same conventions as Table 3 . Each numeric cell is sourced to the paper cited in the System column.

Configuration	Backbone	Easy	Medium	Hard	X-Hard	All
DeepSeek V3 (3-shot)	DeepSeek V3	96.62	53.22	33.33	24.04	51.47
GLM-4 (3-shot)	GLM-4	92.57	67.03	59.08	41.53	66.28
No-CoT SFT	LLaMA3.1-8B+LoRA	96.28	75.30	69.49	67.76	76.01
CoT-SFT	LLaMA3.1-8B+LoRA	96.62	74.93	68.25	73.22	76.01
No-CoT SFT	Qwen3-8B+LoRA	96.62	76.20	70.19	71.58	77.04
CoT-SFT	Qwen3-8B+LoRA	94.95	80.32	81.09	79.25	82.24

Table 5: Case-study results on the Spider test set: difficulty-stratified Execution Accuracy (%). Easy / Medium / Hard / Extra-Hard buckets follow the official Spider difficulty rules. “All EX” is the aggregate over all 2,147 test examples. The two 8B SFT rows differ only in whether the supervision target includes a five-stage reasoning trace; otherwise the protocol is identical [ 20 ] .

Dimension	L0	L1	L1.5	L2	L3
Inference cost	low	low	low–med	medium	high
Error recovery	none	weak	moderate	moderate	strong
Diagnosability	limited	limited	moderate	moderate	strong
Schema-scale tolerance	limited	limited	moderate	moderate	strong
Deployment simplicity	strong	strong	moderate	weaker	weakest
Typical failure	syntax	grounding	supervision	repair quality	control

Table 6: Qualitative comparison of autonomy levels. Cells are derived from the structural definitions in Section 3.1, not from the numeric leaderboard; they are intended to be read alongside Tables 3 and 4 , not in their place.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 173 papers

PExA: Parallel Exploration Agent for Complex Text-to-SQL
Apr 24, 2026Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang +3LLM Inference Efficiency Text-to-SQL
University of California Los Angeles · Bloomberg
Integrating Reasoning and Generalization in Text-to-SQL via Self-Enhanced Fine-Tuning
Jun 14, 2026Feng Lyu, Jinfeng Cen, Sijing Duan +4Execution-Guided Text-to-SQL Self-Training for Language Models
Central South University · Tsinghua University · Nanjing University +1
AV-SQL: Decomposing Complex Text-to-SQL Queries with Agentic Views
Apr 8, 2026Minh Tam Pham, Trinh Pham, Tong Chen +3Text-to-SQL Multi-Agent LLM Systems
Griffith University (Australia) · The University of Queensland (Australia)

Apr 24, 2026J/K move · Enter open · S save

PExA: Parallel Exploration Agent for Complex Text-to-SQL

Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang, Sachith Sri Ram Kothur+2

University of California Los Angeles · Bloomberg

LLM Inference Efficiency Text-to-SQL LLM Agents

Open paper AskPDF arXiv

LLM-based agents for text-to-SQL often struggle with latency-performance trade-off, where performance improvements come at the cost of latency or vice versa. We reformulate text-to-SQL generation within the lens of software test coverage where the original query is prepared with a suite of test cases with simpler, atomic SQLs that are executed in parallel and together ensure semantic coverage of the original query. After iterating on test case coverage, the final SQL is generated only when enough information is gathered, leveraging the explored test case SQLs to ground the final generation. We validated our framework on a state-of-the-art benchmark for text-to-SQL, Spider 2.0, achieving a new state-of-the-art with 70.2% execution accuracy.

Abstract

Text-to-SQL

Explore similar work

PExA: Parallel Exploration Agent for Complex Text-to-SQL

Integrating Reasoning and Generalization in Text-to-SQL via Self-Enhanced Fine-Tuning

AV-SQL: Decomposing Complex Text-to-SQL Queries with Agentic Views