Non-technical stakeholders frequently cannot write the SQL needed to extract insights from operational databases. We built and evaluated a Text-to-SQL agent that closes this gap end to end: a six-node LangGraph StateGraph checks question relevance, fetches the live schema, generates PostgreSQL, validates it with a dry run, retries on failure, executes the query, and narrates the result set in plain English. The agent uses prompt engineering only; no model was fine-tuned. We evaluated it on the Chicago Crime dataset (approximately 8.5 million records, 22 attributes) against a hand-built benchmark of 100 natural language questions with ground-truth SQL, stratified into 30 Easy, 40 Medium and 30 Hard items. Comparing two prompt revisions of the same agent, the revised system (V2) reached a Valid SQL Rate of 93% (from 87%), an Execution Accuracy of 60% under a hybrid relational equivalence metric (from 47%; 19% from 12% under strict JSON matching), and a mean Synthesis Quality of 4.34 out of 5 (from 3.91). The single largest driver was removing a LIMIT 10 instruction from the system prompt, which had been truncating multi-row answers. Error analysis attributes the residual failures to relevance-checker false rejections, ambiguous question semantics, and free-tier API rate limits rather than to the language generation step. We report no comparison against an external baseline system or a public benchmark; the study is a single-model engineering evaluation.
Figures & tables
Column Group
Specific Columns
Handling Strategy
Rationale
Categorical
CRIME_TYPE , DESCRIPTION , LOCATION_DESCRIPTION
Replaced with OTHER OFFENSE
Permits downstream COALESCE operations in SQL
Geospatial
LATITUDE , LONGITUDE , X_COORDINATE , Y_COORDINATE
Taken from a matching BLOCK where available; otherwise replaced with the mean for the BEAT
Allows spatial queries to filter records
Temporal
DATE , UPDATED_ON , YEAR
Missing DATE replaced with the most frequent date in the same YEAR
Marked as “Unknown”. Currently there are no Unknown values, since the source comes with all values filled
Distinguishes unknown status from a recorded value
Table 1: Null value handling strategy by column group. Column names are those of the source data before renaming. Imputed categories, coordinates and dates remain in the snapshot, so counts by category, location or date include imputed values.
Figure 1: Overall system architecture. Apache Airflow ingests the Chicago Crime data weekly into a PostgreSQL database running in Docker; the LangGraph agent receives a user question, queries the database and returns a plain-English answer.
Table
Role
Rows
Key columns
crime_incidents
fact table
8,643,493
38 columns, e.g. INCIDENT_DATE , CRIME_TYPE , ARREST
district_names
lookup
23
district_id , district_name
community_area_names
lookup
77
community_area_id , community_area_name
ward_names
lookup
50
ward_id , ward_name
Table 2: Tables in the benchmark database snapshot (Docker image vigneshwarr/chicago-crime-db ; incidents from 2001-01-01 to 2026-09-29). The fact table has 22 source attributes and 16 derived columns; the lookup tables map district, community-area and ward codes to names.
Figure 2: Internal StateGraph architectures of the two LangGraph agents: (a) agent_1 , the simple pipeline; and (b) agent_2 , with relevance checking, validation and retry. Solid arrows indicate unconditional transitions, and dashed arrows indicate conditional transitions. Out-of-scope questions terminate at the relevance checker. The query validator routes valid queries to execution, returns failed queries to the generator for retry, or routes the workflow to the answer synthesizer to request rephrasing after three failed attempts.
Component
agent_1
agent_2
Schema fetcher
yes
yes
Query generator
yes
yes
Query runner
yes
yes
Answer synthesizer
yes
yes
Relevance checker
no
yes
Query validator
no
yes
Table 3: Components of the two agents. All other settings (model, schema, prompts, answer synthesizer) are identical. Because the agents differ in two components, differences in their results are a joint effect of both.
Node
Purpose
Key Implementation Detail
1. Relevance Checker
Filters non-database questions ( agent_2 only)
Regular-expression and keyword rule; outputs is_relevant flag; routes to END or Schema Fetcher
2. Schema Fetcher
Retrieves live DB schema
SQLDatabase.get_table_info()
3. Query Generator
LLM generates SQL
System prompt varied by prompt stage (Table 6 ); fixed (non-retrieved) examples at stages 3 and 4; temp=0, max_tokens=500
4. Query Validator
Dry-runs SQL ( agent_2 only)
EXPLAIN + data-modification keyword check; increments retry counter (at most 3 generation attempts in total)
5. Query Runner
Executes SQL
db.run(fetch="all") ; the full result is returned
6. Answer Synthesizer
Narrates results in plain English
LLM summarizes the first 10 result rows; returns a fixed message when no query could be produced
Table 4: StateGraph nodes: functions and implementation details. agent_2 uses all six nodes; agent_1 uses nodes 2, 3, 5 and 6.
Component
Technology
Rationale
LLM
Groq API (openai/gpt-oss-120b)
Hosted inference for an open-weight model
Agentic Framework
LangGraph
Explicit state management; transparent execution flow; conditional routing
Database Driver
LangChain SQLDatabase + SQLAlchemy
Abstraction over PostgreSQL dialect; built-in schema introspection
Database
PostgreSQL 15 for all experiments (the production pipeline is described as compatible with version 14 or later)
One sentence: “You are a SQL assistant. Write a SQL query for the user’s question.”
Baseline; no database knowledge beyond the schema in the user message
2. Rules
Stage 1 replaced by a role statement, the foreign-key relationships to the lookup tables, and ten numbered rules (Figure 4 )
PostgreSQL conventions: quoting of upper-case identifiers and string values, boolean and date handling, JOINs on names, CTEs and window functions, percentage formula, no LIMIT unless asked, read-only queries, output format
3. One-shot
Stage 2 followed by one worked question–SQL pair (Figure 5 )
A concrete example of the expected output style
4. Few-shot
Stage 2 followed by four worked question–SQL pairs (Figure 6 )
Examples covering counting, top- N ranking, boolean filters and grouped aggregation with HAVING
Table 6: The four prompt stages. Stages 3 and 4 append examples to the stage 2 prompt unchanged.
Parameter
Value
Rationale
Temperature
0
Minimizes sampling variation
Generation attempts ( agent_2 )
At most 3 in total
Bounds the number of retries and the added latency
Relevance rule ( agent_2 )
At least one action verb and one data-context word
Decides relevance without a model call
LLM Model
openai/gpt-oss-120b
Open-weight model hosted on Groq
Table 7: Key settings held fixed in all experiments.
Metric
Definition
Pre-set goal
VSR (Valid SQL Rate)
Percentage of questions for which the agent produced a query that executed without error; a question for which no query was produced counts as invalid
≥ 95%
EM (normalized result match)
Percentage of questions whose result equals the reference result after the deterministic normalization described below; no model is involved
—
EX (judge-assisted execution accuracy)
Percentage of questions whose result matches the reference exactly (EM) or, for executed results that do not, is judged equivalent by an LLM judge
≥ 85%
SQ (Synthesis Quality)
Mean Likert score (1–5) assigned by the LLM judge to the agent’s plain-English answer, judged against the rows its own query returned
≥ 4.0
Table 8: Evaluation metrics and pre-set goals. EM is deterministic; EX and SQ depend in part on an LLM judge. A question for which no query was produced counts as invalid for VSR and incorrect for EM and EX. Goals were set before development.
Score
Criterion
5
Fully faithful to the result rows and directly answers the question; numbers and names are correct, nothing important is omitted, no facts are invented.
4
Faithful and answers the question, with a minor flaw (slight omission, awkward framing, unnecessary hedging, small rounding looseness).
3
Partly useful: answers only part of the question, ignores part of the result, is vague, or contains a noticeable misreading that does not flip the answer.
2
Substantially misleading: misstates the results, states a conclusion the rows do not support, or answers a different question.
1
Unusable or wrong: fabricated numbers, contradicts the rows, an empty or error response, or reports “no results” or “out of scope” when the question is in scope and data exist.
Table 9: Rubric for Synthesis Quality (SQ). The judge scores the answer against the query result it is shown, not against the reference. An empty result with an answer saying nothing was found is faithful; if the query failed or no query ran, an answer saying “no results found” or “out of scope” scores 1.
openai/gpt-oss-120b on Groq (Groq SDK 0.37.1), temperature 0, developer tier
Database access
SQLAlchemy 2.0.48, psycopg2-binary 2.9.11
Database
PostgreSQL 15 in Docker ( vigneshwarr/chicago-crime-db ), read-only role for the agents
Judge
anthropic/claude-haiku-5-5 via the Anthropic Python SDK
Parallelism
8 questions in parallel within a run
Table 10: Software and configuration used in the experiments.
Agent
Prompt stage
VSR (%)
EX (%)
SQ (1–5)
agent_1
1 simple
67.1±2.2
62.1±2.5
2.33±0.04
agent_1
2 rules
99.0±1.0
93.1±1.2
2.90±0.08
agent_1
3 one-shot
98.5±0.6
91.9±0.5
2.91±0.04
agent_1
4 few-shot
97.9±0.7
91.5±1.4
2.88±0.04
agent_2
1 simple
91.7±0.0
85.0±0.6
2.69±0.04
agent_2
2 rules
91.7±0.0
86.2±0.5
3.52±0.70
Table 11: Performance of the two agents at each prompt stage (mean ± SD over five runs, 96 questions per run). EX is judge-assisted. Pre-set goals: VSR ≥ 95%, EX ≥ 85%, SQ ≥ 4.0. The SQ column is confounded with run order and should not be used to compare agents or stages (Section 3.7 ).
Easy (27)
Medium (39)
Hard (30)
Agent
Stage
VSR
EX
VSR
EX
VSR
EX
agent_1
1 simple
63
63
68
63
69
61
agent_1
2 rules
100
100
99
90
98
91
agent_1
3 one-shot
100
100
98
89
98
88
agent_1
4 few-shot
100
100
99
92
95
83
agent_2
1 simple
96
96
87
82
93
79
Table 12: VSR and EX (%, mean over five runs) by difficulty tier. Tier sizes are given after removing the four reserved questions. The agent_2 ceilings (96, 87 and 93% VSR) reflect the questions rejected by its relevance checker (1 Easy, 5 Medium, 2 Hard). In these tiers one question corresponds to 2.6–3.7 points. SQ is not broken down by tier because of the confound described in Section 3.7 .
agent_1
agent_2
Questions that received a query
96 / 96
88 / 96
Queries produced that executed without error
98.5% (of 96)
100.0% (of 88)
EX among questions that received a query
92.2% (of 96)
94.7% (of 88)
EX over all 96 questions
92.2%
86.8%
Table 13: Comparison of the two agents, stages 2–4 (mean of the three stages). The second and third rows use different question sets for the two agents and are not directly comparable; only the last row uses the same 96 questions for both.
ID
Question
Action phrase
Data word
Why it is rejected
28
List all distinct location risk categories.
list
none
No data-context word.
35
What is the arrest rate for each crime category?
none
arrest, crime
“What is the” is not an action phrase.
37
Which 5 districts (by name) have the highest number of arrests?
none
arrest, district
“Which” and “highest number” are not action phrases.
42
Which districts (by name) had more than 20000 crimes in 2024?
none
crime, district
“Which” and “more than” are not action phrases.
44
Which year had the highest number of homicide offenses?
none
year
“Which” and “highest number” are not action phrases.
62
Which community areas (by name) had more than 5000 property crimes?
none
crime
“Which” and “more than” are not action phrases.
Table 14: The eight questions rejected by the relevance checker in all 20 runs of agent_2 , with the action phrase and data-context words each contains. Seven fail the action-phrase test and one (ID 28) fails the data-context test.
Agent
Stage
No query
Failed to execute
Wrong result
agent_1
1 simple
0
31.6
4.8
agent_1
2 rules
0
1.0
5.6
agent_1
3 one-shot
0
1.4
6.4
agent_1
4 few-shot
0
2.0
6.2
agent_2
1 simple
8.0
0
6.4
agent_2
2 rules
8.0
0
5.2
Table 15: Mean number of questions per run (of 96) with EX = 0, by failure type. For agent_2 , the 8.0 questions without a query are those rejected by the relevance checker. “Wrong result” counts queries that executed but were neither an exact match nor judged equivalent.
Agent
Stage
EM (%)
EX (%)
agent_1
1 simple
52.9
62.1
agent_1
2 rules
77.3
93.1
agent_1
3 one-shot
79.0
91.9
agent_1
4 few-shot
83.5
91.5
agent_2
1 simple
72.1
85.0
agent_2
2 rules
71.9
86.2
Table 16: Normalized result match (EM) and judge-assisted execution accuracy (EX), mean over five runs. EM ignores row order, column names, letter case and whitespace and rounds numbers to two decimals; EX also counts executed results that the LLM judge classified as equivalent. The judge’s decisions were not audited manually, so the EX–EM difference rests on unvalidated judgments.
Runs
Exact copies
Share
agent_1 , stage 1
11 of 65
17%
agent_1 , stages 2–4
71 of 341
21%
agent_2 , stage 1
20 of 110
18%
agent_2 , stage 2, runs 1–2
8 of 44
18%
agent_2 , stage 2 runs 3–5, and stages 3–4
289 of 289
100%
Table 17: Answers to single-number questions (results of at least 1,000) that repeat the query’s number exactly, pooled over runs and grouped by when the runs were made ( agent_1 first, then agent_2 ). Run order, agent and prompt stage were not independently varied, so the cause of the change cannot be identified.
LLM-based agents for text-to-SQL often struggle with latency-performance trade-off, where performance improvements come at the cost of latency or vice versa. We reformulate text-to-SQL generation within the lens of software test coverage where the original query is prepared with a suite of test cases with simpler, atomic SQLs that are executed in parallel and together ensure semantic coverage of the original query. After iterating on test case coverage, the final SQL is generated only when enough information is gathered, leveraging the explored test case SQLs to ground the final generation. We validated our framework on a state-of-the-art benchmark for text-to-SQL, Spider 2.0, achieving a new state-of-the-art with 70.2% execution accuracy.
Text-to-SQL is the task of translating natural language queries into executable SQL for a given database, enabling non-expert users to access structured data without writing SQL manually. Despite rapid advances driven by large language models (LLMs), existing approaches still struggle with complex queries in real-world settings, where database schemas are large and questions require multi-step reasoning over many interrelated tables. In such cases, providing the full schema often exceeds the context window, while one-shot generation frequently produces non-executable SQL due to syntax errors and incorrect schema linking. To address these challenges, we introduce AV-SQL, a framework that decomposes complex Text-to-SQL into a pipeline of specialized LLM agents. Central to AV-SQL is the concept of agentic views: agent-generated Common Table Expressions (CTEs) that encapsulate intermediate query logic and filter relevant schema elements from large schemas. AV-SQL operates in three stages: (1) a rewriter agent compresses and clarifies the input query; (2) a view generator agent processes schema chunks to produce agentic views; and (3) a planner, generator, and revisor agent collaboratively compose these views into the final SQL query. Extensive experiments show that AV-SQL achieves 70.38% execution accuracy on the challenging Spider 2.0 benchmark, outperforming state-of-the-art baselines, while remaining competitive on standard datasets with 85.59% on Spider, 72.16% on BIRD and 63.78% on KaggleDBQA. Our source code is available at https://github.com/pminhtam/AV-SQL.
Minh Tam Pham, Trinh Pham, Tong Chen +3
Griffith University (Australia) · The University of Queensland (Australia)
Text-to-SQL over large analytical databases requires navigating complex schemas, resolving ambiguous queries, and grounding decisions in actual data. Most current systems follow a fixed pipeline where schema elements are retrieved once upfront and the database is only revisited for post-hoc repair, limiting recovery from early mistakes. We present FlexSQL, a text-to-SQL agent whose core design principle is flexible database interaction: the agent can explore schema structure, inspect data values, and run verification queries at any point during reasoning. FlexSQL generates diverse execution plans to cover multiple query interpretations, implements each plan in either SQL or Python depending on the task, and uses a two-tiered repair mechanism that can backtrack from code-level errors to plan-level revisions. On Spider2-Snow, using gpt-oss-120b, FlexSQL achieves a 65.4% score, outperforming strong open-source baselines that use stronger, larger models such as gpt-o3 and DeepSeek-R1. When integrated into a general-purpose coding agent (as skills in Claude Code), our approach yields over 10% relative improvement on Spider2-Snow. Further analysis shows that flexible exploration and flexible execution jointly contribute to the effectiveness of our approach, highlighting flexibility as a key design principle. Our code is available at: https://github.com/StringNLPLAB/FlexSQL
Quang Hieu Pham, Yang He, Ping Nie +5
University of Alberta · Simon Fraser University · University of Waterloo +1