An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database

Vigneshwar Ravi Rao; Rupesh Swarnakar; Fayeq Jeelani Syed†

cs.CLSep 19, 2026

An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database

Authors: Vigneshwar Ravi Rao, Rupesh Swarnakar, Fayeq Jeelani Syed†

Organizations: Luddy School of Informatics, Computing and Engineering, Indiana University, Indianapolis, Indiana, USA

Abstract

Non-technical stakeholders frequently cannot write the SQL needed to extract insights from operational databases. We built and evaluated a Text-to-SQL agent that closes this gap end to end: a six-node LangGraph StateGraph checks question relevance, fetches the live schema, generates PostgreSQL, validates it with a dry run, retries on failure, executes the query, and narrates the result set in plain English. The agent uses prompt engineering only; no model was fine-tuned. We evaluated it on the Chicago Crime dataset (approximately 8.5 million records, 22 attributes) against a hand-built benchmark of 100 natural language questions with ground-truth SQL, stratified into 30 Easy, 40 Medium and 30 Hard items. Comparing two prompt revisions of the same agent, the revised system (V2) reached a Valid SQL Rate of 93% (from 87%), an Execution Accuracy of 60% under a hybrid relational equivalence metric (from 47%; 19% from 12% under strict JSON matching), and a mean Synthesis Quality of 4.34 out of 5 (from 3.91). The single largest driver was removing a LIMIT 10 instruction from the system prompt, which had been truncating multi-row answers. Error analysis attributes the residual failures to relevance-checker false rejections, ambiguous question semantics, and free-tier API rate limits rather than to the language generation step. We report no comparison against an external baseline system or a public benchmark; the study is a single-model engineering evaluation.

Figures & tables

Column Group	Specific Columns	Handling Strategy	Rationale
Categorical	CRIME_TYPE , DESCRIPTION , LOCATION_DESCRIPTION	Replaced with OTHER OFFENSE	Permits downstream COALESCE operations in SQL
Geospatial	LATITUDE , LONGITUDE , X_COORDINATE , Y_COORDINATE	Taken from a matching BLOCK where available; otherwise replaced with the mean for the BEAT	Allows spatial queries to filter records
Temporal	DATE , UPDATED_ON , YEAR	Missing DATE replaced with the most frequent date in the same YEAR	Malformed dates trigger query failures; ensures temporal consistency
Logical	ARREST , DOMESTIC	Marked as “Unknown”. Currently there are no Unknown values, since the source comes with all values filled	Distinguishes unknown status from a recorded value

Table 1: Null value handling strategy by column group. Column names are those of the source data before renaming. Imputed categories, coordinates and dates remain in the snapshot, so counts by category, location or date include imputed values.

Panel of Figure 1 — Figure 1: Overall system architecture. Apache Airflow ingests the Chicago Crime data weekly into a PostgreSQL database running in Docker; the LangGraph agent receives a user question, queries the database and returns a plain-English answer.

Table	Role	Rows	Key columns
crime_incidents	fact table	8,643,493	38 columns, e.g. INCIDENT_DATE , CRIME_TYPE , ARREST
district_names	lookup	23	district_id , district_name
community_area_names	lookup	77	community_area_id , community_area_name
ward_names	lookup	50	ward_id , ward_name

Table 2: Tables in the benchmark database snapshot (Docker image vigneshwarr/chicago-crime-db ; incidents from 2001-01-01 to 2026-09-29). The fact table has 22 source attributes and 16 derived columns; the lookup tables map district, community-area and ward codes to names.

Panel of Figure 2 — Figure 2: Internal StateGraph architectures of the two LangGraph agents: (a) agent_1 , the simple pipeline; and (b) agent_2 , with relevance checking, validation and retry. Solid arrows indicate unconditional transitions, and dashed arrows indicate conditional transitions. Out-of-scope questions terminate at the relevance checker. The query validator routes valid queries to execution, returns failed queries to the generator for retry, or routes the workflow to the answer synthesizer to request rephrasing after three failed attempts.

Component	agent_1	agent_2
Schema fetcher	yes	yes
Query generator	yes	yes
Query runner	yes	yes
Answer synthesizer	yes	yes
Relevance checker	no	yes
Query validator	no	yes

Table 3: Components of the two agents. All other settings (model, schema, prompts, answer synthesizer) are identical. Because the agents differ in two components, differences in their results are a joint effect of both.

Node	Purpose	Key Implementation Detail
1. Relevance Checker	Filters non-database questions ( agent_2 only)	Regular-expression and keyword rule; outputs is_relevant flag; routes to END or Schema Fetcher
2. Schema Fetcher	Retrieves live DB schema	SQLDatabase.get_table_info()
3. Query Generator	LLM generates SQL	System prompt varied by prompt stage (Table 6 ); fixed (non-retrieved) examples at stages 3 and 4; temp=0, max_tokens=500
4. Query Validator	Dry-runs SQL ( agent_2 only)	EXPLAIN + data-modification keyword check; increments retry counter (at most 3 generation attempts in total)
5. Query Runner	Executes SQL	db.run(fetch="all") ; the full result is returned
6. Answer Synthesizer	Narrates results in plain English	LLM summarizes the first 10 result rows; returns a fixed message when no query could be produced

Table 4: StateGraph nodes: functions and implementation details. agent_2 uses all six nodes; agent_1 uses nodes 2, 3, 5 and 6.

Component	Technology	Rationale
LLM	Groq API (openai/gpt-oss-120b)	Hosted inference for an open-weight model
Agentic Framework	LangGraph	Explicit state management; transparent execution flow; conditional routing
Database Driver	LangChain SQLDatabase + SQLAlchemy	Abstraction over PostgreSQL dialect; built-in schema introspection
Database	PostgreSQL 15 for all experiments (the production pipeline is described as compatible with version 14 or later)	Supports the query patterns in the benchmark
Containerization	Docker	Environment reproducibility; isolation
Data Pipeline	Apache Airflow	Scheduled weekly ingestion; automatic error handling; audit trail

Table 5: Tools and frameworks used in the system.

Stage	Prompt content	What it adds
1. Simple	One sentence: “You are a SQL assistant. Write a SQL query for the user’s question.”	Baseline; no database knowledge beyond the schema in the user message
2. Rules	Stage 1 replaced by a role statement, the foreign-key relationships to the lookup tables, and ten numbered rules (Figure 4 )	PostgreSQL conventions: quoting of upper-case identifiers and string values, boolean and date handling, JOINs on names, CTEs and window functions, percentage formula, no LIMIT unless asked, read-only queries, output format
3. One-shot	Stage 2 followed by one worked question–SQL pair (Figure 5 )	A concrete example of the expected output style
4. Few-shot	Stage 2 followed by four worked question–SQL pairs (Figure 6 )	Examples covering counting, top- $N$ ranking, boolean filters and grouped aggregation with HAVING

Table 6: The four prompt stages. Stages 3 and 4 append examples to the stage 2 prompt unchanged.

Parameter	Value	Rationale
Temperature	0	Minimizes sampling variation
Generation attempts ( agent_2 )	At most 3 in total	Bounds the number of retries and the added latency
Relevance rule ( agent_2 )	At least one action verb and one data-context word	Decides relevance without a model call
LLM Model	openai/gpt-oss-120b	Open-weight model hosted on Groq

Table 7: Key settings held fixed in all experiments.

Metric	Definition	Pre-set goal
VSR (Valid SQL Rate)	Percentage of questions for which the agent produced a query that executed without error; a question for which no query was produced counts as invalid	$\geq$ 95%
EM (normalized result match)	Percentage of questions whose result equals the reference result after the deterministic normalization described below; no model is involved	—
EX (judge-assisted execution accuracy)	Percentage of questions whose result matches the reference exactly (EM) or, for executed results that do not, is judged equivalent by an LLM judge	$\geq$ 85%
SQ (Synthesis Quality)	Mean Likert score (1–5) assigned by the LLM judge to the agent’s plain-English answer, judged against the rows its own query returned	$\geq$ 4.0

Table 8: Evaluation metrics and pre-set goals. EM is deterministic; EX and SQ depend in part on an LLM judge. A question for which no query was produced counts as invalid for VSR and incorrect for EM and EX. Goals were set before development.

Score	Criterion
5	Fully faithful to the result rows and directly answers the question; numbers and names are correct, nothing important is omitted, no facts are invented.
4	Faithful and answers the question, with a minor flaw (slight omission, awkward framing, unnecessary hedging, small rounding looseness).
3	Partly useful: answers only part of the question, ignores part of the result, is vague, or contains a noticeable misreading that does not flip the answer.
2	Substantially misleading: misstates the results, states a conclusion the rows do not support, or answers a different question.
1	Unusable or wrong: fabricated numbers, contradicts the rows, an empty or error response, or reports “no results” or “out of scope” when the question is in scope and data exist.

Table 9: Rubric for Synthesis Quality (SQ). The judge scores the answer against the query result it is shown, not against the reference. An empty result with an answer saying nothing was found is faithful; if the query failed or no query ran, an answer saying “no results found” or “out of scope” scores 1.

Component	Version / setting
Agent framework	LangGraph 1.0.10, LangChain Core 1.2.17, LangChain Community 0.4.1, LangChain Groq 1.1.2
LLM inference	openai/gpt-oss-120b on Groq (Groq SDK 0.37.1), temperature 0, developer tier
Database access	SQLAlchemy 2.0.48, psycopg2-binary 2.9.11
Database	PostgreSQL 15 in Docker ( vigneshwarr/chicago-crime-db ), read-only role for the agents
Judge	anthropic/claude-haiku-5-5 via the Anthropic Python SDK
Parallelism	8 questions in parallel within a run

Table 10: Software and configuration used in the experiments.

Agent	Prompt stage	VSR (%)	EX (%)	SQ (1–5)
agent_1	1 simple	$67.1\pm 2.2$	$62.1\pm 2.5$	$2.33\pm 0.04$
agent_1	2 rules	$99.0\pm 1.0$	$93.1\pm 1.2$	$2.90\pm 0.08$
agent_1	3 one-shot	$98.5\pm 0.6$	$91.9\pm 0.5$	$2.91\pm 0.04$
agent_1	4 few-shot	$97.9\pm 0.7$	$91.5\pm 1.4$	$2.88\pm 0.04$
agent_2	1 simple	$91.7\pm 0.0$	$85.0\pm 0.6$	$2.69\pm 0.04$
agent_2	2 rules	$91.7\pm 0.0$	$86.2\pm 0.5$	$3.52\pm 0.70$

Table 11: Performance of the two agents at each prompt stage (mean

\pm

SD over five runs, 96 questions per run). EX is judge-assisted. Pre-set goals: VSR

\geq

95%, EX

\geq

85%, SQ

\geq

4.0. The SQ column is confounded with run order and should not be used to compare agents or stages (Section 3.7 ).

		Easy (27)		Medium (39)		Hard (30)
Agent	Stage	VSR	EX	VSR	EX	VSR	EX
agent_1	1 simple	63	63	68	63	69	61
agent_1	2 rules	100	100	99	90	98	91
agent_1	3 one-shot	100	100	98	89	98	88
agent_1	4 few-shot	100	100	99	92	95	83
agent_2	1 simple	96	96	87	82	93	79

Table 12: VSR and EX (%, mean over five runs) by difficulty tier. Tier sizes are given after removing the four reserved questions. The agent_2 ceilings (96, 87 and 93% VSR) reflect the questions rejected by its relevance checker (1 Easy, 5 Medium, 2 Hard). In these tiers one question corresponds to 2.6–3.7 points. SQ is not broken down by tier because of the confound described in Section 3.7 .

	agent_1	agent_2
Questions that received a query	96 / 96	88 / 96
Queries produced that executed without error	98.5% (of 96)	100.0% (of 88)
EX among questions that received a query	92.2% (of 96)	94.7% (of 88)
EX over all 96 questions	92.2%	86.8%

Table 13: Comparison of the two agents, stages 2–4 (mean of the three stages). The second and third rows use different question sets for the two agents and are not directly comparable; only the last row uses the same 96 questions for both.

ID	Question	Action phrase	Data word	Why it is rejected
28	List all distinct location risk categories.	list	none	No data-context word.
35	What is the arrest rate for each crime category?	none	arrest, crime	“What is the” is not an action phrase.
37	Which 5 districts (by name) have the highest number of arrests?	none	arrest, district	“Which” and “highest number” are not action phrases.
42	Which districts (by name) had more than 20000 crimes in 2024?	none	crime, district	“Which” and “more than” are not action phrases.
44	Which year had the highest number of homicide offenses?	none	year	“Which” and “highest number” are not action phrases.
62	Which community areas (by name) had more than 5000 property crimes?	none	crime	“Which” and “more than” are not action phrases.

Table 14: The eight questions rejected by the relevance checker in all 20 runs of agent_2 , with the action phrase and data-context words each contains. Seven fail the action-phrase test and one (ID 28) fails the data-context test.

Agent	Stage	No query	Failed to execute	Wrong result
agent_1	1 simple	0	31.6	4.8
agent_1	2 rules	0	1.0	5.6
agent_1	3 one-shot	0	1.4	6.4
agent_1	4 few-shot	0	2.0	6.2
agent_2	1 simple	8.0	0	6.4
agent_2	2 rules	8.0	0	5.2

Table 15: Mean number of questions per run (of 96) with EX = 0, by failure type. For agent_2 , the 8.0 questions without a query are those rejected by the relevance checker. “Wrong result” counts queries that executed but were neither an exact match nor judged equivalent.

Agent	Stage	EM (%)	EX (%)
agent_1	1 simple	52.9	62.1
agent_1	2 rules	77.3	93.1
agent_1	3 one-shot	79.0	91.9
agent_1	4 few-shot	83.5	91.5
agent_2	1 simple	72.1	85.0
agent_2	2 rules	71.9	86.2

Table 16: Normalized result match (EM) and judge-assisted execution accuracy (EX), mean over five runs. EM ignores row order, column names, letter case and whitespace and rounds numbers to two decimals; EX also counts executed results that the LLM judge classified as equivalent. The judge’s decisions were not audited manually, so the EX–EM difference rests on unvalidated judgments.

Runs	Exact copies	Share
agent_1 , stage 1	11 of 65	17%
agent_1 , stages 2–4	71 of 341	21%
agent_2 , stage 1	20 of 110	18%
agent_2 , stage 2, runs 1–2	8 of 44	18%
agent_2 , stage 2 runs 3–5, and stages 3–4	289 of 289	100%

Table 17: Answers to single-number questions (results of at least 1,000) that repeat the query’s number exactly, pooled over runs and grouped by when the runs were made ( agent_1 first, then agent_2 ). Run order, agent and prompt stage were not independently varied, so the cause of the change cannot be identified.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 169 papers

PExA: Parallel Exploration Agent for Complex Text-to-SQL
Apr 24, 2026Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang +3LLM Inference Efficiency Text-to-SQL
University of California Los Angeles · Bloomberg
AV-SQL: Decomposing Complex Text-to-SQL Queries with Agentic Views
Apr 8, 2026Minh Tam Pham, Trinh Pham, Tong Chen +3Text-to-SQL Multi-Agent LLM Systems
Griffith University (Australia) · The University of Queensland (Australia)
FlexSQL: Flexible Exploration and Execution Make Better Text-to-SQL Agents
May 4, 2026Quang Hieu Pham, Yang He, Ping Nie +5Execution-Guided Text-to-SQL Text-to-SQL
University of Alberta · Simon Fraser University · University of Waterloo +1

Apr 24, 2026J/K move · Enter open · S save

PExA: Parallel Exploration Agent for Complex Text-to-SQL

Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang, Sachith Sri Ram Kothur+2

University of California Los Angeles · Bloomberg

LLM Inference Efficiency Text-to-SQL LLM Agents

Open paper AskPDF arXiv

LLM-based agents for text-to-SQL often struggle with latency-performance trade-off, where performance improvements come at the cost of latency or vice versa. We reformulate text-to-SQL generation within the lens of software test coverage where the original query is prepared with a suite of test cases with simpler, atomic SQLs that are executed in parallel and together ensure semantic coverage of the original query. After iterating on test case coverage, the final SQL is generated only when enough information is gathered, leveraging the explored test case SQLs to ground the final generation. We validated our framework on a state-of-the-art benchmark for text-to-SQL, Spider 2.0, achieving a new state-of-the-art with 70.2% execution accuracy.

Abstract

Text-to-SQL

Explore similar work

PExA: Parallel Exploration Agent for Complex Text-to-SQL

AV-SQL: Decomposing Complex Text-to-SQL Queries with Agentic Views

FlexSQL: Flexible Exploration and Execution Make Better Text-to-SQL Agents