From Questions to Queries to Insights: A Schema-Grounded Agentic Workflow for Natural-Language Crash Data Analysis

Jingwen Zhu; Keshu Wu; Steven T. Parker; Bin Ran; David A. Noyce

cs.CLOct 26, 2026

From Questions to Queries to Insights: A Schema-Grounded Agentic Workflow for Natural-Language Crash Data Analysis

Authors: Jingwen Zhu, Keshu Wu, Steven T. Parker, Bin Ran, David A. Noyce

Organizations: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI, United States

Abstract

Transportation agencies maintain extensive crash databases to identify safety priorities, evaluate countermeasures, and guide resource allocation. Making these data accessible to practitioners requires flexible ways to translate analytical questions into reliable results. Existing dashboards support predefined analyses, while additional requests often depend on specialists who understand database queries, coded fields, and agency conventions. This study presents CrashDB-QA, a fully local, schema-grounded agentic workflow that enables natural-language analysis of crash databases. Specialized components interpret questions, generate and validate read-only SQL, execute queries, and explain the results. A shared knowledge base grounds query construction in the agency's data dictionary and safety-analysis conventions, while recorded intermediate outputs support inspection and auditing. An optional extension, CrashDB-QA+, checks the alignment among the question, query, and execution result and supplies evidence for iterative repair. We evaluate the workflow on a frozen snapshot of 4.4 million Wisconsin crash records using a new benchmark of 300 questions across ten analytical categories and nine local open-weight models. Domain grounding substantially improves query accuracy, and a code-specialized 7B model provides the best accuracy-latency balance among the tested models. Compositional analyses remain more difficult than routine retrieval, particularly comparisons between groups or periods. Verification improves performance when the Judge uses authoritative field definitions and the Refiner receives execution evidence, whereas additional checking alone offers no consistent benefit. These findings demonstrate the potential of locally deployed, domain-grounded workflows to broaden access to crash analysis and identify the reasoning and verification capabilities needed for reliable use.

Figures & tables

Figure 1: CrashDB-QA architecture. The six-component core connects question interpretation, grounded query construction, controlled execution, and result explanation. The optional CrashDB-QA+ extension, shown in orange, adds verification and iterative repair.

(a) Column entries $\mathcal{K}_{\mathrm{col}}$
Column $c$	Description $D_{c}$	Stored values $V_{c}$	Associated words $W_{c}$
CNTYNAME	County of the crash	$72$ names, e.g. 'MILWAUKEE'	county , counties
ALCFLAG	Police suspected alcohol use	'Y' , 'UNKN' , or NULL	alcohol , drunk , OWI
CYCLFLAG	Motorcycle involvement	'Y' or NULL	motorcycle , motorbike
TOTFATL	Number of fatalities	Integer	fatalities , deaths , killed
(b) Conventions $\mathcal{K}_{\mathrm{conv}}$

Table 1: Examples of column knowledge and analytical conventions in

\mathcal{K}

Panel of Table 2 — Table 2: Roles and prompts of the four language-model agents. Role instructions are quoted in abridged form.

Figure 2: Example of execution-informed refinement. An admissible query initially ranks counties by crash count. The Judge identifies the mismatch with the requested severity-weighted measure, and refinement produces a revised query that passes verification on the second iteration. The dashed red path marks the refinement loop.

Code	Category	Representative question
Tier 1: retrieval through standard filtering and aggregation
C1	Descriptive count	How many alcohol-related crashes occurred in Dane County in 2023?
C2	Aggregation / breakdown	How many crashes occurred in each county in 2023?
C3	Ranking / top- $N$	Which five counties had the most crashes in 2023?
C4	Temporal trend	How has the number of fatal crashes changed year over year?
C5	Spatial / location	How many crashes occurred at each intersection type in Dane County?

Table 3: Representative questions across the ten benchmark categories.

Figure 3: Benchmark taxonomy and representative executable-but-wrong errors. Categories C1–C5 emphasize standard retrieval and aggregation, while C6–C10 require additional composition of conditions, groupings, or derived quantities.

		Tier 1: Retrieval					Tier 2: Reasoning
Model	Regime	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	Overall	Inv.
Qwen-2.5-Coder 7B	code	96.7	86.7	93.3	70.0	90.0	90.0	73.3	86.7	43.3	76.7	80.7	3
Qwen-2.5 7B	general	96.7	80.0	83.3	70.0	90.0	96.7	63.3	90.0	43.3	50.0	76.3	13
DeepSeek-R1 8B	reasoning	100.0	66.7	73.3	76.7	83.3	70.0	86.7	66.7	33.3	76.7	73.3	37
DeepSeek-Coder 6.7B	code	96.7	70.0	76.7	70.0	90.0	56.7	73.3	60.0	26.7	63.3	68.3	24
Llama-3.1 8B	general	100.0	73.3	70.0	60.0	86.7	70.0	70.0	83.3	20.0	43.3	67.7	10

Table 4: Execution accuracy of the grounded CrashDB-QA core by model and analytical category (%). Each category contains

30

questions, for

300

questions in total. Inv. denotes the number of questions producing non-executable queries.

Figure 4: Performance of nine local models on the

300

-question benchmark. (a) Question outcomes. (b) Accuracy on the retrieval and reasoning tiers, with overall accuracy shown at right.

Figure 5: Category-level accuracy by model, ordered by overall performance. Each circle shows retrieval categories on the right and reasoning categories on the left; concentric rings indicate

25

-percentage-point increments.

Figure 6: Stage-level latency across local models. Ribbon widths represent time attributed to pipeline stages before and after database execution.

Panel of Figure 7 — Figure 7: Execution accuracy versus (a) parameter count and (b) median end-to-end latency on a logarithmic scale. Colors identify models, and marker shapes identify training regimes.

Judge design	TP	FP	FN	TN	Precision	Recall	False-accept
Original Judge	225	33	17	22	$0.872$	$0.930$	$0.600$
Dictionary-grounded Judge	228	31	14	24	$0.880$	$0.942$	$0.564$
Evidence-checklist variant	239	51	3	4	$0.824$	$0.988$	$0.927$

Table 5: RQ1: Judge verdicts against execution-match labels for the same

297

executable core outputs. TP and FP are pass verdicts on correct and incorrect outputs; FN and TN are fail verdicts on those outputs. False-accept rate is FP/(FP+TN).

Figure 8: RQ1: Comparison of Judge designs on

297

executable outputs. (a) Verdicts on the

55

incorrect and

242

correct outputs. (b) False-accept rate versus recall; lower false acceptance and higher recall are preferable.

Configuration	Judge	Feedback	Acc. (%)	Repairs	Regressions	$p$
Core only	—	—	$80.7$	—	—	—
Pre-registered configurations
Rule checks	rules only	critique	$81.0$	$1$	$0$	$1$
LLM Judge	LLM	critique	$80.7$	$2$	$2$	$1$
Voting, $k{=}5$ , + rules	LLM + rules	critique	$75.0$	$6$	$23$	$0.002$
Diagnostic improvements: deterministic heuristics disabled

Table 6: RQ2: Verification configurations compared with the CrashDB-QA core on

300

questions. All loop configurations use

N=3

and one greedy candidate unless stated otherwise. Repairs and regressions are counted relative to the core;

p

denotes the unadjusted exact McNemar test.

Figure 9: RQ2: Effects of verification and refinement. (a) Final execution accuracy, with the core shown as a dashed reference line. (b) Repairs and regressions relative to the core.

Data Analysis Agents

Data analysis agents use language models and tools to explore datasets, run analyses, and produce answers or reports grounded in evidence.

See all 106 papers

May 20, 2026

Broadening Access to Transportation Safety Data with Generative AI: A Schema-Grounded Framework for Spatial Natural Language Queries

Transportation safety analysis requires integrating crash records, roadway attributes, and geospatial data through GIS-based workflows, but access remains uneven across agencies and community stakeholders. Technical prerequisites create a gap between analytical tools central to safety planning and the practitioners able to use them. Local agencies, school committees, and residents may have safety concerns but limited capacity to retrieve, filter, map, and analyze relevant data. Generative AI offers a way to narrow this divide, but its public-sector use raises questions about reliability, reproducibility, and governance. This paper presents a schema-grounded natural language interface for transportation safety analysis, using a large language model (LLM) to interpret user intent while preserving deterministic, reviewable execution against an authoritative database. User queries are translated into structured semantic frames, validated by a rule-based layer, compiled into a typed directed acyclic graph of spatial operations, and executed against a PostGIS database. This bounded design separates language interpretation from deterministic execution, keeping results reproducible and schema-grounded while removing access barriers. The framework is evaluated using a statewide Massachusetts transportation safety database integrating crash records, roadway attributes, and geospatial layers including schools, bus stops, crosswalks, and municipal boundaries. All queries executed successfully; the validation layer corrects errors in 29% of evaluation queries, reflecting the gap between flexible natural language and strict schema-grounded requirements. The results suggest that combining natural language accessibility with deterministic execution is a practical direction for broadening access to transportation safety data, with implications for trustworthy AI in public-sector planning.

Mahdi Azhdari, Eric J. Gonzales

Department of Civil and Environmental Engineering, University of Massachusetts Amherst, Amherst, Massachusetts, USA

Grounded Text Generation

Ask PDF

Oct 2, 2026

A Governed Agent Architecture for Natural-Language Dashboard Generation over Enterprise DataBrain

We present a DataBrain-specific prototype that turns natural-language analytic requests into inspectable dashboards. An installable Codex plugin and standalone Agent Runtime combine schema-constrained model decisions with typed tools, persistent state, and deterministic Hooks for approval, audit, checkpointing, recovery, and failure handling. The pipeline resolves entities, discovers metadata, enforces read-only SQL, composes dashboards, and applies static checks, dynamic preflight, and browser inspection. The model proposes actions while deterministic software controls execution and records state transitions. We evaluate the workflow on frozen real-DataBrain tasks and controlled Hook faults. Strict success was 6/8 on metadata and SQL tasks: metadata selection passed 4/4, all four SQL tasks met semantic criteria, and 2/4 met the exact output-column contract. The final release passed 4/4 single-panel dashboard tasks, one two-panel task, and one existing-dashboard refinement; a parameterised task exceeded its step limit. All ten fault scenarios met their specified outcomes without unapproved external side effects. Model inference accounted for over 97% of observed runtime in every reported group. These small, DataBrain-specific results do not establish production readiness, general text-to-SQL accuracy, or an efficiency advantage over manual dashboard construction.

Yiou Wu, Zezhi Tang, Ningwei Bai +1

Department of Computer Science, University College London, London, United Kingdom

Data Analysis Agents LLM Agent Orchestration Enterprise Analytics LLM Agent Reliability Text-to-SQL

Ask PDF

Sep 19, 2026

An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database

Non-technical stakeholders frequently cannot write the SQL needed to extract insights from operational databases. We built and evaluated a Text-to-SQL agent that closes this gap end to end: a six-node LangGraph StateGraph checks question relevance, fetches the live schema, generates PostgreSQL, validates it with a dry run, retries on failure, executes the query, and narrates the result set in plain English. The agent uses prompt engineering only; no model was fine-tuned. We evaluated it on the Chicago Crime dataset (approximately 8.5 million records, 22 attributes) against a hand-built benchmark of 100 natural language questions with ground-truth SQL, stratified into 30 Easy, 40 Medium and 30 Hard items. Comparing two prompt revisions of the same agent, the revised system (V2) reached a Valid SQL Rate of 93% (from 87%), an Execution Accuracy of 60% under a hybrid relational equivalence metric (from 47%; 19% from 12% under strict JSON matching), and a mean Synthesis Quality of 4.34 out of 5 (from 3.91). The single largest driver was removing a LIMIT 10 instruction from the system prompt, which had been truncating multi-row answers. Error analysis attributes the residual failures to relevance-checker false rejections, ambiguous question semantics, and free-tier API rate limits rather than to the language generation step. We report no comparison against an external baseline system or a public benchmark; the study is a single-model engineering evaluation.

Vigneshwar Ravi Rao, Rupesh Swarnakar, Fayeq Jeelani Syed†

Luddy School of Informatics, Computing and Engineering, Indiana University, Indianapolis, Indiana, USA

Text-to-SQL Tool-Using Language Agents Text-to-SQL Evaluation LLM Agent Evaluation

Ask PDF

Abstract

Data Analysis Agents

Explore similar work

Broadening Access to Transportation Safety Data with Generative AI: A Schema-Grounded Framework for Spatial Natural Language Queries

A Governed Agent Architecture for Natural-Language Dashboard Generation over Enterprise DataBrain

An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database