From Questions to Queries to Insights: A Schema-Grounded Agentic Workflow for Natural-Language Crash Data Analysis

Jingwen Zhu; Keshu Wu; Steven T. Parker; Bin Ran; David A. Noyce

cs.CLOct 26, 2026

From Questions to Queries to Insights: A Schema-Grounded Agentic Workflow for Natural-Language Crash Data Analysis

Authors: Jingwen Zhu, Keshu Wu, Steven T. Parker, Bin Ran, David A. Noyce

Organizations: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI, United States

Abstract

Transportation agencies maintain extensive crash databases to identify safety priorities, evaluate countermeasures, and guide resource allocation. Making these data accessible to practitioners requires flexible ways to translate analytical questions into reliable results. Existing dashboards support predefined analyses, while additional requests often depend on specialists who understand database queries, coded fields, and agency conventions. This study presents CrashDB-QA, a fully local, schema-grounded agentic workflow that enables natural-language analysis of crash databases. Specialized components interpret questions, generate and validate read-only SQL, execute queries, and explain the results. A shared knowledge base grounds query construction in the agency's data dictionary and safety-analysis conventions, while recorded intermediate outputs support inspection and auditing. An optional extension, CrashDB-QA+, checks the alignment among the question, query, and execution result and supplies evidence for iterative repair. We evaluate the workflow on a frozen snapshot of 4.4 million Wisconsin crash records using a new benchmark of 300 questions across ten analytical categories and nine local open-weight models. Domain grounding substantially improves query accuracy, and a code-specialized 7B model provides the best accuracy-latency balance among the tested models. Compositional analyses remain more difficult than routine retrieval, particularly comparisons between groups or periods. Verification improves performance when the Judge uses authoritative field definitions and the Refiner receives execution evidence, whereas additional checking alone offers no consistent benefit. These findings demonstrate the potential of locally deployed, domain-grounded workflows to broaden access to crash analysis and identify the reasoning and verification capabilities needed for reliable use.

Figures & tables

Figure 1: CrashDB-QA architecture. The six-component core connects question interpretation, grounded query construction, controlled execution, and result explanation. The optional CrashDB-QA+ extension, shown in orange, adds verification and iterative repair.

(a) Column entries $\mathcal{K}_{\mathrm{col}}$
Column $c$	Description $D_{c}$	Stored values $V_{c}$	Associated words $W_{c}$
CNTYNAME	County of the crash	$72$ names, e.g. 'MILWAUKEE'	county , counties
ALCFLAG	Police suspected alcohol use	'Y' , 'UNKN' , or NULL	alcohol , drunk , OWI
CYCLFLAG	Motorcycle involvement	'Y' or NULL	motorcycle , motorbike
TOTFATL	Number of fatalities	Integer	fatalities , deaths , killed
(b) Conventions $\mathcal{K}_{\mathrm{conv}}$

Table 1: Examples of column knowledge and analytical conventions in

\mathcal{K}

Panel of Table 2 — Table 2: Roles and prompts of the four language-model agents. Role instructions are quoted in abridged form.

Figure 2: Example of execution-informed refinement. An admissible query initially ranks counties by crash count. The Judge identifies the mismatch with the requested severity-weighted measure, and refinement produces a revised query that passes verification on the second iteration. The dashed red path marks the refinement loop.

Code	Category	Representative question
Tier 1: retrieval through standard filtering and aggregation
C1	Descriptive count	How many alcohol-related crashes occurred in Dane County in 2023?
C2	Aggregation / breakdown	How many crashes occurred in each county in 2023?
C3	Ranking / top- $N$	Which five counties had the most crashes in 2023?
C4	Temporal trend	How has the number of fatal crashes changed year over year?
C5	Spatial / location	How many crashes occurred at each intersection type in Dane County?

Table 3: Representative questions across the ten benchmark categories.

Figure 3: Benchmark taxonomy and representative executable-but-wrong errors. Categories C1–C5 emphasize standard retrieval and aggregation, while C6–C10 require additional composition of conditions, groupings, or derived quantities.

		Tier 1: Retrieval					Tier 2: Reasoning
Model	Regime	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	Overall	Inv.
Qwen-2.5-Coder 7B	code	96.7	86.7	93.3	70.0	90.0	90.0	73.3	86.7	43.3	76.7	80.7	3
Qwen-2.5 7B	general	96.7	80.0	83.3	70.0	90.0	96.7	63.3	90.0	43.3	50.0	76.3	13
DeepSeek-R1 8B	reasoning	100.0	66.7	73.3	76.7	83.3	70.0	86.7	66.7	33.3	76.7	73.3	37
DeepSeek-Coder 6.7B	code	96.7	70.0	76.7	70.0	90.0	56.7	73.3	60.0	26.7	63.3	68.3	24
Llama-3.1 8B	general	100.0	73.3	70.0	60.0	86.7	70.0	70.0	83.3	20.0	43.3	67.7	10

Table 4: Execution accuracy of the grounded CrashDB-QA core by model and analytical category (%). Each category contains

30

questions, for

300

questions in total. Inv. denotes the number of questions producing non-executable queries.

Figure 4: Performance of nine local models on the

300

-question benchmark. (a) Question outcomes. (b) Accuracy on the retrieval and reasoning tiers, with overall accuracy shown at right.

Figure 5: Category-level accuracy by model, ordered by overall performance. Each circle shows retrieval categories on the right and reasoning categories on the left; concentric rings indicate

25

-percentage-point increments.

Figure 6: Stage-level latency across local models. Ribbon widths represent time attributed to pipeline stages before and after database execution.

Panel of Figure 7 — Figure 7: Execution accuracy versus (a) parameter count and (b) median end-to-end latency on a logarithmic scale. Colors identify models, and marker shapes identify training regimes.

Judge design	TP	FP	FN	TN	Precision	Recall	False-accept
Original Judge	225	33	17	22	$0.872$	$0.930$	$0.600$
Dictionary-grounded Judge	228	31	14	24	$0.880$	$0.942$	$0.564$
Evidence-checklist variant	239	51	3	4	$0.824$	$0.988$	$0.927$

Table 5: RQ1: Judge verdicts against execution-match labels for the same

297

executable core outputs. TP and FP are pass verdicts on correct and incorrect outputs; FN and TN are fail verdicts on those outputs. False-accept rate is FP/(FP+TN).

Figure 8: RQ1: Comparison of Judge designs on

297

executable outputs. (a) Verdicts on the

55

incorrect and

242

correct outputs. (b) False-accept rate versus recall; lower false acceptance and higher recall are preferable.

Configuration	Judge	Feedback	Acc. (%)	Repairs	Regressions	$p$
Core only	—	—	$80.7$	—	—	—
Pre-registered configurations
Rule checks	rules only	critique	$81.0$	$1$	$0$	$1$
LLM Judge	LLM	critique	$80.7$	$2$	$2$	$1$
Voting, $k{=}5$ , + rules	LLM + rules	critique	$75.0$	$6$	$23$	$0.002$
Diagnostic improvements: deterministic heuristics disabled

Table 6: RQ2: Verification configurations compared with the CrashDB-QA core on

300

questions. All loop configurations use

N=3

and one greedy candidate unless stated otherwise. Repairs and regressions are counted relative to the core;

p

denotes the unadjusted exact McNemar test.

Figure 9: RQ2: Effects of verification and refinement. (a) Final execution accuracy, with the core shown as a dashed reference line. (b) Repairs and regressions relative to the core.

Data Analysis Agents

Data analysis agents use language models and tools to explore datasets, run analyses, and produce answers or reports grounded in evidence.

See all 106 papers

Broadening Access to Transportation Safety Data with Generative AI: A Schema-Grounded Framework for Spatial Natural Language Queries
May 20, 2026Mahdi Azhdari, Eric J. GonzalesGrounded Text Generation
Department of Civil and Environmental Engineering, University of Massachusetts Amherst, Amherst, Massachusetts, USA
A Governed Agent Architecture for Natural-Language Dashboard Generation over Enterprise DataBrain
Oct 2, 2026Yiou Wu, Zezhi Tang, Ningwei Bai +1Data Analysis Agents LLM Agent Orchestration
Department of Computer Science, University College London, London, United Kingdom
An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database
Sep 19, 2026Vigneshwar Ravi Rao, Rupesh Swarnakar, Fayeq Jeelani Syed†Text-to-SQL Tool-Using Language Agents
Luddy School of Informatics, Computing and Engineering, Indiana University, Indianapolis, Indiana, USA

May 20, 2026J/K move · Enter open · S save

Broadening Access to Transportation Safety Data with Generative AI: A Schema-Grounded Framework for Spatial Natural Language Queries

Mahdi Azhdari, Eric J. Gonzales

Department of Civil and Environmental Engineering, University of Massachusetts Amherst, Amherst, Massachusetts, USA

Grounded Text Generation

Open paper AskPDF arXiv

Transportation safety analysis requires integrating crash records, roadway attributes, and geospatial data through GIS-based workflows, but access remains uneven across agencies and community stakeholders. Technical prerequisites create a gap between analytical tools central to safety planning and the practitioners able to use them. Local agencies, school committees, and residents may have safety concerns but limited capacity to retrieve, filter, map, and analyze relevant data. Generative AI offers a way to narrow this divide, but its public-sector use raises questions about reliability, reproducibility, and governance. This paper presents a schema-grounded natural language interface for transportation safety analysis, using a large language model (LLM) to interpret user intent while preserving deterministic, reviewable execution against an authoritative database. User queries are translated into structured semantic frames, validated by a rule-based layer, compiled into a typed directed acyclic graph of spatial operations, and executed against a PostGIS database. This bounded design separates language interpretation from deterministic execution, keeping results reproducible and schema-grounded while removing access barriers. The framework is evaluated using a statewide Massachusetts transportation safety database integrating crash records, roadway attributes, and geospatial layers including schools, bus stops, crosswalks, and municipal boundaries. All queries executed successfully; the validation layer corrects errors in 29% of evaluation queries, reflecting the gap between flexible natural language and strict schema-grounded requirements. The results suggest that combining natural language accessibility with deterministic execution is a practical direction for broadening access to transportation safety data, with implications for trustworthy AI in public-sector planning.

Abstract

Data Analysis Agents

Explore similar work

Broadening Access to Transportation Safety Data with Generative AI: A Schema-Grounded Framework for Spatial Natural Language Queries

A Governed Agent Architecture for Natural-Language Dashboard Generation over Enterprise DataBrain

An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database