Verifying Text-to-SQL Queries under Changing Business Semantics

Genliang Zhu; Xiaobo Chen; Ziyuan Wang; Chu Wang

cs.AIOct 30, 2026

Verifying Text-to-SQL Queries under Changing Business Semantics

Authors: Genliang Zhu, Xiaobo Chen, Ziyuan Wang, Chu Wang

Organizations: Accentrust, Vancouver, Canada · Georgia Institute of Technology, School of Computer Science, Atlanta, Georgia, USA · University of Massachusetts Amherst, Amherst, Massachusetts, USA · Duke University, Durham, North Carolina, USA ·

Abstract

In text-to-SQL and AI-agent analytics, successful SQL execution does not establish semantic correctness under changing business definitions. We study revalidation of fixed queries after changes to refund attribution, tax treatment, eligible populations and exchange-rate dates. An executable verification profile binds the requested revision and knowledge cutoff to legal counterexample data and exact monetary comparison. A preregistered confirmation on 32 newly authored revision families and two local producer bundles retains 192 query-definition cells: 144 nonconforming, 36 finite-suite agreeing and 12 unresolved. Of the nonconforming cells, 63 agree on the current snapshot. At an eight-ledger budget, obligation-guided checking detects 137 discrepancies versus 127 for generic ordering, a paired difference of 6.9 percentage points (family bootstrap 95% interval [2.1, 13.6]; exact family sign-flip p=0.0156). Every ledger arm rejects zero of the 36 agreeing cells. Current-obligation and equal-weight scoring match the guided result, identifying coverage rather than revision-specific priority as the observed source of benefit. Controlled assays, a natural-query pilot and repeated samples provide separate mechanism and robustness evidence. A legal discrepancy refutes conformance; agreement remains restricted to the recorded suite. The study establishes an auditable business-semantic revalidation profile within its authored schema and local-model scope.

Figures & tables

Claim	Executed evidence	Interpretation
Revision-bound conformance	Separate policy evaluator, legality checks and old/new query–definition matrix	Legal discrepancy refutes the requested profile; passing tests do not prove full-domain conformance.
Coverage component	Controlled decomposition; natural-query pilot and repeated samples	Current-only/equal-weight results distinguish coverage from priority weighting.
Local confirmation	32 new families, two bundles, 192 cells; eight-ledger paired comparison	137/144 versus 127/144 detection; authored-schema/local-bundle scope.
Native support	Original ParSEval, PV-SQL and MetricFlow qualification paths	Support measurements, not an effectiveness ranking of complete systems.

Table 1. Claims, executed evidence and interpretation. Qualification and future designs do not enlarge the empirical denominator.

Work	Established object	Obligation here
EntSQL / Spider 2.0	Enterprise knowledge and workflows	Explicit revision interventions
Metric-layer agents	Governed representation and compilation	Strong native compilation path
XData / ParSEval	Distinguishing test databases	Same constraints and budget
SpotIt / SpotIt+	Verification and constraints	Temporal and business controls
Cosette / VeriEQL	Equivalence and bounded reasoning	Explicit verification scope
PV-SQL / SQLDriller	Checks, probing, and correction	Fixed checks versus full repair

Table 2. Closest-work boundaries and evaluation obligations. Established components are credited rather than presented as new foundations.

Panel of Figure 1 — Figure 1. Primary labels are frozen independently before arm execution and remain hidden. The final replay checks frozen decisions without altering those labels or guiding search. Approved request and revision lead to definition differences, legal ledgers, and read-only execution. Independent replay subsequently checks the frozen operational decision; primary labels were independently frozen before this path.

Control	Detected	Rejected	Unresolved	SQL calls	Seconds
Current snapshot only	0/10	0/14	0	24	0.963
Generic legal bank order	6/10	0/14	1	192	7.525
Definition-guided bank order	10/10	0/14	0	192	7.810
Current-obligation cover	10/10	0/14	0	192	8.100
Equal-weight revision cover	10/10	0/14	0	192	7.602

Table 3. Authored controlled assay. Detection uses 10 finite-suite unequal queries; rejection is evaluated on 14 finite-suite agreement queries, not globally proved queries. Time includes selection and all operational executions on this host. The current-snapshot control uses one ledger; the other controls use eight.

Ledgers	Generic	Guided	Current cover	Equal weight	SQL calls
1	0/10	0/10	0/10	0/10	24
2	3/10	8/10	8/10	8/10	48
4	4/10	10/10	10/10	10/10	96
8	6/10	10/10	10/10	10/10	192

Table 4. Nested operational execution prefixes on the same development queries. Each detection denominator is 10; SQL calls are per arm. This is a budget manipulation check, not independent confirmation. All failures and full eight-ledger costs remain recorded.

Definition difference	Current ledger	Legal witness
Sales / receipts	100/100	100/0
Refund period	100/100	100/80
Tax treatment	100/100	120/100
FX date	150/150	60/180

Table 5. Independently labeled examples: candidate SQL / requested metric, in base-currency minor units. Witnesses come from full-suite scientific labeling, not necessarily the eight-ledger operational selection.

Transition/control	O/N	N/O
Sales / receipts	15	15
Refund attribution	11	11
Tax treatment	20	20
FX date	3	3
Eligible statuses	3	3
Combined changes	21	21

Table 6. Unequal results among 38 legal ledgers per cell. Columns denote query/definition versions (O: old; N: new). Zero means agreement in this finite suite, not domain-wide equivalence. The legal-data profile is unchanged across versions.

Model	Cell	Cells	Nonconf.	D0-latent	Agree	Unres.
gpt-oss-20b	old $\to$ v1	24	20	10	3	1
gpt-oss-20b	old $\to$ v2	24	22	9	1	1
gpt-oss-20b	new $\to$ v2	24	17	10	6	1
gemma-4-12b	old $\to$ v1	24	14	4	8	2
gemma-4-12b	old $\to$ v2	24	20	6	2	2
gemma-4-12b	new $\to$ v2	24	11	4	13	0

Table 7. Independent finite-suite labels of final natural SQL on the 24 frozen test families (374 legal ledgers per family; test families were not used for prompt tuning). “D0-latent” counts nonconforming cells whose value agrees with the definition on the current snapshot.

Method	old $\to$ v1	old $\to$ v2	new $\to$ v2	All	False rej.
Nonconforming / agreeing cells	34	42	28	104	33
Current snapshot $D_{0}$	20	27	14	61	0
Generic legal (8)	30	42	23	95	0
Obligation-guided (8)	33	42	26	101	0
Current-obligation (8)	33	42	26	101	0
Equal-weight (8)	33	42	26	101	0

Table 8. Detection of labeled nonconforming cells by cell type and false rejection of finite-suite-agreement cells (test split). Ledger arms never reject an agreeing cell by construction of replayable evidence; judges can.

Reviewer	Detected	False rej.	Abstain	Dev-selected
gpt-oss-20b, base prompt	103	29	0
gpt-oss-20b, checklist	48	4	0
gpt-oss-20b, checklist + $D_{0}$ execution	68	1	0	yes
gemma-4-12b, base prompt	104	32	0
gemma-4-12b, checklist	100	25	4	yes (tie)
gemma-4-12b, checklist + $D_{0}$ execution	102	24	3

Table 9. LLM-judge variants on the 144 test cells (104 nonconforming, 33 agreeing). Variants were tuned on development families only; the primary variant per judge was fixed by a written dev rule (maximum detection minus false-rejection rate, ties to the simpler prompt) before test verdicts were run. Ledger checks are shown for reference.

Check	Detected / 144	False rejection / 36
Current snapshot	81	0
Generic legal order	127	0
Obligation-guided order	137	0
Current-obligation scoring	137	0
Equal-weight scoring	137	0

Table 10. Preregistered local confirmation: 192 cells from 32 new revision families and two model bundles. Detection is conditional on 144 finite-suite nonconforming cells; false rejection is evaluated on 36 agreeing cells. Twelve unresolved cells remain in the full denominator.

Stratum	Snapshot	Generic	Coverage
gpt-oss-20b	42/80	70/80	78/80
gemma-4-12b	39/64	57/64	59/64
tax	11/18	17/18	18/18
refundAttribution	6/19	14/19	18/19
fxDate	12/22	20/22	20/22
statuses	11/17	16/17	17/17

Table 11. Confirmation stratified by producer and revision axis. Entries are detected/nonconforming cells; revision axes are family attributes, not inferred SQL error causes.

Order	1	2	4	8
Snapshot	81	81	81	81
Generic	81	99	113	127
Obligation coverage	81	118	129	137
Current-obligation	81	117	129	137
Equal-weight	81	118	129	137

Table 12. Post-hoc detection at prefixes of the already executed confirmation orders. Counts use the same 144 nonconforming cells; no new SQL is executed.

Arm	Selection (s)	Execution (s)	Comparisons
Snapshot	0.00	11.15	192
Generic	0.08	87.94	1536
Obligation coverage	2.00	87.89	1536
Current-obligation	1.73	88.25	1536
Equal-weight	1.98	88.33	1536

Table 13. Recorded confirmation checking work. All selected executions were charged, including those after a discrepancy. Times are shared-host sums, not randomized speed comparisons.

Object	Frozen/executed record	Consequence
H19/H25 definitions	Registration documents pre-freeze corrections after no compatible assigned snapshot was found	Final 19 coinciding/13 separating strata; no outcome-conditioned replacement is reported.
Protocol and generation	Protocol creation: 12 Oct., 14:36:40; first producer START: 16:59:13	Recorded ordering places protocol before collected generation.
Candidate bank	Registered text: 74+300 ledgers; executed pool: 81+300	A literal size discrepancy; all arms share the 81-candidate pool and eight-ledger limit.
Labels and checking	Label digest recorded before arm START at 17:30:55	Labels remain independent of selection and all 192 primary cells remain in scope.
Editorial reaggregation	Added 13 Oct. from existing records only	Stratification, prefixes and leave-one-family-out are post hoc; registered inference is unchanged.

Table 14. Confirmation chronology. Times are recorded local UTC

-07{:}00

timestamps, not independent timestamp attestations. Source bindings and all original records are retained.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Field	Meaning
Metric / revision	Explicit identifier and strictly matching string or integer revision
Basis	Sales recognition or gross payment receipts
Refund attribution	Posting period or original sale period for sales; posting for receipts
Tax mode	Net or gross sales; no inferred partial-payment tax decomposition
FX date	Original sale date or contribution event date
Population	Explicit eligible sale statuses

Appendix

Table 15. Implemented policy fields. Revision and request applicability are independent of representation-format versions.

Text-to-SQL Evaluation

See all 29 papers

What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study
Jul 7, 2026Robert RichardsonLLM Answer Verification Selective Prediction
Affiliation
Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL
Sep 22, 2026Jiamiao Liu, Dewen Qiao, Yu Zhang +1Probability Calibration Benchmark Auditing
Department of Information, Xinqiao Hospital Army Medical University (Third Military Medical University) Chongqing 400037, China
VeriPIE: Checkable Query Plans and Evidence-Guided Selection for Text-to-SQL
Oct 12, 2026Piyush Jha, Cruise Song, Zhengyu Li +1Execution-Guided Text-to-SQL Text-to-SQL
Georgia Tech

Jul 7, 2026J/K move · Enter open · S save

What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

Robert Richardson

Affiliation

LLM Answer Verification Selective Prediction Text-to-SQL LLM Uncertainty Estimation Text-to-SQL Evaluation

Open paper AskPDF arXiv

Evaluating uncertainty in AI-generated SQL queries requires estimating whether a query is correct, where correct means it executes to the same result as a human-written reference. We study which signals predict correctness on hard multi-table text-to-SQL, using AUROC to measure how well each ranks correct queries above incorrect ones. On BIRD and Spider, black-box signals such as string, structural, and execution self-consistency, a schema-relevance score, and query executability all fall between about 0.61 and 0.68 AUROC, with string self-consistency strongest at 0.675; white-box log-probability is similar (0.67). The signals that move past this ceiling are verification-based: an LLM judge scores from 0.72 (GPT-4o-mini) to 0.78 (Claude). Judges from different providers make different errors, so a two-provider ensemble reaches 0.82 AUROC with a well-calibrated probability (expected calibration error 0.03) and supports useful abstention frontiers (for example, answering 27% of questions at 24% selective risk) where self-consistency offers no valid low-risk subset. The pattern holds across two benchmarks, two generators, and two judge providers. We also ask whether a verifier can be trained. Fine-tuned verifiers, both encoder and generative, reach about 0.77 to 0.79 AUROC in-distribution but fall to about 0.66 on unseen schemas; scaling to 7B, adding schema diversity, distilling a strong judge's rationales, and cross-benchmark training all fail to close that gap. Cross-schema transfer appears to track model scale and reasoning rather than fine-tuning. In practice, correctness uncertainty for text-to-SQL lives in reasoning-based signals: a fine-tuned verifier is a good in-domain tool, but a verifier that generalizes across schemas currently means a large frozen reasoning model.

Abstract

Text-to-SQL Evaluation

Explore similar work

What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL

VeriPIE: Checkable Query Plans and Evidence-Guided Selection for Text-to-SQL