Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider

Raghu Venkat; Tricha Anjali

cs.AIOct 12, 2026

Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider

Authors: Raghu Venkat, Tricha Anjali

Organizations: Numberz.ai Inc., Alpharetta, Georgia, USA

Abstract

Conversational analytics lets people ask questions of records spread across many systems and get answers in plain language; it is useful only if each answer can be trusted and explained. A production system of this kind attaches a confidence score to every answer it writes over a database and uses it to show the answer, show it with a warning, or withhold it. The score's weights, caps and thresholds were set by design. This paper audits that design. On the 1,534 questions of BIRD dev, answered by GPT-5.1 with a benchmark-adapted production prompt and five samples per question, the designed score separates right from wrong answers weakly (AUROC 0.624, 95% CI 0.595-0.652) and withholds 35.7% of correct answers. Nearly all of its ranking signal comes from agreement among the samples, and most of it is lost where agreement is combined by a minimum with a checklist that drops to zero on any unsupported claim, including claims in commentary. We froze a rule in which agreement with the shown answer is the score and deterministic checks act as vetoes and caps, and pre-registered a test on Spider dev before any Spider answer existed. The primary hypothesis held: AUROC rose from 0.566 to 0.725 (paired difference +0.159, 95% CI +0.097 to +0.221), and correct answers withheld fell from 49.0% to 8.8% with PASS accuracy 95.7%. Under one matching policy the choice of agreement statistic matters little; the gain comes from how agreement enters the score. Whether added checks help depends on how correctness is defined, while they always cost unqualified coverage. A pre-registered empty-result check did not meet its effect-size criterion on Spider. Correctness here means the core answer; labels are model-judged, and a blinded two-person audit, not yet complete, will be reported in a revised version.

Figures & tables

Panel of Figure 1 — Figure 1: What changes. The records and the people stay the same; the question stops needing a script. Panel 3 is the requirement this paper is concerned with: every number traceable, and the system willing to say when it cannot answer.

Panel of Figure 2 — Figure 2: Many kinds of record, one form. Each source is read by the reader suited to it, and every value is held with its time, its evidence class, where it was found and what it was computed from. Disagreements between sources are settled by a precedence the organisation declares, with the losing value kept visible.

Panel of Figure 3 — Figure 3: The setting. Records from several systems and the owners’ recorded definitions feed a model that writes queries. Checks outside the model grade the answer and its trace; a gate shows the answer, shows it with a warning, or withholds it with a reason. This paper measures the checks and the gate.

Score	AUROC (95% CI)	pass accuracy	pass share	Correct withheld	Wrong withheld
Designed	0.624 (0.595–0.652)	77.9%	58.5%	35.7%	43.7%
v2 rule	0.694 (0.666–0.723)	77.4%	80.0%	14.0%	25.4%
Agreement alone	0.721 (0.692–0.748)	81.3%	78.9%	6.3%	32.9%
Agreement-first	0.730 (0.702–0.756)	83.6%	62.0%	8.3%	37.3%
v3 (empty result)	0.740 (0.713–0.768)	84.6%	61.3%	8.3%	37.3%
Fitted (exploratory)	0.718 (0.685–0.749)	86.0%	53.3%	17.7%	51.5%

Table 1: BIRD dev, 29 September run, 1,515 labelled answers. The rules below the line were derived from these answers and are in-sample here.

Change	AUROC	pass acc.	pass share
None (designed)	0.624	77.9%	58.5%
Drop GROUNDED	0.631	82.5%	48.2%
Drop SCHEMA	0.625	79.9%	56.1%
Drop COMPLETE	0.624	78.1%	58.4%
Drop SAFETY	0.555	75.5%	62.2%
SAFETY without $L_{2}$	0.572	75.7%	61.8%

Table 2: Single changes to the designed score, recomputed from saved components (BIRD, 29 September run).

Error	Withheld	Not pass	Own check
Fake table	100.0%	100.0%	100.0%
Bad join	100.0%	100.0%	100.0%
Stand-in as value	100.0%	100.0%	100.0%
Wrong number	100.0%	100.0%	100.0%
Wrong time window	90.0%	95.0%	83.3%
Wrong aggregate	75.0%	81.7%	96.7%

Table 3: Planted errors, 60 per kind: share the designed score withheld, share not passed, and share where the check built for that error fired.

Score	AUROC (95% CI)	pass accuracy	pass share	Correct withheld	Wrong withheld
Designed	0.566 (0.503–0.626)	93.9%	49.7%	49.0%	54.8%
Agreement-first (primary)	0.725 (0.663–0.787)	95.7%	73.6%	8.8%	35.6%
v3 (empty result)	0.767 (0.707–0.821)	96.8%	70.7%	8.8%	35.6%
Agreement alone	0.749 (0.688–0.809)	95.5%	90.4%	3.7%	35.6%
Not pre-registered
v2 rule	0.682 (0.623–0.740)	93.2%	85.1%	13.7%	15.1%

Table 4: Spider dev, pre-registered confirmation, 983 labelled answers.

	Observed	Verdict
H1	AUROC difference +0.159 (95% CI +0.097 to +0.221); with truncated pairs counted as disagreeing, agreement-first AUROC 0.721	Pass
H2	correct withheld 8.8% vs 49.0%; pass accuracy 95.7% vs 93.9% (+1.8 points)	Pass
H3	empty result fired on 37 answers (3.8%); right 73.0% of the time vs 92.6% overall, 19.6 points below	Fail
H4	designed 0.566 $<$ agreement-first 0.725 $<$ agreement alone 0.749 $<$ v3 0.767	Not as predicted

Table 5: Pre-registered hypotheses on Spider dev.

Evaluation	$n$	Acc.	Designed	Agree-first	Difference (95% CI)
BIRD, 29 Sep (in-sample)	1,515	72.9%	0.624	0.730	+0.106 (+0.077 to +0.134)
BIRD, 29 Sep, rebuilt labels	1,441	82.6%	0.631	0.756	+0.124 (+0.089 to +0.158)
BIRD, rebuilt harness	1,424	82.5%	0.589	0.755	+0.165 (+0.129 to +0.204)
Spider (pre-registered)	983	92.6%	0.566	0.725	+0.159 (+0.097 to +0.221)

Table 6: AUROC of agreement-first against the designed score in every evaluation.

Evaluation	$L_{2}$ alone	$\min(L_{1},L_{2})$	Weighted mean	With caps (designed)
BIRD, 29 Sep	0.696	0.620	0.638	0.624
BIRD, rebuilt harness	0.715	0.583	0.606	0.589
Spider	0.725	0.565	0.584	0.566

Table 7: AUROC of the designed score rebuilt one step at a time.

Evaluation	Score	pass	flag	abstain
Spider	Designed	459 / 30	5 / 3	446 / 40
	Agreement alone	849 / 40	27 / 7	34 / 26
	Agreement-first	692 / 31	138 / 16	80 / 26
BIRD, rebuilt	Designed	593 / 98	24 / 17	558 / 134
	Agreement alone	970 / 107	95 / 35	110 / 107
	Agreement-first	752 / 61	289 / 74	134 / 114

Table 8: Gate decisions by correctness (right / wrong). flag answers are shown with a warning; abstain answers are withheld.

Panel of Figure 4 — Figure 4: Accuracy and coverage: accuracy of the answers shown as the threshold is lowered. Markers, and every vertex of the designed score’s curve, are attainable thresholds; the lines between them are visual guides.

	Designed		Agreement alone		Agreement-first
Evaluation	ECE	Brier	ECE	Brier	ECE	Brier
BIRD, 29 Sep	0.155	0.209	0.130	0.183	0.098	0.175
BIRD, rebuilt harness	0.157	0.166	0.097	0.134	0.093	0.133
Spider	0.187	0.125	0.057	0.066	0.088	0.077

Table 9: Expected calibration error (10 bins) and Brier score.

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 244 papers

What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study
Jul 7, 2026Robert RichardsonLLM Answer Verification Selective Prediction
Affiliation
Not One Belief: Answer Payloads, Persistence, and Dissociated Confidence Signals in Conversational Question Answering
Oct 26, 2026Duc-Vinh TranConversational QA Belief Updating in LLMs
School of Information and Communication Technology, Hanoi University of Science and Technology, Hanoi, 100000, Vietnam.
Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline
Sep 9, 2026Haowei Liu, Hsin-Tai Wu, Yi FangLLM-as-a-Judge LLM Auditing
Santa Clara University · Independent Researcher

Jul 7, 2026J/K move · Enter open · S save

What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

Robert Richardson

Affiliation

LLM Answer Verification Selective Prediction Text-to-SQL LLM Uncertainty Estimation Text-to-SQL Evaluation

Open paper AskPDF arXiv

Evaluating uncertainty in AI-generated SQL queries requires estimating whether a query is correct, where correct means it executes to the same result as a human-written reference. We study which signals predict correctness on hard multi-table text-to-SQL, using AUROC to measure how well each ranks correct queries above incorrect ones. On BIRD and Spider, black-box signals such as string, structural, and execution self-consistency, a schema-relevance score, and query executability all fall between about 0.61 and 0.68 AUROC, with string self-consistency strongest at 0.675; white-box log-probability is similar (0.67). The signals that move past this ceiling are verification-based: an LLM judge scores from 0.72 (GPT-4o-mini) to 0.78 (Claude). Judges from different providers make different errors, so a two-provider ensemble reaches 0.82 AUROC with a well-calibrated probability (expected calibration error 0.03) and supports useful abstention frontiers (for example, answering 27% of questions at 24% selective risk) where self-consistency offers no valid low-risk subset. The pattern holds across two benchmarks, two generators, and two judge providers. We also ask whether a verifier can be trained. Fine-tuned verifiers, both encoder and generative, reach about 0.77 to 0.79 AUROC in-distribution but fall to about 0.66 on unseen schemas; scaling to 7B, adding schema diversity, distilling a strong judge's rationales, and cross-benchmark training all fail to close that gap. Cross-schema transfer appears to track model scale and reasoning rather than fine-tuning. In practice, correctness uncertainty for text-to-SQL lives in reasoning-based signals: a fine-tuned verifier is a good in-domain tool, but a verifier that generalizes across schemas currently means a large frozen reasoning model.

Abstract

Confidence Estimation in Language Models

Explore similar work

What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

Not One Belief: Answer Payloads, Persistence, and Dissociated Confidence Signals in Conversational Question Answering

Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline