Conversational analytics lets people ask questions of records spread across many systems and get answers in plain language; it is useful only if each answer can be trusted and explained. A production system of this kind attaches a confidence score to every answer it writes over a database and uses it to show the answer, show it with a warning, or withhold it. The score's weights, caps and thresholds were set by design. This paper audits that design. On the 1,534 questions of BIRD dev, answered by GPT-5.1 with a benchmark-adapted production prompt and five samples per question, the designed score separates right from wrong answers weakly (AUROC 0.624, 95% CI 0.595-0.652) and withholds 35.7% of correct answers. Nearly all of its ranking signal comes from agreement among the samples, and most of it is lost where agreement is combined by a minimum with a checklist that drops to zero on any unsupported claim, including claims in commentary. We froze a rule in which agreement with the shown answer is the score and deterministic checks act as vetoes and caps, and pre-registered a test on Spider dev before any Spider answer existed. The primary hypothesis held: AUROC rose from 0.566 to 0.725 (paired difference +0.159, 95% CI +0.097 to +0.221), and correct answers withheld fell from 49.0% to 8.8% with PASS accuracy 95.7%. Under one matching policy the choice of agreement statistic matters little; the gain comes from how agreement enters the score. Whether added checks help depends on how correctness is defined, while they always cost unqualified coverage. A pre-registered empty-result check did not meet its effect-size criterion on Spider. Correctness here means the core answer; labels are model-judged, and a blinded two-person audit, not yet complete, will be reported in a revised version.
Figures & tables
Figure 1: What changes. The records and the people stay the same; the question stops needing a script. Panel 3 is the requirement this paper is concerned with: every number traceable, and the system willing to say when it cannot answer.
Figure 2: Many kinds of record, one form. Each source is read by the reader suited to it, and every value is held with its time, its evidence class, where it was found and what it was computed from. Disagreements between sources are settled by a precedence the organisation declares, with the losing value kept visible.
Figure 3: The setting. Records from several systems and the owners’ recorded definitions feed a model that writes queries. Checks outside the model grade the answer and its trace; a gate shows the answer, shows it with a warning, or withholds it with a reason. This paper measures the checks and the gate.
Score
AUROC (95% CI)
pass accuracy
pass share
Correct withheld
Wrong withheld
Designed
0.624 (0.595–0.652)
77.9%
58.5%
35.7%
43.7%
v2 rule
0.694 (0.666–0.723)
77.4%
80.0%
14.0%
25.4%
Agreement alone
0.721 (0.692–0.748)
81.3%
78.9%
6.3%
32.9%
Agreement-first
0.730 (0.702–0.756)
83.6%
62.0%
8.3%
37.3%
v3 (empty result)
0.740 (0.713–0.768)
84.6%
61.3%
8.3%
37.3%
Fitted (exploratory)
0.718 (0.685–0.749)
86.0%
53.3%
17.7%
51.5%
Table 1: BIRD dev, 29 September run, 1,515 labelled answers. The rules below the line were derived from these answers and are in-sample here.
Change
AUROC
pass acc.
pass share
None (designed)
0.624
77.9%
58.5%
Drop GROUNDED
0.631
82.5%
48.2%
Drop SCHEMA
0.625
79.9%
56.1%
Drop COMPLETE
0.624
78.1%
58.4%
Drop SAFETY
0.555
75.5%
62.2%
SAFETY without L2
0.572
75.7%
61.8%
Table 2: Single changes to the designed score, recomputed from saved components (BIRD, 29 September run).
Error
Withheld
Not pass
Own check
Fake table
100.0%
100.0%
100.0%
Bad join
100.0%
100.0%
100.0%
Stand-in as value
100.0%
100.0%
100.0%
Wrong number
100.0%
100.0%
100.0%
Wrong time window
90.0%
95.0%
83.3%
Wrong aggregate
75.0%
81.7%
96.7%
Table 3: Planted errors, 60 per kind: share the designed score withheld, share not passed, and share where the check built for that error fired.
Table 6: AUROC of agreement-first against the designed score in every evaluation.
Evaluation
L2 alone
min(L1,L2)
Weighted mean
With caps (designed)
BIRD, 29 Sep
0.696
0.620
0.638
0.624
BIRD, rebuilt harness
0.715
0.583
0.606
0.589
Spider
0.725
0.565
0.584
0.566
Table 7: AUROC of the designed score rebuilt one step at a time.
Evaluation
Score
pass
flag
abstain
Spider
Designed
459 / 30
5 / 3
446 / 40
Agreement alone
849 / 40
27 / 7
34 / 26
Agreement-first
692 / 31
138 / 16
80 / 26
BIRD, rebuilt
Designed
593 / 98
24 / 17
558 / 134
Agreement alone
970 / 107
95 / 35
110 / 107
Agreement-first
752 / 61
289 / 74
134 / 114
Table 8: Gate decisions by correctness (right / wrong). flag answers are shown with a warning; abstain answers are withheld.
Figure 4: Accuracy and coverage: accuracy of the answers shown as the threshold is lowered. Markers, and every vertex of the designed score’s curve, are attainable thresholds; the lines between them are visual guides.
Designed
Agreement alone
Agreement-first
Evaluation
ECE
Brier
ECE
Brier
ECE
Brier
BIRD, 29 Sep
0.155
0.209
0.130
0.183
0.098
0.175
BIRD, rebuilt harness
0.157
0.166
0.097
0.134
0.093
0.133
Spider
0.187
0.125
0.057
0.066
0.088
0.077
Table 9: Expected calibration error (10 bins) and Brier score.
Evaluating uncertainty in AI-generated SQL queries requires estimating whether a query is correct, where correct means it executes to the same result as a human-written reference. We study which signals predict correctness on hard multi-table text-to-SQL, using AUROC to measure how well each ranks correct queries above incorrect ones. On BIRD and Spider, black-box signals such as string, structural, and execution self-consistency, a schema-relevance score, and query executability all fall between about 0.61 and 0.68 AUROC, with string self-consistency strongest at 0.675; white-box log-probability is similar (0.67). The signals that move past this ceiling are verification-based: an LLM judge scores from 0.72 (GPT-4o-mini) to 0.78 (Claude). Judges from different providers make different errors, so a two-provider ensemble reaches 0.82 AUROC with a well-calibrated probability (expected calibration error 0.03) and supports useful abstention frontiers (for example, answering 27% of questions at 24% selective risk) where self-consistency offers no valid low-risk subset. The pattern holds across two benchmarks, two generators, and two judge providers. We also ask whether a verifier can be trained. Fine-tuned verifiers, both encoder and generative, reach about 0.77 to 0.79 AUROC in-distribution but fall to about 0.66 on unseen schemas; scaling to 7B, adding schema diversity, distilling a strong judge's rationales, and cross-benchmark training all fail to close that gap. Cross-schema transfer appears to track model scale and reasoning rather than fine-tuning. In practice, correctness uncertainty for text-to-SQL lives in reasoning-based signals: a fine-tuned verifier is a good in-domain tool, but a verifier that generalizes across schemas currently means a large frozen reasoning model.
Conversational QA systems should fix errors when users push back and signal which answers to trust. We split each follow-up into its speech act (re-ask, agree, or challenge) and its payload (an answer string, correct or not), and measure four confidence signals on the same transcripts: verbalized confidence, length-normalized likelihood, a frozen hidden-state probe, and P(True). Answers change when the user supplies an answer string and almost never under the speech act alone: in every retained full-size cell, the correct answer converts at least four times as many wrong answers as re-asking or agreement. The four signals disagree. On Qwen2.5-7B the frozen probe rises with answer persistence rather than truth; this holds on held-out questions, with Llama-3-8B (cleaned answers) as support, but not on four other models. Among 81 Qwen2.5-7B errors that survive one re-ask, re-asking, a fresh chat, or abstain repair almost none, while a gold answer repairs 75-95%. In two preregistered replications, every repair rate without an answer string stays below 0.15; a larger post-hoc rerun excludes 0.15 except for the empty elicit. With BM25 TriviaQA evidence, sentences that contain the answer repair 67-76% of persistent errors versus at most 12% when they do not. An end-to-end contain-gate cuts harm on TriviaQA, but a deployable gate that does not know the gold answer repairs few errors.
Duc-Vinh Tran
School of Information and Communication Technology, Hanoi University of Science and Technology, Hanoi, 100000, Vietnam.
Production text-to-SQL pipelines often end with an LLM-as-judge whose agreement with human annotators has never actually been measured. When we checked ours, the deployed gpt-4o-mini judge agreed with two-author gold at only Cohen's kappa = 0.04 on a disagreement-enriched set and 0.42 on a uniform-random spot-check, over-flagging 77.1% of the human-FAITHFUL cases in the enriched set. Most of its over-flags trace back to a single mechanism we call GRADE-HALLUCINATION. A self-hosted Qwen3.6-27B replacement (kappa = 0.72) lands in the same range as Claude Opus 4.7 (kappa = 0.71); the head-to-head is underpowered at n = 96, but for the deployment decision that hardly matters, since Qwen costs roughly 1/300 as much per call. Ensembling does not help for free. Pairing the weak judge with a stronger one degrades agreement, whereas three strong judges under unanimity routing reach kappa = 0.79 at 89.7% auto-coverage. Applied out-of-domain, the same audit recipe flags 25.5% of BIRD-financial's expert-authored gold SQLs as candidate gold-SQL issues under our annotation protocol. Code and pre-registration are at https://github.com/JamesL404/synca-audit.