Production text-to-SQL pipelines often end with an LLM-as-judge whose agreement with human annotators has never actually been measured. When we checked ours, the deployed gpt-4o-mini judge agreed with two-author gold at only Cohen's kappa = 0.04 on a disagreement-enriched set and 0.42 on a uniform-random spot-check, over-flagging 77.1% of the human-FAITHFUL cases in the enriched set. Most of its over-flags trace back to a single mechanism we call GRADE-HALLUCINATION. A self-hosted Qwen3.6-27B replacement (kappa = 0.72) lands in the same range as Claude Opus 4.7 (kappa = 0.71); the head-to-head is underpowered at n = 96, but for the deployment decision that hardly matters, since Qwen costs roughly 1/300 as much per call. Ensembling does not help for free. Pairing the weak judge with a stronger one degrades agreement, whereas three strong judges under unanimity routing reach kappa = 0.79 at 89.7% auto-coverage. Applied out-of-domain, the same audit recipe flags 25.5% of BIRD-financial's expert-authored gold SQLs as candidate gold-SQL issues under our annotation protocol. Code and pre-registration are at https://github.com/JamesL404/synca-audit.
Figures & tables
Figure 1: The SynCA production pipeline (top) and our recommended judge-stage repair (bottom). The production pipeline runs Q-Gen, multi-model consensus, debate, and a final faithfulness judge over 1,482 accepted question–SQL pairs. The audit finds the final judge ( gpt-4o-mini ) at κ=0.04 versus two-author human gold. The repair replaces it with self-hosted Qwen3.6-27B ( κ=0.72 , ∼ 1/300 of Opus’s per-call cost); high-stakes deployments route the residual to a 3-strong ensemble reaching κ=0.79 at 89.7% auto-coverage.
Subset
Role / provenance
n
Final records
all pipeline outputs
1,598
Accepted
passed consensus+debate (released benchmark)
1,482
Rejected
dropped at consensus or debate
116
234-set
two-author gold, disagreement-enriched calib.
234
200 spot-check
two-author gold, uniform-random ( κ=0.87 )
200
382 active sample
pre-registered stratified draw from the 1,248-pool
382
Table 1: SynCA subsets and their provenance.
Judge
Conditional (n=234)
Unconditional (n=200)
gpt-4o-mini
0.04 [-0.02, 0.09]
0.42 [0.24, 0.58]
gpt-5.4-mini
0.63 [0.49, 0.76]
0.67 [0.52, 0.80]
Qwen3.6-27B
0.72 [0.59, 0.83]
0.94 [0.86, 0.99]
Claude Opus 4.7
0.71 [0.57, 0.83]
0.93 [0.86, 0.99]
Table 2: Per-judge κ vs two-author consensus gold (bootstrap 95% CIs, B=5,000 ; binary, minor-issue folded to faithful ). Qwen ≈ Opus at this measurement resolution (6–7 split on the 13 conditional disagreements). gpt-4o-mini over-flags (FNR 77.1%/FPR 14.3%); gpt-5.4-mini under-flags (FNR 4.7%/FPR 35.7%).
Variant
κ [95% CI]
over-
under-
baseline (production)
0.04 [-0.02, 0.09]
0.77
0.14
CoT
0.32 [ - 0.16, 0.49]
0.04
0.71
rubric
0.33 [ - 0.16, 0.47]
0.11
0.58
no-preview (best)
0.37 [ - 0.20, 0.52]
0.07
0.60
strict
0.23 [ - 0.07, 0.38]
0.07
0.74
Table 3: Four-variant prompt sweep on gpt-4o-mini (234-set; BCa 95% CIs, B=5,000 ; binary, minor-issue folded). Every variant flips the failure mode; none closes the gap to Qwen’s κ=0.72 .
Strategy
κ (95% CI)
UNF F1
cov.
gpt-5.4-mini alone
0.631 [0.49,0.76]
0.701
1.000
+ gpt-4o-mini AND
0.572 [0.42,0.71]
0.640
1.000
+ gpt-4o-mini OR
0.067 [0.02,0.11]
0.341
1.000
Qwen3.6-27B alone
0.716 [0.59,0.83]
0.763
1.000
Claude Opus 4.7 alone
0.705 [0.57,0.83]
0.750
1.000
3-strong MAJORITY (drop weak)
0.731 [0.60,0.84]
0.774
1.000
Table 4: Ensemble strategies on the 234 records with verdicts from all four judges. Unanimity routing auto-decides when all k judges agree, sends others to human. The 3-strong rule achieves κ=0.788 at 89.7% auto-coverage.
Judge
raw
BUG
OVER (sp. → rch.)
gpt-4o-mini
19.8
48
5.7 / 3.8
gpt-5.4-mini
46.2
74
17.0 / 11.3
Qwen3.6-27B
39.6
85
7.5 / 8.5
Opus 4.7
32.1
70
7.5 / 7.5
Table 5: BIRD-financial ( n=106 ), all %. raw : raw flag rate. BUG : bug-detect on 27 buggy gold SQLs (two-author classification of 65 flags). OVER : corrected over-flag rate, sparse / rich schema.
Table 7: Qwen3.6-27B per-class precision on the 200-question spot-check (Wilson 95% CIs). † minor-issue folded to faithful .
Variant
n
κ vs gold
κ vs baseline
baseline
234
0.716 [0.59, 0.83]
1.000
cot
234
0.789 [0.67, 0.89]
0.807
rubric
234
0.754 [0.63, 0.86]
0.871
no_preview
234
0.673 [0.52, 0.80]
0.828
Appendix
Table 8: Qwen3.6-27B prompt-variation κ on the 234-set (BCa CIs, B=5,000 ). All four CIs overlap; Qwen verdicts are stable across prompt choice.
Figure 2: Optimization trajectory: best κ across the seven main runs.
Figure 3: Confusion matrices vs human gold ( faithful = positive). gpt-4o-mini produces 148 false flags; Qwen has the fewest errors of both types.
Figure 4: Over-flag failure-mode taxonomy. Grade hallucination dominates (66%); both upgraded judges fix nearly every category.
Figure 5: Coverage– κ Pareto frontier: learned router (5-fold CV on 234-set) versus unanimity routing. The learned router’s best point (91.0% / 0.790) is within Monte-Carlo noise of the rule’s (89.7% / 0.788).
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.