JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification

Tianxin Zhou; Ruixi Lin

cs.CLAug 20, 2026

JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification

Authors: Tianxin Zhou, Ruixi Lin

Organizations: Independent Researcher

Abstract

Panels of inexpensive LLM judges increasingly make accept-or-escalate decisions. In factuality settings, accepting a claim because several reference-free judges agree can create a hidden risk: agreement may reflect shared false-negative blind spots rather than independent evidence. We introduce JuryProbe, an empirical consensus-risk diagnostic for reference-free factuality judge panels, paired with a calibration-based routing policy. JuryProbe estimates consensus risk from a labeled calibration probe using false-negative-only (FN-only) judge correlation and false-consensus lift; when flagged high-risk, reference-free majority accepts are routed to the same judges with trusted references. On audited FEVER corruptions, reference-free panels show correlated false negatives (FN-only correlations 0.402 and 0.368; lifts 3.13x and 18.13x), while unanimous false consensus drops to zero under a trusted-reference best-case diagnostic on both minimal-pair and non-minimal-pair evidence. In flagged settings, the routed policy is by construction equivalent to grounding every reference-free majority accept (verified in 34/34 splits): improvement comes from accept-conditioned grounding, while the diagnostic determines whether to activate it. A fixed, pre-specified rule flags 8-10 of 10 splits across synthetic, benchmark-authored, and scientific families and 0 of 10 on a negative control, where standing down avoids 28% of reference acquisitions at a 0.004 increase in false accepts. False-accept reduction persists under weak BM25 retrieval at substantial coverage cost, while stale stand-down labels require periodic recalibration. JuryProbe provides no formal risk guarantee and does not establish reliable stand-down on natural panels; its supported contribution is an empirical diagnostic of high-risk panel error dependence.

Figures & tables

Figure 1: JuryProbe-routed policy. Consensus risk is estimated once on a calibration probe and fixed during deployment.

Family	$N_{\mathrm{corr}}$	$N_{\mathrm{det}}$	Corr.	FC Rate	FC Lift	$p$
Number	300	214	0.402	0.159	3.13 $\times$	$<0.001$
Entity	300	286	0.368	0.031	18.13 $\times$	$<0.001$
Attribute	300	296	0.263	0.003	54.55 $\times$	0.018

Table 1: Reference-free consensus-risk statistics.

N_{\mathrm{corr}}

is the number of corrupted claims,

N_{\mathrm{det}}

is the detectable corrupted subset used for difficulty-controlled false-consensus analysis, Corr. denotes FN-only correlation, and FC denotes false consensus.

Family	RF Corr.	Grounded Corr.	RF FC Rate	Grounded FC Rate	RF FC Lift
Number	0.386	0.000	0.159	0.000	3.13 $\times$
Entity	0.393	-0.003	0.031	0.000	18.13 $\times$

Table 2: Trusted-reference grounding diagnostic on the confirmatory families. RF denotes reference-free judging, Corr. denotes FN-only correlation, and FC denotes unanimous false consensus. The same judge panel evaluates the same corrupted claims with and without trusted references. Grounded FC lift is not reported because, in both families, the grounded false-consensus rate and the corresponding independence baseline are zero, making the lift ratio undefined.

Family	Policy	False Accept	True Accept	Residual FC	Verifier Calls
Number	RF Majority	$0.427{\pm}0.029$	$0.581{\pm}0.029$	$0.201{\pm}0.022$	$0{\pm}0$
Number	RF Unanimity	$0.201{\pm}0.022$	$0.273{\pm}0.027$	$0.201{\pm}0.022$	$0{\pm}0$
Number	Disagreement-Routed	$0.201{\pm}0.022$	$0.762{\pm}0.029$	$0.201{\pm}0.022$	$420{\pm}19$
Number	Random-Routed	$0.212{\pm}0.013$	$0.792{\pm}0.019$	$0.100{\pm}0.009$	$454{\pm}14$
Number	Ground-All-Accepts	$0.000{\pm}0.000$	$0.581{\pm}0.029$	$0.000{\pm}0.000$	$454{\pm}14$
Number	JuryProbe-Routed	$\mathbf{0.000{\pm}0.000}$	$0.581{\pm}0.029$	$\mathbf{0.000{\pm}0.000}$	$454{\pm}14$

Table 3: Held-out policy evaluation over 10 split seeds. Values are mean

\pm

standard deviation across splits. Residual FC denotes residual false-consensus rate after policy decisions. Verifier Calls denotes the number of grounded judge calls used by the policy. Always Grounded uses actual grounded verifier outputs from the same judge panel, not oracle labels. The Ground-All-Accepts rows match JuryProbe-Routed exactly because every split in both families is flagged high-risk, making the two policies identical by construction.

Policy	False Accept	True Accept	References / Split
RF Majority	$0.013{\pm}0.006$	$0.552{\pm}0.023$	$0$
JuryProbe-Routed (stand-down)	$0.013{\pm}0.006$	$0.552{\pm}0.023$	$\mathbf{0}$
Ground-All-Accepts	$0.009{\pm}0.005$	$0.552{\pm}0.023$	$84.7{\pm}4.0$
Always Grounded	$0.138{\pm}0.025$	$1.000{\pm}0.000$	$300$

Table 4: Stand-down comparison on the Self-Contained Contradiction negative control (10 splits; 300-item deployment halves). All values are executed grounded-panel results, not oracle labels. References / Split counts unique claims requiring a trusted reference.

Family	Type	Flagged	FN Corr	FC Lift
Number	programmatic FEVER corruption	10/10	$0.385{\pm}0.028$	$2.60{\pm}0.20$
Entity	programmatic FEVER corruption	10/10	$0.348{\pm}0.062$	$11.44{\pm}5.27$
Attribute	programmatic FEVER corruption	8/10	$0.265{\pm}0.112$	–
SciFact	external scientific benchmark	10/10	$0.319{\pm}0.022$	$5.57{\pm}0.70$
FEVER-Refutes	benchmark-authored refutations	8/10	$0.306{\pm}0.073$	$23.55{\pm}13.18$
CREAK	benchmark-authored commonsense	8/10	$0.318{\pm}0.100$	–

Table 5: Cross-family evaluation of the fixed rule (frozen thresholds, no per-family tuning): flagged splits out of 10, with cross-split mean

\pm

standard deviation of calibration-side FN-only correlation and lift. Entries marked – are unstable under near-zero permutation nulls (Appendix F ).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Panel	Fact.	Grounded	Escal.	Consensus Risk
Kohli (2026)	✓	x	x	x	Partial
Trust or Escalate	✓	x	Partial	✓	x
PoLL	✓	x	x	x	x
LongFact	x	✓	x	x	x
SAFE	x	✓	✓	x	x
FActScore	x	✓	✓	x	x

Appendix

Table 6: Positioning of JuryProbe relative to related work. Panel = LLM judge panel; Fact. = factuality-oriented evaluation; Grounded = grounded verification; Escal. = selective escalation; Consensus Risk = routing based on measured correlated-failure risk.

Check	Result	Purpose
Threshold sensitivity	Number: 10/10; Entity: 10/10	Not threshold-fragile
Grounded specificity	Number: 0/10; Entity: 0/10	Not always high-risk
Random-Routed stability	100 trials per split	Stable budget-matched baseline
Grounded evaluation integrity	No oracle, no gold fallback, no missing imputation	Actual verifier outputs only
Strong judge slice	$\rho_{\mathrm{FN}}=0.252$ , $L_{\mathrm{FC}}=2.17\times$ , $p<0.001$	Stronger RF judges do not remove risk

Appendix

Table 7: Additional robustness and artifact checks. Threshold sensitivity reports high-risk detections over correlation thresholds from 0.10 to 0.25 and lift thresholds from 1.25 to 2.00. Grounded specificity reports high-risk detections when the same risk estimator is applied to grounded verifier outputs.

Family	Corrupted claim	Reference	Outcome
Number	Pacific Rim was released July 14, 2013.	Pacific Rim was released July 12, 2013.	RF: 3/3 accept; G: 0/3 accept
Number	Alex Rodriguez was suspended for 169 games.	Alex Rodriguez was suspended for 211 games.	RF: 3/3 accept; G: 0/3 accept
Entity	Steve Jobs’s birth date is October 28, 1955.	Bill Gates’s birth date is October 28, 1955.	RF: 3/3 accept; G: 0/3 accept
Entity	Elton John was named MusiCares’ person of the year in 2013.	Bruce Springsteen was named MusiCares’ person of the year in 2013.	RF: 3/3 accept; G: 0/3 accept

Appendix

Table 8: Representative false-consensus cases. RF denotes reference-free judging, and G denotes grounded judging with the same judge panel and trusted reference.

Family	Subset	N	RF Corr.	RF FC Rate	RF FC Lift
Number	GPT-4o detectable	214	0.386	0.159	3.13 $\times$
Number	All corrupted	300	0.402	0.193	2.69 $\times$
Number	Grounded-detectable	300	0.402	0.193	2.69 $\times$
Entity	GPT-4o detectable	286	0.393	0.031	18.13 $\times$
Entity	All corrupted	300	0.368	0.030	13.19 $\times$
Entity	Grounded-detectable	300	0.368	0.030	13.19 $\times$

Appendix

Table 9: Alternative detectability definitions. Across all detectability definitions, the qualitative conclusion remains unchanged: reference-free panel exhibits excess false consensus, while no grounded false consensus is observed.

Metric	Reference-free	Benchmark rationale	Full abstract
All-3 false consensus	0.053 (10/190)	0.000 (0/190)	0.000 (0/190)
FC lift	5.56 $\times$	0.00 $\times$	0.00 $\times$
Mean pairwise FN correlation	0.312	0.247	0.108
Per-judge false accept	0.23 / 0.10 / 0.43	0.07 / 0.01 / 0.06	0.06 / 0.02 / 0.06
Clean true accept (per-judge mean)	0.598	0.737	0.791
Final parse failures	1/1,140	0/1,140	0/1,140

Appendix

Table 10: SciFact grounding diagnostic (190 contradicted / 190 supported claims). Per-judge false-accept rates list the three judges in the order Llama / Qwen / Gemma.

Family	Flagged	FN Corr	Lift	Permutation $p$
Number	10/10	$0.385{\pm}0.028$	$2.60{\pm}0.20$	$0.000{\pm}0.000$
Entity	10/10	$0.348{\pm}0.062$	$11.44{\pm}5.27$	$0.011{\pm}0.016$
Attribute	8/10	$0.265{\pm}0.112$	$79.55{\pm}87.23$ (unstable)	$0.209{\pm}0.417$
SciFact	10/10	$0.319{\pm}0.022$	$5.57{\pm}0.70$	$0.000{\pm}0.000$
FEVER-Refutes	8/10	$0.306{\pm}0.073$	$23.55{\pm}13.18$	$0.016{\pm}0.031$
CREAK	8/10	$0.318{\pm}0.100$	$92.97{\pm}64.29$ (unstable)	$0.205{\pm}0.419$

Appendix

Table 11: Full cross-family splitwise statistics (mean

\pm

standard deviation over 10 splits). Lift values marked unstable are dominated by near-zero independence nulls and are not comparable across families.

Target family	Share	Audit 25	Audit 50	Audit 95	Stale RF FA
Number	0%	0	0	0	$0.013{\pm}0.008$
Number	10%	2	7	9	$0.054{\pm}0.032$
Number	25%	6	8	10	$0.116{\pm}0.038$
Number	50%	7	10	10	$0.217{\pm}0.044$
Number	100%	8	10	10	$0.429{\pm}0.051$
Entity	0%	0	0	0	$0.013{\pm}0.008$

Appendix

Table 12: Recalibration flags (out of 10 splits) by target share and labeled-audit size, with the stale reference-free false-accept rate of the shifted stream.

Panel	Metric	Events	Rate	Wilson 95%
Main	RF false accept	40/190	0.211	[0.159, 0.274]
Main	RF clean accept	113/190	0.595	[0.524, 0.662]
Main	RF all-3 false consensus	10/190	0.053	[0.029, 0.094]
Main	Ground-All-Accepts false accept	3/190	0.016	[0.005, 0.045]
Main	Ground-All-Accepts clean accept	96/190	0.505	[0.435, 0.576]
Larger	RF false accept	44/190	0.232	[0.177, 0.297]

Appendix

Table 13: Pooled out-of-fold rates on 190 contradicted / 190 supported SciFact claims, with Wilson 95% intervals.

Panel	Fold	FN corr	All-3 count	Lift	$p$	RF FA	GAA FA
Main	1	0.296	7	6.34	0.0003	13/38	0/38
Main	2	0.357	10	5.91	0.0003	5/38	0/38
Main	3	0.228	5	4.60	0.0023	12/38	1/38
Main	4	0.372	10	5.98	0.0003	4/38	1/38
Main	5	0.297	8	4.66	0.0003	6/38	1/38
Larger	1	0.413	13	7.60	0.0003	14/38	1/38

Appendix

Table 14: Fold-level calibration statistics for both SciFact panels (grouped five-fold protocol; all folds flagged).

Policy	False Accept	True Accept	Surviving RF-unanimous FA	Grounded Claims
RF Majority	0.211	0.595	0.053 (10)	0
RF Unanimity	0.053	0.368	0.053 (10)	0
Routed + benchmark rationale	0.026	0.479	0.005 (1)	153
Routed + benchmark full abstract	0.016	0.505	0.005 (1)	153
Routed + BM25@1	0.016	0.295	0.005 (1)	153
Routed + BM25@3	0.021	0.326	0.011 (2)	153

Appendix

Table 15: SciFact reference-source stress test (full frozen subset). Surviving RF-unanimous FA counts corrupted claims unanimously accepted reference-free that also pass grounded majority (count in parentheses). Grounded Claims counts items routed to grounded verification.

Policy	Held-out FA	Full-subset FA	Held-out TA	Full-subset TA
RF Majority	$0.193{\pm}0.041$	0.211	$0.580{\pm}0.071$	0.595
Routed + rationale	$0.028{\pm}0.025$	0.026	$0.490{\pm}0.076$	0.479
Routed + full abstract	$0.010{\pm}0.013$	0.016	$0.502{\pm}0.080$	0.505
Routed + BM25@1	$0.013{\pm}0.018$	0.016	$0.278{\pm}0.025$	0.295
Routed + BM25@3	$0.020{\pm}0.023$	0.021	$0.310{\pm}0.027$	0.326

Appendix

Table 16: SciFact held-out deployment-half policy check versus full-subset rates.

Explore similar work

Aug 18, 2026

Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

Using LLMs as judges has become standard practice for evaluating model outputs at scale. This is particularly common for subjective, open-ended tasks such as assessing helpfulness or alignment, where no single reference answer exists. However, objective tasks introduce a distinct reliability challenge for reference-free LLM judging. In the absence of a reference answer, the judge evaluates factual correctness either through its parametric knowledge or through tool augmentation. Although the former enables efficient evaluation, the judge may hallucinate or lack sufficient evidence for its verdict. Conversely, tool augmentation can provide additional evidence but introduces extra computational cost and requires an appropriate mechanism to determine when and how that evidence should be used reliably. More importantly, neither approach alone provides formal control over the risk of accepted verdicts or guarantees their reliability at a specified level. We propose a risk-controlled framework that calibrates uncertainty thresholds on a held-out set so that the false discovery rate among accepted verdicts remains below a user-specified level~

α

with high probability, using finite-sample Clopper--Pearson intervals. When the parametric mode is not sufficiently confident, the instance is routed to a retrieval-augmented mode, where the judge gathers web evidence and re-evaluates the instance under a second calibrated threshold. The finite-sample guarantee carries over to this two-threshold routing without additional assumptions. Across open-domain QA benchmarks and judges of varying scales, the framework maintains the target error rate while achieving substantially higher coverage than single-mode baselines.

Sher Badshah, Ali Emami, Hassan Sajjad

Dalhousie University · New York University Abu Dhabi · Emory University

LLM-as-a-Judge Selective Prediction LLM Uncertainty Estimation FDR Control

Ask PDF

Sep 22, 2026

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly. We study JEV-as-a-Judge: evaluation with JEV, a decision-only judge that returns label probabilities instead of text, and whose confidence decides whether to accept its verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges, with blinded human adjudication, JEV comes within three points of GPT-6 wherever a verdict can be read off the text, at 0.36% of its fee and a 0.15-second median latency, and falls behind where the verdict must be derived, as in math, code, and logic. Its confidence marks this boundary. With a threshold frozen in advance, accepting confident verdicts and escalating the rest is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs at 41% of its fee, and in a pre-specified live test on two new workloads the cascade matches GPT-6's accuracy exactly. Confidence routing weakens on style-adversarial pairs and reference-free prose; we close with a simple recipe for validating thresholds locally.

Yubo Li, Yidi Miao, Ramayya Krishnan +1

Carnegie Mellon University

LLM-as-a-Judge Language Model Generation Evaluation Selective Prediction LLM Uncertainty Estimation Automated Evaluation

Ask PDF

Sep 18, 2026

Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus

Consensus among LLM judges is often taken as strong evidence that a decision is correct. This assumes that judges make their errors independently. In practice, LLM judges are often trained and evaluated in similar ways, so they can make the same mistakes. We study how this dependency affects the reliability of consensus. We find substantial error correlation across both open-weight and frontier LLM judges. In our main bank of ten judges, the average pairwise correlation between judge errors is 0.21. As a result, the ten judges only provide roughly as much statistical information as 3.5 independent judges. The dependency is even stronger among the high-accuracy frontier judges we evaluate, including judges from different providers. In up to 28% of our comparisons, ignoring shared errors leads to the conclusion that one system is significantly better, while accounting for them does not. We also find that the pattern of errors matters. Errors shared by most judges and errors concentrated among a smaller group affect consensus differently and favor different voting methods. Measuring the overall amount of correlation alone is therefore insufficient. Our results suggest a simple approach: use a small set of trusted examples to estimate judge accuracy and identify shared mistakes. These shared errors should then be considered when analyzing the results, and the voting method should be chosen using trusted examples before it is applied to new data.

Elias Hossain, Niloofar Yousefi, Ser-Nam Lim

University of Central Florida

LLM-as-a-Judge LLM Reliability Language Model Ensembles

Ask PDF