RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection

Elia Onofri; Roberto Di Pietro

cs.CLSep 30, 2026

RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection

Authors: Elia Onofri, Roberto Di Pietro

Organizations: Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division, King Abdullah University of Science and Technology (KAUST) Thuwal 23955, Saudi Arabia

Abstract

Automatic evaluation of faithfulness increasingly relies on a large language model acting as a judge, yet the most reliable judges are proprietary frontier models, costly and ill-suited to high-throughput monitoring. We investigate whether a panel of cheap open-weight judges (4--9B) can be aggregated to stand in for a frontier one, what the substitution sacrifices, and when it is worth making. We propose RAIM, an aggregation scheme robust to the members' correlated errors, coupling a cross-fitted stacked logistic regression with an admissibility test that, read from the members' own outputs, identifies when aggregating them improves on their best member and stays within reach of the frontier judge. We instantiate RAIM with ten judges from disjoint families across eight faithfulness benchmarks. Against Claude Sonnet, the panel retains a median 93% of its Cohen's $κ$ and gives up only 2.9 points of balanced accuracy on average; read as paired differences, it clearly improves on one benchmark and clearly worsens on three (only two by a non-negligible margin), leaving four unresolved. At a sixty-fourth of the frontier's inference price, the operative expense is a one-time in-domain calibration on 50--100 labelled records. The panel is also competitive with purpose-trained detectors on their home benchmarks (within 1.3 accuracy points of GPT-4o and 1.9 of the LLM-AggreFact leader), and beats the strongest one we reran by 6 points on our grounded sets. Whether aggregation pays depends on the members themselves: where several capable members err on different items, the panel improves on its best judge and approaches the frontier; where one dominates, the stacker recovers the leader, and only there does the frontier remain materially ahead. Both conditions are read off the calibration set at no further cost, so a cheap panel can stand in for a frontier one wherever this audit admits it.

Figures & tables

Figure 1 . The RAIM protocol at a glance. Ten cheap open-weight judges (

4

–

9

B) each score a candidate against its source, and a cross-fitted stacked logistic regression combines their votes into one verdict. The same calibration set yields an admissibility test, read off the members, which identifies a regime in which aggregating them pays. The verdict is read against the CV-best single judge of the panel, and against a paid frontier judge the panel is meant to replace.

		label-free			single judge, label-selected			stacked panel		frontier ( Sonnet )		paired $\Delta$ (panel $-$ comparator)
Dataset	$n$	avg. mem.	unw.	D–S	best (avg)	best (CV)	oracle	$\kappa$	bacc	$\kappa$	bacc	avg ( $\kappa$ )	best ( $\kappa$ )	Sonnet ( $\kappa$ )	Sonnet ( bacc )
Core grounded
MedHallu	2000	0.590	0.708	0.716	0.656	0.693 †	0.693	0.725	86.3	0.761	88.1	$\mathbf{+0.135}$	$\mathbf{+0.032}$ ‡	$\mathbf{-0.036}$	$\mathbf{-1.79}$
WiCE	222	0.385	0.449	0.486	0.547	0.484 †	0.547	0.540	77.0	0.548	77.4	$\mathbf{+0.155}$	$+0.056$	$-0.008$	$-0.39$
RAGTruth	1362	0.181	0.179	0.325	0.399	0.399	0.399	0.409	70.4	0.601	80.0	$\mathbf{+0.228}$	$+0.010$	$\mathbf{-0.192}$	$\mathbf{-9.62}$
XSum	546	0.312	0.395	0.420	0.377	0.400 †	0.437	0.437	71.9	0.418	70.9	$\mathbf{+0.125}$	$+0.037$	$+0.019$	$+0.96$

Table 1 . Comparator ladder over the eight benchmarks. Columns are ordered by the labelled information each comparator consumes.

\kappa

is given as a decimal and bacc as a percentage.

Figure 2 . Single-member competence per dataset (Cohen’s

\kappa

vs gold). Violins and strips carry the ten judges’

\kappa

, with the stacked panel (

\bullet

), the oracle best single (

\blacksquare

), and the frontier judge Sonnet (

\star

; § 4 ) overlaid. Datasets are ordered by descending mean single-member competence.

Figure 3 . The regime ordering. (a) The eight datasets (markers) scattered by the members’ mean pairwise error-correlation

\bar{\phi}

(

x

-axis) against the number of members at

\kappa\geq\kappa_{0}=0.30

(

y

-axis). The gold lines are the thresholds

\phi_{0}=0.40

and

c=\nicefrac{{1}}{{2}}

(five members), and bound the admissible region (shaded quadrant); the hatched area is the invariance region, the positions of the gold point at which no dataset changes side, unbounded to the right. Varying

\kappa_{0}

moves the points rather than an edge (Appendix C ). (b) The panel’s paired delta against the CV-best single (

\bullet

), the average member (

\blacksquare

), and the frontier judge (

\blackdiamond

), reported per dataset with

95\%

paired cluster-bootstrap intervals.

		quality ( bacc ) by scope			cost ( usd )		scope
Route	size	LLM-AggreFact -4	grounded 6	suite 8	per 1k	one-time	open	ungr.
purpose-trained and prompted comparators
Bespoke-MiniCheck-7B (leaderboard)	7B	$71.4$	—	n/a	$0.004^{\lx@paragraphsign}$	fine-tune	$\bullet$	$\circ$
MiniCheck-Flan-T5-L [ 73 ]	0.8B	$68.8$ $(68.4^{\dagger})$	$66.6^{\dagger}$	n/a	$0.004^{\lx@paragraphsign}$	fine-tune	$\bullet$	$\circ$
Granite Guardian 3.3 (leaderboard)	8B	$69.5$	—	—	$0.004^{\lx@paragraphsign}$	fine-tune	$\bullet$	$\odot$
Prometheus-2 [ 39 ]	7B	$59.3^{\dagger}$	$60.0^{\dagger}$	$60.4^{\dagger}$	$0.004^{\lx@paragraphsign}$	fine-tune	$\bullet$	$\bullet$

Table 2 . Quality, cost and applicability of each route. bacc is evaluated over three nested groups: the four LLM-AggreFact sets, the six grounded sets and the full suite. Published cells (top) are from the literature (Tables 8 and 9 ;

\kappa

in Table 12 ).

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

App.	What it carries	Supports
The account, its extensions, and its bounds
B	The competence/correlation account in full: the item-level recovery test that carries it, the cross-dataset predictor that does not, the regime ordering read dataset by dataset, the member-level views, the panel-size sweep, and the effective-independence comparison against a published frontier panel	§ 3.3
C	The identity that makes the correlation coordinate free, the obstruction that keeps the competence one from being so, the threshold family and its check on $1{,}888$ (sub-)panels, the labelling budget for both the test and the aggregator, and which half of the rule these eight datasets actually test	§ 3.3 , § 4
D	Matched base-versus-instruct probe: how instruction tuning moves the panel’s error-correlation	§ 3.3
E	Leave-one-dataset-out transfer, and why the in-domain advantage does not survive it	§ 3.3
External comparisons and their calibration

Appendix

Table 3 . Roadmap to the appendices: what each one carries, and the sections of the main text it supports.

members correct	$P(\text{recovers})$	$n$
0	0.000	529
1	0.058	327
2	0.162	235
3	0.101	189
4	0.157	210
5	0.225	160

Appendix

Table 4 . Item-level mechanism test on the

1{,}998

items on which the CV-best single judge is wrong. Items are pooled over the eight datasets, so that any recovery must come from the other members. (a) Recovery probability against the number of members individually correct on the item, counting only those that returned a verdict; (b) the same relation within each dataset.

		Spearman $\rho$ (two-sided $p$ ) against the (stacked $-$ best-single) $\kappa$ gap
subset	$n$	$\bar{\phi}$	members $\kappa\geq 0.30$	$(1-\bar{\phi})\times$ members
all eight	8	$-0.36$ (0.39)	$+0.48$ (0.23)	$+0.47$ (0.24)
without CNN	7	$-0.61$ (0.15)	$+0.74$ (0.06)	$+0.71$ (0.07)
without CNN and ExpertQA	6	$-0.37$ (0.47)	$+0.58$ (0.23)	$+0.54$ (0.27)

Appendix

Table 5. Cross-dataset predictor of the panel’s gain over its best single judge. Each coordinate is taken separately and as the composite; rows drop, in turn, the two sets held back in § 2 .

Figure 4 . Pairwise error-correlation $\phi$ between the ten members, on all eight datasets in ascending $\bar{\phi}$ (panels share the member order and the colour scale; the diagonal is masked). The panels run from FActScore (lowest $\bar{\phi}$ ) to CNN (highest).

Figure 5 . Leave-one-out change in stacked $\kappa$ from dropping each member ( $\Delta=\text{stacked without member}-\text{full stacked}$ ): red marks a load-bearing member, whose removal costs the panel, blue one the panel can spare. Every cell carries its value, and each column’s extremes are set in bold. The colour scale is clipped at $\pm 0.07$ for legibility, Gemma on CNN at $-0.158$ being off-scale.

Figure 6 . The two budgets a deployer sets by dataset. (a) Stacked

\kappa

against panel size

K

, members added in a greedy decorrelating order and the stacker refitted at each

K

(Appendix B ). (b) Stacked

\kappa

against the labelled records the aggregator is fitted on, drawn within the reported stacker’s own folds and pooled across them, interquartile range over

R=100

draws shaded. The dotted line reports the dataset’s label-free majority vote on the same folds; markers encode what a budget beats.

		full sample			$\Pr$ (call matches) at a budget of
Dataset	$n$	$\bar{\phi}$	spread	regime	10	25	50	100	200
MedHallu	2000	$0.335$	$10$	admissible	$0.70$	$0.78$	$0.88$	$0.96$	$0.99$
WiCE	222	$0.320$	$9$	admissible	$0.54$	$0.67$	$0.81$	$0.97$	$1.00$
RAGTruth	1362	$0.546$	$2$	not admissible	$0.98$	$1.00$	$1.00$	$1.00$	$1.00$
XSum	546	$0.494$	$4$	not admissible	$0.79$	$0.87$	$0.94$	$0.99$	$1.00$
CNN	114	$0.854$	$0$	not admissible	$0.99$	$1.00$	$1.00$	$1.00$	—

Appendix

Table 6 . Recovery of the regime call from a limited labelling budget. Per dataset and budget,

R=200

calibration samples of that many labelled records are drawn as whole clusters (§ 2 ), both coordinates of § 3.3 being re-estimated on each.

Dataset	train	unw.	labelled records the stacker is fitted on
			10	25	50	100	200	400	800	full
MedHallu	800	$0.708$	$0.706$	$0.709$	$0.713$	$0.718$	$0.721$	$0.724$	$0.725$	$0.725$
WiCE	176	$0.450$	$0.367$	$0.466$	$0.502$	$0.521$	—	—	—	$0.541$
RAGTruth	544	$0.179$	$0.365$	$0.383$	$0.395$	$0.402$	$0.405$	$0.408$	—	$0.408$
XSum	377	$0.396$	$0.284$	$0.377$	$0.405$	$0.418$	$0.429$	—	—	$0.440$
CNN	87	$0.070$	$0.076$	$0.196$	$0.304$	—	—	—	—	$0.351$

Appendix

Table 7 . Gold-label learning curve of the aggregator: mean Cohen’s

\kappa

over

R=100

draws per budget, budgets being in labelled records , the clustering unit of § 2 . Folds are the five of the reported stacker, the budget drawn from each fold’s training clusters and the out-of-fold predictions pooled before

\kappa

is taken.

Figure 7 . Within-panel pairwise error-correlation

\phi

(the 28 member pairs) for the matched base and instruct panels, per dataset (each dataset under its own scoring frame, § 2 ; constrained decoding). The shift

\Delta\phi

(instruct

-

base) is summarised in Appendix D .

Figure 8 . Leave-one-dataset-out transfer on the four core grounded sets: in-domain stacked and transferred ( lodo ), against the CV-best single judge and the average member per dataset.

System	size	CNN	XSum	WiCE	ExpertQA	mean	source
published, from the leaderboard and the cited papers
Bespoke-MiniCheck-7B	7B	$65.5$	$77.8$	$83.0$	$59.2$	$71.4$	leaderboard
MiniCheck-Flan-T5-L	0.8B	$69.9$	$74.3$	$72.2$	$59.0$	$68.8$	[ 73 ]
FactCG-DeBERTa-L	0.4B	$70.1$	$73.9$	$74.2$	$59.1$	$69.3$	[ 45 ] , leaderboard
HalluGuard-4B †	4B	$70.7$	$75.5$	$80.5$	$59.4$	$71.5$	[ 7 ]
Granite Guardian 3.3	8B	$67.0$	$74.9$	$76.6$	$59.6$	$69.5$	leaderboard

Appendix

Table 8 . The four LLM-AggreFact sets in leaderboard shape (balanced accuracy), the only sets on which we score the same rows as the published results. Upper block published; lower block measured here under the one protocol and per-dataset frame of § 2 .

System		F1
MedHallu — published
GPT-4o [ 59 ]	frontier, prompted	$87.7$
Qwen2.5-14B [ 59 ]	single open judge	$85.2$
GPT-4o-mini [ 59 ]	budget api	$84.1$
Qwen2.5-7B [ 59 ]	our panel member	$83.9$
Gemma-2-9B [ 59 ]	our panel member	$83.8$

Appendix

Table 9 . Hallucinated-class F1 on MedHallu and RAGTruth , published and measured here. Published rows are transcribed from the cited papers ( RAGTruth : response-level F1 on its QA subtask, which we adopt); ours are computed on our balanced construction (Appendix J ), an abstention counting as a missed detection. Slices differ: MedHallu ’s published table pools the

10{,}000

-pair superset against our

1{,}000

-row pqa_labeled slice, and RAGTruth ’s sits on its QA test split, at a hallucinated rate of

0.178

against our

0.5

, so its two blocks are not comparable in level, F1 moving with prevalence. Three published MedHallu detectors are members of our panel (§ 4 ).

	published anchor				ours ( bacc )
Dataset	strongest reported system	value	metric	what blocks a direct comparison	panel	Sonnet
TruthfulQA	GPT-judge (fine-tuned GPT-3-6.7B ) [ 50 ]	$90$ – $96$	accuracy	judges model generations after fine-tuning on in-domain human ratings; we score the dataset’s own reference answers zero-shot	$73.6$	$81.1$
FActScore	retrieval-augmented estimator [ 55 ]	—	system-level error	retrieves Wikipedia evidence by construction, approximating the human score to within $2\%$ at corpus level; we supply no evidence and score per claim	$75.2$	$77.6$

Appendix

Table 10 . Published anchors on the two benchmarks without a comparable published baseline: per set, the strongest anchor we found, the metric its authors report, and the structural difference that blocks the comparison. MedHallu and RAGTruth , reported in hallucinated-class F1, are in Table 9 .

	unweighted panel bacc			members at $\kappa\geq 0.30$		Sonnet $\kappa$
Dataset	def_on	claim	$\Delta$	def_on	claim	def_on	claim
XSum	$51.6$	$69.8$	$+18.1$ [+14.5, +21.7]	$0/10$	$4/10$	$0.269$	$0.418$
WiCE	$56.8$	$72.5$	$+15.8$ [+10.0, +22.0]	$0/10$	$9/10$	$0.539$	$0.548$
CNN	$50.0$	$53.5$	$+3.5$ [+0.8, +7.0]	$0/10$	$0/10$	$0.261$	$0.469$
ExpertQA	$51.8$	$57.3$	$+5.5$ [+3.2, +7.7]	$0/10$	$0/10$	$0.137$	$0.165$

Appendix

Table 11 . Frame contrast on the four LLM-AggreFact sets: the question–answer rubric ( def_on ) against the claim-support frame. Sonnet ’s

\kappa

is read from the judge records the main results use.

	purpose-trained judge $\kappa$					panel $\kappa$
Dataset	Prometheus-2	JudgeLM	Auto-J	MiniCheck- Flan-T5-L ⋄	Bespoke- MiniCheck-7B ⋄	stacked	best single (CV)	Sonnet
MedHallu	0.448	-0.204	0.003	0.262	-0.010	0.725	0.693	0.761
WiCE	0.224	-0.009	0.002	0.459	0.214	0.540	0.484	0.548
RAGTruth	0.004	0.028	-0.027	0.262	0.338	0.409	0.399	0.601
XSum	0.169	0.051	0.053	0.484	0.267	0.437	0.400	0.418
CNN	0.226	0.052	-0.000	0.349	0.103	0.349	0.298	0.469

Appendix

Table 12. Purpose-trained cheap judges off their home benchmark, against the panel and the frontier judge: Cohen’s

\kappa

on identical balanced instances, each dataset under its own scoring frame (claim-support on the LLM-AggreFact sets). Each purpose-trained judge runs off the shelf through its native interface (Appendix G ), thresholded at its documented operating point.

	Cohen’s $\kappa$				$\Delta$ (panel $-$ Sonnet )
Dataset	stacked	Qwen2.5-32B	Qwen2.5-72B	Sonnet	$\kappa$	bacc
MedHallu	0.725	0.614	0.669	0.761	$-0.036$ * [-0.069, -0.003]	$-1.8$ * [-3.5, -0.1]
WiCE	0.540	0.522	0.629	0.548	$-0.008$ [-0.140, +0.123]	$-0.4$ [-6.8, +6.1]
RAGTruth	0.409	0.484	0.351	0.601	$-0.192$ * [-0.239, -0.144]	$-9.6$ * [-12.0, -7.2]
XSum	0.437	0.452	0.446	0.418	$+0.019$ [-0.069, +0.102]	$+1.0$ [-3.3, +5.1]
CNN	0.349	0.380	0.349	0.469	$-0.120$ [-0.275, +0.048]	$-6.1$ [-13.8, +2.1]

Appendix

Table 13 . Performance of the stacked panel, the open single judges at

32

B and

72

B (4-bit awq ), and the proprietary frontier judge. Each dataset is scored under its own frame.

Figure 9 . Open-judge scaling against the frontier judge: the cheap stacked panel, the single open judges Qwen2.5-32B (

\triangledown

) and Qwen2.5-72B (

\triangle

, joined by the scaling segment), and the proprietary frontier Sonnet (

\star

), per dataset (Cohen’s

\kappa

; each dataset under its own scoring frame, § 2 ). The ladder is read in Appendix H .

Dataset	$n$	tier	frame	content
MedHallu	2000	core grounded	QA	medical question answering with synthesised hallucinations [ 59 ]
WiCE	222	core grounded	claim	sentence-level entailment of Wikipedia claims against cited evidence [ 34 ]
RAGTruth	1362	core grounded	QA	retrieval-augmented responses annotated for unsupported content [ 58 ]
XSum	546	core grounded	claim	AggreFact factuality annotations on extreme-summarisation output [ 72 , 57 ]
CNN	114	grounded	claim	AggreFact factuality annotations on CNN/DailyMail summaries [ 72 ]
ExpertQA	1462	grounded	claim	expert-curated questions with attributed answers [ 52 ]

Appendix

Table 14 . The eight benchmarks (§ 2 ).

method	MedHallu	WiCE	RAGTruth	XSum	CNN	ExpertQA	FActScore	TruthfulQA
unweighted panel	0.708	0.449	0.179	0.395	0.070	0.146	0.485	0.374
accuracy-weighted	0.710	0.468	0.303	0.395	0.053	0.190	0.485	0.411
log-odds-weighted	0.711	0.468	0.303	0.394	0.053	0.191	0.473	0.409
average member	0.590	0.385	0.181	0.312	0.088	0.154	0.259	0.274
best-on-average single	0.656	0.547	0.399	0.377	0.298	0.217	0.444	0.471
best single (CV)	0.693	0.484	0.399	0.400	0.298	0.240	0.444	0.471

Appendix

Table 15 . Full method ladder: Cohen’s

\kappa

against gold. Each dataset is scored under its own frame; central estimates, the

95\%

cluster-bootstrap intervals being in the released artefacts.

method	MedHallu	WiCE	RAGTruth	XSum	CNN	ExpertQA	FActScore	TruthfulQA
unweighted panel	85.4	72.5	59.0	69.8	53.5	57.3	74.3	68.7
accuracy-weighted	85.5	73.4	65.2	69.7	52.6	59.5	74.3	70.5
log-odds-weighted	85.6	73.4	65.2	69.7	52.6	59.6	73.6	70.5
average member	77.2	63.9	58.6	65.0	54.2	55.2	62.6	63.5
best-on-average single	82.8	77.1	69.9	68.9	64.9	60.9	70.3	73.4
best single (CV)	70.0	73.9	69.9	69.9	64.9	62.0	70.3	73.4

Appendix

Table 16 . Full method ladder: balanced accuracy against gold. Each dataset is scored under its own frame; central estimates, the

95\%

cluster-bootstrap intervals being in the released artefacts.

	Cohen’s $\kappa$			rescue		unsupervised aggregator
Dataset	unw.	D–S	stk.	$\Delta_{\text{stk}-\text{unw}}$	$\Delta_{\text{unw}-\text{best}}$	$\Delta_{\text{stk}-\text{DS}}$	$\Delta_{\text{DS}-\text{best}}$
MedHallu	0.708	0.716	0.725	$+0.017$ [+0.00, +0.03]	$+0.015$ [-0.01, +0.04]	$+0.009$ [-0.01, +0.03]	$+0.023$ [-0.01, +0.05]
WiCE	0.449	0.486	0.540	$+0.090$ [-0.01, +0.19]	$-0.035$ [-0.14, +0.08]	$+0.054$ [-0.04, +0.15]	$+0.002$ [-0.10, +0.11]
RAGTruth	0.179	0.325	0.409	$+0.229$ * [+0.18, +0.27]	$-0.220$ * [-0.26, -0.18]	$+0.084$ * [+0.04, +0.12]	$-0.074$ * [-0.11, -0.04]
XSum	0.395	0.420	0.437	$+0.042$ [-0.02, +0.10]	$-0.005$ [-0.07, +0.06]	$+0.018$ [-0.03, +0.06]	$+0.020$ [-0.04, +0.08]
FActScore	0.485	0.515	0.503	$+0.018$ [-0.05, +0.08]	$+0.041$ [-0.06, +0.14]	$-0.012$ [-0.05, +0.02]	$+0.071$ [-0.01, +0.15]

Appendix

Table 17 . The stacked panel against the two label-free ways of combining the same ten votes (Cohen’s

\kappa

, each dataset under its own frame). The rescue and unsupervised aggregator blocks are read in § 3.1 .

Figure 10 . The panel member by member (rows in roster order, datasets by descending mean single-member competence); the bracketed value in each cell is that member’s rank within its column, $1$ being the best of the ten, since the columns differ widely in level. (a) Each member’s own Cohen’s $\kappa$ against gold. (b) The mean cross-fit coefficient $\beta_{i}$ the stacker of § 2 assigns it, on the same folds as the reported results. Individual ranks in (b) are noisy across folds and should not be read finely; the columns’ broad structure is what carries the information.

Dataset	source	upstream pool	items	clusters	per cl.	balance
one matched pair per source record: a supported and a hallucinated candidate sharing its question and, on the grounded sets, its evidence
MedHallu	MedHallu/pqa_labeled [ 59 ]	$1{,}000$ questions	$2{,}000$	$1{,}000$	$2$	$1{,}000$ / $1{,}000$
RAGTruth	RAGTruth-processed , QA [ 58 ]	$5{,}034$ resp. / $839$ q.	$1{,}362$	$681$	$2$	$681$ / $681$
TruthfulQA	truthful_qa/generation [ 50 ]	$817$ questions	$1{,}634$	$817$	$2$	$817$ / $817$
FActScore	InstructGPT.jsonl [ 55 ]	$183$ biographies	$330$	$165$	$2$	$165$ / $165$
classes pooled across documents and the majority subsampled to the minority count (seed $0$ ), clusters recovered from the shared document

Appendix

Table 18 . Reduction of each benchmark to balanced binary items (Appendix J ). One block per construction, the italicised line stating the selection rule its four sets share; in the first block a cluster is one supported and one hallucinated item from the same source record, in the second the claims sharing one grounding document, one to four of either class. Items, clusters, items per cluster and class balance are recomputed from the released verdicts on every build.

MedHallu — 0:pos and 0:neg , same cluster
question	Do mitochondria play a role in remodelling lace plant leaves during programmed cell death?
evidence	Programmed cell death (PCD) is the regulated death of cells within an organism. The lace plant (Aponogeton madagascariensis) produces perforations in …
candidate, supported	Results depicted mitochondrial dynamics in vivo as PCD progresses within the lace plant, and highlight the correlation o …
candidate, hallucinated	Mitochondria regulate the formation of perforations in lace plant leaves through the modulation of calcium channels and …
RAGTruth — 0:pos and 0:neg , same cluster
question	butcher shop phone number

Appendix

Table 19 . One item from each dataset paired within a source record (Appendix J ). The supported and the hallucinated candidate answer the same question, against the same evidence where the set supplies any, and share a cluster, so the pair differs in the candidate and in nothing else. Every line is one prompt and one verdict, no judge seeing two candidates at once.

WiCE — 1:pos and 0:neg , different clusters
question	Each player received a key to the city from Mayor Bill de Blasio.
evidence	TITLE: Megan Rapinoe - #SheBelieves: historic ticker-tape parade in NYC for U.S. women’s national soccer team - Pictures - CBS News PUBLISHER: https:/ …
candidate, supported	Each player received a key to the city from Mayor Bill de Blasio.
candidate, hallucinated	The diocese is currently a titular see of the Patriarchate of Constantinople, and Gerasimos Papadopoulos was titular Bis …
XSum — 0:pos and 1:neg , different clusters
question	The number of recorded homicides in Scotland has fallen to its lowest level for more than 40 years, according to the Sco …

Appendix

Table 20 . One item from each dataset balanced across a pool of claims (Appendix J ). The question is the claim, and the nearest unsupported item is a different claim about a different document. Every line is one prompt and one verdict, no judge seeing two candidates at once.

#	tag	checkpoint	developer	ref.
00	Llama	meta-llama/Llama-3.1-8B-Instruct	Meta	[ 51 ]
01	Qwen	Qwen/Qwen2.5-7B-Instruct	Alibaba	[ 62 ]
02	Mistral	mistralai/Mistral-7B-Instruct-v0.3	Mistral AI	[ 31 ]
03	Gemma	google/gemma-2-9b-it	Google	[ 17 ]
04	Phi	microsoft/Phi-4-mini-instruct	Microsoft	[ 54 ]
05	Yi	01-ai/Yi-1.5-9B-Chat-16K	01.AI	[ 1 ]

Appendix

Table 21 . The ten-judge roster: one instruction-tuned open-weight checkpoint per model family, no two sharing a developer, held fixed across every dataset and experiment (§ 2 ). Every member’s native context window covers the

8192

-token cap the panel is run at.

member	MedHallu	WiCE	RAGTruth	XSum	CNN	ExpertQA	FActScore	TruthfulQA
Llama	1.000	0.806	1.000	0.982	0.956	0.966	0.997	0.998
Qwen	1.000	0.991	1.000	0.998	1.000	1.000	1.000	0.999
Mistral	1.000	0.986	0.996	0.996	1.000	0.989	1.000	1.000
Gemma	1.000	0.995	1.000	1.000	1.000	1.000	0.973	0.998
Phi	0.998	0.523	0.999	1.000	1.000	0.756	1.000	0.996
Yi	1.000	1.000	1.000	1.000	1.000	0.997	1.000	1.000

Appendix

Table 22 . Per-member coverage, the fraction of items with a parseable verdict. Each dataset is scored under its own frame.

		abstentions		stacked $\kappa$
Dataset	$n$	count	rate	$0.5$	categ.	$\Delta\kappa$ (categ. $-$ $0.5$ )	moved
MedHallu	2000	567	$2.8\%$	$0.725$	$0.729$	$+0.004$ $[-0.002,+0.010]$	12
RAGTruth	1362	123	$0.9\%$	$0.408$	$0.407$	$-0.002$ $[-0.007,+0.004]$	5
WiCE	222	179	$8.1\%$	$0.541$	$0.532$	$-0.009$ $[-0.054,+0.036]$	7
XSum	546	53	$1.0\%$	$0.440$	$0.447$	$+0.007$ $[-0.011,+0.026]$	8
CNN	114	5	$0.4\%$	$0.351$	$0.351$	$+0.000$ $[+0.000,+0.000]$	0

Appendix

Table 23 . Abstention encoded as a category rather than as the scalar

0.5

. The two arms differ in the meta-learner’s feature map alone, sharing folds, seed, items, estimator, penalty and decision threshold. Intervals are

95\%

paired cluster-bootstrap intervals over

B=2000

resamples of row_id (§ 2 ).

Explore similar work

May 28, 2026cs.CL

Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels

LLM-as-a-judge panels aggregate votes from multiple models, with the expectation that diverse models yield more reliable evaluations. We develop a framework to measure the true informational value of such panels and quantify how far their reliability falls short of the independent-voting ideal. Testing a panel of 9 frontier LLMs from 7 model families on three natural language inference datasets (each with 100 human annotations per item), we find that the 9 judges effectively provide only about 2 independent votes' worth of information. Roughly three-quarters of the panel's nominal independence is lost because the models make the same mistakes on the same items. The consequences are stark: the panel's actual accuracy falls 8-22 percentage points short of what independent voting would achieve, and the best single judge matches or outperforms the full panel across all conditions. Neither adding more judges nor using smarter aggregation algorithms helps -- established methods close at most 11% of this gap, even with access to the correct answers. We quantify these findings using the Kish effective sample size (n_eff) and a Condorcet null model, and show the deficit is robust across prompt variants, temperatures, chain-of-thought reasoning, and a pairwise preference task (RewardBench). The bottleneck is correlated judges, not the aggregation algorithm, implying that scaling up panels cannot substitute for genuinely independent evaluation.

Guneet Kohli

Apple

Llm-As-A-Judge Judges Large Language Model Evaluation Majority Voting Textual Dataset

Ask PDF

Feb 9, 2026cs.LG

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they implicitly assume that judges provide independent estimates of true quality. However, in practice, LLM judges exhibit correlated errors caused by shared latent confounders -- such as verbosity, stylistic preferences, or training artifacts -- causing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes. To address this, we introduce CARE, a confounder-aware aggregation framework that explicitly models LLM judge scores as arising from both a latent true-quality signal and shared confounding factors. Rather than heuristically re-weighting judges, CARE separates quality from confounders without access to ground-truth labels. We provide theoretical guarantees for identifiability and finite-sample recovery under shared confounders, and we quantify the systematic bias incurred when aggregation models omit confounding latent factors. Across 12 public benchmarks spanning continuous scoring, binary classification, and pairwise preference settings, CARE improves aggregation accuracy, reducing error by up to 26.8%. Code is released in https://github.com/SprocketLab/CARE.

Jitian Zhao, Changho Shin, Tzu-Heng Huang +2

Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.

Llm-As-A-Judge Large Language Model Evaluation

Ask PDF

Jul 9, 2026cs.AI

When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

LLM-as-judge (Zheng et al., 2023) is increasingly the default for evaluating AI systems in enterprise pipelines, often scaled to ensembles (Verga et al., 2024) or "mixture-of-experts" (Shazeer et al., 2017) panels of judges. These systems share a key assumption: that consistency -- agreement among judges, or among a model's own samples -- indicates correctness. We show this assumption is unreliable. Agreement is not accuracy: a model can agree with itself, and different models can agree with each other, out of shared bias, a memorized heuristic, or an option-position prior rather than truth. We ask when agreement is nonetheless a usable proxy, in a large-scale cross-runner study: 53 runners drew K=50 samples for assigned overlapping cases across comparisons of model tier, prompting, and scale on GPQA Diamond and AIME -- 265,000 samples. Using majority-correctness as the deployment label and a hierarchical runner-clustered bootstrap, agreement is a positive but weak predictor (rho 0.20-0.59, all positive under item-clustered resampling) whose usefulness is regime-dependent: best for unsaturated mid-tier models and for allocating compute, and worst -- over-confident yet no more accurate -- for the most consistent frontier model (agreement >=0.8 on 77% of GPQA case-result entries, 48% of those wrong). An exploratory cross-family check on three Claude tiers shows the same frontier over-confidence, with confident errors recurring across providers above a marginal-preserving null. Self-consistency is thus a conditional proxy for correctness, not a standalone confidence score. We publicly release the de-identified per-run rows and answer distributions.

Kaihua Ding

University of Pennsylvania

Self-Consistency Judges Frontier Models Artificial Intelligence Systems Majority Voting

Ask PDF