AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory

Junhyung Ahn

cs.AINov 3, 2026

AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory

Authors: Junhyung Ahn

Organizations: School of Information Sciences University of Illinois Urbana-Champaign

Abstract

LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias. Judge pools mix judges of very different reliability, while the human-labeled sets that validate them hold a few hundred items or fewer. In this regime the per-item mean weights weak judges as heavily as strong ones, and confusion-table models such as Dawid-Skene must estimate a per-judge error table from too few items. We propose AGORA (Aggregation Of Reliability-weighted Assessments), an item-response-theory aggregator that places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds, so judge reliability is learned from all items jointly and without human labels. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items. It remains accurate as pools grow more uneven, a label-free dispersion statistic indicates in advance when trust weighting helps, and its learned trust weights track per-judge human validity.

Figures & tables

Figure 1: AGORA on an illustrative pool of three judges rating five items on a 0–4 scale. (1) The input is the judge-score matrix alone. (2) From all items jointly, AGORA fits a graded-response model per judge, drawn as its expected rating against latent quality

\theta

. The thresholds

\beta_{j}

encode severity (B is lenient, A severe), and the discrimination

\alpha_{j}

encodes trust (bars:

\alpha_{j}^{2}

); the horseshoe prior shrinks the flat judge C’s trust toward zero. (3) For each item, every rating places

\theta

in that judge’s threshold interval, with influence set by the judge’s trust. The posterior concentrates where A and B agree, so the EAP estimate

\hat{\theta}_{3}

ignores C’s outlying 0 (circled). (4)

\hat{\theta}_{i}

is mapped back to the rating scale. Against a human reference used only for evaluation, the AGORA scores reproduce the human ranking (Spearman

\rho=1.00

), whereas the per-item mean, shifted by C’s outliers, reaches

\rho=0.70

. Judge parameters and the human reference are illustrative; all other quantities are computed from them.

	SummEval		Topical-Chat		WMT22
Method	full	$N{=}50$	full	$N{=}50$	full	$N{=}50$
Mean	0.521	0.526	0.672	0.668	0.416	0.402
Median	0.523	0.529	0.643	0.643	0.406	0.402
Trimmed Mean	0.538	0.545	0.674	0.672	0.418	0.400
Dawid–Skene	0.555	0.529	0.695	0.678	0.423	0.399
CARE	0.544	0.534	0.688	0.662	0.418	0.405

Table 1: Main results on the primary five-judge pool. Dimension-averaged Spearman

\rho

against human references. full : the full evaluation set (SummEval

N{=}1{,}600

, Topical-Chat

N{=}360

, WMT22

N{=}19{,}725

N{=}50

: mean over 50 random item subsets of 50 items, shared by all methods (paired tests in § 5.1 ). Bold : best point estimate in each column.

Figure 2: Data efficiency on the primary five-judge pool in the small-sample regime (

N\leq 400

). Dimension-averaged gap to the per-item mean,

\Delta\rho=\rho_{\mathrm{method}}-\rho_{\mathrm{Mean}}

, as

N

shrinks (larger

N

on the left). AGORA is the top aggregator at every size; Dawid–Skene loses margin as

N

shrinks.

Figure 3: Accuracy at

N{=}50

as agreement dispersion

\delta

increases, over random five- and seven-judge pools with

0

–

8

uninformative judges (uniformly random ratings) added; points are means within

\delta

quintiles. AGORA remains stable while the per-item mean and Dawid–Skene decline. Dashed line:

\delta

of the primary pool.

Dataset	$N$	Mean	Dawid–Skene	AGORA
SummEval	50	0.510	0.501	0.518
	100	0.545	0.546	0.555
	200	0.542	0.545	0.547
Topical-Chat	50	0.715	0.703	0.718
	100	0.697	0.699	0.703
	200	0.703	0.708	0.709

Table 2: Thirty random five-judge pools: dimension-averaged Spearman

\rho

, averaged over pools and over 15, 12, and 10 item subsets at

N{=}50

, 100, and 200. AGORA has the highest average at every size.

	SummEval		Topical-Chat
Read-out	full	$N{=}50$	full	$N{=}50$
R0: per-item mean	0.521	0.526	0.672	0.668
R1: latent, equal trust	0.533	0.540	0.672	0.684
R2: latent, $\alpha_{j}^{2}$ trust	0.556	0.550	0.705	0.696

Table 3: Read-out decomposition: the fit is held fixed and only the trust vector changes. At full

N

the fitted-discrimination read-out (R2

-

R1) supplies most of the gain over the mean; at

N{=}50

both the latent read-out and the fitted trust contribute.

	clean (full $N$ )		adversarial ( $m{=}2$ )
prior on $\alpha$	SE	TC	SE	TC
flat (MLE)	0.551	0.697	0.527	0.663
horseshoe	0.554	0.705	0.535	0.680

Table 4: Discrimination-prior ablation (

m

: number of injected adversarial judges). The regularized horseshoe matches the unpenalized (flat MLE) fit on clean pools and improves on it under adversarial corruption.

	Mean	$\bar{\rho}_{j}$ -wt.	inv-var	spectral	AGORA
SummEval (full)	0.521	0.550	0.541	0.548	0.556
Topical-Chat (full)	0.672	0.686	0.674	0.683	0.705

Table 5: AGORA vs. label-free continuous reliability weightings (Spearman

\rho

, primary pool, full

N

): mean-correlation (

\bar{\rho}_{j}

), inverse-variance, and a spectral meta-learner. AGORA is best on both datasets.

		primary pool $+$ gemma			top-5 pool
Dataset	$N$	Mean	DS	AGORA	gemma	AGORA
SummEval	50	0.541	0.549	0.569	0.572	0.588
	200	0.543	0.562	0.573	0.548	0.582
	full	0.552	0.574	0.578	0.563	0.586
Topical-Chat	50	0.697	0.701	0.717	0.708	0.750
	200	0.705	0.722	0.732	0.704	0.745

Table 6: Pools containing the strongest judge, gemma-4-31B , scored on held-out items (75%; 10 and 5 item subsets at

N{=}50

and 200). Left: the primary pool plus gemma . Right: the five judges with the highest human validity on a separate calibration quarter, against its best member, gemma alone.

Figure 4: Trust weights track judge validity on the full 14-judge pool. Each point is one judge on one dimension: post-hoc validity

r_{j}

(Spearman with human labels) against the fitted trust weight

\alpha_{j}^{2}

;

\bar{\rho}_{s}

is their dimension-averaged Spearman correlation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Perturbation	SE	TC
slab $c\in\{3,8,12\}$	0.0000	0.0001
grid $\in\{100,200,250\}$	0.0000	0.0000
EM iters $\in\{8,16,20\}$	0.0002	0.0000
warm-start (5 seeds)	0.0000	0.0000

Appendix

Table 7: Estimation sensitivity: the maximum change in full-

N

Spearman

\rho

when each fixed setting is varied from its default (

c{=}5

, grid

150

T{=}12

) and when the warm-start is randomly perturbed. Base

\rho

0.556

(SummEval),

0.705

(Topical-Chat), and

0.426

(WMT22); every shift is at most

0.0002

Judge	SummEval	Topical-Chat	WMT22
gpt-5	0.530	0.629	0.427
gemini-2.5-pro	0.550	0.655	0.421
claude-sonnet-4-6	0.531	0.700	0.417
DeepSeek-V3	0.491	0.652	0.384
Qwen3-Next-80B	0.495	0.567	0.358
Llama-4-Scout	0.422	0.440	0.312

Appendix

Table 8: Per-judge Spearman

\rho

vs. human labels on the full pool (dimension-averaged).

Method	Spearman $\rho$	Kendall $\tau_{b}$	pairwise acc.
Mean	0.416	0.3418	0.619
GLAD	0.419	0.3362	0.644
Dawid–Skene	0.423	0.3396	0.645
AGORA	0.426	0.3419	0.647

Appendix

Table 9: WMT22 (full

N

) under three agreement metrics. AGORA ranks first on all three; the per-item mean trails on pairwise accuracy.

Panel of Figure 5 — Figure 5: Two parameterizations on the same small evaluation set ( $N{=}50$ items, $J{=}5$ judges, $K{=}5$ ). Left: confusion-matrix aggregators give each judge a $K\!\times\!K$ error table; with few items most cells are prior-dominated, so the estimate becomes less data-efficient. Right: AGORA maps every item onto one latent quality $\theta$ and summarizes each judge with a discrimination and ordered thresholds, each informed by all $N$ items; the read-out then weights each judge by $\alpha_{j}^{2}$ , down-weighting unreliable ones.

Dataset	Single best	AGORA
SummEval (full)	0.526	0.556
SummEval ( $N{=}50$ )	0.514	0.550
Topical-Chat (full)	0.700	0.705
Topical-Chat ( $N{=}50$ )	0.657	0.696
WMT22 (full)	0.417	0.426
WMT22 ( $N{=}50$ )	0.396	0.409

Appendix

Table 10: AGORA vs. the label-free single-best judge (mean-correlation argmax), Spearman

\rho

on the primary pool under the same 50-subsample protocol. AGORA leads at every size on all three datasets; on the multi-dimensional benchmarks the margin grows as

N

shrinks.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 581 papers

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Feb 9, 2026Jitian Zhao, Changho Shin, Tzu-Heng Huang +2LLM Evaluation LLM-as-a-Judge
Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.
Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
Mar 5, 2026Sunishchal Dev, Andrew Sloan, Joshua Kavner +2LLM-as-a-Judge LLM Evaluation
RAND Corporation, Santa Monica, CA, USA
VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge Aggregation
Oct 31, 2026Changho Shin, Frederic SalaLLM-as-a-Judge Bias in LLM-as-a-Judge
Department of Computer Science, Princeton University, Princeton, NJ · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI

Feb 9, 2026J/K move · Enter open · S save

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

Jitian Zhao, Changho Shin, Tzu-Heng Huang, Satya Sai Srinath Namburi GNVV+1

Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation Language Model Ensembles

Open paper AskPDF arXiv

LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they implicitly assume that judges provide independent estimates of true quality. However, in practice, LLM judges exhibit correlated errors caused by shared latent confounders -- such as verbosity, stylistic preferences, or training artifacts -- causing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes. To address this, we introduce CARE, a confounder-aware aggregation framework that explicitly models LLM judge scores as arising from both a latent true-quality signal and shared confounding factors. Rather than heuristically re-weighting judges, CARE separates quality from confounders without access to ground-truth labels. We provide theoretical guarantees for identifiability and finite-sample recovery under shared confounders, and we quantify the systematic bias incurred when aggregation models omit confounding latent factors. Across 12 public benchmarks spanning continuous scoring, binary classification, and pairwise preference settings, CARE improves aggregation accuracy, reducing error by up to 26.8%. Code is released in https://github.com/SprocketLab/CARE.

Abstract

LLM-as-a-Judge

Explore similar work

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge Aggregation