Who can we trust? LLM-as-a-jury for Comparative Assessment

Mengjie Qian; Guangzhi Sun; Mark J. F. Gales; Kate M. Knill

cs.CLFeb 18, 2026

Who can we trust? LLM-as-a-jury for Comparative Assessment

Authors: Mengjie Qian, Guangzhi Sun, Mark J. F. Gales, Kate M. Knill

Organizations: Department of Engineering, University of Cambridge, UK.

Abstract

Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and evaluation aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address this, we study the LLM-asa-jury setting and propose BT-sigma, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone. Experiments on benchmark NLG evaluation datasets show that BT-sigma consistently outperforms averaging-based aggregation methods, and that the learned discriminators strongly correlate with independent measures of the cycle consistency of LLM judgments. Further analysis reveals that BT-sigma can be interpreted as an unsupervised calibration mechanism that improves aggregation by modelling judge reliability.

Figures & tables

Figure 1: Hard/Soft Bradley-Terry and BT- $\sigma$ in the LLM-as-a-jury setting. Hard BT uses binary comparisons ( $p(i\succ j)\in\{0,1\}$ ), while soft BT uses probabilistic preferences. BT- $\sigma$ extends soft BT with judge-specific discriminators $\sigma_{k}$ to model judge reliability.

Model	Method	COH	CON	FLU	REL	ALL
Llama-3.1-8B	Avg-Prob	43.87	31.89	26.32	42.39	36.12
	hard BT	46.88	45.47	37.09	50.69	45.03
	soft BT	48.40	45.12	36.80	51.02	45.33
Llama-3.2-3B	Avg-Prob	27.72	5.93	7.12	30.61	17.84
	hard BT	13.31	0.28	17.24	33.15	16.00
	soft BT	13.36	0.53	14.50	34.40	15.70

Table 1: Spearman correlations (SRC) of each individual LLM judge on SummEval. The best results for each aspect on each model are underlined.

Figure 2: Cycle inconsistency rate on different LLMs, evaluated on SummEval (COH).

Panel of Figure 3 — Figure 3: SRC barchart for different aggregation methods on SummEval and Topical-Chat (ALL aspects).

	SummEval				Topical-Chat
Method	COH	CON	FLU	REL	COH	CNT	ENG	NAT
Avg-Prob	52.55	41.75	36.21	50.09	56.01	49.39	61.62	51.53
hard BT	51.26	45.72	40.07	52.32	59.31	50.25	62.57	56.90
soft BT	53.94	47.86	42.69	53.11	60.05	53.87	61.87	58.20
Temp-BT	56.21	47.40	41.88	55.14	56.88	52.21	63.86	60.65
BT- $\sigma$	57.38	47.47	42.99	54.15	59.02	56.30	63.49	60.56

Table 2: Aggregation performance measured by Spearman correlations (SRC) between aggregated rankings and human judgments, evaluated on SummEval and Topical-Chat.

Figure 4: Scatter plot of $1/\sigma_{k}$ and LLM performance with Avg-Prob, measured by SRC, on SummEval (ALL).

Dataset	Metric	COH	CON	FLU	REL	ALL
SummEval	PCC	84.11	54.05	60.75	71.92	72.21
SummEval	SRC	80.95	66.67	76.19	76.19	85.71
		COH	CNT	ENG	NAT	ALL
Topical-Chat	PCC	58.68	73.01	59.79	70.09	67.41
Topical-Chat	SRC	40.48	73.81	61.90	76.19	59.52

Table 3: Correlation between learned discriminator

1/\sigma_{k}

and LLM rankings (PCC/SRC, %) on SummEval and Topical-Chat.

Dataset	Metric	COH	CON	FLU	REL	ALL
SummEval	PCC	90.17	90.33	85.72	88.25	90.29
SummEval	SRC	97.62	97.62	88.10	97.62	95.24
		COH	CNT	ENG	NAT	ALL
Topical-Chat	PCC	90.17	89.99	90.41	81.61	93.09
Topical-Chat	SRC	85.71	92.86	92.86	83.33	90.48

Table 4: Correlation between learned discriminator

1/\sigma_{k}

and

1-\text{CycleRate}

(PCC/SRC, %) on SummEval and Topical-Chat.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Cycle inconsistency rate on different LLMs, evaluated on SummEval.

Figure 6: Scatter plot of $1/\sigma_{k}$ and $1-\text{CycleRate}$ on SummEval.

LLM	Method	COH	CNT	ENG	NAT	ALL
Llama-3.1-8B	Avg-Prob	51.82	44.06	62.61	46.90	51.34
	hard BT	49.65	41.87	61.92	51.70	51.29
	soft BT	50.57	45.92	58.73	49.62	51.21
Llama-3.2-3B	Avg-Prob	31.83	10.92	26.28	14.82	20.96
	hard BT	26.09	6.61	30.70	15.93	19.83
	soft BT	22.90	3.48	26.55	15.14	17.02

Appendix

Table 5: SRC of each individual LLM judge on Topical-Chat. The best results for each aspect on each model are underlined.

Panel of Figure 7 — Figure 7: Cycle inconsistency rate on different LLMs, evaluated on Topical-Chat.

Panel of Figure 8 — Figure 8: Scatter plots of judge discriminators on Topical-Chat.

# Judges	Avg-Prob	hard BT	soft BT	BT- $\sigma$	hard BT- $\sigma$
8	54.64	57.26	58.50	59.92	60.15
7	49.81	52.99	53.99	55.13	52.76
6	53.24	56.63	58.49	59.46	56.82
5	53.46	57.20	56.66	59.51	55.77
4	55.27	58.85	58.53	59.09	63.27
3	55.64	58.56	58.96	58.90	63.47

Appendix

Table 6: Overall SRC on Topical-Chat for different jury sizes.

Table 15

Figure 9: Scatter plot of $1/\sigma_{k}$ and LLM performance with Avg-Prob, measured by SRC on NovelEval (Relevance).

Figure 10: Cycle inconsistency rate on different LLMs, evaluated on NovelEval (Relevance)

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 550 papers

Evaluating the Reliability and Fidelity of Automated Judgment Systems of Large Language Models
Mar 23, 2026Tom Biskupski, Stephan KleberLLM-as-a-Judge Automated Evaluation
Hochschule der Medien Stuttgart, Germany · Ravensburg-Weingarten University of Applied Sciences Weingarten, Germany
A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge
Oct 13, 2026Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli +6Bias in LLM-as-a-Judge LLM-as-a-Judge
University of Pisa · University of Trento · KU Leuven +1
Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
Apr 16, 2026Manan Gupta, Dhruv KumarLLM Evaluation LLM-as-a-Judge
1BITS Pilani, Pilani Campus, India

Mar 23, 2026J/K move · Enter open · S save

Evaluating the Reliability and Fidelity of Automated Judgment Systems of Large Language Models

Tom Biskupski, Stephan Kleber

Hochschule der Medien Stuttgart, Germany · Ravensburg-Weingarten University of Applied Sciences Weingarten, Germany

LLM-as-a-Judge Automated Evaluation LLM Reliability Human-AI Agreement

Open paper AskPDF arXiv

A Large Language Model (LLM) as judge evaluates the quality of victim Machine Learning (ML) models, specifically LLMs, by analyzing their outputs. An LLM as judge is the combination of one model and one specifically engineered judge prompt that contains the criteria for the analysis. The resulting automation of the analysis scales up the complex evaluation of the victim models' free-form text outputs by faster and more consistent judgments compared to human reviewers. Thus, quality and security assessments of LLMs can cover a wide range of the victim models' use cases. Being a comparably new technique, LLMs as judges lack a thorough investigation for their reliability and agreement to human judgment. Our work evaluates the applicability of LLMs as automated quality assessors of victim LLMs. We test the efficacy of 37 differently sized conversational LLMs in combination with 5 different judge prompts, the concept of a second-level judge, and 5 models fine-tuned for the task as assessors. As assessment objective, we curate datasets for eight different categories of judgment tasks and the corresponding ground-truth labels based on human assessments. Our empirical results show a high correlation of LLMs as judges with human assessments, when combined with a suitable prompt, in particular for GPT-4o, several open-source models with

\geqslant

32B parameters, and a few smaller models like Qwen2.5 14B.

Abstract

LLM-as-a-Judge

Explore similar work

Evaluating the Reliability and Fidelity of Automated Judgment Systems of Large Language Models

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations