A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

Cesare Barbera; Luca Stradiotti; Lorenzo Cascioli; Lorenzo Perini; Niek Tax; Daniel Haimovich; Jesse Davis; Andrea Passerini

cs.LGOct 13, 2026

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

Authors: Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli, Lorenzo Perini, Niek Tax, Daniel Haimovich, Jesse Davis, Andrea Passerini,

Organizations: University of Pisa · University of Trento · KU Leuven · Meta Platforms, Inc.

Abstract

Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a scalable alternative when human evaluation is too costly. Unfortunately, LLM judgments can exhibit systematic biases. Prior work has documented position, verbosity or model family of the candidate answers influencing the judge decision rather than their quality, while additional biases may remain undocumented. $\textit{Multicalibration}$ provides a natural framework to address this problem: each bias corresponds to miscalibration on an identifiable group, $\textit{e.g.}$ , family bias manifests as overconfidence in answers generated by models belonging to the same family of the judge. The challenge is to fix miscalibration simultaneously across many, potentially unknown groups. To this end, we introduce $\mathrm{CaLLM}$ , a post hoc framework that multicalibrates an LLM judge, thereby bounding all such biases simultaneously. We prove that $\mathrm{CaLLM}$ bounds the bias of every group whose error structure is captured by an available representation, and that augmenting it with explicit bias groups further reduces the residual bias. We then derive theoretical conditions under which $\mathrm{CaLLM}$ improves the judge's accuracy, rather than trading it for calibration. Finally, we evaluate $\mathrm{CaLLM}$ across pairwise preference benchmarks, showing that it reduces the multicalibration error on both specified and unspecified bias groups, while improving predictive performance of the judge.

Figures & tables

Figure 1: CaLLM ’s pipeline. We feed CaLLM with some representation $\phi(X)$ of the input (e.g., embeddings $E(X)$ and base scores $f_{0}(X)$ ). The final output is a multi-calibrated score $f^{\star}(X)$ .

Figure 2: Average per-axis

\text{MCE}_{\%}

(

\pm

s.e.) for each method across datasets. PKU doesn’t contain the information regarding the LLMs that generated the responses, so we do not report Family bias.

	PKU		AEX		PPE		RWB		MTB
	CaLLM	CaLLM $+g$	CaLLM	CaLLM $+g$	CaLLM	CaLLM $+g$	CaLLM	CaLLM $+g$	CaLLM	CaLLM $+g$
Position	2.8 $\pm$ 0.2	2.6 $\pm$ 0.1	3.7 $\pm$ 0.3	2.5 $\pm$ 0.3	5.4 $\pm$ 0.2	3.7 $\pm$ 0.3	6.9 $\pm$ 1.1	4.7 $\pm$ 0.5	13.4 $\pm$ 1.6	12.5 $\pm$ 1.5
Verbosity	5.6 $\pm$ 0.3	2.8 $\pm$ 0.2	4.5 $\pm$ 0.3	2.5 $\pm$ 0.2	7.5 $\pm$ 0.4	4.0 $\pm$ 0.2	5.2 $\pm$ 0.4	4.7 $\pm$ 0.4	13.5 $\pm$ 1.3	12.3 $\pm$ 0.9
Family	–	–	2.6 $\pm$ 0.2	2.5 $\pm$ 0.2	4.8 $\pm$ 0.3	4.6 $\pm$ 0.4	5.2 $\pm$ 0.3	4.6 $\pm$ 0.1	11.9 $\pm$ 0.8	11.6 $\pm$ 0.2

Table 1: Comparison between CaLLM and CaLLM

\!\!+g

in terms of

MCE_{\%}

for documented biases. PKU doesn’t contain the information regarding the LLMs that generated the responses, so we do not report Family bias.

	PKU		AEX		PPE		RWB		MTB
	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$
CaLLM	.69 $\pm$ .01	.57 $\pm$ .01	.68 $\pm$ .01	.60 $\pm$ .01	.67 $\pm$ .01	.61 $\pm$ .01	.93 $\pm$ .01	.21 $\pm$ .01	.81 $\pm$ .01	.41 $\pm$ .02
$f_{0}$	.63 $\pm$ .01	.73 $\pm$ .01	.63 $\pm$ .01	.76 $\pm$ .01	.64 $\pm$ .01	.74 $\pm$ .01	.86 $\pm$ .01	.36 $\pm$ .02	.79 $\pm$ .01	.49 $\pm$ .02
BPC	.64 $\pm$ .01	.67 $\pm$ .01	.65 $\pm$ .01	.69 $\pm$ .01	.65 $\pm$ .01	.68 $\pm$ .01	.88 $\pm$ .01	.29 $\pm$ .01	.80 $\pm$ .01	.43 $\pm$ .02
CalibraEval	.64 $\pm$ .01	.65 $\pm$ .01	.65 $\pm$ .01	.69 $\pm$ .01	.65 $\pm$ .01	.66 $\pm$ .01	.88 $\pm$ .01	.30 $\pm$ .01	.81 $\pm$ .01	.43 $\pm$ .02
PORTIA	.61 $\pm$ .01	.72 $\pm$ .01	.65 $\pm$ .01	.72 $\pm$ .01	.66 $\pm$ .01	.70 $\pm$ .01	.87 $\pm$ .01	.31 $\pm$ .02	.77 $\pm$ .02	.55 $\pm$ .05

Table 2: Average ACC and log-loss

\mathcal{L}_{\log}

(

\pm

s.e.) across datasets. CaLLM always matches or improves both metrics over all baselines, showing that multicalibration also improves predictive performance.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Average per-axis

\text{MCE}_{\%}

(

\pm

s.e.) on the specified and unspecified groups for each method across datasets.

Figure 4: Average per-group

\text{MCE}_{\%}

(

\pm

s.e.) for CaLLM and the mitigation baselines across datasets with Qwen 2.5-7B-Instruct used as the judge. The self-enhancement axis contains AEX and PPE only: PKU , RWB and MTB do not contain qwen -family responses.

	PKU		AEX		PPE		RWB		MTB
	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$
Position	2.4 $\pm$ 0.2	2.4 $\pm$ 0.2	6.8 $\pm$ 0.4	2.4 $\pm$ 0.3	8.3 $\pm$ 0.5	3.7 $\pm$ 0.5	10.0 $\pm$ 1.2	7.9 $\pm$ 1.1	11.8 $\pm$ 0.8	12.0 $\pm$ 0.8
Verbosity	6.3 $\pm$ 0.5	2.8 $\pm$ 0.2	4.6 $\pm$ 0.3	2.3 $\pm$ 0.1	7.2 $\pm$ 0.4	4.0 $\pm$ 0.3	9.7 $\pm$ 1.0	7.4 $\pm$ 0.8	14.4 $\pm$ 0.7	13.5 $\pm$ 1.1
Family	–	–	2.7 $\pm$ 0.3	2.1 $\pm$ 0.2	5.7 $\pm$ 0.9	4.7 $\pm$ 0.9	–	–	–	–

Appendix

Table 3:

\text{MCE}_{\%}

per bias group (

\pm 1

s.e.) for CaLLM and CaLLM

\!\!+g

using the Qwen 2.5-7B-Instruct judge. The family axis contains results for AEX and PPE only: PKU , RWB and MTB contain no qwen -family response.

	PKU		AEX		PPE		RWB		MTB
	Acc	$\mathcal{L}_{\log}$	Acc	$\mathcal{L}_{\log}$	Acc	$\mathcal{L}_{\log}$	Acc	$\mathcal{L}_{\log}$	Acc	$\mathcal{L}_{\log}$
CaLLM	.68 $\pm$ .01	.60 $\pm$ .01	.61 $\pm$ .01	.65 $\pm$ .01	.61 $\pm$ .01	.66 $\pm$ .01	.84 $\pm$ .01	.37 $\pm$ .01	.80 $\pm$ .01	.45 $\pm$ .02
$f_{0}$	.66 $\pm$ .01	.72 $\pm$ .01	.58 $\pm$ .01	.88 $\pm$ .01	.59 $\pm$ .01	.87 $\pm$ .01	.80 $\pm$ .01	.50 $\pm$ .02	.80 $\pm$ .01	.51 $\pm$ .02
BPC	.66 $\pm$ .01	.65 $\pm$ .01	.59 $\pm$ .01	.71 $\pm$ .01	.60 $\pm$ .01	.72 $\pm$ .01	.80 $\pm$ .01	.42 $\pm$ .01	.80 $\pm$ .01	.46 $\pm$ .02
CalibraEval	.66 $\pm$ .01	.63 $\pm$ .01	.59 $\pm$ .01	.68 $\pm$ .01	.60 $\pm$ .01	.69 $\pm$ .01	.80 $\pm$ .01	.43 $\pm$ .01	.80 $\pm$ .01	.47 $\pm$ .02
PORTIA	.59 $\pm$ .01	.71 $\pm$ .01	.59 $\pm$ .01	.75 $\pm$ .01	.60 $\pm$ .01	.74 $\pm$ .01	.76 $\pm$ .01	.50 $\pm$ .01	.70 $\pm$ .03	.59 $\pm$ .06

Appendix

Table 4: Average accuracy and log-loss (

\pm 1

s.e.) across datasets using Qwen 2.5-7B-Instruct as judge.

Figure 5: Average per-group

\text{MCE}_{\%}

(

\pm 1

standard error) for CaLLM (embeddings from the judge’s last hidden state), CaLLM(Qwen3-0.6B) (embeddings from Qwen3-Embedding-0.6B ) and, as a reference, the best of the Fig. 2 baselines across datasets. Family is undefined on PKU , since it contains no response from the judge’s own family.

	PKU		AEX		PPE		RWB		MTB
	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$
Position	2.7 $\pm$ 0.2	2.2 $\pm$ 0.1	4.2 $\pm$ 0.3	2.1 $\pm$ 0.2	5.7 $\pm$ 0.3	4.1 $\pm$ 0.2	6.7 $\pm$ 1.0	5.6 $\pm$ 0.5	14.2 $\pm$ 2.5	11.9 $\pm$ 1.5
Verbosity	7.4 $\pm$ 0.6	2.4 $\pm$ 0.1	5.5 $\pm$ 0.3	2.3 $\pm$ 0.2	8.6 $\pm$ 0.5	4.3 $\pm$ 0.1	7.9 $\pm$ 0.7	5.2 $\pm$ 0.5	14.3 $\pm$ 1.7	11.7 $\pm$ 1.4
Family	–	–	2.3 $\pm$ 0.1	2.2 $\pm$ 0.1	5.1 $\pm$ 0.3	5.1 $\pm$ 0.3	6.0 $\pm$ 0.6	6.6 $\pm$ 0.7	10.4 $\pm$ 0.8	11.1 $\pm$ 1.7

Appendix

Table 5:

\text{MCE}_{\%}

per bias group (

\pm 1

standard error) for CaLLM and CaLLM

\!\!+g

across datasets using Qwen3-Embedding-0.6B embeddings. Self-enhancement is undefined on PKU because the dataset does not contain muse -family responses.

	PKU		AEX		PPE		RWB		MTB
	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$
CaLLM	.69 $\pm$ .01	.57 $\pm$ .01	.68 $\pm$ .01	.60 $\pm$ .01	.67 $\pm$ .01	.61 $\pm$ .01	.93 $\pm$ .01	.21 $\pm$ .01	.81 $\pm$ .01	.41 $\pm$ .02
CaLLM (Qwen3-0.6B)	.69 $\pm$ .01	.58 $\pm$ .01	.66 $\pm$ .01	.61 $\pm$ .01	.66 $\pm$ .01	.62 $\pm$ .01	.91 $\pm$ .01	.27 $\pm$ .01	.81 $\pm$ .01	.42 $\pm$ .02
best baseline	.64 $\pm$ .01	.65 $\pm$ .01	.65 $\pm$ .01	.69 $\pm$ .01	.66 $\pm$ .01	.66 $\pm$ .01	.88 $\pm$ .01	.29 $\pm$ .01	.81 $\pm$ .01	.43 $\pm$ .02

Appendix

Table 6: Average accuracy and log-loss (

\pm

s.e.) for CaLLM and its ablated variant CaLLM(Qwen3-0.6B) across datasets. best baseline is the strongest of the Fig. 2 baselines on each (dataset, metric) pair.

Figure 6: Average per-group

\text{MCE}_{\%}

(

\pm 1

s.e.) across datasets for CaLLM and the calibration and multicalibration baselines. Family is undefined on PKU , since it contains no response from the judge’s own family.

	PKU		AEX		PPE		RWB		MTB
	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$
CaLLM	.69 $\pm$ .01	.57 $\pm$ .01	.68 $\pm$ .01	.60 $\pm$ .01	.67 $\pm$ .01	.61 $\pm$ .01	.93 $\pm$ .01	.21 $\pm$ .01	.81 $\pm$ .01	.41 $\pm$ .02
$f_{0}$	.63 $\pm$ .01	.73 $\pm$ .01	.63 $\pm$ .01	.76 $\pm$ .01	.64 $\pm$ .01	.74 $\pm$ .01	.86 $\pm$ .01	.36 $\pm$ .02	.79 $\pm$ .01	.49 $\pm$ .02
Temperature	.64 $\pm$ .01	.63 $\pm$ .01	.65 $\pm$ .01	.64 $\pm$ .01	.65 $\pm$ .01	.63 $\pm$ .01	.88 $\pm$ .01	.30 $\pm$ .01	.80 $\pm$ .01	.43 $\pm$ .01
Platt	.64 $\pm$ .01	.63 $\pm$ .01	.65 $\pm$ .01	.64 $\pm$ .01	.65 $\pm$ .01	.63 $\pm$ .01	.88 $\pm$ .01	.30 $\pm$ .01	.80 $\pm$ .01	.43 $\pm$ .01
LOGR	.64 $\pm$ .01	.62 $\pm$ .01	.65 $\pm$ .01	.63 $\pm$ .01	.65 $\pm$ .01	.63 $\pm$ .01	.88 $\pm$ .01	.30 $\pm$ .01	.81 $\pm$ .01	.43 $\pm$ .01

Appendix

Table 7: Average ACC and log-loss

\mathcal{L}_{\log}

(

\pm

s.e.) across datasets. CaLLM always matches or improves both metrics over all calibration and multicalibration strategies.

	PKU	AEX	PPE	RWB	MTB
Position	0.50 $\pm$ 0.01	0.76 $\pm$ 0.01	0.76 $\pm$ 0.01	0.93 $\pm$ 0.01	0.66 $\pm$ 0.05
Verbosity	0.92 $\pm$ 0.01	0.90 $\pm$ 0.01	0.89 $\pm$ 0.01	0.95 $\pm$ 0.01	0.94 $\pm$ 0.02
Family	-	0.98 $\pm$ 0.01	0.98 $\pm$ 0.01	0.99 $\pm$ 0.01	0.98 $\pm$ 0.02

Appendix

Table 8: Recoverability results across datasets.

		$\text{MCE}_{\%}$			$\text{MCE}_{\sigma}$
Data	Method	Position	Verbosity	Family	Position	Verbosity	Family
PKU	CaLLM	2.78 $\pm$ 0.23	5.55 $\pm$ 0.28	–	3.85 $\pm$ 0.31	7.68 $\pm$ 0.38	–
	$f_{0}$	12.34 $\pm$ 0.40	22.47 $\pm$ 0.28	–	18.08 $\pm$ 0.61	32.92 $\pm$ 0.47	–
	BPC	7.91 $\pm$ 0.25	19.25 $\pm$ 0.39	–	11.22 $\pm$ 0.36	27.28 $\pm$ 0.56	–
	CalibraEval	7.66 $\pm$ 0.19	20.51 $\pm$ 0.35	–	10.18 $\pm$ 0.26	27.26 $\pm$ 0.47	–
	PORTIA	10.26 $\pm$ 0.10	21.06 $\pm$ 0.31	–	14.73 $\pm$ 0.19	30.22 $\pm$ 0.45	–

Appendix

Table 9: Average per-group

\text{MCE}_{\%}

and

\text{MCE}_{\sigma}

(

\pm

standard error) for CaLLM and the mitigation baselines across datasets. The family axis does not show PKU since it does not contain muse -family responses.

		$\text{MCE}_{\%}$			$\text{MCE}_{\sigma}$
Dataset	Method	Position	Verbosity	Family	Position	Verbosity	Family
PKU	CaLLM	2.78 $\pm$ 0.23	5.55 $\pm$ 0.28	–	3.85 $\pm$ 0.31	7.68 $\pm$ 0.38	–
PKU	CaLLM $\!\!+g$	2.56 $\pm$ 0.10	2.81 $\pm$ 0.18	–	3.58 $\pm$ 0.14	3.92 $\pm$ 0.25	–
AEX	CaLLM	3.72 $\pm$ 0.35	4.49 $\pm$ 0.27	2.58 $\pm$ 0.16	4.27 $\pm$ 0.41	5.14 $\pm$ 0.29	2.96 $\pm$ 0.19
AEX	CaLLM $\!\!+g$	2.52 $\pm$ 0.30	2.55 $\pm$ 0.24	2.50 $\pm$ 0.19	2.90 $\pm$ 0.34	2.93 $\pm$ 0.29	2.87 $\pm$ 0.22
PPE	CaLLM	5.41 $\pm$ 0.18	7.48 $\pm$ 0.41	4.77 $\pm$ 0.34	2.63 $\pm$ 0.10	3.62 $\pm$ 0.19	2.32 $\pm$ 0.17

Appendix

Table 10: Average per-group

\text{MCE}_{\%}

and

\text{MCE}_{\sigma}

(

\pm

standard error) for CaLLM and CaLLM

\!\!+g

. The family axis does not show PKU since it does not contain muse -family responses.

		$\text{MCE}_{\%}$		$\texttt{MCE}_{\sigma}$
Dataset	Method	Specified	Unspecified	Specified	Unspecified
PKU	CaLLM	5.55 $\pm$ 0.28	3.89 $\pm$ 0.36	7.68 $\pm$ 0.38	5.38 $\pm$ 0.50
	$f_{0}$	22.46 $\pm$ 0.28	12.34 $\pm$ 0.40	32.91 $\pm$ 0.47	18.08 $\pm$ 0.61
	BPC	19.23 $\pm$ 0.40	7.91 $\pm$ 0.25	27.26 $\pm$ 0.56	11.22 $\pm$ 0.36
	CalibraEval	20.49 $\pm$ 0.35	7.66 $\pm$ 0.19	27.24 $\pm$ 0.47	10.18 $\pm$ 0.26
	PORTIA	21.04 $\pm$ 0.31	10.26 $\pm$ 0.10	30.20 $\pm$ 0.45	14.73 $\pm$ 0.19

Appendix

Table 11: Average per-group

\text{MCE}_{\%}

and

\text{MCE}_{\sigma}

(

\pm

standard error) for CaLLM and the mitigation baselines across datasets over the three biases documented in the literature ( i.e. position, verbosity, and family) and the unspecified groups.

		$\text{MCE}_{\%}$			$\text{MCE}_{\sigma}$
Dataset	Method	Position	Verbosity	Family	Position	Verbosity	Family
PKU	CaLLM	2.39 $\pm$ 0.17	6.32 $\pm$ 0.53	–	3.17 $\pm$ 0.23	8.39 $\pm$ 0.69	–
	$f_{0}$	19.81 $\pm$ 0.28	19.80 $\pm$ 0.28	–	31.48 $\pm$ 0.45	31.45 $\pm$ 0.45	–
	BPC	8.86 $\pm$ 0.15	9.43 $\pm$ 0.13	–	12.91 $\pm$ 0.21	13.73 $\pm$ 0.20	–
	CalibraEval	8.13 $\pm$ 0.33	11.05 $\pm$ 0.38	–	10.95 $\pm$ 0.48	14.87 $\pm$ 0.49	–
	PORTIA	10.47 $\pm$ 0.10	17.09 $\pm$ 0.14	–	14.50 $\pm$ 0.12	23.66 $\pm$ 0.21	–

Appendix

Table 12: Average per-group

\text{MCE}_{\%}

and

\text{MCE}_{\sigma}

(

\pm

standard error) for CaLLM and the mitigation baselines across datasets with Qwen 2.5-7B-Instruct used as the judge. The family axis contains AEX and PPE only: PKU , RWB and MTB do not contain qwen -family responses.

		$\text{MCE}_{\%}$			$\text{MCE}_{\sigma}$
Dataset	Method	Position	Verbosity	Family	Position	Verbosity	Family
PKU	CaLLM	2.78 $\pm$ 0.23	5.55 $\pm$ 0.28	–	3.85 $\pm$ 0.31	7.68 $\pm$ 0.38	–
	CaLLM (Qwen3-0.6B)	2.73 $\pm$ 0.22	7.39 $\pm$ 0.62	–	3.74 $\pm$ 0.31	10.12 $\pm$ 0.84	–
	best baseline	7.66 $\pm$ 0.19	19.25 $\pm$ 0.39	–	10.18 $\pm$ 0.26	27.26 $\pm$ 0.47	–
AEX	CaLLM	3.72 $\pm$ 0.35	4.49 $\pm$ 0.27	2.58 $\pm$ 0.16	4.27 $\pm$ 0.41	5.14 $\pm$ 0.29	2.96 $\pm$ 0.19
	CaLLM (Qwen3-0.6B)	4.17 $\pm$ 0.31	5.49 $\pm$ 0.28	2.28 $\pm$ 0.14	4.70 $\pm$ 0.36	6.19 $\pm$ 0.31	2.57 $\pm$ 0.16

Appendix

Table 13: Average per-group

\text{MCE}_{\%}

and

\text{MCE}_{\sigma}

(

\pm 1

s.e.) for CaLLM (embeddings from the judge’s last hidden state), CaLLM(Qwen3-0.6B) (embeddings from Qwen3-Embedding-0.6B ) and, as a reference, the best of the Fig. 2 baselines across datasets. Family is undefined on PKU , since it contains no response from the judge’s own family.

		$\text{MCE}_{\%}$			$\text{MCE}_{\sigma}$
Dataset	Method	Position	Verbosity	Family	Position	Verbosity	Family
PKU	CaLLM	2.78 $\pm$ 0.23	5.55 $\pm$ 0.28	–	3.85 $\pm$ 0.31	7.68 $\pm$ 0.38	–
	$f_{0}$	12.34 $\pm$ 0.40	22.47 $\pm$ 0.28	–	18.08 $\pm$ 0.61	32.92 $\pm$ 0.47	–
	Temperature	5.56 $\pm$ 0.28	19.84 $\pm$ 0.20	–	7.10 $\pm$ 0.36	25.32 $\pm$ 0.29	–
	Platt	5.29 $\pm$ 0.20	18.64 $\pm$ 0.51	–	6.76 $\pm$ 0.26	23.80 $\pm$ 0.64	–
	LOGR	2.87 $\pm$ 0.15	17.71 $\pm$ 0.48	–	3.68 $\pm$ 0.19	22.73 $\pm$ 0.61	–

Appendix

Table 14: Average per-group

\text{MCE}_{\%}

and

\text{MCE}_{\sigma}

(

\pm 1

s.e.) for CaLLM and the calibration and multicalibration baselines across datasets. Family is undefined on PKU , since it contains no response from the judge’s own family.

Bias in LLM-as-a-Judge

Bias in large language model (LLM)-as-a-judge evaluation studies systematic distortions in automated assessments of model outputs.

See all 115 papers

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration
Sep 10, 2026Gemma Zhang, Prachi Badarayani, Asmi Kumar +2LLM Evaluation LLM-as-a-Judge
Microsoft · Massachusetts Institute of Technology
Bias and Uncertainty in LLM-as-a-Judge Estimation
May 7, 2026James FiedlerLLM Evaluation LLM-as-a-Judge
Indeed Inc.
Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges
May 10, 2026Yanran LiLLM Evaluation LLM-as-a-Judge
Department of Biostatistics Columbia University

Sep 10, 2026J/K move · Enter open · S save

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

Gemma Zhang, Prachi Badarayani, Asmi Kumar, Sadid Hasan+1

Microsoft · Massachusetts Institute of Technology

LLM Evaluation LLM-as-a-Judge Language Model Calibration LLM Reliability Language Model Bias Evaluation Bias in LLM-as-a-Judge+1

Open paper AskPDF arXiv

LLMs are increasingly used as automated judges for model training and evaluation, yet individual judges exhibit systematic biases that undermine reliability. Much of prior work has studied biases in pairwise LLM-as-a-judge settings; in this paper, we focus on absolute scoring tasks, which mirror more realistic use cases. Across four benchmarks and six models (36 judge-examinee pairs), we show that a model's task accuracy strongly predicts its judging accuracy (Pearson

r \geq 0.90

on most models) and inversely predicts its directional bias (

r \leq -0.83

), but that accuracy alone does not ensure fair evaluation: more capable examinee models consistently receive more lenient judgments from all judges (

r \geq 0.83

). To address this, we propose calibrated weighted majority voting (WMV), an ensemble evaluation method that aggregates multiple LLM judges weighted by online estimates of their false-positive and false-negative rates. We introduce a disagreement-based estimator that derives these error rates purely from inter-judge agreement patterns, requiring no ground-truth labels or task metadata. In a simulated experiment with shifting task distributions, our label-free WMV tracks an oracle with perfect error-rate knowledge to within 0.5 percentage points on average, outperforming both individual judges and unweighted majority voting. These results demonstrate that principled multi-judge calibration can simultaneously improve accuracy and correct for systematic leniency without requiring labeled data, offering a scalable path to reliable automated evaluation as model capabilities increase.

Abstract

Bias in LLM-as-a-Judge

Explore similar work

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

Bias and Uncertainty in LLM-as-a-Judge Estimation

Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges