A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

Cesare Barbera; Luca Stradiotti; Lorenzo Cascioli; Lorenzo Perini; Niek Tax; Daniel Haimovich; Jesse Davis; Andrea Passerini

cs.LGOct 13, 2026

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

Authors: Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli, Lorenzo Perini, Niek Tax, Daniel Haimovich, Jesse Davis, Andrea Passerini,

Organizations: University of Pisa · University of Trento · KU Leuven · Meta Platforms, Inc.

Abstract

Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a scalable alternative when human evaluation is too costly. Unfortunately, LLM judgments can exhibit systematic biases. Prior work has documented position, verbosity or model family of the candidate answers influencing the judge decision rather than their quality, while additional biases may remain undocumented. $\textit{Multicalibration}$ provides a natural framework to address this problem: each bias corresponds to miscalibration on an identifiable group, $\textit{e.g.}$ , family bias manifests as overconfidence in answers generated by models belonging to the same family of the judge. The challenge is to fix miscalibration simultaneously across many, potentially unknown groups. To this end, we introduce $\mathrm{CaLLM}$ , a post hoc framework that multicalibrates an LLM judge, thereby bounding all such biases simultaneously. We prove that $\mathrm{CaLLM}$ bounds the bias of every group whose error structure is captured by an available representation, and that augmenting it with explicit bias groups further reduces the residual bias. We then derive theoretical conditions under which $\mathrm{CaLLM}$ improves the judge's accuracy, rather than trading it for calibration. Finally, we evaluate $\mathrm{CaLLM}$ across pairwise preference benchmarks, showing that it reduces the multicalibration error on both specified and unspecified bias groups, while improving predictive performance of the judge.

Figures & tables

Figure 1: CaLLM ’s pipeline. We feed CaLLM with some representation $\phi(X)$ of the input (e.g., embeddings $E(X)$ and base scores $f_{0}(X)$ ). The final output is a multi-calibrated score $f^{\star}(X)$ .

Figure 2: Average per-axis

\text{MCE}_{\%}

(

\pm

s.e.) for each method across datasets. PKU doesn’t contain the information regarding the LLMs that generated the responses, so we do not report Family bias.

	PKU		AEX		PPE		RWB		MTB
	CaLLM	CaLLM $+g$	CaLLM	CaLLM $+g$	CaLLM	CaLLM $+g$	CaLLM	CaLLM $+g$	CaLLM	CaLLM $+g$
Position	2.8 $\pm$ 0.2	2.6 $\pm$ 0.1	3.7 $\pm$ 0.3	2.5 $\pm$ 0.3	5.4 $\pm$ 0.2	3.7 $\pm$ 0.3	6.9 $\pm$ 1.1	4.7 $\pm$ 0.5	13.4 $\pm$ 1.6	12.5 $\pm$ 1.5
Verbosity	5.6 $\pm$ 0.3	2.8 $\pm$ 0.2	4.5 $\pm$ 0.3	2.5 $\pm$ 0.2	7.5 $\pm$ 0.4	4.0 $\pm$ 0.2	5.2 $\pm$ 0.4	4.7 $\pm$ 0.4	13.5 $\pm$ 1.3	12.3 $\pm$ 0.9
Family	–	–	2.6 $\pm$ 0.2	2.5 $\pm$ 0.2	4.8 $\pm$ 0.3	4.6 $\pm$ 0.4	5.2 $\pm$ 0.3	4.6 $\pm$ 0.1	11.9 $\pm$ 0.8	11.6 $\pm$ 0.2

Table 1: Comparison between CaLLM and CaLLM

\!\!+g

in terms of

MCE_{\%}

for documented biases. PKU doesn’t contain the information regarding the LLMs that generated the responses, so we do not report Family bias.

	PKU		AEX		PPE		RWB		MTB
	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$
CaLLM	.69 $\pm$ .01	.57 $\pm$ .01	.68 $\pm$ .01	.60 $\pm$ .01	.67 $\pm$ .01	.61 $\pm$ .01	.93 $\pm$ .01	.21 $\pm$ .01	.81 $\pm$ .01	.41 $\pm$ .02
$f_{0}$	.63 $\pm$ .01	.73 $\pm$ .01	.63 $\pm$ .01	.76 $\pm$ .01	.64 $\pm$ .01	.74 $\pm$ .01	.86 $\pm$ .01	.36 $\pm$ .02	.79 $\pm$ .01	.49 $\pm$ .02
BPC	.64 $\pm$ .01	.67 $\pm$ .01	.65 $\pm$ .01	.69 $\pm$ .01	.65 $\pm$ .01	.68 $\pm$ .01	.88 $\pm$ .01	.29 $\pm$ .01	.80 $\pm$ .01	.43 $\pm$ .02
CalibraEval	.64 $\pm$ .01	.65 $\pm$ .01	.65 $\pm$ .01	.69 $\pm$ .01	.65 $\pm$ .01	.66 $\pm$ .01	.88 $\pm$ .01	.30 $\pm$ .01	.81 $\pm$ .01	.43 $\pm$ .02
PORTIA	.61 $\pm$ .01	.72 $\pm$ .01	.65 $\pm$ .01	.72 $\pm$ .01	.66 $\pm$ .01	.70 $\pm$ .01	.87 $\pm$ .01	.31 $\pm$ .02	.77 $\pm$ .02	.55 $\pm$ .05

Table 2: Average ACC and log-loss

\mathcal{L}_{\log}

(

\pm

s.e.) across datasets. CaLLM always matches or improves both metrics over all baselines, showing that multicalibration also improves predictive performance.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Average per-axis

\text{MCE}_{\%}

(

\pm

s.e.) on the specified and unspecified groups for each method across datasets.

Figure 4: Average per-group

\text{MCE}_{\%}

(

\pm

s.e.) for CaLLM and the mitigation baselines across datasets with Qwen 2.5-7B-Instruct used as the judge. The self-enhancement axis contains AEX and PPE only: PKU , RWB and MTB do not contain qwen -family responses.

	PKU		AEX		PPE		RWB		MTB
	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$
Position	2.4 $\pm$ 0.2	2.4 $\pm$ 0.2	6.8 $\pm$ 0.4	2.4 $\pm$ 0.3	8.3 $\pm$ 0.5	3.7 $\pm$ 0.5	10.0 $\pm$ 1.2	7.9 $\pm$ 1.1	11.8 $\pm$ 0.8	12.0 $\pm$ 0.8
Verbosity	6.3 $\pm$ 0.5	2.8 $\pm$ 0.2	4.6 $\pm$ 0.3	2.3 $\pm$ 0.1	7.2 $\pm$ 0.4	4.0 $\pm$ 0.3	9.7 $\pm$ 1.0	7.4 $\pm$ 0.8	14.4 $\pm$ 0.7	13.5 $\pm$ 1.1
Family	–	–	2.7 $\pm$ 0.3	2.1 $\pm$ 0.2	5.7 $\pm$ 0.9	4.7 $\pm$ 0.9	–	–	–	–

Appendix

Table 3:

\text{MCE}_{\%}

per bias group (

\pm 1

s.e.) for CaLLM and CaLLM

\!\!+g

using the Qwen 2.5-7B-Instruct judge. The family axis contains results for AEX and PPE only: PKU , RWB and MTB contain no qwen -family response.

	PKU		AEX		PPE		RWB		MTB
	Acc	$\mathcal{L}_{\log}$	Acc	$\mathcal{L}_{\log}$	Acc	$\mathcal{L}_{\log}$	Acc	$\mathcal{L}_{\log}$	Acc	$\mathcal{L}_{\log}$
CaLLM	.68 $\pm$ .01	.60 $\pm$ .01	.61 $\pm$ .01	.65 $\pm$ .01	.61 $\pm$ .01	.66 $\pm$ .01	.84 $\pm$ .01	.37 $\pm$ .01	.80 $\pm$ .01	.45 $\pm$ .02
$f_{0}$	.66 $\pm$ .01	.72 $\pm$ .01	.58 $\pm$ .01	.88 $\pm$ .01	.59 $\pm$ .01	.87 $\pm$ .01	.80 $\pm$ .01	.50 $\pm$ .02	.80 $\pm$ .01	.51 $\pm$ .02
BPC	.66 $\pm$ .01	.65 $\pm$ .01	.59 $\pm$ .01	.71 $\pm$ .01	.60 $\pm$ .01	.72 $\pm$ .01	.80 $\pm$ .01	.42 $\pm$ .01	.80 $\pm$ .01	.46 $\pm$ .02
CalibraEval	.66 $\pm$ .01	.63 $\pm$ .01	.59 $\pm$ .01	.68 $\pm$ .01	.60 $\pm$ .01	.69 $\pm$ .01	.80 $\pm$ .01	.43 $\pm$ .01	.80 $\pm$ .01	.47 $\pm$ .02
PORTIA	.59 $\pm$ .01	.71 $\pm$ .01	.59 $\pm$ .01	.75 $\pm$ .01	.60 $\pm$ .01	.74 $\pm$ .01	.76 $\pm$ .01	.50 $\pm$ .01	.70 $\pm$ .03	.59 $\pm$ .06

Appendix

Table 4: Average accuracy and log-loss (

\pm 1

s.e.) across datasets using Qwen 2.5-7B-Instruct as judge.

Figure 5: Average per-group

\text{MCE}_{\%}

(

\pm 1

standard error) for CaLLM (embeddings from the judge’s last hidden state), CaLLM(Qwen3-0.6B) (embeddings from Qwen3-Embedding-0.6B ) and, as a reference, the best of the Fig. 2 baselines across datasets. Family is undefined on PKU , since it contains no response from the judge’s own family.

	PKU		AEX		PPE		RWB		MTB
	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$	CaLLM	CaLLM $\!\!+g$
Position	2.7 $\pm$ 0.2	2.2 $\pm$ 0.1	4.2 $\pm$ 0.3	2.1 $\pm$ 0.2	5.7 $\pm$ 0.3	4.1 $\pm$ 0.2	6.7 $\pm$ 1.0	5.6 $\pm$ 0.5	14.2 $\pm$ 2.5	11.9 $\pm$ 1.5
Verbosity	7.4 $\pm$ 0.6	2.4 $\pm$ 0.1	5.5 $\pm$ 0.3	2.3 $\pm$ 0.2	8.6 $\pm$ 0.5	4.3 $\pm$ 0.1	7.9 $\pm$ 0.7	5.2 $\pm$ 0.5	14.3 $\pm$ 1.7	11.7 $\pm$ 1.4
Family	–	–	2.3 $\pm$ 0.1	2.2 $\pm$ 0.1	5.1 $\pm$ 0.3	5.1 $\pm$ 0.3	6.0 $\pm$ 0.6	6.6 $\pm$ 0.7	10.4 $\pm$ 0.8	11.1 $\pm$ 1.7

Appendix

Table 5:

\text{MCE}_{\%}

per bias group (

\pm 1

standard error) for CaLLM and CaLLM

\!\!+g

across datasets using Qwen3-Embedding-0.6B embeddings. Self-enhancement is undefined on PKU because the dataset does not contain muse -family responses.

	PKU		AEX		PPE		RWB		MTB
	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$
CaLLM	.69 $\pm$ .01	.57 $\pm$ .01	.68 $\pm$ .01	.60 $\pm$ .01	.67 $\pm$ .01	.61 $\pm$ .01	.93 $\pm$ .01	.21 $\pm$ .01	.81 $\pm$ .01	.41 $\pm$ .02
CaLLM (Qwen3-0.6B)	.69 $\pm$ .01	.58 $\pm$ .01	.66 $\pm$ .01	.61 $\pm$ .01	.66 $\pm$ .01	.62 $\pm$ .01	.91 $\pm$ .01	.27 $\pm$ .01	.81 $\pm$ .01	.42 $\pm$ .02
best baseline	.64 $\pm$ .01	.65 $\pm$ .01	.65 $\pm$ .01	.69 $\pm$ .01	.66 $\pm$ .01	.66 $\pm$ .01	.88 $\pm$ .01	.29 $\pm$ .01	.81 $\pm$ .01	.43 $\pm$ .02

Appendix

Table 6: Average accuracy and log-loss (

\pm

s.e.) for CaLLM and its ablated variant CaLLM(Qwen3-0.6B) across datasets. best baseline is the strongest of the Fig. 2 baselines on each (dataset, metric) pair.

Figure 6: Average per-group

\text{MCE}_{\%}

(

\pm 1

s.e.) across datasets for CaLLM and the calibration and multicalibration baselines. Family is undefined on PKU , since it contains no response from the judge’s own family.

	PKU		AEX		PPE		RWB		MTB
	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$	ACC	$\mathcal{L}_{\log}$
CaLLM	.69 $\pm$ .01	.57 $\pm$ .01	.68 $\pm$ .01	.60 $\pm$ .01	.67 $\pm$ .01	.61 $\pm$ .01	.93 $\pm$ .01	.21 $\pm$ .01	.81 $\pm$ .01	.41 $\pm$ .02
$f_{0}$	.63 $\pm$ .01	.73 $\pm$ .01	.63 $\pm$ .01	.76 $\pm$ .01	.64 $\pm$ .01	.74 $\pm$ .01	.86 $\pm$ .01	.36 $\pm$ .02	.79 $\pm$ .01	.49 $\pm$ .02
Temperature	.64 $\pm$ .01	.63 $\pm$ .01	.65 $\pm$ .01	.64 $\pm$ .01	.65 $\pm$ .01	.63 $\pm$ .01	.88 $\pm$ .01	.30 $\pm$ .01	.80 $\pm$ .01	.43 $\pm$ .01
Platt	.64 $\pm$ .01	.63 $\pm$ .01	.65 $\pm$ .01	.64 $\pm$ .01	.65 $\pm$ .01	.63 $\pm$ .01	.88 $\pm$ .01	.30 $\pm$ .01	.80 $\pm$ .01	.43 $\pm$ .01
LOGR	.64 $\pm$ .01	.62 $\pm$ .01	.65 $\pm$ .01	.63 $\pm$ .01	.65 $\pm$ .01	.63 $\pm$ .01	.88 $\pm$ .01	.30 $\pm$ .01	.81 $\pm$ .01	.43 $\pm$ .01

Appendix

Table 7: Average ACC and log-loss

\mathcal{L}_{\log}

(

\pm

s.e.) across datasets. CaLLM always matches or improves both metrics over all calibration and multicalibration strategies.

	PKU	AEX	PPE	RWB	MTB
Position	0.50 $\pm$ 0.01	0.76 $\pm$ 0.01	0.76 $\pm$ 0.01	0.93 $\pm$ 0.01	0.66 $\pm$ 0.05
Verbosity	0.92 $\pm$ 0.01	0.90 $\pm$ 0.01	0.89 $\pm$ 0.01	0.95 $\pm$ 0.01	0.94 $\pm$ 0.02
Family	-	0.98 $\pm$ 0.01	0.98 $\pm$ 0.01	0.99 $\pm$ 0.01	0.98 $\pm$ 0.02

Appendix

Table 8: Recoverability results across datasets.

		$\text{MCE}_{\%}$			$\text{MCE}_{\sigma}$
Data	Method	Position	Verbosity	Family	Position	Verbosity	Family
PKU	CaLLM	2.78 $\pm$ 0.23	5.55 $\pm$ 0.28	–	3.85 $\pm$ 0.31	7.68 $\pm$ 0.38	–
	$f_{0}$	12.34 $\pm$ 0.40	22.47 $\pm$ 0.28	–	18.08 $\pm$ 0.61	32.92 $\pm$ 0.47	–
	BPC	7.91 $\pm$ 0.25	19.25 $\pm$ 0.39	–	11.22 $\pm$ 0.36	27.28 $\pm$ 0.56	–
	CalibraEval	7.66 $\pm$ 0.19	20.51 $\pm$ 0.35	–	10.18 $\pm$ 0.26	27.26 $\pm$ 0.47	–
	PORTIA	10.26 $\pm$ 0.10	21.06 $\pm$ 0.31	–	14.73 $\pm$ 0.19	30.22 $\pm$ 0.45	–

Appendix

Table 9: Average per-group

\text{MCE}_{\%}

and

\text{MCE}_{\sigma}

(

\pm

standard error) for CaLLM and the mitigation baselines across datasets. The family axis does not show PKU since it does not contain muse -family responses.

		$\text{MCE}_{\%}$			$\text{MCE}_{\sigma}$
Dataset	Method	Position	Verbosity	Family	Position	Verbosity	Family
PKU	CaLLM	2.78 $\pm$ 0.23	5.55 $\pm$ 0.28	–	3.85 $\pm$ 0.31	7.68 $\pm$ 0.38	–
PKU	CaLLM $\!\!+g$	2.56 $\pm$ 0.10	2.81 $\pm$ 0.18	–	3.58 $\pm$ 0.14	3.92 $\pm$ 0.25	–
AEX	CaLLM	3.72 $\pm$ 0.35	4.49 $\pm$ 0.27	2.58 $\pm$ 0.16	4.27 $\pm$ 0.41	5.14 $\pm$ 0.29	2.96 $\pm$ 0.19
AEX	CaLLM $\!\!+g$	2.52 $\pm$ 0.30	2.55 $\pm$ 0.24	2.50 $\pm$ 0.19	2.90 $\pm$ 0.34	2.93 $\pm$ 0.29	2.87 $\pm$ 0.22
PPE	CaLLM	5.41 $\pm$ 0.18	7.48 $\pm$ 0.41	4.77 $\pm$ 0.34	2.63 $\pm$ 0.10	3.62 $\pm$ 0.19	2.32 $\pm$ 0.17

Appendix

Table 10: Average per-group

\text{MCE}_{\%}

and

\text{MCE}_{\sigma}

(

\pm

standard error) for CaLLM and CaLLM

\!\!+g

. The family axis does not show PKU since it does not contain muse -family responses.

		$\text{MCE}_{\%}$		$\texttt{MCE}_{\sigma}$
Dataset	Method	Specified	Unspecified	Specified	Unspecified
PKU	CaLLM	5.55 $\pm$ 0.28	3.89 $\pm$ 0.36	7.68 $\pm$ 0.38	5.38 $\pm$ 0.50
	$f_{0}$	22.46 $\pm$ 0.28	12.34 $\pm$ 0.40	32.91 $\pm$ 0.47	18.08 $\pm$ 0.61
	BPC	19.23 $\pm$ 0.40	7.91 $\pm$ 0.25	27.26 $\pm$ 0.56	11.22 $\pm$ 0.36
	CalibraEval	20.49 $\pm$ 0.35	7.66 $\pm$ 0.19	27.24 $\pm$ 0.47	10.18 $\pm$ 0.26
	PORTIA	21.04 $\pm$ 0.31	10.26 $\pm$ 0.10	30.20 $\pm$ 0.45	14.73 $\pm$ 0.19

Appendix

Table 11: Average per-group

\text{MCE}_{\%}

and

\text{MCE}_{\sigma}

(

\pm

standard error) for CaLLM and the mitigation baselines across datasets over the three biases documented in the literature ( i.e. position, verbosity, and family) and the unspecified groups.

		$\text{MCE}_{\%}$			$\text{MCE}_{\sigma}$
Dataset	Method	Position	Verbosity	Family	Position	Verbosity	Family
PKU	CaLLM	2.39 $\pm$ 0.17	6.32 $\pm$ 0.53	–	3.17 $\pm$ 0.23	8.39 $\pm$ 0.69	–
	$f_{0}$	19.81 $\pm$ 0.28	19.80 $\pm$ 0.28	–	31.48 $\pm$ 0.45	31.45 $\pm$ 0.45	–
	BPC	8.86 $\pm$ 0.15	9.43 $\pm$ 0.13	–	12.91 $\pm$ 0.21	13.73 $\pm$ 0.20	–
	CalibraEval	8.13 $\pm$ 0.33	11.05 $\pm$ 0.38	–	10.95 $\pm$ 0.48	14.87 $\pm$ 0.49	–
	PORTIA	10.47 $\pm$ 0.10	17.09 $\pm$ 0.14	–	14.50 $\pm$ 0.12	23.66 $\pm$ 0.21	–

Appendix

Table 12: Average per-group

\text{MCE}_{\%}

and

\text{MCE}_{\sigma}

(

\pm

standard error) for CaLLM and the mitigation baselines across datasets with Qwen 2.5-7B-Instruct used as the judge. The family axis contains AEX and PPE only: PKU , RWB and MTB do not contain qwen -family responses.

		$\text{MCE}_{\%}$			$\text{MCE}_{\sigma}$
Dataset	Method	Position	Verbosity	Family	Position	Verbosity	Family
PKU	CaLLM	2.78 $\pm$ 0.23	5.55 $\pm$ 0.28	–	3.85 $\pm$ 0.31	7.68 $\pm$ 0.38	–
	CaLLM (Qwen3-0.6B)	2.73 $\pm$ 0.22	7.39 $\pm$ 0.62	–	3.74 $\pm$ 0.31	10.12 $\pm$ 0.84	–
	best baseline	7.66 $\pm$ 0.19	19.25 $\pm$ 0.39	–	10.18 $\pm$ 0.26	27.26 $\pm$ 0.47	–
AEX	CaLLM	3.72 $\pm$ 0.35	4.49 $\pm$ 0.27	2.58 $\pm$ 0.16	4.27 $\pm$ 0.41	5.14 $\pm$ 0.29	2.96 $\pm$ 0.19
	CaLLM (Qwen3-0.6B)	4.17 $\pm$ 0.31	5.49 $\pm$ 0.28	2.28 $\pm$ 0.14	4.70 $\pm$ 0.36	6.19 $\pm$ 0.31	2.57 $\pm$ 0.16

Appendix

Table 13: Average per-group

\text{MCE}_{\%}

and

\text{MCE}_{\sigma}

(

\pm 1

s.e.) for CaLLM (embeddings from the judge’s last hidden state), CaLLM(Qwen3-0.6B) (embeddings from Qwen3-Embedding-0.6B ) and, as a reference, the best of the Fig. 2 baselines across datasets. Family is undefined on PKU , since it contains no response from the judge’s own family.

		$\text{MCE}_{\%}$			$\text{MCE}_{\sigma}$
Dataset	Method	Position	Verbosity	Family	Position	Verbosity	Family
PKU	CaLLM	2.78 $\pm$ 0.23	5.55 $\pm$ 0.28	–	3.85 $\pm$ 0.31	7.68 $\pm$ 0.38	–
	$f_{0}$	12.34 $\pm$ 0.40	22.47 $\pm$ 0.28	–	18.08 $\pm$ 0.61	32.92 $\pm$ 0.47	–
	Temperature	5.56 $\pm$ 0.28	19.84 $\pm$ 0.20	–	7.10 $\pm$ 0.36	25.32 $\pm$ 0.29	–
	Platt	5.29 $\pm$ 0.20	18.64 $\pm$ 0.51	–	6.76 $\pm$ 0.26	23.80 $\pm$ 0.64	–
	LOGR	2.87 $\pm$ 0.15	17.71 $\pm$ 0.48	–	3.68 $\pm$ 0.19	22.73 $\pm$ 0.61	–

Appendix

Table 14: Average per-group

\text{MCE}_{\%}

and

\text{MCE}_{\sigma}

(

\pm 1

s.e.) for CaLLM and the calibration and multicalibration baselines across datasets. Family is undefined on PKU , since it contains no response from the judge’s own family.

Bias in LLM-as-a-Judge

Bias in large language model (LLM)-as-a-judge evaluation studies systematic distortions in automated assessments of model outputs.

See all 115 papers

Sep 10, 2026

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

LLMs are increasingly used as automated judges for model training and evaluation, yet individual judges exhibit systematic biases that undermine reliability. Much of prior work has studied biases in pairwise LLM-as-a-judge settings; in this paper, we focus on absolute scoring tasks, which mirror more realistic use cases. Across four benchmarks and six models (36 judge-examinee pairs), we show that a model's task accuracy strongly predicts its judging accuracy (Pearson

r \geq 0.90

on most models) and inversely predicts its directional bias (

r \leq -0.83

), but that accuracy alone does not ensure fair evaluation: more capable examinee models consistently receive more lenient judgments from all judges (

r \geq 0.83

). To address this, we propose calibrated weighted majority voting (WMV), an ensemble evaluation method that aggregates multiple LLM judges weighted by online estimates of their false-positive and false-negative rates. We introduce a disagreement-based estimator that derives these error rates purely from inter-judge agreement patterns, requiring no ground-truth labels or task metadata. In a simulated experiment with shifting task distributions, our label-free WMV tracks an oracle with perfect error-rate knowledge to within 0.5 percentage points on average, outperforming both individual judges and unweighted majority voting. These results demonstrate that principled multi-judge calibration can simultaneously improve accuracy and correct for systematic leniency without requiring labeled data, offering a scalable path to reliable automated evaluation as model capabilities increase.

Gemma Zhang, Prachi Badarayani, Asmi Kumar +2

Microsoft · Massachusetts Institute of Technology

LLM Evaluation LLM-as-a-Judge Language Model Calibration LLM Reliability Language Model Bias Evaluation Bias in LLM-as-a-Judge+1

Ask PDF

May 7, 2026

Bias and Uncertainty in LLM-as-a-Judge Estimation

LLM-as-a-Judge evaluation has become a standard tool for assessing base model performance. However, characterizing performance via the naive estimator, i.e., raw judge outputs, is systematically biased. Recent work has proposed estimators to correct this bias, but their reliability depends critically on judge quality and, for model comparisons, on calibration stability. Sharing calibration across compared models is practically attractive but can introduce severe bias, including cases where the comparison estimate points in the wrong direction with high apparent confidence. We study these failure modes through analytical results, simulations over judge quality (

J

) and cross-model calibration instability (

ΔJ

), and a real-data MMLU-Pro case study with sign reversal. We propose

J

and

ΔJ

as diagnostics for when corrected estimates, especially shared-calibration comparisons, are likely unreliable, and provide reporting guidance for LaaJ evaluation.

James Fiedler

Indeed Inc.

LLM Evaluation LLM-as-a-Judge Language Model Calibration LLM Uncertainty Estimation Bias in LLM-as-a-Judge

Ask PDF

May 10, 2026

Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges

Multi-judge evaluation is increasingly used to assess LLMs and reward models, and the prevailing heuristic is to curate: keep the most accurate judges and discard weaker ones. We show that this heuristic can reverse when the target is not point accuracy, but calibrated probabilistic evaluation from a labeled calibration set. Holding the aggregation and calibration procedures fixed, we compare accuracy-ranked top-

k

judge selection with using the full judge panel. Across four labeled pairwise-evaluation benchmarks spanning LLM-as-judge and reward-model settings, the calibrated full panel consistently outperforms accuracy-based selection. On RewardBench2, retaining all judges achieves negative log-likelihood (NLL) of

0.006

versus

0.013

under top-5 selection, halving the calibration error. This advantage persists after judge-family deduplication and against stronger same-pipeline subset search. We explain this reversal with oracle analyses showing that the optimal calibrated risk under proper scoring rules cannot increase when additional judge signals are made available, and that even below-chance judges can be useful when their biases are learnable and their signals are non-redundant. The resulting operating principle is simple: in multi-judge evaluation with labeled calibration data, do not discard weak judges by accuracy alone; keep them when they are parseable, non-redundant, and calibratable.

Yanran Li

Department of Biostatistics Columbia University

LLM Evaluation LLM-as-a-Judge Pairwise Preference Evaluation Language Model Calibration Language Model Ensembles

Ask PDF

Abstract

Bias in LLM-as-a-Judge

Explore similar work

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

Bias and Uncertainty in LLM-as-a-Judge Estimation

Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges