Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

José Pombal; Ricardo Rei; André F. T. Martins

cs.CLApr 8, 2026

Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

Authors: José Pombal, Ricardo Rei, André F. T. Martins

Organizations: Sword Health · Instituto de Telecomunicações · Instituto Superior Técnico, Universidade de Lisboa · TransPerfect ·

Abstract

LLM-as-a-judge has become the de facto approach for evaluating LLM outputs. However, judges are known to exhibit self-preference bias (SPB): they tend to favor outputs produced by themselves or by models from their own family. This skews evaluations and, thus, hinders model development, especially in settings of recursive self-improvement. We present the first study of SPB in rubric-based evaluation, an increasingly popular benchmarking paradigm where judges issue binary verdicts on individual evaluation criteria, instead of assigning holistic scores or rankings. Using IFEval and LiveCodeBench, benchmarks with programmatically verifiable rubrics, we show that SPB persists even when evaluation criteria are entirely objective: among rubrics where generators fail, judges can be more than 50% more likely to incorrectly mark them as satisfied when the output is their own. We also find that, similarly to other evaluation paradigms, ensembling multiple judges helps mitigate SPB, but without fully eliminating it. On HealthBench, a medical chat benchmark with subjective rubrics, we observe that SPB skews model scores by up to 10 points, a potentially decisive margin when ranking frontier models. We analyze the factors that drive SPB in this setting, finding that negative rubrics and subjective topics like communication and emergency referrals are particularly susceptible.

Figures & tables

Figure 1: Illustration of three LLM-as-a-Judge paradigms: pairwise comparison (PWC), direct assessment (DA) and rubric-based evaluation (RB).

	IFEval		LiveCodeBench		HealthBench
Judge	MRA	HSPP-Rub. (Self)	MRA	HSPP-Rub. (Self)	MRA	HSPP-Rub. (Self)
$\bm{\cdot}$ Gemma-4B	0.89 [0.89, 0.90]	1.03 [1.00, 1.06]	0.46 [0.45, 0.46]	1.54 [1.38, 1.73]	0.71 [0.70, 0.71]	1.16 [1.14, 1.17]
$\bm{\cdot}$ Gemma-12B	0.89 [0.88, 0.89]	1.03 [0.95, 1.11]	0.74 [0.74, 0.75]	0.97 [0.93, 1.01]	0.77 [0.77, 0.77]	1.03 [1.01, 1.05]
$\bm{\cdot}$ Gemma-27B	0.89 [0.89, 0.90]	1.08 [1.02, 1.14]	0.77 [0.77, 0.78]	1.03 [0.97, 1.08]	0.83 [0.83, 0.83]	1.10 [1.08, 1.12]
$\bm{\cdot}$ Llama-Scout	0.84 [0.83, 0.85]	1.05 [0.91, 1.20]	0.77 [0.77, 0.77]	1.17 [1.14, 1.20]	0.83 [0.83, 0.83]	0.80 [0.79, 0.82]
$\bm{\cdot}$ Llama-Mav	0.88 [0.87, 0.88]	1.12 [0.98, 1.27]	0.79 [0.78, 0.79]	1.34 [1.30, 1.38]	0.90 [0.90, 0.90]	0.71 [0.69, 0.73]

Table 1: Rubric-level metrics per judge across IFEval, LiveCodeBench, and HealthBench (SR), with 95% confidence intervals (rubric-level bootstrap, 1000 resamples) in grey. MRA = Mean Rubric Accuracy; HSPP-Rub. (Self) = rubric-level HSPP ratio for self-generated outputs. Family-level ratios are in Appendix D .

Panel of Figure 2 — Figure 2: Performance vs. self-preference bias on IFEval across judge paradigms. SR shows also shows GPT-5 and Claude models (light orange squares), which we did not run in other methods for cost purposes. Left: Instance-level (MIPA vs. HSPP-Instance ratio). Right: Rubric-level (MRA vs. HSPP-Rubric ratio).

Panel of Figure 3 — Figure 3: Impact of a 5-family committee on rubric-level self-preference bias (left) and accuracy (right) on IFEval (SR). Lines connect each committee member’s individual performance to its committee-aggregated performance.

Panel of Figure 4 — Figure 4: HSPP-Instance (left) and HSPP-Rubric (right) when filtering rubrics by inter-judge pairwise agreement threshold on IFEval (SR, 12 judges, mean over all judges).

		Mean-MRA bin (proxy for evaluation “hardness”)
Judge	Original	0.5–0.6	0.6–0.7	0.7–0.8	0.8–0.9
$\bm{\cdot}$ GPT-5	20.15 [16.57, 24.58]	13.74 [9.94, 19.29]	14.66 [8.94, 21.48]	7.36 [1.50, 15.55]	0.00 [0.00, 0.00]
$\bm{\cdot}$ GPT-120B	5.97 [4.91, 7.18]	1.13 [0.59, 1.84]	1.98 [0.83, 3.24]	3.98 [1.48, 7.33]	7.35 [1.34, 21.10]
$\bm{\cdot}$ Claude-Sonnet	1.78 [1.58, 1.99]	1.86 [1.56, 2.14]	1.33 [1.11, 1.60]	2.06 [1.45, 2.89]	2.76 [0.89, 5.22]
$\bm{\cdot}$ Qwen-235B	0.83 [0.62, 1.07]	0.78 [0.40, 1.23]	0.93 [0.58, 1.38]	0.25 [0.06, 0.61]	2.28 [0.79, 4.49]

Table 2: LiveCodeBench rubric-level HSPP (Self) stratified by generation hardness: ratios are recomputed using only (generator, instance) cells whose mean rubric accuracy across all 12 judges falls in the bin. “Original” is the unstratified ratio from Table 1 . Null values occur when there are no instances in the bin for that generator.

Panel of Figure 5 — Figure 5: Centered score delta matrix for HealthBench (SR, weighted scoring), scaled by $\times 100$ . Each cell shows judge $j$ ’s deviation for generator $g$ , centered by $j$ ’s average bias. Orange = relative overestimation; blue = relative underestimation. Bordered cells indicate self-evaluation, while dotted areas represent within-family evaluation. See confidence intervals in Table 28 .

Panel of Figure 6 — Figure 6: Interaction coefficients $\textit{is\_self}\times X$ (posterior mean) of a mixed-effects logistic regression of rubric-level overestimation on HealthBench, with random intercepts on judge and instance (Appendix K ). Positive values mean property $X$ amplifies overestimation more when the judge evaluates its own output , i.e., it reinforces self-preference. Left to right: rubric properties (negative polarity, rubric length); rubric axes (reference = Accuracy); themes (reference = untagged). For readability, only the lower half of each 95% interval is drawn and the $y$ -axis is compressed above 1; full intervals are in Table 24 .

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Prompt template for single-rubric (SR) evaluation on IFEval.

Panel of Figure 8 — Figure 8: Prompt template for all-rubrics (AR) evaluation on IFEval.

Panel of Figure 9 — Figure 9: Prompt template for direct assessment (DA) evaluation on IFEval.

Panel of Figure 10 — Figure 10: Prompt template for pairwise comparison (PWC) evaluation on IFEval.

Panel of Figure 11 — Figure 11: Prompt template for single-rubric (SR) evaluation on HealthBench.

	Self	Other
	l2w / l2t (%)	l2w / l2t (%)
$\bm{\cdot}$ Gemma-4B	1.9 / 98.1	1.4 / 98.6
$\bm{\cdot}$ Gemma-12B	4.5 / 95.5	3.2 / 96.8
$\bm{\cdot}$ Gemma-27B	2.0 / 98.0	2.6 / 97.4
$\bm{\cdot}$ Llama-Scout	6.8 / 93.2	6.2 / 93.8
$\bm{\cdot}$ Llama-Mav	3.3 / 96.7	5.1 / 94.9

Appendix

Table 3: Breakdown of instance-level overestimation sub-types on IFEval (SR, 12 judges). Each cell shows the percentage of overestimations attributable to each sub-type: loss

\to

win (l2w) or loss

\to

tie (l2t).

Judge	Pairwise F1 (Bal.)
$\bm{\cdot}$ Gemma-27B	0.61
$\bm{\cdot}$ Llama-Mav	0.64
$\bm{\cdot}$ Qwen-235B	0.68
$\bm{\cdot}$ GPT-5	0.64
$\bm{\cdot}$ Claude-Sonnet	0.64

Appendix

Table 4: HealthBench meta-evaluation results (pairwise model balanced F1) for a subset of judges.

	HSPP-Rub. (Fam.)
Judge	IFEval	LiveCodeBench	HealthBench
$\bm{\cdot}$ Gemma-4B	1.03 [1.01, 1.05]	1.27 [1.15, 1.38]	1.22 [1.21, 1.23]
$\bm{\cdot}$ Gemma-12B	1.04 [0.99, 1.09]	0.94 [0.91, 0.96]	1.02 [1.01, 1.03]
$\bm{\cdot}$ Gemma-27B	1.03 [0.97, 1.07]	1.04 [1.01, 1.07]	1.10 [1.09, 1.12]
$\bm{\cdot}$ Llama-Scout	1.06 [0.93, 1.18]	1.20 [1.17, 1.22]	0.78 [0.77, 0.80]
$\bm{\cdot}$ Llama-Mav	1.11 [1.01, 1.21]	1.20 [1.17, 1.23]	0.72 [0.71, 0.74]

Appendix

Table 5: Family-level rubric HSPP ratios with 95% confidence intervals (rubric-level bootstrap, 1000 resamples).

Judge	MIPA	HSPP-Inst. (Self)	HSPP-Inst. (Fam.)
$\bm{\cdot}$ Gemma-4B	0.83 [0.81, 0.84]	1.03 [0.99, 1.07]	1.03 [1.00, 1.06]
$\bm{\cdot}$ Gemma-12B	0.81 [0.79, 0.83]	1.03 [0.97, 1.11]	1.05 [1.00, 1.10]
$\bm{\cdot}$ Gemma-27B	0.82 [0.81, 0.84]	1.10 [1.05, 1.15]	1.03 [0.98, 1.08]
$\bm{\cdot}$ Llama-Scout	0.77 [0.75, 0.79]	1.01 [0.92, 1.10]	1.06 [0.99, 1.12]
$\bm{\cdot}$ Llama-Mav	0.79 [0.77, 0.81]	1.20 [1.08, 1.30]	1.16 [1.07, 1.24]
$\bm{\cdot}$ Qwen-4B	0.78 [0.76, 0.80]	1.02 [0.91, 1.11]	1.03 [0.95, 1.10]

Appendix

Table 6: IFEval instance-level metrics (SR) per judge with 95% confidence intervals (instance-level bootstrap, 1000 resamples).

Member	HSPP individual	HSPP committee	$\Delta$ (committee $-$ individual)
Gemma-27B	1.08 [1.02, 1.14]	1.08 [0.94, 1.24]	-0.00 [-0.16, +0.17]
Llama-Mav	1.12 [0.98, 1.27]	0.97 [0.78, 1.17]	-0.15 [-0.40, +0.09]
Qwen-235B	1.30 [1.11, 1.48]	1.19 [0.99, 1.39]	-0.10 [-0.37, +0.17]
GPT-5	1.47 [0.48, 2.53]	1.10 [0.84, 1.36]	-0.37 [-1.49, +0.69]
Claude-Sonnet	0.98 [0.64, 1.32]	0.99 [0.79, 1.18]	+0.01 [-0.40, +0.40]

Appendix

Table 7: IFEval: rubric-level self-preference of each committee member vs. the majority-vote committee on the member’s outputs, sharing the member’s non-family baseline.

\Delta<0

means ensembling reduces self-preference. 95% CIs from the rubric-level bootstrap (1000 resamples); the

\Delta

interval is the difference of the two replicate distributions (conservative).

Member	HSPP individual	HSPP committee	$\Delta$ (committee $-$ individual)
Gemma-27B	1.03 [0.97, 1.08]	0.41 [0.34, 0.48]	-0.62 [-0.70, -0.53]
Llama-Mav	1.34 [1.30, 1.38]	0.48 [0.41, 0.55]	-0.86 [-0.94, -0.78]
Qwen-235B	0.83 [0.62, 1.07]	0.95 [0.80, 1.11]	+0.12 [-0.17, +0.37]
GPT-5	20.15 [16.57, 24.58]	4.18 [3.64, 4.69]	-15.97 [-20.31, -12.50]
Claude-Sonnet	1.78 [1.58, 1.99]	1.29 [1.11, 1.48]	-0.50 [-0.76, -0.20]

Appendix

Table 8: LiveCodeBench: same analysis as Table 7 .

Independent variable	Coef	95% CI
Ensemble size	-0.0070	[-0.0092, -0.0047]
# families	+0.0136	[+0.0106, +0.0166]
Mean MRA	+2.3279	[+2.1669, +2.4889]

Appendix

Table 9: OLS of committee-mean HSPP-Rub. Self on ensemble properties, IFEval (

R^{2}=0.252

Independent variable	Coef	95% CI
Ensemble size	-0.0243	[-0.1121, +0.0635]
# families	+0.0475	[-0.0687, +0.1636]
Mean MRA	+17.6089	[+16.2813, +18.9365]

Appendix

Table 10: OLS of committee-mean HSPP-Rub. Self on ensemble properties, LiveCodeBench (

R^{2}=0.227

Benchmark	Rubric subset	# items	$\alpha$
IFEval	All rubrics	10,008	0.306
IFEval	$\geq$ 1 generator failed	4,104	0.314
IFEval	All generators failed	60	0.308
LiveCodeBench	All rubrics	69,775	0.353
LiveCodeBench	$\geq$ 1 generator failed	54,612	0.322
LiveCodeBench	All generators failed	1,850	0.154

Appendix

Table 11: Krippendorff

\alpha

(binary, nominal) of the five family-largest judges, by rubric subset.

Model	Reasoning	MRA	HSPP-Rub Self	HSPP-Rub Fam
GPT-5	default	0.91 [0.91, 0.92]	1.47 [0.48, 2.53]	1.16 [0.63, 1.75]
GPT-5	low	0.90 [0.90, 0.91]	1.71 [0.73, 2.78]	1.28 [0.75, 1.90]
GPT-5	medium	0.91 [0.91, 0.92]	1.48 [0.46, 2.55]	1.17 [0.62, 1.76]
GPT-5	high	0.92 [0.91, 0.92]	1.60 [0.57, 2.82]	1.26 [0.72, 1.96]
Claude-Sonnet	default	0.84 [0.83, 0.85]	0.98 [0.64, 1.32]	1.10 [0.90, 1.37]
Claude-Sonnet	low	0.92 [0.91, 0.92]	1.56 [1.15, 2.08]	1.40 [1.12, 1.71]

Appendix

Table 12: IFEval (SR) rubric-level metrics by judge reasoning effort, with 95% CIs (rubric-level bootstrap, 1000 resamples).

Judge	Score (as gen)	Group	HSPP-Rub Self $\cdot$ default	HSPP-Rub Self $\cdot$ same-group	N comparators
Gemma-27B	0.586	High	1.10 [1.08, 1.12]	1.03 [1.00, 1.05]	3
Gemma-12B	0.572	Mid	1.03 [1.01, 1.05]	0.99 [0.97, 1.02]	3
Gemma-4B	0.477	Low	1.16 [1.14, 1.17]	1.35 [1.33, 1.37]	3
Llama-Mav	0.400	Low	0.71 [0.69, 0.73]	0.71 [0.69, 0.74]	2
Llama-Scout	0.392	Low	0.80 [0.79, 0.82]	0.82 [0.80, 0.84]	2
Qwen-235B	0.605	High	1.12 [1.07, 1.18]	0.95 [0.90, 1.00]	3

Appendix

Table 13: HealthBench HSPP-Rub. Self with the default (non-family) comparator set vs. only same-quality-tertile comparators. 95% CIs from the rubric-level bootstrap (1000 resamples).

Pair (GPT-5 vs …)	GPT-5 performance	Comparator performance	HSPP, all ref-negatives	HSPP, joint ref-negatives	R (joint pool)
GPT-120B	0.71 [0.71, 0.72]	0.63 [0.62, 0.64]	1.26 [1.17, 1.37]	1.30 [1.18, 1.46]	7,479
Claude-Sonnet	0.71 [0.71, 0.72]	0.54 [0.53, 0.55]	1.52 [1.40, 1.65]	1.46 [1.32, 1.62]	8,794
Qwen-235B	0.71 [0.71, 0.72]	0.61 [0.60, 0.61]	1.33 [1.23, 1.44]	1.44 [1.30, 1.59]	8,335

Appendix

Table 14: HealthBench: GPT-5’s one-vs-one self-preference against similar-capability generators. “Joint ref-negatives” restricts to rubrics failed by both generators (paired bootstrap);

R

= joint pool size.

Pair (GPT-5 vs …)	GPT-5 performance	Comparator performance	HSPP, all ref-negatives	HSPP, joint ref-negatives	R (joint pool)
GPT-120B	0.93 [0.90, 0.96]	0.89 [0.85, 0.93]	5.49 [4.03, 8.09]	7.14 [3.92, 23.52]	221
Claude-Sonnet	0.93 [0.90, 0.96]	0.84 [0.80, 0.88]	6.23 [4.68, 8.62]	5.45 [3.07, 12.80]	209
Qwen-235B	0.93 [0.90, 0.96]	0.83 [0.78, 0.88]	19.05 [12.68, 34.20]	12.33 [6.45, 39.00]	247

Appendix

Table 15: LiveCodeBench: same analysis as Table 14 .

Generator	Pass rate	% fails in MRA $<0.7$ bins	% fails in MRA $\geq 0.8$ bins
$\bm{\cdot}$ GPT-5	0.942	65.3%	21.4%
$\bm{\cdot}$ GPT-120B	0.900	68.3%	18.3%
$\bm{\cdot}$ Claude-Sonnet	0.848	68.7%	8.9%
$\bm{\cdot}$ Qwen-235B	0.834	43.9%	24.4%
$\bm{\cdot}$ Claude-Haiku	0.816	62.1%	10.1%
$\bm{\cdot}$ Qwen-30B	0.763	35.8%	27.2%

Appendix

Table 16: LiveCodeBench: distribution of each generator’s failed tests across generation-hardness cells (mean rubric accuracy across all 12 judges, as in Table 2 ). Generator pass rate correlates negatively with the mean MRA on its completions weighted by the percentage of failed tests (Spearman

\rho=-0.71

p=0.010

): better generators’ mistakes are harder to judge.

Judge	0.0–0.1 (n=44)	0.1–0.2 (n=58)	0.2–0.3 (n=61)	0.3–0.4 (n=65)	0.4–0.5 (n=109)	0.5–0.6 (n=386)	0.6–0.7 (n=289)	0.7–0.8 (n=439)	0.8–0.9 (n=768)	0.9–1.0 (n=4,273)
$\bm{\cdot}$ Gemma-4B	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.01 [1.00, 1.05]	1.01 [1.00, 1.04]	1.01 [1.00, 1.03]	1.04 [1.01, 1.07]	0.97 [0.79, 1.10]	1.75 [0.79, 3.06]
$\bm{\cdot}$ Gemma-12B	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	0.90 [0.61, 1.19]	1.01 [0.91, 1.09]	1.05 [0.92, 1.15]	0.88 [0.63, 1.10]	1.66 [1.13, 2.65]	–
$\bm{\cdot}$ Gemma-27B	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.08 [1.00, 1.22]	1.05 [1.02, 1.10]	1.08 [1.03, 1.14]	1.11 [0.91, 1.26]	0.96 [0.32, 1.61]	0.00 [0.00, 0.00]
$\bm{\cdot}$ Llama-Scout	1.01 [1.00, 1.05]	1.03 [1.00, 1.09]	0.84 [0.00, 1.21]	0.81 [0.28, 1.26]	0.79 [0.00, 1.42]	1.11 [0.91, 1.30]	1.05 [0.75, 1.33]	0.81 [0.33, 1.34]	4.90 [1.32, 12.34]	0.00 [0.00, 0.00]
$\bm{\cdot}$ Llama-Mav	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.04 [1.00, 1.14]	1.08 [1.02, 1.16]	1.07 [0.61, 1.41]	1.14 [0.91, 1.36]	1.15 [0.84, 1.39]	1.15 [0.55, 1.75]	1.74 [0.77, 3.74]	0.00 [0.00, 0.00]
$\bm{\cdot}$ Qwen-4B	1.01 [1.00, 1.06]	1.01 [1.00, 1.04]	0.92 [0.00, 1.41]	1.30 [1.07, 1.50]	0.98 [0.45, 1.54]	0.94 [0.74, 1.15]	1.01 [0.75, 1.26]	1.31 [0.77, 1.84]	3.15 [0.71, 7.04]	0.00 [0.00, 0.00]

Appendix

Table 17: IFEval: HSPP-Rub. Self by mean-MRA bin over (generator, instance) cells;

n

= cells per bin. 95% CIs from the rubric-level bootstrap (1000 resamples).

Judge	0.3–0.4 (n=3)	0.4–0.5 (n=20)	0.5–0.6 (n=175)	0.6–0.7 (n=1,942)	0.7–0.8 (n=15,267)	0.8–0.9 (n=32,064)	0.9–1.0 (n=10,529)
$\bm{\cdot}$ Gemma-4B	1.00 [1.00, 1.00]	0.97 [0.89, 1.00]	1.18 [1.10, 1.27]	1.07 [1.03, 1.10]	1.07 [1.05, 1.09]	1.11 [1.09, 1.13]	1.20 [1.10, 1.32]
$\bm{\cdot}$ Gemma-12B	–	1.20 [1.00, 1.60]	0.88 [0.70, 1.09]	0.94 [0.89, 1.00]	0.96 [0.93, 0.98]	1.02 [0.99, 1.04]	1.14 [1.01, 1.27]
$\bm{\cdot}$ Gemma-27B	–	–	0.91 [0.62, 1.18]	1.05 [0.98, 1.11]	1.05 [1.02, 1.08]	1.10 [1.06, 1.13]	1.14 [1.01, 1.29]
$\bm{\cdot}$ Llama-Scout	–	1.07 [0.00, 2.09]	1.09 [0.87, 1.28]	0.87 [0.80, 0.95]	0.86 [0.83, 0.89]	0.76 [0.73, 0.79]	0.71 [0.62, 0.82]
$\bm{\cdot}$ Llama-Mav	–	1.25 [0.00, 2.65]	1.04 [0.70, 1.40]	0.83 [0.75, 0.93]	0.77 [0.74, 0.80]	0.68 [0.66, 0.71]	0.58 [0.48, 0.67]
$\bm{\cdot}$ Qwen-4B	1.00 [1.00, 1.00]	–	0.96 [0.81, 1.11]	0.91 [0.86, 0.96]	0.95 [0.93, 0.97]	0.92 [0.90, 0.95]	0.90 [0.80, 1.00]

Appendix

Table 18: HealthBench: same analysis as Table 17 .

Judge	0.2–0.3 (n=2)	0.3–0.4 (n=7)	0.4–0.5 (n=29)	0.5–0.6 (n=138)	0.6–0.7 (n=426)	0.7–0.8 (n=575)	0.8–0.9 (n=513)	0.9–1.0 (n=434)
$\bm{\cdot}$ Gemma-4B	–	2.04 [1.31, 3.03]	1.49 [1.23, 1.79]	2.50 [1.72, 3.49]	2.46 [1.91, 3.09]	2.25 [1.75, 2.82]	1.61 [1.22, 2.15]	0.39 [0.28, 7.03]
$\bm{\cdot}$ Gemma-12B	–	–	1.01 [0.92, 1.09]	1.29 [1.18, 1.40]	1.07 [1.02, 1.12]	1.11 [1.03, 1.19]	1.16 [1.01, 1.31]	1.46 [1.01, 1.85]
$\bm{\cdot}$ Gemma-27B	–	–	1.10 [0.99, 1.18]	1.21 [1.12, 1.29]	1.23 [1.15, 1.31]	1.02 [0.92, 1.12]	1.19 [0.95, 1.49]	5.06 [2.83, 9.59]
$\bm{\cdot}$ Llama-Scout	–	0.97 [0.91, 1.00]	–	1.09 [1.05, 1.13]	1.04 [1.01, 1.07]	1.28 [1.23, 1.33]	1.14 [1.01, 1.27]	1.05 [0.72, 3.38]
$\bm{\cdot}$ Llama-Mav	–	1.02 [1.00, 1.04]	1.19 [1.09, 1.27]	1.14 [1.07, 1.20]	1.06 [1.01, 1.11]	1.53 [1.46, 1.62]	1.59 [1.43, 1.76]	0.00 [0.00, 0.00]
$\bm{\cdot}$ Qwen-4B	–	–	1.42 [1.29, 1.55]	2.17 [1.83, 2.51]	1.89 [1.60, 2.24]	2.29 [1.81, 2.91]	2.17 [1.45, 3.24]	–

Appendix

Table 19: LiveCodeBench: same analysis as Table 17 .

Benchmark	Q1 (shortest)	Q2	Q3	Q4 (longest)
IFEval	1.07 [0.78, 1.44]	1.22 [0.97, 1.59]	1.17 [0.97, 1.43]	1.02 [0.88, 1.21]
HealthBench	0.96 [0.94, 0.99]	1.01 [0.98, 1.04]	1.05 [1.02, 1.08]	1.10 [1.07, 1.12]
LiveCodeBench	0.90 [0.79, 1.11]	3.12 [2.56, 3.89]	2.66 [2.20, 3.24]	5.32 [4.43, 6.62]

Appendix

Table 20: Aggregate HSPP-Rub. Self by quartile of the judge’s own output length (12 judges). 95% CIs from the rubric-level bootstrap (1000 resamples).

Benchmark	Q1 (lowest ppl)	Q2	Q3	Q4 (highest ppl)
IFEval	0.94 [0.89, 1.03]	0.44 [0.00, 1.42]	3.00 [0.00, 7.05]	1.51 [0.00, 5.49]
HealthBench	1.03 [1.01, 1.06]	0.76 [0.07, 1.45]	0.23 [0.00, 0.50]	0.17 [0.00, 0.40]
LiveCodeBench	1.48 [1.10, 2.61]	0.70 [0.00, 2.27]	17.63 [0.00, 44.17]	0.00 [0.00, 0.00]

Appendix

Table 21: Aggregate HSPP-Rub. Self by within-judge perplexity quartile of the evaluated output (9 open-weights judges).

Benchmark	Recognised as self	Not recognised
IFEval	1.16 [1.06, 1.29]	1.09 [0.99, 1.21]
HealthBench	1.01 [0.99, 1.03]	0.78 [0.67, 0.94]
LiveCodeBench	1.88 [1.75, 2.02]	0.72 [0.67, 0.78]

Appendix

Table 22: Aggregate HSPP-Rub. Self split by the judge’s self-recognition prediction on its own outputs (9 open-weights judges).

Judge	HSPP-Rub Self (original)	HSPP-Rub Self (LOFO)	HSPP-Rub Self (LOO)
Gemma-4B	1.16 [1.14, 1.17]	1.16 [1.15, 1.18]	1.17 [1.15, 1.19]
Gemma-12B	1.03 [1.01, 1.05]	1.03 [1.01, 1.05]	1.00 [0.98, 1.02]
Gemma-27B	1.10 [1.08, 1.12]	1.10 [1.08, 1.12]	1.09 [1.07, 1.12]
Llama-Scout	0.80 [0.79, 0.82]	0.79 [0.77, 0.81]	0.82 [0.80, 0.84]
Llama-Mav	0.71 [0.69, 0.73]	0.70 [0.68, 0.72]	0.70 [0.68, 0.72]
Qwen-4B	0.96 [0.95, 0.98]	0.95 [0.93, 0.97]	0.95 [0.93, 0.97]

Appendix

Table 23: HealthBench HSPP-Rub. Self under the paper’s ensemble reference vs. LOFO and LOO references. 95% CIs from the rubric-level bootstrap (1000 resamples).

Effect	Posterior mean	Posterior SD	95% interval	Odds ratio
is_self $\times$ Axis: communication_quality	-0.033	0.080	[-0.190, +0.124]	0.968
is_self $\times$ Axis: completeness	-0.008	0.029	[-0.065, +0.049]	0.992
is_self $\times$ Axis: context_awareness	+0.010	0.041	[-0.072, +0.091]	1.010
is_self $\times$ Axis: instruction_following	-0.054	0.086	[-0.223, +0.115]	0.948
is_self $\times$ Theme: communication	+1.877	0.894	[+0.124, +3.630]	6.534
is_self $\times$ Theme: complex	+0.411	0.153	[+0.111, +0.712]	1.509

Appendix

Table 24: Interaction coefficients is_self

\times

X

of the HealthBench mixed-effects regression (main effects included in the model but omitted here). Positive

\Rightarrow

property

X

reinforces self-preference.

Judge	Self FP / eligible	Self FPR	Others FP / eligible (pooled, non-family)	Mean per-gen other FPR	HSPP-Rub Self (point)
Gemma-4B	141 / 143	0.9860	651 / 679	0.9577	1.03
Gemma-12B	96 / 109	0.8807	580 / 679	0.8553	1.03
Gemma-27B	94 / 99	0.9495	598 / 679	0.8761	1.08
Llama-Scout	61 / 86	0.7093	596 / 877	0.6771	1.05
Llama-Mav	53 / 67	0.7910	624 / 877	0.7056	1.12
Qwen-4B	68 / 92	0.7391	549 / 783	0.6971	1.06

Appendix

Table 25: IFEval: absolute values and sample counts behind HSPP-Rub. Self.

Judge	Self FP / eligible	Self FPR	Others FP / eligible (pooled, non-family)	Mean per-gen other FPR	HSPP-Rub Self (point)
Gemma-4B	12,098 / 21,563	0.5611	81,575 / 172,510	0.4837	1.16
Gemma-12B	7,040 / 17,712	0.3975	65,899 / 172,510	0.3870	1.03
Gemma-27B	6,506 / 17,128	0.3798	58,508 / 172,510	0.3449	1.10
Llama-Scout	6,330 / 25,364	0.2496	55,334 / 178,486	0.3110	0.80
Llama-Mav	4,271 / 25,063	0.1704	42,884 / 178,486	0.2410	0.71
Qwen-4B	8,455 / 19,964	0.4235	75,698 / 174,640	0.4397	0.96

Appendix

Table 26: HealthBench: absolute values and sample counts behind HSPP-Rub. Self.

Judge	Self FP / eligible	Self FPR	Others FP / eligible (pooled, non-family)	Mean per-gen other FPR	HSPP-Rub Self (point)
Gemma-4B	409 / 2,944	0.1389	1,222 / 11,959	0.0904	1.54
Gemma-12B	1,172 / 1,962	0.5973	7,469 / 11,959	0.6149	0.97
Gemma-27B	926 / 1,772	0.5226	6,318 / 11,959	0.5089	1.03
Llama-Scout	2,300 / 2,905	0.7917	8,339 / 13,637	0.6782	1.17
Llama-Mav	1,629 / 2,095	0.7776	6,692 / 13,637	0.5789	1.34
Qwen-4B	532 / 1,849	0.2877	1,561 / 14,488	0.1299	2.21

Appendix

Table 27: LiveCodeBench: absolute values and sample counts behind HSPP-Rub. Self.

Bias in LLM-as-a-Judge

Bias in large language model (LLM)-as-a-judge evaluation studies systematic distortions in automated assessments of model outputs.

See all 118 papers

Quantifying and Mitigating Self-Preference Bias of LLM Judges
Apr 24, 2026Jinming Yang, Zheng Hu, Chuxian Qiu +3LLM-as-a-Judge Language Model Bias Evaluation
CompleX Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
Feb 2, 2026Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno +1LLM-as-a-Judge Language Model Bias Evaluation
OMRON SINIC X · NexaScience
Self- and Other-Labels Induce Bidirectional Bias in LLM Judges
Jun 6, 2026Songeun Chae, Min Kim, Donghoon Jung +2Bias in LLM-as-a-Judge Language Model Bias Evaluation
School of Digital Humanities and Computational Social Sciences, KAIST, South Korea

Apr 24, 2026J/K move · Enter open · S save

Quantifying and Mitigating Self-Preference Bias of LLM Judges

Jinming Yang, Zheng Hu, Chuxian Qiu, Zhenyu Deng+2

CompleX Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China

LLM-as-a-Judge Language Model Bias Evaluation Bias in LLM-as-a-Judge Automated Evaluation

Open paper AskPDF arXiv

LLM-as-a-Judge has become a dominant approach in automated evaluation systems, playing critical roles in model alignment, leaderboard construction, quality control, and so on. However, the scalability and trustworthiness of this approach can be substantially distorted by Self-Preference Bias (SPB), which is a directional evaluative deviation in which LLMs systematically favor or disfavor their own generated outputs during evaluation. Existing measurements rely on costly human annotations and conflate generative capability with evaluative stance, and thus are impractical for large-scale deployment in real-world systems. To address this issue, we introduce a fully automated framework to quantifying and mitigating SPB, which constructs equal-quality pairs of responses with negligible quality differences, enabling statistical disentanglement of discriminability from bias propensity without human gold standards. Empirical analysis across 20 mainstream LLMs reveals that advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB. To mitigate this bias, we propose a structured multi-dimensional evaluation strategy grounded in cognitive load decomposition, which reduces SPB by 31.5% on average.

Abstract

Bias in LLM-as-a-Judge

Explore similar work

Quantifying and Mitigating Self-Preference Bias of LLM Judges

Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge

Self- and Other-Labels Induce Bidirectional Bias in LLM Judges