Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency

Bruno Brocai; Maria Becker

cs.CLSep 29, 2026

Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency

Authors: Bruno Brocai, Maria Becker

Organizations: Heidelberg University German Department

Abstract

Large Language Model judges are widely used to rank texts and text-generating systems through pairwise comparison, and their reliability is typically assessed via three proxies: position bias, transitivity, and pairwise agreement (self- or human-labeled). Because these proxies drive judge selection and benchmarking, a substantial literature reporting that judges perform poorly on them risks steering practitioners away from otherwise capable evaluators. We argue this assessment is misleading. Under the Bradley--Terry geometry underlying pairwise aggregation, each proxy is dominated by close-rank-gap pairs, where inconsistency is information-theoretically expected and individual verdicts contribute little to the aggregate ranking; far-gap pairs carry the ranking signal but barely move the proxies. We formalize this argument and validate it in a controlled simulation and on two human-rated corpora: the proxies correlate only weakly with ranking accuracy against gold, and their predictive component concentrates in the far-gap regime. Judges should therefore be assessed on rank-gap-conditional metrics, ideally against human rankings. Code at https://github.com/brunobrocai/PairDifficulty.

Figures & tables

Figure 1: Simulation results for BT Judges with identical pairwise agreement. Top shows how disagreement between judges and humans is differently distributed, while overall agreement is exactly equal (bottom left). Despite this, ranking correlation with the human gold varies substantially depending on where disagreement occurs (bottom right).

		Consistency ( $1-$ flip)			IPP	% non-transitive triads
	Judge	all	close	far	all	cyc	mix	ineq	$\rho$
			(Q1 gap)	(Q4 gap)
CLEAR	GPT-5.4-nano	.78	.72	.90	.61	0.0	0.8	9.9	$+0.691$
	GPT-5.4-mini	.80	.66	.96	.97	0.0	0.4	9.5	$+0.825$
	Ministral-3-3B	.79	.70	.92	.89	0.0	0.4	6.2	$+0.755$
	Ministral-3-14B	.84	.73	.97	.89	0.0	0.8	4.1	$+0.788$

Table 1: Inconsistency as a close-pair phenomenon does not track ranking accuracy. Consistency is how often the judge chooses the same text across the swapped pairs; IPP is inconsistent-pair primacy, with .5 indicating perfect balance. close / far are the same rate restricted to the bottom/top quartile of the gold gap

|g_{a}-g_{b}|

. Triad-% is the proportion of all triads.

\rho

is the Spearman correlation of the LLM-derived BT skill against human gold.

Proxy	$\rho_{s}$ vs ranking $\rho$	$p_{\text{Holm}}$
Consistency, all	$+0.559$	$0.198$
Consistency, close	$+0.245$	$0.892$
Consistency, far	$+0.897$	$0.001^{*}$
Primacy (IPP)	$+0.126$	$0.892$
Intransitivity	$-0.608$	$0.174$

Table 2: Spearman correlation between inconsistencies and the model’s ability to replicate human gold ranking, and the associated Holm-corrected significance. Two-sided permutation

p

(10,000 reshuffles) over the 12 judge-corpus points, Holm-corrected across the five proxies. With N=12 the test is well-powered for strong effects but has limited power against moderate effects. We note this as a limitation but emphasize the practical implication: a metric whose correlation with gold ranking is too weak to survive correction at this scale is too noisy to be used as a model-selection signal in practice.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Params	Access
GPT-5.4 nano OpenAI (2026)	undisclosed	API
GPT-5.4 mini OpenAI (2026)	undisclosed	API
Ministral 3 3B Liu et al. (2026)	3B	Open weights
Ministral 3 14B Liu et al. (2026)	14B	Open weights
Gemma 3 4B Gemma Team et al. (2025)	4B	Open weights
Gemma 3 12B Gemma Team et al. (2025)	12B	Open weights

Appendix

Table 3: Judge models.

	Consistency ( $1-$ flip)			$\rho$ vs gold
Judge	close	close	far	all	tied
	(gap $=0$ )	(gap $=1$ )	(gap $\geq 2$ )	verdicts	excluded
	28.2%	44.9%	26.8%
GPT-5.4-nano	.74	.78	.85	$+0.677$	$+0.735$
GPT-5.4-mini	.62	.73	.84	$+0.674$	$+0.726$
Ministral-3-3B	.65	.76	.94	$+0.726$	$+0.786$

Appendix

Table 4: ASAP 2.0 robustness check with gold-tied pairs removed. Consistency is reported for the three gold-gap buckets.

Figure 2: Simulation results for BT Judges with identical pairwise agreement that coinflip below a certain pair gap. Top shows how disagreement between judges and humans is differently distributed, while overall agreement is exactly equal (bottom left). Despite this, ranking correlation with the human gold varies substantially depending on where disagreement occurs (bottom right).

Explore similar work

Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
Feb 2, 2026Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno +1Llm-As-A-Judge Model Judgments
OMRON SINIC X · NexaScience
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
Apr 23, 2026Abel YagubyanLlm-As-A-Judge Large Language Model Judges
Independent Researcher
Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels
Sep 15, 2026David Ababio Awuni, Luke E. K. Achenie, Benjamin Tei Partey +2Judges Llm-As-A-Judge

Feb 2, 2026, cs.CLJ/K move · Enter open · S save

Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge

Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno, Yoshitaka Ushiku

OMRON SINIC X · NexaScience

Llm-As-A-Judge Model Judgments Position Bias Grading

Open paper AskPDF arXiv

Large language models are widely employed as evaluators, a paradigm commonly referred to as LLM-as-a-judge. Prior research has predominantly examined point-wise or pair-wise evaluation protocols; in contrast, our focus is on rubric-based evaluation, which has been attracting increasing attention owing to its utility for training models in domains where verification is otherwise difficult. In this work, we show that rubric-based evaluation implicitly resembles a multiple-choice setting and therefore exhibits position bias: LLMs tend to prefer score options that appear at specific positions within the rubric list. Through controlled experiments across multiple models and datasets, we demonstrate that this position bias is consistent. Its direction, however, is model-specific: some judges favor the first option, while others favor the last. We further identify a second, orthogonal axis of bias: when a prompt scores several criteria simultaneously, the ordering of the criteria itself shifts the resulting scores. We additionally explore permuting the order of the rubric options as a means of mitigating position bias, and find that although the bias can be attenuated, improvements in the correlation between model judgments and human annotations are obtained primarily for models that exhibit strong bias. Our results recast rubric-based LLM-as-a-judge as a multiple-choice problem with measurable, model-specific position bias, and we further confirm that only a small number of random order permutations are sufficient to reduce the error introduced by this bias for the majority of models.

Abstract

Explore similar work

Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge

The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation

Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels