Pairwise language-model judges can gather evidence through direct comparison, reasoning, or reference-based verification, but no single protocol is best across benchmarks and judge backbones. We introduce Backbone-Adaptive Evidence Routing (BAER), which adapts the evidence mechanism while preserving candidate symmetry: swapping the two responses may reverse the preference but cannot change its strength. BAER separates each expert's signed preference from candidate-invariant reliability and builds three symmetric heads: evidence stacking, reliability-based expert routing, and candidate-blind reference verification. Development data select one head for each benchmark--backbone condition, and that choice is frozen before testing. Across four benchmarks and two 8B judge backbones, BAER achieves the highest test accuracy among the compared methods in all eight conditions, with full prediction coverage and gains of 0.87--7.32 points over the strongest external baseline. The results show that adapting how evidence is gathered is more reliable than fixing one judging protocol everywhere.
Figures & tables
Figure 1: Motivation for condition-adaptive evidence. Conflicting experts and shared errors call for different mechanisms, illustrated by BAER’s invariant expert routing and candidate-blind reference verification. Head assignments match the deployed JudgeBench conditions; examples are schematic.
Figure 2: Overview of BAER. Development data freeze one evidence head per benchmark–backbone condition. Stacking combines signed protocol features, routing selects one expert per pair using candidate-invariant reliability, and reference verification solves the instruction without seeing either candidate, then checks each independently. All heads return a candidate-symmetric signed score.
Method
RewardBench
JudgeBench
HH-RLHF
UltraFeedback
Qwen 3-8B
Llama 3.1-8B
Qwen 3-8B
Llama 3.1-8B
Qwen 3-8B
Llama 3.1-8B
Qwen 3-8B
Llama 3.1-8B
Direct bidirectional
81.9
75.2
58.5
49.6
60.1
56.8
91.2
62.3
Self-consistency
81.9
75.5
58.5
50.0
60.1
57.1
91.2
62.3
Chain-of-thought
82.3
62.9
66.7
37.4
59.6
53.3
89.2
56.5
Rubric-based
40.7
66.7
31.3
24.8
51.9
50.5
45.0
49.4
Score-then-choose
42.3
68.9
36.6
34.1
49.1
52.8
45.0
52.0
Table 1: Full-test pairwise accuracy (%) on the benchmark–backbone columns. Bold marks the best result, underline marks the strongest external baseline, and † marks a significant exact two-sided McNemar comparison against the strongest baseline in that column ( p<.05 ).
Figure 3: JudgeBench semantic-head development. Accuracy gaps are relative to the strongest external baseline on matched IDs; zero denotes parity. (a) Selection results for the best variant per family ( n=128 ; other heads use n=136 and are named in (b)). CF = counterfactual reconciliation, Orbit = complete orbit, Anchor = blind anchor, Audit = independent pointwise audit. (b) Selection and calibration ( n=238 ) for the three advancing heads. Only candidate-blind reference verification retains a positive gap; the Llama reference head was not evaluated.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
Large Language Model judges are widely used to rank texts and text-generating systems through pairwise comparison, and their reliability is typically assessed via three proxies: position bias, transitivity, and pairwise agreement (self- or human-labeled). Because these proxies drive judge selection and benchmarking, a substantial literature reporting that judges perform poorly on them risks steering practitioners away from otherwise capable evaluators. We argue this assessment is misleading. Under the Bradley--Terry geometry underlying pairwise aggregation, each proxy is dominated by close-rank-gap pairs, where inconsistency is expected and errors cost little aggregate ranking accuracy; far-gap pairs determine the ranking accuracy but barely move the proxies. We formalize this argument and validate it in a controlled simulation and on two human-rated corpora: the proxies correlate only weakly with ranking accuracy against gold, and their predictive component concentrates in the far-gap regime. Judges should therefore be assessed on rank-gap-conditional metrics, ideally against human rankings. Code at https://github.com/brunobrocai/PairDifficulty
LLM judges are often asked to extract criteria and evidence before choosing between candidate answers. This workflow assumes that the intermediate record preserves the information needed for a later verdict. For reasoning-capable models, visible field order does not reveal internal decision order, so we test an observable alternative: persist the evidence in one call and make it the exclusive input to the next. Across 24,000 judgments over HelpSteer3, FeedbackQA, and CoVal, we compare standard pairwise judging, structured one-call judging, two-call evidence locking, and three-call pointwise locking with Claude Sonnet 4.5 and GPT-5. Evidence locking reduces agreement with released human preferences by 4 to 6 percentage points and increases answer-order inconsistency by 8 to 10 points relative to structured one-call judging. Pointwise locking is also harmful, while structured evidence elicitation remains close to standard judging. The result holds for both judges and all three datasets. Persisted evidence can support auditability, but it should not replace the source answers at decision time.
LLM-as-judge evaluation has become standard practice for open-ended model assessment; however, judges exhibit systematic biases that cannot be averaged out by increasing the number of scenarios or generations. These biases are often similar in magnitude to the model differences that benchmarks are designed to detect, resulting in unreliable rankings when single-judge evaluations are used. We introduce a variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components. Based on this analysis, CyclicJudge, a round-robin assignment of judges to scenarios, is demonstrated to be the optimal strategy for a fixed judge panel and judge-call budget: the score recovers the panel mean exactly while matching the cost of single-judge evaluation. Empirical results on MT-Bench and MindEval validate the effectiveness of CyclicJudge as predicted, across both general-purpose and domain-specific evaluation settings.
Ziyi Zhu, Olivier Tieleman, Alexey Bukhtiyarov +1
Slingshot AI · Department of Engineering, University of Cambridge