Preference benchmarks are built by hiring annotators, and the identity of those annotators is treated as an implementation detail. We measure what that detail buys. On the 2,885 MultiPref items where both pools are internally unanimous, so no tie-breaking convention is consulted at all, expert and crowd annotators assign a different majority label to 23.6% and name the opposite winner on 9.2%; on the 246 comparably unanimous MT-Bench cells, benchmark authors and recruited experts differ on 30.5% and reverse on 8.5%. Yet on both corpora the resulting model leaderboards are bit-identical: Kendall tau = 1.00 with zero of six models displaced. That invariance is far weaker evidence than it looks, and we quantify how weak. Switching pools moves a model's win rate by 1.9pp (SD), one adjacent pair in our own leaderboard sits 0.8pp apart and had a 38% chance of swapping, and an item-level bootstrap displaces at least one model in 28% of resamples. The observed zero is the common outcome, not a property of aggregation: on the same measured perturbation, a ten-model leaderboard is displaced with probability 0.86 and a twenty-model leaderboard with probability 0.9997. Reporting a six-model leaderboard is safe; the safety does not generalise, and everything that consumes labels per item is not safe at any size. We make the distinction precise, show that a widely used dataset's stated assumption of no intra-group annotator variability is false, and show that an LLM judge tracks the crowd pool over the expert pool on all three models we test, including one from a different vendor. All code, per-call outputs, and pre-registered decision rules will be released upon acceptance.
Figures & tables
Figure 1: Two pools annotating the same items disagree substantially ( A ) and produce identical model rankings ( B ). Hatching marks outright winner reversals, a strict subset of majority-label changes. Panel B shows MultiPref ; MT-Bench likewise displaces zero of six. Panel B is a six-model leaderboard, and § 4 shows that is doing much of the work: the same perturbation displaces a ten-model board with probability 0.86 .
Corpus
Pool contrast
Items
Majority differs
Winner reverses
Kendall τ
MultiPref
expert vs. crowd
2,885
23.6%
9.2%
1.0000
MT-Bench
authors vs. experts
246
30.5%
8.5%
1.0000
Table 1: The same pattern on two corpora with different pool contrasts. Winner reversal excludes ties on both sides, so it counts only items where both pools name a decisive and opposite winner. MultiPref is restricted to items where both pools are internally unanimous, and MT-Bench likewise, so no tie-breaking convention enters either corpus.
Tie-break rule
Majority differs
Winner reverses
Kendall τ (displaced)
A > B > Tie (what we ran)
44.9%
29.8%
1.0000 ( 0/6 )
Both pools unanimous (no rule)
23.6%
9.2%
1.0000 ( 0/6 )
Force ties to Tie
41.2%
2.5%
1.0000 ( 0/6 )
B > Tie > A (mirror)
36.6%
9.4%
1.0000 ( 0/6 )
A > Tie > B (hypothetical)
49.8%
16.8%
0.4667 ( 4/6 )
Table 2: Every tie-breaking convention we tried, on all 10,461 MultiPref comparisons. The first row is the rule our leaderboard actually used. Only the last, a hypothetical rule ranking A over Tie over B, moves the ranking; its mirror image does not, identifying it as an artifact of the corpus’s 1.78:1 B-skew.
Models on the leaderboard
6
10
20
30
50
Gap between adjacent models
4.0 pp
2.2 pp
1.1 pp
0.7 pp
0.4 pp
P(at least one displaced)
0.28
0.86
0.9997
1.00
1.00
Table 3: Probability that changing the annotator pool displaces at least one model, as a function of leaderboard size, holding the measured perturbation ( SD=2.6 pp on the pairwise difference) fixed and spreading K models uniformly across the observed win-rate span. The K=6 row is reproduced independently by an item-level bootstrap of the real corpus ( 0.282 ), which is why we are willing to read the rest of the column.
Judge
Items
Agree (expert)
Agree (crowd)
Difference [95% CI]
Qwen3.6-35B-A3B (Qwen)
1,383
0.469
0.538
−0.069[−0.103,−0.034]
Qwen3.6-27B (Qwen)
1,884
0.427
0.479
−0.052[−0.082,−0.022]
Gemma-4-26B-A4B (Google)
1,904
0.432
0.469
−0.037[−0.067,−0.008]
Table 4: LLM-judge alignment with each pool on MultiPref , on items where the judge is self-consistent across presentation orders, within each model’s pre-registered 2,500-item stratified sample. Negative difference means the judge agrees more with the crowd majority. All three clear the pre-registered bar, which is conjunctive over agreement and erasure. Under the committed primary estimator (midpoint order-combination, all items rather than the self-consistent subset) the agreement difference is −0.026 , −0.043 and −0.022 for the three judges in table order; the two whose agreement CI touches zero survive the conjunctive bar on erasure ( +2.3 pp and +2.8 pp), together with the self-consistent subset shown here. Every judge is negative under both estimators.