Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena

Soovadeep Bakshi; Apurba Bose; Aishwarya Bose; Sandip Nandi

cs.CLOct 13, 2026

Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena

Authors: Soovadeep Bakshi, Apurba Bose, Aishwarya Bose, Sandip Nandi

Organizations: Independent Researcher.

Abstract

Chatbot Arena ranks language models with the Bradley-Terry model, which gives each model one score and so assumes Luce's choice axiom. Choice theory predicts where this fails: near-copies are told apart more decisively than their scores imply. On 1.8 million Arena votes, votes between versions of one model are 1.37 times as decisive as Bradley-Terry predicts, between sizes of one generation 1.26 times and between models of one lab 1.12 times; a release a year later gives consistent multipliers, and the effect survives alternative tie treatments, prompt subsets and topics, and timing checks. How alike two models' answers are worded and formatted tracks the effect. A correction with four to six parameters predicts held-out votes better than Bradley-Terry and than models with hundreds of parameters, mostly by modelling ties; frozen and carried to new releases in a test fixed in advance, its gains are concentrated in ties, and better head-to-head odds are not established. The leaderboard order barely moves. With language-model judges and three-answer menus, a near-copy takes votes mostly from its twin when choices are pooled over prompts (log-odds shift +0.10, 95% interval 0.05 to 0.16); within a prompt the shift is smaller and clear for two of four judges.

Figures & tables

Figure 1: Decisiveness of votes between related models relative to unrelated pairs ( Equation 2 , ties dropped, pairs with at least 50 non-tied votes), with 95% intervals from resampling pairs. Both releases point the same way; the newer one has a thirteenth of the votes.

Check	version	size	lab
Main (ties dropped, $\geq$ 50 votes)	1.37	1.26	1.12
All ties kept (Davidson)	1.21	1.23	1.13
Ties as half wins (Arena)	1.19	1.10	1.06
$\geq$ 200 votes per pair	1.44	1.26	1.13
English only	1.33	1.17	1.11
De-duplicated prompts	1.37	1.25	1.10

Table 1: Decisiveness multipliers (version / size / same lab) on the 2024 log under alternative choices. Sources: ties , subsets , timing , clean , mixweight and gonogo.txt in results/2024/ .

	100k (2024)	140k (2025)
	481 pairs	981 pairs
Lineage labels (3 df)	15.5	13.3
Wording only	9.6, $\times$ 1.08	66.9, $\times$ 1.12
Style only	14.1, $\times$ 1.07	38.1, $\times$ 1.09
Meaning, first 128 tokens	2.2, $\times$ 1.03	26.8, $\times$ 1.07
Meaning, whole answer	–	20.3, $\times$ 1.06

Table 2: Decisiveness predicted by answer similarity: likelihood-ratio gain over Bradley–Terry and multiplier per standard deviation. “Adds” is the gain from adding the measure to the model in parentheses. Sources: results/*/similarity.txt ; 2024 meaning from results/2024/meaning/ .

Figure 2: Held-out gain over plain Bradley–Terry on votes between related models (millinats per vote). (a, b) Three-outcome loss of the fixes and rivals (mean and range over seeds or cutoffs; parameter counts in parentheses). (c, d) Each part of the fix alone (2024: lineage; 140k: similarity), with 95% intervals resampling model pairs. All votes: Table 3 ; all cells: Table 4 .

Figure 3: Frozen transfer to target votes between related models. Left: win-odds gain of the frozen correction over plain scores (95% intervals resampling labs). Right: calibration slope with plain scores (gray) and with the frozen correction (orange); 1 is calibrated. Details: Table 5 .

Figure 4: Left: similarity effect

\delta

per judge against the strength-matched placebo, pooled over prompts (mean over 16 designs) and within prompts (Mantel–Haenszel over design-by-prompt strata); diamonds: mean of the four judges. Right: the choice model’s nest parameter on three-answer menus;

\lambda=1

is Luce,

\lambda<1

the red bus. Bars: 95% intervals from resampling prompts.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Model-free check on the 2024 log. Each point is a model pair: the direct head-to-head log-odds against the log-odds implied through shared opponents. Dashed: Bradley–Terry (slope 1). Solid: inverse-variance weighted fit through the origin. Unrelated pairs follow the product rule; related pairs are more lopsided than the chain implies.

2024 log (129 models)
	random votes	unseen pairs	later votes
Lineage fix	+0.96 / +6.6	+1.13 / +8.4	+0.58 / +11.2
Covariance	+0.04 / +0.4	$-$ 0.78 / $-$ 0.1	+0.03 / +0.5
Gen. preference	+0.18 / +0.4	$-$ 0.05 / +0.2	+0.06 / +0.6
Per-pair scale	+0.09 / +0.4	–	–
140k release (53 models)

Appendix

Table 3: Held-out gain in three-outcome log loss over plain Bradley–Terry over all test votes and over votes between relatives (millinats per vote; means over seeds or cutoffs; rivals tuned on validation data). Sources: results/*/fix_heldout_validated.csv .

2024 log
	random votes	unseen pairs	later votes
lineage, ties only	+6.3 / 0.00	+8.0 / $-$ 0.02	+10.9 / +0.04
95% interval	(+4.2, +8.4) / ( $-$ 0.03, +0.02)	(+4.7, +11.6) / ( $-$ 0.06, +0.01)	(+7.6, +13.8) / ( $-$ 0.01, +0.08)
lineage, decisiveness only	+0.05 / +0.29	$-$ 0.01 / +0.22	$-$ 0.24 / +0.14
95% interval	( $-$ 0.04, +0.14) / (+0.13, +0.43)	( $-$ 0.14, +0.11) / (+0.03, +0.41)	( $-$ 0.54, $-$ 0.03) / ( $-$ 0.33, +0.51)
lineage, both	+6.6 / +0.33	+8.2 / +0.17	+10.9 / $-$ 0.05

Appendix

Table 4: Each part of the fixes alone: held-out gain over plain Bradley–Terry on votes between relatives, three-outcome / win-odds log loss (millinats per vote, pooled over seeds or cutoffs), with 95% intervals from resampling model pairs below each row, and the calibration slope on non-tied votes between relatives (200 pair draws). The similarity fix is the six-parameter version of Section 6 (wording, length gap and style).

setting	pairs / labs	correction	three-way gain	win-odds gain	slope	rmse (pp)	votes worth
2024 log $\to$ 140k	125 / 9	lineage	+9.74 ( $-$ 0.48, +19.30)	+0.14 ( $-$ 2.92, +1.68)	1.11 $\to$ 0.95	3.1 $\to$ 3.2	240 $\to$ 219
	125 / 9	temperature	+9.84 ( $-$ 0.20, +19.54)	+0.36 ( $-$ 2.19, +1.59)	1.11 $\to$ 0.96	3.1 $\to$ 3.0	240 $\to$ 248
100k $\to$ 140k	125 / 9	similarity	+14.00 (+1.16, +27.29)	+1.39 ( $-$ 1.32, +3.02)	1.11 $\to$ 1.00	3.1 $\to$ 2.8	240 $\to$ 290
2024 log, new releases (60th pct.)	86 / 10	lineage	+7.76 (+5.86, +10.51)	$-$ 0.64 ( $-$ 1.51, +1.73)	1.03 $\to$ 0.88	2.2 $\to$ 2.8	432 $\to$ 277
	86 / 10	temperature	+7.65 (+5.76, +10.38)	$-$ 0.86 ( $-$ 1.77, +1.63)	1.03 $\to$ 0.87	2.2 $\to$ 3.0	432 $\to$ 244
2024 log, new releases (70th pct.)	68 / 10	lineage	+9.26 (+7.73, +11.48)	+0.38 ( $-$ 0.36, +1.69)	1.08 $\to$ 0.92	2.0 $\to$ 2.3	546 $\to$ 409

Appendix

Table 5: Frozen transfer: gains over plain Bradley–Terry on target votes between related models (millinats per vote; 95% intervals resampling labs), calibration slope on non-tied votes, rmse of the predicted head-to-head win rate (percentage points) and the number of head-to-head votes it is worth, plain

\to

corrected.

	140k release	2024, new releases
related pairs (non-tied votes), labs	125 (10,328), 9	68 (18,824), 10
reliability $\lambda$ (all / 50% / 25% of votes)	0.99 / 0.98 / 0.96–0.97	1.00 / 0.99 / 0.99
$k^{\ast}$ (all votes)	1.11 (1.02 to 1.22) [0.89 to 1.28]	1.10 (1.03 to 1.17) [1.02 to 1.24]
$k^{\ast}$ with 50% / 25% of votes (two seeds)	1.09–1.10 / 1.07–1.14	1.09–1.11 / 1.07–1.12
$k^{\ast}$ version	1.12 (0.41 to 1.89) [0.53 to 1.94]	1.43 (0.91 to 2.02) [1.14 to 2.46]
$k^{\ast}$ size	1.10 (0.95 to 1.30) [0.89 to 1.30]	1.24 (1.12 to 1.39) [1.16 to 1.61]

Appendix

Table 6: Out-of-sample decisiveness on the two targets (multipliers on the log-odds scale; 95% intervals resampling pairs in parentheses, labs in brackets). Gains: win-odds log loss over plain scores on related non-tied votes, millinats per vote; “best” multipliers are fitted to the target’s own related votes, an in-sample upper bound for frozen shared or relation-specific multipliers applied to fixed target scores, not for better preference prediction in general. A few lab draws contain no version (5) or size (2) pair of the 2024 target and are left out of those intervals.

	140k release	2024, new releases
joint estimation: in-sample $-\,k^{\ast}/\lambda$	$+0.034$ ( $+0.011$ to $+0.074$ ) [ $-0.016$ to $+0.075$ ]	$+0.019$ ( $-0.001$ to $+0.039$ ) [ $+0.007$ to $+0.043$ ]
shift: source shared $-$ target in-sample	$-0.01$ ( $-0.12$ to $+0.09$ ) [ $-0.22$ to $+0.23$ ]	$+0.05$ ( $-0.03$ to $+0.13$ ) [ $-0.12$ to $+0.14$ ]
overshoot: source shared $-\,k^{\ast}$	$+0.04$ ( $-0.08$ to $+0.15$ ) [ $-0.16$ to $+0.25$ ]	$+0.07$ ( $-0.01$ to $+0.15$ ) [ $-0.09$ to $+0.17$ ]
source version $-\,k^{\ast}$ version	$+0.29$ ( $-0.52$ to $+0.97$ ) [ $-0.52$ to $+1.31$ ]	$-0.02$ ( $-0.71$ to $+0.60$ ) [ $-1.04$ to $+0.61$ ]
source size $-\,k^{\ast}$ size	$+0.15$ ( $-0.07$ to $+0.32$ ) [ $-0.07$ to $+0.37$ ]	$+0.05$ ( $-0.12$ to $+0.17$ ) [ $-0.33$ to $+0.16$ ]
source lab $-\,k^{\ast}$ lab	$+0.01$ ( $-0.13$ to $+0.14$ ) [ $-0.22$ to $+0.27$ ]	$+0.09$ ( $-0.01$ to $+0.16$ ) [ $-0.16$ to $+0.21$ ]

Appendix

Table 7: Paired bootstrap of the differences behind the explanation (95% intervals resampling pairs in parentheses, labs in brackets; 1,000 draws each). None excludes zero under both schemes.

	GPT-4.1 nano	Claude Haiku 5.5	Gemini 2.5 FL	DeepSeek V3.2
$\delta$ , random $U$	$+0.13$ ( $+0.09$ , $+0.19$ )	$+0.08$ ( $-0.04$ , $+0.18$ )	$+0.11$ ( $+0.02$ , $+0.19$ )	$+0.07$ ( $-0.02$ , $+0.14$ )
$\delta$ , matched $U$	$+0.14$ ( $+0.08$ , $+0.20$ )	$+0.07$ ( $-0.04$ , $+0.18$ )	$+0.08$ ( $-0.03$ , $+0.16$ )	$+0.13$ ( $+0.06$ , $+0.21$ )
designs with $\delta>0$	14 of 16	13 of 16	11 of 16	13 of 16
$\lambda$ , version pairs	0.75 (0.64, 0.87)	0.67 (0.56, 0.77)	0.85 (0.75, 0.98)	0.87 (0.75, 1.02)
$\lambda$ , size pairs	0.86 (0.77, 0.97)	1.03 (0.84, 1.24)	0.66 (0.56, 0.76)	1.10 (0.91, 1.31)
$\lambda$ , same-lab pairs	0.84 (0.73, 0.97)	0.99 (0.81, 1.21)	0.81 (0.67, 0.99)	0.97 (0.82, 1.17)

Appendix

Table 8: Judge experiment per judge: similarity effect

\delta

against the random and the matched placebo, and the choice model’s nest parameters

\lambda

for pairs and for three-answer menus, with 95% intervals from resampling prompts.

	Version	Size	Same lab
model pairs	2	6	6
score correlation	0.50	0.27	0.33
excess over unrelated	$+0.36$	$+0.13$	$+0.19$
	(0.24, 0.50)	(0.06, 0.20)	(0.13, 0.26)
implied $k$	1.31	1.08	1.13
decoy log odds ratio	$+0.17$	$-0.03$	$+0.04$

Appendix

Table 9: PRISM human ratings by relation, with 95% intervals from resampling participants. Implied

k

applies Proposition 1 with

\rho_{0}=0.14

Discrete Choice Modeling

Discrete choice modeling studies how people select among alternatives and how preferences can be estimated from observed choices.

See all 45 papers

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings
Aug 16, 2025Jenny Y. Huang, Yunyi Shen, Dennis Wei +1Pairwise Preference Learning LLM Evaluation
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology · MIT-IBM Watson AI Lab · IBM Research
When LLM Judge Scores Look Good but Best-of-N Decisions Fail
Mar 12, 2026Eddie LandesbergLLM Evaluation LLM-as-a-Judge
Personalized Benchmarking: Evaluating LLMs by Individual Preferences
Apr 21, 2026Cristina Garbacea, Heran Wang, Chenhao TanLLM Evaluation User Preference Modeling
University of Chicago

Aug 16, 2025J/K move · Enter open · S save

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

Jenny Y. Huang, Yunyi Shen, Dennis Wei, Tamara Broderick

Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology · MIT-IBM Watson AI Lab · IBM Research

Pairwise Preference Learning LLM Evaluation Pairwise Comparison Pairwise Preference Evaluation Learning to Rank Pairwise Ranking+1

Open paper AskPDF arXiv

We propose a method for evaluating the robustness of widely used LLM ranking systems -- variants of a Bradley--Terry model -- to dropping a worst-case very small fraction of preference data. Our approach is computationally fast and easy to adopt. When we apply our method to matchups from popular LLM ranking platforms, including Chatbot Arena and derivatives, we find that the rankings of top-performing models can be remarkably sensitive to the removal of a small fraction of preferences; for instance, dropping just 0.003% of human preferences can change the top-ranked model on Chatbot Arena. Our robustness check identifies the specific preferences most responsible for such ranking flips, allowing for inspection of these influential preferences. We observe that the rankings derived from MT-bench preferences are notably more robust than those from Chatbot Arena, likely due to MT-bench's use of expert annotators and carefully constructed prompts. Finally, we find that neither rankings based on crowdsourced human evaluations nor those based on LLM-as-a-judge preferences are systematically more sensitive than the other.

Abstract

Discrete Choice Modeling

Explore similar work

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

When LLM Judge Scores Look Good but Best-of-N Decisions Fail

Personalized Benchmarking: Evaluating LLMs by Individual Preferences