Users often complain that AI assistants such as ChatGPT are getting worse. Some versions do change, but human evaluation leaves a second possibility untested: the people judging may change. We separate the two on Chatbot Arena, a public platform where people compare answers from two anonymous large language models (LLMs) and pick the better one or say both are bad. We study 1,260,967 blind votes cast from May 2023 to August 2024 on pairs of models whose versions are fixed: dated vendor snapshots and models with fixed open weights. For the same pair of unchanged models, the share of votes saying "both are bad" rises by 1.71 percentage points a month on a mean of 17.1%, and by 1.91 points a month in 2024. In fourteen pairs observed often in both 2023 and 2024, it rose in every one, from 15.5% to 23.3%. The ranking does not trend: in 2024, the stronger model's share of decisive votes is flat. The rise holds within the same voter and on the same pair, and across arrival cohorts from their first recorded vote. It holds with prompt controls, in every major language, for dated vendor snapshots, open-weight models and models with stable answer length, and between interface changes. The drift is shared across models: month effects estimated from other models' pairs remove it from each model's own pairs. Platform-wide, the average model would be expected to score 78% against the 2023 average, yet the share of "both are bad" votes ended near where it began. The pattern fits a standard that follows the category's progress, so a product that stands still is judged as declining. Absolute verdicts on AI drifted while relative comparisons held steady, an evaluation drift that matters for LLM leaderboards and for the human feedback used to train models.
Figures & tables
Blind battles, 1 May 2023 to 14 Aug 2024
1,795,852
between two frozen models
1,260,967
between two dated API snapshots
430,370
between two dated OpenAI or Anthropic snapshots
220,514
between two fixed open-weight models
268,423
Models
129
dated API snapshots
33
Table 1: Sample
Figure 1: Better models, steady verdicts. (a) Average Bradley–Terry rating (Elo scale) of the models in each month’s battles, and the rating of the best model with at least 500 battles that month. (b) Change since May 2023 in the share of “both are bad” votes: raw, across all battles (blue), and within the same frozen model pair (orange; calendar-month effects from a regression with pair fixed effects; shaded band is the 95% confidence interval, standard errors clustered by pair and week).
Absolute verdict (share of all votes)
Relative verdict
“Both are bad”
One answer wins
Tie
Stronger model wins
A. May 2023 to August 2024
Change per month (pp)
+1.71∗∗∗
−1.62∗∗∗
−0.09
+0.13
(0.18)
(0.14)
(0.22)
(0.13)
Mean (%)
17.1
63.9
19.0
66.7
Votes
1,260,967
1,260,967
1,260,967
646,299
Table 2: Verdicts on frozen model pairs over time: absolute and relative
Figure 2: Every pair judged often in both periods became “both are bad” more often. Frozen model pairs with at least 50 votes in May–October 2023 and in February–August 2024; vote counts in parentheses (early/late). Bars are 95% binomial confidence intervals.
Figure 3: Absolute verdicts drift; the relative verdict is flat from late 2023. Calendar-month effects (relative to May 2023) from regressions with model-pair fixed effects. (a) Shares of all votes on frozen pairs. (b) Share of decisive votes won by the higher-rated model, pairs whose ratings differ by at least 20 Elo points. Shaded bands are 95% confidence intervals, standard errors clustered by pair and week. The grey area marks 2024.
Specification
pp/month
(SE)
Mean (%)
Obs.
Main estimate
+1.71∗∗∗
(0.18)
17.1
1,260,967
Prompts
Prompt content: language fixed effects, code, math, seven difficulty tags
+1.63∗∗∗
(0.15)
17.1
1,260,967
and conversation: multi-turn, prompt length
+1.57∗∗∗
(0.15)
17.1
1,260,967
and answers: refusal, answer length
+1.59∗∗∗
(0.15)
17.1
1,260,967
Single-turn conversations, prompt-content controls and prompt length
+1.60∗∗∗
(0.17)
17.2
1,087,718
Table 3: Robustness: change in “both are bad” per month within frozen model pairs
Figure 4: One estimate, many cuts. Change in the share of “both are bad” votes per month within frozen model pairs, with 95% confidence intervals. The dashed line marks the main estimate.
Figure 5: Newcomers arrive pickier. Calendar-month effects (relative to May 2023) on “both are bad” within frozen model pairs, for all votes and for each judge identifier’s first recorded vote by month of arrival. Shaded bands are 95% confidence intervals, standard errors clustered by pair and week.
Regressor (outcome: “both are bad”, pp)
Per
Coef.
(SE)
Votes
Same identifier over time
Identifiers seen over ≥ 1 month (identifier and pair FE)
month
+1.90∗∗∗
(0.13)
365,074
Identifiers seen over ≥ 3 months
month
+1.74∗∗∗
(0.16)
197,193
Same identifier on the same pair, judged ≥ 1 month apart (identifier-by-pair FE)
month
+2.30∗∗∗
(0.68)
11,642
with prompt-content controls
month
+2.24∗∗∗
(0.67)
11,642
Identifiers seen over ≥ 1 month, controlling for log(1 + earlier votes)
month
+1.50∗∗∗
(0.13)
365,074
Table 4: Who becomes pickier
Figure 6: A shared drift. For each frozen model with at least 10,000 votes over at least four months: the trend in “both are bad” on the model’s own pairs (orange) and the trend after subtracting calendar-month effects estimated only from frozen pairs that do not involve the model (blue). Pair fixed effects; 95% confidence intervals, standard errors clustered by pair and week.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Model
First
Last
Appearances
Dated API snapshots (33)
gpt-4-0314 †
2023-05
2024-06
59,852
gpt-3.5-turbo-0314 †
2023-05
2023-06
5,841
gpt-4-0613 †
2023-06
2024-08
96,284
gpt-3.5-turbo-0613 †
2023-06
2024-03
40,549
claude-2.0 †
2023-07
2024-03
13,323
Appendix
Table 11
Model
Class
Months
Answers
%/month
(SE)
Stable
chatglm-6b
open weights
3.4
2,885
−22.7
(5.2)
alpaca-13b
open weights
3.7
3,360
−15.4
(4.0)
oasst-pythia-12b
open weights
3.4
3,609
−14.2
(5.9)
fastchat-t5-3b
open weights
3.6
2,894
−7.8
(4.6)
gpt-4-0125-preview
snapshot
6.6
66,961
−4.0
(1.0)
qwen1.5-72b-chat
open weights
4.2
31,750
−3.9
(1.6)
Appendix
Table 12
Prompt language
pp/month
(SE)
Mean (%)
Votes
English
+1.53
(0.15)
17.2
716,620
Chinese
+2.38
(0.33)
18.6
132,682
Russian
+1.78
(0.31)
19.0
113,583
unknown
+1.72
(0.70)
10.5
69,549
German
+1.59
(0.33)
16.4
35,544
Korean
+3.75
(0.67)
15.8
31,229
Appendix
Table 7: Change in “both are bad” per month within frozen pairs, by prompt language
Date
Commit
Lines
Description
Jump (pp)
(SE)
2023-05-08
c1985314
74
Improve docs and format code
−3.09
(1.77)
2023-06-14
07abdb08
58
Misc improvements to the gradio web server
−2.06
(2.60)
2023-06-24
78fa7b4e
50
Update gradio version to 3.35
+2.88
(2.33)
2023-11-12
e0b351a4
246
UI and model change
−1.91
(0.99)
2023-12-09
d82f7ec3
77
Update UI and new models
+0.76
(1.02)
2023-12-24
a28563b9
110
Add new models (Perplexity, gemini) and Separate GPT versions
+0.25
(0.92)
Appendix
Table 8: Jumps in “both are bad” at major changes to the anonymous battle page
Date
New top-rated model
Jump (pp)
(SE)
Votes ( ± 21 days)
2023-11-07
gpt-4-1106-preview
+1.26
(1.56)
26,931
2024-04-10
gpt-4-turbo-2024-04-09
−1.29
(0.63)
311,723
2024-05-14
gpt-4o-2024-05-13
+4.95
(0.91)
427,307
2024-07-18
gemini-1.5-pro-exp-0801
+1.09
(1.21)
199,469
2024-08-07
chatgpt-4o-latest
+8.70
(0.46)
147,034
Appendix
Table 9: Jumps in “both are bad” at arrivals of a new top-rated model
Observed controls
Maximum R2
δ for zero trend
Trend at δ=1 (pp/month)
Prompt content and conversation
0.026 (1.3 × controlled R2 )
35.9
+1.53
0.040 (2 × controlled R2 )
10.8
+1.42
0.050
7.1
+1.35
0.100
2.7
+0.98
Prompt content only
0.022 (1.3 × controlled R2 )
67.5
+1.61
0.035 (2 × controlled R2 )
20.2
+1.55
Appendix
Table 10: Coefficient-stability sensitivity for the main trend
We propose a method for evaluating the robustness of widely used LLM ranking systems -- variants of a Bradley--Terry model -- to dropping a worst-case very small fraction of preference data. Our approach is computationally fast and easy to adopt. When we apply our method to matchups from popular LLM ranking platforms, including Chatbot Arena and derivatives, we find that the rankings of top-performing models can be remarkably sensitive to the removal of a small fraction of preferences; for instance, dropping just 0.003% of human preferences can change the top-ranked model on Chatbot Arena. Our robustness check identifies the specific preferences most responsible for such ranking flips, allowing for inspection of these influential preferences. We observe that the rankings derived from MT-bench preferences are notably more robust than those from Chatbot Arena, likely due to MT-bench's use of expert annotators and carefully constructed prompts. Finally, we find that neither rankings based on crowdsourced human evaluations nor those based on LLM-as-a-judge preferences are systematically more sensitive than the other.
Jenny Y. Huang, Yunyi Shen, Dennis Wei +1
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology · MIT-IBM Watson AI Lab · IBM Research
Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down. But the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores -- so every drift alarm is ambiguous between a worse product and a changed judge. We resolve the ambiguity with a fixed, human-labeled anchor set that the current judge re-scores at a steady interleave, a second betting e-process on the judge-versus-human gap, and a guard-window rule returning a verdict in {none, system, judge}. We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard, and process orthogonality. On two real judge changes, a silent version bump is detected as judge drift in 60/60 runs with zero judge-to-system misattribution, and a contaminating strict-prompt change is correctly attributed on 110 of 120 runs at guard width 300 -- while the industry-default rolling z-test false-alarms on 75% of drift-free streams. Every experiment replicates on a second domain (TL;DR summarization) with nothing re-tuned, and where the domains differ the differences are the ones the race predicts: the strict-prompt change shifts scores harder there, so the anchors fire faster and attribution becomes perfect (240/240). The monitor runs at approximately 0.64 of the cost of strong-judging every item, or 0.21 in a cheaper-but-deafer regime.
Large language models are often used as judges to score candidate responses, then validated with a single global metric such as correlation with reference labels. This can be misleading when the real deployment task is best-of-n selection within a prompt. In a 5,000-prompt best-of-4 benchmark from Chatbot Arena, a judge with moderate global correlation (r = 0.47) captures only 21.0% of the improvement that perfect selection would achieve over random choice. The gap arises because global agreement is driven largely by prompt-level baseline effects, while selection depends on within-prompt ranking: within-prompt correlation is only r_within = 0.27, and coarse pointwise scoring creates ties in 67% of pairwise comparisons. In a matched-pair best-of-2 audit, explicit pairwise judging recovers much of this lost signal, raising recovery from 21.1% to 61.2%. For judge-based selection, the relevant audit should report within-prompt signal, tie rates, and recovery/top-1 accuracy, not global agreement alone.