Users often complain that AI assistants such as ChatGPT are getting worse. Some versions do change, but human evaluation leaves a second possibility untested: the people judging may change. We separate the two on Chatbot Arena, a public platform where people compare answers from two anonymous large language models (LLMs) and pick the better one or say both are bad. We study 1,260,967 blind votes cast from May 2023 to August 2024 on pairs of models whose versions are fixed: dated vendor snapshots and models with fixed open weights. For the same pair of unchanged models, the share of votes saying "both are bad" rises by 1.71 percentage points a month on a mean of 17.1%, and by 1.91 points a month in 2024. In fourteen pairs observed often in both 2023 and 2024, it rose in every one, from 15.5% to 23.3%. The ranking does not trend: in 2024, the stronger model's share of decisive votes is flat. The rise holds within the same voter and on the same pair, and across arrival cohorts from their first recorded vote. It holds with prompt controls, in every major language, for dated vendor snapshots, open-weight models and models with stable answer length, and between interface changes. The drift is shared across models: month effects estimated from other models' pairs remove it from each model's own pairs. Platform-wide, the average model would be expected to score 78% against the 2023 average, yet the share of "both are bad" votes ended near where it began. The pattern fits a standard that follows the category's progress, so a product that stands still is judged as declining. Absolute verdicts on AI drifted while relative comparisons held steady, an evaluation drift that matters for LLM leaderboards and for the human feedback used to train models.
Figures & tables
Blind battles, 1 May 2023 to 14 Aug 2024
1,795,852
between two frozen models
1,260,967
between two dated API snapshots
430,370
between two dated OpenAI or Anthropic snapshots
220,514
between two fixed open-weight models
268,423
Models
129
dated API snapshots
33
Table 1: Sample
Figure 1: Better models, steady verdicts. (a) Average Bradley–Terry rating (Elo scale) of the models in each month’s battles, and the rating of the best model with at least 500 battles that month. (b) Change since May 2023 in the share of “both are bad” votes: raw, across all battles (blue), and within the same frozen model pair (orange; calendar-month effects from a regression with pair fixed effects; shaded band is the 95% confidence interval, standard errors clustered by pair and week).
Absolute verdict (share of all votes)
Relative verdict
“Both are bad”
One answer wins
Tie
Stronger model wins
A. May 2023 to August 2024
Change per month (pp)
+1.71∗∗∗
−1.62∗∗∗
−0.09
+0.13
(0.18)
(0.14)
(0.22)
(0.13)
Mean (%)
17.1
63.9
19.0
66.7
Votes
1,260,967
1,260,967
1,260,967
646,299
Table 2: Verdicts on frozen model pairs over time: absolute and relative
Figure 2: Every pair judged often in both periods became “both are bad” more often. Frozen model pairs with at least 50 votes in May–October 2023 and in February–August 2024; vote counts in parentheses (early/late). Bars are 95% binomial confidence intervals.
Figure 3: Absolute verdicts drift; the relative verdict is flat from late 2023. Calendar-month effects (relative to May 2023) from regressions with model-pair fixed effects. (a) Shares of all votes on frozen pairs. (b) Share of decisive votes won by the higher-rated model, pairs whose ratings differ by at least 20 Elo points. Shaded bands are 95% confidence intervals, standard errors clustered by pair and week. The grey area marks 2024.
Specification
pp/month
(SE)
Mean (%)
Obs.
Main estimate
+1.71∗∗∗
(0.18)
17.1
1,260,967
Prompts
Prompt content: language fixed effects, code, math, seven difficulty tags
+1.63∗∗∗
(0.15)
17.1
1,260,967
and conversation: multi-turn, prompt length
+1.57∗∗∗
(0.15)
17.1
1,260,967
and answers: refusal, answer length
+1.59∗∗∗
(0.15)
17.1
1,260,967
Single-turn conversations, prompt-content controls and prompt length
+1.60∗∗∗
(0.17)
17.2
1,087,718
Table 3: Robustness: change in “both are bad” per month within frozen model pairs
Figure 4: One estimate, many cuts. Change in the share of “both are bad” votes per month within frozen model pairs, with 95% confidence intervals. The dashed line marks the main estimate.
Figure 5: Newcomers arrive pickier. Calendar-month effects (relative to May 2023) on “both are bad” within frozen model pairs, for all votes and for each judge identifier’s first recorded vote by month of arrival. Shaded bands are 95% confidence intervals, standard errors clustered by pair and week.
Regressor (outcome: “both are bad”, pp)
Per
Coef.
(SE)
Votes
Same identifier over time
Identifiers seen over ≥ 1 month (identifier and pair FE)
month
+1.90∗∗∗
(0.13)
365,074
Identifiers seen over ≥ 3 months
month
+1.74∗∗∗
(0.16)
197,193
Same identifier on the same pair, judged ≥ 1 month apart (identifier-by-pair FE)
month
+2.30∗∗∗
(0.68)
11,642
with prompt-content controls
month
+2.24∗∗∗
(0.67)
11,642
Identifiers seen over ≥ 1 month, controlling for log(1 + earlier votes)
month
+1.50∗∗∗
(0.13)
365,074
Table 4: Who becomes pickier
Figure 6: A shared drift. For each frozen model with at least 10,000 votes over at least four months: the trend in “both are bad” on the model’s own pairs (orange) and the trend after subtracting calendar-month effects estimated only from frozen pairs that do not involve the model (blue). Pair fixed effects; 95% confidence intervals, standard errors clustered by pair and week.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Model
First
Last
Appearances
Dated API snapshots (33)
gpt-4-0314 †
2023-05
2024-06
59,852
gpt-3.5-turbo-0314 †
2023-05
2023-06
5,841
gpt-4-0613 †
2023-06
2024-08
96,284
gpt-3.5-turbo-0613 †
2023-06
2024-03
40,549
claude-2.0 †
2023-07
2024-03
13,323
Appendix
Table 11
Model
Class
Months
Answers
%/month
(SE)
Stable
chatglm-6b
open weights
3.4
2,885
−22.7
(5.2)
alpaca-13b
open weights
3.7
3,360
−15.4
(4.0)
oasst-pythia-12b
open weights
3.4
3,609
−14.2
(5.9)
fastchat-t5-3b
open weights
3.6
2,894
−7.8
(4.6)
gpt-4-0125-preview
snapshot
6.6
66,961
−4.0
(1.0)
qwen1.5-72b-chat
open weights
4.2
31,750
−3.9
(1.6)
Appendix
Table 12
Prompt language
pp/month
(SE)
Mean (%)
Votes
English
+1.53
(0.15)
17.2
716,620
Chinese
+2.38
(0.33)
18.6
132,682
Russian
+1.78
(0.31)
19.0
113,583
unknown
+1.72
(0.70)
10.5
69,549
German
+1.59
(0.33)
16.4
35,544
Korean
+3.75
(0.67)
15.8
31,229
Appendix
Table 7: Change in “both are bad” per month within frozen pairs, by prompt language
Date
Commit
Lines
Description
Jump (pp)
(SE)
2023-05-08
c1985314
74
Improve docs and format code
−3.09
(1.77)
2023-06-14
07abdb08
58
Misc improvements to the gradio web server
−2.06
(2.60)
2023-06-24
78fa7b4e
50
Update gradio version to 3.35
+2.88
(2.33)
2023-11-12
e0b351a4
246
UI and model change
−1.91
(0.99)
2023-12-09
d82f7ec3
77
Update UI and new models
+0.76
(1.02)
2023-12-24
a28563b9
110
Add new models (Perplexity, gemini) and Separate GPT versions
+0.25
(0.92)
Appendix
Table 8: Jumps in “both are bad” at major changes to the anonymous battle page
Date
New top-rated model
Jump (pp)
(SE)
Votes ( ± 21 days)
2023-11-07
gpt-4-1106-preview
+1.26
(1.56)
26,931
2024-04-10
gpt-4-turbo-2024-04-09
−1.29
(0.63)
311,723
2024-05-14
gpt-4o-2024-05-13
+4.95
(0.91)
427,307
2024-07-18
gemini-1.5-pro-exp-0801
+1.09
(1.21)
199,469
2024-08-07
chatgpt-4o-latest
+8.70
(0.46)
147,034
Appendix
Table 9: Jumps in “both are bad” at arrivals of a new top-rated model
Observed controls
Maximum R2
δ for zero trend
Trend at δ=1 (pp/month)
Prompt content and conversation
0.026 (1.3 × controlled R2 )
35.9
+1.53
0.040 (2 × controlled R2 )
10.8
+1.42
0.050
7.1
+1.35
0.100
2.7
+0.98
Prompt content only
0.022 (1.3 × controlled R2 )
67.5
+1.61
0.035 (2 × controlled R2 )
20.2
+1.55
Appendix
Table 10: Coefficient-stability sensitivity for the main trend