What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations
Authors: Xiaonan Xu, Wenjing Wu
Organizations: College of Computing, Georgia Institute of Technology, Atlanta, GA, 30332, USA · Department of Computer Science, University of Colorado Boulder, Boulder, CO, 80309, USA
Context: Software systems that depend on commercial large language model APIs must migrate to successor versions when vendors deprecate older models. Migration decisions typically rely on aggregate benchmark scores, which compress heterogeneous item-level behaviour into a single net figure. Objective: We measure what that compression conceals. Method: On three pairwise upgrades in the GPT-5.4 to GPT-5.6 Sol product sequence, we query 900 public benchmark items (graduate-level knowledge, olympiad mathematics, instruction following) 50 times per item per model, classify each item as reliably improved, reliably regressed, practically equivalent, or inconclusive under false-discovery-rate control and a practical-significance threshold, and calibrate the results against a label-permutation null. Results: Across all nine migration-benchmark cells, reliable improvements and reliable regressions coexist. Edges with aggregate gains of up to 7.3 percentage points contain up to 8.3% reliably regressed items; edges with aggregate losses contain up to 10.7% reliably improved items. On the instruction-following benchmark, the gap between strict and loose scoring widens by 3.9 percentage points on the latest migration: a 3.9-point regression under strict scoring shrinks to 0.04 points under loose scoring. Conclusion: Migration decisions based on aggregate scores alone miss substantial bidirectional item-level change. The complete response-level archive and per-item scoring outputs are released.
Figures & tables
Edge
Benchmark
Δ
P+
P−
Pchg
Equiv.
Incon.
Null 95
5.4 → 5.5
SuperGPQA
+ 2.3
8.2
5.0
13.2
77.0
9.8
0.0
Omni-MATH hard
+ 7.3
17.0
6.0
23.0
59.0
18.0
0.0
IFBench
+ 1.9
11.3
8.3
19.7
55.7
24.7
0.0
5.5 → Sol
SuperGPQA
+ 1.4
7.6
4.4
12.0
80.4
7.6
0.0
Omni-MATH hard
− 4.1
9.0
15.0
24.0
67.0
9.0
0.0
IFBench
− 3.9
6.7
13.3
20.0
57.0
23.0
0.0
Table 1: Aggregate change and item-level classification for each migration edge and benchmark. Δ is the change in mean strict accuracy in percentage points. P+ , P− , and Pchg are the shares of items classified as reliably improved, reliably regressed, and reliably changed under the Fisher/BH/ ε=0.2 criteria; Equiv. and Incon. are the practically-equivalent and inconclusive shares. Null 95 is the 95th percentile of Pchg under the permutation null (1,000 replications). All shares in percent; Δ is computed before rounding
Figure 1: Item-level reliable improvement and regression alongside aggregate change for each migration edge and benchmark. Bars extending right (left) show the share of items reliably improved (regressed); the central marker shows the aggregate change Δ
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Slot
Candidate
Estimable
Share (%)
Outcome
Knowledge
SuperGPQA
13/30
43.3
Selected
GPQA Diamond
8/30
26.7
Not selected
SuperGPQA hard
7/30
23.3
Not selected
MMLU-Pro
2/30
6.7
Not selected
Mathematics
Omni-MATH hard
21/30
70.0
Selected
OlymMATH
16/30
53.3
Not selected
Appendix
Table 2: Benchmark screening: estimable items out of 30 per candidate ( K=10 , three models). One benchmark per slot was selected, requiring a share of at least 35% and taking the highest-share candidate per slot
Edge
Benchmark
Items
K=50 rel. changes
Single-draw rel. changes
I → C
C → I
5.4 → 5.5
SuperGPQA
500
66
0
27
20
Omni-MATH hard
100
23
0
14
4
IFBench
300
59
0
42
19
5.5 → Sol
SuperGPQA
500
60
0
28
19
Omni-MATH hard
100
24
0
6
10
IFBench
300
60
0
25
47
Appendix
Table 3: Single-draw comparison. Reliable changes under the full K=50 procedure, reliable changes detected from single draws, and raw flips between the single draws of source and target model (I → C: incorrect to correct; C → I: correct to incorrect)
K=50 classification
I → C
C → I
Unchanged
Total
Reliably improved
120
8
127
255
Reliably regressed
10
87
105
202
Practically equivalent
25
26
1817
1868
Inconclusive
66
54
255
375
Total
221
175
2304
2700
Appendix
Table 4: Raw single-draw flips cross-tabulated against the K=50 classification, aggregated over the nine migration–benchmark cells (2,700 item–edge pairs)
Edge
Benchmark
ε
P+
P−
Pchg
5.4 → 5.5
SuperGPQA
0.10
8.4
5.2
13.6
0.15
8.4
5.2
13.6
0.20
8.2
5.0
13.2
0.25
8.0
4.6
12.6
Omni-MATH hard
0.10
19.0
6.0
25.0
0.15
19.0
6.0
25.0
Appendix
Table 5: P+ , P− , and Pchg per migration–benchmark cell for four values of the practical-significance threshold ε . All shares in percent
Mean
Total tokens
Benchmark
Model
Input
Output
Mean
Median
P5
P95
SuperGPQA
GPT-5.4
543.2
555.2
1098.4
758
468
2692
GPT-5.5
543.2
543.5
1086.7
824
494
2767
Sol
543.8
409.7
953.5
702
476
2244
Omni-MATH hard
GPT-5.4
444.8
7653.3
8098.1
5732
1521
20779
GPT-5.5
444.8
3705.3
4150.1
3684
991
8751
Appendix
Table 6: Per-request token counts by benchmark and model. Mean input and output tokens, and mean, median, 5th and 95th percentile of total tokens
Edge
Benchmark
Source
Target
Change
5.4 → 5.5
SuperGPQA
1098.4
1086.7
− 11.7
Omni-MATH hard
8098.1
4150.1
− 3948.0
IFBench
1405.3
1357.1
− 48.1
5.5 → Sol
SuperGPQA
1086.7
953.5
− 133.2
Omni-MATH hard
4150.1
4211.0
+ 60.9
IFBench
1357.1
1847.0
+ 489.8
Appendix
Table 7: Mean total tokens per request on each migration edge
Benchmark
Model
N
Correct
Wrong
Format
Refusal
Trunc.
API err.
SuperGPQA
GPT-5.4
25000
15841
8913
246
0
0
0
GPT-5.5
25000
16424
8205
371
0
0
0
Sol
25000
16780
7932
288
0
0
0
Omni-MATH hard
GPT-5.4
5000
2285
2689
26
0
0
0
GPT-5.5
5000
2648
2326
26
0
0
0
Sol
5000
2445
2555
0
0
0
0
Appendix
Table 8: Six-category response classification by benchmark and model (counts; N is items ×K=50 )
Benchmark
Stratum
Items
5.4
5.5
Sol
Δ5.4→5.5
Δ5.5→Sol
Δ5.4→Sol
SuperGPQA
Difficulty easy
156
64.0
69.3
72.4
+ 5.3
+ 3.1
+ 8.4
Difficulty middle
204
69.8
70.8
72.6
+ 0.9
+ 1.8
+ 2.7
Difficulty hard
140
53.2
54.3
53.3
+ 1.0
− 1.0
+ 0.1
Agronomy
10
23.2
23.4
27.4
+ 0.2
+ 4.0
+ 4.2
Economics
17
74.0
76.7
78.6
+ 2.7
+ 1.9
+ 4.6
Education
10
50.6
67.2
72.8
+ 16.6
+ 5.6
+ 22.2
Appendix
Table 9: Mean strict accuracy (%) per model and change per edge (percentage points) within benchmark-native strata
We adapted the Reliable Change Index (RCI; Jacobson and Truax, 1991) from clinical psychology to item-level LLM version comparison on 2,000 MMLU-Pro items (K=10 samples at T=0.7). Two within-family pairs were tested: Llama 3 to 3.1 (+1.6 points) and Qwen 2.5 to 3 (+2.8 points). On the full benchmark, most items showed no reliable change (79% and 72%). However, over half the items were floor/ceiling. Among analysable items, change was bidirectional with large effect sizes: 34% improved and 28% deteriorated for Llama; 47% improved and 39% deteriorated for Qwen (median |delta p| = 0.50 and 0.90). Churn was asymmetric by difficulty: low-accuracy items improved, high-accuracy items deteriorated. Domain-level decomposition revealed family-specific reversals: Llama lost physics while Qwen lost law. Greedy single-shot evaluation missed 42% of reliably changed items and falsely flagged 25% of unchanged items. The aggregate accuracy gain is the net residual of opposing item-level movements. We recommend reporting churn rate alongside aggregate accuracy.
Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems. We challenge this assumption by auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, and sycophancy. We find systematic API--interface differences in both accuracy and consistency. On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test--retest agreement than corresponding interface evaluations. For ChatGPT, the performance difference between API and interface access rivals the API-only difference between GPT 5.3 and GPT 5.4. Put differently, switching access surfaces can degrade performance as much as downgrading a full model version. We further test whether exposed API controls can reproduce interface behavior by varying system prompts, sampling parameters, and reasoning settings. These controls shift behavior in some cases but do not reliably eliminate the gap. Our findings document a context-validity gap: measurements obtained through APIs do not necessarily generalize to corresponding deployed interfaces, complicating the use of API evaluations as proxies for deployed systems.
Frontier LLMs are updated frequently and typically outperform their predecessors in aggregate. But aggregate gains say little about individual samples: an update can still cause sample-level regression, where a response correct under the old model becomes incorrect under the new one. This paper studies how to predict such regressions from signals available at inference time. We compare single-model signals (confidence, logit margin, attention entropy) against cross-version signals (output KL divergence, likelihood drift, token-level KL, representation drift) under a unified added-value test that isolates each signal's gain over a confidence baseline. Across six benchmarks in three task families (multiple-choice question answering, or MCQ; math reasoning; code generation) and six model update pairs, we find that (1) signal effectiveness is task-dependent: confidence is strongest on MCQ and simpler math, while likelihood/KL signals give the most frequent gains on harder math and code; (2) no signal is universally best across model updates either; and (3) some cross-version signals stay informative even when confidence fails, including without labels, which supports a proof-of-concept selective fallback that routes high-risk samples back to the old model. Practitioners can use these task-level patterns to choose which regression signal to trust for a given update. Code is available at https://github.com/jiashengsally/llm-regression-signals.