What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations

Xiaonan Xu; Wenjing Wu

cs.SEAug 18, 2026

What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations

Authors: Xiaonan Xu, Wenjing Wu

Organizations: College of Computing, Georgia Institute of Technology, Atlanta, GA, 30332, USA · Department of Computer Science, University of Colorado Boulder, Boulder, CO, 80309, USA

Abstract

Context: Software systems that depend on commercial large language model APIs must migrate to successor versions when vendors deprecate older models. Migration decisions typically rely on aggregate benchmark scores, which compress heterogeneous item-level behaviour into a single net figure. Objective: We measure what that compression conceals. Method: On three pairwise upgrades in the GPT-5.4 to GPT-5.6 Sol product sequence, we query 900 public benchmark items (graduate-level knowledge, olympiad mathematics, instruction following) 50 times per item per model, classify each item as reliably improved, reliably regressed, practically equivalent, or inconclusive under false-discovery-rate control and a practical-significance threshold, and calibrate the results against a label-permutation null. Results: Across all nine migration-benchmark cells, reliable improvements and reliable regressions coexist. Edges with aggregate gains of up to 7.3 percentage points contain up to 8.3% reliably regressed items; edges with aggregate losses contain up to 10.7% reliably improved items. On the instruction-following benchmark, the gap between strict and loose scoring widens by 3.9 percentage points on the latest migration: a 3.9-point regression under strict scoring shrinks to 0.04 points under loose scoring. Conclusion: Migration decisions based on aggregate scores alone miss substantial bidirectional item-level change. The complete response-level archive and per-item scoring outputs are released.

Figures & tables

Edge	Benchmark	$\Delta$	$P^{+}$	$P^{-}$	$P^{\mathit{chg}}$	Equiv.	Incon.
5.4 $\to$ 5.5	SuperGPQA	$+$ 2.3	8.2	5.0	13.2	77.0	9.8
	Omni-MATH hard	$+$ 7.3	17.0	6.0	23.0	59.0	18.0
	IFBench	$+$ 1.9	11.3	8.3	19.7	55.7	24.7
5.5 $\to$ Sol	SuperGPQA	$+$ 1.4	7.6	4.4	12.0	80.4	7.6
	Omni-MATH hard	$-$ 4.1	9.0	15.0	24.0	67.0	9.0
	IFBench	$-$ 3.9	6.7	13.3	20.0	57.0	23.0

Table 1: Aggregate change and item-level classification for each migration edge and benchmark.

\Delta

is the change in mean strict accuracy in percentage points.

P^{+}

P^{-}

, and

P^{\mathit{chg}}

are the shares of items classified as reliably improved, reliably regressed, and reliably changed under the Fisher/BH/

\varepsilon{=}0.2

criteria; Equiv. and Incon. are the practically-equivalent and inconclusive shares. Null 95 is the 95th percentile of

P^{\mathit{chg}}

under the permutation null (1,000 replications). All shares in percent;

\Delta

is computed before rounding

Figure 1: Item-level reliable improvement and regression alongside aggregate change for each migration edge and benchmark. Bars extending right (left) show the share of items reliably improved (regressed); the central marker shows the aggregate change

\Delta

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Slot	Candidate	Estimable	Share (%)	Outcome
Knowledge	SuperGPQA	13/30	43.3	Selected
	GPQA Diamond	8/30	26.7	Not selected
	SuperGPQA hard	7/30	23.3	Not selected
	MMLU-Pro	2/30	6.7	Not selected
Mathematics	Omni-MATH hard	21/30	70.0	Selected
	OlymMATH	16/30	53.3	Not selected

Appendix

Table 2: Benchmark screening: estimable items out of 30 per candidate (

K{=}10

, three models). One benchmark per slot was selected, requiring a share of at least 35% and taking the highest-share candidate per slot

Edge	Benchmark	Items	$K{=}50$ rel. changes	I $\to$ C	C $\to$ I
5.4 $\to$ 5.5	SuperGPQA	500	66	27	20
	Omni-MATH hard	100	23	14	4
	IFBench	300	59	42	19
5.5 $\to$ Sol	SuperGPQA	500	60	28	19
	Omni-MATH hard	100	24	6	10
	IFBench	300	60	25	47

Appendix

Table 3: Single-draw comparison. Reliable changes under the full

K{=}50

procedure, reliable changes detected from single draws, and raw flips between the single draws of source and target model (I

\to

C: incorrect to correct; C

\to

I: correct to incorrect)

$K{=}50$ classification	I $\to$ C	C $\to$ I	Unchanged	Total
Reliably improved	120	8	127	255
Reliably regressed	10	87	105	202
Practically equivalent	25	26	1817	1868
Inconclusive	66	54	255	375
Total	221	175	2304	2700

Appendix

Table 4: Raw single-draw flips cross-tabulated against the

K{=}50

classification, aggregated over the nine migration–benchmark cells (2,700 item–edge pairs)

Edge	Benchmark	$\varepsilon$	$P^{+}$	$P^{-}$	$P^{\mathit{chg}}$
5.4 $\to$ 5.5	SuperGPQA	0.10	8.4	5.2	13.6
		0.15	8.4	5.2	13.6
		0.20	8.2	5.0	13.2
		0.25	8.0	4.6	12.6
	Omni-MATH hard	0.10	19.0	6.0	25.0
		0.15	19.0	6.0	25.0

Appendix

Table 5:

P^{+}

P^{-}

, and

P^{\mathit{chg}}

per migration–benchmark cell for four values of the practical-significance threshold

\varepsilon

. All shares in percent

Benchmark	Model	Input	Output	Mean	Median	P5	P95
		Mean		Total tokens
SuperGPQA	GPT-5.4	543.2	555.2	1098.4	758	468	2692
	GPT-5.5	543.2	543.5	1086.7	824	494	2767
	Sol	543.8	409.7	953.5	702	476	2244
Omni-MATH hard	GPT-5.4	444.8	7653.3	8098.1	5732	1521	20779
	GPT-5.5	444.8	3705.3	4150.1	3684	991	8751

Appendix

Table 6: Per-request token counts by benchmark and model. Mean input and output tokens, and mean, median, 5th and 95th percentile of total tokens

Edge	Benchmark	Source	Target	Change
5.4 $\to$ 5.5	SuperGPQA	1098.4	1086.7	$-$ 11.7
	Omni-MATH hard	8098.1	4150.1	$-$ 3948.0
	IFBench	1405.3	1357.1	$-$ 48.1
5.5 $\to$ Sol	SuperGPQA	1086.7	953.5	$-$ 133.2
	Omni-MATH hard	4150.1	4211.0	$+$ 60.9
	IFBench	1357.1	1847.0	$+$ 489.8

Appendix

Table 7: Mean total tokens per request on each migration edge

Benchmark	Model	$N$	Correct	Wrong	Format
SuperGPQA	GPT-5.4	25000	15841	8913	246
	GPT-5.5	25000	16424	8205	371
	Sol	25000	16780	7932	288
Omni-MATH hard	GPT-5.4	5000	2285	2689	26
	GPT-5.5	5000	2648	2326	26
	Sol	5000	2445	2555	0

Appendix

Table 8: Six-category response classification by benchmark and model (counts;

N

is items

\times

K{=}50

)

Benchmark	Stratum	Items	5.4	5.5	Sol	$\Delta_{5.4\to 5.5}$	$\Delta_{5.5\to\mathrm{Sol}}$	$\Delta_{5.4\to\mathrm{Sol}}$
SuperGPQA	Difficulty easy	156	64.0	69.3	72.4	$+$ 5.3	$+$ 3.1	$+$ 8.4
	Difficulty middle	204	69.8	70.8	72.6	$+$ 0.9	$+$ 1.8	$+$ 2.7
	Difficulty hard	140	53.2	54.3	53.3	$+$ 1.0	$-$ 1.0	$+$ 0.1
	Agronomy	10	23.2	23.4	27.4	$+$ 0.2	$+$ 4.0	$+$ 4.2
	Economics	17	74.0	76.7	78.6	$+$ 2.7	$+$ 1.9	$+$ 4.6
	Education	10	50.6	67.2	72.8	$+$ 16.6	$+$ 5.6	$+$ 22.2

Appendix

Table 9: Mean strict accuracy (%) per model and change per edge (percentage points) within benchmark-native strata

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation
Apr 30, 2026Jon-Paul CacioliLLM Evaluation
Independent Researcher, Melbourne, Australia
API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces
Sep 8, 2026Jennifer Wang, Joachim Baumann, Daniel E. Ho +1LLM Evaluation Language Model Generation Evaluation
Stanford University Stanford, CA, USA
No Universal Signal Predicts Sample-Level LLM Regression under Version Updates
Aug 11, 2026Jia Sheng, Yiwei LuLanguage Model Error Detection LLM Reliability
University of Ottawa · Vector Institute

Apr 30, 2026J/K move · Enter open · S save

Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation

Jon-Paul Cacioli

Independent Researcher, Melbourne, Australia

LLM Evaluation

Open paper AskPDF arXiv

We adapted the Reliable Change Index (RCI; Jacobson and Truax, 1991) from clinical psychology to item-level LLM version comparison on 2,000 MMLU-Pro items (K=10 samples at T=0.7). Two within-family pairs were tested: Llama 3 to 3.1 (+1.6 points) and Qwen 2.5 to 3 (+2.8 points). On the full benchmark, most items showed no reliable change (79% and 72%). However, over half the items were floor/ceiling. Among analysable items, change was bidirectional with large effect sizes: 34% improved and 28% deteriorated for Llama; 47% improved and 39% deteriorated for Qwen (median |delta p| = 0.50 and 0.90). Churn was asymmetric by difficulty: low-accuracy items improved, high-accuracy items deteriorated. Domain-level decomposition revealed family-specific reversals: Llama lost physics while Qwen lost law. Greedy single-shot evaluation missed 42% of reliably changed items and falsely flagged 25% of unchanged items. The aggregate accuracy gain is the net residual of opposing item-level movements. We recommend reporting churn rate alongside aggregate accuracy.

Abstract

LLM Evaluation

Explore similar work

Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation

API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces

No Universal Signal Predicts Sample-Level LLM Regression under Version Updates