Evaluating Model Capabilities After Benchmark Saturation

Morio Hamada; Jehyeok Yeon; Matteo Migliarini; Justin Olive; Magda Dubois

cs.AIOct 14, 2026

Evaluating Model Capabilities After Benchmark Saturation

Authors: Morio Hamada, Jehyeok Yeon, Matteo Migliarini, Justin Olive, Magda Dubois

Organizations: LASR Labs · ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center ·

Abstract

Rapid advances in AI capabilities are causing benchmarks to saturate. Once several models approach perfect scores, a benchmark loses its power to differentiate between them, and with it its utility for informing AI safety cases and frontier policies. Yet benchmark items produce far more information than the scores we typically extract from them: a full transcript of how each model solved the task. We investigate whether these transcripts can be used to assess model capabilities even after benchmark scores have saturated. We simulate saturation retrospectively using historical benchmark and model releases, and test whether protocols that access only correctly solved transcripts from saturated benchmarks can predict capabilities measured on held-out, unsaturated items. We evaluate representative contemporary approaches for extracting information from model transcripts: pairwise judging, single-transcript absolute scoring, and predictions from deterministic transcript features or embeddings. We find that saturated transcripts retain capability information beyond benchmark accuracy. Amongst our evaluated protocols, pairwise judging performs best, reducing prediction error by 5.3 times compared to a baseline using only saturated benchmark scores. This capability-relevant information is distributed across transcript sections, but concentrated in specific channels of information such as reasoning and tool-calls. However, the predictive power of protocols does not always generalise across distribution shifts -- for example by underestimating the capabilities of the first reasoning models introduced in early 2025. Saturated benchmarks thus retain capability-predicting signal in their task transcripts, but extending the lifetime of existing benchmarks requires methods designed to extract these signals robustly.

Figures & tables

Figure 1: Overview of our evaluation setting. As benchmark scores saturate, it becomes infeasible to distinguish the capabilities of models. Using earlier models as references, protocols predict target model capability on harder, unsaturated items, using as information how the models solve the tasks.

Figure 2: Overview of the prediction task and evaluation methodology. (A) Models are grouped into consecutive semesters; for each target period, the preceding semester serves as the reference set. (B) Items are split by whether the top-3 reference models answer correctly: easy items form the saturated set, of which only correctly solved transcripts

\mathcal{T}

are provided to a capability-extraction protocol

\mathcal{P}

to produce a prediction

\hat{\theta}

; the remaining items form the unsaturated set, on which an IRT model is fit to obtain the ground-truth capability

\theta^{*}

for both references and targets.

Domain	Benchmarks
Science	GPQA-Diamond ( Rein et al., 2024 ) , ARC-Challenge ( Clark et al., 2018 ) , HLE-Gold [Science] ( Zhai et al., 2026 )
Mathematics	MMLU-Pro [Maths] ( Wang et al., 2024 ) , AIME 2024 ( Jia, 2024 ) , AIME 2025, AIME 2026, APEX 2025, APEX Shortlist ( Dekoninck et al., 2026 ) , HLE-Gold [Maths]
Coding	AgentBench OS ( Liu et al., 2024 ) , SWE-Bench ( Jimenez et al., 2024 ) , Terminal-Bench 2.1 ( Merrill et al., 2026 )
Cybersecurity	CyBench ( Zhang et al., 2025 ) , InterCode ( Yang et al., 2023 )

Table 1: Evaluation domains and benchmarks. Benchmarks used in each capability domain.

Protocol	Signal extracted
Baseline protocol	–
Transcript features	Deterministic properties (e.g., transcript length and prose features)
Transcript embeddings	Text-embedding representation of a single transcript
Absolute scoring	Absolute scalar score given to a single transcript
Pairwise judging	Relative capability evidence from comparisons between two transcripts

Table 2: Prediction protocols. All non-baseline protocols extract transcript-level signals and calibrate them to target capability

\theta^{*}

using the reference models.

Figure 3: Pairwise judges extract capability-predicting signal from saturated transcripts. Prediction performance of each protocol for 16 target models (squared error: lower is better). Each point is one target model; circle and box: mean and 95% CI. **:

p<.01

, one-sample

t

-test of LLR against the baseline protocol; n.s. : not significant.

Figure 4: Effect of removing transcript sections on prediction error. Change in squared prediction error of pairwise judging when the first, middle, or last third of every transcript is removed, relative to the complete transcript ( dashed line ). Grey points show the baseline prediction. *:

p<.05

; **:

p<.01

Figure 5: Effect of removing transcript channels on prediction error. A. Share of transcript in each channel per domain. B. Change in prediction error of pairwise judging when one channel is removed from every transcript, relative to the complete transcript (dashed line). Each point is the mean over a domain’s target models and the box its 95% paired

t

-interval. *:

p<.05

, **:

p<.01

, paired

t

-test of squared error between channel-removed and full transcripts.

Figure 6: Prediction errors vary substantially across historical target periods. Prediction error for each protocol when calibrated on the preceding reference population and evaluated on the 8 models of each target period. Values above 10 are drawn in the hatched band.

Figure 7: Pairwise judges underestimate maths capabilities of early-2025 reasoning models. Left, Predicted vs true capability for the three non-reasoning targets, five reasoning targets, and reasoning targets judged with reasoning withheld. Diagonal : perfect prediction. Right, Prediction error per group (means

\pm

95%

t

-intervals). **:

p<.01

, two-sample

t

-test.

Figure 8: Pairwise judging generalises across target model capabilities. Prediction error as target capabilities exceed that of the strongest reference model. Targets are grouped into four equal-count bins. Error increases with capability distance for all protocols except pairwise judging.

Figure 9: Pairwise judge predictions generalise to closed-source targets. Change in squared error from each target’s baseline error (below zero: better than the baseline protocol) for six closed-source targets (coloured) and eight open-weight targets from the same semester.

Protocol	Pooled	Science	Maths	Coding	Cyber
Baseline protocol	1.37	0.65	1.32	2.14	1.39
Deterministic features	0.86	0.43	0.76	0.70*	1.57
Embedding	1.01	0.40**	0.66	1.69	1.29**
Absolute scoring	1.01	0.31	0.85	1.41	1.48
Pairwise judging	0.26 **	0.21	0.42	0.14 ***	0.26 **

Table 3: Prediction error per protocol and domain. MSE per domain. Lowest prediction errors per domain are bolded. //:

p<.05/.01/.001

, one-sample

t

-tests on LLR.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Quantification of squared error and log-likelihood ratio for an example target model. Prediction of one target model’s capability ( black : measured

\theta_{m}^{*}

) from reference models in the preceding semester ( grey diamonds ), plotted against release date. For each protocol, predictive uncertainty is represented by a Student-

t

distribution with

n-1

degrees of freedom, centred on the protocol’s prediction and scaled using leave-one-model-out prediction errors on the reference models ( grey : baseline protocol; blue : pairwise judging; orange : deterministic features). Pairwise judging and deterministic features miss

\theta_{m}^{*}

by similar amounts in opposite directions (squared error 0.12 and 0.14, respectively), but the narrower distribution for pairwise judging assigns greater density to

\theta_{m}^{*}

, resulting in a larger log-likelihood ratio relative to the baseline protocol (0.39 versus 0.19).

Period	Model	Science	Maths	Coding	Cyber
2023	gpt-4-0613	$-0.91$ $\pm 0.10$	$-1.85$ $\pm 0.21$	–	–
	mixtral-8x7b-instruct-v0.1	$-1.27$ $\pm 0.12$	$-2.75$ $\pm 0.28$	–	–
	gpt-3.5-turbo-0125	$-1.68$ $\pm 0.13$	$-2.52$ $\pm 0.25$	–	–
	nous-hermes-2-yi-34b	$-1.64$ $\pm 0.17$	$-3.07$ $\pm 0.31$	–	–
	qwen-72b-chat	$-2.03$ $\pm 0.18$	$-2.90$ $\pm 0.29$	–	–
	llama-2-70b-chat-hf	$-1.85$ $\pm 0.13$	$-3.18$ $\pm 0.32$	–	–

Appendix

Table 4: Models used in the retrospective evaluation. Models are grouped by evaluation period. Entries give the estimated global capability

\theta

\pm

the half-width of its 95% credible interval. Within each period and domain, the highest

\theta

is in bold and the second and third are underlined, with shading of matching strength. Closed-source models are shown in italics. A dash indicates that the model is not evaluated in that domain.

Figure 13: (Continued.) Coding (top) and Cyber (bottom).

Panel of Figure 14 — Figure 14: Item difficulty parameters from the global IRT fit. Science and Maths are shown in the top row; Coding and Cyber in the bottom row. For each domain, the upper panel shows the distribution of item difficulty $b_{i}$ by benchmark, with the number of items shown in brackets. The lower panels show discrimination $a_{i}$ against difficulty $b_{i}$ .

Figure 15: Model capabilities by release date. Estimated model capabilities in each domain against release date, based on the global IRT fit (see Appendix A.2 ), coloured by evaluation period. Error bars show posterior standard deviation.

Figure 16: Benchmark accuracy by evaluation period. Points show individual models and lines show period means.

Feature	Definition
Length	Log model-written transcript length
Reasoning length	Log visible-reasoning length
Visible reasoning	Indicator for non-empty visible reasoning
Personal pronouns	Rate of first-person pronouns
Hedging	Rate of hedging expressions
Backtracking	Rate of self-correction expressions

Appendix

Table 5: Deterministic transcript features.

Protocol	MSE [95% CI]	$\Delta$ SE	$p$	LLR	$p$
Baseline protocol	1.37 [0.88, 1.87]	–	–	–	–
Deterministic features	0.86 [0.52, 1.21]	$-0.51$	.064	$-0.26$ (0.30)	.404
Embedding	1.01 [0.61, 1.41]	$-0.36$	$<$ .001	0.04 (0.06)	.464
Absolute scoring	1.01 [0.67, 1.35]	$-0.36$	.002	$-0.21$ (0.16)	.200
Pairwise judge	0.26 [0.18, 0.34]	$-1.12$	$<$ .001	0.68 (0.21)	.005

Appendix

Table 6: Prediction error per protocol. (Accompaniment to Figure 3 .) MSE : Mean squared error over the 16 target models of late 2025 and early 2026 (averaged over four domains).

\Delta

SE: mean squared error minus the baseline’s, paired

t

-test (

n=16

target models). LLR : mean log likelihood ratio of the protocol’s prediction against the baseline protocol (standard error in parentheses).

p

: one-sample

t

-test on LLR against 0. We report statistics based on LLR in Section 4.1 , to penalise confidently incorrect predictions.

Domain	Target period	Baseline protocol	Deterministic features	Embedding	Absolute scoring	Pairwise judge
Science	Early 2024	1.65 [1.05, 2.27]	1.66 [1.07, 2.29]	1.65 [1.05, 2.25]	1.61 [1.00, 2.20]	0.11 [0.01, 0.24]
	Late 2024	0.54 [0.36, 0.74]	0.58 [0.37, 0.84]	0.54 [0.37, 0.74]	0.46 [0.32, 0.63]	0.54 [0.36, 0.75]
	Early 2025	1.16 [0.61, 1.72]	38.06 [4.01, 82.83]	0.77 [0.38, 1.20]	4.14 [1.52, 6.84]	0.65 [0.24, 1.08]
	Late 2025	0.85 [0.46, 1.23]	0.28 [0.14, 0.46]	0.45 [0.26, 0.66]	0.42 [0.24, 0.63]	0.17 [0.02, 0.37]
	Early 2026	0.45 [0.27, 0.64]	0.57 [0.18, 1.11]	0.35 [0.18, 0.54]	0.20 [0.06, 0.37]	0.25 [0.13, 0.38]
Maths	Early 2024	1.61 [0.98, 2.25]	1.97 [0.81, 3.13]	1.42 [0.78, 2.14]	2.41 [1.19, 3.76]	1.61 [0.97, 2.23]

Appendix

Table 7: Prediction error per protocol across target periods. Accompaniment to Figure 6 . Mean squared error over the 8 target models of each period, with 95% percentile-bootstrap intervals (2,000 resamples). Lowest error per row underlined. Cell colours show mean squared error on a logarithmic scale from 0.1 (blue) to 10+ (red).

Semester	Domain	Protocol	Best	2nd Best	3rd Best
2024-I	Science	Pairwise	gpt-4-0613	mixtral-8x7b-instruct-v0.1	nous-hermes-2-yi-34b
	Science	Absolute	nous-hermes-2-yi-34b	gpt-4-0613	mixtral-8x7b-instruct-v0.1
	Maths	Pairwise	gpt-3.5-turbo-0125	mixtral-8x7b-instruct-v0.1	gpt-4-0613
	Maths	Absolute	gpt-3.5-turbo-0125	gpt-4-0613	mixtral-8x7b-instruct-v0.1
2024-II	Science	Pairwise	gpt-4o-2024-05-13	gpt-4-turbo-2024-04-09	wizardlm-2-8x22b
	Science	Absolute	gpt-4o-2024-05-13	gpt-4-turbo-2024-04-09	wizardlm-2-8x22b

Appendix

Table 8: Candidate and picked judges. For each target semester and domain, the three references (models from the preceding semester) with the highest global capability

\theta

, which serve as candidate judges for pairwise judging and absolute scoring, ordered by reference leave-one-out error. The candidate with the lowest error on the reference set (“ Best ”) is the judge used for the task.

Figure 17: Pairwise judging is robust to the choice of judge. Squared prediction error for the 16 target models from 2025-II and 2026-I when using the strongest, second-strongest, or third-strongest reference model as judge. Small points show individual target models, averaged across domains; large circles show means and boxes show 95%

t

-intervals.

Figure 18: Protocol efficacy varies across domains. Posterior MSE ratio of each protocol relative to the baseline protocol, pooled across domains ( filled circles ) and separately by domain ( open markers ), estimated using the hierarchical Gamma GLM. Error bars show 95% credible intervals; the dashed line marks equal error to the baseline.

Protocol	Pooled	Science	Maths	Coding	Cybersecurity	$\sigma_{p}$
Deterministic features	1.00 [0.63, 1.68]	1.03 [0.64, 1.83]	0.98 [0.60, 1.62]	1.03 [0.62, 2.00]	0.98 [0.60, 1.64]	0.17 [0.01, 0.70]
Embedding	0.69 [0.52, 0.95]	0.70 [0.53, 0.93]	0.65 [0.47, 0.85]	0.68 [0.51, 0.90]	0.73 [0.56, 1.03]	0.13 [0.01, 0.57]
Absolute scoring	0.71 [0.50, 1.03]	0.65 [0.41, 0.92]	0.72 [0.51, 1.04]	0.68 [0.46, 0.96]	0.79 [0.56, 1.20]	0.18 [0.01, 0.68]
Pairwise judging	0.42 [0.20, 0.93]	0.53 [0.28, 1.17]	0.73 [0.37, 1.68]	0.21 [0.09, 0.55]	0.27 [0.12, 0.63]	0.62 [0.12, 1.22]

Appendix

Table 9: Hierarchical estimates of protocol performance by domain. Posterior median MSE ratio relative to the baseline protocol, with 95% credible intervals in brackets.

\sigma_{p}

shows the between-domain standard deviation.

Figure 19: Pairwise judges’ prediction performance beats other protocols reliably in Coding and Cyber . a, Ratio of pairwise judging relative to each other protocol, pooled and by domain; values below one favour pairwise judging. b, Between-domain standard deviation

\sigma_{p}

for each protocol on the log-MSE scale. Pairwise judging performance is also the most variable across domains. Error bars show 95% credible intervals.

Figure 20: Pairwise judging transfers asymmetrically across domains. Squared prediction error relative to the saturated-item baseline when transcripts from one source domain are used to predict capability in another target domain. Outlined markers indicate within-domain predictions using transcripts from the target domain itself. Error bars show 95% intervals; the dashed line marks equal error to the baseline.

Figure 21: Effect of the number of evaluated items. Squared prediction error of pairwise judging and absolute scoring (the judge selected by reference leave-one-model-out error, as in the main analysis) as the number of judged saturated items varies across periods and domains: median over 30 random item subsets, band 2.5–97.5 percentile; dashed line is baseline protocol error of each task.

Figure 22: Sensitivity to the reference population. Squared prediction error of pairwise judging as the reference set is progressively restricted to stronger or weaker reference models: median over the third of reference subsets keeping the strongest (top) or weakest (bottom) models, band 2.5–97.5 percentile; dashed line : the baseline protocol prediction for each task (all references).

Figure 23: Removal of self-judgements does not influence results. Prediction performance of each protocol for 16 target models (squared error: lower is better), without self-judgements for absolute scoring and pairwise judging. This did not influence our results (cf. Figure 3 ). Each point is one target model; circle and box: mean and 95% CI.

Figure 24: Single transcript sections carry part of the capability information. Change in prediction error of pairwise judging when only the first, middle, or last third of every transcript is shown, or the full transcript, relative to the baseline protocol ( dashed line ). Points show mean changes across target models and boxes show 95% paired

t

-intervals. *:

p<.05

; **:

p<.01

; ***:

p<.001

Figure 25: Capability signals across transcript positions for deterministic features, embeddings and absolute scoring. Change in squared prediction error for deterministic features ( top ), embeddings ( middle ) and absolute scoring ( bottom ) after removing the first, middle, or last third of every transcript, relative to the complete-transcript condition ( dashed line ). Points show mean changes across target models and boxes show 95% paired

t

-intervals. *:

p<.05

; ***:

p<.001

Figure 26: Capability signals across channels for deterministic features, embeddings and absolute scoring. Change in squared prediction error for deterministic features ( top ), embeddings ( middle ) and absolute scoring ( bottom ) after removing one transcript channel, relative to the complete-transcript condition ( dashed line ). Deterministic features and embeddings are refitted using the corresponding channel-removed reference transcripts; for absolute scoring, the judge re-rates the channel-removed transcripts and is recalibrated on the same references. Maths and Science contain no tool calls. Points show mean changes across target models and boxes show 95% paired

t

-intervals. *:

p<.05

; **:

p<.01

Figure 27: Tightening the saturation boundary. Squared prediction error of each protocol relative to the baseline as the number of models required to solve an item increases. Levels 3–8 require the

k

strongest reference models to solve the item; “all” requires every reference and target model in the period to solve it.

\theta^{*}

is refitted at every level, and level 3 corresponds to the main-text setting. Bands show bootstrap 95% intervals across the 16 target models from late 2025 and early 2026.

	Models that must solve an item
Target domain	3	4	5	6	7	8	all 16
Science ( $n$ : 200 $\to$ 140)	0.33	0.40	0.41	0.51	0.64	1.20	1.18
Maths ( $n$ : 200 $\to$ 102)	0.32	0.34	0.38	0.63	0.69	1.00	0.81
Coding ( $n$ : 159 $\to$ 52)	0.07	0.07	0.09	0.11	0.16	0.23	0.19
Cybersecurity ( $n$ : 149 $\to$ 94)	0.19	0.19	0.17	0.31	0.39	0.16	0.40
Pooled	0.19	0.20	0.22	0.32	0.39	0.65	0.49

Appendix

Table 10: Pairwise judging under stricter saturation boundaries. Squared prediction error relative to the corresponding baseline at each saturation level, with

\theta^{*}

refitted separately at every level. Late 2025 and early 2026 are pooled. The reported

n

values give the number of observed items summed across the two periods at level 3 and under the “all” condition.

Figure 28: Pairwise judging degrades with the distance between references and targets. In Science , squared error per target; circle and box: mean and 95%

t

-interval over the eight targets of a task. Left : early 2026 targets predicted from each earlier reference period. Right : references fixed at late 2024 (light) or early 2025 (dark), targets up to two periods older (negative) or newer (positive). Tasks one period ahead are those of the main text.

Figure 29: Only pairwise judging turns cost into accuracy. Prediction error against USD cost, pooled over every task of each domain and averaged over time periods. The judge is the reference with the lowest leave-one-out error. Dashed line : baseline protocol. Pairwise judging and absolute scoring are shown as curves over 10–100% of the judged items, with cost proportional to the items judged. Deterministic features and embeddings are single points.

Figure 30: Performance of the pairwise judge ensemble. Squared prediction error of the ensemble of the three candidate judges (mean of their predictions; coloured markers) and of the judge selected by leave-one-model-out error (grey dots), on the items all three judged; grey tick, the baseline protocol. Right panel shows the paired difference (ensemble minus single judge; negative = ensemble is better). Ensemble error is generally similar to the single judge’s, but substantially reduces the error of the selected judge in Maths 2024-I (1.61 to 0.58) and 2025-I (3.97 to 2.75).

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Feb 18, 2026Mubashara Akhtar, Anka Reuel, Prajna Soni +34LLM Evaluation Benchmark Design
ETH Zurich · ETH AI Center · Stanford University +23
Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles
Oct 18, 2026Kazem Faghih, Yize Cheng, Shoumik Saha +3LLM Evaluation LLM Reliability
Department of Computer Science, University of Maryland, College Park
Position: State-of-the-Art Claims Require State-of-the-Art Evidence
May 17, 2026YongKyung OhBenchmark Validity
Medical & Imaging Informatics (MII), University of California, Los Angeles (UCLA)

Feb 18, 2026J/K move · Enter open · S save

When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja+33

ETH Zurich · ETH AI Center · Stanford University+23

LLM Evaluation Benchmark Design

Open paper AskPDF arXiv

Artificial intelligence benchmarks are an important mechanism to measure model progress and guide deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find that nearly half of our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test data. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches.

Abstract

LLM Evaluation

Explore similar work

When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles

Position: State-of-the-Art Claims Require State-of-the-Art Evidence