When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

Mubashara Akhtar; Anka Reuel; Prajna Soni; Sanchit Ahuja; Pawan Sasanka Ammanamanchi; Ruchit Rawal; Vilém Zouhar; Srishti Yadav

cs.AIFeb 18, 2026

When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

Authors: Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, Pawan Sasanka Ammanamanchi, Ruchit Rawal, Vilém Zouhar, Srishti Yadav, +29 more

Abstract

Artificial intelligence benchmarks are an important mechanism for measuring model progress and guiding deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find that nearly half of the our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test data. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches.

Explore similar work

Position: State-of-the-Art Claims Require State-of-the-Art Evidence
May 17, 2026YongKyung OhArtificial Intelligence Benchmarks State-Of-The-Art
Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation
Sep 10, 2026Koutian Wu, Junjie Zhou, Ergan Shang +6Artificial Intelligence Benchmarks Large Language Model Evaluation
AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems
May 24, 2026Michael Hardy, Anka Reuel, Lijin Zhang +6Artificial Intelligence Benchmarks Leaderboard

May 17, 2026 · cs.LGJ/K move · Enter open · S save

Position: State-of-the-Art Claims Require State-of-the-Art Evidence

YongKyung Oh

State-of-the-Art (SOTA) claims pervade Artificial Intelligence (AI) and Machine Learning (ML) research. These claims rest on benchmark evaluations, where models are ranked by aggregate scores across tasks. Public benchmarks or leaderboards are the most visible instance, but the same structure appears in paper tables throughout the literature. However, such minimal evidence often cannot support these strong claims. We identify a widespread claim-evidence gap in AI benchmarking. Claiming SOTA carries implicit assumptions beyond mean score superiority, suggesting that a model meaningfully outperforms alternatives across most tasks. However, a marginal improvement in the mean score merely indicates a top average rank rather than true superiority. Analyzing ten cross-domain benchmarks from public leaderboards, we found that in more than half of top-model comparisons, at least one commonly assumed property of superiority does not hold. These properties include meaningful effect size, consistency across tasks, or robustness to dataset removal. Instead, aggregate gains are frequently driven by outlier datasets. This fragility persists even in benchmarks with many tasks. We argue that claim language should reflect the strength of the underlying evidence. This requires no additional experiments, only honest reporting of what results actually show, enabling more precise and interpretable comparisons across models.

Artificial Intelligence Benchmarks State-Of-The-Art Position

Open paper Ask about it PDF

Abstract

Explore similar work

Position: State-of-the-Art Claims Require State-of-the-Art Evidence

Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems