From Human-Level AI Tales to AI Leveling Human Scales
Authors: Peter Romero, Fernando Martínez-Plumed, Zachary R. Tidler, Matthieu Téhénan, Sipeng Chen, Álvaro David Gómez Antón, Luning Sun, Manuel Cebrian, Lexin Zhou, Yael Moros Daval, Daniel Romero-Alvarado, Félix Martí Pérez, Kevin Wei, José Hernández-Orallo
Organizations: Valencian Research Institute of Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · The Psychometrics Centre, University of Cambridge, UK · Georgia Institute of Technology · University of Cambridge, Department of Computer Sciences and Technology · Carnegie Mellon University · Center for Automation and Robotics, Spanish National Research Council · Department of Computer Science, Princeton University · Harvard University · Leverhulme Centre for the Future of Intelligence, University of Cambridge, UK
Comparing AI models to "human level" is often misleading when benchmark scores are incommensurate or human baselines are drawn from a narrow population. To address this, we propose a framework that calibrates items against the 'world population' and report performance on a common, human-anchored scale. Concretely, we build on a set of multi-level scales for different capabilities where each level should represent a probability of success of the whole world population on a logarithmic scale with a base B. We calibrate each scale for each capability (reasoning, comprehension, knowledge, volume, etc.) by compiling publicly released human test data spanning education and reasoning benchmarks (PISA, TIMSS, ICAR, UKBioBank, and ReliabilityBench). The base B is estimated by extrapolating between samples with two demographic profiles using LLMs, with the hypothesis that they condense rich information about human populations. We evaluate the quality of different mappings using group slicing and post-stratification. The new techniques allow for the recalibration and standardization of scales relative to the whole-world population.
Figures & tables
Figure 1: Calibrated annotations of benchmarks can be used to generate profiles of AI systems on human-referenced scales (top). In this paper we calibrate 18 dimensions of capability and knowledge, going from level 0 (near-universal success) to level 5 ≈ 1-in- B5 people succeeding, with B being normalized according to the human distribution taken from several tests with human results (bottom). The calibration uses this source human sample and their demographics through an LLM to extrapolate to the whole world population to calibrate the scales.
International assessment of 15-year-olds in reading, mathematics, and science (2009); ability estimated via item response modeling with a focus on practical skills.
57 countries; 4,500–40,000 students per country.
Text-only items: 12 math, 32 reading, 20 science.
TIMSS
Grade 4/8 mathematics & science; released items from 2003 and 2011; multiple-choice and constructed responses spanning content and cognitive domains.
300 text-only questions across 30 countries; 27 prompt variants each; evaluation subset of 5,000 variants.
ICAR ( Condon & Revelle, 2014 ) .
Public-domain cognitive tests; we use Letter & Number Series (9) and Verbal Reasoning (16), all multiple-choice and purely textual.
Validation sample 97,000 (199 countries); additional cohorts: English 145,000, Chinese 240, German 106.
25 items used to compare item success across languages/contexts.
UK Biobank ( Sudlow et al., 2015 )
Fluid Intelligence (Verbal–Numerical Reasoning): 13 multiple-choice items under a 2-minute limit, probing verbal and numerical reasoning.
502,649 participants; completed by n=168,415; 20,000 repeated after 4 years.
13 text-only timed items as a human reference.
ReliabilityBench ( Zhou et al., 2024 )
LLM reliability benchmark with human difficulty judgments across five subdomains; provides human success rates; all items non-visual.
189 adults in US/UK (64% female; ages 19–78); no individual-level covariates.
401 questions; 27 prompt variants each; used for calibration only.
Table 1: Summary of datasets and the text-only subsets used in this work.
Model
N
MAE
RMSE
rPearson
rSpearman
gpt-5-chat
2993
0.030
0.044
0.976
0.968
llama-4
3121
0.033
0.052
0.971
0.963
gpt-4.1
3123
0.040
0.058
0.958
0.944
deepseek-v3.1
3124
0.043
0.085
0.922
0.914
grok-3
3115
0.043
0.068
0.939
0.920
Table 2: Validation results on ICAR benchmark (lower MAE/RMSE is better, higher r is better).
#Groups
N
MAE
RMSE
rPearson
rSpearman
7
33.143
0.068
0.086
0.948
0.894
Table 3: ICAR ground truth / baseline — anonymized summary across focal groups.
Figure 2: Human-calibrated bases B for groups of dimensions. For each plot we only use the examples for which any of the dimensions of that group is dominant, and using harmonic mean when the group contains more than one dimension. The x -axis shows the level, as annotated following the ADeLe rubrics, and the y -axis shows the corresponding levels as coming from the LLM estimate. By fitting a linear function using the means of the levels ( ⋆ ) we can derive the human-calibrated base for each group. We see that many are smaller than 10, as assumed in the original scales, suggesting that this calibration should be applied when comparing values for different dimensions. Mind Modeling & Social Cognition has a negative slope probably because it has the smallest number of examples for a good estimate at levels 3 and 4.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Section
Content
Introduction overall
We have PISA results from 57 countries (30 OECD + 27 partner countries).
Demographics
Students aged 15 years 3 months to 16 years 2 months, attending at least Grade 7 or equivalent. About 400,000--450,000 students were assessed, representing ∼ 20 million 15-year-olds globally (stratified sampling).
Item introduction
Consider the following question that was asked to all these students:
Item
Question: The picture shows the footprints of a man walking. The pacelength P is the distance between the rear of two consecutive footprints. For men, the formula n/P=140 gives an approximate relationship between n and P , where: n = number of steps per minute, and P = pacelength in metres. Bernard knows his pacelength is 0.80 metres. The formula applies to Bernard’s walking. Task: Calculate Bernard’s walking speed in metres per minute and in kilometres per hour. Show your working out.
Percentage success rate
This question had a 19% success rate in the PISA sample.
Reference group
—
Appendix
Table 4: Example PISA item and setup for extrapolation task.
Dimension (Broad)
Dimension (Specific)
Demand description
AS
Attention and Scan
AS
Attention and Scan
Focus on or locate specific elements within a given stream of information or environment in the whole process of solving a task.
CE
Comprehension and Expression
CEc
Verbal Comprehension
Understand text, stories or the semantic content of other representations of ideas in different formats or modalities.
CEe
Verbal Expression
Generate and articulate ideas, stories, or semantic content in different formats or modalities.
CL
Conceptualisation, Learning and Abstraction
CL
Conceptualisation, Learning and Abstraction
Build new concepts, engage in inductive and analogical reasoning, map relationships between domains, and generate abstractions from concrete examples.
MC
Metacognition and Critical Thinking
MCr
Identifying Relevant Information
Recognise what information helps solve the task or does not, and how this recognition process unfolds as they work toward the solution.
MCt
Critical Thinking Processes
Monitor or regulate multiple thought processes to answer the question effectively, ranging from simple recall to high-level critical thinking.
Appendix
Table 5: Cognitive demand dimensions in the ADeLe framework. Adapted from ( Zhou et al., 2025 ) .
Figure 3: Mapping of dominant ADeLe dimension from extrapolated calibrations (what model thinks about how the world would perform) against real world outcomes of test takers from ICAR Logical Reasoning
Figure 4: Mapping of dominant ADeLe dimension from extrapolated calibrations (what model thinks about how the world would perform) against real world outcomes of test takers from ICAR Verbal Reasoning
Model
N
MAE
RMSE
rPearson
rSpearman
gpt-5-chat
2993
0.030
0.044
0.976
0.968
llama-4
3121
0.033
0.052
0.971
0.963
gpt-4.1
3123
0.040
0.058
0.958
0.944
deepseek-v3.1
3124
0.043
0.085
0.922
0.914
grok-3
3115
0.043
0.068
0.939
0.920
Appendix
Table 6: Validation results on ICAR benchmark (lower MAE/RMSE is better, higher r is better).
Model
N
MAE
RMSE
rPearson
rSpearman
deepseek-v3.1
674
0.166
0.180
0.933
0.912
llama-4
669
0.168
0.184
0.920
0.893
gpt-5-chat
673
0.173
0.186
0.938
0.933
gpt-4.1
672
0.264
0.282
0.857
0.829
grok-3
671
0.336
0.355
0.802
0.780
Appendix
Table 7: Calibration results on ICAR benchmark (lower MAE/RMSE is better, higher r is better).
#Groups
N
MAE
RMSE
rPearson
rSpearman
7
33.143
0.068
0.086
0.948
0.894
Appendix
Table 8: ICAR ground truth / baseline — anonymized summary across focal groups.
Model
N
MAE
RMSE
rPearson
rSpearman
gpt-5-chat
18,976
0.087
0.111
0.825
0.783
gpt-4.1
18,962
0.111
0.141
0.792
0.735
deepseek-v3.1
18,928
0.111
0.147
0.763
0.714
llama-4
18,946
0.112
0.147
0.781
0.730
grok-3
18,910
0.112
0.140
0.818
0.772
Appendix
Table 9: Validation results on PISA benchmark (lower MAE/RMSE is better, higher r is better).
Model
N
MAE
RMSE
rPearson
rSpearman
deepseek-v3.1
1,719
0.132
0.144
0.938
0.940
llama-4
1,721
0.246
0.269
0.770
0.761
gpt-5-chat
1,721
0.257
0.268
0.902
0.909
grok-3
1,720
0.323
0.340
0.780
0.784
gpt-4.1
1,719
0.333
0.346
0.845
0.861
Appendix
Table 10: Calibration results on PISA benchmark (lower MAE/RMSE is better, higher r is better).
Figure 5: Mapping of dominant ADeLe dimension from extrapolated calibrations (what model thinks about how the world would perform) against real world outcomes of test takers from PISA results.
Model
N
MAE
RMSE
rPearson
rSpearman
deepseek-v3.1
351
0.184
0.210
0.908
0.899
llama-4
351
0.250
0.289
0.844
0.881
gpt-5-chat
351
0.265
0.312
0.781
0.779
gpt-4.1
351
0.366
0.402
0.766
0.760
grok-3
351
0.412
0.467
0.596
0.589
Appendix
Table 11: Calibration results on UK Biobank benchmark (lower MAE/RMSE is better, higher r is better).
Figure 6: Mapping of dominant ADeLe dimension from extrapolated calibrations (what model thinks about how the world would perform) against real world outcomes of test takers from UK BioBank.
Model
N
MAE
RMSE
rPearson
rSpearman
gpt-4.1
10,867
0.103
0.231
0.771
0.846
deepseek-v3.1
8,125
0.274
0.414
0.281
0.318
gpt-5-chat
8,129
0.278
0.410
0.228
0.257
grok-3
8,131
0.278
0.417
0.218
0.257
llama-4
8,130
0.279
0.412
0.245
0.275
Appendix
Table 12: Calibration results on ReliabilityBench (lower MAE/RMSE is better, higher r is better).
Figure 7: Mapping of dominant ADeLe dimension from extrapolated calibrations (what model thinks about how the world would perform) against real world outcomes of test takers from ReliabilityBench.
Model
N
MAE
RMSE
rPearson
rSpearman
deepseek-v3.1
4,689
0.121
0.149
0.706
0.663
gpt-5-chat
4,694
0.124
0.159
0.637
0.576
grok-3
4,688
0.154
0.209
0.529
0.500
gpt-4.1
4,698
0.154
0.205
0.554
0.511
llama-4
4,673
0.164
0.217
0.550
0.512
Appendix
Table 13: Validation results on TIMSS benchmark (lower MAE/RMSE is better, higher r is better).
Model
N
MAE
RMSE
rPearson
rSpearman
deepseek-v3.1
57
0.088
0.110
0.871
0.831
gpt-5-chat
57
0.432
0.482
0.435
0.475
grok-3
57
0.435
0.486
0.073
0.142
llama-4
57
0.445
0.494
0.441
0.661
gpt-4.1
57
0.445
0.496
0.172
0.271
Appendix
Table 14: Calibration results on TIMSS benchmark (lower MAE/RMSE is better, higher r is better).
Figure 8: Mapping of dominant ADeLe dimension from extrapolated calibrations (what model thinks about how the world would perform) against real world outcomes of test takers from TIMSS.
#Countries
N
MAE
RMSE
rPearson
rSpearman
87
55.22
0.129
0.149
0.828
0.790
Appendix
Table 15: TIMSS ground truth / baseline - anonymized summary across all countries.
Figure 9: Cluster Analysis of TIMSS outcomes by country. The “High Alignment” cluster (Green) includes Anglosphere nations where models successfully rank item difficulty ( r≈0.59 ), while the “Low Alignment” cluster (Red) includes distinct curricula (e.g., Singapore) where alignment collapses ( r≈0.02 ).
Dimension
Slope ( m )
Int. ( β0 )
Base ( B )
R2
Scaling
Volume
1.50
-0.55
31.95
0.65
High
Attention & Scan
1.24
0.15
17.19
0.86
High
Metacognition
0.83
0.50
6.70
0.64
Stand.
Knowledge
0.71
0.80
5.08
0.77
Stand.
Conceptualisation
0.50
0.27
3.17
0.98
Stand.
Atypicality
0.48
2.55
2.99
0.12
Stand.
Appendix
Table 16: Calibrated scaling parameters derived from means-based fit. A high Base ( B ) implies difficulty grows exponentially faster than human ratings. The Intercept ( β0 ) reflects intrinsic entry cost. R2 indicates the goodness of fit.
Figure 10: The Calibration Process. The progression from raw theoretical annotations to empirically aligned scales. This transformation proves that cognitive dimensions follow consistent scaling laws, but with distinct bases ranging from B≈1 to B≈32 .
AI benchmarks increasingly leverage item-level statistical models, particularly item response theory (IRT), to estimate capabilities, rank systems, select informative examples, and diagnose benchmark quality. However, AI benchmark data often departs from the data regimes encountered in human testing, for which standard IRT estimation tools were originally developed: these benchmarks typically involve fewer evaluated models, far more items, and capability distributions that may be skewed, clustered, or multimodal. We examine how these regime mismatches challenge the reliability of IRT modeling for AI evaluation. Using item parameters and capability distributions derived from six widely used LLM benchmarks, we simulate response matrices under three common IRT models and compare four estimators used in recent benchmark studies: marginal maximum likelihood, Markov chain Monte Carlo, variational inference, and a neural pseudo-Siamese estimator. Across 18,000 simulation runs, we systematically evaluate computational feasibility, scalability, and the reliability of IRT inferences about model rankings, predicted performance, and item characteristics. Results show that classical estimators can become computationally infeasible for large benchmarks, whereas scalable estimators can produce unreliable item-level and ranking inferences with small or non-normally distributed model sets. This study identifies when latent trait models reliably support or risk distorting AI benchmarking claims, and what sample sizes and diagnostics are needed for trustworthy use.
Han Jiang, Sunbeom Kwon, Jinwen Luo +2
Johns Hopkins University · University of Illinois Urbana-Champaign · University of California, Los Angeles
The rapid release of both language models and benchmarks makes it increasingly costly to evaluate every model on every dataset. In practice, models are often evaluated on different samples, making scores difficult to compare across studies. To address this, we propose a framework based on multidimensional Item Response Theory (IRT) that uses anchor items to calibrate new benchmarks to the evaluation suite while holding previously calibrated item parameters fixed. Our approach supports a realistic evaluation setting in which datasets are introduced over time and models are evaluated only on the datasets available at the time of evaluation, while a fixed anchor set for each dataset is used so that results from different evaluation periods can be compared directly. In large-scale experiments on more than 400 models, our framework predicts full-evaluation performance within 2-3 percentage points using only 100 anchor questions per dataset, with Spearman ρ≥0.9 for ranking preservation preservation. This shows that benchmark suites can grow over time while preserving score comparability, since adding a new dataset requires running existing models only on that dataset's anchors. Code and data are available at: https://eliyahabba.github.io/growing-pains/
Eliya Habba, Itay Itzhak, Asaf Yehudai +5
The Hebrew University of Jerusalem · Technion · IBM Research +2
Artificial intelligence benchmarks are an important mechanism to measure model progress and guide deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find that nearly half of our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test data. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches.
Mubashara Akhtar, Anka Reuel, Prajna Soni +34
ETH Zurich · ETH AI Center · Stanford University +23