From Human-Level AI Tales to AI Leveling Human Scales

Peter Romero; Fernando Martínez-Plumed; Zachary R. Tidler; Matthieu Téhénan; Sipeng Chen; Álvaro David Gómez Antón; Luning Sun; Manuel Cebrian

cs.LGFeb 21, 2026

From Human-Level AI Tales to AI Leveling Human Scales

Authors: Peter Romero, Fernando Martínez-Plumed, Zachary R. Tidler, Matthieu Téhénan, Sipeng Chen, Álvaro David Gómez Antón, Luning Sun, Manuel Cebrian,

Organizations: Valencian Research Institute of Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · The Psychometrics Centre, University of Cambridge, UK · Georgia Institute of Technology · University of Cambridge, Department of Computer Sciences and Technology ·

Abstract

Comparing AI models to "human level" is often misleading when benchmark scores are incommensurate or human baselines are drawn from a narrow population. To address this, we propose a framework that calibrates items against the 'world population' and report performance on a common, human-anchored scale. Concretely, we build on a set of multi-level scales for different capabilities where each level should represent a probability of success of the whole world population on a logarithmic scale with a base $B$ . We calibrate each scale for each capability (reasoning, comprehension, knowledge, volume, etc.) by compiling publicly released human test data spanning education and reasoning benchmarks (PISA, TIMSS, ICAR, UKBioBank, and ReliabilityBench). The base $B$ is estimated by extrapolating between samples with two demographic profiles using LLMs, with the hypothesis that they condense rich information about human populations. We evaluate the quality of different mappings using group slicing and post-stratification. The new techniques allow for the recalibration and standardization of scales relative to the whole-world population.

Figures & tables

Figure 1: Calibrated annotations of benchmarks can be used to generate profiles of AI systems on human-referenced scales (top). In this paper we calibrate 18 dimensions of capability and knowledge, going from level 0 (near-universal success) to level 5

\approx

1-in-

B^{5}

people succeeding, with

B

being normalized according to the human distribution taken from several tests with human results (bottom). The calibration uses this source human sample and their demographics through an LLM to extrapolate to the whole world population to calibrate the scales.

Dataset	Description	Scale / coverage	This work (subset)
PISA ( Lundahl & Serder, 2020 ; Breakspear, 2014 )	International assessment of 15-year-olds in reading, mathematics, and science (2009); ability estimated via item response modeling with a focus on practical skills.	57 countries; 4,500–40,000 students per country.	Text-only items: 12 math, 32 reading, 20 science.
TIMSS	Grade 4/8 mathematics & science; released items from 2003 and 2011; multiple-choice and constructed responses spanning content and cognitive domains.	2003: G8=48 countries, G4=26 c.; 2011: G4=57 c., G8=56 c. ( $>$ 360,000 students)	300 text-only questions across 30 countries; 27 prompt variants each; evaluation subset of 5,000 variants.
ICAR ( Condon & Revelle, 2014 ) .	Public-domain cognitive tests; we use Letter & Number Series (9) and Verbal Reasoning (16), all multiple-choice and purely textual.	Validation sample 97,000 (199 countries); additional cohorts: English 145,000, Chinese 240, German 106.	25 items used to compare item success across languages/contexts.
UK Biobank ( Sudlow et al., 2015 )	Fluid Intelligence (Verbal–Numerical Reasoning): 13 multiple-choice items under a 2-minute limit, probing verbal and numerical reasoning.	502,649 participants; completed by n=168,415; 20,000 repeated after 4 years.	13 text-only timed items as a human reference.
ReliabilityBench ( Zhou et al., 2024 )	LLM reliability benchmark with human difficulty judgments across five subdomains; provides human success rates; all items non-visual.	189 adults in US/UK (64% female; ages 19–78); no individual-level covariates.	401 questions; 27 prompt variants each; used for calibration only.

Table 1: Summary of datasets and the text-only subsets used in this work.

Model	$\mathbf{N}$	$\mathbf{MAE}$	$\mathbf{RMSE}$	$\mathbf{r}_{Pearson}$	$\mathbf{r}_{Spearman}$
gpt-5-chat	2993	0.030	0.044	0.976	0.968
llama-4	3121	0.033	0.052	0.971	0.963
gpt-4.1	3123	0.040	0.058	0.958	0.944
deepseek-v3.1	3124	0.043	0.085	0.922	0.914
grok-3	3115	0.043	0.068	0.939	0.920

Table 2: Validation results on ICAR benchmark (lower MAE/RMSE is better, higher

r

is better).

#Groups	$\mathbf{\overline{{N}}}$	$\mathbf{\overline{MAE}}$	$\mathbf{\overline{RMSE}}$	$\mathbf{\overline{r}}_{Pearson}$	$\mathbf{\overline{r}}_{Spearman}$
7	33.143	0.068	0.086	0.948	0.894

Table 3: ICAR ground truth / baseline — anonymized summary across focal groups.

Figure 2: Human-calibrated bases

B

for groups of dimensions. For each plot we only use the examples for which any of the dimensions of that group is dominant, and using harmonic mean when the group contains more than one dimension. The

x

-axis shows the level, as annotated following the ADeLe rubrics, and the

y

-axis shows the corresponding levels as coming from the LLM estimate. By fitting a linear function using the means of the levels (

\star

) we can derive the human-calibrated base for each group. We see that many are smaller than 10, as assumed in the original scales, suggesting that this calibration should be applied when comparing values for different dimensions. Mind Modeling & Social Cognition has a negative slope probably because it has the smallest number of examples for a good estimate at levels 3 and 4.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Section	Content
Introduction overall	We have PISA results from 57 countries (30 OECD + 27 partner countries).
Demographics	Students aged 15 years 3 months to 16 years 2 months, attending at least Grade 7 or equivalent. About 400,000--450,000 students were assessed, representing $\sim$ 20 million 15-year-olds globally (stratified sampling).
Item introduction	Consider the following question that was asked to all these students:
Item	Question: The picture shows the footprints of a man walking. The pacelength $P$ is the distance between the rear of two consecutive footprints. For men, the formula $n/P=140$ gives an approximate relationship between $n$ and $P$ , where: $n$ = number of steps per minute, and $P$ = pacelength in metres. Bernard knows his pacelength is 0.80 metres. The formula applies to Bernard’s walking. Task: Calculate Bernard’s walking speed in metres per minute and in kilometres per hour. Show your working out.
Percentage success rate	This question had a 19% success rate in the PISA sample.
Reference group	—

Appendix

Table 4: Example PISA item and setup for extrapolation task.

	Dimension (Broad)		Dimension (Specific)	Demand description
AS	Attention and Scan	AS	Attention and Scan	Focus on or locate specific elements within a given stream of information or environment in the whole process of solving a task.
CE	Comprehension and Expression	CEc	Verbal Comprehension	Understand text, stories or the semantic content of other representations of ideas in different formats or modalities.
		CEe	Verbal Expression	Generate and articulate ideas, stories, or semantic content in different formats or modalities.
CL	Conceptualisation, Learning and Abstraction	CL	Conceptualisation, Learning and Abstraction	Build new concepts, engage in inductive and analogical reasoning, map relationships between domains, and generate abstractions from concrete examples.
MC	Metacognition and Critical Thinking	MCr	Identifying Relevant Information	Recognise what information helps solve the task or does not, and how this recognition process unfolds as they work toward the solution.
		MCt	Critical Thinking Processes	Monitor or regulate multiple thought processes to answer the question effectively, ranging from simple recall to high-level critical thinking.

Appendix

Table 5: Cognitive demand dimensions in the ADeLe framework. Adapted from ( Zhou et al., 2025 ) .

Figure 3: Mapping of dominant ADeLe dimension from extrapolated calibrations (what model thinks about how the world would perform) against real world outcomes of test takers from ICAR Logical Reasoning

Figure 4: Mapping of dominant ADeLe dimension from extrapolated calibrations (what model thinks about how the world would perform) against real world outcomes of test takers from ICAR Verbal Reasoning

Model	$\mathbf{N}$	$\mathbf{MAE}$	$\mathbf{RMSE}$	$\mathbf{r}_{Pearson}$	$\mathbf{r}_{Spearman}$
gpt-5-chat	2993	0.030	0.044	0.976	0.968
llama-4	3121	0.033	0.052	0.971	0.963
gpt-4.1	3123	0.040	0.058	0.958	0.944
deepseek-v3.1	3124	0.043	0.085	0.922	0.914
grok-3	3115	0.043	0.068	0.939	0.920

Appendix

Table 6: Validation results on ICAR benchmark (lower MAE/RMSE is better, higher

r

is better).

Model	$\mathbf{N}$	$\mathbf{MAE}$	$\mathbf{RMSE}$	$\mathbf{r}_{Pearson}$	$\mathbf{r}_{Spearman}$
deepseek-v3.1	674	0.166	0.180	0.933	0.912
llama-4	669	0.168	0.184	0.920	0.893
gpt-5-chat	673	0.173	0.186	0.938	0.933
gpt-4.1	672	0.264	0.282	0.857	0.829
grok-3	671	0.336	0.355	0.802	0.780

Appendix

Table 7: Calibration results on ICAR benchmark (lower MAE/RMSE is better, higher

r

is better).

#Groups	$\mathbf{\overline{{N}}}$	$\mathbf{\overline{MAE}}$	$\mathbf{\overline{RMSE}}$	$\mathbf{\overline{r}}_{Pearson}$	$\mathbf{\overline{r}}_{Spearman}$
7	33.143	0.068	0.086	0.948	0.894

Appendix

Table 8: ICAR ground truth / baseline — anonymized summary across focal groups.

Model	$\mathbf{N}$	$\mathbf{MAE}$	$\mathbf{RMSE}$	$\mathbf{r}_{Pearson}$	$\mathbf{r}_{Spearman}$
gpt-5-chat	18,976	0.087	0.111	0.825	0.783
gpt-4.1	18,962	0.111	0.141	0.792	0.735
deepseek-v3.1	18,928	0.111	0.147	0.763	0.714
llama-4	18,946	0.112	0.147	0.781	0.730
grok-3	18,910	0.112	0.140	0.818	0.772

Appendix

Table 9: Validation results on PISA benchmark (lower MAE/RMSE is better, higher

r

is better).

Model	$\mathbf{N}$	$\mathbf{MAE}$	$\mathbf{RMSE}$	$\mathbf{r}_{Pearson}$	$\mathbf{r}_{Spearman}$
deepseek-v3.1	1,719	0.132	0.144	0.938	0.940
llama-4	1,721	0.246	0.269	0.770	0.761
gpt-5-chat	1,721	0.257	0.268	0.902	0.909
grok-3	1,720	0.323	0.340	0.780	0.784
gpt-4.1	1,719	0.333	0.346	0.845	0.861

Appendix

Table 10: Calibration results on PISA benchmark (lower MAE/RMSE is better, higher

r

is better).

Figure 5: Mapping of dominant ADeLe dimension from extrapolated calibrations (what model thinks about how the world would perform) against real world outcomes of test takers from PISA results.

Model	$\mathbf{N}$	$\mathbf{MAE}$	$\mathbf{RMSE}$	$\mathbf{r}_{Pearson}$	$\mathbf{r}_{Spearman}$
deepseek-v3.1	351	0.184	0.210	0.908	0.899
llama-4	351	0.250	0.289	0.844	0.881
gpt-5-chat	351	0.265	0.312	0.781	0.779
gpt-4.1	351	0.366	0.402	0.766	0.760
grok-3	351	0.412	0.467	0.596	0.589

Appendix

Table 11: Calibration results on UK Biobank benchmark (lower MAE/RMSE is better, higher

r

is better).

Figure 6: Mapping of dominant ADeLe dimension from extrapolated calibrations (what model thinks about how the world would perform) against real world outcomes of test takers from UK BioBank.

Model	$\mathbf{N}$	$\mathbf{MAE}$	$\mathbf{RMSE}$	$\mathbf{r}_{Pearson}$	$\mathbf{r}_{Spearman}$
gpt-4.1	10,867	0.103	0.231	0.771	0.846
deepseek-v3.1	8,125	0.274	0.414	0.281	0.318
gpt-5-chat	8,129	0.278	0.410	0.228	0.257
grok-3	8,131	0.278	0.417	0.218	0.257
llama-4	8,130	0.279	0.412	0.245	0.275

Appendix

Table 12: Calibration results on ReliabilityBench (lower MAE/RMSE is better, higher

r

is better).

Figure 7: Mapping of dominant ADeLe dimension from extrapolated calibrations (what model thinks about how the world would perform) against real world outcomes of test takers from ReliabilityBench.

Model	$\mathbf{N}$	$\mathbf{MAE}$	$\mathbf{RMSE}$	$\mathbf{r}_{Pearson}$	$\mathbf{r}_{Spearman}$
deepseek-v3.1	4,689	0.121	0.149	0.706	0.663
gpt-5-chat	4,694	0.124	0.159	0.637	0.576
grok-3	4,688	0.154	0.209	0.529	0.500
gpt-4.1	4,698	0.154	0.205	0.554	0.511
llama-4	4,673	0.164	0.217	0.550	0.512

Appendix

Table 13: Validation results on TIMSS benchmark (lower MAE/RMSE is better, higher

r

is better).

Model	$\mathbf{N}$	$\mathbf{MAE}$	$\mathbf{RMSE}$	$\mathbf{r}_{Pearson}$	$\mathbf{r}_{Spearman}$
deepseek-v3.1	57	0.088	0.110	0.871	0.831
gpt-5-chat	57	0.432	0.482	0.435	0.475
grok-3	57	0.435	0.486	0.073	0.142
llama-4	57	0.445	0.494	0.441	0.661
gpt-4.1	57	0.445	0.496	0.172	0.271

Appendix

Table 14: Calibration results on TIMSS benchmark (lower MAE/RMSE is better, higher

r

is better).

Figure 8: Mapping of dominant ADeLe dimension from extrapolated calibrations (what model thinks about how the world would perform) against real world outcomes of test takers from TIMSS.

#Countries	$\mathbf{\overline{{N}}}$	$\mathbf{\overline{MAE}}$	$\mathbf{\overline{RMSE}}$	$\mathbf{\overline{r}}_{Pearson}$	$\mathbf{\overline{r}}_{Spearman}$
87	55.22	0.129	0.149	0.828	0.790

Appendix

Table 15: TIMSS ground truth / baseline - anonymized summary across all countries.

Figure 9: Cluster Analysis of TIMSS outcomes by country. The “High Alignment” cluster (Green) includes Anglosphere nations where models successfully rank item difficulty ( $r\approx 0.59$ ), while the “Low Alignment” cluster (Red) includes distinct curricula (e.g., Singapore) where alignment collapses ( $r\approx 0.02$ ).

Dimension	Slope ( $m$ )	Int. ( $\beta_{0}$ )	Base ( $B$ )	$\mathbf{R^{2}}$	Scaling
Volume	1.50	-0.55	31.95	0.65	High
Attention & Scan	1.24	0.15	17.19	0.86	High
Metacognition	0.83	0.50	6.70	0.64	Stand.
Knowledge	0.71	0.80	5.08	0.77	Stand.
Conceptualisation	0.50	0.27	3.17	0.98	Stand.
Atypicality	0.48	2.55	2.99	0.12	Stand.

Appendix

Table 16: Calibrated scaling parameters derived from means-based fit. A high Base (

B

) implies difficulty grows exponentially faster than human ratings. The Intercept (

\beta_{0}

) reflects intrinsic entry cost.

R^{2}

indicates the goodness of fit.

Panel of Figure 10 — Figure 10: The Calibration Process. The progression from raw theoretical annotations to empirically aligned scales. This transformation proves that cognitive dimensions follow consistent scaling laws, but with distinct bases ranging from $B\approx 1$ to $B\approx 32$ .

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Can We Trust Item Response Theory for AI Evaluation?
Jul 16, 2026Han Jiang, Sunbeom Kwon, Jinwen Luo +2LLM Evaluation Item Response Theory
Johns Hopkins University · University of Illinois Urbana-Champaign · University of California, Los Angeles
Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration
Apr 14, 2026Eliya Habba, Itay Itzhak, Asaf Yehudai +5LLM Evaluation Benchmark Design
The Hebrew University of Jerusalem · Technion · IBM Research +2
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Feb 18, 2026Mubashara Akhtar, Anka Reuel, Prajna Soni +34LLM Evaluation Benchmark Design
ETH Zurich · ETH AI Center · Stanford University +23

Jul 16, 2026J/K move · Enter open · S save

Can We Trust Item Response Theory for AI Evaluation?

Han Jiang, Sunbeom Kwon, Jinwen Luo, Ziang Xiao+1

Johns Hopkins University · University of Illinois Urbana-Champaign · University of California, Los Angeles

LLM Evaluation Item Response Theory

Open paper AskPDF arXiv

AI benchmarks increasingly leverage item-level statistical models, particularly item response theory (IRT), to estimate capabilities, rank systems, select informative examples, and diagnose benchmark quality. However, AI benchmark data often departs from the data regimes encountered in human testing, for which standard IRT estimation tools were originally developed: these benchmarks typically involve fewer evaluated models, far more items, and capability distributions that may be skewed, clustered, or multimodal. We examine how these regime mismatches challenge the reliability of IRT modeling for AI evaluation. Using item parameters and capability distributions derived from six widely used LLM benchmarks, we simulate response matrices under three common IRT models and compare four estimators used in recent benchmark studies: marginal maximum likelihood, Markov chain Monte Carlo, variational inference, and a neural pseudo-Siamese estimator. Across 18,000 simulation runs, we systematically evaluate computational feasibility, scalability, and the reliability of IRT inferences about model rankings, predicted performance, and item characteristics. Results show that classical estimators can become computationally infeasible for large benchmarks, whereas scalable estimators can produce unreliable item-level and ranking inferences with small or non-normally distributed model sets. This study identifies when latent trait models reliably support or risk distorting AI benchmarking claims, and what sample sizes and diagnostics are needed for trustworthy use.

Abstract

LLM Evaluation

Explore similar work

Can We Trust Item Response Theory for AI Evaluation?

Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration

When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation