The LLM community often reports benchmark results as if they are synonymous with general model capabilities. However, benchmarks can have problems that distort performance, like test set contamination and annotator error. How can we know that a benchmark is a reliable indicator of some capability that we want to measure? This question concerns the construct validity of LLM benchmarks, and it requires separating benchmark results from capabilities when we model and predict LLM performance. Both social scientists and computer scientists propose formal models - latent factor models and scaling laws - for identifying the capabilities underlying benchmark scores. However, neither technique is satisfactory for construct validity. Latent factor models ignore scaling laws, and as a result, the capabilities they extract often proxy model size. Scaling laws ignore measurement error, and as a result, the capabilities they extract are both uninterpretable and overfit to the observed benchmarks. This thesis presents the structured capabilities model, the first model to extract interpretable and generalisable capabilities from a large collection of LLM benchmark results. I fit this model and its two alternatives on a large sample of results from the OpenLLM Leaderboard. Structured capabilities outperform latent factor models on parsimonious fit indices, and exhibit better out-of-distribution benchmark prediction than scaling laws. These improvements are possible because neither existing approach separates model scale from capabilities in the appropriate way. Model scale should inform capabilities, as in scaling laws, and these capabilities should inform observed results up to measurement error, as in latent factor models. In combining these two insights, structured capabilities demonstrate better explanatory and predictive power for quantifying construct validity in LLM evaluations.
Figures & tables
Figure 1: High-level picture of my contribution . My proposed model, the structured capabilities model, addresses problems in both latent factor and observational scaling law approaches. These existing approaches do not sufficiently separate model scale from capability estimates, which harms both their explanatory and their predictive power.
Figure 2: The scale of language model training, 2010 to 2025 . The dashed black line shows the introduction of the Transformer in 2017. The solid red lines show trends for training compute before the Transformer (order of magnitude increase every 5 years) and after the Transformer (order of magnitude increase every 1.2 years). Influential LLMs are annotated on the plot with red points. Data from Epoch AI ( Epoch AI, 2025 ) .
Subtask
# Questions
# Multiple-choice options
Boolean expressions
250
2
Causal judgement
187
2
Date understanding
250
6
Disambiguation QA
250
3
Formal fallacies
250
2
Geometric shapes
250
11
Table 1: BIG-Bench Hard subtasks in my dataset .
Figure 3: Illustrated three-parameter item response curve . This curve represents Equation 7 . “Difficulty” ( β=0 ) represents how difficult an item is relative to others by affecting a lateral transformation in p(θ) across all ability scores. “Discrimination” ( α=1 ) represents the item’s ability to differentiate a narrow range of ability scores by affecting the slope of the sigmoidal curve dθdp(θ) . “Guessing probability” ( c=1/5=0.2 ) represents the rate of success at limθ→0p(θ) , which for multiple-choice questions will be the odds of guessing correctly from 1/c options.
Figure 4: Raw BBH score correlation matrix . The correlation metric is Spearman’s rank correlation ρ . The labels along the y -axis are acronym abbreviations for the full subtask names, which are given along the x -axis.
Figure 5: Logistic scaling law fits for two subtasks . Ruin names has the highest observed correlation with the logistic fit ( R2=0.52 ); Web of lies has the lowest correlation ( R2=0.12 ).
Figure 6: BBH score residuals correlation matrix . Score residuals are taken from the logistic fits against the log -parameter size displayed in Figure 13 . The correlation metric is Spearman’s rank correlation coefficient, ρ . The labels along the y -axis are acronym abbreviations for the full subtask names, which are given along the x -axis.
Figure 7: Scree plots for BBH score matrices .
Eq.
Str.
Log.
χ2↓
CFI ↑
RMSEA ↓
SRMR ↓
AIC ↓
BIC ↓
( 13a )
✗
✗
2984
0.9737
0.0876
0.0134
-217242
-216578
( 13b )
✓
✗
3615
0.9680
0.0894
0.0163
-220369
-219673
( 13c )
✗
✓
1547
0.9836
0.0623
0.0110
222038
222703
( 13d )
✓
✓
2127
0.9782
0.0679
0.0135
218743
219439
Table 2: Fit measures for the four tested EFA models . The Str. column indicates whether the model includes model parameter size log(n) as a structural regression parameter. The Log. column indicates whether the dataset uses the transformed matrix B′ in place of B . The fourth row, with Equation 13d , shows the structured capabilities model. Highlighted boldface values for χ2 , CFI , RMSEA , and SRMR indicate the best overall model, since these absolute model fit metrics share the same range. Highlighted boldface values for AIC and BIC indicate the most parsimonious model for a given data matrix, either B or B′ , where the likelihood is comparable.
Figure 8: Linear latent factor model results . This model corresponds to Equation 13a from Experiment A. Factor 1 dominates, explaining the largest proportion of the total variance ( 0.72 ) and correlating strongest with parameter size ( R2=0.474 ). Factor 2 loads strongest on some more challenging subtasks (like Tracking shuffled objects) and also correlates strongly with model size ( R2=0.419 ).
Figure 9: Structured capabilities model results . This model corresponds to Equation 13d from Experiment A. Factors 1 and 2 share appreciable proportions of the total variance ( 0.41 and 0.36 ), followed by factor 3 ( 0.14 ). All three of these factors correlate strongly with parameter size ( R2=0.379,0.554,0.416 ).
MSEtrain
MSEtest
Withheld subtask
SC
OSL
Size
SC
OSL
Size
Boolean expressions
1.056
1.153
2.487
0.206
0.239
1.144
Causal judgement
0.476
0.469
0.846
0.338
0.293
0.687
Date understanding
0.464
0.462
1.419
0.250
0.149
1.022
Disambiguation QA
1.536
1.519
2.592
0.608
0.680
1.764
Formal fallacies
0.961
0.984
1.490
0.862
1.092
4.411
Table 3: Prediction errors for held-out subtasks . Each column shows the mean-squared error for predicting held-out subtasks with each different capability estimation method. SC refers to my structured capabilities model with latent factor capability estimates ( Equation 16a ). OSL refers to the PCA decomposition approach from Ruan et al. (2024) ’s observational scaling laws ( Equation 16b ). Size refers to Owen (2024) ’s logistic scaling laws using model parameter size ( Equation 17 ). Highlighted boldface values indicate the lowest errors for each of MSEtrain and MSEtest .
Figure 10: Parameter estimates for the observational scaling law model . The heldout task to predict is Geometric shapes.
Figure 11: Parameter estimates for the structured capabilities model . The held-out task to predict is Geometric shapes.
Figure 12: Variance explained by estimated capabilities . For both models the held-out task to predict is Geometric Shapes.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Dataset description
Citation
GSM8K
Grade school-level arithmetic word problems.
Cobbe et al. (2021)
MATH
Challenging high school-level competition mathematics problems.
Hendrycks et al. (2021)
FrontierMath
Unpublished, expert-level research mathematics problems.
Glazer et al. (2024)
GPQA
“Google-Proof Q&A.” PhD-level biology, physics, and chemistry multiple choice questions.
Rein et al. (2024)
MMLU-Pro
Broad knowledge and reasoning benchmark with questions spanning 14 academic subjects.
Table 4: All mentioned benchmark datasets, with citations .
Figure 13: Raw BBH scores against log -parameter size . The red line represents a logistic fit using the functional form of Equation 8 . R2 values range from 0.12 (Web of lies) to 0.52 (Ruin names).
Recent work in machine learning increasingly attributes human-like capabilities such as reasoning or theory of mind to large language models (LLMs) on the basis of benchmark performance. This paper examines this practice through the lens of construct validity, understood as the problem of linking theoretical capabilities to their empirical measurements. It contrasts three influential frameworks: the nomological account developed by Cronbach and Meehl, the inferential account proposed by Messick and refined by Kane, and Borsboom's causal account. I argue that the nomological account provides the most suitable foundation for current LLM capability research. It avoids the strong ontological commitments of the causal account while offering a more substantive framework for articulating construct meaning than the inferential account. I explore the conceptual implications of adopting the nomological account for LLM research through a concrete case: the assessment of reasoning capabilities in LLMs.
Timo Freiesleben
Munich Center for Mathematical Philosophy (MCMP) LMU Munich
Evaluating large language models (LLMs) relies heavily on benchmark scores, yet aggregate metrics can obscure whether individual items serve their intended measurement purpose. BSDProbe provides an item-level framework for benchmark structural diagnosis that jointly characterizes correctness transitions and their supporting response evidence. Repeated responses along an empirically ordered model axis yield complementary correctness and normalized entropy trajectories, from which four-dimensional profiles of boundary position, boundary width, boundary-signal validity, and order consistency are constructed for benchmark-level aggregation. Experiments on six benchmarks identify varying proportions of potentially anomalous items. With Qwen3 as both the selection and reference axis, BSDProbe achieves a 17.9% mean relative improvement in normalized discrimination ratio over the best baseline per benchmark and matches or exceeds all baselines in ranking agreement with the full benchmark.
The rapid development and growing deployment of large language models (LLMs) have made it increasingly important to understand their capabilities. A common approach is to evaluate LLMs using assessment instruments originally designed to measure skills and competencies in humans, such as standardized exams, and to use performance on these instruments as evidence for generalizable claims about LLMs' underlying abilities on the same skills the assessments are intended to measure in humans. However, from a validity perspective, such inferences require that the relationship between observed performance and underlying constructs established for humans also holds for LLMs. In particular, a necessary condition for transferring score interpretations is similarity in the latent structure of responses to the assessment. In this study, we examine whether this condition holds in two educational contexts: high-school chemistry and a quantitative reasoning section of a university entrance exam. Using a case study design, we compare human response data with responses generated by six multimodal LLMs. Our analytical approach combines exploratory factor analysis, factor congruence, and resampling to assess latent structure similarity across human learners and LLMs. Across both instruments, we find systematic differences between human and LLM factor structures, showing evidence that the analyzed assessments may not measure the same constructs for humans and LLMs. These findings call into question the validity of evaluation practices that use educational assessments to make claims about AI capabilities.