Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilities

Xu Zhang; Xudong Gong; Jiacheng Qin; Qiang Wang; JiaQi Liao; Zhe Wang; Dawei Feng; Bo Ding

cs.AIApr 14, 2026

Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilities

Authors: Xu Zhang, Xudong Gong, Jiacheng Qin, Qiang Wang, JiaQi Liao, Zhe Wang, Dawei Feng, Bo Ding

Organizations: College of Computer Science and Technology, National University of Defense Technology, Changsha, Hunan, China · State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Humanities and Social Sciences, School of Public Administration, Beihang University, Beijing, China · National Key Laboratory of Parallel and Distributed Computing, Changsha, Hunan, China

Abstract

Current evaluations of large language models aggregate performance across diverse tasks into single scores. This obscures fine-grained ability variation, limiting targeted model improvement and ability-guided selection for specific tasks. Motivated by this gap, we propose a cognitive diagnostic framework that estimates model abilities across multiple fine-grained dimensions. For mathematics, we construct a 35-dimensional ability taxonomy grounded in cognitive theory and domain knowledge. The framework employs multidimensional Item Response Theory with an item-ability association matrix to estimate fine-grained ability levels, which in turn enable prediction of performance on unseen items (questions of benchmark). Evaluated on 41 models, our approach demonstrates strong criterion validity, consistent ability estimates across benchmarks, and accurate prediction of unseen items with AUC ranging from 0.80 to 0.89 within benchmarks and from 0.77 to 0.86 across benchmarks, substantially exceeding trivial baselines. The framework generalizes across scientific domains, producing consistent diagnostic performance in physics (27 dimensions), chemistry (58 dimensions), and computer science (12 dimensions). This work establishes a principled framework for fine-grained assessment of abilities, with potential applications in targeted training, ability-guided model selection, and ability-aware benchmark design.

Figures & tables

Figure 1: A diagnostic framework for evaluating LLMs via fine-grained mathematical abilities. Centered on Bloom’s taxonomy, we decompose Mathematical Abilities into two core dimensions: Knowledge (e.g., Algebra, Calculus) and Cognitive (e.g., Multi-step Problem Solving, Critical Reasoning). Each is further subdivided into interpretable sub-skills. The framework links benchmark questions (left) to their underlying abilities and maps them to real-world problem domains (right), enabling granular diagnosis beyond aggregate scores and bridging the gap between evaluation benchmarks and practical task demands.

Figure 2: Architecture of the fine-grained ability assessment framework. The framework comprises three stages: (a) Response Matrix Construction, where models are evaluated on benchmarks to generate a binary model-item response matrix; (b) Association Matrix Construction, in which each item of benchmark is mapped to fine-grained abilities (e.g., domain knowledge or cognitive processes) via expert annotation and prior knowledge, forming a Q-matrix; and (c) Fine-Grain Ability Evaluation, where a cognitive diagnostic model estimates latent ability profiles by jointly analyzing the response and association matrices.

Figure 3: Structure of the IRT-based fine-grained ability evaluation model. Based on the IRT method, a simple fully connected neural network is implemented on the Interaction Layer to fit model performance in downstream tasks, thereby achieving fine-grained ability evaluation.

Panel of Figure 4 — Figure 4: Distribution of item coverage across 35 fine-grained abilities in three mathematical benchmarks. The distributions are inconsistent across knowledge-oriented and cognition-oriented dimensions, with pronounced differences among individual dimensions. Left : benchmark-knowledge ability associations, indicating distinct assessment emphases across benchmarks. Right : benchmark-cognition ability associations, exhibiting greater overlap in assessment tendencies while retaining benchmark-specific emphases.

Figure 5: Correlation between estimated fine-grained abilities and model performance on associated benchmark items. Spearman’s rank correlation coefficient ( $\rho$ ) measures the association between each ability dimension (columns) and model accuracy on its linked items across three benchmarks (rows). Color coding : Blue indicates $\rho>0.7$ (strong), yellow indicates $0.5<\rho\leq 0.7$ (moderate), and red indicates $0.3<\rho\leq 0.5$ (weak). Significance markers : ** denotes $p<0.01$ , * denotes $p<0.05$ . N/A indicates that no items in the benchmark are associated with that dimension (i.e., invalid for evaluation). Dimensions with consistent strong correlation across benchmarks reflect stable diagnostic signals; weak correlations often arise from low item coverage (<10 items).

Figure 6: Cross-benchmark correlation of estimated fine-grained ability levels. Spearman’s

\rho

between paired benchmarks, computed over 19 of 35 ability dimensions with

>10

items in all benchmarks. Color: purple = MMLU-MATH vs MATH500, blue = MMLU-MATH vs MMLU-Pro-MATH, red = MMLU-Pro-MATH vs MATH500. Shape: squares (

p<0.01

), triangles (

p<0.05

), diamonds (

p>0.05

). Dashed lines mark thresholds:

\rho=0.3

(weak),

0.5

(moderate),

0.7

(strong). 14 of these 19 dimensions show

\rho>0.5

across all pairwise comparisons.

Figure 7: AUC distribution across 41 LLMs for nine prediction scenarios. Boxplots show median, IQR, and outliers (whiskers = 1.5

\times

IQR). First three: within-benchmark prediction on MMLU-MATH, MATH500, MMLU-Pro-MATH; remaining six: cross-benchmark prediction over all ordered benchmark pairs. All scenarios significantly outperform baselines under identical item assumptions: accuracy-based (AUC = 0.50), standard IRT (AUC = 0.50), and random prediction (mean AUC = 0.49).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Category	Series	Model	Parameter Size
1	Online Model	DeepSeek	deepseek-v3.2	N/A
2		DeepSeek	deepseek-v3	N/A
3		GPT	gpt-4o	N/A
4			gpt-5.1	N/A
5			gpt-o1	N/A
6			gpt-o3	N/A

Appendix

Table 1: List of evaluated language models with unique identifiers.

ID	Subject	Name	# Items	Source	Answer Format
1	Math	MMLU-MATH	748	cais/mmlu	Multiple-choice
2		MMLU-Pro-MATH	1351	TIGER-Lab/MMLU-Pro	Multiple-choice
3		MATH500	500	HuggingFaceH4/MATH-500	Free-response
4	Chemistry	MMLU-Pro-Chemistry	1132	TIGER-Lab/MMLU-Pro	Multiple-choice
5	Physics	MMLU-Pro-Physics	1299	TIGER-Lab/MMLU-Pro	Multiple-choice
6	Computer science	MMLU-Pro-CS	410	TIGER-Lab/MMLU-Pro	Multiple-choice

Appendix

Table 2: List of evaluated benchmarks with detailed metadata.

Dimension	MMLU-MATH		MMLU-Pro-MATH		MATH500
	Count	Ratio	Count	Ratio	Count	Ratio
ability_0	61	0.0816	55	0.0407	20	0.0400
ability_1	340	0.4545	300	0.2221	125	0.2500
ability_2	77	0.1029	112	0.0829	84	0.1680
ability_3	183	0.2447	189	0.1399	167	0.3340
ability_4	23	0.0307	57	0.0422	64	0.1280

Appendix

Table 3: Item coverage of per ability dimension across mathematical benchmarks.

Model	MMLU-MATH	MATH500	MMLU-Pro-MATH
0	0.8711 $\pm$ 0.0489	0.8732 $\pm$ 0.0401	0.8522 $\pm$ 0.0303
1	0.8672 $\pm$ 0.0356	0.8434 $\pm$ 0.0863	0.8683 $\pm$ 0.0288
2	0.8548 $\pm$ 0.0338	0.8049 $\pm$ 0.0681	0.8584 $\pm$ 0.0222
3	0.8560 $\pm$ 0.0593	0.8557 $\pm$ 0.0405	0.8573 $\pm$ 0.0398
4	0.8583 $\pm$ 0.0607	0.8634 $\pm$ 0.0654	0.8547 $\pm$ 0.0316
5	0.8474 $\pm$ 0.0331	0.8586 $\pm$ 0.0552	0.8517 $\pm$ 0.0480

Appendix

Table 4: Within-benchmark unseen item prediction performance per model (AUC

\pm

std over 10 random splits with different seeds).

Model	MMLU $\rightarrow$ MATH500	MMLU $\rightarrow$ MMLU-Pro	MATH500 $\rightarrow$ MMLU	MATH500 $\rightarrow$ MMLU-Pro	MMLU-Pro $\rightarrow$ MMLU	MMLU-Pro $\rightarrow$ MATH500
0	0.7686 $\pm$ 0.0070	0.8345 $\pm$ 0.0035	0.8254 $\pm$ 0.0027	0.8305 $\pm$ 0.0023	0.8448 $\pm$ 0.0057	0.7870 $\pm$ 0.0022
1	0.7942 $\pm$ 0.0031	0.8329 $\pm$ 0.0032	0.8132 $\pm$ 0.0017	0.8106 $\pm$ 0.0022	0.8330 $\pm$ 0.0030	0.8287 $\pm$ 0.0045
2	0.7766 $\pm$ 0.0033	0.8385 $\pm$ 0.0035	0.8498 $\pm$ 0.0032	0.8203 $\pm$ 0.0017	0.8521 $\pm$ 0.0038	0.7953 $\pm$ 0.0028
3	0.7784 $\pm$ 0.0041	0.8425 $\pm$ 0.0046	0.8390 $\pm$ 0.0017	0.8268 $\pm$ 0.0019	0.8636 $\pm$ 0.0046	0.7917 $\pm$ 0.0052
4	0.8085 $\pm$ 0.0052	0.8288 $\pm$ 0.0041	0.8299 $\pm$ 0.0019	0.8110 $\pm$ 0.0018	0.8309 $\pm$ 0.0047	0.8201 $\pm$ 0.0028
5	0.7774 $\pm$ 0.0036	0.8457 $\pm$ 0.0045	0.8158 $\pm$ 0.0049	0.8172 $\pm$ 0.0020	0.8145 $\pm$ 0.0042	0.8197 $\pm$ 0.0024

Appendix

Table 5: Cross-benchmark unseen item prediction performance per model (AUC

\pm

std over repeat for 10 times).

Explore similar work

Cost-Efficient Estimation of General Abilities Across Benchmarks
Apr 1, 2026Michael Krumdick, Adam Wiemerslage, Seth Ebner +2LLM Evaluation Item Response Theory
Kensho Technologies · MIT
Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
Oct 26, 2025Peiyu Li, Xiuxiu Tang, Si Chen +4LLM Evaluation Psychometric Validation
University of Notre Dame, Notre Dame, USA
LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model
Aug 23, 2026Ergan Shang, Weijing Tang, Yinqiu HeLLM Evaluation Language Model Generation Evaluation
Department of Statistics and Data Science, Carnegie Mellon University · Department of Statistics, University of Madison

Apr 1, 2026J/K move · Enter open · S save

Cost-Efficient Estimation of General Abilities Across Benchmarks

Michael Krumdick, Adam Wiemerslage, Seth Ebner, Charles Lovering+1

Kensho Technologies · MIT

LLM Evaluation Item Response Theory

Open paper AskPDF arXiv

Thousands of diverse benchmarks have been developed to measure the quality of large language models (LLMs). Yet prior work has demonstrated that LLM performance is often sufficiently explained by a small set of latent factors, or abilities. This suggests the potential for more efficient and principled benchmarking, but it remains difficult to compare the quality of different methods. Motivated by predictive validity, we argue that the quality of a benchmarking framework should be grounded in how efficiently it enables the prediction of model performance on unseen tasks. To analyze this objective, we collect the "Wide-scale Item Level Dataset" (WILD), a dataset of item-model response pairs, comprising evaluations of 65 models on 109,564 unique items spanning 163 tasks drawn from 27 datasets. This dataset enables the first analysis of how different techniques can predict a model's performance on a large, diverse collection of unseen tasks under different budget constraints. We demonstrate that combining a modified multidimensional item response theory (IRT) model with adaptive item selection driven by optimal experimental design can predict performance on 112 held-out benchmark tasks with a mean absolute error (MAE) of less than 7%, and can do so after observing only 16 items. We further demonstrate that incorporating cost-aware discount factors into our selection criteria can reduce the total tokens needed to reach 7% MAE from 141,000 tokens to only 22,000, an 85% reduction in evaluation cost.

Abstract

Explore similar work

Cost-Efficient Estimation of General Abilities Across Benchmarks

Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model