SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models

Berke Arda; Peng Cui; Qiaoyuan Zheng; Rudolf Debelak; Mubashara Akhtar; Mrinmaya Sachan

cs.CLOct 20, 2026

SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models

Authors: Berke Arda, Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mubashara Akhtar, Mrinmaya Sachan

Organizations: Department of Computer Science ETH Zurich · Machine Learning and Optimization Lab EPFL · ETH AI Center ETH Zurich

Abstract

Current evaluation of LLMs on benchmarks typically aggregates performance over individual examples into a single scalar score. This approach is overly coarse and ignores the diverse scope of skills covered by benchmark datasets. To address this limitation and better exploit benchmark data, we propose SkillEval, a new evaluation framework that performs cross-model, skill-level analysis. The framework integrates advances from two areas: (1) it leverages the meta-cognitive capabilities of LLMs to automatically discover interpretable skills from questions, and (2) it applies cognitive diagnosis models (CDMs) from psychometrics to estimate per-skill abilities of LLMs. We conduct experiments on 3,811 LLMs evaluated on 9,523 items across five benchmarks spanning diverse domains (MATH, BBH, GPQA, MuSR, IFEval). Compared to the traditional average-based evaluation paradigm, SkillEval produces reliable, more calibrated and informative yet concise model profiles that more accurately predict model behavior on unseen items. We also demonstrate the utility of SkillEval in two downstream tasks: (1) LLM routing, which dynamically assigns each query to a capable but efficient model, reducing inference cost by over 60% while achieving improved performance, and (2) Computerized Adaptive Testing (CAT), where an unseen model can be efficiently evaluated using a trained CDM. Furthermore, we show that these skill profiles provide useful insights, for example, how benchmarks share skills and how instruction-tuned models differ from their base models.

Figures & tables

Figure 1: We first extract skills from the questions and construct a question-to-skill mapping (i.e., the

Q

-matrix). Then, using this

Q

-matrix together with the performance matrix of models on items, we train a neural CDM to estimate each model’s skill-level abilities

\theta

Table 1: Comparison between the LLM-extracted skill set (

{\rm K}=100

) and an expert-defined mathematics skill set (the Common Core standards;

{\rm K}=406

Model	Flat	Per-LLM	Per-dim
Model	Flat	Per-LLM	Raw	Orthogonal	Full
IRTNet ( $d{=}232$ )	0.02 ( $\pm$ 0.06)	0.02 ( $\pm$ 0.06)	0.00 ( $\pm$ 0.01)	0.43 ( $\pm$ 0.22)	0.43 ( $\pm$ 0.21)
IRTNet ( $d{=}512$ )	0.00 ( $\pm$ 0.04)	0.00 ( $\pm$ 0.04)	0.00 ( $\pm$ 0.00)	0.17 ( $\pm$ 0.02)	0.17 ( $\pm$ 0.01)
IRTNet ( $d{=}1024$ )	0.01 ( $\pm$ 0.00)	0.01 ( $\pm$ 0.00)	0.00 ( $\pm$ 0.00)	0.10 ( $\pm$ 0.02)	0.10 ( $\pm$ 0.02)
SkillEval ( ${\rm K}=100$ )	0.99 ( $\pm$ 0.00)	0.99 ( $\pm$ 0.01)	0.96 ( $\pm$ 0.00)	0.96 ( $\pm$ 0.00)	0.96 ( $\pm$ 0.00)

Table 2: Mean Pearson correlations between ability matrices

\Theta

across run pairs, with standard deviations across pairs in parentheses. Flat correlates all entries of the two ability matrices after arranging them into vectors. Per-LLM correlates each LLM’s ability profile between runs and averages over LLMs. These summaries can be high when some dimensions consistently take larger values than others. Per-dim instead compares values across LLMs separately for each dimension, then averages the correlations over dimensions. It therefore tests whether each dimension assigns similar relative values to LLMs across runs. Raw uses the dimensions as learned. Orthogonal applies the rotation or reflection that best aligns one run’s ability matrix with the other ( Schönemann, 1966 ) , and Full additionally centres and rescales the matrices before alignment ( Gower, 1975 ) .

Model	AUC		Acc
Model	ID	OOD	ID	OOD
IRT (2PL)	86.16	–	78.76	–
IRTNet ( $d{=}232$ )	81.38	75.04	74.21	69.05
IRTNet ( $d{=}512$ )	81.36	75.14	74.21	69.07
IRTNet ( $d{=}1024$ )	81.74	75.35	74.68	69.38
SkillEval ( ${\rm K}=100$ )	81.73	72.15	75.57	67.62

Table 3: AUC and accuracy (Acc) under two settings: ID (interaction-wise split) and OOD (item-wise split). For Acc, a response is predicted correct when its predicted probability is at least

0.5

. IRT 2PL has no item features and therefore cannot generalise to held-out items; its OOD column is left blank. OOD values are means over seeds, five for SkillEval and three for IRTNet. ID values are single runs.

Figure 2: Predicted versus observed benchmark accuracy on held-out items in the OOD setting. Each point represents one LLM. The horizontal axis shows its observed accuracy on the benchmark’s held-out items; the vertical axis shows the mean predicted probability of a correct answer on those items. The diagonal indicates exact agreement, and $r$ is the Pearson correlation across $3{,}811$ LLMs. IRTNet uses $d{=}232$ .

Method	Acc@1	Acc@3	Acc@5	Acc@10
Random	40.47	–	–	–
Strongest single LLM	61.57	–	–	–
IRT 2PL	61.10	77.74	81.94	85.93
KNN	62.05	78.01	83.10	88.98
IRTNet, 3 seeds	63.08	77.31	81.21	86.44
	( $\pm$ 0.98)	( $\pm$ 0.55)	( $\pm$ 1.60)	( $\pm$ 0.76)

Table 4: Routing performance on

1{,}905

held-out items with

3{,}811

candidate LLMs. All scores are percentages. Rows marked “3 seeds” report the mean over three trainings with different seeds, with the standard deviation in parentheses; IRTNet uses

d{=}1024

, its best size in Table 3 . The strongest single LLM and KNN involve no training or randomness, Random is a single random draw, and the other rows are single training runs. Best results are bold .

Figure 3: Accuracy vs. inference cost for three routers using the same threshold rule. Both axes are expressed relative to the strongest single LLM selected on training items. Each point corresponds to one threshold

\tau

Figure 4: Profiling

200

held-out LLMs from

N

answers. (a) Mean absolute benchmark-score error across LLMs and the five benchmarks; the dotted line uses all

7{,}618

pool answers with the same estimator. (b) Pearson correlation with profiles estimated from all answers without the prior, computed across LLMs for each skill and averaged over skills.

Figure 5: Estimated mastery differences between base and instruction-tuned models. ( a ) Benchmark means, assigning each skill to its primary benchmark by item count; stars indicate unadjusted

p

-values (

*\,p{<}0.05

**\,p{<}0.01

***\,p{<}0.001

). ( b ) The

14/100

skills significant after Bonferroni correction (

\alpha{=}5{\times}10^{-4}

), sorted by mean difference and coloured by primary benchmark; names are abbreviated. Error bars:

\pm 1

SEM across

81

matched model pairs. Omitted skills are not necessarily unchanged.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$M$	Number of LLMs ( $M{=}3{,}811$ )
$N$	Number of items ( $N{=}9{,}523$ ); also the number of answers used to profile a new LLM
$K$	Number of skill clusters ( $K{=}100$ default)
$i\in\{1,\dots,M\}$	LLM index
$j\in\{1,\dots,N\}$	Item index
$k\in\{1,\dots,K\}$	Skill index

Appendix

Table 5: Notation summary.

Skill cluster	Items	Example items
Quadratic Formula (MATH-primary, 429/451)	451	“Find the domain of $f(x)=\sqrt{\sqrt{x^{2}-16}-3}$ ” “If $-6\leq a\leq-2$ and $3\leq b\leq 5$ , find the greatest value of $\frac{a+b}{a}$ ”
Truthfulness of Nested Statements (BBH-primary, 1836/1880)	1,880	“Then he tells Billy: ‘Be sure that you do not come in at all tomorrow morning’ ” (causal judgement) “Teressa lies. Willian says Teressa lies. Jamey says Willian lies. Ryan says Jamey lies. Kandi says Ryan lies. Does Kandi tell the truth?” (web of lies)
Structural Isomers in Organic Compounds (GPQA-primary, 442/488)	488	“Identify product D [from NMR spectrum]: (A) 2-bromonaphthalene (B) 2-bromo-1,2,3,4-tetrahydronaphthalene…”
Word Count Constraints (IFEval-primary, 414/418)	418	“Write a social media post about the hajj pilgrimage. Use all lowercase and include the word ‘story’ at least twice”

Appendix

Table 6: Representative skill clusters with example items. Each cluster aggregates items from multiple benchmarks that share a common skill requirement.

	Both	Skill 1	Skill 2	Neither
BBH (conjunctive)	0.74	0.33	0.22	0.10
IFEval (compensatory)	0.93	0.97	0.42	0.12

Appendix

Table 7: Accuracy by mastery group on two held-out items. The BBH item is conjunctive (needing both skills). The IFEval item is compensatory (either skill suffices).

$K$	Test AUC	Acc@1	Acc@3	Acc@5	Acc@10
50	0.720	0.640	0.783	0.829	0.877
100	0.717	0.657	0.780	0.829	0.875
150	0.719	0.654	0.778	0.831	0.887
200	0.716	0.661	0.797	0.846	0.885
300	0.719	0.672	0.794	0.843	0.882
400	0.721	0.669	0.802	0.841	0.887

Appendix

Table 8: Effect of the number of skill clusters

K

on diagnostic AUC and routing Acc@

k

. Diagnostic fit is essentially constant across

K

, and routing accuracy rises modestly with

K

. Each row is a single training run with batch size

1024

; the

K{=}100

row is the main model, so its test AUC differs from the five-seed mean in Table 3 .

	Error			Agreement		AUC
$N$	Random	Adaptive	Count	Random	Adaptive	Random	Adaptive
0	0.095	0.095	0.095	0.000	0.000	0.682	0.682
5	0.087	0.079	0.202	0.207	0.301	0.682	0.684
10	0.082	0.068	0.218	0.252	0.351	0.682	0.686
20	0.075	0.061	0.217	0.329	0.417	0.682	0.687
50	0.068	0.056	0.154	0.396	0.470	0.684	0.688

Appendix

Table 9: Profiling

200

held-out LLMs from

N

answers. Error: mean absolute benchmark-score error over LLMs and benchmarks. Agreement: mean per-skill Pearson correlation with profiles estimated from all pool answers without the prior. AUC: mean per-LLM test-item AUC. Both selection methods are evaluated up to

500

answers; random-selection results are averaged over three seeds. The all-answers error and AUC use the estimator with the prior.

Benchmark	$n$	Random	Strongest	IRT 2PL	SkillEval
MATH	264	0.152	0.375	0.530	0.542
BBH	1159	0.471	0.716	0.690	0.735
GPQA	246	0.280	0.370	0.402	0.407
MuSR	136	0.301	0.529	0.419	0.544
IFEval	100	0.450	0.810	0.680	0.820
Overall	1905	0.405	0.616	0.611	0.657

Appendix

Table 10: Routing Acc@1 by source benchmark on the held-out test items. Strongest is the train-selected top-1 LLM. SkillEval beats the single-LLM baseline and IRT 2PL on all five benchmarks.

Figure 6: Distribution of “weak-beats-strong” items by source benchmark.

38\%

of held-out items (

n{=}723/1{,}905

) admit at least one

\leq{}13

B LLM that out-performs the global strongest model, spanning

41

skill clusters: BBH

329

, MATH

156

, GPQA

155

, MuSR

64

, IFEval

19

Held-out	$n$	Random	Strongest	SkillEval (transfer)	Reference
MuSR	756	0.415	0.467	0.470	0.544
IFEval	536	0.437	0.731	0.410	0.820
Combined	1292	0.400	0.577	0.445	—

Appendix

Table 11: Cross-benchmark transfer with MuSR and IFEval held out entirely. “Reference” is the full-training Acc@1 from § 4.1 . Transfer succeeds on MuSR (overlap with BBH multi-step reasoning) but fails on IFEval, where

14

of the

62

assigned skills receive no training items.

Figure 7: Skill coverage per benchmark. Dark segments show skills unique to that benchmark; light segments show shared skills. IFEval contributes

13

unique skills, while BBH has only

1

. MATH + IFEval together cover

87/100

skills.

Figure 9: Mean estimated mastery difference (instruction-tuned $-$ base) per model family and benchmark. Blue = improved, red = degraded. The IFEval column is positive for every evaluated family; the MuSR column is negative for every evaluated family.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

A Framework for Evaluating Agentic Skills at Scale
Jun 16, 2026Maksim Shaposhnikov, Nicolas Fortuin, Simon Stipcich +3LLM Agent Skill Evaluation LLM Agent Evaluation
Tessl London, United Kingdom
OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents
May 22, 2026Jiahao Ying, Boxian Ai, Wei Tang +2LLM Agent Skill Evaluation LLM Agent Evaluation
Singapore Management University · Institute of Trustworthy Embodied AI, Fudan University · Joy Future Academy, JD
Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilities
Apr 14, 2026Xu Zhang, Xudong Gong, Jiacheng Qin +5LLM Evaluation Cognitive Diagnostic Models
College of Computer Science and Technology, National University of Defense Technology, Changsha, Hunan, China · State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Humanities and Social Sciences, School of Public Administration, Beihang University, Beijing, China +1

Jun 16, 2026J/K move · Enter open · S save

A Framework for Evaluating Agentic Skills at Scale

Maksim Shaposhnikov, Nicolas Fortuin, Simon Stipcich, Maria I. Gorinova+2

Tessl London, United Kingdom

LLM Agent Skill Evaluation LLM Agent Evaluation Instruction Following Agent Benchmarks

Open paper AskPDF arXiv

Agent skills -- structured, reusable knowledge artifacts that augment LLM agent capabilities -- have been rapidly adopted in industry, yet their cross-domain impact and use across commercial and open-source models remain under-studied, and no reusable methodology exists for evaluating an individual skill. In this work, we present an evaluation framework that lets a skill author construct realistic tasks to rigorously assess the aspects of a skill that matter most to them, and that estimates skill utility by solving those tasks. Further, we apply our evaluation approach at scale to 500 real-world skills, generating 1,000 tasks derived from the skills' content, along with instruction-following and goal-completion scoring rubrics. Using these metrics, we evaluate how 19 agent-model configurations, both proprietary and open-source, perform on the tasks. Our results show that models vary widely in how closely they adhere to the instructions encoded in skills, leading to substantial differences in their performance gains. Furthermore, we show that access to a skill significantly changes model behavior compared to the no-skill setup, providing an essential mechanism for encoding opinionated workflows into LLM agents. We release our evaluation dataset to support future work on agent skills.

Abstract

LLM Evaluation

Explore similar work

A Framework for Evaluating Agentic Skills at Scale

OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents

Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilities