Embedding-Based Educational Taxonomies

Kia Karbasi; Kaveh Karbasi; Gregory Pottie

cs.CLOct 24, 2026

Embedding-Based Educational Taxonomies

Authors: Kia Karbasi, Kaveh Karbasi, Gregory Pottie

Organizations: University of California, Los Angeles · Tempus AI, Inc.

Abstract

Educational taxonomies organize educational content, such as exercise problems, into concepts, which are also called skills or knowledge components. These taxonomies define learning objectives in educational applications such as intelligent tutoring systems. Developed through human expertise and iterative refinement, they also provide fundamental structure for curricula, assessment, and models of student learning. Recent advances in semantic text embeddings make it possible to construct alternative educational taxonomies computationally from the content itself. In this paper, we investigate this possibility by clustering embeddings of math questions to construct embedding-based educational taxonomies. We compare these taxonomies with the existing Common Core State Standards (CCSS)-based knowledge-component taxonomies in two ASSISTments datasets and evaluate their utility in Deep Knowledge Tracing, a downstream task that relies on taxonomic mappings of questions to knowledge components. The embedding-based taxonomies exhibit substantial but incomplete correspondence with the existing taxonomies, revealing both preserved and alternative organizations of educational content. When used for knowledge tracing, they improve predictive performance by approximately 3-5 AUC percentage points. These results demonstrate the potential of semantic embeddings for automatically constructing educational taxonomies that both recover structure present in human-developed taxonomies and provide useful alternative representations for downstream educational modeling.

Figures & tables

Figure 1: Knowledge tracing with an induced taxonomy and sequence relabeling. Panel (a) conceptually illustrates DKT tracking induced KCs; the illustrated mastery states are conceptual and do not represent measured student knowledge. Panel (b) shows the actual relabeling procedure for DKT evaluation: the original KC label in each interaction is replaced with the induced label while questions, responses, sequence order, student splits, and evaluation remain unchanged.

Dataset	Total questions	PB questions	Total KCs	PB KCs	Students	Responses
ASSISTments 2012	179,999	32,792	265	162	23,845	978,271
ASSISTments 2017	3,162	638	102	68	1,703	237,092

Table 1: Dataset and taxonomy summary. Total counts describe the full releases; PB counts describe the Problem Bodies text-linked question/KC subsets used for text-based analyses. Student and response counts describe the text-linked subset used for DKT evaluation. For both datasets, we construct induced taxonomies using outputs from OpenAI’s text-embedding-3-small and text-embedding-3-large , shortened through the API’s dimensions parameter to 768 and 1,536 dimensions, respectively OpenAI (2026) ; we use the PCA dimensions

\{10,20,40,60,80,100,120,160\}

, with

K

values

\{50,100,150,200,250,300\}

for 2012 and

\{20,40,60,80,100,150\}

for 2017. PCA and

k

-means use a random seed of 42;

k

-means uses 10 initializations and at most 300 iterations. The reported (PCA,

K

) settings in Table 2 are (100, 300) for 2012 and (10, 100) for 2017. The PCA dimension and

K

are fixed before each DKT Bayesian search and are not selected by it.

Figure 2: Distribution of questions from the 25 largest categories in the 2012 original taxonomy across the corresponding induced taxonomy. The figure uses text-embedding-3-large , PCA $=100$ , and $K=300$ . The left heatmap shows question counts and the right heatmap shows the share of each original category in each induced category. We include the three induced categories containing the most questions from each original category and combine the rest as Other . The identifiers for induced categories are arbitrary labels and do not indicate an order.

Dataset	Embedding	PCA	$K$	Homogeneity	Completeness	NMI	ARI	AUC with original taxonomy	Highest AUC with induced taxonomy	$\Delta$ AUC
ASSISTments 2012	3-large	100	300	.930	.717	.809	.293	.7367	.7841	+.0474
ASSISTments 2017	3-large	10	100	.681	.509	.582	.066	.7131	.7403	+.0272

Table 2: Taxonomy comparison and DKT results at the reported settings. Scores for taxonomy agreement are averaged across the two embedding models at each listed PCA dimension and

K

. The embedding column identifies the model used for the induced taxonomy with the highest observed test AUC. The induced value is the highest observed test AUC across the tested combinations of embedding model, PCA dimension, and

K

\Delta

AUC is induced minus original.

Figure 3: Mean homogeneity and completeness as the induced taxonomy uses more categories. At each

K

, the lines show averages over the eight PCA dimensions and two embedding models; bands show the full range across those 16 settings and are not confidence intervals. In 2012, homogeneity generally increases while completeness decreases. In 2017, both generally increase, although inverse purity and pairwise recall decline over the same

K

range. Appendix Figure 7 gives the full results for each embedding model and PCA dimension.

Panel of Figure 4 — Figure 4: Change in test AUC when using the induced taxonomy instead of the original taxonomy. The figure uses text-embedding-3-large , which produced the highest observed test AUC in both datasets. Each cell shows the change in test AUC for one PCA and $K$ setting. The test AUC at every displayed setting exceeds the baseline with the original taxonomy. We also test text-embedding-3-small , giving 96 induced taxonomies per dataset in total.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Measure	What the score measures
ARI	Adjusted Rand index: agreement between pairs of questions, adjusted for chance; higher is closer.
NMI / AMI	Normalized / adjusted mutual information: how much information the two taxonomies share; AMI adjusts for chance; higher is closer.
Homogeneity	Whether each induced category mostly contains questions from one original category; higher is closer.
Completeness	Whether questions from each original category mostly stay together in one induced category; higher is closer.
V-measure	Harmonic mean of homogeneity and completeness; higher is closer.
Macro purity	Average share with the most common original label in each induced category, weighting induced categories equally; higher is closer.

Appendix

Table 3: Measures used to compare the original and induced taxonomies Rosenberg and Hirschberg (2007) ; Vinh et al. (2010) ; Meilă (2007) ; Fowlkes and Mallows (1983) . Increasing

K

can raise some scores and lower others, so the scores should be interpreted together.

Dataset	Best ARI	Best AMI	Best V/NMI	Best FM	Lowest VI
2012	.490 (120, 50)	.812 (160, 150)	.830 (160, 150)	.503 (120, 50)	1.554 (160, 150)
2017	.114 (100, 40)	.371 (100, 80)	.663 (120, 150)	.165 (80, 20)	2.789 (120, 150)

Appendix

Table 4: Best agreement scores across the 48 settings, averaged across the two embedding models. ARI is the adjusted Rand index; AMI and NMI denote adjusted and normalized mutual information, respectively; FM is Fowlkes–Mallows; and VI is variation of information. Parentheses give the PCA dimension and

K

. Lower VI means a closer match; higher values mean a closer match for the other measures. NMI and V-measure have the same values here and reach their highest values at the same setting. Table 6 shows these and the purity measures at the smallest and largest values of

K

. Figure 6 shows ARI and NMI separately for each embedding model and dataset.

Dataset (PCA, $K$ )	Homog.	Complete.	V/NMI	Global pur.	Macro pur.	Global inv.	Macro inv.	ARI	AMI	FM	Pair $F_{1}$	VI
2012 $(100,300)$	.930	.717	.809	.888	.898	.406	.617	.293	.776	.393	.299	1.872
2017 $(10,100)$	.681	.509	.582	.530	.546	.266	.486	.066	.272	.123	.083	3.290

Appendix

Table 5: Structural scores at the two PCA/

K

settings used for the reported DKT results. Each score is averaged across the two embedding models. “Homog.” means homogeneity, “Complete.” means completeness, “pur.” means purity, and “inv.” means inverse purity. Table 3 defines ARI, AMI, and NMI; FM means Fowlkes–Mallows. VI uses natural logarithms and is the only score for which lower values mean a closer match. V-measure and NMI have the same values here.

Figure 5: Distribution of questions from the 25 largest categories in the 2017 original taxonomy across the corresponding induced taxonomy. The figure uses text-embedding-3-large , PCA $=10$ , and $K=100$ . The left heatmap shows question counts and the right heatmap shows the share of each original category in each induced category. We include the three induced categories containing the most questions from each original category and combine the rest as Other . Induced category identifiers are arbitrary labels and do not indicate an order.

Panel of Figure 6 — Figure 6: ARI and NMI for every PCA dimension and value of $K$ , shown separately for each embedding model in 2012 (a) and 2017 (b). Table 4 gives the highest values after averaging scores across the two models. In 2017, ARI and NMI reach their highest values at different settings, and, for each metric, each embedding model’s highest value differs from the highest value obtained after averaging scores across the two models.

Panel of Figure 7 — Figure 7: Homogeneity and completeness across values of $K$ , shown separately for each combination of embedding model and PCA dimension in 2012 (a) and 2017 (b). In 2012, homogeneity generally rises while completeness falls. In 2017, both generally rise across the tested settings, although global inverse purity and pairwise recall fall from the smallest to the largest tested value of $K$ , as shown in Table 6 . Outlined circles mark the smallest tested $K$ , and squares mark the largest.

Dataset/ $K$	Homog.	Complete.	Global purity	Global inverse	Pair precision	Pair recall	VI	Size spread
2012/50	.725	.821	.572	.717	.426	.516	1.854	.522
2012/300	.922	.710	.880	.392	.839	.176	1.948	.575
2017/20	.417	.484	.345	.439	.175	.146	3.460	.418
2017/150	.797	.548	.674	.270	.442	.042	2.901	.476

Appendix

Table 6: Scores at the smallest and largest tested values of

K

. Each value is the simple average over eight PCA dimensions and both embedding models. In both datasets, increasing

K

raises homogeneity, global purity, and pairwise precision. It lowers global inverse purity and pairwise recall. In 2012, completeness falls and VI rises. In 2017, completeness rises and VI falls. “Homog.” means homogeneity, “Complete.” means completeness, and VI means variation of information. Size spread is the standard deviation of the induced cluster sizes divided by their mean; a larger value means that the cluster sizes differ more.

Figure 8: Rank correlations between each taxonomy-agreement score and test AUC, both averaged across the two embedding models. “After” indicates that rank patterns linked to PCA dimension and $K$ have been removed. Cells show coefficients; Table 7 reports significance after correction.

Measure	2012 before	2012 after	2017 before	2017 after
ARI	$-.858^{*}$	$-.109$	$-.908^{*}$	$-.273$
AMI	$.041$	$.024$	$.138$	$-.436^{*}$
NMI	$.374^{*}$	$.038$	$.816^{*}$	$-.418^{*}$
Homogeneity	$.906^{*}$	$.125$	$.857^{*}$	$-.425^{*}$
Completeness	$-.847^{*}$	$.097$	$.620^{*}$	$-.443^{*}$
V-measure	$.374^{*}$	$.038$	$.816^{*}$	$-.418^{*}$

Appendix

Table 7: Associations, across the 48 shared settings of PCA dimension and

K

in each dataset, between each taxonomy-agreement score and test AUC, both averaged across the two embedding models. “Before” is the direct Spearman correlation; “After” is the correlation after removing rank patterns linked to PCA dimension and

K

. An asterisk marks

p<.05

after the Benjamini–Hochberg correction, applied separately by dataset and analysis stage. VI is a distance measure, so a positive coefficient indicates that a greater difference between the taxonomies is associated with higher test AUC.

Taxonomy Induction

See all 19 papers

CogTax: A Four-Level Cognitive Taxonomy for Command-Line Computing Education
Jun 30, 2026Manuel Alonso-Carracedo, Ruben Fernandez-Boullon, Pedro Celard +2Hierarchical Classification Computer Science Education
Universidade de Vigo, Spain · IFCAE, Universidade de Vigo, Spain · aComputer Science Department, IFCAE-Instituto de Investigación en Física, Computación y Ciencia Aeroespacial, Universidade de Vigo, Ourense, 32004, Spain +1
K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs
May 10, 2026Hao Liang, Qihan Lin, Zhaoyang Han +5AI in Education Knowledge Augmentation for Language Models
1Peking University · Institute for Advanced Algorithms Research, Shanghai · 3OriginHub Technology +1
Cross-Dataset Bloom Question Classification: Supervised Models and Prompted LLMs
Apr 22, 2026Abdolali Faraji, Mohammadreza Molavi, Zohreh Rasoulkhani +2LLM Evaluation Educational Assessment
Leibniz Information Centre for Science and Technology (TIB) · University of Genoa

Jun 30, 2026J/K move · Enter open · S save

CogTax: A Four-Level Cognitive Taxonomy for Command-Line Computing Education

Manuel Alonso-Carracedo, Ruben Fernandez-Boullon, Pedro Celard, Francisco J. Rodriguez-Martinez+1

Universidade de Vigo, Spain · IFCAE, Universidade de Vigo, Spain · aComputer Science Department, IFCAE-Instituto de Investigación en Física, Computación y Ciencia Aeroespacial, Universidade de Vigo, Ourense, 32004, Spain+1

Hierarchical Classification Computer Science Education Educational Assessment

Open paper AskPDF arXiv

As computing education expands beyond traditional programming into operational domains such as systems administration and command-line environments, existing pedagogical frameworks struggle to capture a dimension that is critical in these contexts: the real-world consequences of learner actions. Existing cognitive taxonomies classify learning objectives by mental operations but do not account for system impact, leaving a critical gap in command-line education where conceptually simple commands can have severe consequences. This work presents CogTax, a four-level cognitive taxonomy that integrates two dimensions: cognitive complexity, derived from Bloom's Revised Taxonomy, and operational impact, which distinguishes observational, reversible, structural, and administrative operations. The four progressive levels range from safe read-only inspection to advanced system management requiring integration of multiple abstract models. Then, the taxonomy level is defined as the maximum of these dimensions, ensuring that both conceptual understanding and operational awareness are addressed. CogTax gives instructors a principled framework for sequencing course material and calibrating assessment difficulty, and gives students an explicit reference for self-assessment and gap identification. To demonstrate that taxonomy levels are automatically assignable, making the framework scalable without manual expert annotation, a classifier that combines syntactic representations derived from abstract syntax trees with semantic embeddings is trained. Evaluated on 585 expert-annotated Linux/bash commands, this combined approach achieves 89% accuracy, outperforming either representation alone, and demonstrates cross-language extensibility through structural equivalences across command languages.

Abstract

Taxonomy Induction

Explore similar work

CogTax: A Four-Level Cognitive Taxonomy for Command-Line Computing Education

K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs

Cross-Dataset Bloom Question Classification: Supervised Models and Prompted LLMs