Does Deep Knowledge Tracing Learn Semantic Relations Between Knowledge Components?
Organizations: University of California, Los Angeles · Sharif University of Technology
Abstract
Deep Knowledge Tracing (DKT) is a knowledge tracing (KT) model that updates a shared latent state from sequences of interactions, each pairing a knowledge component (KC) with a response. An interaction with one KC can therefore affect predictions for other KCs. We ask: Are the KC relations learned by DKT semantic? We construct text-derived semantic KC representations using nine pipelines to test whether the conclusions hold across alternative ways of constructing those representations. The first test compares DKT-derived KC relation graphs with semantic relation structures induced by pairwise cosine similarities among the KC representations. Across the evaluated datasets, embedding models, and pipelines, the two structures align only weakly and inconsistently. The second test introduces semantic information into DKT; the pretrained text-derived representations do not consistently outperform the standard DKT baseline. In the reduced-data experiments, semantic initialization yields a higher test area under the ROC curve (AUC) than the baseline at eight of nine nested sample sizes for ASSISTments 2017. KC-supervised fine-tuned BERT produces small positive AUC differences in some matched comparisons, but the study does not establish their stability across runs. These effects remain limited and inconsistent. The findings suggest that behavioral, temporal, structural, or psychometric signals may also shape DKT's learned relations. DKT-derived relations are not primarily semantic: the tested text-derived semantic structures do not strongly recover the DKT-derived relation structure.
Figures & tables
| Dataset | # Questions | # Questions in PB | # KCs | # KCs in PB |
|---|---|---|---|---|
| ASSISTments 2009 | 26,688 | 7,836 | 123 | 76 |
| ASSISTments 2012 | 179,999 | 32,792 | 265 | 162 |
| ASSISTments 2017 | 3,162 | 638 | 102 | 68 |
| Family | Model | Dimensionality | Dataset Scope |
|---|---|---|---|
| OpenAI | text-embedding-3-small | 1,536 | All three |
| OpenAI | text-embedding-3-large | 3,072 | All three |
| OpenAI | text-embedding-ada-002 | 1,536 | All three |
| Sentence- Transformer | all-MiniLM-L6-v2 | 384 | All three |
| Sentence- Transformer | all-mpnet-base-v2 | 768 | All three |
| Sentence- Transformer | multi-qa-mpnet-base-dot-v1 | 768 | All three |
| Pipeline | Execution order | Question-text operation | Question representation step | KC construction step | Response-state construction step |
|---|---|---|---|---|---|
| P1 | question KC SA | raw question text | embed raw question | average | mirror |
| P2 | question KC SA | raw question text | embed raw question | average | stack antonym |
| P3 | question KC SA | raw question text | embed raw question | sample single question | mirror |
| P4 | question KC SA | raw question text | embed raw question | sample single question | stack antonym |
| P5 | KC question SA | raw question text | embed stuffed KC text | stuff window | mirror |
| P6 | KC question SA | raw question text | embed stuffed KC text | stuff window | stack antonym |
| Dataset | Embedding model | Pipeline | Pearson | Spearman | Top-10% overlap |
|---|---|---|---|---|---|
| ASSISTments 2009 | bert_ft_assist2009 | 5 | 0.0573 | 0.0276 | 0.1151 |
| ASSISTments 2012 | embed_english_v3.0 | 3 | 0.0894 | 0.0782 | 0.1571 |
| ASSISTments 2017 | all_MiniLM_L6_v2 | 5 | 0.0247 | 0.0114 | 0.1404 |
| Dataset | Variant | Embedding | Pipe. | SemOut | Test AUC | AUC | Test Acc. |
| ASSISTments 2009 | |||||||
| 2009 | Baseline DKT | – | – | No | 0.8582 | +0.0000 | 0.7851 |
| 2009 | Pretrained Embedding Initialization | text-embedding-3-small | P1 | No | 0.8550 | -0.0031 | 0.7851 |
| 2009 | Pretrained Embedding Initialization + Regularization | text-embedding-3-small | P1 | No | 0.8567 | -0.0015 | 0.7865 |
| 2009 | Projection Bottleneck | text-embedding-3-large | P4 | No | 0.8552 | -0.0030 | 0.7870 |
| 2009 | Adaptive recurrent capacity | embed-v4.0 | P6 | No | 0.8342 | -0.0240 | 0.7708 |
| Dataset | Mean AUC | Mean Acc. | Positive AUC splits | Largest AUC | Pipeline | Smallest AUC |
|---|---|---|---|---|---|---|
| ASSISTments 2009 | +0.0004 | +0.0000 | 5 / 9 | 40% | P4 | 20% |
| ASSISTments 2012 | +0.0002 | -0.0028 | 6 / 9 | 80% | P3 | 40% |
| ASSISTments 2017 | +0.0096 | +0.0020 | 8 / 9 | 30% | P3 | 20% |
| Dataset | #KCs | Train | Test | Test Acc. | Macro F1 | Weighted F1 |
|---|---|---|---|---|---|---|
| ASSISTments 2009 | 65 | 4297 | 921 | 0.9967 | 0.9665 | 0.9963 |
| ASSISTments 2012 | 130 | 13894 | 2978 | 0.9795 | 0.8434 | 0.9772 |
| ASSISTments 2017 | 51 | 448 | 96 | 0.1354 | 0.1095 | 0.0860 |
| Dataset | Condition | Selected pipeline | Test AUC | AUC | Test Acc. | Acc. |
|---|---|---|---|---|---|---|
| ASSISTments 2009 | Matched baseline | – | 0.8552 | – | 0.7875 | – |
| ASSISTments 2009 | Fine-tuned BERT | P4 | 0.8581 | +0.0029 | 0.7875 | +0.0000 |
| ASSISTments 2012 | Matched baseline | – | 0.7363 | – | 0.7384 | – |
| ASSISTments 2012 | Fine-tuned BERT | P4 | 0.7364 | +0.0001 | 0.7371 | -0.0013 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Term or display | Scope and aggregation unit |
|---|---|
| Configuration | One dataset-specific combination of embedding model, representation pipeline, and architecture variant. |
| RQ1 alignment sweep | The full set of evaluated dataset–embedding-model–representation-pipeline–architecture-variant combinations; coverage varies because not every model–pipeline–architecture combination is present in every dataset. |
| RQ2.1 full-data sweep | Approximately 1,065 distinct evaluated full-data configurations represented across the 21 experiment exports summarized in Table 13 . |
| Displayed embedding-model summary | The five model means reported in Table 14 ; this display does not define which configurations contribute to the pipeline aggregate. |
| Embedding-model mean | Reported mean across each displayed model’s evaluated dataset–pipeline configurations. |
| Pipeline mean | Reported across-dataset aggregate for that pipeline. |
| Pipeline | Selected architecture | Test AUC | Test Acc. |
|---|---|---|---|
| P1 | Pretrained Embedding Initialization + Regularization | 0.8567 | 0.7865 |
| P2 | Semantic Output Layer + Projection Bottleneck | 0.8321 | 0.7702 |
| P3 | Projection Bottleneck | 0.8542 | 0.7845 |
| P4 | Projection Bottleneck | 0.8552 | 0.7870 |
| P5 | Projection Bottleneck | 0.8535 | 0.7847 |
| P6 | Semantic Output Layer + Projection Bottleneck | 0.8349 | 0.7727 |
| Pipeline | Selected architecture | Test AUC | Test Acc. |
|---|---|---|---|
| P1 | Semantic Output Layer | 0.7356 | 0.7376 |
| P2 | Semantic Output Layer + Projection Bottleneck | 0.7358 | 0.7377 |
| P3 | Semantic Output Layer + Projection Bottleneck | 0.7358 | 0.7374 |
| P4 | Semantic Output Layer + Projection Bottleneck | 0.7354 | 0.7376 |
| P5 | Semantic Output Layer + Projection Bottleneck | 0.7353 | 0.7374 |
| P6 | Semantic Output Layer + Projection Bottleneck | 0.7358 | 0.7371 |
| Pipeline | Selected architecture | Test AUC | Test Acc. |
|---|---|---|---|
| P1 | Pretrained Embedding Initialization | 0.7205 | 0.6806 |
| P2 | Semantic Output Layer | 0.7171 | 0.6757 |
| P3 | Pretrained Embedding Initialization | 0.7207 | 0.6811 |
| P4 | Projection Bottleneck | 0.7196 | 0.6808 |
| P5 | Projection Bottleneck | 0.7137 | 0.6774 |
| P6 | Adaptive recurrent capacity | 0.7197 | 0.6803 |
| Embedding model | Pearson | Spearman | Top-10% overlap |
|---|---|---|---|
| bert_ft_assist2012 | 0.0321 | 0.0260 | 0.1304 |
| text-embedding-3-large | 0.0297 | 0.0260 | 0.1268 |
| bert_ft_assist2009 | 0.0217 | 0.0118 | 0.0999 |
| all-MiniLM-L12-v2 | 0.0205 | 0.0178 | 0.1260 |
| embed-v4.0 | 0.0187 | 0.0185 | 0.1229 |
| Pipeline | Pearson | Spearman | Top-10% overlap |
|---|---|---|---|
| 5 | 0.0243 | 0.0233 | 0.1229 |
| 9 | 0.0238 | 0.0223 | 0.1214 |
| 3 | 0.0227 | 0.0191 | 0.1228 |
| 7 | 0.0189 | 0.0165 | 0.1208 |
| 1 | 0.0175 | 0.0142 | 0.1216 |
| 6 | 0.0167 | 0.0237 | 0.1231 |
Knowledge Tracing
Knowledge tracing models how learners’ mastery of skills changes over time from their interactions with educational tasks.
See all 35 papers