MERIT: Memory-Enhanced Retrieval for Interpretable Knowledge Tracing
Authors: Runze Li, Kedi Chen, Guwei Feng, Mo Yu, Jun Wang, Wei Zhang
Organizations: School of Computer Science and Technology, East China Normal University, China · Shanghai Innovation Institute, China · WeChat AI, Tencent, China
Knowledge Tracing (KT) models students' evolving knowledge states to predict future performance, serving as a foundation for personalized education. While traditional deep learning models achieve high accuracy, they often lack interpretability. Large Language Models (LLMs) offer strong reasoning capabilities but struggle with limited context windows and hallucinations. Furthermore, existing LLM-based methods typically require expensive fine-tuning, limiting scalability and adaptability to new data. We propose MERIT (Memory-Enhanced Retrieval for Interpretable Knowledge Tracing), a training-free framework combining frozen LLM reasoning with structured pedagogical memory. Rather than updating parameters, MERIT transforms raw interaction logs into an interpretable memory bank. The framework uses semantic denoising to categorize students into latent cognitive schemas and constructs a paradigm bank where representative error patterns are analyzed offline to generate explicit Chain-of-Thought (CoT) rationales. During inference, a hierarchical routing mechanism retrieves relevant contexts, while a logic-augmented module applies semantic constraints to calibrate predictions. By grounding the LLM in interpretable memory, MERIT achieves state-of-the-art performance on real-world datasets without gradient updates. This approach reduces computational costs and supports dynamic knowledge updates, improving the accessibility and transparency of educational diagnosis.
Figures & tables
Figure 1. Overview of the MERIT framework. The architecture integrates a frozen LLM with an external cognitive memory across four stages. Cognitive Schema Discovery first discretizes the student space using semantic denoising and density-based clustering. Interpretative Memory Bank Construction then transforms expert reasoning traces into a static retrieval database. Subsequently, Hierarchical Cognitive Retrieval applies global centroid routing and local hybrid search to maintain domain consistency. Finally, Logic-Augmented Reasoning incorporates semantic difficulty calibration and explicit boundary constraints to regulate the prediction.
Dataset
ASSIST2009
ASSIST2012
Eedi
BePKT
# Responses
0.4m
2.7m
17.8m
23.9k
# Sequences
8.3k
67.1k
475.4k
1.9k
# Questions
6.9k
53.1k
2.7k
0.5k
# Concepts
200
265
386
100
Table 1. Statistics of the preprocessed datasets.
Method
Category
ASSIST09
ASSIST12
Eedi
BePKT
AUC
ACC
F1
AUC
ACC
F1
AUC
ACC
F1
AUC
ACC
F1
DKT
DL-based
0.7485
0.7315
0.7985
0.7512
0.7385
0.7993
0.7286
0.7065
0.8061
0.6985
0.6852
0.5647
AKT
DL-based
0.7684
0.7185
0.7925
0.7657
0.7374
0.7925
0.7388
0.7132
0.7994
0.6975
0.6845
0.5591
SAKT
DL-based
0.7582
0.7365
0.8005
0.7616
0.7361
0.8027
0.7411
0.7165
0.8082
0.7042
0.6904
0.5712
LPKT
DL-based
0.7533
0.7317
0.8038
0.7485
0.7248
0.8044
0.7457
0.7222
0.8106
0.6958
0.6807
0.5730
IKT
DL-based
0.7439
0.7194
0.7937
0.7583
0.7386
0.7795
0.7338
0.7135
0.7859
0.6885
0.6730
0.5614
Table 2. Overall performance comparison on ASSIST09, ASSIST12, Eedi, and BePKT. The best results are bolded , and the second best are underlined . Methods marked with * are training-free.
Method
ASSIST09
ASSIST12
Eedi
BePKT
AUC
ACC
F1
AUC
ACC
F1
AUC
ACC
F1
AUC
ACC
F1
Base (w/o Retrieval)
0.6708
0.4794
0.3131
0.6749
0.4383
0.3601
0.7333
0.6855
0.7458
0.7392
0.6875
0.2038
w/o Global Routing
0.6715
0.4782
0.3297
0.6804
0.4500
0.3838
0.7312
0.6815
0.7433
0.7461
0.6925
0.2166
w/o Interpretative Traces
0.6709
0.4879
0.3482
0.6792
0.4487
0.3822
0.7295
0.6850
0.7460
0.7573
0.6800
0.1688
w/o Logic Constraints
0.7015
0.5012
0.3622
0.6887
0.4613
0.4035
0.7329
0.6935
0.7675
0.7617
0.6950
0.2278
MERIT Gemini-2.5-Flash
0.8244
0.7554
0.8054
0.7778
0.7441
0.8122
0.7969
0.7603
0.8139
0.7827
0.7850
0.6228
Table 3. Comprehensive ablation study results. The shaded rows denote the full MERIT framework, distinguishing it from the baseline (w/o Retrieval) and ablated variants.
Figure 2. Parameter sensitivity analysis on ASSISTments 2009 using Gemini-2.5-Flash. (Left) Impact of retrieval volume k . Performance peaks at k=3 , suggesting that a compact context window limits noise from irrelevant samples. (Right) Sensitivity to hybrid search weight α . A weight of 0.7 optimally balances latent semantic features with symbolic keyword matching.
Figure 3. UMAP visualization of cognitive schema discovery. (a) Embeddings from raw logs overlap significantly, suggesting statistical artifacts (e.g., IDs, scores) obscure underlying patterns. (b) Conversely, Semantic Denoising yields distinct, compact clusters, confirming effective separation of cognitive signals from noise.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Description
Problem Definition
U,E
Set of students and set of exercises, respectively.
Su
Interaction sequence for student u : Su={s1,s2,…,st} .
st=(et,rt,dt)
Interaction tuple at step t , consisting of exercise content et , response rt , and difficulty dt .
Concept: Median Difficulty: 1.0 (Max) Risk: Extreme Difficulty Spike .
Baseline Error
DKT predicts High Probability (0.85) due to strong positive momentum bias.
Phase 2: MERIT Evidence Retrieval (The Reality Check)
Retrieved Paradigm
Source: Memory Bank Paradigm #5 (Cluster #5) Historical Context: "Student had perfect streak on trivial items: Stem/Leaf Plot ( d=0.0 ), Median ( d=0.0 )…" Outcome: False on next question (Venn Diagram, d=1.0 ). Retrieved Analysis: ↪ Key Pattern: "Difficulty Spike Failure" ↪ Reasoning: "The student’s perfect streak on trivial (0.0 difficulty) questions created a misleading sense of momentum . The failure was caused by an extreme and simultaneous spike in concept difficulty (0.0 → 1.0)."
Appendix
Table 5. Detailed execution trace for the case study (Student ID: 712). The table contrasts the surface-level observation that misled the baseline model with the deep, memory-augmented reasoning process of MERIT. By retrieving a structurally similar paradigm (Paradigm #5), MERIT identifies the "Difficulty Spike" pattern and corrects the prediction.
Knowledge Tracing (KT) models students' knowledge states based on learning interactions to predict performance. While deep learning-based KT models have boosted predictive accuracy, most models rely on deterministic vector embeddings and opaque latent state transitions, limiting interpretability regarding how specific past behaviors influence predictions. To address this limitation, we propose Probabilistic Logical Knowledge Tracing (PLKT), an interpretable KT framework that formulates prediction as a goal-conditioned evidence reasoning process over historical learning behaviors. Instead of representing knowledge states as deterministic vector embeddings, PLKT employs robust Beta-distributed probabilistic embeddings to represent student knowledge states. This probabilistic foundation allows us to model the uncertainty of historical behaviors and perform explicit logical operations (e.g., conjunction), constructing transparent reasoning paths that reveal how specific past interactions contribute to the prediction. Extensive experiments show that PLKT outperforms state-of-the-art KT methods while achieving superior interpretability. Our code is available at https://anonymous.4open.science/r/PLKT-D3CE/.
Siyu Wu, Cong Xu, Wei Zhang
1Shanghai Institute of AI Education, East China Normal University, Shanghai, 200241, China · 2Shool of Computer Sicence and Technology, East China Normal University, Shanghai, 200241, China
Knowledge Tracing (KT) infers a student's knowledge state from past interactions to predict future performance. Conventional Deep Learning (DL)-based KT models are typically tied to platform-specific identifiers and latent representations, making them hard to transfer and interpret. Large Language Model (LLM)-based methods can be either ungrounded under prompting or overly domain-dependent under fine-tuning. In addition, most existing KT methods are developed and evaluated under a same-distribution assumption. In real deployments, educational data often arise from heterogeneous platforms with substantial distribution shift, which often degrades generalization. To this end, we propose RAG-KT, a retrieval-augmented paradigm that frames cross-platform KT as reliable context constrained inference with LLMs. It builds a unified multi-source structured context with cross-source alignment via Question Group abstractions and retrieves complementary rich and reliable context for each prediction, enabling grounded prediction and interpretable diagnosis. Experiments on three public KT benchmarks demonstrate consistent gains in accuracy and robustness, including strong performance under cross-platform conditions.
Recent advances in large language models (LLMs) have led to the development of AI-powered tutoring systems that provide interactive support via dialogue. To enable these tutoring systems to provide personalized support, it is essential to assess student performance at each turn, motivating knowledge tracing (KT) in dialogue settings. However, existing dialogue-based KT approaches often ignore question difficulty modeling and rely on opaque latent representations from LLMs, hindering accurate and interpretable prediction. In this work, we propose an interpretable difficulty-aware conversational KT framework built upon LLMs, which explicitly models students' abilities and the difficulty of tutor-posed tasks at each turn. The framework incorporates the original textual question and the next tutor-posed task to estimate the student's knowledge state and the difficulty of the upcoming turn. Furthermore, it integrates Item Response Theory to map LLM's outputs into student ability and question difficulty parameters, enabling interpretable prediction of student performance grounded in cognitive theories of learning. We evaluate the framework on two tutor-student dialogue datasets. Both quantitative and qualitative results show that our framework outperforms existing KT baselines, meanwhile generating interpretable outputs consistent with cognitive theory.