Can a System-One LLM Perform Knowledge Tracing When Few or No Learners Are Logged?

Unggi Lee; Haeun Park

cs.CLOct 8, 2026

Can a System-One LLM Perform Knowledge Tracing When Few or No Learners Are Logged?

Authors: Unggi Lee, Haeun Park

Organizations: Korea University Sejong Campus · Korea University

Abstract

Knowledge tracing (KT) models need many logged learners, so a new course or platform starts without a usable model. In LLM-based KT the LLM generates the answer, which we call System-Two; it is either fine-tuned on the target data or reasons and votes over ten samples, which is slow and gives coarse probabilities. We ask whether an off-the-shelf System-One LLM, which returns a probability for a typed question directly in a single pass, can perform KT when few or no learners are logged. On seven datasets, Jev without any data from the target platform reaches a mean AUC of .706, above the best of 28 deep KT models trained on 8 learners (.689) and above System-Two Thinking-KT on all seven datasets (.650) at about 1/100 of its API cost. Adding examples and a similar-learner statistic from the logged learners (JevKT) raises this to .722; JevKT stays significantly ahead of deep KT up to 16 learners and ahead on average up to 64, and supervised KT catches up between 64 and 128 learners. Among the readers we tested, the gain is specific to Jev, since three other LLMs queried with the byte-identical typed request through the official System-One adapter fall below it on all seven datasets, and reader swaps and contamination checks find no evidence that the input format or memorised data explain the gain. For new learners the advantage holds from their first interactions, whereas on unseen items with all learners logged, deep KT remains ahead.

Figures & tables

Panel of Figure 1 — Figure 1: Three ways to predict whether a learner answers the next item correctly. (a) Deep KT is trained on the logs of many learners. (b) System-Two LLM-KT samples ten reasoning traces and reads a probability from vote frequencies. (c) This study poses the prediction to an off-the-shelf System-One model (Jev) as one typed question about a serialised learner state, either alone (Jev 0-shot ) or with 64 KC-matched examples and a similar-learner statistic (JevKT = Jev + 64-shot + kNN); the probability is read from a single pass, without gradient training on the target dataset.

Panel of Figure 2 — Figure 2: How JevKT was selected: mean $\Delta$ AUC over the state-only input (7 datasets) vs. number of shots $k$ . Filled: development split (used for selection); hollow: test (not used for selection). Other inputs: Appendix C .

$N$	Method	ASSIST09	NIPS34	XES3G5M	DBE-KT22	Alg05	Junyi	ASSIST17	Avg	Wins
	Best deep KT †	–	–	–	–	–	–	–	–
	Best LLM-based KT ‡	.715	.662	.699	.688	.656	.671	.608	.671
0	Jev 0-shot	.738	.691	.725	.742	.698	.700	.648	.706	7/7
	Best deep KT †	.700	.653	.689	.793	.711	.652	.624	.689
	Best LLM-based KT ‡	.707	.662	.699 ∘	.699	.656 ∘	.671 ∘	.608 ∘	.672
8	JevKT	.735	.707	.720	.821	.734	.701	.638	.722	7/7

Table 1: Cold-start AUC (P1) of Jev against the best deep KT and the best LLM-based KT baseline with

N

training learners (mean over 5 seeds). Without training data, Jev 0-shot is above LLM-based KT on all seven datasets and above deep KT trained on 8 learners on average; JevKT stays ahead of deep KT up to

N{=}64

on average. Wins : datasets on which Jev beats every baseline in its block. † Best of 28 pyKT models, selected on test. ‡ LOKT, the strongest LLM-based KT baseline. ∘

N{=}0

value repeated (no answer options). § Reported by LLM-KT ( Wang et al., 2026 ) on its own split; not ranked. All models: Appendix M . Best in bold , second best underlined .

Panel of Figure 3 — Figure 3: Cold start for new learners and new items (mean AUC over 7 datasets; few training learners, P1, is in Table 1 ). (a) P2: AUC by the position of the target in a new learner’s sequence with $N{=}8$ training learners; Jev 0-shot uses no training learners. (b) P3: targets on items held out of training. All methods are scored on identical targets; per-dataset results in Appendix D . † Best of 28 pyKT models selected on test per cell, which favours DLKT.

Panel of Figure 4 — Figure 4: Model or format? On the same Qwen3.5-9B weights, a typed one-pass head (B0) is worse than the plain readout (A) and Jev distillation (C) does not beat A, both with full data and in cold start; on the identical state, Jev is the stronger reader (35/35 cells).

Reader	$N{=}0$	$N{=}8$	Below Jev
Jev	.706	.717	–
Adapter + GPT-4o-mini	.680	–	7/7
Adapter + Gemini-2.5-FL	.670	–	7/7
Adapter + DeepSeek-V4-F	.626	.634	7/7
JEV-9B (Jev-distilled)	.687	.699	6/7
Laya (open-source Jev)	.536	.553	7/7

Table 2: The same typed request read by other models (mean AUC over 7 datasets). Below Jev : datasets on which the reader is below Jev 0-shot at

N{=}0

N{=}8

rows of Jev and the adapter use a 1,000-target subsample (budget). † Trained on the 8 learners. Per dataset: Table 12 . Best in bold , second best underlined .

	Think $-$ No-think		Jev $-$ Thinking-KT
Dataset	budget 1024	budget 2048	(budget 2048)
ASSIST09	$-$ .013	$-$ .015	+.038 ∗
NIPS34	+.014	+.005	+.038 ∗
XES3G5M	$-$ .020	$-$ .007	+.072 ∗
DBE-KT22	$-$ .016	$-$ .026	+.073 ∗
Algebra05	+.023	+.036	+.053 ∗

Table 3: Reasoning does not help Qwen3.5-9B on KT; a single System-One pass does better.

\Delta

AUC (Holm-corrected paired bootstrap;

{}^{*}p{<}.05

). Think

-

No-think : Thinking-KT with reasoning budget 1024 / 2048 minus the same model without reasoning. Jev

-

Thinking-KT : one Jev call vs. 10 sampled reasoning generations. Cost : per million predictions of Thinking-KT at each budget (no-think: $707) and of one Jev call.

Panel of Figure 5 — Figure 5: Reliability of the predicted probabilities (7 datasets pooled, 10 equal-mass bins for display; ECE in Table 13 uses 15; diagonal = perfect calibration). Vote frequencies of System-Two LLM-KT pile up at 0 and 1, while the single-pass readout of Jev follows the diagonal with a slight underconfidence. Per-dataset ECE and Brier: Table 13 .

Method	Training	Ready after	Train $	Latency	Pred/s	$/1M
JevKT	✗	0 / 0	–	325 ms	72.23	$886
Jev 0-shot	✗	0 / 0	–	200 ms	105	$44
LOKT	✗	0 / 0	–	1.3 s	3.39	$707
Thinking-KT	✗	0 / 0	–	41.7 s	0.12	$4,360
JEV-9B (vLLM)	✗	0 / 0	– ‡	204 ms	8.27	$7.39
CLST	✓	110.8 s / –	$<$ $0.01 / –	104 ms	9.6	$6.37

Table 4: Efficiency. Training : gradient training on target data is needed. Ready after and Train

: time and cost until the first prediction for a new course with

N{=}8

/ all learners. Latency (median, one prediction), throughput, and cost per million predictions. API methods: OpenRouter prices. DLKT, CLST and JEV-9B run on one RTX 3090, priced at the RunPod Community Cloud on-demand rate (

0.22/h, September 2026), which favours them. LOKT and Thinking-KT (budget 2048) use 10 votes; CLST fine-tunes Mistral-7B with LoRA. ‡ Distilled once from Jev, not per course.

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Learners	Interactions	Items	KCs	Base rate
ASSIST2009	3,002	277k	17,705	123	.63
NIPS34	4,918	1.38M	948	57	.54
XES3G5M	5,000 ∗	1.54M	7,374	847	.79
DBE-KT22	1,138	158k	212	93	.78
Algebra2005	567	607k	173k	112	.74
Junyi	5,000 ∗	1.33M	24,683	1,326	.71

Appendix

Table 5: Datasets. Learners with at least 10 interactions; base rate = share of correct answers among test targets.

Input to Jev	A09	NIPS	XES	DBE	Alg05	Junyi	A17	Mean
Jev 0-shot (no evidence, no shots)	.738	.691	.725	.742	.698	.700	.648	.706
+ item/KC statistics	.741	.708	.729	.786	.723	.704	.660	.721
+ similar-learner kNN	.738	.733	.730	.816	.733	.702	.649	.729
+ 16 KC-matched shots	.739	.709	.729	.797	.723	.702	.641	.720
+ 64 KC-matched shots	.739	.724	.720	.813	.727	.703	.643	.724
+ 64 shots + kNN (JevKT)	.737	.734	.720	.834	.738	.704	.646	.730

Appendix

Table 6: Input ablation under cold start (

N{=}16

training learners, mean over 5 seeds, same early-window targets as P1). Each row adds one input to the learner’s history; Jev 0-shot uses no training learners. Best in bold , second best underlined .

Input added to the state	dev	test
+ kNN, + 64 KC-matched shots (JevKT)	+.062	+.066
+ kNN, + 32 KC-matched shots	+.060	+.067
+ statistics + kNN, + 16 KC-matched shots	+.055	+.062
+ kNN	+.050	+.051
$H{=}50$ , + statistics + kNN	+.046	+.049
+ statistics + kNN	+.046	+.048

Appendix

Table 7: Input-design exploration for Jev. Mean

\Delta

AUC over 7 datasets relative to the state only (AUC dev .703 / test .710). The configuration was selected on the development split by a rule fixed in advance; test values are shown for reference and were not used for selection. 18 runs with LOKT option weights (TCOW) did not change AUC and are omitted. Best in bold , second best underlined .

Panel of Figure 6 — Figure 6: P1 per dataset (first panel: mean). AUC as a function of the number of training learners $N$ (identical early-window targets, mean over 5 seeds); the shaded “all” point uses the full training data on the same targets. With few learners, training-free JevKT outperforms the best of 28 DLKT models and stays at or above Jev 0-shot (dashed); supervised KT catches up between $N{=}64$ and $128$ on average and leads with full data. † Best of 28 pyKT models selected on test per cell, which favours DLKT.

Panel of Figure 7 — Figure 7: The cold-start advantage of training-free JevKT (Jev + 64-shot + kNN) shrinks as more learners are logged. $\Delta$ AUC against the best of 28 pyKT models (selected on test) on identical early-window targets; thin lines are datasets, the thick line is the mean over datasets with all 7 cells available.

Dataset	$N$	CLST	Jev	Best DLKT †	Paper
ASSIST09	0	.728	.738	–	–
	8	.658	.735	.700	.689
	16	.682	.737	.706	–
	32	.686	.737	.724	–
	64	.735	.745	.734	–
NIPS34	0	.623	.691	–	–

Appendix

Table 8: CLST (our re-implementation, fine-tuned Mistral-7B with LoRA) on the three datasets of the original paper, on our early-window targets (mean over 5 seeds;

N{=}0

: one run). Jev = JevKT (

N{\geq}8

) or Jev 0-shot (

N{=}0

). Paper : CLST’s reported AUC at

N{=}8

on its own splits. † Best of the 11 core pyKT models. Best in bold , second best underlined .

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
Position 1
JevKT	.709 $\pm$ .059	.644 $\pm$ .047	.512 $\pm$ .057	–	.799 $\pm$ .022	.517 $\pm$ .024	.660 $\pm$ .066	.640
Jev 0-shot ( $N{=}0$ )	.689 $\pm$ .103	.492 $\pm$ .094	.533 $\pm$ .102	–	.650 $\pm$ .086	.466 $\pm$ .119	.654 $\pm$ .090	.580
Best deep KT †	.749 $\pm$ .049	.558 $\pm$ .056	.542 $\pm$ .049	–	.663 $\pm$ .016	.510 $\pm$ .030	.607 $\pm$ .080	.605
Positions 2–5
JevKT	.771 $\pm$ .008	.729 $\pm$ .041	.666 $\pm$ .006	.768 $\pm$ .042	.714 $\pm$ .013	.657 $\pm$ .007	.689 $\pm$ .032	.714

Appendix

Table 9: P2 (new learners), every cell. AUC by the position of the target in a new learner’s sequence with

N{=}8

training learners (Jev 0-shot uses none), mean

\pm

SD over 5 seeds (Jev 0-shot : learner-bootstrap SD of its single run); same predictions as P1. Position 1 has only 41–50 targets per dataset, so its AUC is noisy; – = one class only. † Best of 28 pyKT models, chosen on the whole

(N,\text{dataset})

cell. Best in bold , second best underlined .

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
AUC
JevKT	.728 $\pm$ .029	.725 $\pm$ .014	.707 $\pm$ .021	.687 $\pm$ .047	.775 $\pm$ .015	.731 $\pm$ .039	.639 $\pm$ .012	.713
Jev 0-shot	.706 $\pm$ .034	.717 $\pm$ .018	.718 $\pm$ .016	.705 $\pm$ .048	.737 $\pm$ .018	.714 $\pm$ .034	.629 $\pm$ .013	.704
LOKT ‡	.688 $\pm$ .036	.683 $\pm$ .027	.692 $\pm$ .015	.682 $\pm$ .054	.649 $\pm$ .030	.653 $\pm$ .020	.599 $\pm$ .012	.664
Thinking-KT	.670 $\pm$ .026	.667 $\pm$ .029	.651 $\pm$ .007	.669 $\pm$ .034	.689 $\pm$ .009	.662 $\pm$ .037	.593 $\pm$ .012	.657
Best deep KT †	.746 $\pm$ .019	.739 $\pm$ .015	.725 $\pm$ .003	.697 $\pm$ .014	.806 $\pm$ .002	.728 $\pm$ .026	.665 $\pm$ .016	.729

Appendix

Table 10: P3 (new items), every cell. Targets whose item was held out of training and of all examples and statistics (20% of items, 1,000 targets per seed, all training learners); mean

\pm

SD over 3 item-split seeds. † Best of the 11 core pyKT models selected on test AUC per dataset (bottom row). ‡ LOKT uses option weights from the training learners where options exist (ASSIST09, NIPS34, DBE-KT22) and no training data elsewhere. Best in bold , second best underlined .

		Full data		Cold start
	Weights / readout	$H{=}25$	+ evidence	$N{=}8$
A	Qwen3.5-9B, plain label readout	.688	.752	.700
B0	Qwen3.5-9B, typed one-pass head (no training)	.630	.701	.659
C	JEV-9B (Jev-distilled Qwen3.5-9B)	.679	.741	.699
Reader swap (identical serialised state)
	Qwen3.5-9B reads the same 64-shot + kNN state	.689	.727	.699

Appendix

Table 11: Model or format? Mean AUC over 7 datasets. Rows A–C share the Qwen3.5-9B weights; the reader swap feeds the identical serialised state to Qwen3.5-9B or Jev. Full data: state with the last 25 interactions only, or with evidence. Cold start:

N{=}8

training learners (A–C: kNN evidence, mean of 5 seeds; swap: 64-shot + kNN state, seed 0). Best in bold , second best underlined .

Reader	A09	NIPS	XES	DBE	Alg05	Junyi	A17	Mean
$N{=}0$ : no learners logged
Jev 0-shot	.738	.691	.725	.742	.698	.700	.648	.706
Adapter + GPT-4o-mini	.716	.669	.691	.699	.674	.685	.627	.680
Adapter + Gemini-2.5-Flash-Lite	.711	.664	.669	.710	.645	.669	.620	.670
Adapter + DeepSeek-V4-Flash	.671	.610	.601	.619	.612	.666	.604	.626
JEV-9B (Jev-distilled)	.741	.677	.695	.705	.691	.690	.611	.687

Appendix

Table 12: Is the result specific to Jev? AUC on the cold-start targets (P1). Adapter rows send the byte-identical typed request to other LLMs through TypeSafe’s official System-One adapter; JEV-9B is Qwen3.5-9B distilled from Jev; Laya is the open-source Jev. ‡ 1,000-target subsample and seeds 0–2 (budget), with Jev on the same targets and seeds; other

N{=}8

rows use all targets and 5 seeds. † Best of 28 pyKT models selected on test. Best in bold , second best underlined .

Method	A09	NIPS	XES	DBE	Alg05	Junyi	A17	ECE	Brier
Thinking-KT (System Two, $N{=}0$ )	.261	.266	.257	.199	.232	.251	.303	.253	.282
LOKT ( $N{=}0$ )	.249	.270	.175	.188	.179	.155	.244	.209	.249
Jev 0-shot ( $N{=}0$ )	.092	.048	.094	.113	.107	.098	.037	.084	.187
Best DLKT † ( $N{=}8$ )	.060	.071	.064	.083	.060	.111	.086	.076	.192
JevKT ( $N{=}8$ )	.079	.061	.067	.078	.076	.077	.054	.070	.182

Appendix

Table 13: Calibration on the cold-start targets: expected calibration error (ECE, 15 equal-mass bins) per dataset, and mean ECE and Brier score over the 7 datasets (lower is better).

N{=}8

rows are averaged over 5 seeds. † Best of the pyKT models selected on test AUC. Best in bold , second best underlined .

Targets	A09	NIPS	XES	DBE	Alg05	Junyi	A17	Mean	Wins
Item answered by the $N$ learners	+.024	+.056	+.032	+.027	+.028	+.066	+.017	+.036	7/7
Item not answered by them	+.035	–	+.028	–	+.018	+.050	+.009	+.028	5/5
Learner accuracy so far $<.5$	+.047	+.096	+.036	–	+.036	+.067	+.017	+.050	6/6
Learner accuracy $.5$ – $.8$	+.024	+.059	+.033	$-$ .001	+.014	+.037	+.033	+.029	6/7
Learner accuracy $\geq.8$	+.071	+.127	+.051	+.059	+.026	+.045	–	+.063	6/6

Appendix

Table 14: Where does JevKT gain over the best deep KT model at

N{=}8

\Delta

AUC (JevKT

-

best DLKT, mean over 5 seeds) on subsets of the cold-start targets; slices with fewer than 100 targets are omitted (–). Wins : datasets with

\Delta{>}0

$N$	$\Delta$ AUC	Holm-sig.
8	+.033	7/7
16	+.023	6/7
32	+.016	3/7
64	+.004	2/7
128	$-$ .003	1/7
256	$-$ .008	1/7

Appendix

Table 15: JevKT

-

best DLKT under cold start (P1): mean

\Delta

AUC over 7 datasets and the number of datasets on which JevKT is significantly better (one-sided paired learner-cluster bootstrap, 1,000 resamples, Holm over datasets,

p{<}.05

). The best DLKT model is chosen on test per cell (optimistic for DLKT), among pyKT models with at least three finished seeds.

Method	$N{=}0$	$N{=}8$	$N{=}16$	$N{=}32$	$N{=}64$	$N{=}128$	$N{=}256$	all
JevKT (Jev 0-shot at $N{=}0$ )	.706 [.695, .717]	.722 [.712, .731]	.730 [.720, .740]	.739 [.728, .748]	.747 [.737, .757]	.753 [.744, .763]	.758 [.749, .768]	.770 [.760, .780]
Best deep KT †	–	.689 [.679, .698]	.707 [.697, .717]	.723 [.713, .733]	.743 [.733, .752]	.756 [.747, .765]	.766 [.758, .775]	.798 [.790, .807]
Tuned deep KT ¶	–	.652 [.643, .660]	.688 [.679, .698]	.709 [.699, .718]	.737 [.728, .747]	–	–	–
BKT	–	.632 [.622, .641]	.647 [.638, .657]	.660 [.649, .670]	.675 [.665, .685]	.685 [.675, .695]	.693 [.682, .703]	.705 [.694, .716]
PFA	–	.593 [.584, .601]	.620 [.611, .629]	.647 [.638, .657]	.665 [.655, .675]	.677 [.668, .688]	.686 [.675, .696]	.697 [.687, .708]
LOKT	.671 [.659, .684]	–	–	–	–	–	–	–

Appendix

Table 16: P1: mean AUC over the 7 datasets with its 95% confidence interval (learner-cluster bootstrap,

B{=}1{,}000

; seeds averaged inside each resample, datasets resampled independently). The interval covers the sampling of targets and learners at fixed seeds. † Best of 28 pyKT models selected on test per cell. ¶ Best of four pyKT models

\times

eight configurations chosen on validation learners (

N{\leq}64

). LOKT and Thinking-KT use no data from the

N

learners and are shown at

N{=}0

. Best in bold , second best underlined .

	ID probes	De-identification AUC
Dataset	flags	D0	D1	D2	D3	total $\Delta$	Holm $p$
ASSIST09	none	0.736	0.740	0.737	–	+0.002	1.000
NIPS34	none	0.686	0.682	0.686	–	+0.001	1.000
XES3G5M	none	0.717	0.712	0.704	0.706	$-$ 0.010	0.920
DBE-KT22	none	0.719	0.714	0.712	0.687	$-$ 0.033	0.084
Algebra05	none	0.740	0.729	0.728	–	$-$ 0.011	0.282

Appendix

Table 17: Contamination checks for Jev. ID probes : flags from the ID-only difficulty and skill probes and from row continuation (Holm-corrected, four LLMs). De-identification : Jev 0-shot AUC on the original data (D0), with IDs relabelled (D1), names removed (D2) and item text removed (D3, where text exists); total

\Delta

= D0

\rightarrow

last level, one-sided Holm

p

for a drop.

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
Jev 0-shot	.738 $\pm$ .014	.691 $\pm$ .014	.725 $\pm$ .013	.742 $\pm$ .018	.698 $\pm$ .018	.700 $\pm$ .015	.648 $\pm$ .013	.706 $\pm$ .006
Laya (open-source Jev)	.528 $\pm$ .014	.545 $\pm$ .014	.522 $\pm$ .016	.645 $\pm$ .018	.524 $\pm$ .016	.532 $\pm$ .015	.458 $\pm$ .013	.536 $\pm$ .006
System-Two LLM-KT, prompted
Thinking-KT	.700 $\pm$ .014	.641 $\pm$ .014	.663 $\pm$ .014	.666 $\pm$ .019	.663 $\pm$ .018	.642 $\pm$ .016	.576 $\pm$ .013	.650 $\pm$ .006
LOKT	.715 $\pm$ .014	.662 $\pm$ .014	.699 $\pm$ .014	.688 $\pm$ .021	.656 $\pm$ .019	.671 $\pm$ .016	.608 $\pm$ .013	.671 $\pm$ .006

Appendix

Table 18: P1 with

N{=}0

training learners: AUC of every model, a single deterministic run (

\pm

= learner-bootstrap SD), on the 2,000 shared early targets per dataset. ‡ Three datasets of the original CLST paper. (k) = only

k

seeds finished. Mean = mean over the 7 datasets (– if a dataset is missing). qDKT and QIKT do not fit Algebra05 in 24 GB of GPU memory (–). Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.735 $\pm$ .003	.707 $\pm$ .022	.720 $\pm$ .006	.821 $\pm$ .003	.734 $\pm$ .003	.701 $\pm$ .003	.638 $\pm$ .013	.722 $\pm$ .004
Jev 0-shot	.738 $\pm$ .014 ∘	.691 $\pm$ .014 ∘	.725 $\pm$ .013 ∘	.742 $\pm$ .018 ∘	.698 $\pm$ .018 ∘	.700 $\pm$ .015 ∘	.648 $\pm$ .013 ∘	.706 $\pm$ .006 ∘
Laya (open-source Jev)	.588 $\pm$ .009	.570 $\pm$ .006	.515 $\pm$ .005	.645 $\pm$ .025	.530 $\pm$ .008	.497 $\pm$ .002	.526 $\pm$ .004	.553 $\pm$ .005
System-Two LLM-KT, prompted
Thinking-KT	.700 $\pm$ .014 ∘	.641 $\pm$ .014 ∘	.663 $\pm$ .014 ∘	.666 $\pm$ .019 ∘	.663 $\pm$ .018 ∘	.642 $\pm$ .016 ∘	.576 $\pm$ .013 ∘	.650 $\pm$ .006 ∘

Appendix

Table 19: P1 with

N{=}8

training learners: AUC of every model, mean

\pm

SD over 5 learner-subset seeds, on the 2,000 shared early targets per dataset. ‡ Three datasets of the original CLST paper. (k) = only

k

seeds finished. ¶ Best of four pyKT models

\times

eight hyper-parameter configurations chosen on the validation learners of each cell (no test information). Mean = mean over the 7 datasets (– if a dataset is missing). qDKT and QIKT do not fit Algebra05 in 24 GB of GPU memory (–). ∘ The method uses no data from the

N

learners here (Jev 0-shot and Thinking-KT everywhere; LOKT on datasets without answer options), so its

N{=}0

run is repeated. Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.737 $\pm$ .009	.734 $\pm$ .013	.720 $\pm$ .006	.834 $\pm$ .008	.738 $\pm$ .004	.704 $\pm$ .005	.646 $\pm$ .005	.730 $\pm$ .004
Jev 0-shot	.738 $\pm$ .014 ∘	.691 $\pm$ .014 ∘	.725 $\pm$ .013 ∘	.742 $\pm$ .018 ∘	.698 $\pm$ .018 ∘	.700 $\pm$ .015 ∘	.648 $\pm$ .013 ∘	.706 $\pm$ .006 ∘
Laya (open-source Jev)	.602 $\pm$ .001	.557 $\pm$ .008	.525 $\pm$ .010	.635 $\pm$ .013	.528 $\pm$ .005	.496 $\pm$ .003	.526 $\pm$ .008	.553 $\pm$ .002
System-Two LLM-KT, prompted
Thinking-KT	.700 $\pm$ .014 ∘	.641 $\pm$ .014 ∘	.663 $\pm$ .014 ∘	.666 $\pm$ .019 ∘	.663 $\pm$ .018 ∘	.642 $\pm$ .016 ∘	.576 $\pm$ .013 ∘	.650 $\pm$ .006 ∘

Appendix

Table 20: P1 with

N{=}16

training learners: AUC of every model, mean

\pm

SD over 5 learner-subset seeds, on the 2,000 shared early targets per dataset. ‡ Three datasets of the original CLST paper. (k) = only

k

seeds finished. ¶ Best of four pyKT models

\times

N

learners here (Jev 0-shot and Thinking-KT everywhere; LOKT on datasets without answer options), so its

N{=}0

run is repeated. Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.737 $\pm$ .006	.752 $\pm$ .005	.728 $\pm$ .007	.838 $\pm$ .004	.751 $\pm$ .002	.709 $\pm$ .003	.656 $\pm$ .009	.739 $\pm$ .002
Jev 0-shot	.738 $\pm$ .014 ∘	.691 $\pm$ .014 ∘	.725 $\pm$ .013 ∘	.742 $\pm$ .018 ∘	.698 $\pm$ .018 ∘	.700 $\pm$ .015 ∘	.648 $\pm$ .013 ∘	.706 $\pm$ .006 ∘
Laya (open-source Jev)	.602 $\pm$ .004	.558 $\pm$ .006	.522 $\pm$ .007	.629 $\pm$ .020	.523 $\pm$ .005	.498 $\pm$ .005	.517 $\pm$ .002	.550 $\pm$ .003
System-Two LLM-KT, prompted
Thinking-KT	.700 $\pm$ .014 ∘	.641 $\pm$ .014 ∘	.663 $\pm$ .014 ∘	.666 $\pm$ .019 ∘	.663 $\pm$ .018 ∘	.642 $\pm$ .016 ∘	.576 $\pm$ .013 ∘	.650 $\pm$ .006 ∘

Appendix

Table 21: P1 with

N{=}32

training learners: AUC of every model, mean

\pm

SD over 5 learner-subset seeds, on the 2,000 shared early targets per dataset. ‡ Three datasets of the original CLST paper. (k) = only

k

seeds finished. ¶ Best of four pyKT models

\times

N

learners here (Jev 0-shot and Thinking-KT everywhere; LOKT on datasets without answer options), so its

N{=}0

run is repeated. Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.745 $\pm$ .005	.762 $\pm$ .006	.739 $\pm$ .007	.838 $\pm$ .002	.762 $\pm$ .004	.715 $\pm$ .005	.667 $\pm$ .009	.747 $\pm$ .002
Jev 0-shot	.738 $\pm$ .014 ∘	.691 $\pm$ .014 ∘	.725 $\pm$ .013 ∘	.742 $\pm$ .018 ∘	.698 $\pm$ .018 ∘	.700 $\pm$ .015 ∘	.648 $\pm$ .013 ∘	.706 $\pm$ .006 ∘
Laya (open-source Jev)	.600 $\pm$ .003	.571 $\pm$ .004	.515 $\pm$ .008	.607 $\pm$ .007	.533 $\pm$ .004	.509 $\pm$ .004	.506 $\pm$ .006	.549 $\pm$ .002
System-Two LLM-KT, prompted
Thinking-KT	.700 $\pm$ .014 ∘	.641 $\pm$ .014 ∘	.663 $\pm$ .014 ∘	.666 $\pm$ .019 ∘	.663 $\pm$ .018 ∘	.642 $\pm$ .016 ∘	.576 $\pm$ .013 ∘	.650 $\pm$ .006 ∘

Appendix

Table 22: P1 with

N{=}64

training learners: AUC of every model, mean

\pm

SD over 5 learner-subset seeds, on the 2,000 shared early targets per dataset. ‡ Three datasets of the original CLST paper. (k) = only

k

seeds finished. ¶ Best of four pyKT models

\times

N

learners here (Jev 0-shot and Thinking-KT everywhere; LOKT on datasets without answer options), so its

N{=}0

run is repeated. Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.745 $\pm$ .002	.764 $\pm$ .005	.747 $\pm$ .006	.837 $\pm$ .004	.774 $\pm$ .006	.727 $\pm$ .005	.680 $\pm$ .003	.753 $\pm$ .001
Jev 0-shot	.738 $\pm$ .014 ∘	.691 $\pm$ .014 ∘	.725 $\pm$ .013 ∘	.742 $\pm$ .018 ∘	.698 $\pm$ .018 ∘	.700 $\pm$ .015 ∘	.648 $\pm$ .013 ∘	.706 $\pm$ .006 ∘
Laya (open-source Jev)	.596 $\pm$ .003	.575 $\pm$ .002	.511 $\pm$ .007	.605 $\pm$ .003	.530 $\pm$ .004	.520 $\pm$ .006	.507 $\pm$ .004	.549 $\pm$ .002
System-Two LLM-KT, prompted
Thinking-KT	.700 $\pm$ .014 ∘	.641 $\pm$ .014 ∘	.663 $\pm$ .014 ∘	.666 $\pm$ .019 ∘	.663 $\pm$ .018 ∘	.642 $\pm$ .016 ∘	.576 $\pm$ .013 ∘	.650 $\pm$ .006 ∘

Appendix

Table 23: P1 with

N{=}128

training learners: AUC of every model, mean

\pm

SD over 5 learner-subset seeds, on the 2,000 shared early targets per dataset. ‡ Three datasets of the original CLST paper. (k) = only

k

seeds finished. Mean = mean over the 7 datasets (– if a dataset is missing). qDKT and QIKT do not fit Algebra05 in 24 GB of GPU memory (–). ∘ The method uses no data from the

N

learners here (Jev 0-shot and Thinking-KT everywhere; LOKT on datasets without answer options), so its

N{=}0

run is repeated. Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.746 $\pm$ .005	.769 $\pm$ .004	.752 $\pm$ .005	.838 $\pm$ .002	.779 $\pm$ .001	.738 $\pm$ .002	.687 $\pm$ .004	.758 $\pm$ .001
Jev 0-shot	.738 $\pm$ .014 ∘	.691 $\pm$ .014 ∘	.725 $\pm$ .013 ∘	.742 $\pm$ .018 ∘	.698 $\pm$ .018 ∘	.700 $\pm$ .015 ∘	.648 $\pm$ .013 ∘	.706 $\pm$ .006 ∘
Laya (open-source Jev)	.586 $\pm$ .006	.579 $\pm$ .005	.512 $\pm$ .005	.602 $\pm$ .002	.533 $\pm$ .005	.534 $\pm$ .004	.504 $\pm$ .001	.550 $\pm$ .001
System-Two LLM-KT, prompted
Thinking-KT	.700 $\pm$ .014 ∘	.641 $\pm$ .014 ∘	.663 $\pm$ .014 ∘	.666 $\pm$ .019 ∘	.663 $\pm$ .018 ∘	.642 $\pm$ .016 ∘	.576 $\pm$ .013 ∘	.650 $\pm$ .006 ∘

Appendix

Table 24: P1 with

N{=}256

training learners: AUC of every model, mean

\pm

SD over 5 learner-subset seeds, on the 2,000 shared early targets per dataset. ‡ Three datasets of the original CLST paper. (k) = only

k

seeds finished. Mean = mean over the 7 datasets (– if a dataset is missing). qDKT and QIKT do not fit Algebra05 in 24 GB of GPU memory (–). ∘ The method uses no data from the

N

learners here (Jev 0-shot and Thinking-KT everywhere; LOKT on datasets without answer options), so its

N{=}0

run is repeated. Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.780 $\pm$ .012	.774 $\pm$ .011	.758 $\pm$ .013	.829 $\pm$ .014	.780 $\pm$ .016	.771 $\pm$ .013	.701 $\pm$ .013	.770 $\pm$ .005
Jev 0-shot	.738 $\pm$ .014 ∘	.691 $\pm$ .014 ∘	.725 $\pm$ .013 ∘	.742 $\pm$ .018 ∘	.698 $\pm$ .018 ∘	.700 $\pm$ .015 ∘	.648 $\pm$ .013 ∘	.706 $\pm$ .006 ∘
Laya (open-source Jev)	.581 $\pm$ .015	.579 $\pm$ .014	.509 $\pm$ .015	.596 $\pm$ .018	.538 $\pm$ .018	.512 $\pm$ .015	.505 $\pm$ .014	.546 $\pm$ .006
System-Two LLM-KT, prompted
Thinking-KT	.700 $\pm$ .014 ∘	.641 $\pm$ .014 ∘	.663 $\pm$ .014 ∘	.666 $\pm$ .019 ∘	.663 $\pm$ .018 ∘	.642 $\pm$ .016 ∘	.576 $\pm$ .013 ∘	.650 $\pm$ .006 ∘

Appendix

Table 25: P1 with

N{=}\mathrm{all}

training learners: AUC of every model, mean

\pm

SD over seeds (pyKT; Jev, LLM and CLST rows are single runs), on the 2,000 shared early targets per dataset. § Reported by the fine-tuned LLM-KT ( Wang et al., 2026 ) (all training data, its own split; F1 not reported); not comparable to our targets and not ranked. ‡ Three datasets of the original CLST paper. (k) = only

k

seeds finished. Mean = mean over the 7 datasets (– if a dataset is missing). qDKT and QIKT do not fit Algebra05 in 24 GB of GPU memory (–). ∘ The method uses no data from the

N

learners here (Jev 0-shot and Thinking-KT everywhere; LOKT on datasets without answer options), so its

N{=}0

run is repeated. Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
Jev 0-shot	.707 / .779	.636 / .688	.751 / .840	.836 / .908	.734 / .827	.751 / .840	.646 / .423	.723 / .758
Laya (open-source Jev)	.394/.203	.442/.057	.429/.470	.723/.828	.373/.355	.341/.269	.473/ .385	.454/.367
System-Two LLM-KT, prompted
Thinking-KT	.655/.704	.611 /.615	.662/.753	.756/.849	.678/.769	.658/.755	.617/.265	.663/.673
LOKT	.637/.677	.604/.575	.713 / .803	.777 / .863	.703/.804	.742 / .833	.636 /.349	.687 / .700

Appendix

Table 26: P1 with

N{=}0

training learners: ACC / F1 of every model (threshold .5, F1 of the correct class; mean over seeds), same cells as Table 18 . Best in bold , second best underlined .

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.711 / .790	.650 / .694	.750 /.850	.855 / .920	.765 / .856	.760 /.848	.635/.317	.732 /.753
Jev 0-shot	.707 / .779 ∘	.636 /.688 ∘	.751 /.840 ∘	.836/.908 ∘	.734/.827 ∘	.751 /.840 ∘	.646 /.423 ∘	.723 / .758 ∘
Laya (open-source Jev)	.371/.058	.448/.091	.326/.226	.699/.808	.334/.228	.282/.080	.609/.080	.438/.224
System-Two LLM-KT, prompted
Thinking-KT	.655/.704 ∘	.611/.615 ∘	.662/.753 ∘	.756/.849 ∘	.678/.769 ∘	.658/.755 ∘	.617/.265 ∘	.663/.673 ∘

Appendix

Table 27: P1 with

N{=}8

training learners: ACC / F1 of every model (threshold .5, F1 of the correct class; mean over seeds), same cells as Table 19 . Best in bold , second best underlined .

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.716 / .800	.667 / .721	.753 / .853	.860 / .923	.770 / .859	.760 /.849	.642 /.300	.738 / .758
Jev 0-shot	.707 /.779 ∘	.636/.688 ∘	.751/.840 ∘	.836/.908 ∘	.734/.827 ∘	.751 /.840 ∘	.646 /.423 ∘	.723 / .758 ∘
Laya (open-source Jev)	.380/.088	.452/.117	.347/.283	.743/.844	.336/.239	.279/.073	.609/.088	.449/.247
System-Two LLM-KT, prompted
Thinking-KT	.655/.704 ∘	.611/.615 ∘	.662/.753 ∘	.756/.849 ∘	.678/.769 ∘	.658/.755 ∘	.617/.265 ∘	.663/.673 ∘

Appendix

Table 28: P1 with

N{=}16

training learners: ACC / F1 of every model (threshold .5, F1 of the correct class; mean over seeds), same cells as Table 20 . Best in bold , second best underlined .

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.713 / .801	.684 / .732	.751 / .853	.861 / .923	.775 / .861	.755 /.845	.646 /.342	.741 /.765
Jev 0-shot	.707/.779 ∘	.636/.688 ∘	.751 /.840 ∘	.836/.908 ∘	.734/.827 ∘	.751 /.840 ∘	.646 /.423 ∘	.723/.758 ∘
Laya (open-source Jev)	.380/.091	.443/.064	.364/.326	.692/.804	.347/.269	.280/.074	.607/.103	.445/.247
System-Two LLM-KT, prompted
Thinking-KT	.655/.704 ∘	.611/.615 ∘	.662/.753 ∘	.756/.849 ∘	.678/.769 ∘	.658/.755 ∘	.617/.265 ∘	.663/.673 ∘

Appendix

Table 29: P1 with

N{=}32

training learners: ACC / F1 of every model (threshold .5, F1 of the correct class; mean over seeds), same cells as Table 21 . Best in bold , second best underlined .

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.716 / .806	.695 / .747	.753 / .854	.862 / .924	.779/.863	.756 /.847	.650 /.373	.744 /.773
Jev 0-shot	.707/.779 ∘	.636/.688 ∘	.751/.840 ∘	.836/.908 ∘	.734/.827 ∘	.751/.840 ∘	.646/.423 ∘	.723/.758 ∘
Laya (open-source Jev)	.384/.105	.443/.069	.374/.356	.630/.752	.357/.286	.279/.071	.605/.100	.439/.248
System-Two LLM-KT, prompted
Thinking-KT	.655/.704 ∘	.611/.615 ∘	.662/.753 ∘	.756/.849 ∘	.678/.769 ∘	.658/.755 ∘	.617/.265 ∘	.663/.673 ∘

Appendix

Table 30: P1 with

N{=}64

training learners: ACC / F1 of every model (threshold .5, F1 of the correct class; mean over seeds), same cells as Table 22 . Best in bold , second best underlined .

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.716 / .803	.695 / .744	.756 / .855	.864/ .925	.782/.865	.760 / .852	.661 /.388	.748 /.776
Jev 0-shot	.707/.779 ∘	.636/.688 ∘	.751/.840 ∘	.836/.908 ∘	.734/.827 ∘	.751/.840 ∘	.646/.423 ∘	.723/.758 ∘
Laya (open-source Jev)	.387/.115	.444/.071	.381/.370	.634/.755	.357/.288	.285/.086	.604/.106	.442/.256
System-Two LLM-KT, prompted
Thinking-KT	.655/.704 ∘	.611/.615 ∘	.662/.753 ∘	.756/.849 ∘	.678/.769 ∘	.658/.755 ∘	.617/.265 ∘	.663/.673 ∘

Appendix

Table 31: P1 with

N{=}128

training learners: ACC / F1 of every model (threshold .5, F1 of the correct class; mean over seeds), same cells as Table 23 . Best in bold , second best underlined .

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.717/.800	.699/ .746	.760 / .857	.865/ .925	.787/.868	.763 / .853	.668 /.399	.751 /.778
Jev 0-shot	.707/.779 ∘	.636/.688 ∘	.751/.840 ∘	.836/.908 ∘	.734/.827 ∘	.751/.840 ∘	.646/.423 ∘	.723/.758 ∘
Laya (open-source Jev)	.387/.115	.443/.065	.380/.363	.631/.753	.346/.260	.290/.103	.603/.104	.440/.252
System-Two LLM-KT, prompted
Thinking-KT	.655/.704 ∘	.611/.615 ∘	.662/.753 ∘	.756/.849 ∘	.678/.769 ∘	.658/.755 ∘	.617/.265 ∘	.663/.673 ∘

Appendix

Table 32: P1 with

N{=}256

training learners: ACC / F1 of every model (threshold .5, F1 of the correct class; mean over seeds), same cells as Table 24 . Best in bold , second best underlined .

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.738/ .813	.699/.743	.769/.860	.868/ .927	.784/.866	.779 / .862	.674/.418	.759/.784
Jev 0-shot	.707/.779 ∘	.636/.688 ∘	.751/.840 ∘	.836/.908 ∘	.734/.827 ∘	.751/.840 ∘	.646/.423 ∘	.723/.758 ∘
Laya (open-source Jev)	.389/.116	.447/.081	.367/.337	.624/.747	.344/.250	.281/.082	.603/.106	.436/.246
System-Two LLM-KT, prompted
Thinking-KT	.655/.704 ∘	.611/.615 ∘	.662/.753 ∘	.756/.849 ∘	.678/.769 ∘	.658/.755 ∘	.617/.265 ∘	.663/.673 ∘

Appendix

Table 33: P1 with

N{=}\mathrm{all}

training learners: ACC / F1 of every model (threshold .5, F1 of the correct class; mean over seeds), same cells as Table 25 . Best in bold , second best underlined .

Method	1	2–5	6–10	11–15	16–30	31–49
System-One LLM
Jev 0-shot	.580	.687	.712	.710	.704	.717
Laya (open-source Jev)	.572	.539	.615	.547	.533	.528
System-Two LLM-KT, prompted
Thinking-KT	.598	.641	.614	.659	.662	.662
LOKT	.603	.664	.683	.684	.668	.676

Appendix

Table 34: P2 with

N{=}0

: AUC by the position of the target in a new learner’s sequence, mean over the 7 datasets (

\pm

= SD over seeds), for every model with all 7 datasets. (k) = only

k

seeds finished; Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	1	2–5	6–10	11–15	16–30	31–49
System-One LLM
JevKT	.640 $\pm$ .023	.714 $\pm$ .012	.726 $\pm$ .007	.723 $\pm$ .008	.725 $\pm$ .006	.728 $\pm$ .003
Jev 0-shot ∘	.580	.687	.712	.710	.704	.717
Laya (open-source Jev)	.556 $\pm$ .012	.599 $\pm$ .014	.639 $\pm$ .014	.574 $\pm$ .005	.545 $\pm$ .002	.522 $\pm$ .005
System-Two LLM-KT, prompted
Thinking-KT ∘	.598	.641	.614	.659	.662	.662

Appendix

Table 35: P2 with

N{=}8

: AUC by the position of the target in a new learner’s sequence, mean over the 7 datasets (

\pm

= SD over seeds), for every model with all 7 datasets. (k) = only

k

seeds finished; ∘ Uses no data from the

N

learners (LOKT: on 4 of the 7 datasets), so the

N{=}0

run is repeated (fully or in part). Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	1	2–5	6–10	11–15	16–30	31–49
System-One LLM
JevKT	.658 $\pm$ .027	.719 $\pm$ .011	.735 $\pm$ .008	.721 $\pm$ .003	.733 $\pm$ .006	.737 $\pm$ .003
Jev 0-shot ∘	.580	.687	.712	.710	.704	.717
Laya (open-source Jev)	.564 $\pm$ .012	.605 $\pm$ .006	.642 $\pm$ .005	.580 $\pm$ .009	.544 $\pm$ .004	.522 $\pm$ .004
System-Two LLM-KT, prompted
Thinking-KT ∘	.598	.641	.614	.659	.662	.662

Appendix

Table 36: P2 with

N{=}16

: AUC by the position of the target in a new learner’s sequence, mean over the 7 datasets (

\pm

= SD over seeds), for every model with all 7 datasets. (k) = only

k

seeds finished; ∘ Uses no data from the

N

learners (LOKT: on 4 of the 7 datasets), so the

N{=}0

run is repeated (fully or in part). Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	1	2–5	6–10	11–15	16–30	31–49
System-One LLM
JevKT	.652 $\pm$ .023	.726 $\pm$ .005	.737 $\pm$ .007	.735 $\pm$ .008	.741 $\pm$ .003	.746 $\pm$ .003
Jev 0-shot ∘	.580	.687	.712	.710	.704	.717
Laya (open-source Jev)	.555 $\pm$ .014	.610 $\pm$ .006	.645 $\pm$ .005	.575 $\pm$ .004	.536 $\pm$ .005	.520 $\pm$ .002
System-Two LLM-KT, prompted
Thinking-KT ∘	.598	.641	.614	.659	.662	.662

Appendix

Table 37: P2 with

N{=}32

: AUC by the position of the target in a new learner’s sequence, mean over the 7 datasets (

\pm

= SD over seeds), for every model with all 7 datasets. (k) = only

k

seeds finished; ∘ Uses no data from the

N

learners (LOKT: on 4 of the 7 datasets), so the

N{=}0

run is repeated (fully or in part). Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	1	2–5	6–10	11–15	16–30	31–49
System-One LLM
JevKT	.668 $\pm$ .017	.731 $\pm$ .004	.748 $\pm$ .005	.738 $\pm$ .005	.748 $\pm$ .002	.756 $\pm$ .005
Jev 0-shot ∘	.580	.687	.712	.710	.704	.717
Laya (open-source Jev)	.577 $\pm$ .011	.610 $\pm$ .007	.641 $\pm$ .004	.580 $\pm$ .008	.535 $\pm$ .004	.514 $\pm$ .004
System-Two LLM-KT, prompted
Thinking-KT ∘	.598	.641	.614	.659	.662	.662

Appendix

Table 38: P2 with

N{=}64

: AUC by the position of the target in a new learner’s sequence, mean over the 7 datasets (

\pm

= SD over seeds), for every model with all 7 datasets. (k) = only

k

seeds finished; ∘ Uses no data from the

N

learners (LOKT: on 4 of the 7 datasets), so the

N{=}0

run is repeated (fully or in part). Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	1	2–5	6–10	11–15	16–30	31–49
System-One LLM
JevKT	.677 $\pm$ .015	.731 $\pm$ .008	.759 $\pm$ .006	.743 $\pm$ .005	.753 $\pm$ .004	.764 $\pm$ .002
Jev 0-shot ∘	.580	.687	.712	.710	.704	.717
Laya (open-source Jev)	.590 $\pm$ .018	.612 $\pm$ .003	.638 $\pm$ .006	.584 $\pm$ .007	.532 $\pm$ .002	.515 $\pm$ .003
System-Two LLM-KT, prompted
Thinking-KT ∘	.598	.641	.614	.659	.662	.662

Appendix

Table 39: P2 with

N{=}128

: AUC by the position of the target in a new learner’s sequence, mean over the 7 datasets (

\pm

= SD over seeds), for every model with all 7 datasets. (k) = only

k

seeds finished; ∘ Uses no data from the

N

learners (LOKT: on 4 of the 7 datasets), so the

N{=}0

run is repeated (fully or in part). Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	1	2–5	6–10	11–15	16–30	31–49
System-One LLM
JevKT	.679 $\pm$ .015	.736 $\pm$ .008	.763 $\pm$ .006	.748 $\pm$ .003	.760 $\pm$ .002	.768 $\pm$ .003
Jev 0-shot ∘	.580	.687	.712	.710	.704	.717
Laya (open-source Jev)	.594 $\pm$ .009	.610 $\pm$ .002	.643 $\pm$ .009	.585 $\pm$ .007	.535 $\pm$ .003	.516 $\pm$ .002
System-Two LLM-KT, prompted
Thinking-KT ∘	.598	.641	.614	.659	.662	.662

Appendix

Table 40: P2 with

N{=}256

: AUC by the position of the target in a new learner’s sequence, mean over the 7 datasets (

\pm

= SD over seeds), for every model with all 7 datasets. (k) = only

k

seeds finished; ∘ Uses no data from the

N

learners (LOKT: on 4 of the 7 datasets), so the

N{=}0

run is repeated (fully or in part). Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	1	2–5	6–10	11–15	16–30	31–49
System-One LLM
JevKT	.689	.745	.773	.758	.771	.785
Jev 0-shot ∘	.580	.687	.712	.710	.704	.717
Laya (open-source Jev)	.589	.608	.646	.572	.534	.508
System-Two LLM-KT, prompted
Thinking-KT ∘	.598	.641	.614	.659	.662	.662

Appendix

Table 41: P2 with

N{=}\mathrm{all}

: AUC by the position of the target in a new learner’s sequence, mean over the 7 datasets (

\pm

= SD over seeds), for every model with all 7 datasets. (k) = only

k

seeds finished; ∘ Uses no data from the

N

learners (LOKT: on 4 of the 7 datasets), so the

N{=}0

run is repeated (fully or in part). Best in bold , second best underlined (among cells with

\geq

3 seeds).

Method	A09	NIPS34	XES3G5M	DBE	Alg05	Junyi	A17	Mean
System-One LLM
JevKT	.728 $\pm$ .029	.725 $\pm$ .014	.707 $\pm$ .021	.687 $\pm$ .047	.775 $\pm$ .015	.731 $\pm$ .039	.639 $\pm$ .012	.713
Jev 0-shot	.706 $\pm$ .034	.717 $\pm$ .018	.718 $\pm$ .016	.705 $\pm$ .048	.737 $\pm$ .018	.714 $\pm$ .034	.629 $\pm$ .013	.704
System-Two LLM-KT, prompted
Thinking-KT	.670 $\pm$ .026	.667 $\pm$ .029	.651 $\pm$ .007	.669 $\pm$ .034	.689 $\pm$ .009	.662 $\pm$ .037	.593 $\pm$ .012	.657
LOKT §	.688 $\pm$ .036	.683 $\pm$ .027	.692 $\pm$ .015	.682 $\pm$ .054	.649 $\pm$ .030	.653 $\pm$ .020	.599 $\pm$ .012	.664

Appendix

Table 42: P3 (new items): AUC of every model on the held-out items, mean

\pm

SD over 3 item-split seeds ( (k) = only

k

finished). § Option weights from the training learners where options exist. Best in bold , second best underlined (among cells with

\geq

3 seeds).

Explore similar work

May 10, 2026cs.AI

Explainable Knowledge Tracing via Probabilistic Embeddings and Pattern-based Reasoning

Knowledge Tracing (KT) models students' knowledge states based on learning interactions to predict performance. While deep learning-based KT models have boosted predictive accuracy, most models rely on deterministic vector embeddings and opaque latent state transitions, limiting interpretability regarding how specific past behaviors influence predictions. To address this limitation, we propose Probabilistic Logical Knowledge Tracing (PLKT), an interpretable KT framework that formulates prediction as a goal-conditioned evidence reasoning process over historical learning behaviors. Instead of representing knowledge states as deterministic vector embeddings, PLKT employs robust Beta-distributed probabilistic embeddings to represent student knowledge states. This probabilistic foundation allows us to model the uncertainty of historical behaviors and perform explicit logical operations (e.g., conjunction), constructing transparent reasoning paths that reveal how specific past interactions contribute to the prediction. Extensive experiments show that PLKT outperforms state-of-the-art KT methods while achieving superior interpretability. Our code is available at https://anonymous.4open.science/r/PLKT-D3CE/.

Siyu Wu, Cong Xu, Wei Zhang

1Shanghai Institute of AI Education, East China Normal University, Shanghai, 200241, China · 2Shool of Computer Sicence and Technology, East China Normal University, Shanghai, 200241, China

Representation Learning Interpretable ML Knowledge Tracing

Ask PDF

May 1, 2026cs.CL

Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialogues

Recent advances in large language models (LLMs) have led to the development of AI-powered tutoring systems that provide interactive support via dialogue. To enable these tutoring systems to provide personalized support, it is essential to assess student performance at each turn, motivating knowledge tracing (KT) in dialogue settings. However, existing dialogue-based KT approaches often ignore question difficulty modeling and rely on opaque latent representations from LLMs, hindering accurate and interpretable prediction. In this work, we propose an interpretable difficulty-aware conversational KT framework built upon LLMs, which explicitly models students' abilities and the difficulty of tutor-posed tasks at each turn. The framework incorporates the original textual question and the next tutor-posed task to estimate the student's knowledge state and the difficulty of the upcoming turn. Furthermore, it integrates Item Response Theory to map LLM's outputs into student ability and question difficulty parameters, enabling interpretable prediction of student performance grounded in cognitive theories of learning. We evaluate the framework on two tutor-student dialogue datasets. Both quantitative and qualitative results show that our framework outperforms existing KT baselines, meanwhile generating interpretable outputs consistent with cognitive theory.

Shuyan Huang, Alexander Scarlatos, Jaewook Lee +1

University of Massachusetts Amherst

Intelligent Tutoring Systems Interpretable ML Knowledge Tracing

Ask PDF

Jul 6, 2026cs.AI

Detecting Answer-Driven Reasoning in LLM-Based Educational Tutors via Truncated Chain-of-Thought Auditing

Large language model (LLM) tutors often produce fluent step-by-step explanations, but a correct and pedagogically formatted response does not guarantee that the answer was derived from the student-facing problem. In realistic tutoring systems, the model may also have access to teacher notes, answer keys, rubrics, or retrieved solution artifacts. We study whether such private answer information can make tutor explanations answer-driven: the final answer is behaviorally available before the written explanation has justified it. Using Truncated Reasoning AUC Evaluation (TRACE), which probes how early a chain-of-thought prefix can pass a verifier, we evaluate 1000 GSM8K test problems under three paired tutoring contexts: question-only, correct answer-key, and wrong answer-key. At fixed fractions of each generated explanation, we force the model to answer immediately and verify the response against the gold numeric answer. With Qwen2.5-3B-Instruct, answer-key access raises median TRACE AUC from 0.375 to 0.900 and makes the gold answer available at the first 10% prefix in 997 of 1000 cases. The effect remains strong on the 746 examples where both question-only and answer-key explanations end with the correct answer. These results support truncated CoT auditing as a lightweight process-level diagnostic for answer-driven reasoning in math tutoring explanations.

Bonan Shen, Dingyan Shang, Youting Wang +1

Independent Researcher United States · Northeastern University Boston, MA, USA · Syracuse University Syracuse, NY, USA

CoT Faithfulness Intelligent Tutoring Systems LLM Auditing LLM Mathematical Reasoning Reasoning Trace Analysis

Ask PDF