Prediction Is Not Calibration: Item Parameter Recovery and Representation Coupling in Knowledge Tracing

Wenrui Yuan; Stéphanie M. van den Berg; Johannes Steinrücke

cs.LGOct 21, 2026

Prediction Is Not Calibration: Item Parameter Recovery and Representation Coupling in Knowledge Tracing

Authors: Wenrui Yuan, Stéphanie M. van den Berg, Johannes Steinrücke

Organizations: University of Twente, Enschede, The Netherlands

Abstract

Knowledge tracing (KT) models can estimate item parameters through an item response decoder trained solely for next-response prediction, yet predictive accuracy does not guarantee parameter recovery. We study 2PL, GPCM, and NRM decoders paired with LSTM, Transformer, and DKVMN encoders. Response-model derivatives show why discrimination and option slope updates depend directly on the scale and spread of estimated ability. Affine logit identities show how item-dependent shifts and rescaling in estimated ability are absorbed into item parameters without changing response probabilities. In simulations, difficulty, mean threshold, and correct-option intercept recovery remain high under shuffled order across representation widths, whereas discrimination and keyed option slope contrast recovery require wider item representations and degrade further under fixed item order. Separating the sequence item representation (width $d_S$ ) from the item parameter key (width $d_M$ ), and pairing a wide key with a thin sequence representation ( $\mathrm{SK}(64,8)$ ), reduces fixed-order ability distortions and improves recovery and cross-order consistency without sacrificing test accuracy or increasing model size relative to a shared width-64 baseline ( $\mathrm{SH}(64)$ ). On TIMSS and EdNet, widening $d_M$ consistently improves agreement with an offline IRT reference, while the trade-off from narrowing $d_S$ depends on the encoder's reliance on sequence width for prediction. These findings show why item parameters, especially discrimination and option slopes, must be evaluated separately from response prediction.

Figures & tables

Panel of Figure 1 — Figure 1: Item representation paths in SH and SK. The sequence encoder summarizes interactions before $t$ as an ability estimate, while the parameter readout estimates the parameter vector of the current item $q_{i,t}$ . SH( $w$ ) supplies both paths from one item table of width $w$ . SK( $d_{M},d_{S}$ ) uses a parameter key of width $d_{M}$ for the item parameter readout and an item table of width $d_{S}$ for the sequence encoder. The remaining components and the prediction loss are unchanged.

Method	Item output	Use in response prediction	Evaluation of the item output
Deep-IRT [ 28 ]	Item difficulty $\beta_{q}$	1PL equation with discrimination fixed at 3.0	External real-data comparisons, with no simulated recovery of $\beta_{q}$
Independent-network Deep-IRT [ 24 ]	Item difficulty and skill difficulty	1PL equation with separate student and item networks	Simulated difficulty recovery and test prediction under fixed discrimination
QIKT [ 6 ]	Question-specific acquisition/problem-solving and general mastery scores	Additive sigmoid over the three scores with no learnable response-layer weights	Test prediction and component ablations, with no item parameter recovery
DKT as implicit dynamic MIRT [ 25 ]	Item coefficient vector and intercept in item-output DKT	Inner product with a dynamic multidimensional learner state	Test accuracy and AUC across decoder choices, with no comparison with generating or offline IRT parameters
BKT with IRT and skill discovery [ 13 ]	Item-specific guessing/slipping offsets and item-to-skill assignments	BKT observation probabilities with learner ability and item offsets	BKT parameter and skill-assignment recovery, with no item discrimination estimation
Option Tracing [ 12 ]	Selected option probabilities	Categorical prediction of the selected option	Test prediction, with no NRM option slope or intercept estimation

Table 1: Item outputs and their evaluation in representative knowledge tracing models that use item response terminology or an item response equation.

Figure 2: Item order and 2PL parameter estimation. The left column plots

\phi_{q}

and

s_{q}

against

\beta_{q}

and

\alpha_{q}

\mathrm{SK}(8,8)

. The middle column compares the uncorrected and affine-corrected estimates. The right column varies

d_{M}

while

d_{S}=8

. Blue denotes fixed order and gray denotes shuffled order. Corresponding plots for the GPCM and NRM appear in Appendix A.1 .

Fixed item order
	2PL			GPCM			NRM
Width choice	$\rho_{\mathrm{half}}$	$\rho_{\mathrm{unc}}$	$\rho_{\mathrm{corr}}$	$\rho_{\mathrm{half}}$	$\rho_{\mathrm{unc}}$	$\rho_{\mathrm{corr}}$	$\rho_{\mathrm{half}}$	$\rho_{\mathrm{unc}}$	$\rho_{\mathrm{corr}}$
SK $d_{M}{=}8$ , $d_{S}{=}8$	0.961	0.626	0.934	0.957	0.617	0.931	0.955	0.426	0.919
SK $d_{M}{=}8$ , $d_{S}{=}64$	0.977	0.385	0.916	0.969	0.341	0.909	0.960	0.252	0.915
SK $d_{M}{=}64$ , $d_{S}{=}8$	0.831	0.954	0.973	0.798	0.951	0.977	0.848	0.933	0.969
SK $d_{M}{=}64$ , $d_{S}{=}64$	0.948	0.848	0.962	0.920	0.860	0.963	0.933	0.838	0.963

Table 2: Split-half agreement of the ability offset

\phi_{q}

and recovery of location parameters before and after the affine correction.

\rho_{\mathrm{half}}

compares

\phi_{q}

across two disjoint halves of the test split.

\rho_{\mathrm{unc}}

and

\rho_{\mathrm{corr}}

give uncorrected and corrected recovery. Each entry averages 75 training runs.

Figure 3: Ability offsets and gains under

\mathrm{SH}(64)

\mathrm{SK}(64,64)

, and

\mathrm{SK}(64,8)

. Colors identify the Spearman correlation plotted. Solid lines denote fixed order and dashed lines shuffled order. Each point on a line averages five datasets and five folds, where faint points show individual runs and darker points show dataset means. The NRM location comparison uses generating

a_{q,k^{\star}}>0.2

Figure 4: Simulated item parameter recovery relative to SH(8). The 2PL and GPCM rows report

\Delta_{\mathrm{SH}(8)}\rho_{\alpha}^{\mathrm{sim}}

, and the NRM row reports

\Delta_{\mathrm{SH}(8)}\rho_{a^{\mathrm{key}}}^{\mathrm{sim}}

. Blue curves hold

d_{S}

fixed while

d_{M}

varies, and the gray dashed curve varies the shared width

w

in SH. Curves join means for each width choice, and faint points show the 25 training runs. Twenty of 4,500 runs fall outside the displayed row ranges. Absolute recovery is shown in Figure 5 and Appendix A.5 .

Figure 5: Absolute recovery of $\alpha_{q}$ and $a_{q}^{\mathrm{key}}$ over the complete $\mathrm{SK}(d_{M},d_{S})$ width grid on simulated data. Each tile averages 25 training runs. Color limits are shared across encoders within each response model.

		Test accuracy				Recovery, $\alpha_{q}$ or $a_{q}^{\mathrm{key}}$				Cost
Model	Encoder	SK(64,8)	vs SH(8)	vs SH(64)	vs SK(8,64)	SK(64,8)	vs SH(8)	vs SH(64)	vs SK(8,64)	Ratio
2PL	LSTM	.714	+.003	+.001	+.004	.898	+.344	+.098	+.485	.62
2PL	Transformer	.713	+.004	+.000	+.001	.806	+.433	+.011	+.464	.96
2PL	DKVMN	.715	-.000	+.002	+.002	.914	+.162	+.133	+.529	.68
GPCM	LSTM	.502	+.015	+.008	+.028	.941	+.223	+.099	+.557	.62
GPCM	Transformer	.492	+.033	+.001	+.028	.900	+.463	+.015	+.522	.96

Table 3: Simulated test accuracy and parameter recovery for SK(64,8). Differences from SH(8), SH(64), and SK(8,64) are paired and average 25 training runs. Cost is the trainable parameter ratio to SH(64). Bootstrap intervals are reported in Appendix A.7 .

Figure 6: Agreement with the offline IRT reference relative to SH(8). The 2PL and GPCM rows report

\Delta_{\mathrm{SH}(8)}\rho_{\alpha}^{\mathrm{MML}}

, and the NRM row reports

\Delta_{\mathrm{SH}(8)}\rho_{a^{\mathrm{key}}}^{\mathrm{MML}}

. Blue curves hold

d_{S}

fixed while

d_{M}

varies, and the gray dashed curve varies

w

in SH. Curves join means for each width choice, and faint points show individual training runs. Eleven of 1,260 runs fall outside the displayed row ranges. Absolute agreement is reported in Appendix A.5 .

Study	Encoder	$\mathrm{Acc}_{\rm SK}$	$\Delta\mathrm{Acc}$	$\rho_{\rm SK}^{\rm MML}$	$\Delta\rho^{\rm MML}$	Cost
EdNet 2PL	LSTM	0.641	+0.021	0.846	+0.043	0.46
EdNet 2PL	Transformer	0.617	-0.018	0.827	-0.031	0.77
EdNet 2PL	DKVMN	0.653	+0.003	0.856	+0.006	0.48
TIMSS GPCM	LSTM	0.579	-0.002	0.325	+0.027	0.35
TIMSS GPCM	Transformer	0.580	-0.001	0.518	+0.022	0.89
TIMSS GPCM	DKVMN	0.584	+0.006	0.545	+0.090	0.37

Table 4: SK(64,8) test accuracy and agreement with the offline IRT reference, with paired differences from SH(64). LSTM and Transformer use 25 pairs for 2PL and GPCM and five for NRM. DKVMN uses five pairs per study. Cost is the trainable parameter ratio to SH(64).

Figure 7: Test accuracy and recovery of 2PL and GPCM discrimination

\alpha_{q}

and the NRM keyed option slope contrast

a_{q}^{\mathrm{key}}

on simulated data. Circles denote SK and triangles denote SH. Color indicates

d_{M}

for SK and

w

for SH; large markers show width-choice means and faint markers show means for the five generated datasets.

Figure 8: Test accuracy and agreement of

\alpha_{q}

and

a_{q}^{\mathrm{key}}

with the offline IRT reference on real data. Markers follow Figure 7 , with faint markers showing individual training runs and the gray line marking split-half agreement between two offline estimates from separate response halves.

Figure 9: Item parameter recovery, offline agreement, and trainable parameter count relative to SH(8). Horizontal position is

\Delta_{\mathrm{SH}(8)}\rho_{\xi}

, using simulated recovery in the top row and agreement with the offline IRT reference in the bottom row. The vertical axis is logarithmic. Solid SK curves vary

d_{M}

with

d_{S}=8

, while dashed SH curves vary

w

. Arrows follow increasing width from 8 to 64. Absolute recovery and agreement are reported in Appendix A.8 .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Item order and GPCM parameter estimation. The left column plots

\phi_{q}

and

s_{q}

against the mean step threshold

\bar{\beta}_{q}

and discrimination

\alpha_{q}

. The middle column compares the uncorrected and affine-corrected estimates. The right column varies

d_{M}

while

d_{S}=8

Figure 11: Item order and NRM parameter estimation. The left column uses the correct option location

-c_{q,k_{q}^{\star}}/a_{q,k_{q}^{\star}}

and keyed option slope contrast

a_{q}^{\mathrm{key}}

. The middle column applies Eq. ( 25 ). The right column uses

a_{q}^{\mathrm{key}}

. The location panels exclude 35 items with generating

a_{q,k_{q}^{\star}}\leq 0.2

to avoid dividing by a correct option slope near zero.

Figure 12: Spearman agreement

\rho_{\xi}^{\mathrm{ord}}

between item estimates from fixed and shuffled orders. Colors identify the parameters. Lines join the means for

\mathrm{SH}(64)

\mathrm{SK}(64,64)

, and

\mathrm{SK}(64,8)

. Faint points are paired runs and darker points average the five folds within each generated dataset.

Model	Order	Common	Itemwise	Demeaned	Logit RMSE
2PL	Fixed	0.666	0.444	0.492	0.248
2PL	Shuffled	0.554	0.564	0.578	0.263
GPCM	Fixed	0.470	0.334	0.360	0.395
GPCM	Shuffled	0.371	0.378	0.384	0.408
NRM	Fixed	0.721	0.477	0.533	0.276
NRM	Shuffled	0.618	0.628	0.637	0.291

Appendix

Table 5: Affine fit error on test learners not used to fit the transformation. The three middle columns report ability RMSE normalized by the test-half standard deviation, and Logit RMSE uses the itemwise fit on category-centered logits. Each row averages 150 runs, with the two fitting directions averaged within run before folds and datasets.

Model	Order	$\rho_{\zeta}$	$\rho_{\zeta}^{\rm corr}$	$\rho_{\xi}$	$\rho_{\xi}^{\rm corr}$
2PL	Fixed	0.790	0.977	0.732	0.784
2PL	Shuffled	0.974	0.970	0.743	0.720
GPCM	Fixed	0.784	0.980	0.791	0.883
GPCM	Shuffled	0.982	0.978	0.832	0.820
NRM	Fixed	0.679	0.974	0.475	0.726
NRM	Shuffled	0.962	0.959	0.548	0.519

Appendix

Table 6: Run-level recovery before and after the itemwise affine correction. Here

\zeta_{q}

is 2PL difficulty, GPCM mean step threshold, or NRM correct-option intercept. The parameter

\xi_{q}

is discrimination

\alpha_{q}

for 2PL and GPCM or the keyed option slope contrast

a_{q}^{\mathrm{key}}

for NRM. Items with nonpositive fitted gains are excluded from the recovery correlations but retained in the prediction-error check.

Figure 13: DKVMN ability readout and parameter recovery under random item order. The first panel reports Pearson correlation between the per-item mean ability estimate

\overline{\hat{\theta}}_{q}

and generating difficulty. The remaining panels report Spearman recovery. Faint points show 15 runs per condition, darker points average the three initializations within each of five datasets, and large markers show their mean.

Figure 14: Ranks assigned to the same items by SH(64) and SK(64,8). Faint points are paired runs and open circles average each item’s ranks over pairs. The diagonal denotes equal ranks. LSTM and Transformer have 25 pairs for 2PL and GPCM and five for NRM. DKVMN has five pairs in each column.

Model	Encoder	$n$	$\Delta\mathrm{Acc}$	$\Delta\mathrm{NLL}$	Minimum	Maximum
2PL	LSTM	25	+0.0007	-0.0022	-0.0030	-0.0007
2PL	Transformer	25	+0.0004	+0.0007	-0.0016	+0.0034
2PL	DKVMN	25	+0.0022	-0.0033	-0.0042	-0.0021
GPCM	LSTM	5	+0.0138	-0.0307	-0.0374	-0.0261
GPCM	Transformer	5	+0.0040	-0.0060	-0.0178	+0.0031
GPCM	DKVMN	5	+0.0053	-0.0086	-0.0125	-0.0060

Appendix

Table 7: Test prediction changes from SH(64) to SK(64,8). Means weight the five generated datasets equally. The last two columns report the minimum and maximum dataset-mean log-loss changes across the five generated datasets. The 2PL and NRM reuse 25 runs per model. GPCM uses five supplementary runs per model, one fold from each dataset, with raw unsorted thresholds retained.

Study	Encoder	$n$	$\rho_{\rm SH,SK}$	Rank change	$\Delta\mathrm{Acc}$	$\Delta\mathrm{NLL}$
EdNet 2PL	LSTM	25	0.858	8.7	+0.0211	-0.0765
EdNet 2PL	Transformer	25	0.861	8.3	-0.0179	+0.0681
EdNet 2PL	DKVMN	5	0.936	5.6	+0.0034	-0.0055
TIMSS GPCM	LSTM	25	0.822	10.1	-0.0016	+0.0031
TIMSS GPCM	Transformer	25	0.824	10.5	-0.0009	+0.0014
TIMSS GPCM	DKVMN	5	0.724	13.3	+0.0061	-0.0070

Appendix

Table 8: Matched SK(64,8) minus SH(64) comparisons. Rank change is the mean across runs of the median absolute item-rank change, in percentile points. The ranks use all 31 TIMSS or 250 EdNet items. Here

n

denotes the number of paired training runs, and positive

\Delta\mathrm{NLL}

indicates higher test log loss.

Figure 15: Absolute recovery across

d_{M}

and

d_{S}

on simulated data. Blue lines hold

d_{S}

fixed while

d_{M}

varies. The gray dashed line is SH as

w

varies. The NRM row uses

a_{q}^{\mathrm{key}}

Figure 16: Absolute agreement with the offline IRT reference across

d_{M}

and

d_{S}

on real data. Encodings follow Figure 15 . The gray band gives the agreement between two offline estimates from separate response halves.

Figure 17: Agreement with the offline IRT reference over the complete SK width grid on real data. Color limits are shared across encoders within a dataset and response model.

Figure 18: Agreement among item parameter estimates from training runs with different folds and initializations. Each tile averages pairwise Spearman correlations within one SK width combination. Color limits are shared across encoders within a dataset and response model.

Figure 19: Absolute item parameter recovery, offline agreement, and trainable parameter count. The top row reports simulated recovery, and the bottom row reports agreement with the offline IRT reference. Trainable parameter count is relative to SH(8) on the logarithmic vertical axis. Series and arrow direction follow Figure 9 .

		SK, $d_{S}=8$				SH
Dataset and model	Encoder	$d_{M}{=}8$	16	32	64	$w{=}8$	16	32	64
EdNet 2PL	LSTM	10.5	12.5	16.5	24.6	10.5	16.5	28.7	53.0
	Transformer	46.4	48.4	52.5	60.5	46.4	50.9	60.0	78.2
	DKVMN	8.9	10.9	15.0	23.0	8.9	14.1	24.8	47.7
TIMSS GPCM	LSTM	7.0	7.3	7.8	8.9	7.0	9.6	14.8	25.1
	Transformer	42.9	43.2	43.7	44.8	42.9	44.0	46.1	50.3

Appendix

Table 11: Trainable parameters, in thousands, for the real-data width study. SK holds

d_{S}=8

while

d_{M}

varies. SH varies its shared width

w

. Counts are exact and do not vary across training runs.

Knowledge Tracing

Knowledge tracing models how learners’ mastery of skills changes over time from their interactions with educational tasks.

See all 31 papers

Aug 10, 2026

Multimodal Item Parameter Estimation using Simulated Response Probabilitie

We present results from reconstructing multiple-choice model (MCM) and three-parameter logistic (3PL) model curves using a fine-tuned multimodal large language model (LLM) based on Qwen3.5. The model is prompted and fine-tuned to replicate choice probabilities across a large training corpus of multiple-choice items containing both image and text stimuli, conditioned on a labeled set of student ability levels. By learning to reproduce the systematic error patterns of students across a discrete range of abilities, the LLM implicitly captures the underlying response probabilities encoded in the 3PL and MCM curves. This allows us to accurately approximate item difficulty on a held-out test set directly from the model's predicted option probabilities.

Christopher Ormerod, YoungKoung Kim

College Board

Educational Assessment Multimodal Large Language Models Item Response Theory

Ask PDF

Jun 12, 2026

Recovering Stranded Discrimination in Knowledge Tracing: Per-Item Bias Correction via Empirical-Bayes Shrinkage

Deployed knowledge-tracing models are typically frozen after training, yet systematic per-item logit bias arises, from limited per-item expressivity in backbone architectures and from post-deployment shifts in item properties, degrading prediction quality. Global post-hoc calibrators such as Platt scaling, temperature scaling, and isotonic regression improve probability estimates but leave discriminative ability, as measured by AUC, unchanged. This AUC invariance is a structural consequence of monotone score-only transforms; recovering the stranded discrimination requires conditioning on item identity. We propose SLC (State-space Logit Correction), which converts binary observations to Gaussian pseudo-observations via Laplace/IRLS, applies empirical-Bayes shrinkage through a Kalman smoother, and fits an offset-Platt link. The state-space formulation also yields a detectability bound that characterizes the Bernoulli information floor, explaining why temporal tracking provides no benefit at current data densities. Across four datasets, five backbones, and three seeds, SLC improves AUC on all four datasets and NLL on three, with the advantage concentrating on sparse items. Cross-domain controls suggest that the same phenomenon can arise beyond education when the deployed backbone leaves entity-level bias.

Xiaoran Yan, Cheng Tang, Atsushi Shimada

Kyushu University, Fukuoka, Japan

Post-Hoc Calibration Empirical Bayes Knowledge Tracing

Ask PDF

Jun 17, 2026

LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment

Item discrimination is a fundamental psychometric property of educational assessment, which measures whether an item meaningfully distinguishes students with higher proficiency from students with lower proficiency. While various existing works have explored whether large language models (LLMs) can estimate item difficulty, it remains unclear whether they can capture item discrimination. In this work, we evaluate 42 proprietary and open-weight LLMs in zero-shot settings using two complementary approaches: direct discrimination prediction, where models explicitly estimate an item's discrimination value from its content, and response-based Classical Test Theory (CTT) calibration, where LLM answers are treated as synthetic student responses to compute discrimination scores. Our results show that direct prediction yields weak alignment with human-calibrated discrimination: the best-performing model reaches only a Spearman correlation of 0.152. Response-based CTT calibration provides a stronger but still limited signal, with the all-persona synthetic respondent pool reaching a Spearman correlation of 0.241. These findings highlight item discrimination as an open challenge for LLM-based psychometric evaluation: current LLMs contain non-random discrimination-relevant signal, but they do not yet reliably capture how assessment items distinguish human students.

Han Chen, Ming Li, Chenguang Wang +4

MBZUAI · University of Maryland · Virginia Tech

LLM Evaluation Educational Assessment Psychometric Validation

Ask PDF

Abstract

Knowledge Tracing

Explore similar work

Multimodal Item Parameter Estimation using Simulated Response Probabilitie

Recovering Stranded Discrimination in Knowledge Tracing: Per-Item Bias Correction via Empirical-Bayes Shrinkage

LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment