MCQ Difficulty Prediction via Modeling Learner Heterogeneity Using Data-Driven Cognitive Profiling

Dhriti Krishnan; Jaromir Savelka

cs.CYApr 13, 2026

MCQ Difficulty Prediction via Modeling Learner Heterogeneity Using Data-Driven Cognitive Profiling

Authors: Dhriti Krishnan, Jaromir Savelka

Organizations: Carnegie Mellon University, Pittsburgh, PA, USA

Abstract

Predicting the difficulty of multiple-choice questions (MCQs) is important for effective assessment, yet current methods typically assume a unimodal student ability distribution, overlooking the heterogeneous nature of student misconceptions. We propose a persona-driven framework that replaces theoretical ability sampling with data-driven cognitive profiling. Using student interactions from the EEDI dataset, we identify behavioral personas via latent class analysis (LCA), then condition a large language model (LLM) to simulate response distributions for each persona. These signals are aggregated with topic context and fed into a Ridge Regression model to predict the item response theory (IRT) difficulty parameter. With five-fold cross-validation, our method improves over a recent baseline (MSE: 0.367 to 0.274; R2: 0.525 to 0.686). The discovered personas are interpretable and offer insights into why items are difficult, with potential applications to diagnostic assessment design.

Figures & tables

Panel of Figure 1 — Figure 1: The Proposed Pipeline. LCA discovers 5 learner personas from student response data (Step 1). An LLM simulates per-persona response distributions (Step 2), which are aggregated into features (Step 3) and used to predict IRT difficulty via Ridge Regression (Step 4). Ground truth is estimated independently via 2PL-IRT (Step 0).

Figure 2: Model selection for psychometric profiling. Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC) for Latent Class models with varying numbers of latent classes. BIC reaches a global minimum at $k=5$ , indicating the optimal number of learner profiles.

Persona	Core Strength	Core Weakness
The Rule Memorizer	Applies formulas (e.g., multiplication) accurately.	Lacks number sense; misconceptions on magnitude.
The Procedural Calculator	Strong on single-step arithmetic.	Fails inverse reasoning and multi-step logic.
The Abstract Reasoner	High logical/proportional reasoning.	Weak automaticity with basic arithmetic.
The Conceptual Reasoner	Strong mathematical intuition (“Why”).	Breakdown in procedural fluency (“How”).
The Fraction Calculator	Solves standard fraction equations well.	Fails proportional reasoning in real contexts.

Table 1: Summary of Discovered Personas. Five profiles with distinct procedural and conceptual capabilities.

Method	MSE	$R^{2}$
Our method	0.274 $\pm$ 0.022	0.686 $\pm$ 0.012
Two-Stage Likelihood [ 8 ]	0.367 $\pm$ 0.082	0.525 $\pm$ 0.101
FTWR [ 8 ]	0.471 $\pm$ 0.149	0.390 $\pm$ 0.181
FT [ 8 ]	0.522 $\pm$ 0.079	0.329 $\pm$ 0.048
LR [ 8 ]	0.688 $\pm$ 0.028	0.084 $\pm$ 0.059

Table 2: Item difficulty prediction performance. Baseline results are reported from [ 8 ]

Explore similar work

Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction
Jun 26, 2026Chenguang Wang, Ming Li, Xinyue Zeng +4Cognitive Modeling Educational Assessment
1Virginia Tech · University of Maryland · 3MBZUAI +1
Multimodal Item Parameter Estimation using Simulated Response Probabilitie
Aug 10, 2026Christopher Ormerod, YoungKoung KimEducational Assessment Multimodal Large Language Models
College Board
LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
Jun 17, 2026Han Chen, Ming Li, Chenguang Wang +4LLM Evaluation Educational Assessment
MBZUAI · University of Maryland · Virginia Tech

Jun 26, 2026J/K move · Enter open · S save

Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

Chenguang Wang, Ming Li, Xinyue Zeng, Zhuochun Li+3

1Virginia Tech · University of Maryland · 3MBZUAI+1

Cognitive Modeling Educational Assessment Large Reasoning Models Reasoning Trace Analysis

Open paper AskPDF arXiv

Predicting human item difficulty is central to educational assessment, where reliable estimates support fairness and effective test construction. Existing methods often depend on costly human calibration or item-level textual representations, providing limited evidence about the cognitive processes that make items difficult. We argue that difficulty should be viewed not only as a property of item text, but also as an observable consequence of the problem-solving burden an item induces. Large Reasoning Models (LRMs) offer scalable process evidence through reasoning traces, but such evidence must be structured to support interpretable modeling. To this end, we introduce Epi2Diff (Episode to Difficulty), a framework that maps LRM reasoning traces into cognitively grounded episode sequences. These episodes group trace segments into functional problem-solving states, enabling difficulty to be modeled through reasoning scale, effort allocation, and state transitions. Epi2Diff extracts compact episode-dynamic features and combines them with semantic item representations for human difficulty prediction. Experiments on four real-world human difficulty datasets show that Epi2Diff consistently outperforms strong baselines, including fine-tuned small language models, LLM in-context learning, and supervised LLM adaptation. On SAT-derived classification benchmarks, Epi2Diff achieves an 8.1% average relative gain over supervised LLM fine-tuning baselines. Further analyses show that harder items induce more effortful, iterative, and implementation-centered episode dynamics, rather than merely longer responses. These results demonstrate that cognitive episodes in LRM reasoning traces provide a predictive and interpretable process representation for human item difficulty, offering a new lens for educational measurement with reasoning models.

Abstract

Explore similar work

Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

Multimodal Item Parameter Estimation using Simulated Response Probabilitie

LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment