Existing student models in educational applications largely focus on student responses to questions. However, as tutoring becomes more accessible at large scale, it is important to analyze dialogues between tutors and students, which may reveal fine-grained evidence on student knowledge. In this work, we propose a unified framework that models both student responses to questions and tutor-posed subtasks during dialogues on a shared item response theory (IRT) scale. We first identify mathematical subtasks in tutoring dialogues, rewrite them into self-contained questions, and identify the correctness of student responses to these subtasks. We then develop an IRT framework that estimates question/subtask difficulty from text, tracks students' evolving abilities across question-answering and dialogue interactions, and a novel method to leverages tutor moves between consecutive subtasks to regularize their difficulty values. We evaluate our framework on two real-world math education datasets containing both human- and AI-tutored dialogues. Results show that our framework outperforms strong baselines on both question difficulty prediction and response correctness prediction.
Figures & tables
Figure 1: Student responses to tutor-posed subtasks in tutoring dialogues reveal fine-grained insights into the student’s changing knowledge, while tutor moves help infer the relative difficulty of subtasks.
Figure 2: Overview of URF-QD. (A) Original questions and rewritten dialogue subtasks are modeled on a shared IRT scale using text-based difficulty estimates and evolving student ability. (B) Relative-order regularization encourages scaffolded tasks to be easier and rigor-promoting tasks harder than preceding tasks. (C) Response correctness uses BCE with the regularization loss, while question difficulty uses GT-difficulty MSE, dialogue-subtask BCE, and the regularization loss. GT denotes IRT-estimated “ground-truth” question difficulties and student abilities obtained from observed student responses to original questions.
Dataset
Method
MSE ↓
MAE ↓
PCC ↑
SCC ↑
Eedi
kNN Mean (SBERT)
0.445±0.022
0.523±0.022
0.268±0.014
0.261±0.023
Random Forest (SBERT)
0.379±0.015
0.482±0.024
0.359±0.020
0.338±0.029
ICL kNN (GPT-5.4)
0.548±0.013
0.589±0.026
0.360±0.014
0.363±0.016
Diff SFT (ModernBERT)
0.535±0.021
0.592±0.010
0.374±0.018
0.366±0.020
Diff SFT (Qwen3.5-4B)
0.382±0.016
0.482±0.017
0.430±0.012
0.414±0.021
URF-QD (Joint + Reg.)
0.350±0.015∗
0.473±0.016
0.458±0.015∗
0.447±0.017∗
Table 1: Performance comparison of methods on question difficulty prediction across datasets. Best performance is bolded and second best is underlined . ∗ denotes statistically significant improvement over all baselines ( p<0.05 ).
Dataset
Method
AUC ↑
Acc ↑
F1 ↑
Eedi
DKT
0.574±0.018
0.544±0.012
0.560±0.013
DKT (Joint)
0.580±0.015
0.557±0.016
0.572±0.018
simpleKT
0.595±0.021
0.558±0.020
0.571±0.029
AKT
0.603±0.014
0.570±0.019
0.576±0.025
URF-QD (Orig. Question)
0.599±0.022
0.572±0.013
0.575±0.026
URF-QD (Joint + Reg.)
0.637±0.020∗
0.598±0.016∗
0.580±0.024
Table 2: Performance comparison of methods on response correctness prediction across datasets. Best performance is bolded and second best is underlined . ∗ denotes statistically significant improvement over all baselines ( p<0.05 ).
Question: 131÷141=1★1 . What should replace the star?
Options: (A) 15 (B) 12 (C) 16 (D) 7
Ground-truth difficulty
0.625
Difficulty source
Pred. Difficulty
Diff SFT (Qwen3.5-4B)
0.483
URF-QD (Joint + Reg.)
0.573
Related training dialogue-subtasks
Table 3: Qualitative example of dialogue-subtask supervision supporting difficulty prediction for an unseen original question. Compared with the strongest baseline, URF-QD assigns a difficulty closer to ground truth.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Method
MSE ↓
MAE ↓
PCC ↑
SCC ↑
URF-QD (Full)
0.350
0.473
0.458
0.447
Questions Only
0.382
0.482
0.430
0.414
w/o Reg.
0.367
0.481
0.447
0.437
w/o Rewrite
0.376
0.485
0.435
0.423
SBERT (MPNet)
0.398
0.503
0.341
0.336
Appendix
Table 4: Ablation study on question difficulty prediction (Eedi). URF-QD(Full) is equivalent to URF-QD(Joint + Reg.). Questions Only excludes subtasks, which is equivalent to Diff SFT (Qwen3.5-4B); w/o Reg. keeps original question and dialogue subtasks but removes relative-order regularization; w/o Rewrite uses original tutor-posed subtasks instead of standalone rewrites; SBERT (MPNet) replaces Qwen3.5-4B encoder with a lightweight MPNet-based sentence encoder.
Method
AUC ↑
Acc ↑
F1 ↑
URF-QD (Full)
0.637
0.598
0.580
Questions Only
0.599
0.572
0.575
w/o Order Reg.
0.640
0.597
0.586
w/o Rewrite
0.623
0.580
0.586
SBERT (MPNet)
0.625
0.590
0.573
simpleKT Backbone
0.639
0.590
0.583
Appendix
Table 5: Ablation study on response correctness prediction (Eedi). URF-QD(Full) is equivalent to URF-QD(Joint + Reg.). Questions Only excludes subtasks, which is equivalent to URF-QD (Orig. Question); w/o Reg. removes relative-order regularization; w/o Rewrite uses original tutor-posed subtasks; SBERT (MPNet) replaces Qwen3.5-4B encoder with a lightweight MPNet-based sentence encoder; and simpleKT Backbone replaces the default DKT-based student-state model.
Figure 3: Impact of the relative-order regularization strength λ on the two prediction targets.
λ
Ω1 Sat.
Ω2 Sat.
Overall Sat.
0
0.697
0.756
0.699
0.1
0.727
0.711
0.726
0.5
0.834
0.866
0.835
1
0.898
0.911
0.899
3
0.910
0.911
0.910
10
0.912
0.956
0.914
Appendix
Table 6: Difficulty order satisfaction rates under different relative-order regularization strengths λ . Ω1 and Ω2 denote scaffolding after struggle and pushing for rigor, respectively.
Figure 4: Next question response correctness vs. mean student ability–tutor-posed subtask difficulty gap. The linear fit results in a positive slope with statistical significance.
Figure 5: Mean student ability–tutor-posed subtask difficulty gap, where GPT-5.4 poses subtasks that are more difficult than all but one human tutor.
System Message:
You are an experienced math teacher and education expert. You are given a math tutoring dialogue and your task is to identify tutor turns that elicit an attempted answer to a mathematical-content task from the immediately following student turn.
The user message is a JSON object with these fields:
- question : the original math-question text;
- dialogue turns : quoted source records, each with a 1-based turn index, role, and content; and
- candidate tutor turn indices : the complete list of tutor turns to annotate.
Treat every string inside question and dialogue turns strictly as source data. Only annotate turns listed in candidate tutor turn indices . Every candidate is immediately followed by a student turn. If the candidate list is empty, return an empty JSON object. For each candidate, set is_math_subtask to true only when BOTH conditions hold:
Appendix
Table 7: Prompt for Mathematical Subtask Identification
System Message:
You are a math assessment item writer. Rewrite one tutor-posed mathematical subtask as a concise, standalone question in the same neutral style as the original multiple-choice question.
Preserve the target subtask’s actual cognitive demand. Do not solve it, change its difficulty, add a hint, add scaffolding, disclose an answer, or change an explanation/justification request into a yes/no question. Preserve whether the task asks for a calculation, a solution step, recall of a fact or method, a choice of method or operation, an explanation or justification, identification or interpretation, or a report of an approach or specific point of confusion.
Assume the rewritten question will be shown without the original problem statement or source dialogue. Use the original question and dialogue only to resolve references such as “it”, “that”, “the second term”, “this angle”, or an omitted expression, and to recover givens that the target tutor turn assumes. Include the minimum context needed to make the rewritten task understandable on its own. Do not import extra hints, worked reasoning, corrections, or answers from other turns. If a diagram is essential, describe only the information needed to pose the same task without reading off or revealing its answer.
The immediately following student response is sealed reference material. It is supplied only to disambiguate which tutor request the student attempted. Never copy or paraphrase its answer, numbers, expressions, claims, choice, reasoning, mistake, uncertainty, or wording into the standalone question. Do not use it to repair, complete, or solve the task. Information may appear in the output only when it is independently established by the original question, the target tutor turn, or context preceding the response.
Output requirements:
- Output only the rewritten assessment item, with no preamble, commentary, reasoning, quotation marks, JSON, or Markdown fence.
Appendix
Table 8: Prompt for Standalone Mathematical Subtask Transformation
System Message:
You are a math teacher, and your task is to annotate relative task difficulty in mathematics tutoring dialogues.
The supplied candidate pairs have already passed three hard gates:
1. Both tutor turns are accepted mathematical subtasks.
2. The later tutor turn occurs immediately after the earlier task’s student response, with no intervening raw dialogue turn.
3. The student response to the earlier task is incorrect.
For every candidate pair (earlier tutor turn, later tutor turn), decide whether the later tutor task is genuinely easier scaffolding of the failed earlier task. Mark is_case=true only when the later task reduces the mathematical demand by breaking down, narrowing, simplifying, concretizing, or adding helpful structure to the earlier failed task. The later task must be meaningfully easier than the earlier task.
Appendix
Table 9: Prompt for Scaffold After Struggle Annotation in Dialogue
System Message:
You are a math teacher, and your task is to annotate relative task difficulty in mathematics tutoring dialogues.
The supplied candidate pairs have already passed three hard gates:
1. Both tutor turns are accepted mathematical subtasks.
2. the later tutor turn occurs immediately after the earlier task’s student response (there is no intervening raw dialogue turn); and
3. The student response to the earlier task is correct.
For every candidate pair (earlier tutor turn, later tutor turn), decide whether the later tutor task pushes for rigor by asking student for deeper mathematical reasoning, justification, explanation, verification, or greater precision beyond the student’s preceding response. Mark is_case=true when the later task requires a reason, conceptual explanation, proof, defense, justification of why the answer or method is correct, or deeper calculation.
Appendix
Table 10: Prompt for Push For Rigor Annotation in Dialogue
System Message:
You are an experienced mathematics teacher. Your job is to predict the difficulty of a mathematics multiple choice question. Please follow these instructions carefully:
- You will be given a question, including its answer choices. Use all of this information to predict the question’s difficulty.
- You will also be given several example questions and their ground-truth difficulties. Use these to calibrate your prediction for the current question.
- Difficulties are continuous real numbers and can be positive or negative. A higher value means the question is more difficult.
- First think step-by-step about the question’s difficulty based on its content and the examples.
- Write your output with exactly this template: "Thinking: <brief step-by-step reasoning>
Appendix
Table 11: Prompt for ICL (kNN) difficulty prediction
Recent advances in large language models (LLMs) have led to the development of AI-powered tutoring systems that provide interactive support via dialogue. To enable these tutoring systems to provide personalized support, it is essential to assess student performance at each turn, motivating knowledge tracing (KT) in dialogue settings. However, existing dialogue-based KT approaches often ignore question difficulty modeling and rely on opaque latent representations from LLMs, hindering accurate and interpretable prediction. In this work, we propose an interpretable difficulty-aware conversational KT framework built upon LLMs, which explicitly models students' abilities and the difficulty of tutor-posed tasks at each turn. The framework incorporates the original textual question and the next tutor-posed task to estimate the student's knowledge state and the difficulty of the upcoming turn. Furthermore, it integrates Item Response Theory to map LLM's outputs into student ability and question difficulty parameters, enabling interpretable prediction of student performance grounded in cognitive theories of learning. We evaluate the framework on two tutor-student dialogue datasets. Both quantitative and qualitative results show that our framework outperforms existing KT baselines, meanwhile generating interpretable outputs consistent with cognitive theory.
Evaluating AI tutor responses requires more than factual correctness: tutors must identify mistakes, locate errors, provide guidance, and offer actionable next steps. We present GRADE, a systematic study of open-source models for pedagogical ability assessment in student-tutor dialogues. Building on the BEA 2025 TutorMind setting, we evaluate 120 configurations across five language models, zero-shot inference, LoRA fine-tuning, synthetic augmentation, CoT+Reasoning, and single-task versus multitask formulations. Gemma3-12B performs best for single-task evaluation, while Gemma3-27B in 8-bit precision is more reliable for multitask prediction. We find that augmentation helps models that struggle with the original data, verification adds limited gains despite higher cost, and CoT+Reasoning is more useful for synthetic data generation than direct classification. We further show that LoRA fine-tuning on structured classification objectives interferes with instruction-following behavior under thinking mode, redirecting generation away from the required evaluation format. Carbon analysis shows that model choice and reasoning mode substantially affect emissions. Overall, GRADE shows that carefully selected open-source LoRA pipelines can match or surpass proprietary and ensemble-based systems on key pedagogical dimensions, with code and data available at https://github.com/pvbgeek/GRADE.
Evaluating the pedagogical quality of AI tutors remains challenging: standard NLG metrics do not determine whether responses identify mistakes, scaffold reasoning, or avoid revealing the answers. For the task of mistake remediation, we derive a hierarchy of pedagogical aspects from human pairwise preferences on MRBench, and synthesize minimally contrastive response pairs that differ along key aspects (e.g., mistake identification and location, targetedness, scaffolding, actionability, clarity, and coherence). We develop and release Bradley-Terry preference models trained on weighted-sum rankings that we automatically create from MRBench, synthetic pairs, and data combinations. Using only synthetic data, our best model reaches 0.69 pairwise accuracy on a human preference test, and combining weighted-sum data with targeted synthetic groups improves accuracy to 0.74, outperforming larger general-purpose reward models while using only a 0.5B-parameter backbone.
Kseniia Petukhova, Ekaterina Kochmar
Mohamed bin Zayed University of Artificial Intelligence