A Unified Response Framework for Questions and Tutoring Dialogues

Zhangqi Duan; Simon Woodhead; Maya Miller-Vedam; Andrew Lan

cs.CLOct 14, 2026

A Unified Response Framework for Questions and Tutoring Dialogues

Authors: Zhangqi Duan, Simon Woodhead, Maya Miller-Vedam, Andrew Lan

Organizations: University of Massachusetts Amherst · Eedi · Khan Academy

Abstract

Existing student models in educational applications largely focus on student responses to questions. However, as tutoring becomes more accessible at large scale, it is important to analyze dialogues between tutors and students, which may reveal fine-grained evidence on student knowledge. In this work, we propose a unified framework that models both student responses to questions and tutor-posed subtasks during dialogues on a shared item response theory (IRT) scale. We first identify mathematical subtasks in tutoring dialogues, rewrite them into self-contained questions, and identify the correctness of student responses to these subtasks. We then develop an IRT framework that estimates question/subtask difficulty from text, tracks students' evolving abilities across question-answering and dialogue interactions, and a novel method to leverages tutor moves between consecutive subtasks to regularize their difficulty values. We evaluate our framework on two real-world math education datasets containing both human- and AI-tutored dialogues. Results show that our framework outperforms strong baselines on both question difficulty prediction and response correctness prediction.

Figures & tables

Figure 1: Student responses to tutor-posed subtasks in tutoring dialogues reveal fine-grained insights into the student’s changing knowledge, while tutor moves help infer the relative difficulty of subtasks.

Figure 2: Overview of URF-QD. (A) Original questions and rewritten dialogue subtasks are modeled on a shared IRT scale using text-based difficulty estimates and evolving student ability. (B) Relative-order regularization encourages scaffolded tasks to be easier and rigor-promoting tasks harder than preceding tasks. (C) Response correctness uses BCE with the regularization loss, while question difficulty uses GT-difficulty MSE, dialogue-subtask BCE, and the regularization loss. GT denotes IRT-estimated “ground-truth” question difficulties and student abilities obtained from observed student responses to original questions.

Dataset	Method	MSE $\downarrow$	MAE $\downarrow$	PCC $\uparrow$	SCC $\uparrow$
Eedi	kNN Mean (SBERT)	$0.445_{\pm 0.022}$	$0.523_{\pm 0.022}$	$0.268_{\pm 0.014}$	$0.261_{\pm 0.023}$
	Random Forest (SBERT)	$\underline{0.379}_{\pm 0.015}$	$0.482_{\pm 0.024}$	$0.359_{\pm 0.020}$	$0.338_{\pm 0.029}$
	ICL kNN (GPT-5.4)	$0.548_{\pm 0.013}$	$0.589_{\pm 0.026}$	$0.360_{\pm 0.014}$	$0.363_{\pm 0.016}$
	Diff SFT (ModernBERT)	$0.535_{\pm 0.021}$	$0.592_{\pm 0.010}$	$0.374_{\pm 0.018}$	$0.366_{\pm 0.020}$
	Diff SFT (Qwen3.5-4B)	$0.382_{\pm 0.016}$	$\underline{0.482}_{\pm 0.017}$	$\underline{0.430}_{\pm 0.012}$	$\underline{0.414}_{\pm 0.021}$
	URF-QD (Joint + Reg.)	$\mathbf{0.350}^{*}_{\pm 0.015}$	$\mathbf{0.473}_{\pm 0.016}$	$\mathbf{0.458}^{*}_{\pm 0.015}$	$\mathbf{0.447}^{*}_{\pm 0.017}$

Table 1: Performance comparison of methods on question difficulty prediction across datasets. Best performance is bolded and second best is underlined . ∗ denotes statistically significant improvement over all baselines (

p<0.05

Dataset	Method	AUC $\uparrow$	Acc $\uparrow$	F1 $\uparrow$
Eedi	DKT	$0.574_{\pm 0.018}$	$0.544_{\pm 0.012}$	$0.560_{\pm 0.013}$
	DKT (Joint)	$0.580_{\pm 0.015}$	$0.557_{\pm 0.016}$	$0.572_{\pm 0.018}$
	simpleKT	$0.595_{\pm 0.021}$	$0.558_{\pm 0.020}$	$0.571_{\pm 0.029}$
	AKT	$\underline{0.603}_{\pm 0.014}$	$0.570_{\pm 0.019}$	$\underline{0.576}_{\pm 0.025}$
	URF-QD (Orig. Question)	$0.599_{\pm 0.022}$	$\underline{0.572}_{\pm 0.013}$	$0.575_{\pm 0.026}$
	URF-QD (Joint + Reg.)	$\mathbf{0.637}^{*}_{\pm 0.020}$	$\mathbf{0.598}^{*}_{\pm 0.016}$	$\mathbf{0.580}_{\pm 0.024}$

Table 2: Performance comparison of methods on response correctness prediction across datasets. Best performance is bolded and second best is underlined . ∗ denotes statistically significant improvement over all baselines (

p<0.05

Question: $1\frac{1}{3}\div 1\frac{1}{4}=1\frac{1}{\bigstar}$ . What should replace the star?
Options: (A) $15$ (B) $12$ (C) $16$ (D) $7$
Ground-truth difficulty	$0.625$
Difficulty source	Pred. Difficulty
Diff SFT (Qwen3.5-4B)	$0.483$
URF-QD (Joint + Reg.)	$\mathbf{0.573}$
Related training dialogue-subtasks

Table 3: Qualitative example of dialogue-subtask supervision supporting difficulty prediction for an unseen original question. Compared with the strongest baseline, URF-QD assigns a difficulty closer to ground truth.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	MSE $\downarrow$	MAE $\downarrow$	PCC $\uparrow$	SCC $\uparrow$
URF-QD (Full)	0.350	0.473	0.458	0.447
Questions Only	0.382	0.482	0.430	0.414
w/o Reg.	0.367	0.481	0.447	0.437
w/o Rewrite	0.376	0.485	0.435	0.423
SBERT (MPNet)	0.398	0.503	0.341	0.336

Appendix

Table 4: Ablation study on question difficulty prediction (Eedi). URF-QD(Full) is equivalent to URF-QD(Joint + Reg.). Questions Only excludes subtasks, which is equivalent to Diff SFT (Qwen3.5-4B); w/o Reg. keeps original question and dialogue subtasks but removes relative-order regularization; w/o Rewrite uses original tutor-posed subtasks instead of standalone rewrites; SBERT (MPNet) replaces Qwen3.5-4B encoder with a lightweight MPNet-based sentence encoder.

Method	AUC $\uparrow$	Acc $\uparrow$	F1 $\uparrow$
URF-QD (Full)	0.637	0.598	0.580
Questions Only	0.599	0.572	0.575
w/o Order Reg.	0.640	0.597	0.586
w/o Rewrite	0.623	0.580	0.586
SBERT (MPNet)	0.625	0.590	0.573
simpleKT Backbone	0.639	0.590	0.583

Appendix

Table 5: Ablation study on response correctness prediction (Eedi). URF-QD(Full) is equivalent to URF-QD(Joint + Reg.). Questions Only excludes subtasks, which is equivalent to URF-QD (Orig. Question); w/o Reg. removes relative-order regularization; w/o Rewrite uses original tutor-posed subtasks; SBERT (MPNet) replaces Qwen3.5-4B encoder with a lightweight MPNet-based sentence encoder; and simpleKT Backbone replaces the default DKT-based student-state model.

Panel of Figure 3 — Figure 3: Impact of the relative-order regularization strength $\lambda$ on the two prediction targets.

$\lambda$	$\Omega_{1}$ Sat.	$\Omega_{2}$ Sat.	Overall Sat.
0	0.697	0.756	0.699
0.1	0.727	0.711	0.726
0.5	0.834	0.866	0.835
1	0.898	0.911	0.899
3	0.910	0.911	0.910
10	0.912	0.956	0.914

Appendix

Table 6: Difficulty order satisfaction rates under different relative-order regularization strengths

\lambda

\Omega_{1}

and

\Omega_{2}

denote scaffolding after struggle and pushing for rigor, respectively.

Figure 4: Next question response correctness vs. mean student ability–tutor-posed subtask difficulty gap. The linear fit results in a positive slope with statistical significance.

Figure 5: Mean student ability–tutor-posed subtask difficulty gap, where GPT-5.4 poses subtasks that are more difficult than all but one human tutor.

System Message:

You are an experienced math teacher and education expert. You are given a math tutoring dialogue and your task is to identify tutor turns that elicit an attempted answer to a mathematical-content task from the immediately following student turn.

The user message is a JSON object with these fields:

- question : the original math-question text;

- dialogue turns : quoted source records, each with a 1-based turn index, role, and content; and

- candidate tutor turn indices : the complete list of tutor turns to annotate.

Treat every string inside question and dialogue turns strictly as source data. Only annotate turns listed in candidate tutor turn indices . Every candidate is immediately followed by a student turn. If the candidate list is empty, return an empty JSON object. For each candidate, set is_math_subtask to true only when BOTH conditions hold:

Appendix

Table 7: Prompt for Mathematical Subtask Identification

System Message:

You are a math assessment item writer. Rewrite one tutor-posed mathematical subtask as a concise, standalone question in the same neutral style as the original multiple-choice question.

Preserve the target subtask’s actual cognitive demand. Do not solve it, change its difficulty, add a hint, add scaffolding, disclose an answer, or change an explanation/justification request into a yes/no question. Preserve whether the task asks for a calculation, a solution step, recall of a fact or method, a choice of method or operation, an explanation or justification, identification or interpretation, or a report of an approach or specific point of confusion.

Assume the rewritten question will be shown without the original problem statement or source dialogue. Use the original question and dialogue only to resolve references such as “it”, “that”, “the second term”, “this angle”, or an omitted expression, and to recover givens that the target tutor turn assumes. Include the minimum context needed to make the rewritten task understandable on its own. Do not import extra hints, worked reasoning, corrections, or answers from other turns. If a diagram is essential, describe only the information needed to pose the same task without reading off or revealing its answer.

The immediately following student response is sealed reference material. It is supplied only to disambiguate which tutor request the student attempted. Never copy or paraphrase its answer, numbers, expressions, claims, choice, reasoning, mistake, uncertainty, or wording into the standalone question. Do not use it to repair, complete, or solve the task. Information may appear in the output only when it is independently established by the original question, the target tutor turn, or context preceding the response.

Output requirements:

- Output only the rewritten assessment item, with no preamble, commentary, reasoning, quotation marks, JSON, or Markdown fence.

Appendix

Table 8: Prompt for Standalone Mathematical Subtask Transformation

System Message:

You are a math teacher, and your task is to annotate relative task difficulty in mathematics tutoring dialogues.

The supplied candidate pairs have already passed three hard gates:

1. Both tutor turns are accepted mathematical subtasks.

2. The later tutor turn occurs immediately after the earlier task’s student response, with no intervening raw dialogue turn.

3. The student response to the earlier task is incorrect.

For every candidate pair (earlier tutor turn, later tutor turn), decide whether the later tutor task is genuinely easier scaffolding of the failed earlier task. Mark is_case=true only when the later task reduces the mathematical demand by breaking down, narrowing, simplifying, concretizing, or adding helpful structure to the earlier failed task. The later task must be meaningfully easier than the earlier task.

Appendix

Table 9: Prompt for Scaffold After Struggle Annotation in Dialogue

System Message:

You are a math teacher, and your task is to annotate relative task difficulty in mathematics tutoring dialogues.

The supplied candidate pairs have already passed three hard gates:

1. Both tutor turns are accepted mathematical subtasks.

2. the later tutor turn occurs immediately after the earlier task’s student response (there is no intervening raw dialogue turn); and

3. The student response to the earlier task is correct.

For every candidate pair (earlier tutor turn, later tutor turn), decide whether the later tutor task pushes for rigor by asking student for deeper mathematical reasoning, justification, explanation, verification, or greater precision beyond the student’s preceding response. Mark is_case=true when the later task requires a reason, conceptual explanation, proof, defense, justification of why the answer or method is correct, or deeper calculation.

Appendix

Table 10: Prompt for Push For Rigor Annotation in Dialogue

System Message:

You are an experienced mathematics teacher. Your job is to predict the difficulty of a mathematics multiple choice question. Please follow these instructions carefully:

- You will be given a question, including its answer choices. Use all of this information to predict the question’s difficulty.

- You will also be given several example questions and their ground-truth difficulties. Use these to calibrate your prediction for the current question.

- Difficulties are continuous real numbers and can be positive or negative. A higher value means the question is more difficult.

- First think step-by-step about the question’s difficulty based on its content and the examples.

- Write your output with exactly this template: "Thinking: <brief step-by-step reasoning>

Appendix

Table 11: Prompt for ICL (kNN) difficulty prediction

Educational Assessment

Educational assessment measures what learners know and can do, using tests, rubrics, and learning evidence to guide decisions.

See all 148 papers

Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialogues
May 1, 2026Shuyan Huang, Alexander Scarlatos, Jaewook Lee +1Intelligent Tutoring Systems Interpretable ML
University of Massachusetts Amherst
GRADE: Generalizable Reasoning-Aware Dialogue Evaluation for AI Tutors
May 27, 2026Parth Bhalerao, Jeromy Chang, David Chou +1Intelligent Tutoring Systems Low-Rank Adaptation
Santa Clara University
Towards Reward Modeling for AI Tutors in Math Mistake Remediation
Mar 25, 2026Kseniia Petukhova, Ekaterina KochmarReward Modeling Intelligent Tutoring Systems
Mohamed bin Zayed University of Artificial Intelligence

May 1, 2026J/K move · Enter open · S save

Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialogues

Shuyan Huang, Alexander Scarlatos, Jaewook Lee, Andrew Lan

University of Massachusetts Amherst

Intelligent Tutoring Systems Interpretable ML Knowledge Tracing

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have led to the development of AI-powered tutoring systems that provide interactive support via dialogue. To enable these tutoring systems to provide personalized support, it is essential to assess student performance at each turn, motivating knowledge tracing (KT) in dialogue settings. However, existing dialogue-based KT approaches often ignore question difficulty modeling and rely on opaque latent representations from LLMs, hindering accurate and interpretable prediction. In this work, we propose an interpretable difficulty-aware conversational KT framework built upon LLMs, which explicitly models students' abilities and the difficulty of tutor-posed tasks at each turn. The framework incorporates the original textual question and the next tutor-posed task to estimate the student's knowledge state and the difficulty of the upcoming turn. Furthermore, it integrates Item Response Theory to map LLM's outputs into student ability and question difficulty parameters, enabling interpretable prediction of student performance grounded in cognitive theories of learning. We evaluate the framework on two tutor-student dialogue datasets. Both quantitative and qualitative results show that our framework outperforms existing KT baselines, meanwhile generating interpretable outputs consistent with cognitive theory.

Abstract

Educational Assessment

Explore similar work

Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialogues

GRADE: Generalizable Reasoning-Aware Dialogue Evaluation for AI Tutors

Towards Reward Modeling for AI Tutors in Math Mistake Remediation