Towards Reward Modeling for AI Tutors in Math Mistake Remediation

Kseniia Petukhova; Ekaterina Kochmar

cs.CLMar 25, 2026

Towards Reward Modeling for AI Tutors in Math Mistake Remediation

Authors: Kseniia Petukhova, Ekaterina Kochmar

Organizations: Mohamed bin Zayed University of Artificial Intelligence

Abstract

Evaluating the pedagogical quality of AI tutors remains challenging: standard NLG metrics do not determine whether responses identify mistakes, scaffold reasoning, or avoid revealing the answers. For the task of mistake remediation, we derive a hierarchy of pedagogical aspects from human pairwise preferences on MRBench, and synthesize minimally contrastive response pairs that differ along key aspects (e.g., mistake identification and location, targetedness, scaffolding, actionability, clarity, and coherence). We develop and release Bradley-Terry preference models trained on weighted-sum rankings that we automatically create from MRBench, synthetic pairs, and data combinations. Using only synthetic data, our best model reaches 0.69 pairwise accuracy on a human preference test, and combining weighted-sum data with targeted synthetic groups improves accuracy to 0.74, outperforming larger general-purpose reward models while using only a 0.5B-parameter backbone.

Figures & tables

Figure 1: Example of annotated tutor responses from MRBench . The Sonnet response is annotated as more actionable (“Yes” vs. “To some extent”) because it prompts the student to recall the prefix for a five-sided shape rather than revealing it directly. In contrast, the Expert response is more encouraging in tone (“Encouraging” vs. “Neutral”).

Dialog Context
Student: yay Tutor: Next, we have to find the value of 6 times 9 + 2. Tutor: What is the value of 6 times 9 + 2? Student: 99
Preferred Response	Tutor Response	MI	ML	RA	PG	AC	HM	CO	TT	Score
Preferred (weighted-sum)	Good try, but remember the order of operations: multiplication should be done before addition. So it should be 6 times 9, then add 2. Let’s try again!	Yes	Yes	No	Yes	Yes	Yes	Yes	Enc.	4.55
		0.50	1.00	0.25	1.00	0.50	0.25	1.00	0.05	4.55
Preferred (human)	Great try! What was your first step?	TSE	No	No	Yes	Yes	Yes	Yes	Enc.	3.30
		0.25	0.00	0.25	1.00	0.50	0.25	1.00	0.05	3.30

Table 1: Example of misalignment between the weighted-sum ranking (Section 3.1 ) and human preference. For each response, the first row reports the original MRBench annotation labels, and the second row reports their weighted contributions (annotation score

\times

dimension weight). Abbreviations: MI – Mistake Identification ; ML – Mistake Location ; RA – Revealing the Answer ; PG – Providing Guidance ; AC – Actionability ; HM – Human-likeness ; CO – Coherence ; TT – Tutor Tone ; TSE – To some extent ; Enc – Encouraging .

Figure 2: Pipeline for synthetic data augmentation. The procedure augments MRBench by generating aspect-specific improvements of suboptimal responses, jointly improved variants, and controlled degradations of optimal responses, thereby constructing structured preference pairs aligned with human annotation preferences. Suboptimal responses are those that do not receive desirable annotations in one or more of the following MRBench dimensions: Revealing the Answer , Providing Guidance , Actionability , and Coherence . In contrast, poor responses receive undesirable annotations across all four of these dimensions.

Block	Group	Pairs
Aspect-wise (worsen $\downarrow$ )	Revealing the answer $\downarrow$	$578$
	Mistake identification & location $\downarrow$	$578$
	Clarity & coherence $\downarrow$	$578$
	Targetedness / core misunderstanding $\downarrow$	$578$
	Scaffolding & actionability $\downarrow$	$578$
Aspect-wise (improve $\uparrow$ )	Mistake identification & location $\uparrow$	$833$

Table 2: Synthetic augmentation results by evaluation aspects. Aspect-wise worsening of the pairs corresponds to Step 3, improvement to Step 1, and joint improvement to Step 2.

\dagger

The “global non-preference” block aggregates comparisons from Steps 4–5, where factually incorrect or mistake-unaware responses are non-preferred.

Training data setup	Qwen2.5-0.5B		Qwen2.5-1.5B		Gemma-2-2B
Training data setup	W.-sum test	Human test	W.-sum test	Human test	W.-sum test	Human test
Weighted-sum	0.70	0.68	0.70	0.69	0.74	0.69
Synthetic	0.66	0.69	0.66	0.68	0.70	0.69
Combined	0.73	0.69	0.74	0.70	0.76	0.67
Weighted-sum & Targetedness $\uparrow$	0.71	0.74	0.71	0.69	0.79	0.68
Weighted-sum & Mistake ID & Location $\uparrow$	0.72	0.74	0.71	0.70	0.76	0.70

Table 3: Reward model performance (pairwise accuracy) across different training data setups and models. External baselines are evaluated only on the human test set. Bold values indicate the best results on the human test set, and underlined values indicate the second-best ones.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Illustrative contrastive examples for the hierarchy of pedagogical aspects.

Table 4: Examples of misalignment between the weighted-sum ranking (Section 3.1 ) and human preference. For each response, the first row reports the original MRBench annotation labels, and the second row reports their weighted contributions (annotation score

\times

Dialog Context
Tutor: Hi, could you please provide a step-by-step solution for the question below? The question is: Jason has to drive home which is 120 miles away. If he drives at 60 miles per hour for 30 minutes, what speed does he have to average for the remainder of the drive to get there in exactly 1 hour 30 minutes? Student: In 30 minutes, Jason covers a distance of 60 x 0.5 = 30 miles. He has 120 - 30 = 90 miles left to cover. To cover 90 miles in 1 hour 30 minutes (or 1.5 hours), he needs to average a speed of 90/1.5 = 60 miles per hour. Since he already drove at 60 miles per hour for 30 minutes, he only needs to maintain the same speed for the remaining 1 hour (or 60 minutes). 60 Tutor: Jason covers a distance of 60 x 0.5 = 30 miles. He has 120 - 30 = 90 miles left to cover. To cover 90 miles in 1 hour 30 minutes (or 1.5 hours) he needs to average a speed of 90/1.5 = 60 miles per hour. Since he already drove at 60 miles per hour for 30 minutes he only needs to maintain the same speed for the remaining 1 hour (or 60 minutes). Tutor: i can see you have basically worked the majority of this out correctly well done. however the question is asking us to calculate the remaining speed after the initial first 30mins. Now knowing this how does this help you? Student: Since he already drove at 60 miles per hour for 30 minutes he only needs to maintain the same speed for the remaining 1 hour (or 60 minutes) to get home in 1 hour 30 minutes. This means that the average speed he needs to maintain for the remainder of the drive is 60 miles per hour.
Correct Solution
Jason drives 60 miles per hour * 0.5 hours = 30 miles initially He needs to drive an additional 120 miles total - 30 miles driven = 90 more miles He has 1.5 hours - 0.5 hours = 1 hour to drive the remaining distance He must average 90 miles per hour to drive the remaining 90 miles in 1 hour 90
Response	Tutor Response
A	I appreciate your effort, but let’s think about this again: if Jason drove at 60 mph for the entire trip, would he arrive in 1 hour 30 minutes as the question states?
B	Great job on understanding the distance and time, but remember, the total travel time is 1 hour 30 minutes, so after the initial 30 minutes, Jason has 1 hour to cover the remaining distance – how does that change the required speed?

Appendix

Table 5: Example of annotator disagreement: one annotator preferred Response B, while three preferred Response A.

Figure 4: Basic prompt template used for LLM-based preference annotation.

Figure 5: Prompt template with guidelines used for LLM-based preference annotation.

Figure 6: Prompt template with a decision checklist used for LLM-based preference annotation.

Figure 7: Prompt template with a decision-making process based on a hierarchy (see Appendix A ) that was used for LLM-based preference annotation in Section 3.2 .

Training data setup	W.-sum test	Human test
Weighted-sum	0.70	0.68
Synthetic	0.66	0.69
Combined	0.73	0.69
Weighted-sum & Targetedness $\uparrow$	0.71	0.74
Weighted-sum & Targetedness $\downarrow$	0.74	0.71
Weighted-sum & Mistake ID & Location $\uparrow$	0.72	0.74

Appendix

Table 6: Reward model performance (pairwise accuracy) for Qwen2.5-0.5B across different training data setups. External baselines are evaluated only on the human test set. Bold values indicate the best results on the human test set.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 313 papers

Towards Pedagogically Aligned LLM Tutors for Math Mistake Remediation
Jun 19, 2026Kseniia Petukhova, Tien Dat Nguyen, Ekaterina KochmarLLM Alignment Intelligent Tutoring Systems
Mohamed bin Zayed University of Artificial Intelligence
MisEdu-RAG: A Misconception-Aware Dual-Hypergraph RAG for Novice Math Teachers
Apr 5, 2026Zhihan Guo, Yuting Lu, Jionghao LinRetrieval-Augmented Generation Knowledge Augmentation for Language Models
The University of Hong Kong, Hong Kong SAR, China · Carnegie Mellon University, Pittsburgh PA 15213, USA · Monash University, Clayton VIC 3800, Australia
GRADE: Generalizable Reasoning-Aware Dialogue Evaluation for AI Tutors
May 27, 2026Parth Bhalerao, Jeromy Chang, David Chou +1Intelligent Tutoring Systems Low-Rank Adaptation
Santa Clara University

Jun 19, 2026J/K move · Enter open · S save

Towards Pedagogically Aligned LLM Tutors for Math Mistake Remediation

Kseniia Petukhova, Tien Dat Nguyen, Ekaterina Kochmar

Mohamed bin Zayed University of Artificial Intelligence

LLM Alignment Intelligent Tutoring Systems Generative AI in Education Instruction Tuning

Open paper AskPDF arXiv

Large language models have strong potential for use in intelligent tutoring systems, but they often fail to follow effective pedagogical strategies, such as guiding students without revealing final answers. We study the application of a two-stage alignment pipeline for math mistake remediation, combining supervised fine-tuning on tutoring dialogs with Direct Preference Optimization on synthetic preference pairs. We construct a dataset that integrates existing tutoring corpora with synthetic data generated along pedagogical dimensions, such as scaffolding and factuality, and study different input configurations that incorporate solution correctness and gold answers. Experiments show that this approach improves both factual accuracy and pedagogical quality over base models and existing tutoring models. Human evaluation further indicates that our best model is competitive with a strong proprietary baseline, while providing additional benefits in terms of openness, transparency, and reproducibility. Our results highlight the effectiveness of preference-based pedagogical alignment, while also revealing challenges in reliably evaluating tutoring quality.

Abstract

Reward Modeling

Explore similar work

Towards Pedagogically Aligned LLM Tutors for Math Mistake Remediation

MisEdu-RAG: A Misconception-Aware Dual-Hypergraph RAG for Novice Math Teachers

GRADE: Generalizable Reasoning-Aware Dialogue Evaluation for AI Tutors