Large Language Models (LLMs) show significant potential in AI mathematical tutoring, yet current evaluations often rely on simplistic metrics or narrow pedagogical scenarios, failing to assess comprehensive, multi-turn teaching effectiveness. In this paper, we introduce KMP-Bench, a comprehensive K-8 Mathematical Pedagogical Benchmark designed to assess LLMs from two complementary perspectives. The first module, KMP-Dialogue, evaluates holistic pedagogical capabilities against six core principles (e.g., Challenge, Explanation, Feedback), leveraging a novel multi-turn dialogue dataset constructed by weaving together diverse pedagogical components. The second module, KMP-Skills, provides a granular assessment of foundational tutoring abilities, including multi-turn problem-solving, error detection and correction, and problem generation. Our evaluations on KMP-Bench reveal a key disparity: while leading LLMs excel at tasks with verifiable solutions, they struggle with the nuanced application of pedagogical principles. Additionally, we present KMP-Pile, a large-scale (150K) dialogue dataset. Models fine-tuned on KMP-Pile show substantial improvement on KMP-Bench, underscoring the value of pedagogically-rich training data for developing more effective AI math tutors.
Figures & tables
Figure 1: The curation pipeline for the KMP-Bench dataset. The process generates four distinct pedagogical components from curated K-8 problems using an LLM guided by human-crafted few-shot examples. After a rigorous quality control process (including LLM-based self-verification and model ensemble validation), these components are organized into Dialogue Flows, which are then manually verified for pedagogical soundness before being woven into final tutoring dialogues.
Figure 2: Statistical distributions of the dataset.
Figure 3: The evaluation framework of KMP-Dialogue. The process evaluates a model’s Tutor Response in the context of a full dialogue history and specific instructions (left panel). An LLM or human evaluator compares the Tutor Response against a Reference Response using the 4 general criteria and the criteria corresponding to the relevant pedagogical principle(s) to determine a final “Win”, “Tie”, or “Lose” outcome.
Figure 4: The evaluation framework for the Mathematical Problem Generation task in KMP-Skills.
Model
Overall Judgement
General- Level Acc
Principle-Level Acc
Overall Acc
Challenge
Explanation
Modelling
Practice
Questioning
Feedback
Closed-Source Models
GPT-4o
60.7
44.5
43.4
62.2
57.5
47.5
59.9
41.3
48.2
Gemini-2.0-Flash
58.5
38.0
51.1
49.9
34.2
42.2
50.8
37.6
41.1
Claude-3.7-Sonnet
87.1
69.8
80.0
79.3
71.2
69.1
79.3
72.2
72.5
LearnLM-2.0-Flash
74.8
58.9
77.3
42.7
24.9
67.6
73.5
63.9
58.6
Table 1: Overall pedagogical performance of various LLMs in KMP-Dialogue. Tutor LLM responses are compared against reference responses from our curated dialogues. The best and second-best results among all the models are highlighted in red and blue, respectively. The Δ row represents the improvement of KMP-LM-7B over Qwen2.5-Math-7B-Instruct.
Model
Size
KMP- FQA
KMP-EC
KMP-Gen
MR-Score
F1-Score
Correction
Follow-up
Similar
Closed-Source Models
GPT-4o
-
84.1
64.4
78.3
90.9
74.8
72.2
GPT‑4o mini
-
79.4
56.9
66.1
84.3
76.5
64.8
Gemini 2.0 Flash
-
91.2
86.3
87.5
96.7
75.0
73.2
LearnLM-1.5-Pro
-
87.7
69.8
74.6
94.4
77.0
75.0
Table 2: Foundational tutoring performance of various LLMs on KMP-Skills. Abbreviations: KMP-FQA denotes Multi-Turn Problem Solving, KMP-EC denotes Error Detection and Correction, and KMP-Gen denotes Problem Generation. The best and second-best results among all the models are highlighted in red and blue, respectively. The Δ row represents the improvement of KMP-LM-7B over Qwen2.5-Math-7B-Instruct.
Model
Grading Method
General-Level Acc
Principle-Level Acc
Overall Acc
Alignment Rate (%)
LearnLM-2.0-Flash
Gemini-2.0-Flash
61.8
60.4
61.1
89.7
Manual
62.5
57.9
60.2
Claude-3.7-Sonnet
Gemini-2.0-Flash
72.5
75.7
74.1
87.1
Manual
72.9
80.5
76.7
DeepSeek-V3
Gemini-2.0-Flash
75.0
75.5
75.2
92.5
Manual
75.8
76.5
76.1
Table 3: Comparison of evaluation results between our LLM evaluator (Gemini-2.0-Flash) and manual human annotation on a subset of KMP-Dialogue. The Alignment Rate measures the percentage of criteria where the LLM evaluator’s judgment aligns with the human ground truth.
Figure 5: The distribution of the task errors.
Figure 6: Distribution of the 6K seed problems used for the KMP-Skills evaluation benchmark. The diagram details the composition of the dataset by mathematical domain, grade level, and original source dataset.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The prompt used for the LLM-based cross-verification of seed math problems.
Figure 8: An example from our human-crafted few-shot prompts used to guide the LLM in generating the ‘Confusion Clarification’ component. It includes the problem context and three types of query-answer pairs (What, How, Why) that the model learns to emulate, targeting definitions, processes, and deeper rationales.
Model
KMP-EC
Task1
Task2
Task3
MR-Score
Closed-Source Models
GPT-4o
64.0
67.6
62.5
64.4
Gemini 2.0 Flash
89.5
87.3
84.4
86.3
LearnLM-1.5-Pro
70.9
72.4
67.8
69.8
Open-Source Models
Appendix
Table 4: Detailed comparison of model performances on Error Detection and Correction task. Task1: MCC score for solution correctness, Task2: accuracy of the first-error-step prediction, Task3: accuracy of identifying the correct error-reason. The best and second-best results among all the models are highlighted in red and blue.
Figure 9: An example illustrating the truncation of a dialogue flow. The pedagogical interaction is deemed complete after turn 8. The line indicates the truncation point, after which subsequent turns consisting of conversational pleasantries are removed from the final dataset.
Figure 10: The prompt used to guide the LLM in generating a Knowledge-Extending Follow-up Question.
Figure 11: The instructional prompt guiding an LLM to perform detailed error analysis and generate a corrected solution for a student’s attempt at a math problem.
Figure 12: The prompt used to guide the LLM in generating Similar Practice Problems.
Figure 13: The detailed instructional prompt used to guide the LLM in generating multi-turn “Confusion Clarification” dialogues.
Figure 14: The detailed instructional prompt used to guide the LLM in generating a structured Dialogue Flow.
Figure 15: The detailed instructional prompt used to guide the LLM in generating a complete, natural language K-8 math tutoring dialogue.
Figure 16: The first part of the grading prompt used to guide an LLM (acting as an expert K-8 mathematics pedagogical evaluator) in the comparative assessment of a Tutor LLM’s response against a reference response.
Figure 17: The second part of the grading prompt used to guide an LLM (acting as an expert K-8 mathematics pedagogical evaluator) in the comparative assessment of a Tutor LLM’s response against a reference response.
Figure 18: Illustrative examples of the general evaluation criteria within the KMP-Dialogue framework.
Figure 19: Illustrative examples of Principle-Specific evaluation criteria for the Practice principle.
Figure 20: The grading prompt for the Mathematical Problem Generation task. It guides the LLM evaluator in its rigorous, fail-by-default assessment of generated problems and solutions.
Model
Overall Judgement
General- Level Acc
Principle-Level Acc
Overall Acc
Challenge
Explanation
Modelling
Practice
Questioning
Feedback
Closed-Source Models
GPT-4o
60.7
44.5
43.4
62.2
57.5
47.5
59.9
41.3
48.2
Gemini-2.0-Flash
58.5
38.0
51.1
49.9
34.2
42.2
50.8
37.6
41.1
Claude-3.7-Sonnet
87.1
69.8
80.0
79.3
71.2
69.1
79.3
72.2
72.5
LearnLM-2.0-Flash
74.8
58.9
77.3
42.7
24.9
67.6
73.5
63.9
58.6
Appendix
Table 5: Comprehensive pedagogical performance of various LLMs in KMP-Dialogue. Tutor LLM responses are compared against reference responses from our curated dialogues. The best and second-best results among all the models are highlighted in red and blue, respectively. The Δ row represents the improvement of KMP-LM-7B over Qwen2.5-Math-7B-Instruct.
Model
Size
KMP- FQA
KMP-EC
KMP-Gen
MR-Score
F1-Score
Correction
Follow-up
Similar
Closed-Source Models
GPT-4o
-
84.1
64.4
78.3
90.9
74.8
72.2
GPT‑4o mini
-
79.4
56.9
66.1
84.3
76.5
64.8
Gemini 2.0 Flash
-
91.2
86.3
87.5
96.7
75.0
73.2
LearnLM-1.5-Pro
-
87.7
69.8
74.6
94.4
77.0
75.0
Appendix
Table 6: Comprehensive foundational tutoring performance of various LLMs on KMP-Skills. Abbreviations: KMP-FQA denotes Multi-Turn Problem Solving, KMP-EC denotes Error Detection and Correction, and KMP-Gen denotes Problem Generation. The best and second-best results among all the models are highlighted in red and blue, respectively. The Δ row represents the improvement of KMP-LM-7B over Qwen2.5-Math-7B-Instruct.
Figure 21: A case study illustrating a “Flawed Scaffolding Strategy.” The tutor, instructed to challenge the student, provides a new problem but also gives away the solution steps and a strong hint, thereby over-scaffolding and reducing the problem’s intended difficulty.
Figure 22: A case study illustrating the “Evasion by Substitution” error. The tutor was instructed to model an algebraic method after the student successfully solved the problem. Instead, it offered detailed feedback and a new problem, failing to perform the required pedagogical action.
Figure 23: A case study illustrating “Evasion by Vague Questioning.” The tutor was instructed to model the solution but instead evaded the task by asking the student to perform the required steps.
Figure 24: A case study illustrating the “Inappropriate Content Selection” error. The tutor models a solution using the abstract combinations formula, which is pedagogically less suitable than the reference response’s more intuitive, step-by-step probability approach.
Figure 25: A case study illustrating a “Factual Error.” The tutor attempts to model a solution but becomes confused, makes multiple logical and mathematical mistakes, and incorrectly concludes that no solution exists.
Figure 26: A case study illustrating the “Low-Quality Interaction” error. The tutor confirms the student’s insight but then asks an overly broad question. The reference response provides better scaffolding by breaking down the next steps, leading to a more effective pedagogical interaction.
Aligning LLMs for math tutoring typically requires RL-based training with multi-GPU infrastructure. We investigate whether training-free prompt optimization-evolving only the system prompt via API calls-can serve as a practical alternative. We adapt 7 published methods and propose 5 education-specialized methods, evaluating these 12 methods under 5 conditions on 2 OOD benchmark suites. All 12 best-per-method configurations surpass the strongest RL-trained baseline (R_total = 0.633), and our ParetoGrad achieves the best Pareto balance across post-test solve rate, leak control, and helpfulness, rather than dominating any single component. Behavioral analysis with an 82-code educational codebook reveals that training-free methods rely on teaching-knowledge patterns at 2-3x the rate of RL-trained models, with a compensating ~10 percentage-point reduction in intent-level scaffolding. We also find a task-dependent reasoning mode effect consistent across training-free and RL-based paradigms. Our approach enables efficient development of pedagogically aligned LLM tutors with prompts alone and minimal compute.
Unggi Lee, Minchul Shin, Yeil Jeong +5
Korea University Sejong Campus · 1Korea University Sejong Campus · Gyeonggi Institute of Education +9
Large Language Models (LLMs) are increasingly used in math education not only as problem solvers but also as assessors of learners' reasoning. However, it remains unclear whether stronger math problem-solving ability is associated with stronger step-level assessment performance. This study examines that relationship using the GSM8K and MATH subsets of PROCESSBENCH, a human-annotated benchmark for identifying the earliest erroneous step in mathematical reasoning. We evaluate two LLM-based math tutor agent settings, instantiated with GPT-4 and GPT-5, in two independent tasks on the same math problems: solving the original problem and assessing a benchmark-provided solution by predicting the earliest erroneous step. Results show a consistent within-model pattern: assessment accuracy is substantially higher on math problem items the same model solved correctly than on items it solved incorrectly, with statistically significant associations across both models and datasets. At the same time, assessment remains more difficult than direct problem solving, especially on error-present solutions. These findings suggest that math problem-solving expertise supports stronger assessment performance, but reliable step-level diagnosis also requires additional capabilities such as step tracking, monitoring, and precise error localization. The results have implications for the design and evaluation of AI-supported Adaptive Instructional Systems (AISs) for formative assessment in math education.
Liang Zhang, Yu Fu, Xinyi Jin
University of Michigan, Ann Arbor, MI, USA · New York, NY, USA · The High School Affiliated to Minzu University of China, Beijing, China
Large language models have strong potential for use in intelligent tutoring systems, but they often fail to follow effective pedagogical strategies, such as guiding students without revealing final answers. We study the application of a two-stage alignment pipeline for math mistake remediation, combining supervised fine-tuning on tutoring dialogs with Direct Preference Optimization on synthetic preference pairs. We construct a dataset that integrates existing tutoring corpora with synthetic data generated along pedagogical dimensions, such as scaffolding and factuality, and study different input configurations that incorporate solution correctness and gold answers. Experiments show that this approach improves both factual accuracy and pedagogical quality over base models and existing tutoring models. Human evaluation further indicates that our best model is competitive with a strong proprietary baseline, while providing additional benefits in terms of openness, transparency, and reproducibility. Our results highlight the effectiveness of preference-based pedagogical alignment, while also revealing challenges in reliably evaluating tutoring quality.
Kseniia Petukhova, Tien Dat Nguyen, Ekaterina Kochmar
Mohamed bin Zayed University of Artificial Intelligence