cs.CLMar 3, 2026
SaveFrom Solver to Tutor: Evaluating the Pedagogical Intelligence of LLMs with KMP-Bench
Organizations: Multimedia Laboratory (MMLab), The Chinese University of Hong Kong · CPII under InnoHK · Shenzhen Loop Area Institute
Abstract
Large Language Models (LLMs) show significant potential in AI mathematical tutoring, yet current evaluations often rely on simplistic metrics or narrow pedagogical scenarios, failing to assess comprehensive, multi-turn teaching effectiveness. In this paper, we introduce KMP-Bench, a comprehensive K-8 Mathematical Pedagogical Benchmark designed to assess LLMs from two complementary perspectives. The first module, KMP-Dialogue, evaluates holistic pedagogical capabilities against six core principles (e.g., Challenge, Explanation, Feedback), leveraging a novel multi-turn dialogue dataset constructed by weaving together diverse pedagogical components. The second module, KMP-Skills, provides a granular assessment of foundational tutoring abilities, including multi-turn problem-solving, error detection and correction, and problem generation. Our evaluations on KMP-Bench reveal a key disparity: while leading LLMs excel at tasks with verifiable solutions, they struggle with the nuanced application of pedagogical principles. Additionally, we present KMP-Pile, a large-scale (150K) dialogue dataset. Models fine-tuned on KMP-Pile show substantial improvement on KMP-Bench, underscoring the value of pedagogically-rich training data for developing more effective AI math tutors.
Figures & tables
Figure 1: The curation pipeline for the KMP-Bench dataset. The process generates four distinct pedagogical components from curated K-8 problems using an LLM guided by human-crafted few-shot examples. After a rigorous quality control process (including LLM-based self-verification and model ensemble validation), these components are organized into Dialogue Flows, which are then manually verified for pedagogical soundness before being woven into final tutoring dialogues.
Figure 2: Statistical distributions of the dataset.
Figure 3: The evaluation framework of KMP-Dialogue. The process evaluates a model’s Tutor Response in the context of a full dialogue history and specific instructions (left panel). An LLM or human evaluator compares the Tutor Response against a Reference Response using the 4 general criteria and the criteria corresponding to the relevant pedagogical principle(s) to determine a final “Win”, “Tie”, or “Lose” outcome.
Figure 4: The evaluation framework for the Mathematical Problem Generation task in KMP-Skills.
| Model | Overall Judgement | General- Level Acc | Principle-Level Acc | Overall Acc | |||||
| Challenge | Explanation | Modelling | Practice | Questioning | Feedback | ||||
| Closed-Source Models | |||||||||
| GPT-4o | 60.7 | 44.5 | 43.4 | 62.2 | 57.5 | 47.5 | 59.9 | 41.3 | 48.2 |
| Gemini-2.0-Flash | 58.5 | 38.0 | 51.1 | 49.9 | 34.2 | 42.2 | 50.8 | 37.6 | 41.1 |
| Claude-3.7-Sonnet | 87.1 | 69.8 | 80.0 | 79.3 | 71.2 | 69.1 | 79.3 | 72.2 | 72.5 |
| LearnLM-2.0-Flash | 74.8 | 58.9 | 77.3 | 42.7 | 24.9 | 67.6 | 73.5 | 63.9 | 58.6 |
Table 1: Overall pedagogical performance of various LLMs in KMP-Dialogue. Tutor LLM responses are compared against reference responses from our curated dialogues. The best and second-best results among all the models are highlighted in red and blue, respectively. The row represents the improvement of KMP-LM-7B over Qwen2.5-Math-7B-Instruct.
| Model | Size | KMP- FQA | KMP-EC | KMP-Gen | |||
| MR-Score | F1-Score | Correction | Follow-up | Similar | |||
| Closed-Source Models | |||||||
| GPT-4o | - | 84.1 | 64.4 | 78.3 | 90.9 | 74.8 | 72.2 |
| GPT‑4o mini | - | 79.4 | 56.9 | 66.1 | 84.3 | 76.5 | 64.8 |
| Gemini 2.0 Flash | - | 91.2 | 86.3 | 87.5 | 96.7 | 75.0 | 73.2 |
| LearnLM-1.5-Pro | - | 87.7 | 69.8 | 74.6 | 94.4 | 77.0 | 75.0 |
Table 2: Foundational tutoring performance of various LLMs on KMP-Skills. Abbreviations: KMP-FQA denotes Multi-Turn Problem Solving, KMP-EC denotes Error Detection and Correction, and KMP-Gen denotes Problem Generation. The best and second-best results among all the models are highlighted in red and blue, respectively. The row represents the improvement of KMP-LM-7B over Qwen2.5-Math-7B-Instruct.
| Model | Grading Method | General-Level Acc | Principle-Level Acc | Overall Acc | Alignment Rate (%) |
| LearnLM-2.0-Flash | Gemini-2.0-Flash | 61.8 | 60.4 | 61.1 | 89.7 |
| Manual | 62.5 | 57.9 | 60.2 | ||
| Claude-3.7-Sonnet | Gemini-2.0-Flash | 72.5 | 75.7 | 74.1 | 87.1 |
| Manual | 72.9 | 80.5 | 76.7 | ||
| DeepSeek-V3 | Gemini-2.0-Flash | 75.0 | 75.5 | 75.2 | 92.5 |
| Manual | 75.8 | 76.5 | 76.1 |
Table 3: Comparison of evaluation results between our LLM evaluator (Gemini-2.0-Flash) and manual human annotation on a subset of KMP-Dialogue. The Alignment Rate measures the percentage of criteria where the LLM evaluator’s judgment aligns with the human ground truth.
Figure 5: The distribution of the task errors.
Figure 6: Distribution of the 6K seed problems used for the KMP-Skills evaluation benchmark. The diagram details the composition of the dataset by mathematical domain, grade level, and original source dataset.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The prompt used for the LLM-based cross-verification of seed math problems.
Figure 8: An example from our human-crafted few-shot prompts used to guide the LLM in generating the ‘Confusion Clarification’ component. It includes the problem context and three types of query-answer pairs (What, How, Why) that the model learns to emulate, targeting definitions, processes, and deeper rationales.
| Model | KMP-EC | |||
| Task1 | Task2 | Task3 | MR-Score | |
| Closed-Source Models | ||||
| GPT-4o | 64.0 | 67.6 | 62.5 | 64.4 |
| Gemini 2.0 Flash | 89.5 | 87.3 | 84.4 | 86.3 |
| LearnLM-1.5-Pro | 70.9 | 72.4 | 67.8 | 69.8 |
| Open-Source Models | ||||
Table 4: Detailed comparison of model performances on Error Detection and Correction task. Task1: MCC score for solution correctness, Task2: accuracy of the first-error-step prediction, Task3: accuracy of identifying the correct error-reason. The best and second-best results among all the models are highlighted in red and blue.
Figure 9: An example illustrating the truncation of a dialogue flow. The pedagogical interaction is deemed complete after turn 8. The line indicates the truncation point, after which subsequent turns consisting of conversational pleasantries are removed from the final dataset.
Figure 10: The prompt used to guide the LLM in generating a Knowledge-Extending Follow-up Question.
Figure 11: The instructional prompt guiding an LLM to perform detailed error analysis and generate a corrected solution for a student’s attempt at a math problem.
Figure 12: The prompt used to guide the LLM in generating Similar Practice Problems.
Figure 13: The detailed instructional prompt used to guide the LLM in generating multi-turn “Confusion Clarification” dialogues.
Figure 14: The detailed instructional prompt used to guide the LLM in generating a structured Dialogue Flow.
Figure 15: The detailed instructional prompt used to guide the LLM in generating a complete, natural language K-8 math tutoring dialogue.
Figure 16: The first part of the grading prompt used to guide an LLM (acting as an expert K-8 mathematics pedagogical evaluator) in the comparative assessment of a Tutor LLM’s response against a reference response.
Figure 17: The second part of the grading prompt used to guide an LLM (acting as an expert K-8 mathematics pedagogical evaluator) in the comparative assessment of a Tutor LLM’s response against a reference response.
Figure 18: Illustrative examples of the general evaluation criteria within the KMP-Dialogue framework.
Figure 19: Illustrative examples of Principle-Specific evaluation criteria for the Practice principle.
Figure 20: The grading prompt for the Mathematical Problem Generation task. It guides the LLM evaluator in its rigorous, fail-by-default assessment of generated problems and solutions.
| Model | Overall Judgement | General- Level Acc | Principle-Level Acc | Overall Acc | |||||
| Challenge | Explanation | Modelling | Practice | Questioning | Feedback | ||||
| Closed-Source Models | |||||||||
| GPT-4o | 60.7 | 44.5 | 43.4 | 62.2 | 57.5 | 47.5 | 59.9 | 41.3 | 48.2 |
| Gemini-2.0-Flash | 58.5 | 38.0 | 51.1 | 49.9 | 34.2 | 42.2 | 50.8 | 37.6 | 41.1 |
| Claude-3.7-Sonnet | 87.1 | 69.8 | 80.0 | 79.3 | 71.2 | 69.1 | 79.3 | 72.2 | 72.5 |
| LearnLM-2.0-Flash | 74.8 | 58.9 | 77.3 | 42.7 | 24.9 | 67.6 | 73.5 | 63.9 | 58.6 |
Table 5: Comprehensive pedagogical performance of various LLMs in KMP-Dialogue. Tutor LLM responses are compared against reference responses from our curated dialogues. The best and second-best results among all the models are highlighted in red and blue, respectively. The row represents the improvement of KMP-LM-7B over Qwen2.5-Math-7B-Instruct.
| Model | Size | KMP- FQA | KMP-EC | KMP-Gen | |||
| MR-Score | F1-Score | Correction | Follow-up | Similar | |||
| Closed-Source Models | |||||||
| GPT-4o | - | 84.1 | 64.4 | 78.3 | 90.9 | 74.8 | 72.2 |
| GPT‑4o mini | - | 79.4 | 56.9 | 66.1 | 84.3 | 76.5 | 64.8 |
| Gemini 2.0 Flash | - | 91.2 | 86.3 | 87.5 | 96.7 | 75.0 | 73.2 |
| LearnLM-1.5-Pro | - | 87.7 | 69.8 | 74.6 | 94.4 | 77.0 | 75.0 |
Table 6: Comprehensive foundational tutoring performance of various LLMs on KMP-Skills. Abbreviations: KMP-FQA denotes Multi-Turn Problem Solving, KMP-EC denotes Error Detection and Correction, and KMP-Gen denotes Problem Generation. The best and second-best results among all the models are highlighted in red and blue, respectively. The row represents the improvement of KMP-LM-7B over Qwen2.5-Math-7B-Instruct.
Figure 21: A case study illustrating a “Flawed Scaffolding Strategy.” The tutor, instructed to challenge the student, provides a new problem but also gives away the solution steps and a strong hint, thereby over-scaffolding and reducing the problem’s intended difficulty.
Figure 22: A case study illustrating the “Evasion by Substitution” error. The tutor was instructed to model an algebraic method after the student successfully solved the problem. Instead, it offered detailed feedback and a new problem, failing to perform the required pedagogical action.
Figure 23: A case study illustrating “Evasion by Vague Questioning.” The tutor was instructed to model the solution but instead evaded the task by asking the student to perform the required steps.
Figure 24: A case study illustrating the “Inappropriate Content Selection” error. The tutor models a solution using the abstract combinations formula, which is pedagogically less suitable than the reference response’s more intuitive, step-by-step probability approach.
Figure 25: A case study illustrating a “Factual Error.” The tutor attempts to model a solution but becomes confused, makes multiple logical and mathematical mistakes, and incorrectly concludes that no solution exists.
Figure 26: A case study illustrating the “Low-Quality Interaction” error. The tutor confirms the student’s insight but then asks an overly broad question. The reference response provides better scaffolding by breaking down the next steps, leading to a more effective pedagogical interaction.
More in this topic
Generative AI in Education
Generative AI in education uses language and multimodal models to support teaching, learning, feedback, assessment, and educational design.
See all 282 papers