cs.AIMar 26, 2026

Is Mathematical Problem-Solving Expertise in Large Language Models Associated with Assessment Performance?

Authors: Liang Zhang, Yu Fu, Xinyi Jin

Organizations: University of Michigan, Ann Arbor, MI, USA · New York, NY, USA · The High School Affiliated to Minzu University of China, Beijing, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Models (LLMs) are increasingly used in math education not only as problem solvers but also as assessors of learners' reasoning. However, it remains unclear whether stronger math problem-solving ability is associated with stronger step-level assessment performance. This study examines that relationship using the GSM8K and MATH subsets of PROCESSBENCH, a human-annotated benchmark for identifying the earliest erroneous step in mathematical reasoning. We evaluate two LLM-based math tutor agent settings, instantiated with GPT-4 and GPT-5, in two independent tasks on the same math problems: solving the original problem and assessing a benchmark-provided solution by predicting the earliest erroneous step. Results show a consistent within-model pattern: assessment accuracy is substantially higher on math problem items the same model solved correctly than on items it solved incorrectly, with statistically significant associations across both models and datasets. At the same time, assessment remains more difficult than direct problem solving, especially on error-present solutions. These findings suggest that math problem-solving expertise supports stronger assessment performance, but reliable step-level diagnosis also requires additional capabilities such as step tracking, monitoring, and precise error localization. The results have implications for the design and evaluation of AI-supported Adaptive Instructional Systems (AISs) for formative assessment in math education.

Figures & tables

Figure 1: Accuracy on problem-solving (final answer vs. gold answer) and step-level assessment (predicted earliest error step vs. human label), by dataset and model (mean over 3 runs).
Figure 2: Step-level assessment accuracy grouped by whether the same LLM model solved the item correctly or incorrectly, by dataset and model (mean over 3 runs).
ModelDatasetχ2\chi^{2}pp (Fisher)Diff (%)95% CI
GPT-4GSM8K39.385.35×10−125.35\times 10^{-12}42.0[35.1, 48.9]
MATH162.321.82×10−361.82\times 10^{-36}38.4[32.6, 44.2]
GPT-5GSM8K12.711.66×10−41.66\times 10^{-4}34.1[20.8, 47.4]
MATH224.338.28×10−518.28\times 10^{-51}45.9[40.4, 51.4]
Table 1: Within-model association between item-level problem-solving success and step-level assessment accuracy. The table reports χ2\chi^{2} statistics, Fisher’s exact test pp values, and differences in assessment accuracy (solved correct −- solved incorrect) with 95% confidence intervals (CI), pooled over three runs.
ModelDatasetAccuracy (error present)Accuracy (no error)F1
GPT-4GSM8K4.791.28.9
MATH10.072.917.5
GPT-5GSM8K4.897.19.2
MATH7.487.413.6
Table 2: Step-level assessment metrics by error presence: accuracy on error-present samples (human label ≠−1\neq-1 ), accuracy on no-error samples (human label =−1=-1 ), and their harmonic mean (F1), pooled over three runs.
ModelDatasetAssess F1 (%)Gap (Solve −- F1)
GPT-4GSM8K8.986.0
MATH17.512.3
GPT-5GSM8K9.288.2
MATH13.616.9
Table 3: Derived summary of step-level assessment difficulty, complementing Fig. 1 . The table reports assessment F1 and the solve–assess gap, computed as solving accuracy minus assessment F1, pooled over three runs.
More in this topic

LLM Mathematical Reasoning

Large language model mathematical reasoning studies how AI systems solve, explain, verify, and generalize across mathematical problems and proofs.

See all 269 papers

Explore similar work

CardsList
Mar 3, 2026

From Solver to Tutor: Evaluating the Pedagogical Intelligence of LLMs with KMP-Bench

Large Language Models (LLMs) show significant potential in AI mathematical tutoring, yet current evaluations often rely on simplistic metrics or narrow pedagogical scenarios, failing to assess comprehensive, multi-turn teaching effectiveness. In this paper, we introduce KMP-Bench, a comprehensive K-8 Mathematical Pedagogical Benchmark designed to assess LLMs from two complementary perspectives. The first module, KMP-Dialogue, evaluates holistic pedagogical capabilities against six core principles (e.g., Challenge, Explanation, Feedback), leveraging a novel multi-turn dialogue dataset constructed by weaving together diverse pedagogical components. The second module, KMP-Skills, provides a granular assessment of foundational tutoring abilities, including multi-turn problem-solving, error detection and correction, and problem generation. Our evaluations on KMP-Bench reveal a key disparity: while leading LLMs excel at tasks with verifiable solutions, they struggle with the nuanced application of pedagogical principles. Additionally, we present KMP-Pile, a large-scale (150K) dialogue dataset. Models fine-tuned on KMP-Pile show substantial improvement on KMP-Bench, underscoring the value of pedagogically-rich training data for developing more effective AI math tutors.

Weikang Shi, Houxing Ren, Junting Pan +8

Multimedia Laboratory (MMLab), The Chinese University of Hong Kong · CPII under InnoHK · Shenzhen Loop Area Institute

Generative AI in EducationIntelligent Tutoring SystemsLLM EvaluationMathematical Reasoning Benchmarks
AskPDF
Apr 6, 2026

How Far Are We? Systematic Evaluation of LLMs vs. Human Experts in Mathematical Contest in Modeling

Large language models (LLMs) have achieved strong performance on reasoning benchmarks, yet their ability to solve real-world problems requiring end-to-end workflows remains unclear. Mathematical modeling competitions provide a stringent testbed for evaluating such end-to-end problem-solving capability. We propose a problem-oriented, stage-wise evaluation framework that assesses LLM performance across modeling stages using expert-verified criteria. We validate the framework's reliability by comparing automatic scores with independent human expert judgments on problems from the China Postgraduate Mathematical Contest in Modeling, demonstrating substantially stronger alignment than existing evaluation schemes. Using this framework, we reveal a comprehension-execution gap in state-of-the-art LLMs: while they perform well in early stages such as problem identification and formulation, they exhibit persistent deficiencies in execution-oriented stages including model solving, code implementation, and result analysis. These gaps persist even with increased model scale. We further trace these failures to insufficient specification, missing verification, and lack of validation, with errors propagating across stages without correction. Our findings suggest that bridging this gap requires approaches beyond model scaling, offering insights for applying LLMs to complex real-world problem solving.

Yuhang Liu, Heyan Huang, Yizhe Yang +3

Beijing Institute of Technology, Beijing, China · Southeast Academy of Information Technology, Putian, China

LLM EvaluationLLM Mathematical Reasoning
AskPDF
Sep 26, 2026

Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning

Large language models (LLMs) can solve every intermediate step of a multi-step math problem on its own and still fail the full problem, even when given a roadmap of the steps and all of their answers. We introduce OracleLadder, a diagnostic evaluation that locates where LLM math reasoning fails by giving the model increasing levels of oracle help. For each problem, a teacher model writes a fixed roadmap of intermediate sub-goals (milestones), and a deterministic symbolic verifier grades every answer. Testing the model with no help, with the roadmap, with the roadmap plus the milestone answers, and on each milestone alone sorts each failure into one of five reasoning gaps. On 354 NuminaMath problems and six models from 8B to 671B parameters (Qwen3, gpt-oss, Llama 3.3, DeepSeek-V3.1), the largest gap for every model is the composition gap, a stricter form of the compositionality gap. It covers 33-48% of problems, and 24-37% after removing problems that an LLM review flags as grading errors. Accuracy and milestone-help recovery rank the two strongest models differently, and two RLVR runs with similar accuracy gains move problems differently. The roadmap effect replicates on MATH500 and AIME 2024/25, per-problem recovery agrees for 83-87% of problems under an independent second teacher, and the help ladder carries over to code generation. We release the data, roadmaps, prompts, and code at https://github.com/slark-prime/OracleLadder.

Zhuohan Wang, Haoran Ma, Tianyu Wu +3

Harvard University · Rutgers University · The Hong Kong University of Science and Technology

LLM Mathematical ReasoningLLM EvaluationMathematical Reasoning Benchmarks
AskPDF