cs.AIMar 26, 2026

Is Mathematical Problem-Solving Expertise in Large Language Models Associated with Assessment Performance?

Authors: Liang Zhang, Yu Fu, Xinyi Jin

Organizations: University of Michigan, Ann Arbor, MI, USA · New York, NY, USA · The High School Affiliated to Minzu University of China, Beijing, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Models (LLMs) are increasingly used in math education not only as problem solvers but also as assessors of learners' reasoning. However, it remains unclear whether stronger math problem-solving ability is associated with stronger step-level assessment performance. This study examines that relationship using the GSM8K and MATH subsets of PROCESSBENCH, a human-annotated benchmark for identifying the earliest erroneous step in mathematical reasoning. We evaluate two LLM-based math tutor agent settings, instantiated with GPT-4 and GPT-5, in two independent tasks on the same math problems: solving the original problem and assessing a benchmark-provided solution by predicting the earliest erroneous step. Results show a consistent within-model pattern: assessment accuracy is substantially higher on math problem items the same model solved correctly than on items it solved incorrectly, with statistically significant associations across both models and datasets. At the same time, assessment remains more difficult than direct problem solving, especially on error-present solutions. These findings suggest that math problem-solving expertise supports stronger assessment performance, but reliable step-level diagnosis also requires additional capabilities such as step tracking, monitoring, and precise error localization. The results have implications for the design and evaluation of AI-supported Adaptive Instructional Systems (AISs) for formative assessment in math education.

Figures & tables

Figure 1: Accuracy on problem-solving (final answer vs. gold answer) and step-level assessment (predicted earliest error step vs. human label), by dataset and model (mean over 3 runs).
Figure 2: Step-level assessment accuracy grouped by whether the same LLM model solved the item correctly or incorrectly, by dataset and model (mean over 3 runs).
ModelDatasetχ2\chi^{2}pp (Fisher)Diff (%)95% CI
GPT-4GSM8K39.385.35×10−125.35\times 10^{-12}42.0[35.1, 48.9]
MATH162.321.82×10−361.82\times 10^{-36}38.4[32.6, 44.2]
GPT-5GSM8K12.711.66×10−41.66\times 10^{-4}34.1[20.8, 47.4]
MATH224.338.28×10−518.28\times 10^{-51}45.9[40.4, 51.4]
Table 1: Within-model association between item-level problem-solving success and step-level assessment accuracy. The table reports χ2\chi^{2} statistics, Fisher’s exact test pp values, and differences in assessment accuracy (solved correct −- solved incorrect) with 95% confidence intervals (CI), pooled over three runs.
ModelDatasetAccuracy (error present)Accuracy (no error)F1
GPT-4GSM8K4.791.28.9
MATH10.072.917.5
GPT-5GSM8K4.897.19.2
MATH7.487.413.6
Table 2: Step-level assessment metrics by error presence: accuracy on error-present samples (human label ≠−1\neq-1 ), accuracy on no-error samples (human label =−1=-1 ), and their harmonic mean (F1), pooled over three runs.
ModelDatasetAssess F1 (%)Gap (Solve −- F1)
GPT-4GSM8K8.986.0
MATH17.512.3
GPT-5GSM8K9.288.2
MATH13.616.9
Table 3: Derived summary of step-level assessment difficulty, complementing Fig. 1 . The table reports assessment F1 and the solve–assess gap, computed as solving accuracy minus assessment F1, pooled over three runs.
More in this topic

LLM Mathematical Reasoning

Large language model mathematical reasoning studies how AI systems solve, explain, verify, and generalize across mathematical problems and proofs.

See all 269 papers

Explore similar work

CardsList

  1. From Solver to Tutor: Evaluating the Pedagogical Intelligence of LLMs with KMP-Bench

    Mar 3, 2026Weikang Shi, Houxing Ren, Junting Pan +8Generative AI in EducationIntelligent Tutoring Systems

    Multimedia Laboratory (MMLab), The Chinese University of Hong Kong · CPII under InnoHK · Shenzhen Loop Area Institute

  2. How Far Are We? Systematic Evaluation of LLMs vs. Human Experts in Mathematical Contest in Modeling

    Apr 6, 2026Yuhang Liu, Heyan Huang, Yizhe Yang +3LLM EvaluationLLM Mathematical Reasoning

    Beijing Institute of Technology, Beijing, China · Southeast Academy of Information Technology, Putian, China

  3. Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning

    Sep 26, 2026Zhuohan Wang, Haoran Ma, Tianyu Wu +3LLM Mathematical ReasoningLLM Evaluation

    Harvard University · Rutgers University · The Hong Kong University of Science and Technology