Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math
Authors: Dingjie Song, Tianlong Xu, Yi-Fan Zhang, Hang Li, Zhiling Yan, Xing Fan, Haoyang Li, Lichao Sun, +1 more
Organizations: Lehigh University, Bethlehem, PA, USA · Squirrel Ai Learning, Bellevue, WA, USA · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Michigan State University, East Lansing, MI, USA · Squirrel Ai Learning, Shanghai, China
Assessing student handwritten scratchwork is crucial for personalized educational feedback but presents unique challenges due to diverse handwriting, complex layouts, and varied problem-solving approaches. Existing educational NLP primarily focuses on textual responses and neglects the complexity and multimodality inherent in authentic handwritten scratchwork. Current multimodal large language models (MLLMs) excel at visual reasoning but typically adopt an "examinee perspective", prioritizing generating correct answers rather than diagnosing student errors. To bridge these gaps, we introduce ScratchMath, a novel benchmark specifically designed for explaining and classifying errors in authentic handwritten mathematics scratchwork. Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students, supporting two key tasks: Error Cause Explanation (ECE) and Error Cause Classification (ECC), with seven defined error types. The dataset is meticulously annotated through rigorous human-machine collaborative approaches involving multiple stages of expert labeling, review, and verification. We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts, especially in visual recognition and logical reasoning. Proprietary models notably outperform open-source models, with large reasoning models showing strong potential for error explanation. All evaluation data and frameworks are publicly available to facilitate further research.
Figures & tables
Figure 1: Overview of this work. (Top) An example illustrating two tasks proposed. (Bottom) Summary of the three research questions addressed in this study.
Figure 2: Overview of ScratchMath (with models’ answer and human labeled answer translated in English). It illustrates dataset structure, tasks (ECE and ECC), multimodal model predictions, and expert human annotations.
Figure 3: An overview of the ScratchMath benchmark construction pipeline.
Practical math, unit conversions, financial calculations. e.g., Cost calculations, Average speed
Statistics and Probability
1.9
4.0
Data collection, probability estimation. e.g., Statistical tables, Frequency probability
Table 2: Distribution and Examples of Mathematical Topics by Category in ScratchMath (translated in English)
Model
#Params
Error Cause Explanation
Error Cause Classification
Average
Avg Rank
Primary
Middle
Rank
Primary
Middle
Rank
Random Guessing
–
0.0
0.0
–
12.5
12.5
–
6.25
–
Human Performance
–
89.3
86.2
–
78.4
81.5
–
83.9
–
Proprietary Models
Gemini 2.0 Flash
–
52.2
46.9
4
38.6
49.0
2
46.7
3
Gemini 2.0 Flash Lite
–
36.0
32.0
8
34.6
46.5
5
37.2
5
Table 3: Performance of state-of-the-art MLLMs on ScratchMath . We use weighted-average accuracy for the ECC task. Boldface and underline mark the best and second-best results for each metric, reported separately for proprietary vs. open-source groups. † : activated parameters of MoE model. ∗ indicates reasoning models.
Scratchwork
Problem Statement
Ref. Ans.
Stu. Ans.
Ground Truth
Model Prediction
Visual Recognition Failure
Solve for x : 4x−3(20−x)=6x−7(9−x)
21
−21
Student solved −6x=−3 as x=−21 instead of 21 .
…Finally x=−6−3=21 . However, the student miscalculated −63+60 as +3 at this step, so they wrote −6x=3 , leading to x=−63=−21 .
Formatting Misinterpretation
If the two roots of a(x+m)2+b=0 are −1 and 4 , then the solutions to a(x+m−3)2+b=0 are x= .
2 or 7
2 and 7
Formatting error (“and” vs. “or”); also mis-shifted the quadratic by m .
… The student, in their scratchpad, mistakenly treated the second equation a⋅(x+m−3)2+b=0 as a⋅(x−3)2+b=0 . They then directly set x−3=−1 and x−3=4 , thus obtaining x=2,7 …
Misaligned Misinterpretation
Brick dimensions 20×11×6 cm, density 1.5 g/cm 3 . Find the weight in kilograms.
1.98
1980
Answered in grams without converting to kg.
…When calculating by hand, he even converted " 1320×1.5 " to " 132×15 " , and although the result was still 1980 , the division by 10 and multiplication by 10 in the process didn’t correspond to the units …
Table 4: Examples (translated in English) illustrating typical model error types in ECE. “Ref. Ans.” and “Stu. Ans.” denote the reference and student answers, respectively. Bold text highlights the erroneous parts of the prediction.
Figure 5: Failure Cases Distribution of o4-mini on ECE
Figure 6: Models’ Performance on different ECC classes
Accurate transcription of handwritten mathematics is crucial for educational AI systems, yet current benchmarks fail to evaluate this capability properly. Most prior studies focus on single-line expressions and rely on lexical metrics such as BLEU, which fail to assess the semantic reasoning across multi-line student solutions. In this paper, we present the first systematic study of multi-line handwritten math Optical Character Recognition (OCR), revealing a critical failure mode of Vision-Language Models (VLMs): over-correction. Instead of faithfully transcribing a student's work, these models often "fix" errors, thereby hiding the very mistakes an educational assessment aims to detect. To address this, we propose PINK (Penalized INK-based score), a semantic evaluation metric that leverages a Large Language Model (LLM) for rubric-based grading and explicitly penalizes over-correction. Our comprehensive evaluation of 15 state-of-the-art VLMs on the FERMAT dataset reveals substantial ranking reversals compared to BLEU: models like GPT-4o are heavily penalized for aggressive over-correction, whereas Gemini 2.5 Flash emerges as the most faithful transcriber. Furthermore, human expert studies show that PINK aligns significantly better with human judgment (55.0% preference over BLEU's 39.5%), providing a more reliable evaluation framework for handwritten math OCR in educational settings.
Jin Seong, Wencke Liermann, Minho Kim +2
Electronics and Telecommunications Research Institute, Republic of Korea
Automated grading systems have enabled scalable assessment for many response types, but handwritten mathematics remains a barrier due to the complexity of multi-step solutions. Vision-capable large language models (LLMs) offer new opportunities here, yet their reliability in authentic instructional settings remains poorly understood. We present an empirical evaluation of an LLM-based grader for handwritten mathematical work using instructor-defined rubrics. Extending a prior pipeline for typed responses, we integrate transcription and rubric-based evaluation of photographic submissions within a single LLM call, evaluating on student work from two university STEM courses. Comparing AI grading decisions against human-assigned ground truth at the rubric-item level, we observe high overall accuracy, with most errors -- 87% in the best model -- attributable to transcription failures rather than rubric misapplication. We categorize common error modes, including image quality issues, hallucinated content, and incorrect handling of equivalent expressions. These findings highlight both the promise and limitations of LLM-based grading for handwritten mathematics, providing guidance for system design, prompt refinement, and deployment in educational settings.
Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to faithfully read real handwriting remains underexplored. Existing OCR benchmarks focus largely on printed text or clean single-line inputs, leaving limited coverage of realistic handwritten OCR scenarios such as multilingual handwriting, writer errors, and structurally complex mathematical expressions. We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR. It covers handwritten text recognition and handwritten mathematical expression recognition across six subtasks and twelve subsets, totaling 77.57K labeled images from public datasets and newly collected student writings. A key component is a difficulty-stratified multi-line formula corpus designed to test robustness under increasing structural complexity. We evaluate thirteen open- and closed-source systems with five complementary metrics under a unified protocol. Results show that current systems remain far from faithful transcription: performance drops sharply on complex multi-line formulas, model rankings vary across language and formula settings, and several generative models hallucinate plausible but visually unsupported corrections. OmniHandwritingOCR provides a challenging testbed for diagnosing language, content, structural, and visual-grounding failure modes of multimodal models in handwritten OCR scenarios.