Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math
Authors: Dingjie Song, Tianlong Xu, Yi-Fan Zhang, Hang Li, Zhiling Yan, Xing Fan, Haoyang Li, Lichao Sun, +1 more
Organizations: Lehigh University, Bethlehem, PA, USA · Squirrel Ai Learning, Bellevue, WA, USA · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Michigan State University, East Lansing, MI, USA · Squirrel Ai Learning, Shanghai, China
Assessing student handwritten scratchwork is crucial for personalized educational feedback but presents unique challenges due to diverse handwriting, complex layouts, and varied problem-solving approaches. Existing educational NLP primarily focuses on textual responses and neglects the complexity and multimodality inherent in authentic handwritten scratchwork. Current multimodal large language models (MLLMs) excel at visual reasoning but typically adopt an "examinee perspective", prioritizing generating correct answers rather than diagnosing student errors. To bridge these gaps, we introduce ScratchMath, a novel benchmark specifically designed for explaining and classifying errors in authentic handwritten mathematics scratchwork. Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students, supporting two key tasks: Error Cause Explanation (ECE) and Error Cause Classification (ECC), with seven defined error types. The dataset is meticulously annotated through rigorous human-machine collaborative approaches involving multiple stages of expert labeling, review, and verification. We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts, especially in visual recognition and logical reasoning. Proprietary models notably outperform open-source models, with large reasoning models showing strong potential for error explanation. All evaluation data and frameworks are publicly available to facilitate further research.
Figures & tables
Figure 1: Overview of this work. (Top) An example illustrating two tasks proposed. (Bottom) Summary of the three research questions addressed in this study.
Figure 2: Overview of ScratchMath (with models’ answer and human labeled answer translated in English). It illustrates dataset structure, tasks (ECE and ECC), multimodal model predictions, and expert human annotations.
Figure 3: An overview of the ScratchMath benchmark construction pipeline.
Practical math, unit conversions, financial calculations. e.g., Cost calculations, Average speed
Statistics and Probability
1.9
4.0
Data collection, probability estimation. e.g., Statistical tables, Frequency probability
Table 2: Distribution and Examples of Mathematical Topics by Category in ScratchMath (translated in English)
Model
#Params
Error Cause Explanation
Error Cause Classification
Average
Avg Rank
Primary
Middle
Rank
Primary
Middle
Rank
Random Guessing
–
0.0
0.0
–
12.5
12.5
–
6.25
–
Human Performance
–
89.3
86.2
–
78.4
81.5
–
83.9
–
Proprietary Models
Gemini 2.0 Flash
–
52.2
46.9
4
38.6
49.0
2
46.7
3
Gemini 2.0 Flash Lite
–
36.0
32.0
8
34.6
46.5
5
37.2
5
Table 3: Performance of state-of-the-art MLLMs on ScratchMath . We use weighted-average accuracy for the ECC task. Boldface and underline mark the best and second-best results for each metric, reported separately for proprietary vs. open-source groups. † : activated parameters of MoE model. ∗ indicates reasoning models.
Scratchwork
Problem Statement
Ref. Ans.
Stu. Ans.
Ground Truth
Model Prediction
Visual Recognition Failure
Solve for x : 4x−3(20−x)=6x−7(9−x)
21
−21
Student solved −6x=−3 as x=−21 instead of 21 .
…Finally x=−6−3=21 . However, the student miscalculated −63+60 as +3 at this step, so they wrote −6x=3 , leading to x=−63=−21 .
Formatting Misinterpretation
If the two roots of a(x+m)2+b=0 are −1 and 4 , then the solutions to a(x+m−3)2+b=0 are x= .
2 or 7
2 and 7
Formatting error (“and” vs. “or”); also mis-shifted the quadratic by m .
… The student, in their scratchpad, mistakenly treated the second equation a⋅(x+m−3)2+b=0 as a⋅(x−3)2+b=0 . They then directly set x−3=−1 and x−3=4 , thus obtaining x=2,7 …
Misaligned Misinterpretation
Brick dimensions 20×11×6 cm, density 1.5 g/cm 3 . Find the weight in kilograms.
1.98
1980
Answered in grams without converting to kg.
…When calculating by hand, he even converted " 1320×1.5 " to " 132×15 " , and although the result was still 1980 , the division by 10 and multiplication by 10 in the process didn’t correspond to the units …
Table 4: Examples (translated in English) illustrating typical model error types in ECE. “Ref. Ans.” and “Stu. Ans.” denote the reference and student answers, respectively. Bold text highlights the erroneous parts of the prediction.
Figure 5: Failure Cases Distribution of o4-mini on ECE
Figure 6: Models’ Performance on different ECC classes