cs.CLMar 1, 2026

The Aftermath of DrawEduMath: Vision Language Models Underperform with Struggling Students and Misdiagnose Errors

Authors: Li Lucy, Albert Zhang, Nathan Anderson, Ryan Knight, Kyle Lo

Organizations: University of Washington · Insource Services · Worcester Polytechnic Institute · Allen Institute for AI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Effective mathematics education requires identifying and responding to students' mistakes. For AI to support pedagogical applications, models must perform well across different levels of student proficiency. Our work provides an extensive, year-long snapshot of how 11 vision-language models (VLMs) perform on DrawEduMath, a QA benchmark involving real students' handwritten, hand-drawn responses to math problems. We find that models' weaknesses concentrate on a core component of math education: student error. All evaluated VLMs underperform when describing work from students who require more pedagogical help, and across all QA, they struggle the most on questions related to assessing student error. Thus, while VLMs may be optimized to be math problem solving experts, our results suggest that they require alternative development incentives to adequately support educational use cases.

Figures & tables

Figure 1: On the left is a math problem, where students are asked to draw x<5/2x<5/2 on a number line. The right side shows two example student responses that differ in correctness. DrawEduMath pairs each math problem with one student response, and prompts VLMs to answer questions about the student response.
Figure 2: VLMs consistently perform worse on answering DrawEduMath benchmark questions pertaining to erroneous student responses. Performance on non-erroneous student responses ( \mathord{\mathchoice{\resizebox{}{442368}{\hbox{\raisebox{0.0pt}{\displaystyle\bigcirc}}}}{\resizebox{}{442368}{\hbox{\raisebox{0.0pt}{\textstyle\bigcirc}}}}{\resizebox{}{341334}{\hbox{\raisebox{0.0pt}{\scriptstyle\bigcirc}}}}{\resizebox{}{268529}{\hbox{\raisebox{0.0pt}{\scriptscriptstyle\bigcirc}}}}} ) is labeled with specific VLMs’ names; that same model’s performance on erroneous student responses is directly below ( □\square ). Error bars are 95% CI.
Figure 3: Content description QA consistently drives the gap in VLM performance between student responses that contain errors versus those that do not. Appendix A includes additional VLMs that expand this finding.
Modelβ1\beta_{1}
Gemini 2.0 Flash0.0944***
Gemini 2.5 Pro Preview0.0888***
Gemini 2.5 Pro0.0889***
GPT-4.10.0865***
GPT-4.5 preview0.0874***
o4-mini0.0612***
Table 1: Estimated effects of student correctness ( β1\beta_{1} ) on VLMs’ accuracy on content description QA, where all p<1.0−12p<1.0^{-12} .
Figure 4: An example of how a student response image (top) is transformed and cleaned up by our digital redrawing process (bottom). This student uses a place value chart to show how digit values change for 345 after division by 100.
Figure 5: Models’ performance for content description QA generally improves after images are redrawn. Error bars are 95% CI.
ModelOriginal ImagesRedrawn Images
Gemini 2.5 Pro-0.089***-0.096***
Claude Sonnet 4.5-0.100***-0.058**
GPT-5-0.055**-0.043**
Llama 4 Scout-0.092***-0.073***
Table 2: Differences in average scores on content description QA between erroneous and non-erroneous student images persist after redrawing. ** p<0.01p<0.01 , *** p<0.001p<0.001 .
Figure 6: Illustrative examples of the phenomenon where models predict answers for erroneous student responses that match true answers for non-erroneous students.
Model%
Gemini 2.5 Pro0.2923
Claude Sonnet 4.50.3519
GPT-50.3125
Llama 4 Scout0.3060
Table 3: The percentage of times for which an incorrect model answer for a content description question and erroneous student image matched the majority (> 50%) of true answers for non-erroneous student images.
Figure 7: Model performance on correctness & error (C&E) QA, with and without natural language description (NLD) support. We evaluate with a subset of input images and captions as described in § 7.1 . Error bars are 95% CI.
Figure 8: VLMs’ performance on the two main subtypes of correctness & error QA, disaggregated by whether a student response is overall correct ( generic ) or correct based on specific aspect of their solution ( binary ). Error bars are 95% CI.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: VLMs’ performance across different subtypes of correctness & error questions, as defined in § 8.1 .
Figure 10: An expanded version of Figure 3 , showing which question categories contribute to the gap in VLM performance between student responses that contain errors versus those that do not. Questions below the dotted line in each subplot are content description QA.
More in this topic

AI in Education

AI in Education studies how artificial intelligence can support teaching, learning, assessment, and educational decision-making.

See all 261 papers

Explore similar work

CardsList

  1. Can Vision Language Models Be Adaptive in Mathematics Education? A Learner Model-based Rubric Study

    May 15, 2026Jie Gao, Yongan Yu, Junzhu Su +3VLM EvaluationMathematics Education

    1McGill University · 2Mila – Quebec AI Institute · 1McGill +1

  2. Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities

    Aug 9, 2026Harish Kashyap, Kiran Byadarhaly, Sriram Chakaravarthy +2LLM EvaluationT2I Generation Evaluation

    Pandita AI Inc., 2540 N First St, San Jose CA 95131, USA

  3. MisEdu-RAG: A Misconception-Aware Dual-Hypergraph RAG for Novice Math Teachers

    Apr 5, 2026Zhihan Guo, Yuting Lu, Jionghao LinRetrieval-Augmented GenerationKnowledge Augmentation for Language Models

    The University of Hong Kong, Hong Kong SAR, China · Carnegie Mellon University, Pittsburgh PA 15213, USA · Monash University, Clayton VIC 3800, Australia