When VLMs 'Fix' Students: Identifying and Penalizing Over-Correction in the Evaluation of Multi-line Handwritten Math OCR

Jin Seong; Wencke Liermann; Minho Kim; Jong-hun Shin; Soojong Lim

cs.CYApr 1, 2026

When VLMs 'Fix' Students: Identifying and Penalizing Over-Correction in the Evaluation of Multi-line Handwritten Math OCR

Authors: Jin Seong, Wencke Liermann, Minho Kim, Jong-hun Shin, Soojong Lim

Organizations: Electronics and Telecommunications Research Institute, Republic of Korea

Abstract

Accurate transcription of handwritten mathematics is crucial for educational AI systems, yet current benchmarks fail to evaluate this capability properly. Most prior studies focus on single-line expressions and rely on lexical metrics such as BLEU, which fail to assess the semantic reasoning across multi-line student solutions. In this paper, we present the first systematic study of multi-line handwritten math Optical Character Recognition (OCR), revealing a critical failure mode of Vision-Language Models (VLMs): over-correction. Instead of faithfully transcribing a student's work, these models often "fix" errors, thereby hiding the very mistakes an educational assessment aims to detect. To address this, we propose PINK (Penalized INK-based score), a semantic evaluation metric that leverages a Large Language Model (LLM) for rubric-based grading and explicitly penalizes over-correction. Our comprehensive evaluation of 15 state-of-the-art VLMs on the FERMAT dataset reveals substantial ranking reversals compared to BLEU: models like GPT-4o are heavily penalized for aggressive over-correction, whereas Gemini 2.5 Flash emerges as the most faithful transcriber. Furthermore, human expert studies show that PINK aligns significantly better with human judgment (55.0% preference over BLEU's 39.5%), providing a more reliable evaluation framework for handwritten math OCR in educational settings.

Figures & tables

Figure 1: BLEU’s Blind Spots in Multi-line Handwritten Math OCR. (a) BLEU penalizes semantically equivalent expressions. (b) BLEU assigns a high score even when the OCR misses a key part of the solution, treating a critical omission as insignificant.

Figure 2: Overview of the proposed PINK scoring system. Step 1: a VLM OCR output over-corrects a student error. Step 2: the auto-grading system assigns a seemingly high score. Step 3: comparison with the oracle score reveals an over-correction event. Step 4: penalties are applied to compute the final PINK score.

Figure 3: Distribution of over-correction instances across rubric items for 15 models , sorted by Penalized Score (Top is Best). Over-corrections are heavily concentrated in the final two stages, particularly in higher-performing models, consistent with autoregressive context accumulation overriding visual evidence.

Figure 4: Correlation between Original Score and Over-correction Ratio. Bubble size represents the scale of the model.

Figure 5: Ranking Reversal of VLMs. The leaderboard based on conventional BLEU scores (left) is dramatically overturned when re-evaluated with our PINK metric (right), which penalizes over-correction. Models like GPT-4o drop significantly, while Gemini 2.5 Flash ascends nine places to the top rank, showing that existing metrics miss the crucial dimension of faithfulness.

Open-source Models
Model	PINK Score $\uparrow$	BLEU Score $\uparrow$	Edit Distance $\downarrow$
Qwen2.5 72B VL	0.920	0.733	0.1978
Qwen2.5 32B VL	0.919	0.734	0.1975
Ovis2 34B	0.916	0.723	0.1997
InternVL3 78B	0.911	0.749	0.1969
InternVL3 38B	0.905	0.771	0.1925

Table 1: Performance comparison of open-source and closed-source models. The table evaluates models using the PINK, BLEU, and Edit Distance scores. The background color intensity indicates the relative performance for each metric.

Panel of Figure 6 — Figure 6: Over-Correction Rate : Comparison between GPT-4o (red) and Gemini 2.5 Flash (green).

Panel of Figure 7 — Figure 7: Human evaluation results. (a) Experts showed a clear overall preference for PINK over BLEU. (b) PINK was consistently preferred across all score ranges, demonstrating its robustness.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Range (points)	Occurrences	Percentage
0–5	22,483	60.4%
5–10	7,038	18.9%
10–15	3,548	9.5%
15–20	4,131	11.1%

Appendix

Table 2: Distribution of over-correction magnitudes across all models.

Figure 8: Ranking stability under different penalty thresholds

T

, where models are penalized when

\Delta S(S_{\text{model}}-S_{\text{oracle}})>T

(i.e., VLM score exceeds student’s true score by more than

T

points). Kendall’s

\tau

measures ranking consistency relative to

T{=}10

baseline. High stability (

\tau>0.9

) in the range

T\in[5,15]

validates our threshold choice.

Figure 9: Model ranking trajectories under varying penalty thresholds. Three patterns emerge: robust (e.g., Gemini 2.5 Flash), minor over-correctors (e.g., GPT-4o), and critical over-correctors (e.g., InternVL3).

Figure 10: Overview of Auto-Grading System’s Rubrics

Figure 11: Human Evaluation: Labeling Tool

Figure 12: Case Study of Over-Correction. (Top) Simple-Level: InternVL3-8B locally fixes the student’s error ( $\sin\to\tan$ ), inflating the score from 34 to 99. (Bottom) Critical-Level: GPT-4o ignores the student’s erroneous reasoning steps and provides a correct summary, inflating the score from 14 to 100.

Figure 13: Visualizing Attention Maps and Penalty Application. We visualize the attention mechanism at the moment of generating the target token, conditioned on the preceding integral context. (Top) InternVL3-8B generates the mathematically correct ‘tan’ (Over-Correction). The attention map shows no activation on the handwritten ‘sin’ , verifying that the contextual pressure caused the model to treat visual evidence as noise. While this yields a high raw score (99), PINK correctly identifies this unfaithful behavior and applies a penalty, dropping the score to 27 . (Bottom) Qwen2.5-VL 8B resists this pressure and faithfully transcribes the error (‘sin’), with attention precisely grounded on the handwritten strokes. This faithful transcription aligns with the Oracle baseline, resulting in a final score of 34 with no penalty .

Figure 14: Over-correction frequency across all 15 tested VLM models.

Figure 15: Ranking changes after applying the over-correction penalty. The PINK score leads to a significant reordering of models, highlighting the impact of penalizing unfaithful corrections.

Stage	Level	Pearson $r$	QWK	Kendall’s $\tau$
Grading	Sample	0.78	0.78	–
OC Penalty	Sample	0.74	0.73	–
PINK Score	VLM	0.99	–	–
Final Rank	VLM	–	–	0.90

Appendix

Table 3: Reproducibility Pipeline. Micro-variances at the sample level cancel out, yielding near-perfect macro agreement. N = 33k (sample), N = 15 (VLM).

Model	PINK Score		Rank
Model	GPT	Qwen	GPT	Qwen
Gemini 2.5 Flash	0.94	0.87	1	3 $\downarrow$
Qwen2.5-72B-VL	0.92	0.88	2	2 -
Qwen2.5-32B-VL	0.92	0.88	3	1 $\uparrow$
Ovis2 34B	0.92	0.85	4	4 -
InternVL3-78B	0.91	0.84	5	6 $\downarrow$

Appendix

Table 4: Leaderboard Stability. Comparison using GPT-5 vs. Qwen3-Next-80B-A3B-Thinking judges.

Metric	Value
PINK Mean (5 runs)	0.872
PINK CV	0.41%
Penalized Score QWK	0.926

Appendix

Table 5: Run-to-run stability. GPT-4o graded 5

\times

by Qwen3-80B (

N

= 1,803).

VLM (Quartile)	Orig	V1	V2	V3	Std
Gemini 2.5 Flash (Q1)	.873	.875	.870	.877	$\pm$ .003
GPT-4o (Q2)	.853	.856	.857	.864	$\pm$ .004
InternVL3-8B (Q3)	.787	.792	.793	.802	$\pm$ .005
Llama-3.2-11B (Q4)	.582	.592	.593	.596	$\pm$ .005

Appendix

Table 6: PINK scores across three rubric prompt variants. Grader: Qwen3-80B. Models selected from each PINK quartile.

VLM (Quartile)	Orig	V1	V2	V3	Std
Qwen2.5-VL-32B (Q1)	.873	.872	.872	.877	$\pm$ .002
Ovis2-8B (Q2)	.804	.807	.799	.810	$\pm$ .005
Qwen2.5-VL-7B (Q3)	.725	.738	.729	.731	$\pm$ .005
LLaMA-3.2-11B (Q4)	.617	.654	.628	.634	$\pm$ .015

Appendix

Table 7: PINK scores across OCR prompt variants (graded by Qwen3-80B,

N=1{,}500

). Quartile ordering is preserved across all variants.

Baseline	PINK	Baseline	Tied	$p$
MathBERTa	124 (62%)	52 (26%)	24 (12%)	$5.75\!\times\!10^{-8}$
Edit Dist.	122 (61%)	57 (29%)	21 (11%)	$1.33\!\times\!10^{-6}$
BLEU	119 (60%)	69 (35%)	12 (6%)	$3.27\!\times\!10^{-4}$

Appendix

Table 8: Human preference: PINK vs. three baselines (200 items, 3 annotators).

p

: two-sided exact binomial test on non-tied majority-vote items.

	MathBERTa	Edit Dist.	BLEU	PINK
MathBERTa	1.00	0.75	0.88	0.47
Edit Dist.	0.75	1.00	0.88	0.52
BLEU	0.88	0.88	1.00	0.50
PINK	0.47	0.52	0.50	1.00

Appendix

Table 9: Kendall’s

\tau

rank correlation across four metrics (15 VLMs).

	PINK Score			OC (%)
Model	Org	Mit	$\Delta$	Org	Mit	$\Delta$
High-PINK (open-source top-3)
Qwen2.5-32B	.88	.85	$-.03$	62%	53%	$-10$
Qwen2.5-72B	.87	.84	$-.03$	61%	50%	$-11$
Ovis2-34B	.85	.85	.00	59%	54%	$-4$
Low-PINK (open-source bottom-3)

Appendix

Table 10: Prompt-based mitigation. OC: samples with

\geq

1 over-corrected item.

\Delta

: mitigated

-

original. Green : improved; Red : degraded.

Figure 16: Auto-Grading Prompt

Figure 17: Auto-Labeling Prompt

Figure 18: OCR Prompt

Figure 19: Mitigated OCR Prompt

Explore similar work

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios
Aug 19, 2026Zinuo Guo, Min Zhang, Bo JiangOptical Character Recognition Multimodal Model Evaluation
East China Normal University Shanghai, China
Automated Grading of Handwritten Mathematics Using Vision-Capable LLMs
May 18, 2026Jacob Levine, Miguel Aenlle, Craig Zilles +2VLM Evaluation Rubric-Based Evaluation
University of Illinois Urbana-Champaign
Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math
Mar 26, 2026Dingjie Song, Tianlong Xu, Yi-Fan Zhang +6Educational Assessment Multimodal Model Evaluation
Lehigh University, Bethlehem, PA, USA · Squirrel Ai Learning, Bellevue, WA, USA · Institute of Automation, Chinese Academy of Sciences, Beijing, China +2

Aug 19, 2026J/K move · Enter open · S save

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios

Zinuo Guo, Min Zhang, Bo Jiang

East China Normal University Shanghai, China

Optical Character Recognition Multimodal Model Evaluation Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to faithfully read real handwriting remains underexplored. Existing OCR benchmarks focus largely on printed text or clean single-line inputs, leaving limited coverage of realistic handwritten OCR scenarios such as multilingual handwriting, writer errors, and structurally complex mathematical expressions. We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR. It covers handwritten text recognition and handwritten mathematical expression recognition across six subtasks and twelve subsets, totaling 77.57K labeled images from public datasets and newly collected student writings. A key component is a difficulty-stratified multi-line formula corpus designed to test robustness under increasing structural complexity. We evaluate thirteen open- and closed-source systems with five complementary metrics under a unified protocol. Results show that current systems remain far from faithful transcription: performance drops sharply on complex multi-line formulas, model rankings vary across language and formula settings, and several generative models hallucinate plausible but visually unsupported corrections. OmniHandwritingOCR provides a challenging testbed for diagnosing language, content, structural, and visual-grounding failure modes of multimodal models in handwritten OCR scenarios.

Abstract

Explore similar work

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios

Automated Grading of Handwritten Mathematics Using Vision-Capable LLMs

Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math