Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to faithfully read real handwriting remains underexplored. Existing OCR benchmarks focus largely on printed text or clean single-line inputs, leaving limited coverage of realistic handwritten OCR scenarios such as multilingual handwriting, writer errors, and structurally complex mathematical expressions. We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR. It covers handwritten text recognition and handwritten mathematical expression recognition across six subtasks and twelve subsets, totaling 77.57K labeled images from public datasets and newly collected student writings. A key component is a difficulty-stratified multi-line formula corpus designed to test robustness under increasing structural complexity. We evaluate thirteen open- and closed-source systems with five complementary metrics under a unified protocol. Results show that current systems remain far from faithful transcription: performance drops sharply on complex multi-line formulas, model rankings vary across language and formula settings, and several generative models hallucinate plausible but visually unsupported corrections. OmniHandwritingOCR provides a challenging testbed for diagnosing language, content, structural, and visual-grounding failure modes of multimodal models in handwritten OCR scenarios.
Figures & tables
Figure 1. Overview of OmniHandwritingOCR. The benchmark focuses on two core tasks, handwritten text recognition (HTR) and handwritten mathematical expression recognition (HMER), covers six subtasks and twelve subsets with 77.57K images.
Figure 2. Construction pipeline of OmniHandwritingOCR, including data collection, filtering, difficulty stratification, annotation, and expert verification.
Figure 3. Representative examples from each subtask, showing the diversity of handwriting styles and the progressive difficulty of formula structures.
Figure 4. Examples of fact-based annotations. Ground-truth labels preserve visible writer mistakes, while Qwen2.5-VL-72B and InternVL3-78B incorrectly produce corrected but visually unsupported transcriptions.
Subtask
Source
Samples
Total
English Text
GNHK
515
11,859
IAM-line
11,344
Chinese Text
private
874
12,529
CASIA-HWDB
11,655
Single-line Formula
CROHME
3,332
15,096
HME100K
9,754
Table 1. Detailed composition of the dataset.
Subset
Len.
Cmd.
Lines
ml-easy
156.9
4.1
3.7
ml-medium
213.0
7.6
7.1
ml-hard
488.8
25.7
13.0
Table 2. Model-agnostic statistics of private multi-line formula subsets. Len. denotes average ground-truth characters, Cmd. denotes average L a T e X commands, and Lines denotes average non-empty text lines.
Benchmark
Type
Tasks
Structure
Num
All
MLF
RIMES-2011-line ( Grosicki et al., 2009 )
HTR
1
single-line only
12,104
-
READ-2016 ( Grüning et al., 2017b )
HTR
1
multi-line only
30,000
-
SCUT-HCCDoc ( Zhang et al., 2020a )
HTR
5
single-line & multi-line
12,253
-
CROHME2023 ( Xie et al., 2023 )
HMER
1
single-line only
13,279
0
MLHME-38K ( TAL Education Group, 2023 )
HMER
2
single-line & multi-line
38,000
9,931
Table 3. Comparison of different handwriting-related benchmarks, where HTR denotes Handwritten Text Recognition, HMER denotes Handwritten Mathematical Expression Recognition, and MLF denotes Handwritten Multi-Line Formula.
Methods
BLEU-4 ↑
F1 Score ↑
1-NED ↑
Overall ↑
CER ↓
WER ↓
Overall ↓
Multimodal large language models (MLLMs)
GPT-4o
39.53
67.66
59.32
55.50
48.53
50.03
49.28
Qwen2.5-VL-72B
56.40
79.98
72.91
69.76
38.88
36.91
37.90
InternVL3-78B
53.76
78.21
72.16
68.04
32.19
34.72
33.46
Gemma-3-27B-it
31.84
60.08
50.97
47.63
65.82
67.83
66.83
DeepSeek-VL2-27B
39.57
63.06
54.90
52.51
76.91
63.49
70.20
Table 4. Experiments (%) on the entire OmniHandwritingOCR. The highest is bold , and the second highest is underlined .
Methods
BLEU-4 ↑
F1 Score ↑
1-NED ↑
CER ↓
WER ↓
en-text
zh-text
en-text
zh-text
en-text
zh-text
en-text
zh-text
en-text
zh-text
Multimodal large language models (MLLMs)
GPT-4o
54.03
33.51
77.44
58.48
81.29
60.66
20.05
43.90
31.93
51.84
Qwen2.5-VL-72B
58.57
71.51
81.84
88.39
84.63
90.71
16.01
10.42
28.21
15.16
InternVL3-78B
51.26
56.59
76.98
77.42
81.14
79.91
20.36
22.53
34.69
29.08
Gemma-3-27B-it
40.07
13.93
69.36
33.06
74.86
29.28
26.12
95.18
42.70
101.30
Table 5. Experiments (%) on the English (en-text) and Chinese text (zh-text). The highest is bold , and the second highest is underlined .
Methods
BLEU-4 ↑
F1 Score ↑
1-NED ↑
CER ↓
WER ↓
sl
ml-easy
sl
ml-easy
sl
ml-easy
sl
ml-easy
sl
ml-easy
Multimodal large language models (MLLMs)
GPT-4o
24.51
61.08
62.26
81.68
51.98
71.17
79.24
29.37
74.50
26.35
Qwen2.5-VL-72B
40.09
64.24
72.98
82.84
63.13
71.73
67.07
35.53
57.53
30.13
InternVL3-78B
56.67
67.00
84.08
85.13
80.70
75.81
27.04
26.04
30.34
23.49
Gemma-3-27B-it
19.36
52.03
53.74
75.96
45.83
64.16
104.31
40.00
108.95
35.03
Table 6. Experiments (%) on single-line (sl) and easy multi-line (ml-easy). The highest is bold , and the second highest is underlined .
Methods
BLEU-4 ↑
F1 Score ↑
1-NED ↑
CER ↓
WER ↓
ml-medium
ml-hard
ml-medium
ml-hard
ml-medium
ml-hard
ml-medium
ml-hard
ml-medium
ml-hard
Multimodal large language models (MLLMs)
GPT-4o
46.84
17.18
73.43
52.68
61.20
29.60
43.05
75.58
43.06
72.52
Qwen2.5-VL-72B
59.98
43.98
80.42
73.39
71.47
55.81
49.32
54.91
42.00
48.42
InternVL3-78B
57.53
33.52
79.35
66.32
69.81
45.59
36.39
60.80
34.83
55.89
Gemma-3-27B-it
43.61
22.03
70.32
58.05
57.32
34.38
56.11
73.21
49.95
69.05
Table 7. Experiments (%) on the medium multi-line formula (ml-medium) and hard multi-line formula (ml-hard).
Figure 5. Overall benchmark performance (%) under BLEU-4, F1, 1-NED, CER, and WER.
While the top model on OmniDocBench now reaches 96.34% overall on printed-document parsing, the ability of current models to handle challenging handwritten documents remains largely uncharacterized. Existing benchmarks focus on isolated text or formulas, overlook handwritten tables and real-world degradation, and report aggregate accuracy without explaining why models fail. We present WildHandBench, a benchmark containing 500 handwritten documents across three structures (free text, tables, formulas), four languages, and nine real-world scenarios. We introduce a Prior-Driven Error (PDE) metric that quantifies whether errors originate from language priors rather than visual evidence. Evaluating 18 state-of-the-art models together with calibrated human baselines, we find: (1) the best model achieves only 71.85% overall; (2) humans outperform all models yet the gap is narrow (77.09% vs. 71.85%); and (3) model errors are qualitatively different from human errors -- 63-91% of model errors are prior-driven versus only 49% for humans, exposing systematic reliance on language priors that conventional accuracy metrics cannot capture.
Accurate transcription of handwritten mathematics is crucial for educational AI systems, yet current benchmarks fail to evaluate this capability properly. Most prior studies focus on single-line expressions and rely on lexical metrics such as BLEU, which fail to assess the semantic reasoning across multi-line student solutions. In this paper, we present the first systematic study of multi-line handwritten math Optical Character Recognition (OCR), revealing a critical failure mode of Vision-Language Models (VLMs): over-correction. Instead of faithfully transcribing a student's work, these models often "fix" errors, thereby hiding the very mistakes an educational assessment aims to detect. To address this, we propose PINK (Penalized INK-based score), a semantic evaluation metric that leverages a Large Language Model (LLM) for rubric-based grading and explicitly penalizes over-correction. Our comprehensive evaluation of 15 state-of-the-art VLMs on the FERMAT dataset reveals substantial ranking reversals compared to BLEU: models like GPT-4o are heavily penalized for aggressive over-correction, whereas Gemini 2.5 Flash emerges as the most faithful transcriber. Furthermore, human expert studies show that PINK aligns significantly better with human judgment (55.0% preference over BLEU's 39.5%), providing a more reliable evaluation framework for handwritten math OCR in educational settings.
Jin Seong, Wencke Liermann, Minho Kim +2
Electronics and Telecommunications Research Institute, Republic of Korea
Large Multimodal Models (LMMs) have recently shown strong performance on Optical Character Recognition (OCR) tasks, demonstrating their promising capability in document literacy. However, their effectiveness in real-world applications remains underexplored, as existing benchmarks adopt task scopes misaligned with practical applications and assume homogeneous acquisition conditions. To address this gap, we introduce CC-OCR V2, a comprehensive and challenging OCR benchmark tailored to real-world document processing. CC-OCR V2 focuses on practical enterprise document processing tasks and incorporates hard and corner cases that are critical yet underrepresented in prior benchmarks, covering 5 major OCR-centric tracks: text recognition, document parsing, document grounding, key information extraction, and document question answering, comprising 7,093 high-difficulty samples. Extensive experiments on 14 advanced LMMs reveal that current models fall short of real-world application requirements. Even state-of-the-art LMMs exhibit substantial performance degradation across diverse tasks and scenarios. These findings reveal a significant gap between performance on current benchmarks and effectiveness in real-world applications. We release the full dataset and evaluation toolkit at https://github.com/eioss/CC-OCR-V2.