Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to faithfully read real handwriting remains underexplored. Existing OCR benchmarks focus largely on printed text or clean single-line inputs, leaving limited coverage of realistic handwritten OCR scenarios such as multilingual handwriting, writer errors, and structurally complex mathematical expressions. We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR. It covers handwritten text recognition and handwritten mathematical expression recognition across six subtasks and twelve subsets, totaling 77.57K labeled images from public datasets and newly collected student writings. A key component is a difficulty-stratified multi-line formula corpus designed to test robustness under increasing structural complexity. We evaluate thirteen open- and closed-source systems with five complementary metrics under a unified protocol. Results show that current systems remain far from faithful transcription: performance drops sharply on complex multi-line formulas, model rankings vary across language and formula settings, and several generative models hallucinate plausible but visually unsupported corrections. OmniHandwritingOCR provides a challenging testbed for diagnosing language, content, structural, and visual-grounding failure modes of multimodal models in handwritten OCR scenarios.
Figures & tables
Figure 1. Overview of OmniHandwritingOCR. The benchmark focuses on two core tasks, handwritten text recognition (HTR) and handwritten mathematical expression recognition (HMER), covers six subtasks and twelve subsets with 77.57K images.
Figure 2. Construction pipeline of OmniHandwritingOCR, including data collection, filtering, difficulty stratification, annotation, and expert verification.
Figure 3. Representative examples from each subtask, showing the diversity of handwriting styles and the progressive difficulty of formula structures.
Figure 4. Examples of fact-based annotations. Ground-truth labels preserve visible writer mistakes, while Qwen2.5-VL-72B and InternVL3-78B incorrectly produce corrected but visually unsupported transcriptions.
Subtask
Source
Samples
Total
English Text
GNHK
515
11,859
IAM-line
11,344
Chinese Text
private
874
12,529
CASIA-HWDB
11,655
Single-line Formula
CROHME
3,332
15,096
HME100K
9,754
Table 1. Detailed composition of the dataset.
Subset
Len.
Cmd.
Lines
ml-easy
156.9
4.1
3.7
ml-medium
213.0
7.6
7.1
ml-hard
488.8
25.7
13.0
Table 2. Model-agnostic statistics of private multi-line formula subsets. Len. denotes average ground-truth characters, Cmd. denotes average L a T e X commands, and Lines denotes average non-empty text lines.
Benchmark
Type
Tasks
Structure
Num
All
MLF
RIMES-2011-line ( Grosicki et al., 2009 )
HTR
1
single-line only
12,104
-
READ-2016 ( Grüning et al., 2017b )
HTR
1
multi-line only
30,000
-
SCUT-HCCDoc ( Zhang et al., 2020a )
HTR
5
single-line & multi-line
12,253
-
CROHME2023 ( Xie et al., 2023 )
HMER
1
single-line only
13,279
0
MLHME-38K ( TAL Education Group, 2023 )
HMER
2
single-line & multi-line
38,000
9,931
Table 3. Comparison of different handwriting-related benchmarks, where HTR denotes Handwritten Text Recognition, HMER denotes Handwritten Mathematical Expression Recognition, and MLF denotes Handwritten Multi-Line Formula.
Methods
BLEU-4 ↑
F1 Score ↑
1-NED ↑
Overall ↑
CER ↓
WER ↓
Overall ↓
Multimodal large language models (MLLMs)
GPT-4o
39.53
67.66
59.32
55.50
48.53
50.03
49.28
Qwen2.5-VL-72B
56.40
79.98
72.91
69.76
38.88
36.91
37.90
InternVL3-78B
53.76
78.21
72.16
68.04
32.19
34.72
33.46
Gemma-3-27B-it
31.84
60.08
50.97
47.63
65.82
67.83
66.83
DeepSeek-VL2-27B
39.57
63.06
54.90
52.51
76.91
63.49
70.20
Table 4. Experiments (%) on the entire OmniHandwritingOCR. The highest is bold , and the second highest is underlined .
Methods
BLEU-4 ↑
F1 Score ↑
1-NED ↑
CER ↓
WER ↓
en-text
zh-text
en-text
zh-text
en-text
zh-text
en-text
zh-text
en-text
zh-text
Multimodal large language models (MLLMs)
GPT-4o
54.03
33.51
77.44
58.48
81.29
60.66
20.05
43.90
31.93
51.84
Qwen2.5-VL-72B
58.57
71.51
81.84
88.39
84.63
90.71
16.01
10.42
28.21
15.16
InternVL3-78B
51.26
56.59
76.98
77.42
81.14
79.91
20.36
22.53
34.69
29.08
Gemma-3-27B-it
40.07
13.93
69.36
33.06
74.86
29.28
26.12
95.18
42.70
101.30
Table 5. Experiments (%) on the English (en-text) and Chinese text (zh-text). The highest is bold , and the second highest is underlined .
Methods
BLEU-4 ↑
F1 Score ↑
1-NED ↑
CER ↓
WER ↓
sl
ml-easy
sl
ml-easy
sl
ml-easy
sl
ml-easy
sl
ml-easy
Multimodal large language models (MLLMs)
GPT-4o
24.51
61.08
62.26
81.68
51.98
71.17
79.24
29.37
74.50
26.35
Qwen2.5-VL-72B
40.09
64.24
72.98
82.84
63.13
71.73
67.07
35.53
57.53
30.13
InternVL3-78B
56.67
67.00
84.08
85.13
80.70
75.81
27.04
26.04
30.34
23.49
Gemma-3-27B-it
19.36
52.03
53.74
75.96
45.83
64.16
104.31
40.00
108.95
35.03
Table 6. Experiments (%) on single-line (sl) and easy multi-line (ml-easy). The highest is bold , and the second highest is underlined .
Methods
BLEU-4 ↑
F1 Score ↑
1-NED ↑
CER ↓
WER ↓
ml-medium
ml-hard
ml-medium
ml-hard
ml-medium
ml-hard
ml-medium
ml-hard
ml-medium
ml-hard
Multimodal large language models (MLLMs)
GPT-4o
46.84
17.18
73.43
52.68
61.20
29.60
43.05
75.58
43.06
72.52
Qwen2.5-VL-72B
59.98
43.98
80.42
73.39
71.47
55.81
49.32
54.91
42.00
48.42
InternVL3-78B
57.53
33.52
79.35
66.32
69.81
45.59
36.39
60.80
34.83
55.89
Gemma-3-27B-it
43.61
22.03
70.32
58.05
57.32
34.38
56.11
73.21
49.95
69.05
Table 7. Experiments (%) on the medium multi-line formula (ml-medium) and hard multi-line formula (ml-hard).
Figure 5. Overall benchmark performance (%) under BLEU-4, F1, 1-NED, CER, and WER.