WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans

Jun Zhang; Qiao Zhao; Cheng Cui; Jianying Qu; Zhongkai Sun; Jianwen Yang; Changda Zhou; ZhuoXin Liu

cs.CVAug 24, 2026

WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans

Authors: Jun Zhang, Qiao Zhao, Cheng Cui, Jianying Qu, Zhongkai Sun, Jianwen Yang, Changda Zhou, ZhuoXin Liu, +1 more

Organizations: Baidu Inc.

Abstract

While the top model on OmniDocBench now reaches 96.34% overall on printed-document parsing, the ability of current models to handle challenging handwritten documents remains largely uncharacterized. Existing benchmarks focus on isolated text or formulas, overlook handwritten tables and real-world degradation, and report aggregate accuracy without explaining why models fail. We present WildHandBench, a benchmark containing 500 handwritten documents across three structures (free text, tables, formulas), four languages, and nine real-world scenarios. We introduce a Prior-Driven Error (PDE) metric that quantifies whether errors originate from language priors rather than visual evidence. Evaluating 18 state-of-the-art models together with calibrated human baselines, we find: (1) the best model achieves only 71.85% overall; (2) humans outperform all models yet the gap is narrow (77.09% vs. 71.85%); and (3) model errors are qualitatively different from human errors -- 63-91% of model errors are prior-driven versus only 49% for humans, exposing systematic reliance on language priors that conventional accuracy metrics cannot capture.

Figures & tables

Benchmark	Lang.	Handwritten	Free Text	Table	Formula	Real Degrad.	Human Baseline
IAM	EN	✓	✓	–	–	–	–
CASIA-HWDB	ZH	✓	✓	–	–	–	–
SCUT-EPT / HCCDoc	ZH	✓	✓	–	–	Partial	–
CROHME / MathWriting	Symbol	✓	–	–	✓	–	–
PubTabNet / FinTabNet	EN	–	–	✓	–	–	–
ICDAR Table	EN	–	–	✓	–	–	–

Table 1: Comparison with representative benchmarks. WildHandBench is, to our knowledge, the first to jointly cover handwritten free text, tables, and formulas with real-scenario degradation and calibrated human baselines.

Figure 1: Overview of the WildHandBench construction pipeline.

Figure 2: Qualitative comparison of handwritten table parsing across general-purpose models, OCR-specialized models, and the human baseline. The right panel illustrates three error types: table structure errors (E1), table content errors (E2), and prior-driven errors (E3).

			Performance Evaluation (%)				Prior-Driven Error (%)
Type	Model	Params	Text Edit $\downarrow$	Table TEDS $\uparrow$	Formula CDM $\uparrow$	Overall $\uparrow$	Text $\downarrow$	Table $\downarrow$	Formula $\downarrow$	Overall $\downarrow$
Proprietary	Claude Opus 4.8	–	53.30	52.27	74.11	57.70	72.78	59.32	91.66	74.59
	Gemini 3.5 Flash	–	27.10	53.63	78.22	68.25	71.36	71.91	94.36	79.21
	Gemini 3.1 Pro	–	24.32	60.44	79.42	71.85	71.19	66.56	94.77	77.51
Open General	InternVL3.5	241B-A28B	46.80	41.22	64.00	52.80	72.62	66.38	90.03	76.34
	Qwen3-VL	235B-A22B	34.30	51.81	69.06	62.19	70.01	62.18	91.33	74.51

Table 2: Recognition performance and prior-driven error for all evaluated models and the human baseline.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Category	zh_hans	en	zh_hant	zh_en_mix	Total
Text	251	99	12	7	369
Table	62	16	0	1	79
Formula	20	29	0	3	52
Total	333	144	12	11	500

Appendix

Table 3: Category

\times

Language joint distribution.

Scenario	Count
Literary essays	93
Letters & notes	81
Medical & health	62
Education & learning	61
Business documents	58
Math & science formulas	55

Appendix

Table 4: Per-scenario sample counts.

Figure 3: Classical poem (calligraphy). A handwritten copy of Su Shi’s Chibi Nostalgia contains non-standard character orderings. Gemini “corrects” the handwriting to the memorized textbook version—swapping the non-standard two-character ordering ( táo làng ) back to the canonical order ( làng táo ). This is a classic canonical-text prior: the model overwrites the actual handwritten content with a remembered standard version. The human baseline faithfully preserves the non-standard forms and marks only truly illegible characters with “?”.

Figure 4: Personal letter. Gemini substitutes “prefix” with “parameter”—a semantic-field substitution in which the model replaces the written word with a semantically related term drawn from its priors rather than transcribing the source faithfully.

Figure 5: Clinical diagnosis. The GT diagnosis is “acute mesenteric lymphadenitis / acute enteritis / terminal ileitis.” Gemini outputs a completely different yet medically plausible diagnosis (“colonic diverticulitis / colonic polyps / terminal ileitis”; only the third item matches). This is the most extreme form of PDE: medical domain-knowledge priors fabricate a coherent but factually wrong diagnosis. The human baseline honestly writes “?” for illegible portions.

Figure 6: Traditional Chinese medicine prescription. Gemini applies a prior-driven substitution, replacing a herb name with a more frequently prescribed alternative.

Figure 7: Political study notes. Gemini rewrites into more formal policy terminology (“self-reliance” $\to$ “self-strengthening”) and swaps the character order inside a four-character phrase (“achievement makes talent” $\to$ “talent makes achievement”)—both linguistically valid but not matching the source. This cleanly contrasts high PDE (semantic rewriting) against low PDE (visual misreading).

Figure 8: Childcare training reflection. Gemini fabricates an entirely different topic: “daily cleaning skills training” becomes “recent first-aid training,” and “infant bathing / umbilical care” becomes “infant airway obstruction care”—the whole passage topic is replaced. PaddleOCR-VL exhibits character-level noise but preserves the correct topic keywords (“daily cleaning,” “bathing”). The human baseline matches the GT exactly.

Figure 9: Elementary-school essay. The student wrote “doing heart-warming things in places you cannot see .” Gemini polishes it to “doing heart-warming things when you most need help ”—semantically “better” but not the original text (PDE). Both Gemini and PaddleOCR-VL are effectively “polishing” the child’s composition.

Figure 10: Mathematical derivation (Lagrange multipliers). Gemini corrects the awkward phrase “in the same direction of proportional” to “in the same direction & proportional.” PaddleOCR-VL misreads “proportional” as “prepatian” and garbles the $\nabla g$ matrix line, but shows no semantic rewriting.

Model Name	Type	Access	Model ID / Version
GPT-5.4	General VLM	API (OpenAI-compat)	gpt-5.4
Gemini-3.1-Pro	General VLM	API (OpenAI-compat)	gemini-3.1-pro-preview
Gemini-3.5-Flash	General VLM	API (OpenAI-compat)	gemini-3.5-flash
Claude-Opus-4-8	General VLM	API (OpenAI-compat)	claude-opus-4-8
Qwen3.5-Plus	General VLM	API (OpenAI-compat)	qwen3.5-plus
Qwen3-VL	General VLM	API (vLLM)	qwen3-vl-235b-a22b-thinking

Appendix

Table 5: Complete list of evaluated models.

Explore similar work

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios
Aug 19, 2026Zinuo Guo, Min Zhang, Bo JiangOptical Character Recognition Multimodal Model Evaluation
East China Normal University Shanghai, China
How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings
May 8, 2026Zhiheng Li, Zongyang Ma, Jiaxian Chen +12Document Parsing Document Image Understanding
CASIA · UCAS · Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information +6
From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms
Apr 14, 2026Nicholas Pather, Joshua Fouché, Sitwala Mundia +5Document Information Extraction Document Image Understanding
CSAM, University of the Witwatersrand & Grai Labs, Johannesburg, South Africa · Grai Labs, Cape Town, South Africa · Faculty of Health Sciences, University of the Witwatersrand, South Africa +5

Aug 19, 2026J/K move · Enter open · S save

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios

Zinuo Guo, Min Zhang, Bo Jiang

East China Normal University Shanghai, China

Optical Character Recognition Multimodal Model Evaluation Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to faithfully read real handwriting remains underexplored. Existing OCR benchmarks focus largely on printed text or clean single-line inputs, leaving limited coverage of realistic handwritten OCR scenarios such as multilingual handwriting, writer errors, and structurally complex mathematical expressions. We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR. It covers handwritten text recognition and handwritten mathematical expression recognition across six subtasks and twelve subsets, totaling 77.57K labeled images from public datasets and newly collected student writings. A key component is a difficulty-stratified multi-line formula corpus designed to test robustness under increasing structural complexity. We evaluate thirteen open- and closed-source systems with five complementary metrics under a unified protocol. Results show that current systems remain far from faithful transcription: performance drops sharply on complex multi-line formulas, model rankings vary across language and formula settings, and several generative models hallucinate plausible but visually unsupported corrections. OmniHandwritingOCR provides a challenging testbed for diagnosing language, content, structural, and visual-grounding failure modes of multimodal models in handwritten OCR scenarios.

Abstract

Explore similar work

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios

How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings

From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms