From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms

Nicholas Pather; Joshua Fouché; Sitwala Mundia; Karl-Günter Technau; Thokozile Malaba; Alex Welte; Ushma Mehta; Bruce A. Bassett

cs.CVApr 14, 2026

From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms

Authors: Nicholas Pather, Joshua Fouché, Sitwala Mundia, Karl-Günter Technau, Thokozile Malaba, Alex Welte, Ushma Mehta, Bruce A. Bassett

Organizations: CSAM, University of the Witwatersrand & Grai Labs, Johannesburg, South Africa · Grai Labs, Cape Town, South Africa · Faculty of Health Sciences, University of the Witwatersrand, South Africa · Empilweni Services and Research Unit, Department of Paediatrics and Child Health, University of the Witwatersrand, Johannesburg, South Africa · Division of Epidemiology and Biostatistics, School of Public Health, Faculty of Health Sciences, University of Cape Town, South Africa · Discipline of Public Health, School of Medicine, University of KwaZulu-Natal, South Africa · Division of Clinical Pharmacology, Department of Medicine, University of Cape Town, South Africa · WITS MIND Institute and CSAM, University of the Witwatersrand, University of Cape Town & Grai Labs, Cape Town, South Africa

Abstract

Manual digitisation of structured handwritten documents is slow and costly. We benchmark 17 leading frontier multi-modal large language models and open-source models against a very challenging real-world medical form that mixes dates; structured, printed text; hand-written responses and significant variability challenges. None of the smaller or older models perform well but the latest Google and OpenAI models reach accuracies around $85\%$ with weighted F1 scores $\simeq 90\%$ across the discrete or predefined fields despite the very challenging nature of the responses. Clear task specific strengths emerge: GPT 5.4 excels in noisy date extraction as well as reliability with the lowest hallucination rate ( $6\%$ ). Claude Sonnet 4.6 had the best average performance across formatted fields (dates and numerical values), while Gemini 3.1 delivered the best overall performance, with the lowest free text error rates (WER = $0.50$ and CER = $0.31$ ) and the strongest results across discrete classification metrics. We further show that prompt optimisation dramatically improves macro precision, recall and F1 by over $60\%$ , but has little impact on weighted metrics (only $\sim2-5\%$ improvement). These results provide evidence that the rapid improvements of multimodal large language models offer a compelling pathway toward fully automated digitisation of complex handwritten workflows that is particularly relevant in low- and middle-income countries.

Figures & tables

Figure 1: Our maternity case record with example realistic synthetic data. The form shows a challenging mix of dates, abbreviations, idiosyncratic responses and structural anomalies such as text written outside the intended field space. Note the handwriting in the synthetic data is often substantially neater than that of the real data set.

Panel of Figure 2 — Figure 2: End-to-end pipeline used for testing frontier LLM’s on our medical benchmark dataset.

Figure 3: Forest plot displaying the median, 16th and 84th percentile for the overall accuracy excluding null fields across the synthetic data set on the optimal prompt. This is indicative that the small, open-source models were the weakest performing by a large margin $\geq 40\%$ from best performing models.

Model	Accuracy	Accuracy (w/o Nulls)	Formatted Accuracy	WER	CER	Null Field Accuracy	Hallucination Rate
3.1 Pro	$0.84_{-0.08}^{+0.07}$	$0.75_{-0.06}^{+0.09}$	$0.84_{-0.18}^{+0.12}$	0.33	0.15	0.93	0.07
3.1 Flash Lite	$0.86$ ${}_{-0.09}^{+0.05}$	$0.83$ ${}_{-0.08}^{+0.03}$	$0.91$ ${}_{-0.17}^{+0.04}$	0.40	0.21	0.91	0.09
2.5 Pro	$0.79_{-0.07}^{+0.06}$	$0.72_{-0.04}^{+0.06}$	$0.76_{-0.12}^{+0.10}$	0.39	0.18	0.87	0.13
2.5 Flash	$0.75_{-0.04}^{+0.07}$	$0.67_{-0.04}^{+0.07}$	$0.71_{-0.12}^{+0.14}$	0.45	0.26	0.86	0.14
GPT 5.4	$0.80_{-0.08}^{+0.05}$	$0.67_{-0.05}^{+0.07}$	$0.73_{-0.13}^{+0.12}$	0.47	0.21	0.91	0.09
GPT 5.2-Pro	$0.76_{-0.08}^{+0.07}$	$0.62_{-0.01}^{+0.06}$	$0.72_{-0.14}^{+0.12}$	0.60	0.38	0.89	0.11

Table 1: Performance metrics of various LLMs on the synthetic dataset using the optimized prompt and template. Metrics show median values with 16th and 84th percentiles; WER is word error rate CER is character error rate—WER and CER were only calculated across free text fields. Hallucination rate indicates proportion of fabricated outputs. The open-source models are shown in the last three rows and are not competitive in every metric. Gemini 3.1 Flash Lite followed by 3.1 Pro were the best performing models across the synthetic data set. Haiku 4.5 and GPT 5-Mini were unable to interpret the discrete option lists, often returning the list verbatim, which drastically decreased performance.

Figure 4: Forest plot displaying the median, 16th and 84th percentile for the overall accuracy excluding null fields across the real data set on the optimal prompt. Frontier Gemini and GPT models performed best with very little performance deviation across these models.

Model	Docs	Accuracy	Accuracy (w/o nulls)	Formatted Accuracy	Formatted Accuracy (w/o nulls)	WER	CER	Null Field Accuracy	Hallucination Rate
3.1 Pro	49	$0.85$ ${}_{-0.05}^{+0.05}$	$0.78$ ${}_{-0.09}^{+0.07}$	$0.88_{-0.16}^{+0.06}$	$0.67_{-0.25}^{+0.12}$	0.50	0.31	0.92	0.08
3.1 Flash Lite	49	$0.84_{-0.05}^{+0.04}$	$0.73_{-0.07}^{+0.09}$	$0.85_{-0.07}^{+0.08}$	$0.72_{-0.10}^{+0.10}$	0.53	0.40	0.92	0.08
2.5 Pro	49	$0.79_{-0.05}^{+0.06}$	$0.74_{-0.08}^{+0.07}$	$0.85_{-0.07}^{+0.06}$	$0.27_{-0.08}^{+0.12}$	0.60	0.40	0.86	0.14
2.5 Flash	49	$0.80_{-0.04}^{+0.04}$	$0.73_{-0.09}^{+0.07}$	$0.82_{-0.10}^{+0.07}$	$0.32_{-0.06}^{+0.13}$	0.60	0.45	0.89	0.11
GPT 5.4	49	$0.85$ ${}_{-0.05}^{+0.04}$	$0.76_{-0.09}^{+0.07}$	$0.89$ ${}_{-0.10}^{+0.07}$	$0.39_{-0.07}^{+0.13}$	0.56	0.43	0.94	0.06
GPT 5-Mini	49	$0.57_{-0.06}^{+0.09}$	$0.31_{-0.07}^{+0.17}$	$0.74_{-0.10}^{+0.11}$	$0.72_{-0.13}^{+0.12}$	0.72	0.64	0.83	0.17

Table 2: Performance metrics of various LLMs on the real-world document dataset using the optimized prompt and template. Metrics include median values with 16th and 84th percentiles; WER is word error rate, CER is character error rate—WER and CER were only calculated for free text fields. Hallucination rate indicates proportion of fabricated outputs. Gemini 3.1 Pro had the best performance overall and across free text fields minimising the CER and WER. GPT 5.4 had the lowest hallucination followed closely by the Gemini 3.1 family. Sonnet 4.6 had the best performance across formatted fields (dates and numerical values).

Model	Macro Precision	Macro Recall	Macro F1	Weighted Precision	Weighted Recall	Weighted F1
3.1 Pro	0.64	0.70	0.66	0.92	0.91	0.91
3.1 Flash Lite	0.58	0.58	0.56	0.87	0.87	0.87
2.5 Pro	0.46	0.50	0.46	0.88	0.84	0.85
2.5 Flash	0.52	0.57	0.51	0.89	0.86	0.87
GPT 5.4	0.54	0.60	0.56	0.89	0.89	0.89
GPT 5-Mini	0.22	0.13	0.15	0.66	0.31	0.40

Table 3: Performance metrics of discrete selection fields across the real data using the optimised prompt indicating the precision, recall and F1 on a macro and weighted level. Gemini 3.1 Pro outperformed all other models across weighted and macro metrics indicating it is the best model for discrete selection data.

Figure 5: Sectional accuracy across the largest sections in the real data set for each model using the optimal prompt. "Investigations" was the weakest performing section as a result of disordered formatted field transcription, the worst performing free text field and inconsistent normalisation and medical shorthands. "Medical and General" was the best performing field as it primarily consists of discrete selection fields.

Model	Birth Companion	Booking Visit	Counselling	Examination	Future Contraception	Gestational Age	Header ID	Investigations	Medical & Gen History	Mental Health	Obstetric & Neonatal History	Vaginal Exam
# Fields	1	2	26	14	8	18	7	43	27	4	37	7
3.1 Pro	$1.00$ ${}_{-0.00}^{+0.00}$	$0.00$ ${}_{-0.00}^{+0.00}$	$0.77_{-0.77}^{+0.23}$	$0.75_{-0.11}^{+0.13}$	$1.00$ ${}_{-0.29}^{+0.00}$	$0.83$ ${}_{-0.34}^{+0.17}$	$1.00$ ${}_{-0.17}^{+0.00}$	$0.53_{-0.15}^{+0.17}$	$0.96$ ${}_{-0.12}^{+0.04}$	$1.00$ ${}_{-0.28}^{+0.00}$	$0.88$ ${}_{-0.03}^{+0.13}$	$0.67$ ${}_{-0.07}^{+0.33}$
3.1 Flash Lite	$1.00$ ${}_{-0.00}^{+0.00}$	$0.00$ ${}_{-0.00}^{+0.00}$	$0.71_{-0.71}^{+0.29}$	$0.77$ ${}_{-0.13}^{+0.15}$	$0.67_{-0.00}^{+0.33}$	$0.50_{-0.31}^{+0.31}$	$0.83_{-0.03}^{+0.17}$	$0.56$ ${}_{-0.13}^{+0.13}$	$0.92_{-0.16}^{+0.08}$	$1.00$ ${}_{-0.33}^{+0.00}$	$0.86_{-0.15}^{+0.07}$	$0.60_{-0.20}^{+0.40}$
2.5 Pro	$1.00$ ${}_{-0.00}^{+0.00}$	$0.00$ ${}_{-0.00}^{+1.00}$	$0.75_{-0.42}^{+0.25}$	$0.71_{-0.10}^{+0.16}$	$1.00$ ${}_{-0.00}^{+0.00}$	$0.83_{-0.30}^{+0.17}$	$0.83_{-0.17}^{+0.00}$	$0.55_{-0.11}^{+0.12}$	$0.88_{-0.39}^{+0.12}$	$1.00$ ${}_{-0.25}^{+0.00}$	$0.83_{-0.18}^{+0.10}$	$0.60_{-0.20}^{+0.40}$
2.5 Flash	$1.00$ ${}_{-0.00}^{+0.00}$	$0.00$ ${}_{-0.00}^{+0.76}$	$0.63_{-0.63}^{+0.38}$	$0.71_{-0.11}^{+0.14}$	$1.00$ ${}_{-0.33}^{+0.00}$	$0.67_{-0.20}^{+0.22}$	$0.67_{-0.05}^{+0.17}$	$0.50_{-0.15}^{+0.13}$	$0.91_{-0.11}^{+0.05}$	$1.00$ ${}_{-0.28}^{+0.00}$	$0.86_{-0.14}^{+0.10}$	$0.60_{-0.20}^{+0.26}$
GPT 5.4	$1.00$ ${}_{-0.00}^{+0.00}$	$0.00$ ${}_{-0.00}^{+0.76}$	$1.00$ ${}_{-1.00}^{+0.00}$	$0.71_{-0.14}^{+0.12}$	$1.00$ ${}_{-0.25}^{+0.00}$	$0.67_{-0.17}^{+0.33}$	$0.83_{-0.17}^{+0.17}$	$0.56$ ${}_{-0.11}^{+0.13}$	$0.88_{-0.15}^{+0.08}$	$1.00$ ${}_{-0.25}^{+0.00}$	$0.79_{-0.15}^{+0.16}$	$0.60_{-0.20}^{+0.30}$

Table 4: Median accuracy scores and performance distribution bounds for each model across specific document sections for real-world data, with null values excluded to measure extraction success on populated fields. "Medical & General History" and "Future Contraception" are in majority discrete selection fields and achieve the highest performance. Investigations achieves the weakest performance due to disordered transcription.

Best Performing Fields (w/o nulls)	Worst Performing Fields (w/o nulls)	Most Hallucinated Fields
Discussed (0.98)	Other (0.00)	EDD method (1.0)
Outcome (0.95)	Screening for gestational diabetes 2 (0.00)	Family History Diabetes (0.98)
Discussed in record (0.95)	Tetox notes (0.03)	Family history TB (0.98)
Permission obtained (0.92)	Screening for gestational diabetes (0.06)	HIV status at booking (0.96)

Table 5: Top 4 best, worst, and most hallucinated fields across the real dataset with the optimal prompt and template. Averaged across all models excluding 5-Mini and Haiku; only fields with

n\geq 5

filled fields are included within the dataset are included. Most hallucinated fields is defined by the hallucination rate. Rates are shown in brackets.

Model	Docs	Accuracy	Accuracy (w/o nulls)	Formatted Accuracy	Formatted Accuracy (w/o nulls)	WER	CER	Null Field Accuracy	Hallucination Rate
3.1 Pro	49	$0.84$ ${}_{-0.08}^{+0.05}$	$0.78$ ${}_{-0.23}^{+0.06}$	$0.88_{-0.13}^{+0.08}$	$0.76_{-0.35}^{+0.20}$	0.34	0.92	0.08	0.08
3.1 Flash Lite	49	$0.80_{-0.04}^{+0.06}$	$0.68_{-0.01}^{+0.05}$	$0.85_{-0.08}^{+0.08}$	$0.71_{-0.14}^{+0.21}$	0.43	0.92	0.08	0.08
2.5 Pro	48	$0.78_{-0.08}^{+0.06}$	$0.73_{-0.13}^{+0.03}$	$0.86_{-0.07}^{+0.08}$	$0.79$ ${}_{-0.14}^{+0.09}$	0.62	0.42	0.84	0.16
2.5 Flash	49	$0.78_{-0.06}^{+0.06}$	$0.66_{-0.06}^{+0.09}$	$0.82_{-0.07}^{+0.07}$	$0.65_{-0.16}^{+0.15}$	0.62	0.46	0.88	0.12
GPT 5.4	48	$0.84$ ${}_{-0.04}^{+0.04}$	$0.73_{-0.07}^{+0.06}$	$0.90$ ${}_{-0.08}^{+0.04}$	$0.79$ ${}_{-0.15}^{+0.15}$	0.44	0.94	0.06	0.06
GPT 5-Mini	49	$0.71_{-0.07}^{+0.13}$	$0.49_{-0.01}^{+0.13}$	$0.75_{-0.09}^{+0.08}$	$0.39_{-0.17}^{+0.19}$	0.61	0.93	0.07	0.07

Table 6: Base (ablated) prompt performance metrics on the real-world document dataset. Metrics include median values with 16th and 84th percentiles; word error rate (WER) and character error rate (CER) measure the ratio of words and characters respectively—CER and WER were only calculated for free text fields. Hallucination rate indicates proportion of incorrect or fabricated outputs. Gemini 3.1 remains the best performing model, suprisingly the WER decreases for some models with this simplified prompt however the CER becomes substantially higher and overall performance suffers a noticeable decrease across most models. Budget models such as Haiku 4.5 and GPT 5-Mini experience a performance increase due to the models not being able to interpret the previous discretised template and returning the list of options rather than choosing the correct one.

Model	Macro Precision	Macro Recall	Macro F1	Weighted Precision	Weighted Recall	Weighted F1	$\Delta$ Macro F1	$\Delta$ Weighted F1
3.1 Pro	0.41	0.39	0.39	0.87	0.86	0.86	-0.27	-0.05
3.1 Flash Lite	0.32	0.28	0.28	0.83	0.82	0.82	-0.28	-0.05
2.5 Pro	0.24	0.23	0.22	0.86	0.78	0.80	-0.24	-0.05
2.5 Flash	0.32	0.31	0.30	0.84	0.81	0.82	-0.21	-0.05
GPT 5.4	0.32	0.31	0.30	0.88	0.86	0.87	-0.26	-0.02
GPT 5-Mini	0.29	0.22	0.24	0.72	0.66	0.66	0.09	0.22

Table 7: Base (ablated) prompt performance on the real-world document dataset showing precision, recall and F1 on a macro and weighted level. The last column shows the drop in weighted and macro F1 score respectively compared to the optimised prompt results. This is indicative of the model performing worse overall with the ablated prompt in discrete selection fields particularly surrounding uncommon selections.

Explore similar work

Aug 19, 2026

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios

Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to faithfully read real handwriting remains underexplored. Existing OCR benchmarks focus largely on printed text or clean single-line inputs, leaving limited coverage of realistic handwritten OCR scenarios such as multilingual handwriting, writer errors, and structurally complex mathematical expressions. We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR. It covers handwritten text recognition and handwritten mathematical expression recognition across six subtasks and twelve subsets, totaling 77.57K labeled images from public datasets and newly collected student writings. A key component is a difficulty-stratified multi-line formula corpus designed to test robustness under increasing structural complexity. We evaluate thirteen open- and closed-source systems with five complementary metrics under a unified protocol. Results show that current systems remain far from faithful transcription: performance drops sharply on complex multi-line formulas, model rankings vary across language and formula settings, and several generative models hallucinate plausible but visually unsupported corrections. OmniHandwritingOCR provides a challenging testbed for diagnosing language, content, structural, and visual-grounding failure modes of multimodal models in handwritten OCR scenarios.

Zinuo Guo, Min Zhang, Bo Jiang

East China Normal University Shanghai, China

Optical Character Recognition Multimodal Model Evaluation Multimodal Large Language Models

Ask PDF

Aug 24, 2026

WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans

While the top model on OmniDocBench now reaches 96.34% overall on printed-document parsing, the ability of current models to handle challenging handwritten documents remains largely uncharacterized. Existing benchmarks focus on isolated text or formulas, overlook handwritten tables and real-world degradation, and report aggregate accuracy without explaining why models fail. We present WildHandBench, a benchmark containing 500 handwritten documents across three structures (free text, tables, formulas), four languages, and nine real-world scenarios. We introduce a Prior-Driven Error (PDE) metric that quantifies whether errors originate from language priors rather than visual evidence. Evaluating 18 state-of-the-art models together with calibrated human baselines, we find: (1) the best model achieves only 71.85% overall; (2) humans outperform all models yet the gap is narrow (77.09% vs. 71.85%); and (3) model errors are qualitatively different from human errors -- 63-91% of model errors are prior-driven versus only 49% for humans, exposing systematic reliance on language priors that conventional accuracy metrics cannot capture.

Jun Zhang, Qiao Zhao, Cheng Cui +6

Baidu Inc.

Document Understanding Document Image Understanding Multimodal Model Evaluation Linguistic Bias in Language Models

Ask PDF

Mar 16, 2026

VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents

We introduce VAREX (VARied-schema EXtraction), a benchmark for evaluating multimodal foundation models on structured data extraction from government forms. VAREX employs a Reverse Annotation pipeline that programmatically fills PDF templates with synthetic values, producing deterministic ground truth validated through three-phase quality assurance. The benchmark comprises 1,777 documents with 1,771 unique schemas across three structural categories, each provided in four input modalities: plain text, layout-preserving text (whitespace-aligned to approximate column positions), document image, or both text and image combined. Unlike existing benchmarks that evaluate from a single input representation, VAREX provides four controlled modalities per document, enabling systematic ablation of how input format affects extraction accuracy -- a capability absent from prior benchmarks. We evaluate 20 models from frontier proprietary models to small open models, with particular attention to models <=4B parameters suitable for cost-sensitive and latency-constrained deployment. Results reveal that (1) below 4B parameters, structured output compliance -- not extraction capability -- is a dominant bottleneck; in particular, schema echo (models producing schema-conforming structure instead of extracted values) depresses scores by 45-65 pp (percentage points) in affected models; (2) extraction-specific fine-tuning at 2B yields +81 pp gains, demonstrating that the instruction-following deficit is addressable without scale; (3) layout-preserving text provides the largest accuracy gain (+3-18 pp), exceeding pixel-level visual cues; and (4) the benchmark most effectively discriminates models in the 60-95% accuracy band. Dataset and evaluation code are publicly available.

Udi Barzelay, Ophir Azulai, Inbar Shapira +4

IBM Research

Document Information Extraction Multimodal Model Evaluation Multimodal Foundation Models

Ask PDF