From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms

Nicholas Pather; Joshua Fouché; Sitwala Mundia; Karl-Günter Technau; Thokozile Malaba; Alex Welte; Ushma Mehta; Bruce A. Bassett

cs.CVApr 14, 2026

From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms

Authors: Nicholas Pather, Joshua Fouché, Sitwala Mundia, Karl-Günter Technau, Thokozile Malaba, Alex Welte, Ushma Mehta, Bruce A. Bassett

Organizations: CSAM, University of the Witwatersrand & Grai Labs, Johannesburg, South Africa · Grai Labs, Cape Town, South Africa · Faculty of Health Sciences, University of the Witwatersrand, South Africa · Empilweni Services and Research Unit, Department of Paediatrics and Child Health, University of the Witwatersrand, Johannesburg, South Africa · Division of Epidemiology and Biostatistics, School of Public Health, Faculty of Health Sciences, University of Cape Town, South Africa · Discipline of Public Health, School of Medicine, University of KwaZulu-Natal, South Africa · Division of Clinical Pharmacology, Department of Medicine, University of Cape Town, South Africa · WITS MIND Institute and CSAM, University of the Witwatersrand, University of Cape Town & Grai Labs, Cape Town, South Africa

Abstract

Manual digitisation of structured handwritten documents is slow and costly. We benchmark 17 leading frontier multi-modal large language models and open-source models against a very challenging real-world medical form that mixes dates; structured, printed text; hand-written responses and significant variability challenges. None of the smaller or older models perform well but the latest Google and OpenAI models reach accuracies around $85\%$ with weighted F1 scores $\simeq 90\%$ across the discrete or predefined fields despite the very challenging nature of the responses. Clear task specific strengths emerge: GPT 5.4 excels in noisy date extraction as well as reliability with the lowest hallucination rate ( $6\%$ ). Claude Sonnet 4.6 had the best average performance across formatted fields (dates and numerical values), while Gemini 3.1 delivered the best overall performance, with the lowest free text error rates (WER = $0.50$ and CER = $0.31$ ) and the strongest results across discrete classification metrics. We further show that prompt optimisation dramatically improves macro precision, recall and F1 by over $60\%$ , but has little impact on weighted metrics (only $\sim2-5\%$ improvement). These results provide evidence that the rapid improvements of multimodal large language models offer a compelling pathway toward fully automated digitisation of complex handwritten workflows that is particularly relevant in low- and middle-income countries.

Figures & tables

Figure 1: Our maternity case record with example realistic synthetic data. The form shows a challenging mix of dates, abbreviations, idiosyncratic responses and structural anomalies such as text written outside the intended field space. Note the handwriting in the synthetic data is often substantially neater than that of the real data set.

Panel of Figure 2 — Figure 2: End-to-end pipeline used for testing frontier LLM’s on our medical benchmark dataset.

Figure 3: Forest plot displaying the median, 16th and 84th percentile for the overall accuracy excluding null fields across the synthetic data set on the optimal prompt. This is indicative that the small, open-source models were the weakest performing by a large margin $\geq 40\%$ from best performing models.

Model	Accuracy	Accuracy (w/o Nulls)	Formatted Accuracy	WER	CER	Null Field Accuracy	Hallucination Rate
3.1 Pro	$0.84_{-0.08}^{+0.07}$	$0.75_{-0.06}^{+0.09}$	$0.84_{-0.18}^{+0.12}$	0.33	0.15	0.93	0.07
3.1 Flash Lite	$0.86$ ${}_{-0.09}^{+0.05}$	$0.83$ ${}_{-0.08}^{+0.03}$	$0.91$ ${}_{-0.17}^{+0.04}$	0.40	0.21	0.91	0.09
2.5 Pro	$0.79_{-0.07}^{+0.06}$	$0.72_{-0.04}^{+0.06}$	$0.76_{-0.12}^{+0.10}$	0.39	0.18	0.87	0.13
2.5 Flash	$0.75_{-0.04}^{+0.07}$	$0.67_{-0.04}^{+0.07}$	$0.71_{-0.12}^{+0.14}$	0.45	0.26	0.86	0.14
GPT 5.4	$0.80_{-0.08}^{+0.05}$	$0.67_{-0.05}^{+0.07}$	$0.73_{-0.13}^{+0.12}$	0.47	0.21	0.91	0.09
GPT 5.2-Pro	$0.76_{-0.08}^{+0.07}$	$0.62_{-0.01}^{+0.06}$	$0.72_{-0.14}^{+0.12}$	0.60	0.38	0.89	0.11

Table 1: Performance metrics of various LLMs on the synthetic dataset using the optimized prompt and template. Metrics show median values with 16th and 84th percentiles; WER is word error rate CER is character error rate—WER and CER were only calculated across free text fields. Hallucination rate indicates proportion of fabricated outputs. The open-source models are shown in the last three rows and are not competitive in every metric. Gemini 3.1 Flash Lite followed by 3.1 Pro were the best performing models across the synthetic data set. Haiku 4.5 and GPT 5-Mini were unable to interpret the discrete option lists, often returning the list verbatim, which drastically decreased performance.

Figure 4: Forest plot displaying the median, 16th and 84th percentile for the overall accuracy excluding null fields across the real data set on the optimal prompt. Frontier Gemini and GPT models performed best with very little performance deviation across these models.

Model	Docs	Accuracy	Accuracy (w/o nulls)	Formatted Accuracy	Formatted Accuracy (w/o nulls)	WER	CER	Null Field Accuracy	Hallucination Rate
3.1 Pro	49	$0.85$ ${}_{-0.05}^{+0.05}$	$0.78$ ${}_{-0.09}^{+0.07}$	$0.88_{-0.16}^{+0.06}$	$0.67_{-0.25}^{+0.12}$	0.50	0.31	0.92	0.08
3.1 Flash Lite	49	$0.84_{-0.05}^{+0.04}$	$0.73_{-0.07}^{+0.09}$	$0.85_{-0.07}^{+0.08}$	$0.72_{-0.10}^{+0.10}$	0.53	0.40	0.92	0.08
2.5 Pro	49	$0.79_{-0.05}^{+0.06}$	$0.74_{-0.08}^{+0.07}$	$0.85_{-0.07}^{+0.06}$	$0.27_{-0.08}^{+0.12}$	0.60	0.40	0.86	0.14
2.5 Flash	49	$0.80_{-0.04}^{+0.04}$	$0.73_{-0.09}^{+0.07}$	$0.82_{-0.10}^{+0.07}$	$0.32_{-0.06}^{+0.13}$	0.60	0.45	0.89	0.11
GPT 5.4	49	$0.85$ ${}_{-0.05}^{+0.04}$	$0.76_{-0.09}^{+0.07}$	$0.89$ ${}_{-0.10}^{+0.07}$	$0.39_{-0.07}^{+0.13}$	0.56	0.43	0.94	0.06
GPT 5-Mini	49	$0.57_{-0.06}^{+0.09}$	$0.31_{-0.07}^{+0.17}$	$0.74_{-0.10}^{+0.11}$	$0.72_{-0.13}^{+0.12}$	0.72	0.64	0.83	0.17

Table 2: Performance metrics of various LLMs on the real-world document dataset using the optimized prompt and template. Metrics include median values with 16th and 84th percentiles; WER is word error rate, CER is character error rate—WER and CER were only calculated for free text fields. Hallucination rate indicates proportion of fabricated outputs. Gemini 3.1 Pro had the best performance overall and across free text fields minimising the CER and WER. GPT 5.4 had the lowest hallucination followed closely by the Gemini 3.1 family. Sonnet 4.6 had the best performance across formatted fields (dates and numerical values).

Model	Macro Precision	Macro Recall	Macro F1	Weighted Precision	Weighted Recall	Weighted F1
3.1 Pro	0.64	0.70	0.66	0.92	0.91	0.91
3.1 Flash Lite	0.58	0.58	0.56	0.87	0.87	0.87
2.5 Pro	0.46	0.50	0.46	0.88	0.84	0.85
2.5 Flash	0.52	0.57	0.51	0.89	0.86	0.87
GPT 5.4	0.54	0.60	0.56	0.89	0.89	0.89
GPT 5-Mini	0.22	0.13	0.15	0.66	0.31	0.40

Table 3: Performance metrics of discrete selection fields across the real data using the optimised prompt indicating the precision, recall and F1 on a macro and weighted level. Gemini 3.1 Pro outperformed all other models across weighted and macro metrics indicating it is the best model for discrete selection data.

Figure 5: Sectional accuracy across the largest sections in the real data set for each model using the optimal prompt. "Investigations" was the weakest performing section as a result of disordered formatted field transcription, the worst performing free text field and inconsistent normalisation and medical shorthands. "Medical and General" was the best performing field as it primarily consists of discrete selection fields.

Model	Birth Companion	Booking Visit	Counselling	Examination	Future Contraception	Gestational Age	Header ID	Investigations	Medical & Gen History	Mental Health	Obstetric & Neonatal History	Vaginal Exam
# Fields	1	2	26	14	8	18	7	43	27	4	37	7
3.1 Pro	$1.00$ ${}_{-0.00}^{+0.00}$	$0.00$ ${}_{-0.00}^{+0.00}$	$0.77_{-0.77}^{+0.23}$	$0.75_{-0.11}^{+0.13}$	$1.00$ ${}_{-0.29}^{+0.00}$	$0.83$ ${}_{-0.34}^{+0.17}$	$1.00$ ${}_{-0.17}^{+0.00}$	$0.53_{-0.15}^{+0.17}$	$0.96$ ${}_{-0.12}^{+0.04}$	$1.00$ ${}_{-0.28}^{+0.00}$	$0.88$ ${}_{-0.03}^{+0.13}$	$0.67$ ${}_{-0.07}^{+0.33}$
3.1 Flash Lite	$1.00$ ${}_{-0.00}^{+0.00}$	$0.00$ ${}_{-0.00}^{+0.00}$	$0.71_{-0.71}^{+0.29}$	$0.77$ ${}_{-0.13}^{+0.15}$	$0.67_{-0.00}^{+0.33}$	$0.50_{-0.31}^{+0.31}$	$0.83_{-0.03}^{+0.17}$	$0.56$ ${}_{-0.13}^{+0.13}$	$0.92_{-0.16}^{+0.08}$	$1.00$ ${}_{-0.33}^{+0.00}$	$0.86_{-0.15}^{+0.07}$	$0.60_{-0.20}^{+0.40}$
2.5 Pro	$1.00$ ${}_{-0.00}^{+0.00}$	$0.00$ ${}_{-0.00}^{+1.00}$	$0.75_{-0.42}^{+0.25}$	$0.71_{-0.10}^{+0.16}$	$1.00$ ${}_{-0.00}^{+0.00}$	$0.83_{-0.30}^{+0.17}$	$0.83_{-0.17}^{+0.00}$	$0.55_{-0.11}^{+0.12}$	$0.88_{-0.39}^{+0.12}$	$1.00$ ${}_{-0.25}^{+0.00}$	$0.83_{-0.18}^{+0.10}$	$0.60_{-0.20}^{+0.40}$
2.5 Flash	$1.00$ ${}_{-0.00}^{+0.00}$	$0.00$ ${}_{-0.00}^{+0.76}$	$0.63_{-0.63}^{+0.38}$	$0.71_{-0.11}^{+0.14}$	$1.00$ ${}_{-0.33}^{+0.00}$	$0.67_{-0.20}^{+0.22}$	$0.67_{-0.05}^{+0.17}$	$0.50_{-0.15}^{+0.13}$	$0.91_{-0.11}^{+0.05}$	$1.00$ ${}_{-0.28}^{+0.00}$	$0.86_{-0.14}^{+0.10}$	$0.60_{-0.20}^{+0.26}$
GPT 5.4	$1.00$ ${}_{-0.00}^{+0.00}$	$0.00$ ${}_{-0.00}^{+0.76}$	$1.00$ ${}_{-1.00}^{+0.00}$	$0.71_{-0.14}^{+0.12}$	$1.00$ ${}_{-0.25}^{+0.00}$	$0.67_{-0.17}^{+0.33}$	$0.83_{-0.17}^{+0.17}$	$0.56$ ${}_{-0.11}^{+0.13}$	$0.88_{-0.15}^{+0.08}$	$1.00$ ${}_{-0.25}^{+0.00}$	$0.79_{-0.15}^{+0.16}$	$0.60_{-0.20}^{+0.30}$

Table 4: Median accuracy scores and performance distribution bounds for each model across specific document sections for real-world data, with null values excluded to measure extraction success on populated fields. "Medical & General History" and "Future Contraception" are in majority discrete selection fields and achieve the highest performance. Investigations achieves the weakest performance due to disordered transcription.

Best Performing Fields (w/o nulls)	Worst Performing Fields (w/o nulls)	Most Hallucinated Fields
Discussed (0.98)	Other (0.00)	EDD method (1.0)
Outcome (0.95)	Screening for gestational diabetes 2 (0.00)	Family History Diabetes (0.98)
Discussed in record (0.95)	Tetox notes (0.03)	Family history TB (0.98)
Permission obtained (0.92)	Screening for gestational diabetes (0.06)	HIV status at booking (0.96)

Table 5: Top 4 best, worst, and most hallucinated fields across the real dataset with the optimal prompt and template. Averaged across all models excluding 5-Mini and Haiku; only fields with

n\geq 5

filled fields are included within the dataset are included. Most hallucinated fields is defined by the hallucination rate. Rates are shown in brackets.

Model	Docs	Accuracy	Accuracy (w/o nulls)	Formatted Accuracy	Formatted Accuracy (w/o nulls)	WER	CER	Null Field Accuracy	Hallucination Rate
3.1 Pro	49	$0.84$ ${}_{-0.08}^{+0.05}$	$0.78$ ${}_{-0.23}^{+0.06}$	$0.88_{-0.13}^{+0.08}$	$0.76_{-0.35}^{+0.20}$	0.34	0.92	0.08	0.08
3.1 Flash Lite	49	$0.80_{-0.04}^{+0.06}$	$0.68_{-0.01}^{+0.05}$	$0.85_{-0.08}^{+0.08}$	$0.71_{-0.14}^{+0.21}$	0.43	0.92	0.08	0.08
2.5 Pro	48	$0.78_{-0.08}^{+0.06}$	$0.73_{-0.13}^{+0.03}$	$0.86_{-0.07}^{+0.08}$	$0.79$ ${}_{-0.14}^{+0.09}$	0.62	0.42	0.84	0.16
2.5 Flash	49	$0.78_{-0.06}^{+0.06}$	$0.66_{-0.06}^{+0.09}$	$0.82_{-0.07}^{+0.07}$	$0.65_{-0.16}^{+0.15}$	0.62	0.46	0.88	0.12
GPT 5.4	48	$0.84$ ${}_{-0.04}^{+0.04}$	$0.73_{-0.07}^{+0.06}$	$0.90$ ${}_{-0.08}^{+0.04}$	$0.79$ ${}_{-0.15}^{+0.15}$	0.44	0.94	0.06	0.06
GPT 5-Mini	49	$0.71_{-0.07}^{+0.13}$	$0.49_{-0.01}^{+0.13}$	$0.75_{-0.09}^{+0.08}$	$0.39_{-0.17}^{+0.19}$	0.61	0.93	0.07	0.07

Table 6: Base (ablated) prompt performance metrics on the real-world document dataset. Metrics include median values with 16th and 84th percentiles; word error rate (WER) and character error rate (CER) measure the ratio of words and characters respectively—CER and WER were only calculated for free text fields. Hallucination rate indicates proportion of incorrect or fabricated outputs. Gemini 3.1 remains the best performing model, suprisingly the WER decreases for some models with this simplified prompt however the CER becomes substantially higher and overall performance suffers a noticeable decrease across most models. Budget models such as Haiku 4.5 and GPT 5-Mini experience a performance increase due to the models not being able to interpret the previous discretised template and returning the list of options rather than choosing the correct one.

Model	Macro Precision	Macro Recall	Macro F1	Weighted Precision	Weighted Recall	Weighted F1	$\Delta$ Macro F1	$\Delta$ Weighted F1
3.1 Pro	0.41	0.39	0.39	0.87	0.86	0.86	-0.27	-0.05
3.1 Flash Lite	0.32	0.28	0.28	0.83	0.82	0.82	-0.28	-0.05
2.5 Pro	0.24	0.23	0.22	0.86	0.78	0.80	-0.24	-0.05
2.5 Flash	0.32	0.31	0.30	0.84	0.81	0.82	-0.21	-0.05
GPT 5.4	0.32	0.31	0.30	0.88	0.86	0.87	-0.26	-0.02
GPT 5-Mini	0.29	0.22	0.24	0.72	0.66	0.66	0.09	0.22

Table 7: Base (ablated) prompt performance on the real-world document dataset showing precision, recall and F1 on a macro and weighted level. The last column shows the drop in weighted and macro F1 score respectively compared to the optimised prompt results. This is indicative of the model performing worse overall with the ablated prompt in discrete selection fields particularly surrounding uncommon selections.

Explore similar work

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios
Aug 19, 2026Zinuo Guo, Min Zhang, Bo JiangOptical Character Recognition Multimodal Model Evaluation
East China Normal University Shanghai, China
WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans
Aug 24, 2026Jun Zhang, Qiao Zhao, Cheng Cui +6Document Understanding Document Image Understanding
Baidu Inc.
VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents
Mar 16, 2026Udi Barzelay, Ophir Azulai, Inbar Shapira +4Document Information Extraction Multimodal Model Evaluation
IBM Research

Aug 19, 2026J/K move · Enter open · S save

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios

Zinuo Guo, Min Zhang, Bo Jiang

East China Normal University Shanghai, China

Optical Character Recognition Multimodal Model Evaluation Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to faithfully read real handwriting remains underexplored. Existing OCR benchmarks focus largely on printed text or clean single-line inputs, leaving limited coverage of realistic handwritten OCR scenarios such as multilingual handwriting, writer errors, and structurally complex mathematical expressions. We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR. It covers handwritten text recognition and handwritten mathematical expression recognition across six subtasks and twelve subsets, totaling 77.57K labeled images from public datasets and newly collected student writings. A key component is a difficulty-stratified multi-line formula corpus designed to test robustness under increasing structural complexity. We evaluate thirteen open- and closed-source systems with five complementary metrics under a unified protocol. Results show that current systems remain far from faithful transcription: performance drops sharply on complex multi-line formulas, model rankings vary across language and formula settings, and several generative models hallucinate plausible but visually unsupported corrections. OmniHandwritingOCR provides a challenging testbed for diagnosing language, content, structural, and visual-grounding failure modes of multimodal models in handwritten OCR scenarios.

Abstract

Explore similar work

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios

WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans

VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents