Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and a Vision-Language Model

Ari Vesalainen; Eetu Mäkelä; Laura Ruotsalainen; Mikko Tolonen

cs.CVFeb 16, 2026

Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and a Vision-Language Model

Authors: Ari Vesalainen, Eetu Mäkelä, Laura Ruotsalainen, Mikko Tolonen

Organizations: University of Helsinki, Department of Computer Science, Finland · University of Helsinki, Department of Digital Humanities, Finland

Abstract

Optical Character Recognition (OCR) of eighteenth-century printed texts remains challenging due to degraded print quality, archaic glyphs, and non-standardized orthography. Although transformer-based OCR systems and Vision-Language Models (VLMs) achieve strong aggregate accuracy, metrics such as Character Error Rate (CER) and Word Error Rate (WER) provide limited insight into their reliability for scholarly use. We compare a dedicated OCR transformer (TrOCR) and a general-purpose Vision-Language Model (Qwen) on line-level historical English texts using length-weighted accuracy metrics and hypothesis driven error analysis. While Qwen achieves lower CER/WER and greater robustness to degraded input, it exhibits selective linguistic regularization and orthographic normalization that may silently alter historically meaningful forms. TrOCR preserves orthographic fidelity more consistently but is more prone to cascading error propagation. Our findings show that architectural inductive biases shape OCR error structure in systematic ways. Models with similar aggregate accuracy can differ substantially in error locality, detectability, and downstream scholarly risk, underscoring the need for architecture-aware evaluation in historical digitization workflows.

Figures & tables

Subset	Model	CER (%)	WER (%)
All images	Qwen	$0.5986\pm 0.0428$	$2.4771\pm 0.1326$
All images	TrOCR	$1.0517\pm 0.0677$	$3.8909\pm 0.1632$
Coloured images	Qwen	$0.5001\pm 0.0465$	$2.2540\pm 0.1681$
Coloured images	TrOCR	$0.8157\pm 0.0602$	$3.3974\pm 0.1933$
Black & white images	Qwen	$0.7679\pm 0.0828$	$2.8654\pm 0.2295$
Black & white images	TrOCR	$1.4578\pm 0.1711$	$4.7496\pm 0.3349$

Table 1: Global OCR accuracy statistics (mean

\pm

95% CI) for Qwen and TrOCR across all images and by image type.

Line length	Model	CER (%)	WER (%)
Short	Qwen	$3.9263\pm 1.5643$	$9.1938\pm 2.6522$
Short	TrOCR	$5.7883\pm 1.3407$	$15.2758\pm 2.5528$
Medium	Qwen	$1.5636\pm 0.3778$	$5.7182\pm 1.1507$
Medium	TrOCR	$2.2191\pm 0.5248$	$7.5938\pm 1.3338$
Long	Qwen	$0.5422\pm 0.0402$	$2.2875\pm 0.1266$
Long	TrOCR	$0.9766\pm 0.0711$	$3.6142\pm 0.1680$

Table 2: Global OCR accuracy statistics (mean

\pm

95% CI) for Qwen and TrOCR by line-length category.

Proxy	Operational definition
Real-word error	Token differs from reference but remains attested in historical lexicon
Non-word error	Token differs from reference and is unattested in historical lexicon
Boundary error	Token split or merge due to incorrect whitespace placement
Glyph confusion	Systematic substitution between historically distinct glyph forms

Table 3: Definition of error proxies used in evaluation.

Panel of Figure 1 — Figure 1: Illustrations of error analysis

Ground Truth	TrOCR Output	QWEN Output
pre \longs ent conduct of affairs, and \longs uch apprehen \longs ions were entertained of futurity,	pre \longs ent conduct of affairs, and \longs uch apprehen \longs ions were entertained of \longs uturity ,	pre \longs ent conduct of affairs, and \longs uch apprehen \longs ions were entertained of futurity,
\longs ummon it, 322. One \longs ummoned under the influence of the covenant	\longs ummon it, 322. One \longs ummened under the influence of the covenant-	\longs ummon it, 322. One \longs ummoned under the influence of the covenant-

Table 4: Representative examples of visual grounding

GT word	OCR word	Context (GT $\rightarrow$ OCR)
tarded	seemed	tarded not the execution $\rightarrow$ seemed not the execution
rigorouſly	merely	rigorouſly enforced $\rightarrow$ merely enforced
Artay	free	Artay, commissions of $\rightarrow$ free commissions of
Groans	Gentleman	Groans of the Britains $\rightarrow$ Gentleman of the Britains
ward	used	ward to assert $\rightarrow$ used to assert

Table 5: Examples of linguistic regularization errors.

Type	Model	Ground Truth	OCR Output
Lexical	Qwen	Antient history	Ancient history
Lexical	Qwen	imploy their thoughts	employ their thoughts
Lexical	Qwen	phænomena	phaenomena
Ligature	Qwen	Celtæ	Celtae
Ligature	Qwen	Cæſar	Caeſar
Ligature	Qwen	Ætius	Aetius

Table 6: Representative examples of orthographic normalization. Qwen exhibits lexically motivated normalization beyond graphemic conventions, whereas TrOCR’s normalization is largely limited to ligature handling.

Model	Ground Truth	OCR Output
Qwen	Veſpaſian, i. 6. and ſucceeded by Julius Frontinus,	and ſucceeded by Julius Frontinus,
Qwen	the revolting barons are ſubdued by the prince a	the revolted barons are ſubdued by the prince at
TrOCR	—; lords of, ſee Lords.	—jects of, ſee hards.
TrOCR	his inequality to fulfil the conjugal duties; and he con-	his incapable to fulfil the conjugal duties; and he con-

Table 7: Representative examples illustrating error propagation and fragmentation (H4). Both models exhibit comparable edit distances, but differ in how errors spread within a line.

Explore similar work

When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents
Jul 27, 2026Marina Gardella, Camilo Mariño, Diego Belzarena +3VLM Evaluation VLM Hallucination
CB · Université Paris-Saclay, ENS Paris-Saclay, CNRS, Centre Borelli, France · UDELAR, CB +4
Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions
May 26, 2026Antonia Karamolegkou, Nicolas Angleraud, Benoît Sagot +1VLM Evaluation Document Understanding
Inria, Paris, France
A Benchmark of State-Space Models vs. Transformers and BiLSTM-based Models for Historical Newspaper OCR
Apr 1, 2026Merveilles Agbeti-Messan, Pierrick Tranouez, Stéphane Nicolas +2Selective SSMs Transformer
LITIS EA4108, University of Rouen Normandy, France · LITIS EA4108, INSA of Rouen Normandy, France

Jul 27, 2026J/K move · Enter open · S save

When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents

Marina Gardella, Camilo Mariño, Diego Belzarena, Ignacio Ramírez+2

CB · Université Paris-Saclay, ENS Paris-Saclay, CNRS, Centre Borelli, France · UDELAR, CB+4

VLM Evaluation VLM Hallucination Optical Character Recognition

Open paper AskPDF arXiv

Optical Character Recognition (OCR) is a key component in the digitization of historical archives. Recently, Vision-Language Models (VLMs) have emerged as strong alternatives to traditional OCR systems, achieving state-of-the-art performance on standard benchmarks. However, their suitability for archival transcription remains insufficiently understood. In this work, we benchmark traditional OCR systems and VLM-based approaches on the Berrutti dataset, a challenging collection of Uruguayan dictatorship-era documents derived from microfilm scans. While VLMs consistently outperform traditional methods in terms of Character Error Rate (CER) and Word Error Rate (WER), we show that these improvements hide a more complex picture. Through a detailed qualitative analysis, we uncover systematic failure modes that are invisible to standard metrics, including orthographic normalization, spurious content generation, and semantic substitutions that preserve fluency while altering meaning. Errors affecting named entities are particularly critical, as they can introduce substantial semantic distortions with minimal impact on CER and WER. These findings reveal a critical gap between quantitative OCR performance and transcription fidelity in real-world archival settings, and highlight the need for evaluation frameworks that go beyond character-level accuracy to capture the semantic reliability of generated transcriptions.

Abstract

Explore similar work

When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents

Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions

A Benchmark of State-Space Models vs. Transformers and BiLSTM-based Models for Historical Newspaper OCR