A Survey of OCR Evaluation Methods and Metrics and the Invisibility of Historical Documents

Fitsum Sileshi Beyene; Christopher L. Dancy

cs.CVMar 26, 2026

A Survey of OCR Evaluation Methods and Metrics and the Invisibility of Historical Documents

Authors: Fitsum Sileshi Beyene, Christopher L. Dancy

Organizations: The Pennsylvania State University University Park, Pennsylvania, USA

Abstract

Optical character recognition (OCR) and document understanding systems increasingly rely on large vision and vision-language models, yet evaluation remains centered on modern, Western, and institutional documents. This emphasis masks system behavior in historical and marginalized archives, where layout, typography, and material degradation shape interpretation. This study examines how OCR and document understanding systems are evaluated, with particular attention to Black historical newspapers. We review OCR and document understanding papers, as well as benchmark datasets, which are published between 2006 and 2025 using the PRISMA framework. We look into how the studies report training data, benchmark design, and evaluation metrics for vision transformer and multimodal OCR systems. During the review, we found that Black newspapers and other community-produced historical documents rarely appear in reported training data or evaluation benchmarks. Most evaluations emphasize character accuracy and task success on modern layouts. They rarely capture structural failures common in historical newspapers, including column collapse, typographic errors, and hallucinated text. To put these findings into perspective, we use previous empirical studies and archival statistics from significant Black press collections to show how evaluation gaps lead to structural invisibility and representational harm. We propose that these gaps occur due to organizational (meso) and institutional (macro) behaviors and structure, shaped by benchmark incentives and data governance decisions.

Figures & tables

Figure 1. PRISMA 2020 flow diagram for the systematic review of OCR evaluation methods and metrics in document understanding (2006–2025). PRISMA 2020 flow diagram

Model	Year	Training Source	Scale	Params	Scripts	Historical Coverage
olmOCR 2	2025	Web PDFs & Internet Archive	~260K pgs	~7B	EN+	Substantial (IA scans; no curation)
PaddleOCR-VL	2025	Synthetic & web corpora	Undisclosed	~0.9B	109+	Limited (no historical eval)
Qwen2.5-VL	2025	Web multimodal corpora	Web-scale	7B/72B	Multi	None reported
GOT-OCR 2.0	2024	Synthetic + modern web	~5M pairs	~580M	EN	None (synthetic focus)
Surya	2024	Scanned document images	Undisclosed	Undiscl.	90+	Limited (no benchmarks)
DocFormer v2	2024	Document imgs (IIT-CDIP)	~11M imgs	200–500M	EN	None reported

Table 1. Training Data Sources and Historical Coverage of Major OCR Systems (2021–2025)

System	Architecture	Historical Scans	Historical Typography	Complex Layouts	Handwriting
Generative / VLM (End-to-End Text Generation)
olmOCR 2	VLM (7B)	82.3% (Math) 47.7% (General)	Supported	83.7% (Multi-col)	Supported (20k pages)
Qwen2.5-VL	VLM (7B+)	65.5% (Bench)	High (Gen. Cap.)	Strong	Remarkable (Script-agnostic)
GOT-OCR 2.0	Encoder-Dec.	22.1% (Fail on noise)	Poor	Poor (reported)	Not evaluated
Nougat	Transformer	Poor (Hallucinates)	N/A	Academic Only	Poor
Donut	Transformer	Poor	Poor	Poor	Weak

Table 2. Comparative Evaluation of Critical Document Analysis Systems (2025)

Benchmark	Languages	Document Types	Annotation Paradigm	Hist. / Archival Coverage	Underrep. Archives
CC-OCR	Multi (28+)	Web PDF, scene, docs	Plain Text	None (modern web-crawl)	No targeted focus
OmniDoc	EN, ZH	Academic, slides, books	Detection & Text	Modern-era (Born-digital)	No targeted focus
OCRBench	EN, ZH	31 scenarios (mixed)	VQA / Key-Value	Limited (Isolated IAM/HKR)	No explicit focus
DUDE	Prim. EN	Multi-page business	QA Pairs	Contemporary (Scanned)	No targeted focus
DUE	English	Tables, graphs	QA / Classif.	None	No targeted focus
DocVQA	English	Industry forms (Tobacco)	QA Pairs (ANLS)	Legacy Litigation (20th C.)	Single-domain

Table 3. Evaluation Coverage of Major OCR and Document Understanding Benchmarks

Error Regime	Model	Technical Failure	Archival Consequence
Geometric Collapse (Linearity Bias)	Tesseract v5	Misinterpreted vertical column rules as text; defaulted to left-to-right “Z-pattern” reading order	Semantic merging of distinct content (editorial poetry conflated with civic reports); granular search rendered ineffective
Decoding Instability	Surya	High-contrast 19th-c. fonts triggered token collapse; generative repetition of character sequences	Introduction of “garbage tokens”
Corrective Hallucination	olmOCR 2 (SOTA VLM)	Overwrote a visual evidence with high-probability tokens (“knowledge bleed?”)	Entities were replaced with a fabricated text;

Table 4. Failure patterns from the The Weekly Advocate (1837)

Publication / Series	Years	Layout Topology	Masthead / Header	Non-Text Elements	Typography Complexity	Scan Degradation
The Weekly Advocate & The Colored American (Transitional newspapers)	1837–1841	4 columns, standard grid frequently interrupted by advertisements & spanning headers	High complexity, ribbon banners with multiple embedded text (e.g., “Established for …” & bold serif headers with sub-text	High variance, heavy horizontal rules, woodcut engraving of the U.S. Capitol, & iconographic glyphs in advertisements	Moderate–high, modern serif (Didone), tight leading, archaic ligatures	bleed-through
Mirror of Liberty (Quarterly magazine)	1838–1841	2 columns, wider column width, magazine aspect ratio	Low complexity, simple typographic header, outline block type	Minimal, primarily text-heavy, sparse rules	Moderate, larger point size, clear typographic hierarchy	low contrast, character dropout
The North Star (Abolitionist)	1847–1851	7 columns, extremely dense layout	Moderate, typographic masthead often obscured by scan artifacts	Structural emphasis, thin and broken vertical rules	Very high, minimal point size, dense setting, italics for emphasis	blur, speckle noise, skew
The Impartial Citizen	1861–1864	5 columns, dense grid with slightly wider margins	High, Gothic/Blackletter masthead distinct from Roman body text	Standard, vertical column rules	High, dense text blocks challenge line segmentation	lighting gradients from microfilm
The Weekly Anglo-African	1859–1861	7 columns, header-centered structure	Moderate, detailed masthead with integrated text	Integrated artwork, motto embedded in engraving	Mixed, display fonts in header vs. standard body text	cleaner lines than other samples

Table 5. Structural, temporal, and archival characteristics of some Black press documents.

Figure 2. Original vs Surya Layout analysis vs Tesseract

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Query Category	Search Query
Primary evaluation queries	“OCR evaluation metrics document understanding”
	“document AI benchmark evaluation”
	“OCR benchmark dataset peer reviewed”
	“document layout analysis benchmark”
	“vision language model OCR evaluation”
Historical & bias-aware queries	“historical document OCR evaluation”

Appendix

Table 6. Search queries used in the PRISMA 2020 systematic review of OCR evaluation literature.

Explore similar work

Feb 16, 2026

Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and a Vision-Language Model

Optical Character Recognition (OCR) of eighteenth-century printed texts remains challenging due to degraded print quality, archaic glyphs, and non-standardized orthography. Although transformer-based OCR systems and Vision-Language Models (VLMs) achieve strong aggregate accuracy, metrics such as Character Error Rate (CER) and Word Error Rate (WER) provide limited insight into their reliability for scholarly use. We compare a dedicated OCR transformer (TrOCR) and a general-purpose Vision-Language Model (Qwen) on line-level historical English texts using length-weighted accuracy metrics and hypothesis driven error analysis. While Qwen achieves lower CER/WER and greater robustness to degraded input, it exhibits selective linguistic regularization and orthographic normalization that may silently alter historically meaningful forms. TrOCR preserves orthographic fidelity more consistently but is more prone to cascading error propagation. Our findings show that architectural inductive biases shape OCR error structure in systematic ways. Models with similar aggregate accuracy can differ substantially in error locality, detectability, and downstream scholarly risk, underscoring the need for architecture-aware evaluation in historical digitization workflows.

Ari Vesalainen, Eetu Mäkelä, Laura Ruotsalainen +1

University of Helsinki, Department of Computer Science, Finland · University of Helsinki, Department of Digital Humanities, Finland

Optical Character Recognition VLM Evaluation

Ask PDF

Jul 27, 2026

When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents

Optical Character Recognition (OCR) is a key component in the digitization of historical archives. Recently, Vision-Language Models (VLMs) have emerged as strong alternatives to traditional OCR systems, achieving state-of-the-art performance on standard benchmarks. However, their suitability for archival transcription remains insufficiently understood. In this work, we benchmark traditional OCR systems and VLM-based approaches on the Berrutti dataset, a challenging collection of Uruguayan dictatorship-era documents derived from microfilm scans. While VLMs consistently outperform traditional methods in terms of Character Error Rate (CER) and Word Error Rate (WER), we show that these improvements hide a more complex picture. Through a detailed qualitative analysis, we uncover systematic failure modes that are invisible to standard metrics, including orthographic normalization, spurious content generation, and semantic substitutions that preserve fluency while altering meaning. Errors affecting named entities are particularly critical, as they can introduce substantial semantic distortions with minimal impact on CER and WER. These findings reveal a critical gap between quantitative OCR performance and transcription fidelity in real-world archival settings, and highlight the need for evaluation frameworks that go beyond character-level accuracy to capture the semantic reliability of generated transcriptions.

Marina Gardella, Camilo Mariño, Diego Belzarena +3

CB · Université Paris-Saclay, ENS Paris-Saclay, CNRS, Centre Borelli, France · UDELAR, CB +4

VLM Evaluation VLM Hallucination Optical Character Recognition

Ask PDF

Oct 12, 2026

DDVR-Net: Degradation-Aware Decoupling and Visibility-Guided Restoration for Historical Document Recognition

Historical document analysis is crucial for cultural heritage preservation and low-resource language digitization. However, real historical documents often suffer from severe degradation, such as ink erosion and structural damage, causing mixed-visibility character sequences. Conventional OCR systems struggle in such evidence-limited scenarios, while contextual language models may over-correct visually clear characters. To address these challenges, we propose DDVR-Net, a degradation-aware decoupling and visibility-guided restoration framework for historical document recognition. DDVR-Net decouples intrinsic glyph structures from physical corruption at the feature level and predicts local visibility to estimate character reliability. A contextual decoder is selectively activated through a visibility-guided gate, restoring degraded characters while preserving clear visual recognition. Furthermore, we construct a low-resource multilingual benchmark covering Tibetan, Mongolian, and Uyghur scripts with 187000 document pairs, combining large-scale synthetic data and progressive training with real-world annotations to reduce the synthetic-to-real gap. Extensive experiments demonstrate that DDVR-Net consistently outperforms existing methods across three languages and diverse reading orders, providing an effective framework and benchmark for low-resource historical document recognition. Code and Benchmark: https://github.com/Shajiu/DDVR-Net

Jiu Sha, Yu Weng, Mengxiao Zhu

Institute of National Security, Minzu University of China, Beijing, China · School of Information Engineering, Minzu University of China, Beijing, China · School of Artificial Intelligence and Computer Science, North China University of Technology, Beijing, China

Optical Character Recognition Historical Document Processing Low-Resource Language Processing

Ask PDF