Organizations: UDELAR, CB · Facultad de Ingenier´ıa, Universidad de la Rep´ublica, Montevideo, Uruguay · Universit´e Paris-Saclay, ENS Paris-Saclay, CNRS, Centre Borelli, France · CB · UDELAR · LU - Hong Kong · Lingnan University, Hong Kong
Optical Character Recognition (OCR) is essential for digitizing historical archives, yet standard evaluation metrics such as Character Error Rate (CER) do not necessarily reflect whether meaningful information, such as names, dates, and identifiers, is preserved. In this work, we investigate entitylevel analysis of OCR errors as a complementary perspective for evaluating OCR quality in historical documents. Using a publicly available benchmark of 175 documents from the Berrutti Archive, augmented with 5,209 manually annotated entity mentions that will be released upon acceptance, we compare sequence-recognition OCR systems, OCR-specialized vision-language models, and general-purpose vision-language models. Beyond global transcription accuracy, we analyze entity preservation and recoverability under varying character-level edit tolerances. Our experiments show that OCR rankings depend on the evaluation criterion: while Qwen 3.6 achieves the highest entity coverage across all tested tolerances, several systems with lower exact-match coverage recover substantially when small errors are tolerated, suggesting that many of their errors correspond to localized character-level deviations rather than fundamentally incorrect entity transcriptions. Conversely, some systems remain unable to recover many entities even under relaxed matching. These findings show that OCR quality cannot be fully characterized by aggregate transcription metrics alone and highlight the importance of evaluating how OCR errors affect the preservation and recoverability of meaningful information.
Figures & tables
Fig. 1: Annotation statistics. Bars indicate the total number of annotated entity mentions for each category, while the accompanying values report the number of documents containing at least one instance of that entity type.
Model
CER (%)
Entity CER (%)
All
Date
ID
Loc.
Org.
Person
Surya-OCR
8.15
9.91
12.67
26.67
2.96
4.84
11.90
Qwen 3.6
9.78
11.51
7.67
14.32
2.74
4.98
17.05
PP-OCRv5
10.73
15.14
19.76
23.09
7.09
9.36
18.53
DocTR
10.82
16.25
12.12
14.66
6.00
6.93
24.08
MinerU2.5-Pro
11.11
20.60
11.03
89.58
3.98
6.51
24.92
TABLE I: CER and Entity CER (%) by entity type on the Berrutti dataset. Best results are in bold and second-best results are underlined. The ⋆ in Qwen 3.6 indicates reasoning mode.
Model
Cov@0 (%)
All
Date
ID
Loc.
Org.
Person
Qwen 3.6
85.39
86.64
79.17
90.65
85.39
84.88
Chandra-OCR
73.09
84.91
58.75
90.97
86.07
58.97
Qwen 3.6 ⋆
68.02
78.66
57.92
77.42
77.06
59.12
MinerU2.5-Pro
67.54
78.02
54.72
89.14
80.97
51.09
Surya-OCR
67.19
73.49
68.06
83.44
73.92
53.34
TABLE II: Cov@0 (%) by entity type on the Berrutti dataset. Best results are highlighted in bold and second-best results are underlined. The ⋆ in Qwen 3.6 indicates reasoning mode.
Fig. 2: Entity coverage as a function of the tolerated edit distance k . The highlighted curves correspond to representative OCR systems, while the gray curves show the remaining evaluated methods.
Optical Character Recognition (OCR) is a key component in the digitization of historical archives. Recently, Vision-Language Models (VLMs) have emerged as strong alternatives to traditional OCR systems, achieving state-of-the-art performance on standard benchmarks. However, their suitability for archival transcription remains insufficiently understood. In this work, we benchmark traditional OCR systems and VLM-based approaches on the Berrutti dataset, a challenging collection of Uruguayan dictatorship-era documents derived from microfilm scans. While VLMs consistently outperform traditional methods in terms of Character Error Rate (CER) and Word Error Rate (WER), we show that these improvements hide a more complex picture. Through a detailed qualitative analysis, we uncover systematic failure modes that are invisible to standard metrics, including orthographic normalization, spurious content generation, and semantic substitutions that preserve fluency while altering meaning. Errors affecting named entities are particularly critical, as they can introduce substantial semantic distortions with minimal impact on CER and WER. These findings reveal a critical gap between quantitative OCR performance and transcription fidelity in real-world archival settings, and highlight the need for evaluation frameworks that go beyond character-level accuracy to capture the semantic reliability of generated transcriptions.
Marina Gardella, Camilo Mariño, Diego Belzarena +3
CB · Université Paris-Saclay, ENS Paris-Saclay, CNRS, Centre Borelli, France · UDELAR, CB +4
Optical Character Recognition (OCR) of eighteenth-century printed texts remains challenging due to degraded print quality, archaic glyphs, and non-standardized orthography. Although transformer-based OCR systems and Vision-Language Models (VLMs) achieve strong aggregate accuracy, metrics such as Character Error Rate (CER) and Word Error Rate (WER) provide limited insight into their reliability for scholarly use. We compare a dedicated OCR transformer (TrOCR) and a general-purpose Vision-Language Model (Qwen) on line-level historical English texts using length-weighted accuracy metrics and hypothesis driven error analysis. While Qwen achieves lower CER/WER and greater robustness to degraded input, it exhibits selective linguistic regularization and orthographic normalization that may silently alter historically meaningful forms. TrOCR preserves orthographic fidelity more consistently but is more prone to cascading error propagation. Our findings show that architectural inductive biases shape OCR error structure in systematic ways. Models with similar aggregate accuracy can differ substantially in error locality, detectability, and downstream scholarly risk, underscoring the need for architecture-aware evaluation in historical digitization workflows.
Ari Vesalainen, Eetu Mäkelä, Laura Ruotsalainen +1
University of Helsinki, Department of Computer Science, Finland · University of Helsinki, Department of Digital Humanities, Finland
Optical character recognition (OCR) and document understanding systems increasingly rely on large vision and vision-language models, yet evaluation remains centered on modern, Western, and institutional documents. This emphasis masks system behavior in historical and marginalized archives, where layout, typography, and material degradation shape interpretation. This study examines how OCR and document understanding systems are evaluated, with particular attention to Black historical newspapers. We review OCR and document understanding papers, as well as benchmark datasets, which are published between 2006 and 2025 using the PRISMA framework. We look into how the studies report training data, benchmark design, and evaluation metrics for vision transformer and multimodal OCR systems. During the review, we found that Black newspapers and other community-produced historical documents rarely appear in reported training data or evaluation benchmarks. Most evaluations emphasize character accuracy and task success on modern layouts. They rarely capture structural failures common in historical newspapers, including column collapse, typographic errors, and hallucinated text. To put these findings into perspective, we use previous empirical studies and archival statistics from significant Black press collections to show how evaluation gaps lead to structural invisibility and representational harm. We propose that these gaps occur due to organizational (meso) and institutional (macro) behaviors and structure, shaped by benchmark incentives and data governance decisions.
Fitsum Sileshi Beyene, Christopher L. Dancy
The Pennsylvania State University University Park, Pennsylvania, USA