The Character Error Vector: Decomposable errors for page-level OCR evaluation

Jonathan Bourne; Mwiza Simbeye; Joseph Nockels

cs.CVApr 7, 2026

The Character Error Vector: Decomposable errors for page-level OCR evaluation

Authors: Jonathan Bourne, Mwiza Simbeye, Joseph Nockels

Abstract

The Character Error Rate (CER) is a key metric for evaluating the quality of Optical Character Recognition (OCR). However, this metric assumes that text has been perfectly parsed, which is often not the case. Under page-parsing errors, CER becomes undefined, limiting its use as a metric and making evaluating page-level OCR challenging, particularly when using data that do not share a labelling schema. We introduce the Character Error Vector (CEV), a bag-of-characters evaluator for OCR. The CEV can be decomposed into parsing and OCR, and interaction error components. This decomposability allows practitioners to focus on the part of the Document Understanding pipeline that will have the greatest impact on overall text extraction quality. The CEV can be implemented using a variety of methods, of which we demonstrate SpACER (Spatially Aware Character Error Rate) and a Character distribution method using the Jensen-Shannon Distance. We validate the CEV's performance against other metrics: first, the relationship with CER; then, parse quality; and finally, as a direct measure of page-level OCR quality. The validation process shows that the CEV is a valuable bridge between parsing metrics and local metrics like CER. We analyse a dataset of archival newspapers made of degraded images with complex layouts and find that state-of-the-art end-to-end models are outperformed by more traditional pipeline approaches. Whilst the CEV requires character-level positioning for optimal triage, thresholding on easily available values can predict the main error source with an F1 of 0.91. We provide the CEV as part of a Python library to support Document understanding research.

Figures & tables

Component	Difference	Comparison	Description
Parsing	$d_{\text{pars}}$	$d(R\parallel Q)$	Errors caused by page regions being missed or captured more than once due to overlapping detections.
OCR	$d_{\text{ocr}}$	$d(S^{*}\parallel Q)$	Transcription errors resulting in deviation from the true character distribution, isolated on GT regions.
Interaction	$d_{\text{int}}$	$d(S\parallel R)$	OCR errors which are model dependent but on the parsing of the document image.
Total	$d_{\text{total}}$	$d(S\parallel Q)$	Combined character error encompassing both parsing and OCR errors.

Table 1: CDD decomposition into parsing and OCR error components when character position information is available.

Symbol	Definition	Error captured
$Q$	GT character vector	No Error
$R$	Vector from the predicted parsing on GT characters	Parsing error only
$S^{*}$	Vector created by performing OCR on GT regions	OCR error only
$S$	Vector created by performing OCR on predicted regions	Combined OCR and Parsing

Table 2: Character vectors used in the CDD decomposition.

Panel of Figure 1 — Figure 1: CDD decomposition when character position information is available. Solid arrows denote component differences; the dashed arrow denotes the total end-to-end divergence.

Property	Value
Images	49
Lines	19,691
Words	175,548
Granularity	Word level
Region Type	Polygon

Table 3: Details on The Spiritualist dataset used in this paper

Model	Type	Parameters	Framework
PaddleOCR	CNN	23.9M	PaddlePaddle
TrOCR	Transformer	334M	PyTorch
EasyOCR	CNN	13.4M	PyTorch
Tesseract	Traditional	—	C++

Table 4: Comparison of OCR Models

Model	Type	Parameters	Framework
DocLayout-YOLO	CNN	15.4M	PyTorch
Heron	Transformer	42.9M	PyTorch
PP-DocLayout-L	Transformer	30.94M	PaddlePaddle
PP-DocLayout-M	Transformer	5.65M	PaddlePaddle
PP-DocLayout-S	Transformer	1.21M	PaddlePaddle

Table 5: Comparison of Document Layout Analysis Models

Model	Type	Parameters	Framework
olmOCR	Transformer	7B	PyTorch
dots.MOCR	Transformer	7B	PyTorch
Granite-Docling	Transformer	258M	PyTorch

Table 6: Comparison of End-to-End OCR Models

Font	Layout	Columns
Ewert, Great Vibes, Kablammo, Roboto, UnifrakturMaguntia	Centered, Justified, Left aligned	1, 2, 3

Table 7: Parameters used in text generation process

Figure 2: Analysis of the impact of spatial uncertainty on the character position of the ground truth. We see that increasing the prediction width increases performance, but that paragraph-level inference is unusable at any scale.

	SpACER $\rho$	CDD $\rho$
Easyocr	0.657	0.542
PaddleOCR	0.945	0.680
Tesseract	0.910	0.704
TrOCR	0.963	0.267

Table 8: Spearman correlation (

\rho

) between CER and

d_{\text{ocr}}

for SpACER and CDD, computed over GT regions.

	SpACER $\rho$	CDD $\rho$
Heron	-0.577	-0.742
PPDoc-L	-0.742	-0.809
PPDoc-M	-0.275	0.285
PPDoc-S	0.181	0.250
YOLO	-0.522	-0.649

Table 9: Spearman correlation (

\rho

) between per-page

d_{\text{pars}}

and COTe score by parsing model. Negative

\rho

indicates that higher COTe (better parsing geometry) corresponds to lower parsing error, as expected.

Figure 3: We can see that SpACER has good correlation with CER at page level, indicating that they are both measuring the same thing. The CDD has low correlation, as it measures change in shape of the distribution not the error rate.

	Easyocr	PaddleOCR	Tesseract	TrOCR
Heron	0.031	0.009	0.009	0.036
PPDoc-L	0.049	0.024	0.029	0.053
PPDoc-M	0.033	0.006 ∗	0.126	0.160
PPDoc-S	0.033	0.006	0.114	0.159
YOLO	0.028	0.015	0.016	0.037

Table 10: Page-level SpACER error (

d_{\text{total}}

) by parsing model (rows) and OCR model (columns). Bold : column best; bold ∗ : overall best.

Figure 4: PPDoc-m and PPDoc-S, fail to correctly parse the images resulting in multiple columns in each crop, such as this example from image 1. All the OCR models fail to produce sensible text, but the failure mode depends on model architecture.

	SpACER	CDD	COTe	[email protected]
Heron	0.013	0.008	0.843	0.074
PPDoc-L	0.025	0.011	0.749	0.084
PPDoc-M	0.001	0.003	0.126	0.000
PPDoc-S	0.001	0.000	0.264	0.000
YOLO	0.015	0.009	0.862	0.099

Table 11: Parsing error (

d_{\text{pars}}

) by parsing model with COTe and [email protected]. SpACER macro is the primary metric; lower is better for SpACER/CDD, higher for COTe/mAP.

	SpACER	CDD	CER
Easyocr	0.047	0.087	0.121
PaddleOCR	0.020	0.020	0.028
Tesseract	0.024	0.056	0.050
TrOCR	0.053	0.031	0.092

Table 12: OCR error (

d_{\text{ocr}}

) by OCR model, averaged over pages using GT regions. SpACER macro is the primary metric; lower is better.

	Easyocr	PaddleOCR	Tesseract	TrOCR
Heron	0.032	0.012	0.012	0.035
PPDoc-L	0.029	0.010	0.012	0.038
PPDoc-M	0.032	0.006	0.103	0.158
PPDoc-S	0.033	0.006 ∗	0.115	0.158
YOLO	0.027	0.014	0.013	0.028

Table 13: Interaction error (

d_{\text{int}}

, SpACER) by parsing model (rows) and OCR model (columns). Bold : column best; bold ∗ : overall best.

Model	SpACER	Output Text
EasyOCR	0.029	thc SPIRIT INFLUENCE. inches of Mr. Saul’$ and Mr. Hurt’s hands, and table, but in difierent parts of the room the table was ON one occasiol;
PaddleOCR	0.007	THE HANDLING OF inches of Mr. Saul’s and Mr. Hurt’s hands, and they SPIRIT INFLUENCE. table, but in different parts of the room; the table was
Tesseract	0.138	SPIRIT INFLUEN oe UNDER [bumed himself, Home then held it withi Month Sieone Tiiestiones. MeL

Table 14: Comparison of OCR model outputs from image 1 of the dataset when using PPDoc-S parsing regions.

Model	SpACER	CDD
Granite	0.864	0.207
M.OCR	0.097	0.057
OlmOCR	0.106	0.044

Table 15: Median SpACER and CDD by end-to-end model (d_total)

Figure 5: The figure shows that even conservative cutoffs for SpACER and COTe can identify if parsing or OCR quality is the main source of error in the pipeline

Explore similar work

Oct 14, 2026

Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR

Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER, which keeps CER's alignment and denominator and charges each grapheme edit a severity from an LLM judge, with no word segmentation. Normalized CER keeps the score of a whole transcription, where it tracks readers as well as semCER or better, and semCER breaks that score down into named edits. The charges track readers' ratings edit by edit in English, Vietnamese, and Chinese and put the edits readers rate most severe near the top of a sorted list. On pairs drawn where CER and semCER disagreed, readers side with semCER more often than with normalized CER in English and Chinese, and equally often in Vietnamese. Across 16 languages, the judge charges a wrong diacritic or tone mark almost nothing in Spanish but nearly a wrong letter's severity in Telugu. Where we can count real words, a general VLM's wrong letter forms one far above chance while a classical pipeline's stays near it. We release the 61 million judged edits, the aggregated reader ratings, and the distilled scorer at https://github.com/nguyenx2t/semcer.

Nguyen Nguyen, Yongyi Zang

University of Rochester · Independent

Optical Character Recognition

Ask PDF

Feb 16, 2026

Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and a Vision-Language Model

Optical Character Recognition (OCR) of eighteenth-century printed texts remains challenging due to degraded print quality, archaic glyphs, and non-standardized orthography. Although transformer-based OCR systems and Vision-Language Models (VLMs) achieve strong aggregate accuracy, metrics such as Character Error Rate (CER) and Word Error Rate (WER) provide limited insight into their reliability for scholarly use. We compare a dedicated OCR transformer (TrOCR) and a general-purpose Vision-Language Model (Qwen) on line-level historical English texts using length-weighted accuracy metrics and hypothesis driven error analysis. While Qwen achieves lower CER/WER and greater robustness to degraded input, it exhibits selective linguistic regularization and orthographic normalization that may silently alter historically meaningful forms. TrOCR preserves orthographic fidelity more consistently but is more prone to cascading error propagation. Our findings show that architectural inductive biases shape OCR error structure in systematic ways. Models with similar aggregate accuracy can differ substantially in error locality, detectability, and downstream scholarly risk, underscoring the need for architecture-aware evaluation in historical digitization workflows.

Ari Vesalainen, Eetu Mäkelä, Laura Ruotsalainen +1

University of Helsinki, Department of Computer Science, Finland · University of Helsinki, Department of Digital Humanities, Finland

Optical Character Recognition VLM Evaluation

Ask PDF

Oct 8, 2026

SP-DocReader: Difference-Aware Self-Play for Precise Document OCR

Accurate page transcription remains difficult for vision language models under limited input and training budgets. We present SP-DocReader, a self-play framework for optical character recognition (OCR) that targets residual errors after supervised fine-tuning. Reading Discrepancy Masking aligns reference and generated model tokens through a longest common subsequence, then scores unmatched positions with their full conditioning prefixes. Focused Fidelity Loss adds direct negative log-likelihood supervision at unmatched ground-truth positions. Only the OCR module is trained, while the backbone remains frozen. We derive the combined gradient to distinguish relative score optimization from direct supervision. Compared with SFT-2, SP-DR-3 reduces Vary-600K character error rate on both backbones. On Qwen3-VL-4B, it reduces character error rate by approximately 54 percent and improves DocVQA Average Normalized Levenshtein Similarity (ANLS) by 3.7 points. These results show the value of focusing self-play training on the discrepancies that remain after supervised fine-tuning.

Wenjie Liao, Xiaohui Song, Liangjie Zhao +1

Guangdong OPPO Mobile Telecommunications Corp.,Ltd · Adelaide University

Optical Character Recognition Vision-Language Models Fine-Tuning

Ask PDF