Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR

Nguyen Nguyen; Yongyi Zang

cs.CLOct 14, 2026

Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR

Authors: Nguyen Nguyen, Yongyi Zang

Organizations: University of Rochester · Independent

Abstract

Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER, which keeps CER's alignment and denominator and charges each grapheme edit a severity from an LLM judge, with no word segmentation. Normalized CER keeps the score of a whole transcription, where it tracks readers as well as semCER or better, and semCER breaks that score down into named edits. The charges track readers' ratings edit by edit in English, Vietnamese, and Chinese and put the edits readers rate most severe near the top of a sorted list. On pairs drawn where CER and semCER disagreed, readers side with semCER more often than with normalized CER in English and Chinese, and equally often in Vietnamese. Across 16 languages, the judge charges a wrong diacritic or tone mark almost nothing in Spanish but nearly a wrong letter's severity in Telugu. Where we can count real words, a general VLM's wrong letter forms one far above chance while a classical pipeline's stays near it. We release the 61 million judged edits, the aggregated reader ratings, and the distilled scorer at https://github.com/nguyenx2t/semcer.

Figures & tables

Figure 1: Equal edit counts can hide errors a reader cannot repair. Six single-edit errors are sorted by semCER ’s charge; CER charges each one equally. The highlighted rows share one substitution: đấng (“being”) becomes the recoverable đáng (“worthy”), whereas Tấm, the heroine of Tấm Cám , becomes Tám, another Vietnamese name.

Figure 2: The metric charges one transcription line edit by edit. A general VLM transcribes an OmniDocBench line, shown as the crop above and the aligned reference and output below. Each edit is shaded, $+$ marks an insertion, $-$ a deletion, and a dash an empty side. Six edits give CER $0.171$ over $35$ graphemes, while their severities sum to $1.90$ and give semCER $0.054$ .

Level	Rating	Description
$0$	$0$	Cosmetic, fully recoverable
$1$	$0.25$	Non-word or obvious error, context recovers
$2$	$0.5$	Valid alternative, reader may notice
$3$	$0.75$	Recovery fails, but error is detectable
$4$	$1$	Reads perfectly, is false, no signal

Table 1: The rubric’s five levels. The judge returns a level, and its rating

\rho_{k}=k/4

is the scale of the severity and of the readers’ ratings.

Figure 3: Sorted by severity, the edits readers rate most severe come first. Each panel gives the share of all reader-rated damage caught after checking a fraction of the single-edit pool in each order, over the first half of the list. The readers’ own order is the ceiling, a random order the diagonal, and the rule marks the

10\%

cut.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Datasets	Lines
Handwriting	IAM ( Marti and Bunke, 2002 ) , RIMES ( Grosicki and El-Abed, 2011 ) , German Handwriting ( Fachhochschule Südwestfalen, 2024 ) , CASIA-HWDB ( Liu et al., 2011 ) , KHATT ( Mahmoud et al., 2014 ) , UNHD ( Ahmed et al., 2019 ) , VNOnDB ( Nguyen et al., 2018 ) , Thai handwriting ( Pongsasit, 2025 ) , BRESSAY ( Neto et al., 2024 ) , School Notebooks ( ai-forever, 2022 ) , ICDAR 2025 IHDR ( IIIT Hyderabad, Centre for Visual Information Technology, 2025 )	104,897
Scene text	HierText ( Long et al., 2022 ) , LSVT ( Sun et al., 2019 ) , RCTW-17 ( Shi et al., 2017 ) , COCO-Text ( Veit et al., 2016 ) , TextOCR ( Singh et al., 2021 ) , CTW ( Yuan et al., 2019 ) , ReCTS ( Zhang et al., 2019 ) , VinText ( Nguyen et al., 2021 ) , MLT-2019 ( Nayef et al., 2019 ) , ThaiOCRBench ( Nonesung et al., 2025 )	11,380
Printed document	OmniDocBench ( Ouyang et al., 2025 ) , VieBookRead ( Do et al., 2025 ) , RETAS ( Yalniz and Manmatha, 2011 ) , Mozhi ( Mathew et al., 2024 )	113,118
Total	25 datasets, 16 languages	229,395

Appendix

Table 3: The evaluation corpus contains 229,395 transcription lines after filtering.

System	Params	Reference
Classical OCR
EasyOCR	–	( JaidedAI, 2020 )
PP-OCRv5	–	( Cui et al., 2025 )
Document-specialized VLM
Qianfan-OCR	4B	( Dong et al., 2026 )
General VLMs

Appendix

Table 4: The evaluation includes classical pipelines, a document-specialized VLM, and general VLMs. 26B-A4B gives the total and active parameters of a mixture-of-experts model.

Study	Lang	Items	Readers	$\alpha$	ICC(2, $k$ )
single-edit	English	87	5	0.71	0.89
	Vietnamese	80	4	0.67	0.87
	Chinese	86	5	0.53	0.81
pair-level	English	178	4	0.79	0.92
	Vietnamese	173	4	0.65	0.88
	Chinese	185	3	0.68	0.84

Appendix

Table 5: The annotation studies reach moderate to high reliability.

\alpha

is Krippendorff’s alpha ( Krippendorff, 2004 ) with ordinal weighting, and ICC(2,

k

) ( Shrout and Fleiss, 1979 ) is the reliability of the readers’ mean, both over readers who rated at least half of a pool’s items.

Partial correlation within band
	English	Vietnamese	Chinese
severity	0.30 ∗	0.33 ∗	0.52 ∗
mode	0.44 ∗	0.23 ∗	0.29 ∗
severity, given the mode	0.13	0.24 ∗	0.45 ∗
Correlation on the whole pool
severity	0.81	0.66	0.82

Appendix

Table 6: Per-edit results on the single-edit pools. The first two blocks are Spearman correlations with readers’ ratings, the mode being the judge’s most likely level, and the last two the top-of-list measures of Section B.1 . An asterisk marks a within-band partial whose interval excludes zero, and a dagger a baseline that severity leads significantly.

Score	English	Vietnamese	Chinese
unnormalized CER	$0.777^{*}$	$0.775^{*}$	$0.791$
whitespace stripped	$0.857$	$0.807$	$0.796$
character filter	$0.894$	$0.822$	$0.756$
normalized CER	$0.925^{*}$	$0.902$	$0.757$
case-folded, punct. kept	$0.838^{*}$	$0.862$	$0.794$
semCER	$0.890$	$0.873$	$0.763$

Appendix

Table 7: Normalization drives agreement with readers over whole transcriptions. Each entry is Spearman’s

\rho

with readers on the pair-level pool. The first four conventions are cumulative and apply before alignment. An asterisk marks a difference from semCER that is significant after Holm correction ( Section A.4 ). The class charge was added after that test, so its dagger marks an uncorrected difference from semCER whose interval excludes zero.

Figure 7: Every margin of the preference test. Each marker is semCER ’s agreement with the readers’ majority minus the comparison’s, with its interval, a tie counting as a miss. Filled markers are significant. The last panel holds the near-miss pairs that are not normalization-proof, and the last three rows score the pairs with the judge’s earlier severities, the distilled scorer, and the second judge.

Agreement with readers on near-miss pairs
	English	Vietnamese	Chinese
Pool, strict majority	173	163	146
Near-miss pairs	70	78	68
Normalization-proof pairs	22	30	30
semCER	0.87 ∗	0.76	0.85 ∗
normalized CER	0.64	0.73	0.59

Appendix

Table 8: Counts behind the preference test. An asterisk marks a significant margin over normalized CER , and the last row is a single reader’s agreement with the strict majority of the other readers.

Single edits
	English	Vietnamese	Chinese
correlation with readers	0.80	0.65	0.83
partial within band	0.37 ∗	0.35 ∗	0.57 ∗
same, given the filter label	0.35 ∗	0.20	0.51 ∗
correlation with the first judge	0.81	0.74	0.83
Whole transcriptions

Appendix

Table 9: The second judge reproduces the per-edit, whole-transcription, and normalization-proof results. Each entry is Qwen3.7-Flash’s value on the rated pools. An override is a pair with a strict reader majority where it picks the output CER rejects. The margin is its agreement with the readers’ majority minus normalized CER ’s, a tie counting as a miss. An asterisk marks an interval that excludes zero, Holm-corrected in the margin row.

VLM	Chinese	Vietnamese	German	French	English	Spanish
Gemma 4	$+0.093^{*}$ (456)	$+0.045$ (108)	$-0.010$ (479)	$-0.029^{*}$ (492)	$-0.011^{*}$ (1,431)	$-0.031^{*}$ (626)
InternVL3.5	$+0.028$ (343)	$+0.162^{*}$ (104)	$+0.002$ (260)	$+0.003$ (321)	$-0.010$ (1,242)	$+0.004$ (533)
Qwen3.5	$+0.051$ (308)	$+0.038$ (84)	$-0.006$ (473)	$-0.021^{*}$ (500)	$-0.011^{*}$ (1,395)	$-0.030$ (624)
Qwen3.6	$+0.012$ (297)	$+0.046$ (77)	$-0.016$ (506)	$-0.037^{*}$ (546)	$-0.010$ (1,309)	$-0.026$ (629)

Appendix

Table 10: Equal-count contrasts, each VLM against EasyOCR. Each entry is the mean severity per edit, VLM minus EasyOCR, on lines where both make the same number of edits, one to five, with paired lines in parentheses. An asterisk marks a contrast that survives Holm correction over the

24

Language	mark-only	content sub.	per edit
Chinese †	–	0.80	0.50
Telugu	0.75	0.81	0.74
Kannada	0.70	0.80	0.70
Tamil	0.66	0.77	0.68
Thai	0.64	0.81	0.66
Marathi	0.56	0.68	0.62

Appendix

Table 11: Severity per edit follows the grapheme load. The judge’s mean severity of a mark-only edit (the grapheme load), a content substitution, and all edits, over lines with one to five edits. Chinese † has no marks, so its load is the content-substitution cost.

Language	Domain	general VLM	classical
English	printed	$0.14$ / $0.05$ ∗	$0.18$ / $0.22$ ∗
English	handwriting	$0.57$ / $0.14$ ∗	–
English	scene text	$0.22$ / $0.12$ ∗	$0.24$ / $0.20$ ∗
German	printed	$0.30$ / $0.03$ ∗	$0.04$ / $0.05$
German	handwriting	$0.45$ / $0.04$ ∗	$0.05$ / $0.05$
French	printed	$0.30$ / $0.04$ ∗	$0.07$ / $0.09$

Appendix

Table 12: General VLM letter errors form real words far above chance, while classical errors stay near it. Each entry is the share of letter errors that form another real word, observed / chance. An asterisk marks a difference from chance whose interval excludes zero, and a dash fewer than

30

letter errors.

Correlation with readers, single edits
	English	Vietnamese	Chinese	All $16$
student	0.79	0.63	0.80
judge	0.81	0.66	0.82
Clear pairs ordered as the judge does (%)
held-out test lines	90	91	93	90
rated-pool lines	90	92	90	91

Appendix

Table 13: The distilled scorer tracks readers and the judge. The test lines are

1{,}623

sampled from the test split, up to

120

per language, each transcribed by one randomly chosen system. The pooled share is

90.3\%

[

89.5

91.1

], and no language falls below

87\%

Data	$192{,}897$ training, $11{,}344$ validation,
	and $22{,}683$ test lines
Backbone	Qwen3.5-0.8B-Base ( Qwen Team, 2026a ) ,
	all weights trained
Input	marked pair, up to $2{,}048$ tokens
Readout	last token and the pooled
	reference and output spans

Appendix

Table 14: Training setup of the distilled scorer.

Explore similar work

The Character Error Vector: Decomposable errors for page-level OCR evaluation
Apr 7, 2026Jonathan Bourne, Mwiza Simbeye, Joseph NockelsAutomated Evaluation Optical Character Recognition
Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and a Vision-Language Model
Feb 16, 2026Ari Vesalainen, Eetu Mäkelä, Laura Ruotsalainen +1Optical Character Recognition VLM Evaluation
University of Helsinki, Department of Computer Science, Finland · University of Helsinki, Department of Digital Humanities, Finland
A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition
May 5, 2026Thibault Bañeras-Roux, Mickael Rouvier, Jane Wottawa +1ASR Evaluation Automatic Speech Recognition
Nantes University, LS2N, France

Apr 7, 2026J/K move · Enter open · S save

The Character Error Vector: Decomposable errors for page-level OCR evaluation

Jonathan Bourne, Mwiza Simbeye, Joseph Nockels

Automated Evaluation Optical Character Recognition Document Image Understanding

Open paper AskPDF arXiv

The Character Error Rate (CER) is a key metric for evaluating the quality of Optical Character Recognition (OCR). However, this metric assumes that text has been perfectly parsed, which is often not the case. Under page-parsing errors, CER becomes undefined, limiting its use as a metric and making evaluating page-level OCR challenging, particularly when using data that do not share a labelling schema. We introduce the Character Error Vector (CEV), a bag-of-characters evaluator for OCR. The CEV can be decomposed into parsing and OCR, and interaction error components. This decomposability allows practitioners to focus on the part of the Document Understanding pipeline that will have the greatest impact on overall text extraction quality. The CEV can be implemented using a variety of methods, of which we demonstrate SpACER (Spatially Aware Character Error Rate) and a Character distribution method using the Jensen-Shannon Distance. We validate the CEV's performance against other metrics: first, the relationship with CER; then, parse quality; and finally, as a direct measure of page-level OCR quality. The validation process shows that the CEV is a valuable bridge between parsing metrics and local metrics like CER. We analyse a dataset of archival newspapers made of degraded images with complex layouts and find that state-of-the-art end-to-end models are outperformed by more traditional pipeline approaches. Whilst the CEV requires character-level positioning for optimal triage, thresholding on easily available values can predict the main error source with an F1 of 0.91. We provide the CEV as part of a Python library to support Document understanding research.

Abstract

Explore similar work

The Character Error Vector: Decomposable errors for page-level OCR evaluation

Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and a Vision-Language Model

A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition