Word-Level Alignment of OCR and Ground Truth under Zoning and Reading-Order Errors
Organizations: Department of Computer Science, University of Nevada, Las Vegas, Las Vegas, NV, USA.
Abstract
OCR post-processing needs OCR text aligned word by word with its ground truth. Data sets rarely provide that mapping, and a monotone aligner can lose its place when zoning errors permute the reading order. We revisit this problem, the Zoning Problem of our 2019 paper, with a protocol that tests correctness under such permutations. We apply OCR-side reading-order permutations with construction-time word-level gold and score correct-match F1, cascade length and recovery. We compare the 2019 location-aware aligner, a re-anchoring extension of it, a RETAS-style baseline (RETAS-lite), a multi-region aligner that combines unique-word and unique n-gram anchors with chunk-to-window assignment, genalog's RETAS and the original Java RETAS. Experiments use synthetic character noise with block moves, paragraph swaps and column interleaving, historical newspaper reading-order permutations, geometry-derived gold for Tesseract OCR, and a small word-box set. Synthetic tests show where local tracking fails after large displacements and where unique-anchor methods collapse when unique anchors are scarce. On our synthetic boilerplate page the 2019 method had the highest F1 among those tested. Historical newspaper and geometry-derived evaluations carry the limitations of approximate gold and the article-order convention. The first multi-region version has a quadratic scaling cost. Local tracking remains fragile under large reading-order changes; the tested anchor-based methods handle many of those cases but fail when unique anchors are scarce. Approximate gold, the article-order convention, missing residual annotation, and the lack of modern multi-column word-level ground truth remain limitations.
Figures & tables
| Aligner | Block move | Paragraph swap | Column interleave |
|---|---|---|---|
| Location-aware (2019) | 0.890 [0.865, 0.913] | 0.012 [0.003, 0.029] | 0.255 [0.242, 0.265] |
| + re-anchoring | 0.922 [0.907, 0.935] | 0.339 [0.338, 0.340] | 0.353 [0.348, 0.358] |
| RETAS-lite | 0.998 [0.996, 0.999] | 0.977 [0.975, 0.977] | 0.907 [0.904, 0.909] |
| genalog (RETAS) | 0.920 [0.920, 0.921] | 0.181 [0.164, 0.215] | 0.500 [0.499, 0.501] |
| Multi-region v1 | 1.000 [0.999, 1.000] | 1.000 [1.000, 1.000] | 0.983 [0.981, 0.985] |
| Block move | Paragraph swap | Column interleave | ||||
|---|---|---|---|---|---|---|
| Aligner | ||||||
| Location-aware (2019) | 25.3 | 0.33 | 129.0 | 0.17 | 48.8 | 0.23 |
| + re-anchoring | 18.2 | 0.53 | 66.5 | 0.17 | 23.6 | 0.33 |
| RETAS-lite | 0.7 | 1.00 | 3.3 | 0.98 | 1.3 | 1.00 |
| genalog (RETAS) | 13.7 | 0.67 | 207.0 | 0.07 | 6.3 | 0.51 |
| Multi-region v1 | 0.0 | 1.00 | 0.0 | 1.00 | 0.1 | 1.00 |
| Aligner | F1 | Runtime (s) | |
|---|---|---|---|
| Location-aware (2019) | 0.573 [0.435, 0.707] | 0.550 | 0.008 |
| + re-anchoring | 0.578 [0.441, 0.712] | 0.552 | 0.031 |
| RETAS-lite | 0.986 [0.972, 0.995] | 0.987 | 0.005 |
| Multi-region v1 | 0.987 [0.969, 0.999] | 0.995 | 0.009 |
| genalog (RETAS) | 0.668 [0.549, 0.792] | 0.643 | 2.544 |
| Java RETAS | 0.590 [0.557, 0.623] | 0.870 | 0.102 |
| Aligner | F1 (all gold) | F1 (high-conf. gold) |
|---|---|---|
| Multi-region v1 | 0.432 [0.404, 0.460] | 0.650 [0.623, 0.675] |
| Multi-region v2 | 0.403 [0.375, 0.432] | 0.607 [0.579, 0.634] |
| RETAS-lite | 0.354 [0.326, 0.381] | 0.504 [0.477, 0.530] |
| genalog (RETAS) | 0.254 [0.230, 0.277] | 0.371 [0.342, 0.399] |
| Java RETAS | 0.176 [0.164, 0.187] | 0.273 [0.260, 0.284] |
| + re-anchoring | 0.042 [0.034, 0.051] | 0.059 [0.050, 0.071] |
| Aligner | F1 (all gold) | F1 (high-conf. gold) |
|---|---|---|
| Multi-region v1 | 0.742 [0.687, 0.795] | 0.948 [0.918, 0.975] |
| Multi-region v2 | 0.735 [0.677, 0.791] | 0.947 [0.917, 0.973] |
| RETAS-lite | 0.698 [0.629, 0.765] | 0.910 [0.869, 0.949] |
| Location-aware (2019) | 0.483 [0.373, 0.592] | 0.694 [0.547, 0.835] |
| genalog (RETAS) | 0.492 [0.341, 0.628] | 0.647 [0.437, 0.833] |
| Java RETAS | 0.378 [0.323, 0.436] | 0.663 [0.609, 0.712] |
| Condition | MR v1 | RETAS-lite | Java RETAS | 2019 | (MR v1 lite) | |
|---|---|---|---|---|---|---|
| Noise 0.10 | 0.972 | 0.893 | 0.476 | 0.264 | +0.079 [+0.076, +0.083] | 0.001 |
| Noise 0.15 | 0.973 | 0.889 | 0.457 | 0.243 | +0.084 [+0.078, +0.091] | 0.001 |
| Noise 0.20 | 0.957 | 0.873 | 0.436 | 0.228 | +0.084 [+0.077, +0.094] | 0.001 |
| Noise 0.25 | 0.962 | 0.870 | 0.416 | 0.268 | +0.093 [+0.086, +0.099] | 0.001 |
| 8 regions | 0.998 | 0.981 | 0.432 | 0.141 | +0.017 [+0.014, +0.021] | 0.001 |
| 15 regions | 0.995 | 0.969 | 0.385 | 0.101 | +0.027 [+0.022, +0.031] | 0.001 |
| Multi-region v1 | Multi-region v2 | RETAS-lite | ||||
|---|---|---|---|---|---|---|
| Tokens (seeds) | F1 | s | F1 | s | F1 | s |
| 1,000 (5) | 0.771 | 0.021 | 0.768 | 0.015 | 0.504 | 0.003 |
| 5,000 (5) | 0.423 | 0.588 | 0.418 | 0.219 | 0.294 | 0.011 |
| 10,000 (5) | 0.334 | 2.533 | 0.326 | 0.036 | 0.326 | 0.036 |
| 25,000 (3) | 0.284 | 14.7 | 0.343 | 0.153 | 0.343 | 0.148 |
| 50,000 (3) | 0.256 | 61.5 | 0.341 | 0.511 | 0.341 | 0.507 |
| Evaluation | v1 F1 | v2 F1 | (v2 v1) |
|---|---|---|---|
| IMPACT test-200, all gold | 0.432 [0.404, 0.460] | 0.403 [0.375, 0.432] | 0.029 [ 0.032, 0.026] |
| IMPACT test-200, high-conf. | 0.650 [0.623, 0.675] | 0.607 [0.579, 0.634] | 0.043 [ 0.048, 0.038] |
| Word-box gold, all | 0.742 [0.687, 0.795] | 0.735 [0.677, 0.791] | 0.007 [ 0.012, 0.002] |
| TREC synthetic (Step 8) | 0.945 [0.935, 0.955] | 0.942 [0.933, 0.952] | 0.002 [ 0.004, 0.001] |
| Boilerplate stress | 0.125 [0.118, 0.134] | 0.159 [0.151, 0.168] | +0.034 [+0.029, +0.041] |
| Long document, 50k tokens a | 0.256 (0.239 to 0.287) | 0.341 (0.320 to 0.363) | +0.085 (+0.077 to +0.096) |
| degrade5 ( 5% CER) | degrade20 ( 20% CER) | |||
|---|---|---|---|---|
| Aligner | Agreement F1 | Matched | Agreement F1 | Matched |
| Location-aware (2019) | 0.888 [0.827, 0.933] | 0.957 | 0.102 [0.076, 0.131] | 0.740 |
| + re-anchoring | 0.893 [0.834, 0.936] | 0.959 | 0.107 [0.081, 0.137] | 0.743 |
| RETAS-lite | 0.934 [0.921, 0.945] | 0.912 | 0.378 [0.354, 0.404] | 0.375 |
| Multi-region v1 | 0.944 [0.934, 0.954] | 0.933 | 0.541 [0.521, 0.561] | 0.671 |
| Java RETAS | 0.450 [0.438, 0.463] | 0.311 | 0.179 [0.170, 0.188] | 0.145 |