Evaluating Cloud and Self-Hosted OCR Systems for Arabic, English, and French Documents: Bounding Box Detection, Text Recognition, and Mathematical Expression Recognition
Organizations: Naseej Innovation Lab, Naseej Technical Company, Riyadh, Saudi Arabia
Abstract
Choosing an OCR system for a multilingual archive requires balancing accuracy against deployment constraints. This study compares cloud and self-hosted systems on three tasks: bounding box detection on 48 Arabic, English, and French pages, text recognition on 315 pages across nine document types, and mathematical expression recognition on 83 English and Arabic samples. Detection was assessed by page-level visual grades, text by character error rate (CER) against a Gemini-generated reference transcription, and mathematics by scores against known answers. Main estimates are accompanied by 95% intervals and paired comparisons. Google Document AI and Surya produced Good line boxes on 47 of 48 (98%) and 46 of 48 (96%) pages. Datalab, Azure Document Intelligence, and Document AI had mean CERs of 0.145-0.152, with no difference detected in unadjusted paired comparisons. Surya had the lowest mean CER among self-hosted systems (0.199). No mathematical test had a statistically clear leader. Strict checks showed that near-perfect page scores could conceal a replaced Arabic variable or Eastern Arabic digits rewritten in Western form. Reliable evaluation therefore requires uncertainty estimates, structural checks, and notation-specific tests in addition to aggregate recognition scores.
Figures & tables
| System | Mean CER | 95% CI | Median | Capped |
|---|---|---|---|---|
| Datalab | 0.145 | [0.120, 0.172] | 0.038 | 0.145 |
| Azure DI | 0.145 | [0.122, 0.171] | 0.051 | 0.143 |
| Document AI | 0.152 | [0.128, 0.176] | 0.051 | 0.152 |
| Surya | 0.199 | [0.158, 0.251] | 0.038 | 0.177 |
| olmOCR 2 | 0.241 | [0.208, 0.276] | 0.096 | 0.235 |
| Tesseract | 0.333 | [0.297, 0.369] | 0.250 | 0.332 |
| Datalab | Azure DI | Document AI | Surya | olmOCR 2 | Tesseract | Qwen3-VL | dots.ocr | KATIB | PaddleOCR-VL | |
|---|---|---|---|---|---|---|---|---|---|---|
| AR-PRINT (119) | 0.129 | 0.142 | 0.139 | 0.203 | 0.225 | 0.368 | 0.363 | 0.602 | 0.373 | 0.326 |
| AR-HAND (10) | 0.170 | 0.171 | 0.177 | 0.251 | 0.373 | 0.759 | 0.504 | 0.483 | 0.307 | 0.444 |
| AR-MANU (10) | 0.225 | 0.342 | 0.339 | 0.381 | 0.738 | 0.874 | 1.217 | 0.831 | 0.658 | 0.775 |
| AR-CALLI (13) | 0.399 | 0.610 | 0.544 | 0.639 | 0.743 | 0.944 | 2.459 | 0.909 | 0.719 | 0.842 |
| AR-TABLE (22) | 0.327 | 0.173 | 0.410 | 0.374 | 0.585 | 0.322 | 0.576 | 0.434 | 0.631 | 0.488 |
| EN-PRINT (26) | 0.067 | 0.049 | 0.035 | 0.057 | 0.013 | 0.042 | 0.018 | 0.035 | 0.201 | 0.084 |
| Gemini | Datalab | PaddleOCR-VL | dots.ocr | olmOCR 2 | Surya | Qwen3-VL | GOT-OCR 2.0 | pix2tex | Azure DI | KATIB | Document AI | Tesseract | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Formulas | 0.915 | 0.911 | 0.849 | 0.806 | 0.799 | 0.795 | 0.769 | 0.611 | 0.605 | 0.440 | 0.438 | 0.331 | 0.180 |
| Arabic pages | 0.996 | 0.998 | 0.252 | 0.995 | 0.990 | 0.997 | 0.823 | 0.088 | 0.020 | 0.489 | 0.737 | 0.654 | 0.365 |
| Arabic notation | 0.358 | 0.553 | 0.055 | 0.161 | 0.832 | 0.351 | 0.743 | 0.011 | 0.000 | 0.975 | 0.735 | 0.877 | 0.672 |
| Real EN pages | 0.998 | 0.994 | 0.907 | 0.999 | 1.000 | 0.999 | 0.746 | 0.862 | 0.087 | 0.836 | 0.858 | 0.604 | 0.431 |
| Real AR page | 0.993 | 0.979 | 0.601 | 0.986 | 0.993 | 0.993 | 0.685 | 0.392 | 0.140 | 0.510 | 0.643 | 0.448 | 0.371 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Optical Character Recognition
Optical character recognition converts text in images into machine-readable text, from printed pages and handwriting to signs and symbols.
See all 195 papers