Handwritten Urdu is hard to recognize because the cursive Nastaliq script has no fixed baseline and no clear character boundaries. We benchmark six systems on the Urdu Nastaliq Handwritten Dataset (UNHD): Tesseract (default configuration), a script-specific recognizer trained on printed Urdu (UTRNet-Large), two open-source vision-language models (VLMs), and two commercial VLMs. All six systems are evaluated without fine-tuning on 2,465 single-line images (Part A), and the four VLMs receive the same fixed prompt (decoding uses temperature 0 where configurable, except for Claude Sonnet 5, whose decoding cannot be configured). Both commercial VLMs reach lower error than the printed-Urdu recognizer and the open-source VLMs. We do not compare against handwriting-trained recognizers, which would likely do better in-domain; we characterize zero-shot behavior, not state of the art. We also describe three distinct failure modes among the ranked systems. In one of them, a commercial model explicitly declines to transcribe a few very hard-to-read images instead of guessing. In a second study (Part B), VLMs correct the output of a separate OCR system using both the source image and the noisy text. Once separated from each model's own reading ability, the benefit is small on Tesseract's output (0.7% to 7.4%) and larger (17.9% and 21.9%) only for the two commercial correctors given UTRNet's text. It appears to depend on how legible the supplied text already is, and a weaker corrector can make a more legible text worse.
Figures & tables
Fig. 1: Part A pipeline.
System
Category
CER
95% CI (CER)
WER
Empty / refusal
Tesseract
Traditional OCR
0.7215
[0.7145, 0.7286]
1.1793
5.1%
Qwen2.5-VL-7B
Open-source VLM
0.6464
[0.6343, 0.6592]
0.9495
0%
Qwen3-VL-8B
Open-source VLM
0.4861
[0.4737, 0.4990]
0.8380
0%
UTRNet-Large
Trained (script-specific)
0.3084
[0.3017, 0.3153]
0.7827
0%
Claude Sonnet 5
Commercial VLM
0.2656
[0.2550, 0.2769]
0.5225
0.20%
Gemini 3.5 Flash-Lite
Commercial VLM
0.1718
[0.1633, 0.1805]
0.3772
0%
TABLE I: Part A results ( N=2,465 ). CER and WER are diacritic-sensitive. CIs are 10,000-resample paired bootstrap intervals.
Failure mode
Seen in
Description
Rate
Empty output
Tesseract
Likely full-page mode on single lines (single-line mode untested)
5.1%
Repetition loop
Qwen2.5-VL, Qwen3-VL
Phrase repeated until the generation cap
1.66% / 0.81%
Refusal on hard images
Claude Sonnet 5
Explicit decline to transcribe
0.20%
None observed
Gemini 3.5 Flash-Lite
No empty or refused output
0%
TABLE II: Failure modes and their rates in Part A.
Corrector
Input
CER
Reduction vs. input (%) [95% CI]
Reduction vs. own PtA (%) [95% CI]
Qwen2.5-VL-7B
Tesseract
0.5984
17.1 [15.0, 19.0]
7.4 [5.0, 9.8]
Qwen2.5-VL-7B
UTRNet
0.3363
−9.0 [ −11.1 , −7.2 ]
48.0 [46.5, 49.5]
Qwen3-VL-8B
Tesseract
0.4829
33.1 [31.3, 34.7]
0.7 [ −2.1 , +3.3 ]
Qwen3-VL-8B
UTRNet
0.2963
3.9 [2.7, 5.2]
39.0 [37.4, 40.7]
Gemini 3.5 Flash-Lite
Tesseract
0.1613
77.6 [76.5, 78.7]
6.1 [3.9, 8.3]
Gemini 3.5 Flash-Lite
UTRNet
0.1410
54.3 [52.4, 56.1]
17.9 [15.7, 20.2]
TABLE III: Part B results. CER reduction vs. the input source and vs. the corrector’s own Part A CER (PtA), with 95% bootstrap CIs. Sonnet 5 rows exclude unusable empty outputs ( n=2,464 and 2,463 ).
Fig. 2: CER reduction vs. the input source and vs. each corrector’s own Part A CER.
Tesseract quality
n
Qwen2.5-VL
Qwen3-VL
Gemini
Claude Sonnet 5
Decent (CER ≤ 0.5)
266
+31.7 [27.0, 36.3]
+1.8 [ −9.0 , 11.3]
+14.0 [7.1, 20.4]
+7.7 [0.7, 14.3]
Bad (0.5–0.9)
1849
+5.9 [3.0, 8.6]
+0.9 [ −2.3 , 4.0]
+6.5 [4.4, 9.3]
+2.6 [ −0.1 , 5.5]
Very poor (CER > 0.9)
350
−0.4 [ −7.4 , 6.0]
−1.0 [ −8.5 , 5.8]
+1.8 [ −3.5 , 6.7]
+3.3 [ −5.5 , 14.8]
TABLE IV: Correction gain by Tesseract input legibility, as percent CER reduction relative to each corrector’s own Part A CER (95% bootstrap CIs).
Optical character recognition (OCR) has advanced rapidly with the rise of vision-language models, yet evaluation has remained concentrated on a small cluster of high- and mid-resource scripts. We introduce GlotOCR Bench, a comprehensive benchmark evaluating OCR generalization across 100+ Unicode scripts. Our benchmark comprises clean and degraded image variants rendered from real multilingual texts. Images are rendered using fonts from the Google Fonts repository, shaped with HarfBuzz and rasterized with FreeType, supporting both LTR and RTL scripts. Samples of rendered images were manually reviewed to verify correct rendering across all scripts. We evaluate a broad suite of open-weight and proprietary vision-language models and find that most perform well on fewer than ten scripts, and even the strongest frontier models fail to generalize beyond thirty scripts. Performance broadly tracks script-level pretraining coverage, suggesting that current OCR systems rely on language model pretraining as much as on visual recognition. Models confronted with unfamiliar scripts either produce random noise or hallucinate characters from similar scripts they already know. We release the benchmark and pipeline for reproducibility. Pipeline Code: https://github.com/cisnlp/glotocr-bench, Benchmark: https://hf.co/datasets/cis-lmu/glotocr-bench.
Amir Hossein Kargaran, Nafiseh Nikeghbal, Jana Diesner +2
Arabic calligraphy is a high-stakes cultural visual domain for Muslim communities, yet VLM evaluations rarely test its fine-grained script traditions. We audit 12 VLMs on a cleaned 1,195-image six-way benchmark using numeric, visual-cue, and Diwani-first prompts. In this closed-set setting, the best numeric-prompt accuracy is 60.9% for closed/API models and 47.2% for open-weight models. The main failure is default-label collapse: every numeric run is dominated by a single label taking 33-63% of predictions, against an 18.8% majority-class baseline. For 8 of 12 models the default is Thuluth; GPT-4o labels 52.9% of all images as Thuluth although it is only 18.1% of the set. Prompting helps unevenly: visual cues move accuracy by more than 10 points for some models but hurt others, and foregrounding Diwani often relocates the default rather than removing it. Arabic calligraphy recognition is therefore a compact, culturally grounded stress test for VLM robustness and for measuring which visual traditions become legible to automated systems.
Niaz Mohaiman Abtahi, Evan Maksud, Reyana Shahid
Advanced AI Lab Limited · United International University
In-the-wild Bengali scene text recognition is largely unmeasured: existing resources target handwritten documents or constrained sign-board parsing, report only aggregate edit-distance metrics, and evaluate either conventional OCR or VLMs, never both on the same in-the-wild data. To address this gap, we introduce BANGLAWILD, a benchmark of 2,535 Bengali scene text images, each paired with a verbatim gold transcription, two categorical axes, four diagnostic attributes, and an orthographically standard form where the in-image text deviates from canonical spelling. We evaluate fifteen VLMs and three conventional OCR systems under three prompting strategies, fine-tune 6 open-source models with LoRA, and complement edit-distance metrics with an LLM-as-a-Judge evaluation. Our results reveal a persistent gap in which larger models within the same family do not outperform smaller ones. Our fifteen-class error taxonomy shows that visual mis-recognition accounts for ~60% of errors in the strongest systems, while conjunct-related errors contribute under 2%, challenging a long-standing assumption in Bengali OCR research; the same visual dominant profile also holds across architectures, including the one conventional baseline that reads Bengali reliably. Prompt language mainly affects cross-script drift and LoRA reduces catastrophic failures in weak models without lifting the ceiling on already competent ones. Code and data will be publicly released.
Sadab Shiper, Tawsif Tashwar Dipto, Mir Md Inzamam +1
Islamic University of Technology, Bangladesh · BRAC University, Bangladesh