When Models Misread Ink: Auditing VLMs for Arabic Calligraphy Recognition

Niaz Mohaiman Abtahi; Evan Maksud; Reyana Shahid

cs.CVOct 15, 2026

When Models Misread Ink: Auditing VLMs for Arabic Calligraphy Recognition

Authors: Niaz Mohaiman Abtahi, Evan Maksud, Reyana Shahid

Organizations: Advanced AI Lab Limited · United International University

Abstract

Arabic calligraphy is a high-stakes cultural visual domain for Muslim communities, yet VLM evaluations rarely test its fine-grained script traditions. We audit 12 VLMs on a cleaned 1,195-image six-way benchmark using numeric, visual-cue, and Diwani-first prompts. In this closed-set setting, the best numeric-prompt accuracy is 60.9% for closed/API models and 47.2% for open-weight models. The main failure is default-label collapse: every numeric run is dominated by a single label taking 33-63% of predictions, against an 18.8% majority-class baseline. For 8 of 12 models the default is Thuluth; GPT-4o labels 52.9% of all images as Thuluth although it is only 18.1% of the set. Prompting helps unevenly: visual cues move accuracy by more than 10 points for some models but hurt others, and foregrounding Diwani often relocates the default rather than removing it. Arabic calligraphy recognition is therefore a compact, culturally grounded stress test for VLM robustness and for measuring which visual traditions become legible to automated systems.

Figures & tables

Closed/API VLMs
Model	Acc. $\uparrow$	Macro F1 $\uparrow$	Thuluth bias	Diwani recall $\uparrow$	Diwani precision $\uparrow$	Diwani F1 $\uparrow$
GPT-4o	50.5	0.504	2.93	15.7	75.6	0.260
GPT-4.1	55.3	0.507	2.05	1.8	66.7	0.036
GPT-5.5	60.9	0.616	2.05	31.8	71.9	0.441
GPT-5.6 Terra	60.3	0.628	2.07	46.5	66.9	0.549
Open-weight VLMs

Table 1: Numeric-prompt results. Thuluth bias is predicted/true Thuluth count; closer to 1.0 is better. Bold marks the best value within each model block.

Figure 1: Numeric-prompt failure structure. Left: stacked counts show which true labels are incorrectly absorbed into Thuluth. Right: Diwani recall remains low for many models despite higher overall accuracy.

Figure 2: Prompt effects relative to the numeric prompt. Visual cues are inconsistent; Diwani-first often raises Diwani recall at the cost of calibration.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

True label	DuwatBench	RuFa	HICMA
Kufic	80	0	26
Naskh	103	0	115
Thuluth	216	0	0
Diwani	217	0	0
Ruq’ah	75	150	0
Nasta’liq	64	149	0

Appendix

Table 2: Source composition of the cleaned evaluation set by true label.

Figure 3: Representative examples from the six calligraphy styles in the cleaned evaluation set.

Figure 4: Representative numeric-prompt confusion matrices for GPT-4.1 and Qwen3-VL-32B.

Figure 5: Per-style recall across prompt conditions for six representative models.

Model	Prompt	Accuracy $\uparrow$	Balanced acc. $\uparrow$	Macro-F1 $\uparrow$	Thuluth ratio	Diwani recall $\uparrow$
GPT-4o	Numeric	50.5	54.1	0.504	2.93	15.7
GPT-4o	Visual-cue	60.8	63.3	0.627	2.02	43.3
GPT-4o	Diwani-first	59.6	62.6	0.607	1.56	41.0
GPT-4.1	Numeric	55.3	58.9	0.507	2.05	1.8
GPT-4.1	Visual-cue	56.6	60.0	0.516	2.11	1.8
GPT-4.1	Diwani-first	56.9	60.5	0.527	1.86	12.0

Appendix

Table 3: Full model–prompt summary. Accuracy and balanced accuracy are percentages; Macro-F1 is unitless; Thuluth ratio is predicted count divided by true count; Diwani recall is a percentage. Bold marks best or closest-to-ideal metric values.

Model	Prompt/run	n	Accuracy $\uparrow$	Balanced acc. $\uparrow$	Macro-F1 $\uparrow$	Thuluth ratio	Diwani recall $\uparrow$
GPT-4o	Closed-set B	1252	65.9	45.9	0.475	1.27	30.8
GPT-5.5	Closed-set B	1252	71.9	52.7	0.576	1.32	29.0
GPT-4o	Alt wording C	1252	49.3	41.6	0.368	0.55	85.3
GPT-5.5	Alt wording C	1252	70.1	43.9	0.469	1.25	47.3
GPT-4o	Label-order D	1252	63.0	49.4	0.492	1.08	43.3
GPT-5.5	Label-order D	1252	72.2	53.3	0.586	1.32	31.7

Appendix

Table 4: Original DuwatBench-only context runs on the clean six-class subset. These are not part of the main 36-run comparison. Bold marks the best value in each metric column.

Cultural Bias in VLMs

See all 14 papers

When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study
Aug 23, 2026Moshiur Farazi, Firoj Alam, Abderrahmane Maaradji +3Optical Character Recognition Document Image Understanding
University of Doha for Science and Technology, Doha, Qatar · Qatar Computing Research Institute, Hamad Bin Khalifa University, Doha, Qatar · Northwestern University in Qatar, Doha, Qatar
ArabicNumBench: Evaluating Arabic Number Reading in Large Language Models
Feb 21, 2026Anas Alhumud, Abdulaziz Alhammadi, Muhammad Badruddin KhanArabic NLP LLM Evaluation
Saudi Data and Artificial Intelligence Authority, Saudi Arabia · Imam Mohammad Ibn Saud Islamic University, Saudi Arabia
BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models
Aug 4, 2026Sadab Shiper, Tawsif Tashwar Dipto, Mir Md Inzamam +1VLM Evaluation Scene Text Recognition
Islamic University of Technology, Bangladesh · BRAC University, Bangladesh

Aug 23, 2026J/K move · Enter open · S save

When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study

Moshiur Farazi, Firoj Alam, Abderrahmane Maaradji, Zakaria Maamar+2

University of Doha for Science and Technology, Doha, Qatar · Qatar Computing Research Institute, Hamad Bin Khalifa University, Doha, Qatar · Northwestern University in Qatar, Doha, Qatar

Optical Character Recognition Document Image Understanding VLM Evaluation

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly being used for document understanding, yet their role in Arabic and Islamic manuscript recognition remains underexplored. To address such a gap in this paper, we evaluate traditional OCR, general-purpose VLMs, Arabic-specialized VLMs, and OCR-conditioned VLM correction across eight Arabic text datasets spanning historical manuscripts, aged printed books, clean print, multi-domain documents, and handwriting. The results show that no single approach dominates across setups. On line-level historical manuscripts, VLMs are close to Tesseract; on page-level manuscript images, they perform better; and in several settings, an OCR-conditioned corrector improves over both standalone OCR and standalone VLMs. The central finding is an OCR-prior recoverability principle: OCR conditioning helps when the OCR output remains visually and textually recoverable, providing anchors that the VLM can refine against the image. It improves recognition on aged print, clean print, mixed-domain Arabic, and some Naskh manuscripts, but degrades performance when the prior is script-mismatched or systematically misleading, as in Maghribi manuscripts and realistic student handwriting. Additional diagnostics show that Arabic VLM-OCR is sensitive to diacritics, preprocessing, generation budget, and repetition loops. These findings support an adaptive OCR-VLM workflow that routes pages according to script, OCR-prior recoverability, length diagnostics, and failure-mode indicators.

Abstract

Cultural Bias in VLMs

Explore similar work

When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study

ArabicNumBench: Evaluating Arabic Number Reading in Large Language Models

BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models