When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study

Moshiur Farazi; Firoj Alam; Abderrahmane Maaradji; Zakaria Maamar; Hamdy Mubarak; Wajdi Zaghouani

cs.CVAug 23, 2026

When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study

Authors: Moshiur Farazi, Firoj Alam, Abderrahmane Maaradji, Zakaria Maamar, Hamdy Mubarak, Wajdi Zaghouani

Organizations: University of Doha for Science and Technology, Doha, Qatar · Qatar Computing Research Institute, Hamad Bin Khalifa University, Doha, Qatar · Northwestern University in Qatar, Doha, Qatar

Abstract

Vision-language models (VLMs) are increasingly being used for document understanding, yet their role in Arabic and Islamic manuscript recognition remains underexplored. To address such a gap in this paper, we evaluate traditional OCR, general-purpose VLMs, Arabic-specialized VLMs, and OCR-conditioned VLM correction across eight Arabic text datasets spanning historical manuscripts, aged printed books, clean print, multi-domain documents, and handwriting. The results show that no single approach dominates across setups. On line-level historical manuscripts, VLMs are close to Tesseract; on page-level manuscript images, they perform better; and in several settings, an OCR-conditioned corrector improves over both standalone OCR and standalone VLMs. The central finding is an OCR-prior recoverability principle: OCR conditioning helps when the OCR output remains visually and textually recoverable, providing anchors that the VLM can refine against the image. It improves recognition on aged print, clean print, mixed-domain Arabic, and some Naskh manuscripts, but degrades performance when the prior is script-mismatched or systematically misleading, as in Maghribi manuscripts and realistic student handwriting. Additional diagnostics show that Arabic VLM-OCR is sensitive to diacritics, preprocessing, generation budget, and repetition loops. These findings support an adaptive OCR-VLM workflow that routes pages according to script, OCR-prior recoverability, length diagnostics, and failure-mode indicators.

Figures & tables

Figure 1: Representative data-level degradation modes in Arabic manuscript scans, drawn from samples in our evaluation corpora. Each mode corresponds to a failure pattern that traditional OCR and VLM systems handle differently.

Study	Script	VLM setting	Corrector	Diacritics	Cross-dataset
European and multilingual historical OCR
Greif et al. 2025	Latin	ZS	✓	✗	✗
Kim et al. 2025	Latin	ZS	✗	✗	✗
Crosilla et al. 2025	Latin multi	ZS	✗	✗	✓
Semnani et al. 2025	46 languages	FT	✗	✗	✓
Kanerva et al. 2025	Latin and Finnish	–	✓	✗	✓

Table 1: Positioning of this work against related studies. The columns indicate script family, VLM setting, whether OCR correction is tested, whether diacritics are analysed, and whether cross-dataset quality comparison is included. ✓ = yes, ✗ = no, – = not applicable.

Dataset	Content	Scale	Annotation	Access
Muharaf *	Historical Arabic manuscripts, 19th–21st c.	${\sim}$ 1,600 pages	Line + polygon	Public
RASAM 2 *	Maghribi manuscripts	${\sim}$ 300 pages	Line, PAGE-XML	Public
KHATT *	Modern Arabic handwriting	1,000 writers	Paragraph	Restricted
Madinah *	Ancient manuscripts, 8 books	40 pages	Page	Public
KITAB-Bench *	Multi-domain Arabic	3,760 samples	Mixed, 13 subsets	Public
SARD *	Synthetic printed Arabic	996 samples	Character + word	Public

Table 2: Arabic text recognition datasets. Datasets marked with * are used in our experiments.

Model	Paradigm	Parameters	Architecture
Tesseract 5 ( Smith 2007 )	Traditional OCR	—	Rule-based + LSTM
Qwen2.5-VL-7B-Instruct ( Bai et al. 2025a ) (Qwen)	General-purpose VLM	7B	Transformer VL
Fanar-2-Oryx-IVU ( FANAR Team et al. 2026 ) (Fanar-2)	Arabic-tuned VLM	7B	Qwen2.5-VL fine-tuned
QARI-OCR ( Wasfy et al. 2025 ; Bai et al. 2025b )	OCR-tuned VLM	4B	Qwen3-VL-4B + LoRA

Table 3: Summary of evaluated systems. Kraken ( Calfa 2024 ) is part of the open-source HTR landscape but is not included in the head-to-head comparison because its default installation does not ship with an Arabic-capable model.

Figure 2: Architecture of the hybrid OCR-corrector pipeline. Tesseract provides a noisy text. The VLM receives this prior together with the source image and outputs a transcription.

Model	With diacritics		Without diacritics
Model	CER $\downarrow$	WER $\downarrow$	CER $\downarrow$	WER $\downarrow$
Traditional OCR
Tesseract 5	69.6	103.7	69.2	103.6
Vision-Language Models
Qwen2.5-VL-7B	86.7	108.0	70.4	104.6
Fanar-2-Oryx-IVU	87.7	109.0	70.8	105.6

Table 4: Text recognition on the Muharaf test set. Corpus-level CER and WER are reported as percentages. Lower is better. Kraken is omitted because its default installation does not ship an Arabic-capable model and produced empty output on all 1,334 lines.

Model	Muharaf line		Madinah page		$\Delta$ CER
Model	CER	WER	CER	WER	$\Delta$ CER
Tesseract 5	69.2	103.6	51.0	85.7	$-18.2$
Qwen2.5-VL-7B	70.4	104.6	40.7	80.2	$-29.7$
Fanar-2	70.8	105.6	39.6	78.7	$-31.2$

Table 5: Comparison between Muharaf line images and Madinah page images. CER and WER are reported without diacritics.

Condition	CER $\downarrow$	WER $\downarrow$	$\Delta$ CER
Standalone systems
Tesseract 5	69.2	103.6	—
Qwen2.5-VL-7B (Qwen)	70.4	104.6	+1.2
Fanar-2-Oryx-IVU (Fanar-2)	70.8	105.6	+1.6
OCR-conditioned correctors
Tesseract $\rightarrow$ Qwen	60.6	93.9	$-8.6$

Table 6: OCR-conditioned correction on Muharaf. Metrics are corpus-level percentages with diacritics normalised.

Condition	CER	Edit-distance signal
Tesseract alone	69.2	—
Qwen standalone	70.4	—
Tesseract $\rightarrow$ Qwen with real prior and image	60.6	baseline
Shuffled prior with real image	81.6	Distance to wrong prior 64.5%
Real prior with image withheld	73.9	Distance to real prior 29.9%

Table 7: Ablations on the use of the OCR prior on Muharaf. The ablations use 1,198 lines with non-empty Tesseract output. CER is normalised.

Dataset	Setting	Tess.	Best VLM	Best corr.	$\Delta$ Rel.	Gap
SARD	Clean printed	4.7	Qwen 3.8	T $\rightarrow$ Q 2.6	+31.6%	+0.9
KITAB-Bench	Mixed	40.2	Qwen 38.7	T $\rightarrow$ Q 32.7	+15.5%	+1.5
Muharaf	Historical Naskh	69.2	Qwen 70.4	T $\rightarrow$ Q 60.6	+12.4%	$-1.2$
RASAM 2	Historical Maghribi	72.2	Qwen 63.9	T $\rightarrow$ Q 75.8	$-18.6$ %	+8.3
QNL Books	Aged printed	33.1	Fanar-2 19.2	T $\rightarrow$ F 15.5	+19.3%	+13.9
Students Essays	Modern student HW	77.4	Qwen 49.4	T $\rightarrow$ F 86.7	$-75.5$ %	+28.0

Table 8: Cross-dataset OCR-prior recoverability analysis. CER values are normalised percentages. The gap is Tesseract CER minus the best standalone VLM CER. Positive relative change means that the corrector improves over the best standalone method. Students Essays uses per-sample medians. The standalone VLM value uses the mitigated outputs on the catastrophic subset. Corrector medians are unmitigated. All other rows use dataset aggregates. SARD uses the 2,048-token rerun.

Figure 3: Relative improvement of the best corrector over the best standalone method. The Tesseract–VLM gap is useful, but it is not a complete decision rule. OCR-prior recoverability also depends on script, document setting, and error type.

Condition	CER $\downarrow$	WER $\downarrow$
Tesseract 5	40.2	77.3
Qwen2.5-VL-7B	38.7	57.5
Tesseract $\rightarrow$ Qwen	32.7	51.9

Table 9: KITAB-Bench aggregate results. Metrics are dataset-level and diacritics normalised.

Subset	N	Tess.	Qwen	Corrector
arabicocr	50	1.2	6.7	4.7
hindawi	200	16.5	21.2	15.6
patsocr	500	17.5	11.3	3.3
isippt	500	18.2	38.8	16.7
synthesizear	500	30.4	22.0	10.7
onlinekhatt	200	61.7	31.6	28.3

Table 10: KITAB-Bench per-subset CER. Values are normalised percentages. Bold marks the best method per subset.

Figure 4: KITAB-Bench per-subset CER. The khattparagraph subset is excluded from the plot because all methods exceed 394% CER.

Condition	CER $\downarrow$	WER $\downarrow$
Tesseract 5	33.1	63.9
Qwen2.5-VL-7B	20.1	35.4
Fanar-2-Oryx-IVU	19.2	34.7
Tesseract $\rightarrow$ Qwen	19.9	36.5
Tesseract $\rightarrow$ Fanar-2	15.5	30.0

Table 11: QNL Books results. The dataset contains 199 page-level samples from 100 aged printed Arabic books.

Model	CER			WER
Model	+D	$-$ D	$\Delta$	+D	$-$ D	$\Delta$
Tesseract 5	69.6	69.2	+0.4	103.7	103.6	+0.1
Qwen2.5-VL-7B	86.7	70.4	+16.3	108.0	104.6	+3.4
Fanar-2	87.7	70.8	+16.9	109.0	105.6	+3.4
QARI-OCR v0.4.0	194.7	167.2	+27.5	232.0	230.2	+1.8

Table 12: Diacritics sensitivity on Muharaf. CER and WER are reported with diacritics retained and with diacritics stripped. The difference is shown in percentage points.

Figure 5: CER with and without tashkeel . VLMs show a large diacritics penalty, while Tesseract is nearly flat.

Dataset	Qwen $\Delta$ pp	Tess. $\Delta$ pp
KHATT	+20.2	—
Muharaf	+16.3	+0.4
RASAM 2	+9.8	+0.1
KITAB-Bench	+6.9	+1.5
SARD	+2.9	+2.3

Table 13: Cross-dataset diacritics penalty for Qwen2.5-VL-7B. The values show the CER increase when diacritics are retained. Tesseract is shown where available.

Preprocessing	CER $\downarrow$	WER $\downarrow$	$\Delta$ CER
P1 Raw baseline	70.4	104.6	—
P5 Sharpen	67.2	100.6	$-4.5$ %
P3 Contrast CLAHE	71.3	103.3	+1.3%
P8 Full pipeline	72.3	109.7	+2.7%
P4 Denoise	72.4	104.0	+2.8%
P2 Binarised	72.8	106.3	+3.4%

Table 14: Effect of preprocessing on Qwen2.5-VL-7B. Muharaf results are corpus-level and diacritics normalised.

Figure 6: The eight preprocessing variants applied to a Muharaf line crop. Only sharpening improves CER. Other operations either smooth or quantise fine Arabic-script cues.

Condition	512 tokens		2048 tokens
Condition	CER $\downarrow$	WER $\downarrow$	CER $\downarrow$	WER $\downarrow$
Tesseract 5	4.7	24.4	4.7	24.4
Qwen2.5-VL-7B	36.3	40.8	3.8	11.2
Tesseract $\rightarrow$ Qwen	36.7	43.7	2.6	12.8

Table 15: SARD aggregate results before and after increasing the VLM generation token size. Tesseract is unaffected by this setting.

Font	Tess.	Qwen 512	Corrector 512
Arial	3.9	51.9	52.7
Calibri	4.0	52.7	53.0
Scheherazade New	4.4	12.5	13.7
Sakkal Majalla	4.5	27.1	28.4
Traditional Arabic	4.5	29.2	28.5
Amiri	7.7	24.0	23.2

Table 16: SARD per-font CER under the original 512-token size. The truncation artefact is concentrated on the long-reference fonts.

Condition	Median CER	Catastrophe rate	Corpus CER	Corpus CER + norep
Tesseract 5	77.4	0.0%	76.3	—
Qwen2.5-VL-7B	55.8	28.0%	199.0	66.6
Fanar-2-Oryx-IVU	56.7	27.5%	183.3	66.9
Tesseract $\rightarrow$ Qwen	90.9	30.5%	228.6	—
Tesseract $\rightarrow$ Fanar-2	86.7	27.0%	208.3	—

Table 17: Students Essays results. CER is normalised. The catastrophe rate is the fraction of samples with per-sample CER above 200%.

Figure 7: Per-sample CER on Students Essays under Qwen2.5-VL-7B before and after setting no_repeat_ngram_size =4. The mitigation reduces the catastrophic tail and lowers corpus CER from 199.0% to 66.6%.

Field	Naive prompt		Calibrated prompt		Category
Field	Non-null	Rate	Non-null	Rate	Category
Visual classification
Language	16/16	100.0	16/16	100.0	Classification
Script type	16/16	100.0	16/16	100.0	Classification
Content extraction
Subject	16/16	100.0	3/16	18.8	Content

Table 18: Zero-shot metadata extraction from 16 Madinah manuscript pages using Qwen, under a naive prompt and a null-enforcing calibrated prompt. Non-null rate measures how often the model attempts an extraction; under the naive prompt this rate is 100% for every field (compliance bias), while the calibrated prompt returns null for most content fields.

Error type	Tess.	Qwen	Fanar-2	QARI
Substitution	49.4	45.3	44.3	16.6
Omission	47.7	11.9	11.9	4.1
Diacritics	2.7	27.0	27.3	16.9
Hallucination	0.1	15.9	16.5	62.4

Table 19: Heuristic error type attribution on Muharaf. Each column is normalised so that the four heuristic counts sum to 100%.

Figure 8: Error type distribution on Muharaf. Tesseract errors are dominated by substitution and omission. VLM errors shift toward diacritics and hallucination.

Explore similar work

Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions
May 26, 2026Antonia Karamolegkou, Nicolas Angleraud, Benoît Sagot +1VLM Evaluation Document Understanding
Inria, Paris, France
From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists
Sep 18, 2026Yuanxiang Huangfu, Hanmeng Zhong, Linqing Chen +1Vision-Language Models Optical Character Recognition
PatSnap Co., LTD. Suzhou, China
From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction
Oct 8, 2026Uddipan Basu Bir, Vincent Christlein, Andreas Maier +1Document Information Extraction VLM Adaptation
Friedrich-Alexander-Universität Erlangen–Nürnberg, Germany

May 26, 2026J/K move · Enter open · S save

Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions

Antonia Karamolegkou, Nicolas Angleraud, Benoît Sagot, Thibault Clérice

Inria, Paris, France

VLM Evaluation Document Understanding Historical Document Analysis VLM Hallucination Vision-Language Grounding Optical Character Recognition

Open paper AskPDF arXiv

Recent work has shown that Vision-Language Models (VLMs) used for optical character recognition (OCR) can generate plausible but visually unsupported text, suggesting reliance on language priors. Comparing open-weight VLMs with traditional OCR baselines on low-resource Ancient Greek critical editions, we show that VLM errors often remain fluent even when wrong, producing plausible Greek substitutions where traditional engines produce local recognition noise. To analyze visual evidence during decoding, we introduce controlled image perturbations and token-level grounding measures based on conditional versus image-free decoding distributions. Under character-level perturbations, VLMs diverge sharply from the perturbed ground truth while traditional OCR remains comparatively faithful; however, token-level analysis shows that prior reliance is model-specific: in an OCR-specialist model, fluent lexical errors are produced with little reliance on the image, whereas general-purpose VLMs remain conditioned on the visual input even when wrong. Decode-time interventions fail to reliably restore grounding, while post-OCR language-model correction improves several systems only by repairing text after generation. Our results extend prior evidence of OCR language-prior reliance to low-resource historical documents and a broader set of models, showing that fluent output is not necessarily visually grounded and motivating interpretability-driven evaluation beyond aggregate accuracy.

Abstract

Explore similar work

Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions

From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists

From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction