DISCO: Document Intelligence Suite for COmparative Evaluation

Kenza Benkirane; Dan Goldwater; Martin Asenov; Aneiss Ghodsi

cs.CLMar 4, 2026

DISCO: Document Intelligence Suite for COmparative Evaluation

Authors: Kenza Benkirane, Dan Goldwater, Martin Asenov, Aneiss Ghodsi

Organizations: Parexel AI Labs, London, United Kingdom · Parexel AI Labs, San Francisco, United-States

Abstract

Document intelligence requires accurate text extraction and reliable reasoning over document content. We introduce DISCO, a Document Intelligence Suite for COmparative Evaluation, that evaluates optical character recognition (OCR) pipelines and vision-language models (VLMs) separately on parsing and question answering across diverse document types, including handwritten text, multilingual scripts, medical forms, infographics, and multi-page documents. Our evaluation shows that performance varies substantially across tasks and document characteristics, underscoring the need for complexity-aware approach selection. OCR pipelines are generally more reliable for handwriting and for long or multi-page documents, where explicit text grounding supports text-heavy reasoning, while VLMs perform better on multilingual text and visually rich layouts. Task-aware prompting yields mixed effects, improving performance on some document types while degrading it on others. These findings provide empirical guidance for selecting document processing strategies based on document structure and reasoning demands.

Figures & tables

Task	Dataset	Description
Parsing	IAM ${}_{\text{DISCO}}$ from Marti and Bunke (2002)	Handwritten text
Parsing	ICDAR 2015 ${}_{\text{DISCO}}$ from Karatzas and others (2019)	Multilingual scene text
Parsing	PubLayNet ${}_{\text{DISCO}}$ from Zhong et al. (2019)	Scientific document pages (layout)
Parsing	RxPad ${}_{\text{DISCO}}$ from Pattin et al. (2026)	Medical forms (French)
QA	DocVQA ${}_{\text{DISCO}}$ from Mathew et al. (2021)	Scanned forms and letters
QA	InfographicVQA ${}_{\text{DISCO}}$ from Mathew et al. (2022)	Infographics

Table 1: Benchmark suite composition: datasets and document types evaluated in DISCO. Note: All DISCO dataset subsets and accompanying evaluation artifacts will be released upon paper acceptance and with the final version of the paper. VisR-Bench and PubLayNet are included in the suite, but we do not report experimental results on them in this paper.

Task	Dataset	$P_{\text{OCR}}$	$P_{\text{VLM-base}}$	$P_{\text{VLM-task}}$
Parsing - $S_{\textrm{CER}}$	IAM ${}_{\text{DISCO}}$	0.087	0.171	0.080
	ICDAR ${}_{\text{DISCO}}$	0.553	0.213	0.073
	RxPad	0.654	0.660	0.659
Task	Dataset	$QA_{\text{OCR}}$	$QA_{\text{VLM-2stage}}$	$QA_{\text{VLM-direct}}$
QA - $S_{\textrm{GT-in-Pred}}$	DocVQA ${}_{\text{DISCO}}$	0.876	0.896	0.908
	InfographicVQA ${}_{\text{DISCO}}$	0.754	0.711	0.785

Table 2: DISCO benchmark results - Parsing

S_{\textrm{CER}}

and QA

S_{\textrm{GT\text{-}in\text{-}Pred}}

across OCR a VLM pipeline variants.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Model performance across phases on IAM ${}_{\text{DISCO}}$ .

Approach	CER $\downarrow$	WER $\downarrow$	ANLS $\uparrow$	$S_{\textrm{CS}}$ $\uparrow$
OCR models	5.53	4.67	0.07	0.45
VLM (generic prompt)	2.13	2.02	0.18	0.56
VLM (task-aware prompt)	0.73	0.85	0.22	0.47

Appendix

Table 3: Parsing average performance on ICDAR Mini. Results averaged across models within each phase.

Script Category	OCR (CER) $\downarrow$	VLM (CER) $\downarrow$	$n$
Chinese	5.04	1.93	94
Chinese, Latin	1.59	0.59	85
Hindi	3.16	1.17	106
Japanese	1.75	1.61	64
Korean	0.89	0.35	67
Latin	1.69	2.07	24

Appendix

Table 4: CER by script category for best-performing OCR ( azure-ai-documentintelligence ) vs best-performing VLM ( gpt-5-mini ). Sample counts in parentheses.

Figure 2: Model performance across phases on ICDAR ${}_{\text{DISCO}}$ . $P_{\text{OCR}}$ : OCR baseline; $P_{\text{VLM-base}}$ : VLM with generic prompting; $P_{\text{VLM-task}}$ : VLM with task-aware prompting. For CER and WER, lower (green) is better; for ANLS and Cosine Similarity, higher is better.

Phase	$S_{\textrm{CER}}$ $\downarrow$	$S_{\textrm{WER}}$ $\downarrow$	$S_{\textrm{CS}}$ $\uparrow$
$P_{\text{OCR}}$	0.654	0.589	0.482
$P_{\text{VLM-base}}$	0.660	0.594	0.476
$P_{\text{VLM-task}}$	0.659	0.583	0.482

Appendix

Table 5: Mean performance metrics across experimental phases on RxPad, averaged across all evaluated models.

Field (French)	GT Presence $\uparrow$	VLM Recall $\uparrow$
mg (Milligrams)	70.0%	100.0%
comprimé (Tablet)	46.5%	100.0%
traitement (Treatment)	31.5%	100.0%
médecin (Doctor)	32.0%	139.1%
ordonnance (Prescription)	13.5%	100.0%
fois par jour (Times/day)	25.0%	4.0%

Appendix

Table 6: Field extraction recall for VLMs in Phase

P_{\text{VLM-task}}

. Values above 100% indicate the model detected more instances than present in ground truth annotations.

Figure 3: Model performance across phases on RxPad. $P_{\text{OCR}}$ : OCR baseline; $P_{\text{VLM-base}}$ : VLM with base prompting; $P_{\text{VLM-task}}$ : VLM with task-aware prompting. For $S_{\textrm{CER}}$ and $S_{\textrm{WER}}$ , lower (green) is better; for $S_{\textrm{CS}}$ , higher is better.

Strategy	Best Configuration	$S_{\textrm{GT\text{-}in\text{-}Pred}}$ $\uparrow$	$S_{\textrm{ANLS}}$ $\uparrow$	$S_{\textrm{EM}}$ $\uparrow$	$S_{\textrm{CS}}$ $\uparrow$
$QA_{\text{OCR}}$	azure-ai-documentintelligence $\rightarrow$ gpt-5-mini	0.876	0.841	0.744	0.855
$QA_{\text{VLM-2stage}}$	gpt-5-mini $\rightarrow$ gpt-5-mini	0.896	0.868	0.779	0.876
$QA_{\text{VLM-direct}}$	gpt-5-mini	0.908	0.632	0.482	0.786

Appendix

Table 7: Best model performance by strategy on DocVQA.

S_{\textrm{GT\text{-}in\text{-}Pred}}

is the primary metric.

Figure 4: Regression performance when predicting QA correctness from the parsed document data (DocVQA).

Strategy	Mean $S_{\textrm{GT\text{-}in\text{-}Pred}}$ $\uparrow$	Mean $S_{\textrm{ANLS}}$ $\uparrow$
$QA_{\text{OCR}}$	0.840	0.720
$QA_{\text{VLM-2stage}}$	0.725	0.480
$QA_{\text{VLM-direct}}$	0.893	0.450

Appendix

Table 8: Mean QA performance by strategy on DocVQA (reported separately for

S_{\textrm{GT\text{-}in\text{-}Pred}}

and

S_{\textrm{ANLS}}

Panel of Figure 5 — Figure 5: DocVQA strategy heatmaps using the primary strategy metric $S_{\textrm{GT\text{-}in\text{-}Pred}}$ for (1) $QA_{\text{OCR}}$ , (2) $QA_{\text{VLM-2stage}}$ , and (3) $QA_{\text{VLM-direct}}$ .

Model / system	$S_{\textrm{GT-in-Extracted-Text}}$ $\uparrow$
azure-ai-documentintelligence	0.8712
mistral-ocr-2505	0.8479
mistral-ocr-2512	0.6146
gpt-5-nano	0.4479
gpt-5-mini	0.8695
claude-3.5-sonnet	0.8677

Appendix

Table 9:

S_{\textrm{GT-in-Extracted-Text}}

from parsed document data on DocVQA.

$S_{\textrm{GT-in-Pred}}$ (primary metric) $\uparrow$
OCR system	$QA_{\text{OCR}}^{\text{generic}}$	$QA_{\text{OCR}}^{\text{cot}}$	$QA_{\text{OCR}}^{\text{task-aware}}$
azure-ai-documentintelligence	0.814	0.846	0.831
mistral-ocr-2505	0.785	0.805	0.794
mistral-ocr-2512	0.574	0.555	0.579
$S_{\textrm{ANLS}}$ (string similarity) $\uparrow$
azure-ai-documentintelligence	0.623	0.833	0.733

Appendix

Table 10: OCR system performance on DocVQA

{}_{\text{DISCO}}

by phase (all using gpt-5-mini for QA).

Panel of Figure 6 — Figure 6: InfographicVQA strategy heatmaps. Each row corresponds to a QA strategy ( $QA_{\text{OCR}}$ , $QA_{\text{VLM-2stage}}$ , $QA_{\text{VLM-direct}}$ ) and reports four metrics: $S_{\textrm{GT\text{-}in\text{-}Pred}}$ , $S_{\textrm{ANLS}}$ , $S_{\textrm{EM}}$ , and $S_{\textrm{CS}}$ (Cosine Similarity).

Strategy	$S_{\textrm{GT\text{-}in\text{-}Pred}}$ $\uparrow$	$S_{\textrm{ANLS}}$ $\uparrow$	$S_{\textrm{EM}}$ $\uparrow$
$QA_{\text{OCR}}$ : OCR $\to$ QA ( azure-ai-documentintelligence )	0.754	0.629	0.515
$QA_{\text{VLM-2stage}}$ : VLM $\to$ QA	0.711	0.585	0.477
$QA_{\text{VLM-direct}}$ : Direct VQA	0.785	0.186	0.102

Appendix

Table 11: Best-performing configuration per strategy ( gpt-5-mini as QA model)

OCR System	$S_{\textrm{GT\text{-}in\text{-}Pred}}$ $\uparrow$	$S_{\textrm{ANLS}}$ $\uparrow$	$S_{\textrm{EM}}$ $\uparrow$
azure-ai-documentintelligence	0.754	0.629	0.515
mistral-ocr-2505	0.368	0.288	0.214
mistral-ocr-2512	0.345	0.273	0.190

Appendix

Table 12: Impact of OCR system on

QA_{\text{OCR}}

pipeline performance (

QA_{\text{OCR}}

-c phase, gpt-5-mini )

Panel of Figure 7 — Figure 7: DUDE strategy heatmaps. Each row corresponds to a QA strategy ( $QA_{\text{OCR}}$ , $QA_{\text{VLM-2stage}}$ , $QA_{\text{VLM-direct}}$ ) and reports four metrics: $S_{\textrm{GT\text{-}in\text{-}Pred}}$ , $S_{\textrm{ANLS}}$ , $S_{\textrm{EM}}$ , and $S_{\textrm{CS}}$ (Cosine Similarity).

Model / parsing method	$S_{\textrm{GT\text{-}in\text{-}Parsed\text{-}Data}}$ $\uparrow$
azure-ai-documentintelligence	0.746
mistral-ocr-2505	0.357
mistral-ocr-2512	0.321
gpt-5-nano	0.625
gpt-5-mini	0.694
claude-3-5-sonnet	0.645

Appendix

Table 13: Ground-truth coverage in parsed data across OCR-based and VLM-based parsing methods.

Strategy	$S_{\textrm{GT\text{-}in\text{-}Pred}}$ $\uparrow$	$S_{\textrm{ANLS}}$ $\uparrow$	$S_{\textrm{EM}}$ $\uparrow$	Substring Match $\uparrow$
$QA_{\text{OCR}}^{\text{strat}}$ (OCR+VLM)	0.514	0.286	0.186	0.538
$QA_{\text{VLM-2stage}}^{\text{strat}}$ (VLM Parse+QA)	0.371	0.213	0.129	0.391
$QA_{\text{VLM-direct}}^{\text{strat}}$ (Direct VQA)	0.493	0.377	0.219	0.524

Appendix

Table 14: Strategy-level performance comparison on DUDE (n=404 per phase)

Phase	OCR Tool	$S_{\textrm{GT\text{-}in\text{-}Pred}}$ $\uparrow$	$S_{\textrm{ANLS}}$ $\uparrow$	$S_{\textrm{EM}}$ $\uparrow$	$S_{\textrm{CS}}$ $\uparrow$
$QA_{\text{OCR}}^{\text{generic}}$	azure-ai-documentintelligence	0.517	0.356	0.205	0.596
$QA_{\text{OCR}}^{\text{generic}}$	mistral-ocr-2505	0.500	0.344	0.196	0.592
$QA_{\text{OCR}}^{\text{generic}}$	mistral-ocr-2512	0.512	0.341	0.196	0.591
$QA_{\text{OCR}}^{\text{cot}}$	azure-ai-documentintelligence	0.470	0.503	0.354	0.630
$QA_{\text{OCR}}^{\text{cot}}$	mistral-ocr-2505	0.470	0.512	0.361	0.634
$QA_{\text{OCR}}^{\text{cot}}$	mistral-ocr-2512	0.468	0.504	0.351	0.635

Appendix

Table 15: OCR tool comparison within

QA_{\text{OCR}}^{\text{strat}}

(OCR

\rightarrow

gpt-5-mini )

Parsing method	Ground truth in parsed data
Azure Intelligence OCR	0.505
Mistral OCR 3	0.391
Mistral OCR 2	0.468
GPT-5 Mini	0.465

Appendix

Table 16: Ground truth coverage in parsed data for different parsing methods.

	Parsing		QA Performance
OCR System	GT-in-Ext	Rank	QA ${}^{\text{cot}}_{\text{OCR}}$	QA ${}^{\text{task}}_{\text{OCR}}$
Azure Intelligence	0.505	1	0.475 (2)	0.567 (2)
Mistral OCR 2	0.468	2	0.475 (2)	0.567 (2)
Mistral OCR 3	0.391	3	0.468 (3)	0.579 (1)

Appendix

Table 17: Relationship between parsing effectiveness (GT-in-Extracted-Text) and QA performance on DUDEDISCO.

GPT-5 Mini (VLM parsing for reference)
OCR System	DocVQA	InfographicVQA	DUDE
Azure Intelligence	0.871	0.754	0.505
Mistral OCR 2 (2505)	0.848	0.357	0.468
Mistral OCR 3 (2512)	0.615	0.321	0.391
GPT-5 Mini	0.869	0.465	0.694

Appendix

Table 18: Ground truth coverage in parsed text by OCR system (GT-in-Extracted-Text metric). Higher values indicate more reliable text extraction.

OCR System	GT-in-Pred $\uparrow$	ANLS $\uparrow$
Mistral OCR 3 → GPT-5 Mini	0.579	0.000
Mistral OCR 2 → GPT-5 Mini	0.567	0.004
Azure Intelligence → GPT-5 Mini	0.567	0.000

Appendix

Table 19: QA1c phase anomaly on DUDEDISCO: high answer containment with zero format compliance across all OCR systems.

	P-a (OCR baseline)			P-b (generic VLM extraction)			P-c (task-aware VLM extraction)
Dataset	CER $\downarrow$	WER $\downarrow$	$S_{\textrm{CS}}$ $\uparrow$	CER $\downarrow$	WER $\downarrow$	$S_{\textrm{CS}}$ $\uparrow$	CER $\downarrow$	WER $\downarrow$	$S_{\textrm{CS}}$ $\uparrow$
IAM (handwriting)	0.0894	0.3011	0.9457	0.1626	0.2234	0.9046	0.0800	0.1098	0.9136
ICDAR (scene text)	5.5261	4.6697	0.4547	2.1256	2.0210	0.5589	0.7336	0.8544	0.4744
RxPad (prescriptions)	0.6541	0.5886	0.4820	0.6601	0.5935	0.4758	0.6591	0.5828	0.4818

Appendix

Table 20: Parsing metrics (cosine similarity, CER, WER) across datasets and phases, extracted from the notebook summary outputs. Where multiple systems are reported within a phase, the row reflects the system with the highest cosine similarity for that phase on that dataset (and its corresponding CER/WER).

Dataset	Phase	System	$S_{\textrm{GT\text{-}in\text{-}Pred}}$ $\uparrow$	$S_{\textrm{ANLS}}$ $\uparrow$	$S_{\textrm{EM}}$ $\uparrow$
DocVQA	$QA_{1}^{a}$	azure_intelligence__gpt-5-mini	0.8405	0.6284	0.4622
DocVQA	$QA_{1}^{b}$	azure_intelligence__gpt-5-mini	0.8763	0.8406	0.7444
DocVQA	$QA_{1}^{c}$	azure_intelligence__gpt-5-mini	0.8571	0.7412	0.5714
DocVQA	$QA_{2}^{a}$	gpt-5-mini__gpt-5-mini	0.8514	0.6159	0.4438
DocVQA	$QA_{2}^{b}$	gpt-5-mini__gpt-5-mini	0.8956	0.8683	0.7791
DocVQA	$QA_{3}^{\text{fewshot}}$	gpt-5-nano	0.8740	0.7455	0.4360

Appendix

Table 21: QA metrics (

S_{\textrm{GT\text{-}in\text{-}Pred}}

S_{\textrm{ANLS}}

S_{\textrm{EM}}

) across datasets and phases, extracted from the notebook summary outputs. For each dataset and phase, the row reflects the best run among those marked [PRIMARY] in the notebook outputs (selected by

S_{\textrm{ANLS}}

, then

S_{\textrm{EM}}

Criterion	DocVQA	InfographicVQA	DUDE	Overall
Parsing Effectiveness (GT-in-Extracted-Text)
Azure Intelligence	1	1	1	3/3
Mistral OCR 2	0	0	$\sim$	0/3
Mistral OCR 3	0	0	0	0/3
GPT-5 Mini (VLM)	$\sim$	0	1	1/3
QA Performance (GT-in-Pred with best prompt)

Appendix

Table 23: Winner summary by dataset and evaluation criterion. 1 = wins, 0 = loses,

\sim

= comparable performance (within 2 percentage points).

Figure 8: Speed vs accuracy trade-off across QA strategies on DocVQA ${}_{\text{DISCO}}$ . $QA_{\text{VLM-direct}}^{\text{}}$ (green) achieves the best efficiency frontier: highest accuracy with lowest latency. $QA_{\text{VLM-2stage}}^{\text{}}$ (olive) incurs 2–4 $\times$ longer inference times due to sequential parsing and reasoning stages.

Strategy	GPT-5-nano	GPT-5-mini	Claude 3.5 Sonnet
$QA_{\text{VLM-direct}}^{\text{}}$ (1 call)	$0.00011	$0.00058	$0.0060
$QA_{\text{VLM-2stage}}^{\text{}}$ (2 VLM calls)	$0.00034	$0.00170	$0.0150
$QA_{\text{OCR}}^{\text{}}$ (OCR + 1 LLM call) $\dagger$	$0.00012	$0.00043	$0.0046

Appendix

Table 24: Per-document cost for single-question QA.

\dagger

OCR cost estimated at $0.001/page (Azure Document Intelligence); text-only LLM calls avoid image token costs.

	Parsing	Question answering (QA)
Primary metric	$S_{\textrm{CS}}$ (Cosine Similarity): semantic similarity between the parsed output and the ground-truth text embeddings. We use $S_{\textrm{CS}}$ as the primary parsing metric because it captures semantic closeness between extracted and reference text, even when the surface form differs (e.g., minor OCR noise, punctuation, or formatting changes).	$S_{\textrm{GT-in-Pred}}$ (Ground-Truth-in-Prediction): binary substring-based metric equal to 1 if the ground-truth answer string appears in the prediction, 0 otherwise. We use $S_{\textrm{GT-in-Pred}}$ as the primary QA metric because it supports a binary correctness check and is robust to verbose LLM outputs (where the correct answer may be embedded in a longer explanation).
Other metrics	CER (Character Error Rate): character-level edit distance normalised by ground-truth length; WER (Word Error Rate): word-level edit distance normalised by ground-truth word count.	EM (Exact Match): 1 if the prediction exactly matches the ground truth, 0 otherwise; ANLS (Average Normalised Levenshtein Similarity): normalised string similarity based on Levenshtein distance.

Appendix

Table 32

Explore similar work

Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing
May 31, 2026Minglai Yang, Xinyan Velocity Yu, Pengyuan Li +22VLM Evaluation Document Parsing
2077AI · University of Arizona · University of Southern California +8
Qianfan-OCR: A Unified End-to-End Model for Document Intelligence
Mar 11, 2026Daxiang Dong, Mingming Zheng, Dong Xu +17Document Image Understanding Optical Character Recognition
OvisOCR2 Technical Report
Jul 15, 2026Shiyin Lu, Yinglun Li, Yu Xia +10Document Parsing Optical Character Recognition
ATH-MaaS, Alibaba Group

May 31, 2026J/K move · Enter open · S save

Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing

Minglai Yang, Xinyan Velocity Yu, Pengyuan Li, Xinyu Guo+21

2077AI · University of Arizona · University of Southern California+8

VLM Evaluation Document Parsing Document Layout Analysis Document Image Understanding

Open paper AskPDF arXiv

Document parsing and recognition are fundamental capabilities for vision-language models (VLMs) and document processing systems. However, existing Optical Character Recognition (OCR) and document parsing benchmarks are increasingly limited in coverage and difficulty: many focus on common document genres or uniformly sampled pages where modern parsers already perform strongly, while offering limited annotation for expert-domain structures such as chemical formula, music notation, complex tables, and cross-page layouts. We introduce Dr. DocBench, a difficulty-aware benchmark for expert-level document parsing. Built from a large-scale multilingual book corpus, Dr. DocBench spans 52 BISAC subject domains and selects challenging documents through parser-failure-based sampling, targeting cases where multiple state-of-the-art systems struggle. It contains 4,514 annotated pages from long documents averaging around 100 pages, with 65k high-quality page- and block-level annotations for layout, reading order, hierarchical relations, and domain-specific visual contents. Evaluations of pipeline-based parsers and general-purpose VLMs show that strong performance on existing benchmarks does not transfer to our expert-level document parsing. Our analysis reveals substantial failures across subjects, content types, and structural attributes, highlighting Dr. DocBench as a comprehensive testbed for diagnosing and advancing document intelligence.

Abstract

Explore similar work

Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing

Qianfan-OCR: A Unified End-to-End Model for Document Intelligence

OvisOCR2 Technical Report