OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets

Jiyuan Shen; Peiyue Yuan; Atin Ghosh; Yifan Mai; Daniel Dahlmeier

cs.CLMar 3, 2026

OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets

Authors: Jiyuan Shen, Peiyue Yuan, Atin Ghosh, Yifan Mai, Daniel Dahlmeier

Organizations: SAP · Stanford University

Abstract

Multimodal Large Language Models (MLLMs) enhance the potential of natural language processing. However, their actual impact on document information extraction remains unclear. In particular, it is unclear whether an MLLM-only pipeline--while simpler--can truly match the performance of traditional OCR+MLLM setups. In this paper, we conduct a large-scale benchmarking study that evaluates various out-of-the-box MLLMs on business-document information extraction. To examine and explore failure modes, we propose an automated hierarchical error analysis framework that leverages large language models (LLMs) to diagnose error patterns systematically. Our findings suggest that OCR may not be necessary for powerful MLLMs, as image-only input can achieve comparable performance to OCR-enhanced approaches. Moreover, we demonstrate that carefully designed schema, exemplars, and instructions can further enhance MLLMs performance. We hope this work can offer practical guidance and valuable insight for advancing document information extraction.

Figures & tables

Figure 1: Example of a document page extracted using our OCR engine.

Figure 2: An example of textual content extracted by our in-house OCR engine.

Figure 3: Hierarchical Error Analysis Framework

Company	Model	Image-only			OCR-only			Image + OCR
Company	Model	Dataset C1	Dataset C2	Mean	Dataset C1	Dataset C2	Mean	Dataset C1	Dataset C2	Mean
Meta	Llama 4 Scout	67.4	69.3	68.4	68.1	69.7	68.9	67.3	69.8	68.6
Meta	Llama 4 Maverick	62.8	68.2	65.5	63.9	68.1	66.0	62.9	68.2	65.5
MistralAI	Pixtral Large (2411)	68.7	57.4	63.1	75.3	71.2	73.3	72.7	68.0	70.4
Amazon	Nova Pro	77.9	65.1	71.5	68.7	65.1	66.9	77.5	66.6	72.1
OpenAI	GPT-4o mini	68.3	64.9	66.6	66.1	70.5	68.3	71.6	70.5	71.1

Table 1: Performance comparison of different MLLMs across evaluation settings: Image, OCR, and Image + OCR as input formats. C1 and C2 refer to two different datasets, while Mean denotes the arithmetic mean of the F1-scores on C1 and C2.

Figure 4: Performance comparison on various size models across different input types. The small shape ( , , ) denotes the arithmetic mean across two different categories of dataset.

+

is the F1-score in C1, while

\times

is for C2.

Model	Latency/Page (Est.)	Cost/Page (Est.)
GPT-4o	$\sim$ 2.2s	$\sim$ $0.006
Claude 3.5 Sonnet	$\sim$ 4.7s	$\sim$ $0.010
Claude 3 Opus	$\sim$ 7.0s	$\sim$ $0.050
Gemini 1.5 Pro	$\sim$ 3.9s	$\sim$ $0.001
Gemini 2.0 Pro	$\sim$ 2.0s	$\sim$ $0.004
Gemini 2.5 Flash	$\sim$ 1.4s	$\sim$ $0.0025

Table 2: Estimated latency and cost per page for different closed-source models.

Figure 5: Error analysis results for three different input modalities.

Google Gemini 1.5 Pro	Initial	Final
Dataset C1	87.3	89.1
Dataset C2	66.4	68.6
Mean	76.8	78.9

Table 3: Performance results for the optimized prompt template with image-only input.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Approx. Doc Count	Avg. Word Density	Page Language Distribution	Document Currencies	Document Countries
C1 + C2	Around 1,000	High density (financial tabular + semi-structured text, $\sim$ 150–400 words per page)	English ( $\sim$ 200), Spanish ( $\sim$ 150), French ( $\sim$ 100), Italian ( $\sim$ 80), German ( $\sim$ 90), Romanian ( $\sim$ 20), Slovak ( $\sim$ 10), Hungarian ( $\sim$ 10), Portuguese ( $\sim$ 10), Mixed/Unknown ( $\sim$ 150), Other ( $<$ 10 each)	Euro ( $\sim$ 70), Indian Rupee ( $\sim$ 70), US Dollar ( $\sim$ 50), British Pound ( $\sim$ 30), Generic/Masked ( $\sim$ 500), Chinese Yuan ( $\sim$ 10), UAE Dirham ( $\sim$ 10), Indonesian Rupiah ( $\sim$ 10), Swiss Franc ( $\sim$ 10), Vietnamese Dong ( $\sim$ 5), Malaysian Ringgit ( $\sim$ 5), Saudi Riyal ( $\sim$ 5), Venezuelan Bolívar ( $\sim$ 5), Australian Dollar ( $\sim$ 5), Philippine Peso ( $\sim$ 5), South Korean Won ( $<$ 5), South African Rand ( $<$ 5), Singapore Dollar ( $<$ 5), Moroccan Dirham ( $<$ 5), New Zealand Dollar ( $<$ 5), Bolivian Boliviano ( $<$ 5), Canadian Dollar ( $<$ 5), Azerbaijani Manat ( $<$ 5), Turkish Lira ( $<$ 5), Hungarian Forint ( $<$ 5), Danish Krone ( $<$ 5), Null/Unspecified ( $\sim$ 20)	Spain ( $\sim$ 120), Romania ( $\sim$ 80), France ( $\sim$ 80), Italy ( $\sim$ 80), Germany ( $\sim$ 90), India ( $\sim$ 70), Netherlands ( $\sim$ 70), US ( $\sim$ 40), UK ( $\sim$ 30), UAE ( $\sim$ 10), China ( $\sim$ 10), Indonesia ( $\sim$ 10), Venezuela ( $\sim$ 10), Saudi Arabia ( $\sim$ 10), Ireland ( $\sim$ 10), Austria ( $\sim$ 5), Switzerland ( $\sim$ 5), Denmark ( $\sim$ 5), Slovakia ( $\sim$ 5), Vietnam ( $\sim$ 5), Malaysia ( $\sim$ 5), Portugal ( $\sim$ 5), Hungary ( $\sim$ 5), Australia ( $\sim$ 5), Philippines ( $\sim$ 5), South Korea ( $<$ 5), South Africa ( $<$ 5), Singapore ( $<$ 5), Morocco ( $<$ 5), Peru ( $<$ 5), New Zealand ( $<$ 5), Bolivia ( $<$ 5), Canada ( $<$ 5), Azerbaijan ( $<$ 5), Turkey ( $<$ 5), Null/Unspecified ( $\sim$ 10)

Appendix

Table 4: Dataset characteristics: document counts, density, language distribution, currencies, and country coverage.

Figure 6: The corresponding image(cropped and censored) for example 1,2 and 3.

Figure 7: The corresponding image(cropped and censored) for example 4.

Figure 8: The corresponding image(cropped and censored) for example 5.

Figure 9: The corresponding image(cropped and censored) for example 6.

Figure 10: The corresponding image(cropped and censored) for example 7.

Figure 11: The corresponding image(cropped and censored) for example 8 and 9.

Document Information Extraction

Document Information Extraction turns text, tables, forms, and images into structured, searchable data.

See all 279 papers

From Pixels to Pairs: A Comprehensive Benchmark of LLM-Driven Key-Value Extraction in Noisy Document Settings
Jul 14, 2026Zahra AnvariLLM Evaluation Document Information Extraction
Independent AI Researcher.
CC-OCR V2: Benchmarking Large Multimodal Models for Literacy in Real-world Document Processing
May 5, 2026Zhipeng Xu, Junhao Ji, Zulong Chen +10Document Understanding Multimodal Large Language Models
Alibaba Group · Northeastern University
OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding
Aug 31, 2026Gengxu Li, Yuan Wu, Yi ChangVisual Reasoning Multimodal Reasoning
School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · 3International Center of Future Science, Jilin University

Jul 14, 2026J/K move · Enter open · S save

From Pixels to Pairs: A Comprehensive Benchmark of LLM-Driven Key-Value Extraction in Noisy Document Settings

Zahra Anvari

Independent AI Researcher.

LLM Evaluation Document Information Extraction Language Model Robustness

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated strong capabilities in document key-value pair (KVP) extraction, yet controlled evaluations of their robustness to optical character recognition (OCR) output remain limited. This leaves an important gap in understanding their reliability in real-world OCR-to-LLM pipelines. Unlike end-to-end Vision-Language Models (VLMs), which jointly perform visual perception and semantic extraction, modular pipelines allow these stages and their errors to be isolated and audited. We introduce a controlled benchmark that distinguishes downstream LLM extraction behavior from upstream OCR degradation. It evaluates 136 experimental configurations and 17,688 document-level inferences generated with deterministic decoding across five instruction-tuned open-weight LLMs (2B-8B parameters), three datasets, and four text-quality conditions. The evaluation combines a full zero-shot comparison, targeted one- to three-shot experiments, and a sensitivity analysis of 40 configurations across 20 frozen demonstration sets. By separating Key Recall (annotated-field recovery) from Exact Match and Value F1 (exact and partial value recovery, respectively), we test whether OCR degradation affects field identification and value reproduction differently across models. Our findings challenge three practical assumptions: (1) clean-text performance reliably predicts real-world robustness, (2) model rankings remain consistent across annotation-derived Gold and OCR-derived text, and (3) additional few-shot demonstrations monotonically improve extraction accuracy. The observed model-ranking reversals and unstable few-shot gains expose important reliability risks under noisy document conditions. We release the benchmarking framework, dataset splits, and evaluation scripts to support reproducible research.

Abstract

Document Information Extraction

Explore similar work

From Pixels to Pairs: A Comprehensive Benchmark of LLM-Driven Key-Value Extraction in Noisy Document Settings

CC-OCR V2: Benchmarking Large Multimodal Models for Literacy in Real-world Document Processing

OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding