From Pixels to Pairs: A Comprehensive Benchmark of LLM-Driven Key-Value Extraction in Noisy Document Settings

Zahra Anvari

cs.CLJul 14, 2026

From Pixels to Pairs: A Comprehensive Benchmark of LLM-Driven Key-Value Extraction in Noisy Document Settings

Authors: Zahra Anvari

Organizations: Independent AI Researcher.

Abstract

Large language models (LLMs) have demonstrated strong capabilities in document key-value pair (KVP) extraction, yet controlled evaluations of their robustness to optical character recognition (OCR) output remain limited. This leaves an important gap in understanding their reliability in real-world OCR-to-LLM pipelines. Unlike end-to-end Vision-Language Models (VLMs), which jointly perform visual perception and semantic extraction, modular pipelines allow these stages and their errors to be isolated and audited. We introduce a controlled benchmark that distinguishes downstream LLM extraction behavior from upstream OCR degradation. It evaluates 136 experimental configurations and 17,688 document-level inferences generated with deterministic decoding across five instruction-tuned open-weight LLMs (2B-8B parameters), three datasets, and four text-quality conditions. The evaluation combines a full zero-shot comparison, targeted one- to three-shot experiments, and a sensitivity analysis of 40 configurations across 20 frozen demonstration sets. By separating Key Recall (annotated-field recovery) from Exact Match and Value F1 (exact and partial value recovery, respectively), we test whether OCR degradation affects field identification and value reproduction differently across models. Our findings challenge three practical assumptions: (1) clean-text performance reliably predicts real-world robustness, (2) model rankings remain consistent across annotation-derived Gold and OCR-derived text, and (3) additional few-shot demonstrations monotonically improve extraction accuracy. The observed model-ranking reversals and unstable few-shot gains expose important reliability risks under noisy document conditions. We release the benchmarking framework, dataset splits, and evaluation scripts to support reproducible research.

Figures & tables

Panel of Figure 1 — Figure 1: The end-to-end benchmark architecture pipeline. A document image is processed through branching OCR engines (or a Gold Text bypass) to construct a prompt. The resulting text is fed into the LLMs, and the predictions are evaluated using three complementary metrics, Key Recall (KR), Exact Match (EM), and Value F1 (VF1), to distinguish annotated-field recovery from exact and partial value recovery. The bottom banner highlights the combinatorial scale of the executed experimental matrix.

Panel of Figure 2 — Figure 2: Example documents from the evaluated test splits: (a) FUNSD, (b) SROIE, and (c) CORD.

Gold excerpt	PaddleOCR excerpt
TA #: T07281A SPONSOR’S NAME: Lorillard TEST ARTICLE IDENTIFICATION: B220 LOT #: 012590 CONDITION OF SHIPMENT: GOOD BROKEN LEAKED	[TA # label omitted] T07281A SPONSOR’S NAME: LORIllard TEST ARTTCKE IDENTIFICATION: : B220 LoT #: 012590 CONDTTION OF SHIMENT: OODEROKEN IZAFED

Figure 3: Four qualitative error-propagation pathways for zero-shot Qwen2.5–7B under PaddleOCR-derived text: (a) key-label loss and corruption in FUNSD document 87137840; (b) character-level value corruption in SROIE document X51006328937; (c) reading-order displacement with cascading value misalignment in CORD document receipt_00045; and (d) OCR-induced error amplification in SROIE document X51006619784. Each panel presents a focused document crop, the corresponding Gold and PaddleOCR excerpts, and the downstream extraction impact. Red marks the relevant OCR-induced changes.

Dataset	OCR Engine	Mean CER	Mean WER
FUNSD	PaddleOCR	0.4199	0.5675
	EasyOCR	0.4974	0.7318
	Tesseract	0.5485	0.7197
SROIE	PaddleOCR	0.3099	0.4577
	EasyOCR	0.4145	0.7148
	Tesseract	0.4334	0.6615

Table 1: Character Error Rate (CER) and Word Error Rate (WER) for the evaluated OCR engines across the three benchmark datasets. Lower values indicate better OCR quality.

Text	Model	KR	EM	VF1
Gold	Gemma 2B	0.2763	0.2252	0.2404
	Gemma 7B	0.3990	0.3080	0.3433
	Mistral–7B	0.5850	0.4627	0.5102
	Qwen2.5–7B	0.6332	0.5040	0.5519
	LLaMA 3–8B	0.4384	0.3032	0.3594
PaddleOCR	Gemma 2B	0.3067	0.1352	0.1887

Table 2: Zero-shot KVP extraction performance on FUNSD across Gold and OCR-derived text. Bold values indicate the best score within each text condition.

Panel of Figure 4 — Figure 4: Few-shot extraction trajectories for Qwen2.5–7B and LLaMA 3–8B across FUNSD, SROIE, and CORD under annotation-derived Gold text and PaddleOCR-derived text. Colors and marker shapes distinguish Key Recall (KR), Exact Match (EM), and Value F1 (VF1); solid filled markers denote Qwen2.5–7B, while dashed open markers denote LLaMA 3–8B. The trajectories demonstrate that the effect of additional demonstrations depends on the model, dataset, metric, and text condition, with no uniformly optimal demonstration count. Axis ranges are shared within each dataset row.

			Gold Text			PaddleOCR
Dataset	Model	Shots	KR	EM	VF1	KR	EM	VF1
FUNSD	Qwen2.5–7B	0	0.6332	0.5040	0.5519	0.5703	0.2477	0.3532
		1	0.6043	0.4991	0.5426	0.5941	0.2672	0.3658
		2	0.6023	0.4963	0.5406	0.5702	0.2492	0.3358
		3	0.6223	0.5110	0.5603	0.6035	0.2852	0.3824
	LLaMA 3–8B	0	0.4384	0.3032	0.3594	0.4653	0.2072	0.2848

Table 3: Few-shot KVP extraction performance across datasets under annotation-derived Gold text and PaddleOCR-derived text. Within each dataset, the same clean training demonstrations were used under both text conditions. Bold values indicate the best score for each model, text condition, and metric across the evaluated shot counts.

Panel of Figure 5 — Figure 5: Confirmatory Sensitivity Analysis: Variability in 3-shot Value F1 performance across demonstration complexity strategies (Qwen2.5-7B on FUNSD Gold). The observed ranges—for example, $53.73\%$ to $61.76\%$ within the Representative strategy—demonstrate substantial sensitivity to demonstration composition even when demonstration count and selection strategy are fixed.

Model	Strategy	$n$	KR	EM	VF1
Qwen2.5-7B	Low	5	$0.5790\pm 0.0219$ [-1pt] $[0.5514,\,0.6119]$	$0.4659\pm 0.0242$ [-1pt] $[0.4272,\,0.4906]$	$0.5146\pm 0.0197$ [-1pt] $[0.4845,\,0.5399]$
	Representative	5	$0.6530\pm 0.0329$ [-1pt] $[0.6169,\,0.6932]$	$0.5437\pm 0.0455$ [-1pt] $[0.4933,\,0.5961]$	$0.5785\pm 0.0374$ [-1pt] $[0.5373,\,0.6176]$
	High	4	$0.6301\pm 0.0075$ [-1pt] $[0.6229,\,0.6395]$	$0.5072\pm 0.0242$ [-1pt] $[0.4754,\,0.5276]$	$0.5506\pm 0.0161$ [-1pt] $[0.5297,\,0.5674]$
	Diverse	5	$0.6190\pm 0.0204$ [-1pt] $[0.6001,\,0.6458]$	$0.5031\pm 0.0202$ [-1pt] $[0.4847,\,0.5341]$	$0.5436\pm 0.0140$ [-1pt] $[0.5318,\,0.5595]$
LLaMA-3-8B	Low	5	$0.6130\pm 0.0302$ [-1pt] $[0.5784,\,0.6558]$	$0.5168\pm 0.0227$ [-1pt] $[0.4928,\,0.5440]$	$0.5531\pm 0.0203$ [-1pt] $[0.5313,\,0.5824]$
	Representative	5	$0.6578\pm 0.0315$ [-1pt] $[0.6254,\,0.7095]$	$0.5459\pm 0.0349$ [-1pt] $[0.5069,\,0.6000]$	$0.5829\pm 0.0297$ [-1pt] $[0.5530,\,0.6321]$

Table 4: Sensitivity of three-shot FUNSD Gold-text performance to demonstration composition. Each cell reports mean

\pm

sample standard deviation, with the observed range shown underneath.

Text	Model	KR	EM	VF1
Gold	Gemma 2B	0.8393	0.4107	0.5824
	Gemma 7B	1.0000	0.5778	0.8479
	Mistral–7B	0.9942	0.5836	0.8408
	Qwen2.5–7B	0.9950	0.6866	0.8930
	LLaMA 3–8B	0.6542	0.3869	0.5510
PaddleOCR	Gemma 2B	0.8624	0.3847	0.5735

Table 5: Zero-shot KVP extraction performance on SROIE across Gold and OCR-derived text. Bold values indicate the best score within each text condition.

Text	Model	KR	EM	VF1
Gold	Gemma 2B	0.3140	0.2673	0.2682
	Gemma 7B	0.7022	0.6121	0.6121
	Mistral–7B	0.7820	0.6327	0.6361
	Qwen2.5–7B	0.9045	0.8736	0.8748
	LLaMA 3–8B	0.8581	0.7383	0.7383
PaddleOCR	Gemma 2B	0.2179	0.1385	0.1385

Table 6: Zero-shot KVP extraction performance on CORD across Gold and OCR-derived text. Bold values indicate the best score within each text condition.

Panel of Figure 6 — Figure 6: Zero-shot Value F1 for Qwen2.5–7B across Gold and OCR-derived inputs. The magnitude of degradation varies substantially across datasets and OCR engines.

Explore similar work

Aug 3, 2026cs.AI

Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction

Intelligent document processing (IDP) with vision-language models (VLMs) hinges on confidence scores trustworthy enough to route extractions between automation and human review. Existing document benchmarks are dominated by clean, high-quality samples, leaving low accuracy regions too sparse for calibration assessment. We introduce ConfBench, the first calibration-specific benchmark for key information extraction (KIE), built by applying 20 controlled degradation pipelines to a diverse document set, yielding 1,346 variants and 70K+ entity-level evaluations spanning the full accuracy spectrum. We evaluate four proprietary and three open-weight VLMs under verbalized and log-probability confidence estimation methods across three input modalities, and find: (i) OCR+Image modality results in more accurate confidence estimates; (ii) model capability is the dominant factor: within the Claude family confidence quality scales monotonically with capability, while across families parameter count is a poor predictor; (iii) calibration quality varies widely across models, from near-perfect to severely overconfident, and per-model post-hoc correction rescales these absolute confidence values for threshold-based routing without altering ranking-based operational metrics; and (iv) log-probability with first-token aggregation consistently outperforms mean-token and margin aggregations. We also introduce ECARB, a review-budget metric translating discriminative gains into operational savings. We release ConfBench publicly to enable systematic study of confidence estimators and calibration methods for trustworthy IDP application deployment.

Priyashree Roy, Sujitha Martin, Mohammad Rostami +6

Amazon Web Services

Confidence Estimation Extraction First Comprehensive Benchmark Benchmark Saturation

Ask PDF

Sep 14, 2026cs.AI

Beyond Accuracy: Robustness, Cost, and Governance Trade-offs for Vision-Language Models in Templated Document Extraction

Vision-language models (VLMs) are increasingly used to extract structured fields from business documents, yet most evaluations report accuracy on clean benchmarks and offer little guidance to practitioners choosing an approach for a given task complexity. We address this gap with a measurement-grounded study and an open-source release. Across eleven systems (three commercial, two reasoning, five open-source VLMs in pretrained and fine-tuned form, and a non-LLM OCR->regex floor) scored on a 750-document held-out pool of synthetic checks, fine-tuning on 3K samples lifts the best open-source VLMs above F1 0.98-above every zero-shot commercial system on this task-while GPT-5 leads the commercial pool on F1 and Claude Sonnet 4.5 collapses on Date. To turn these measurements into actionable choices, we introduce a practitioner-oriented selection framework that maps a task profile (quality, latency, governance, volume) to a recommended approach via filtering and total-cost minimization, illustrated on a hypothetical mid-volume document-extraction scenario.

Kushal Patel, Pushkal Shrivastava, Mackenzie Lees +4

John Hancock, 200 Berkeley St, Boston, MA 02116, USA

Recent Vision-Language Models Visual Document Retrieval Governance

Ask PDF

May 5, 2026cs.CL

CC-OCR V2: Benchmarking Large Multimodal Models for Literacy in Real-world Document Processing

Large Multimodal Models (LMMs) have recently shown strong performance on Optical Character Recognition (OCR) tasks, demonstrating their promising capability in document literacy. However, their effectiveness in real-world applications remains underexplored, as existing benchmarks adopt task scopes misaligned with practical applications and assume homogeneous acquisition conditions. To address this gap, we introduce CC-OCR V2, a comprehensive and challenging OCR benchmark tailored to real-world document processing. CC-OCR V2 focuses on practical enterprise document processing tasks and incorporates hard and corner cases that are critical yet underrepresented in prior benchmarks, covering 5 major OCR-centric tracks: text recognition, document parsing, document grounding, key information extraction, and document question answering, comprising 7,093 high-difficulty samples. Extensive experiments on 14 advanced LMMs reveal that current models fall short of real-world application requirements. Even state-of-the-art LMMs exhibit substantial performance degradation across diverse tasks and scenarios. These findings reveal a significant gap between performance on current benchmarks and effectiveness in real-world applications. We release the full dataset and evaluation toolkit at https://github.com/eioss/CC-OCR-V2.

Zhipeng Xu, Junhao Ji, Zulong Chen +10

Alibaba Group · Northeastern University

Optical Character Recognition Large Multimodal Models Literacy

Ask PDF