VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents

Udi Barzelay; Ophir Azulai; Inbar Shapira; Idan Friedman; Foad Abo Dahood; Madison Lee; Abraham Daniels

cs.CVMar 16, 2026

VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents

Authors: Udi Barzelay, Ophir Azulai, Inbar Shapira, Idan Friedman, Foad Abo Dahood, Madison Lee, Abraham Daniels

Organizations: IBM Research

Abstract

We introduce VAREX (VARied-schema EXtraction), a benchmark for evaluating multimodal foundation models on structured data extraction from government forms. VAREX employs a Reverse Annotation pipeline that programmatically fills PDF templates with synthetic values, producing deterministic ground truth validated through three-phase quality assurance. The benchmark comprises 1,777 documents with 1,771 unique schemas across three structural categories, each provided in four input modalities: plain text, layout-preserving text (whitespace-aligned to approximate column positions), document image, or both text and image combined. Unlike existing benchmarks that evaluate from a single input representation, VAREX provides four controlled modalities per document, enabling systematic ablation of how input format affects extraction accuracy -- a capability absent from prior benchmarks. We evaluate 20 models from frontier proprietary models to small open models, with particular attention to models <=4B parameters suitable for cost-sensitive and latency-constrained deployment. Results reveal that (1) below 4B parameters, structured output compliance -- not extraction capability -- is a dominant bottleneck; in particular, schema echo (models producing schema-conforming structure instead of extracted values) depresses scores by 45-65 pp (percentage points) in affected models; (2) extraction-specific fine-tuning at 2B yields +81 pp gains, demonstrating that the instruction-following deficit is addressable without scale; (3) layout-preserving text provides the largest accuracy gain (+3-18 pp), exceeding pixel-level visual cues; and (4) the benchmark most effectively discriminates models in the 60-95% accuracy band. Dataset and evaluation code are publicly available.

Figures & tables

Figure 1: Varex benchmark overview. A government form (a) is paired with a per-document JSON schema (b) that defines the extraction target, including nested structures via $ref . Forms are programmatically filled with realistic data, and ground truth (c) is derived directly from the fill values. The benchmark spans 1,777 documents with 1,771 unique schemas.

Figure 2: The Reverse Annotation pipeline. Stage 1: Fillable PDF templates are filled with deterministic placeholders ( TXT_001 , TXT_002 , …). Stage 2: An LLM discovers a semantic schema by mapping placeholders to field names. Stage 3: Realistic synthetic values replace placeholders and are injected into form widgets. Stage 4: Each filled document is exported in four modalities.

Benchmark	Docs	Schemas	MM	SJ	GT Method	Domain
FUNSD	199	1	×	×	Manual	Forms
CORD	11K	1	×	Part.	Manual	Receipts
VRDU	3K	2	×	✓	Manual	Gov. forms
DocILE	6.7K	1	×	×	Hybrid	Business
ExtractB.	35	5	×	✓	Manual	Enterprise
Varex	1,777	1,771	✓	✓	Rev. An.	Gov. forms

Table 1: Comparison with publicly available benchmarks. MM = multi-modal inputs; SJ = structured JSON output. To our knowledge, Varex is the first benchmark combining unique per-document schemas, multi-modal inputs, and structured JSON output.

Figure 3: Dataset and evaluation overview. (a) Distribution of extraction fields per document (median 11). (b) Field-level EM% by semantic category for four representative models on Image ( V ); fields are grouped into nine categories by keyword matching ( e.g . , Name includes applicant_name , witness_name , etc.), covering 74% of 21,084 fields; email and monetary values show the widest cross-scale gaps (15–17 pp). (c) Number of vision models (out of 18) achieving perfect extraction per document; 91 documents (5%) receive imperfect scores from all models, largely attributable to residual annotation noise (see Sec. 4.1 ).

Frontier APIs
Model	Size	All	Flat	Nest.	Table	Perf.
Gemini 2.5 Pro [ 6 ]	API	98.0 (98.5)	97.3	98.2	97.7	82.8
Gemini 2.5 Flash [ 6 ]	API	97.3 (97.9)	96.9	97.7	96.5	81.0
GPT-4o [ 10 ]	API	94.8 (97.1)	95.1	95.3	93.5	64.3
Large Open ( $\geq$ 8B)
Qwen3-VL [ 1 ]	8B	96.6 (97.6)	96.6	97.3	95.0	78.2

Table 2: Varex Image (V) results. All = EM% (ANLS%). Perf. = perfect-document rate. Bold = best per column. † Fine-tuned for extraction (base: 9.7%). ‡ Schema echo dominates (Sec. 4.4).

Model	Size	P	S	V	S+V
Gem. 2.5 Flash [ 6 ]	API	93.3	96.5	97.3	97.8
GPT-4o [ 10 ]	API	93.0	95.8	94.8	96.9
Llama 4 Mav. [ 16 ]	17B $\times$ 128E	89.4	95.5	95.6	97.3
Qwen3-VL [ 1 ]	8B	88.0	94.4	96.6	97.1
Ministral [ 17 ]	14B	85.0	92.9	94.8	95.5
InternVL3.5 [ 25 ]	2B	75.8	88.9	85.6	89.4

Table 3: Modality comparison for selected models (EM%). Best modality per model in bold . Text-only baselines in italics .

Figure 4: Output compliance failures in small models (Image V). Dark bars: actual EM; hatched bars: EM on compliant documents. Failures include schema reproduction (dominant in InternVL3.5 1B) and schema-wrapped extraction (dominant in Qwen3-VL 2B). Qwen3-VL 2B drops from 91.5% to 34.2%; InternVL3.5 1B from 72.7% to 28.2%. NuExtract and h2oVL show no gap.

Model	200 DPI	50 DPI	$\Delta$
Gemini 2.5 Pro	98.0	96.3	$-$ 1.7
Gemini 2.5 Flash	97.3	93.8	$-$ 3.5
GPT-4o	94.8	65.7	$-$ 29.1
Qwen3-VL 8B	96.6	56.9	$-$ 39.7
Llama 4 Maverick	95.6	57.3	$-$ 38.3
Ministral 14B	94.8	55.1	$-$ 39.7

Table 4: Resolution robustness on Image V (EM%).

Figure 8: Varex document at standard resolution (200 DPI). A Table-category bond loss application with 11 array rows. All top models achieve 100% EM at this resolution.

Figure 9: Same document at reduced resolution (50 DPI). Serial numbers, dates, and small text become visually ambiguous. Open models in the 8–17B range lose 13–35 pp; Gemini models remain at 100%.

Model	Field	Ground Truth	Prediction
Maverick 17B $\times$ 128E	bonds[1].issue_date	05/22/2012	05/20/2012
	bonds[1].bond_id	BBB 98765432 B	BBB 8765432 B
	customer_number	DPM-2020-16514	DPM 2020-16514
Qwen3-VL 8B	customer_name	Nadia I. Jaimes	Nadia J. Jaimes
Qwen3-VL 8B	customer_number	DPM-2020-16514	DPM-2020-18514
Ministral 14B	customer_name	Nadia I. Jaimes	Nadia I. James

Table 5: Representative extraction errors at 50 DPI on the document in Figs. 8 and 9 . All models shown score 100% at 200 DPI. Error types at 50 DPI: digit confusion, character substitution, and missed fields.

Explore similar work

Apr 28, 2026

The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models

Large Language Models are increasingly being deployed to extract structured data from unstructured and semi-structured sources: parsing invoices, medical records, and converting PDF documents to database entries. Yet existing benchmarks for structured output generation either focus on schema compliance alone, or evaluate value correctness within a single source domain. We introduce SOB (The Structured Output Benchmark), a multi-source benchmark spanning three source modalities: native text, images, and audio conversations. All models receive a text-normalized representation of their context regardless of source modality; this deliberate design isolates structured-output capability from raw vision or speech-processing quality, ensuring a fair, source-agnostic comparison. Our benchmark comprises 5,000 text evaluation records derived from multi-hop QA drawn from a 25,091-record full corpus, 209 image records from OCR-processed PDFs across seven document types including multi-column layouts, dense tables, scanned historical documents, small-print text, and mathematical typesetting, and 115 audio records from the AMI corpus. Each record pairs a natural-language question with a JSON schema that the model must follow and a ground-truth answer verified against the source context. We evaluate 21 frontier and open-weight models across three source domains and seven metrics. Our results reveal a consistent pattern: models achieve near-perfect schema compliance, yet the best Value Accuracy, measured by exact leaf-value match, reaches only 83.0% on text, 67.2% on images, and 23.7% on audio, where longer context makes extraction substantially harder. We release the dataset, evaluation pipeline, and all related code.

Abhinav Kumar Singh, Harsha Vardhan Khurdula, Yoeven D Khemlani +1

1New Delhi, India · JigsawStack, Inc. 2San Francisco, CA, USA · 3Durgapur, WB, India

LLM Evaluation LLM Information Extraction Automated Evaluation Structured Output Generation

Ask PDF

Jul 31, 2026

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on HuggingFace and GitHub.

Boyang Zhang, Adrian Lyjak, Eli Stewart +2

LLM Grounding LLM Agent Evaluation Document Information Extraction

Ask PDF

Aug 18, 2026

Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application

The extraction of structured information from unstructured documents represents a critical component of digital transformations in all sectors. While proprietary solutions dominate commercial applications, a rapidly growing ecosystem of open-source Optical Character Recognition (OCR) engines, Large Language Models (LLMs), and Vision-Language Models (VLMs) offers accessible alternatives. However, systematic evaluations on realistic, multi-step extraction pipelines remain scarce. Responsible usage of such extraction tools require comprehensive evaluations on realistic tasks, especially as these solutions will be key components of applications in the public sector that the EU AI act categorizes as high risk. To address this gap we present a comprehensive benchmark assessing the end-to-end performance of open-source systems on a complex real-world document processing task classified as high risk: Student applications for an international study program. We conduct a comprehensive empirical evaluation with state-of-the-art OCR engines, LLMs and VLMs. Our results reveal that while VLMs generally outperform OCR+LLM pipelines, even state-of-the-art open-source models struggle to handle such tasks reliably in zero-shot settings. Only 4 of 35 configurations achieved F1 scores above 0.5, with the best OCR+LLM pipeline matching top VLM performance, though most OCR+LLM combinations performed substantially worse. Roughly 75% of all configurations scored below 0.25. Model scale influences performance, yet the relationship is non-linear: substantially larger models do not guarantee proportionally better results. Input quality, particularly the structural preservation of OCR output, emerges as a critical factor independent of downstream model capability.

Elias Schubert, Felix Bießmann

Berlin University of Applied Sciences, Luxemburger Str. 10, 13353 Berlin, Germany · Einstein Center Digital Future, Wilhelmstraße 67, 10117 Berlin, Germany

LLM Information Extraction Document Image Understanding LLM Evaluation VLM Evaluation

Ask PDF