cs.CVMar 16, 2026

VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents

Authors: Udi Barzelay, Ophir Azulai, Inbar Shapira, Idan Friedman, Foad Abo Dahood, Madison Lee, Abraham Daniels

Organizations: IBM Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce VAREX (VARied-schema EXtraction), a benchmark for evaluating multimodal foundation models on structured data extraction from government forms. VAREX employs a Reverse Annotation pipeline that programmatically fills PDF templates with synthetic values, producing deterministic ground truth validated through three-phase quality assurance. The benchmark comprises 1,777 documents with 1,771 unique schemas across three structural categories, each provided in four input modalities: plain text, layout-preserving text (whitespace-aligned to approximate column positions), document image, or both text and image combined. Unlike existing benchmarks that evaluate from a single input representation, VAREX provides four controlled modalities per document, enabling systematic ablation of how input format affects extraction accuracy -- a capability absent from prior benchmarks. We evaluate 20 models from frontier proprietary models to small open models, with particular attention to models <=4B parameters suitable for cost-sensitive and latency-constrained deployment. Results reveal that (1) below 4B parameters, structured output compliance -- not extraction capability -- is a dominant bottleneck; in particular, schema echo (models producing schema-conforming structure instead of extracted values) depresses scores by 45-65 pp (percentage points) in affected models; (2) extraction-specific fine-tuning at 2B yields +81 pp gains, demonstrating that the instruction-following deficit is addressable without scale; (3) layout-preserving text provides the largest accuracy gain (+3-18 pp), exceeding pixel-level visual cues; and (4) the benchmark most effectively discriminates models in the 60-95% accuracy band. Dataset and evaluation code are publicly available.

Figures & tables

Figure 1: Varex benchmark overview. A government form (a) is paired with a per-document JSON schema (b) that defines the extraction target, including nested structures via $ref . Forms are programmatically filled with realistic data, and ground truth (c) is derived directly from the fill values. The benchmark spans 1,777 documents with 1,771 unique schemas.
Figure 2: The Reverse Annotation pipeline. Stage 1: Fillable PDF templates are filled with deterministic placeholders ( TXT_001 , TXT_002 , …). Stage 2: An LLM discovers a semantic schema by mapping placeholders to field names. Stage 3: Realistic synthetic values replace placeholders and are injected into form widgets. Stage 4: Each filled document is exported in four modalities.
BenchmarkDocsSchemasMMSJGT MethodDomain
FUNSD1991××ManualForms
CORD11K1×Part.ManualReceipts
VRDU3K2×✓ManualGov. forms
DocILE6.7K1××HybridBusiness
ExtractB.355×✓ManualEnterprise
Varex1,7771,771✓✓Rev. An.Gov. forms
Table 1: Comparison with publicly available benchmarks. MM = multi-modal inputs; SJ = structured JSON output. To our knowledge, Varex is the first benchmark combining unique per-document schemas, multi-modal inputs, and structured JSON output.
Figure 3: Dataset and evaluation overview. (a) Distribution of extraction fields per document (median 11). (b) Field-level EM% by semantic category for four representative models on Image ( V ); fields are grouped into nine categories by keyword matching ( e.g . , Name includes applicant_name , witness_name , etc.), covering 74% of 21,084 fields; email and monetary values show the widest cross-scale gaps (15–17 pp). (c) Number of vision models (out of 18) achieving perfect extraction per document; 91 documents (5%) receive imperfect scores from all models, largely attributable to residual annotation noise (see Sec. 4.1 ).
ModelSizeAllFlatNest.TablePerf.
Frontier APIs
Gemini 2.5 Pro [ 6 ]API98.0 (98.5)97.398.297.782.8
Gemini 2.5 Flash [ 6 ]API97.3 (97.9)96.997.796.581.0
GPT-4o [ 10 ]API94.8 (97.1)95.195.393.564.3
Large Open ( ≥\geq 8B)
Qwen3-VL [ 1 ]8B96.6 (97.6)96.697.395.078.2
Table 2: Varex Image (V) results. All = EM% (ANLS%). Perf. = perfect-document rate. Bold = best per column. † Fine-tuned for extraction (base: 9.7%). ‡ Schema echo dominates (Sec. 4.4).
ModelSizePSVS+V
Gem. 2.5 Flash [ 6 ]API93.396.597.397.8
GPT-4o [ 10 ]API93.095.894.896.9
Llama 4 Mav. [ 16 ]17B ×\times 128E89.495.595.697.3
Qwen3-VL [ 1 ]8B88.094.496.697.1
Ministral [ 17 ]14B85.092.994.895.5
InternVL3.5 [ 25 ]2B75.888.985.689.4
Table 3: Modality comparison for selected models (EM%). Best modality per model in bold . Text-only baselines in italics .
Figure 4: Output compliance failures in small models (Image V). Dark bars: actual EM; hatched bars: EM on compliant documents. Failures include schema reproduction (dominant in InternVL3.5 1B) and schema-wrapped extraction (dominant in Qwen3-VL 2B). Qwen3-VL 2B drops from 91.5% to 34.2%; InternVL3.5 1B from 72.7% to 28.2%. NuExtract and h2oVL show no gap.
Model200 DPI50 DPIΔ\Delta
Gemini 2.5 Pro98.096.3−- 1.7
Gemini 2.5 Flash97.393.8−- 3.5
GPT-4o94.865.7−- 29.1
Qwen3-VL 8B96.656.9−- 39.7
Llama 4 Maverick95.657.3−- 38.3
Ministral 14B94.855.1−- 39.7
Table 4: Resolution robustness on Image V (EM%).
Figure 8: Varex document at standard resolution (200 DPI). A Table-category bond loss application with 11 array rows. All top models achieve 100% EM at this resolution.
Figure 9: Same document at reduced resolution (50 DPI). Serial numbers, dates, and small text become visually ambiguous. Open models in the 8–17B range lose 13–35 pp; Gemini models remain at 100%.
ModelFieldGround TruthPrediction
Maverick 17B ×\times 128Ebonds[1].issue_date05/22/201205/20/2012
bonds[1].bond_idBBB 98765432 BBBB 8765432 B
customer_numberDPM-2020-16514DPM 2020-16514
Qwen3-VL 8Bcustomer_nameNadia I. JaimesNadia J. Jaimes
customer_numberDPM-2020-16514DPM-2020-18514
Ministral 14Bcustomer_nameNadia I. JaimesNadia I. James
Table 5: Representative extraction errors at 50 DPI on the document in Figs. 8 and 9 . All models shown score 100% at 200 DPI. Error types at 50 DPI: digit confusion, character substitution, and missed fields.

Explore similar work

CardsList

  1. The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models

    Apr 28, 2026Abhinav Kumar Singh, Harsha Vardhan Khurdula, Yoeven D Khemlani +1LLM EvaluationLLM Information Extraction

    1New Delhi, India · JigsawStack, Inc. 2San Francisco, CA, USA · 3Durgapur, WB, India

  2. ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

    Jul 31, 2026Boyang Zhang, Adrian Lyjak, Eli Stewart +2LLM GroundingLLM Agent Evaluation

  3. Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application

    Aug 18, 2026Elias Schubert, Felix BießmannLLM Information ExtractionDocument Image Understanding

    Berlin University of Applied Sciences, Luxemburger Str. 10, 13353 Berlin, Germany · Einstein Center Digital Future, Wilhelmstraße 67, 10117 Berlin, Germany