AI agents are changing the requirements for document parsing. What matters is semantic correctness: parsed output must preserve the structure and meaning needed for autonomous decisions, including correct table structure, precise chart data, semantically meaningful formatting, and visual grounding. Existing benchmarks do not fully capture this setting for enterprise automation, relying on narrow document distributions and text-similarity metrics that miss agent-critical failures. We introduce ParseBench, a benchmark of ∼2,000 human-verified pages from enterprise documents spanning insurance, finance, and government, organized around five capability dimensions: tables, charts, content faithfulness, semantic formatting, and visual grounding. Across 14 methods spanning vision-language models, specialized document parsers, and LlamaParse, the benchmark reveals a fragmented capability landscape: no method is consistently strong across all five dimensions. LlamaParse Agentic achieves the highest overall score at 84.9%, and the benchmark highlights the remaining capability gaps across current systems. Dataset and evaluation code are available on https://huggingface.co/datasets/llamaindex/ParseBench and https://github.com/run-llama/ParseBench.
Figures & tables
Figure 1: ParseBench comprises ∼2,000 human-verified enterprise document pages and more than 169K test rules, organized around five capability dimensions: tables, charts, content faithfulness, semantic formatting, and visual grounding. Each dimension targets a distinct class of agent-critical parsing failures.
Benchmark
Year
Scale
Domain
Tables
Charts
Content
Formatting
Grounding
Evaluation
Subtask
PubTabNet [ 32 ]
2020
568K
Academic
✓
TEDS
FinTabNet [ 30 ]
2020
113K
Financial
✓
TEDS
DocLayNet [ 25 ]
2022
80K
Mixed a
∘ b
mAP
ChartQA [ 19 ]
2022
21K
Web
∘ c
Relaxed accuracy
E2E
OmniDocBench [ 24 ]
2024
1,355
Mixed d
✓
✓
∘ b
TEDS, edit dist.
OCRBench v2 [ 8 ]
2025
10K
Mixed
✓
∘ c
✓
TEDS, accuracy, F1
Table 1: Comparison of document parsing benchmarks. Subtask benchmarks evaluate a single capability on a narrow corpus; end-to-end benchmarks target multiple capabilities jointly. ParseBench covers all five capability dimensions on enterprise documents. Scale reflects the primary annotation unit (table/chart images for subtask benchmarks, document pages for end-to-end benchmarks).
Dataset
Metric
Pages
Docs
Rules
Tables
GTRM
503
284
—
Charts
Data Point Match Rate
568
99
4,864
Text
Content Faithfulness, Semantic Formatting
507
507
147,319
Layout
Element Pass Rate
500
321
16,325
Total
2,078
1,180
169,011
Table 2: ParseBench dataset statistics. The Text dataset serves both the Content Faithfulness and Semantic Formatting dimensions with different evaluation rules. Tables uses GTRM , a continuous metric (no discrete rules).
Category
Docs
Pages
% of Pages
SERFF
199
274
54.5%
Public Financial Docs
26
103
20.5%
Government Docs
12
54
10.7%
Other Public Docs
9
24
4.8%
FinTabNet
19
19
3.8%
Academic
6
13
2.6%
Table 3: Table dimension: data source distribution
Figure 2: Illustration of the TableRecordMatch metric. Predicted records and columns are matched to those in the ground truth. Each matched record is scored by binary cell-level agreement.
Figure 3: Illustration of the ChartDataPointMatch evaluation: a chart is parsed into a table, and annotated data points are verified by matching expected values and labels against the table’s rows and columns.
Category
Description
Docs
text_simple
Simple text with some styling
170
text_ocr
Scanned/image docs, various quality
129
text_multicolumns
1–8 columns, different layouts
117
text_multilang
20+ languages, all major scripts
47
text_misc
Unusual content/layout/reading order
33
text_dense
Dense, large docs (e.g., newspapers)
24
Table 4: Content faithfulness and semantic formatting: shared document categories. The same corpus serves both dimensions, with different evaluation rules applied.
Source PDF
Stripped output
Agent misreading
49.9939.99
49.9939.99
Two valid prices; may quote the old one
H 2 O
H2O
Treats “H2O” as an opaque identifier
See note 1
See note1
Reads “note1” as a single token, footnote lost
Total: $12,450
Total: $12,450
Misses that this is a key aggregate value
Figure 4: Formatting loss changes document semantics. Each row shows a source snippet, what a formatting-unaware parser emits, and the resulting misinterpretation by a downstream agent.
Provider
Overall
Tables
Charts
Content Faithful.
Semantic Format.
Visual Ground.
VLMs
OpenAI GPT-5 Mini
46.8
69.8
30.1
82.3
45.8
6.2
Anthropic Haiku 4.5
45.2
77.2
13.8
78.7
49.4
6.7
Google Gemini 3 Flash
71.0
89.9
64.8
86.2
58.4
56.0
Qwen 3 VL
62.0
74.7
28.2
87.6
64.2
55.2 *
Dots OCR 1.5
55.8
85.2
0.9
90.0
47.0
55.8
Table 5: Main ParseBench results (%). Overall is the unweighted mean across all five dimensions. Bold marks the best score across all providers in each column; underlined values mark the second-best. * For visual grounding, Qwen uses a separate layout-only pipeline; 4 pages where that pipeline failed to return usable output are excluded. See Appendix C.2 . † Docling’s visual-grounding score excludes 13 pages with pipeline failures.
Figure 6: Distribution of chart characteristics in the benchmark dataset. Per-chart dimensions are computed over all 1,039 charts; the chart count dimension is computed per page.
Provider
Category
Model / Mode
Notes
OpenAI GPT-5 Mini
VLM (commercial)
gpt-5-mini
Reasoning: minimal (default exceeds cost target)
Anthropic Haiku 4.5
VLM (commercial)
claude-haiku-4-5
Thinking: disabled (default)
Google Gemini 3 Flash
VLM (commercial)
gemini-3-flash
Thinking: minimal (default exceeds cost target)
Qwen 3 VL
VLM (open)
Qwen3-VL-8B-Instruct
H100, vLLM 0.11.2, BF16
Dots OCR 1.5
VLM (open)
dots-ocr-1.5
H100, vLLM 0.16.0, BF16
Docling
OSS Pipeline
Default pipeline
v2.73.1, H100
Appendix
Table 6: Per-provider evaluation configurations. All VLMs use the same standardized prompt per benchmark dimension.
Model
GPU
Serving Framework
Version
Notes
Qwen 3 VL (8B)
NVIDIA H100
vLLM
0.11.2
Full precision (BF16)
Dots OCR 1.5
NVIDIA H100
vLLM
0.16.0
Full precision (BF16)
Docling
NVIDIA H100
Native pipeline
v2.73.1
Chart extraction via Granite Vision 3.3 2B [ 12 ]
Appendix
Table 7: Hardware and software configurations for self-hosted models. All deployments use Modal serverless GPU containers with on-demand scaling.
Provider
Reasoning
Charts
Tables
Text
Layout
Average
(¢/page)
Anthropic Haiku 4.5
disabled †
0.97
2.46
1.53
1.60
1.64
thinking
3.49
4.26
3.16
3.83
3.69
Gemini 3 Flash
minimal
0.49
0.84
0.56
0.69
0.65
high †
2.38
2.29
2.25
2.70
2.41
GPT-5 Mini
minimal
0.70
1.22
0.67
0.91
0.88
Appendix
Table 8: Per-page cost (¢/page) across dataset categories for proprietary VLM configurations. All configurations include layout extraction. † marks default reasoning settings.
Figure 7: Tables: quality vs. cost.
Figure 8: Charts: quality vs. cost.
Figure 9: Content Faithfulness: quality vs. cost.
Figure 10: Semantic Formatting: quality vs. cost.
Figure 11: Visual Grounding: quality vs. cost.
Provider
Pages
Element Pass
Cls.
Loc.
Attr.
Reading Order
All Layout Rules
VLMs
OpenAI GPT-5 Mini
500/500
6.1
11.5
17.8
16.6
32.3
15.2
Anthropic Haiku 4.5
499/500
6.7
13.4
18.8
11.0
39.3
14.5
Google Gemini 3 Flash
497/500
56.0
60.0
68.5
71.9
82.2
66.5
Qwen 3 VL
496/500
55.2
60.7
68.0
70.4
79.9
66.1
Dots OCR 1.5
492/500
55.8
60.0
64.6
65.3
81.0
63.2
Appendix
Table 9: Fine-grained rule-based layout metrics (%). Bold marks the best score in each column; underlined values mark the second-best.
Provider
Pages
mAP@[.50:.95]
AF1
LAP
LAR
VLMs
OpenAI GPT-5 Mini
500/500
2.1
40.8
39.5
44.2
Anthropic Haiku 4.5
499/500
1.9
42.4
40.8
47.2
Google Gemini 3 Flash
497/500
42.8
83.8
84.0
85.4
Qwen 3 VL
496/500
32.1
86.7
91.4
85.1
Dots OCR 1.5
492/500
39.4
77.8
84.9
76.0
Appendix
Table 10: Detection and attribution-only layout diagnostics (%). Bold marks the best score in each column; underlined values mark the second-best.
Figure 12: Difficulty-stratified layout element pass rate on two reported buckets, Easy and Hard , for LlamaParse and the three strongest specialized parsers. The right panel reports LlamaParse’s absolute lead over the next-best method in each bucket.
Signal family
Feature and thresholds
Contribution
Layout density
rule_count >= 45, 60, 80, 110
+1 each
Layout density
picture_count >= 4, 10
+1 each
Layout density
table_count >= 1, 3
+1 each
Structural fragmentation
fragmented_row_count >= 5, 9
+1 each
Structural fragmentation
horizontal_band_count >= 3, 4
+1 each
Structural fragmentation
upward_reset_ratio >= 0.12
+1
Appendix
Table 11: Deterministic page-level heuristic used to score page difficulty for the appendix Easy / Hard analysis. Pages with scores ≤3 are reported as Easy ; all higher-scoring pages are reported as Hard .
Figure 13: Two contrasting predictions where GriTS and TEDS disagree with the table’s downstream usefulness. (a) A semantically wrong prediction that GriTS and TEDS round to ≈1.0 . (b) A semantically equivalent rewrite that GriTS and TEDS heavily penalize. TRM ranks both correctly.
Labels
Value
Relative tolerance
Accepted range
Below upper secondary, Sweden, Adjusted
10
0.1
[9,11]
Below upper secondary, Sweden, Unadjusted
3
0.5
[1.5,4.5]
Appendix
Table 12: Two representative test rules for the OECD chart example (Sweden / Below upper secondary). The wide 0.5 relative tolerance on the Unadjusted rule reflects that the bar is very short ( ≈ 3 score points) and difficult to read precisely from the 3D chart; any estimate between 1.5 and 4.5 is accepted.
Provider
Passed
Primary failure mode
LlamaParse Agentic
8/10
Minor label association gaps
Gemini 3 Flash (high thinking)
3/10
Labels not associated with values
Reducto
3/10
Incorrect values
GPT-5 Mini (minimal reasoning)
0/10
Incorrect values
Gemini 3 Flash (minimal thinking)
0/10
No table extracted
Appendix
Table 13: Per-provider results on the OECD chart example (10 test rules).
Figure 14: Source chart and provider outputs for the OECD chart example. (a) Source chart with Sweden / Below upper secondary marked in red. (b)–(e) Rendered table outputs (top portion shown); Sweden row highlighted in green (pass) or pink (fail). Gemini 3 Flash with minimal thinking omitted (no table, 0/10).
Provider
Content Faithfulness
Text Correctness
Order
LlamaParse Agentic
0.966
0.949
1.000
Textract
0.780
0.720
0.901
Haiku 4.5
0.659
0.876
0.225
Extend
0.577
0.648
0.437
Appendix
Table 14: Per-provider Content Faithfulness scores on the Federal Register example. Each competitor exhibits a distinct failure mode: Textract duplicates content, Haiku 4.5 hallucinates text and jumbles column order, and Extend reads across columns row-wise.
Figure 15: Source page and provider outputs for the Federal Register example. (a) Source 3-column page. (b)–(e) Rendered outputs. LlamaParse Agentic correctly linearizes columns; Textract wraps them into an HTML table; Extend reads row-wise across columns; Haiku 4.5 hallucinates words and jumbles column order.
Provider
Semantic Formatting
Text Styling
Title Accuracy
LlamaParse Agentic
1.000
1.000
1.000
GPT-5 Mini
0.829
1.000
0.658
Haiku 4.5
0.174
0.000
0.348
Textract
0.000
0.000
0.000
Appendix
Table 15: Per-provider Semantic Formatting scores on the infographic example. GPT-5 Mini preserves bold styling but flattens the heading hierarchy; Haiku 4.5 uses HTML tags instead of Markdown; Textract produces plain text with no formatting at all.
Figure 16: Source page and provider outputs for the infographic example. (a) Source page with 3-level heading hierarchy. (b)–(e) Rendered outputs. LlamaParse Agentic preserves hierarchy and bold titles; GPT-5 Mini flattens headings; Haiku 4.5 mixes HTML with Markdown; Textract produces plain text.
Provider
Element Pass Rate
Localization
Classification
Attribution
LlamaParse Agentic
86.4%
93.2%
86.4%
100.0%
Gemini 3 Flash
43.2%
45.5%
45.5%
62.1%
LandingAI
2.3%
4.5%
2.3%
3.4%
Haiku 4.5
0.0%
0.0%
0.0%
0.0%
Appendix
Table 16: Per-provider Element Pass Rate and sub-metric scores on the Sappi annual report page (44 ground truth elements). The element pass rate requires all three sub-checks to pass simultaneously. Gemini 3 Flash detects some elements but misses over half; LandingAI and Haiku 4.5 produce only 6 predictions each and fail almost entirely.
Figure 17: Ground truth and predicted layout overlays for the Sappi annual report page. (a) Ground truth with 44 elements across Picture , Text , Section , and Page-footer classes. (b)–(e) Provider predictions. LlamaParse Agentic produces 48 predictions closely matching GT; Gemini detects only 21; LandingAI and Haiku 4.5 produce only 6 each.
We introduce OfficeQA Pro, a benchmark for evaluating AI agents on grounded, multi-document reasoning over a large and heterogeneous document corpus. The corpus consists of U.S. Treasury Bulletins spanning nearly 100 years, comprising 89,000 pages and over 26 million numerical values. OfficeQA Pro consists of 133 questions that require precise document parsing, retrieval, and analytical reasoning across both unstructured text and tabular data. Frontier LLMs including Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview achieve less than 5% accuracy on OfficeQA Pro when relying on parametric knowledge, and less than 12% with additional access to the web. When provided directly with the document corpus, frontier agents still struggle on over half of questions, scoring 34.1% on average. We find that providing agents with a structured document representation produced by Databricks' ai_parse_document yields a 16.1% average relative performance gain across agents. We conduct additional ablations to study the effects of model selection, table representation, retrieval strategy, and test-time scaling on performance. Despite these improvements, significant headroom remains before agents can be considered reliable at enterprise-grade grounded reasoning.
Krista Opsahl-Ong, Arnav Singhvi, Jasmine Collins +10
Document parsing systems are increasingly deployed in high-stakes, regulated workflows such as mortgage underwriting, financial reporting, supply-chain logistics, and clinical records. Yet most public benchmarks evaluate parsers on clean academic layouts or synthetic prose, and report a single OCR or markdown-level similarity score. Such documents and metrics correlate poorly with what downstream agents actually need: the correct value for a specific field on a messy real-world page. We introduce RealDocBench, a two-track benchmark built from real regulated documents. The QA track contains 1,356 field-level questions over 581 documents spanning four domains, where each question is paired with a typed gold_dict of key-to-value answers and parsers are scored on both per-field and strict per-question accuracy. The layout track contains 1,500 human-verified page images annotated with COCO-style bounding boxes under a nine-class public taxonomy, scored with a Hungarian matcher that includes adjacency-aware split/merge recovery. We evaluate eighteen systems, spanning commercial parsing APIs, general-purpose VLMs, and open-source OCR models, under a uniform extraction-and-scoring protocol, and report accuracy alongside per-page cost and cache-busted latency. RealDocBench exposes a wide performance spread that single-number benchmarks hide, a persistently hard medical sub-domain, and sharp cost/latency trade-offs across operating points. We release the datasets, parser adapters, and evaluation harness to support reproducible, field-level comparison of document parsing systems.
Document parsing and recognition are fundamental capabilities for vision-language models (VLMs) and document processing systems. However, existing Optical Character Recognition (OCR) and document parsing benchmarks are increasingly limited in coverage and difficulty: many focus on common document genres or uniformly sampled pages where modern parsers already perform strongly, while offering limited annotation for expert-domain structures such as chemical formula, music notation, complex tables, and cross-page layouts. We introduce Dr. DocBench, a difficulty-aware benchmark for expert-level document parsing. Built from a large-scale multilingual book corpus, Dr. DocBench spans 52 BISAC subject domains and selects challenging documents through parser-failure-based sampling, targeting cases where multiple state-of-the-art systems struggle. It contains 4,514 annotated pages from long documents averaging around 100 pages, with 65k high-quality page- and block-level annotations for layout, reading order, hierarchical relations, and domain-specific visual contents. Evaluations of pipeline-based parsers and general-purpose VLMs show that strong performance on existing benchmarks does not transfer to our expert-level document parsing. Our analysis reveals substantial failures across subjects, content types, and structural attributes, highlighting Dr. DocBench as a comprehensive testbed for diagnosing and advancing document intelligence.
Minglai Yang, Xinyan Velocity Yu, Pengyuan Li +22
2077AI · University of Arizona · University of Southern California +8