cs.CVApr 9, 2026
SaveParseBench: A Document Parsing Benchmark for AI Agents
Abstract
AI agents are changing the requirements for document parsing. What matters is semantic correctness: parsed output must preserve the structure and meaning needed for autonomous decisions, including correct table structure, precise chart data, semantically meaningful formatting, and visual grounding. Existing benchmarks do not fully capture this setting for enterprise automation, relying on narrow document distributions and text-similarity metrics that miss agent-critical failures. We introduce ParseBench, a benchmark of human-verified pages from enterprise documents spanning insurance, finance, and government, organized around five capability dimensions: tables, charts, content faithfulness, semantic formatting, and visual grounding. Across 14 methods spanning vision-language models, specialized document parsers, and LlamaParse, the benchmark reveals a fragmented capability landscape: no method is consistently strong across all five dimensions. LlamaParse Agentic achieves the highest overall score at 84.9%, and the benchmark highlights the remaining capability gaps across current systems. Dataset and evaluation code are available on https://huggingface.co/datasets/llamaindex/ParseBench and https://github.com/run-llama/ParseBench.
Figures & tables
Figure 1: ParseBench comprises human-verified enterprise document pages and more than 169K test rules, organized around five capability dimensions: tables, charts, content faithfulness, semantic formatting, and visual grounding. Each dimension targets a distinct class of agent-critical parsing failures.
| Benchmark | Year | Scale | Domain | Tables | Charts | Content | Formatting | Grounding | Evaluation | |
| Subtask | PubTabNet [ 32 ] | 2020 | 568K | Academic | TEDS | |||||
| FinTabNet [ 30 ] | 2020 | 113K | Financial | TEDS | ||||||
| DocLayNet [ 25 ] | 2022 | 80K | Mixed a | b | mAP | |||||
| ChartQA [ 19 ] | 2022 | 21K | Web | c | Relaxed accuracy | |||||
| E2E | OmniDocBench [ 24 ] | 2024 | 1,355 | Mixed d | b | TEDS, edit dist. | ||||
| OCRBench v2 [ 8 ] | 2025 | 10K | Mixed | c | TEDS, accuracy, F1 |
Table 1: Comparison of document parsing benchmarks. Subtask benchmarks evaluate a single capability on a narrow corpus; end-to-end benchmarks target multiple capabilities jointly. ParseBench covers all five capability dimensions on enterprise documents. Scale reflects the primary annotation unit (table/chart images for subtask benchmarks, document pages for end-to-end benchmarks).
| Dataset | Metric | Pages | Docs | Rules |
| Tables | GTRM | 503 | 284 | — |
| Charts | Data Point Match Rate | 568 | 99 | 4,864 |
| Text | Content Faithfulness, Semantic Formatting | 507 | 507 | 147,319 |
| Layout | Element Pass Rate | 500 | 321 | 16,325 |
| Total | 2,078 | 1,180 | 169,011 |
Table 2: ParseBench dataset statistics. The Text dataset serves both the Content Faithfulness and Semantic Formatting dimensions with different evaluation rules. Tables uses GTRM , a continuous metric (no discrete rules).
| Category | Docs | Pages | % of Pages |
| SERFF | 199 | 274 | 54.5% |
| Public Financial Docs | 26 | 103 | 20.5% |
| Government Docs | 12 | 54 | 10.7% |
| Other Public Docs | 9 | 24 | 4.8% |
| FinTabNet | 19 | 19 | 3.8% |
| Academic | 6 | 13 | 2.6% |
Table 3: Table dimension: data source distribution
Figure 2: Illustration of the TableRecordMatch metric. Predicted records and columns are matched to those in the ground truth. Each matched record is scored by binary cell-level agreement.
Figure 3: Illustration of the ChartDataPointMatch evaluation: a chart is parsed into a table, and annotated data points are verified by matching expected values and labels against the table’s rows and columns.
| Category | Description | Docs |
| text_simple | Simple text with some styling | 170 |
| text_ocr | Scanned/image docs, various quality | 129 |
| text_multicolumns | 1–8 columns, different layouts | 117 |
| text_multilang | 20+ languages, all major scripts | 47 |
| text_misc | Unusual content/layout/reading order | 33 |
| text_dense | Dense, large docs (e.g., newspapers) | 24 |
Table 4: Content faithfulness and semantic formatting: shared document categories. The same corpus serves both dimensions, with different evaluation rules applied.
| Source PDF | Stripped output | Agent misreading |
| 39.99 | 39.99 | Two valid prices; may quote the old one |
| H 2 O | H2O | Treats “H2O” as an opaque identifier |
| See note 1 | See note1 | Reads “note1” as a single token, footnote lost |
| Total: $12,450 | Total: $12,450 | Misses that this is a key aggregate value |
Figure 4: Formatting loss changes document semantics. Each row shows a source snippet, what a formatting-unaware parser emits, and the resulting misinterpretation by a downstream agent.
| Provider | Overall | Tables | Charts | Content Faithful. | Semantic Format. | Visual Ground. |
| VLMs | ||||||
| OpenAI GPT-5 Mini | 46.8 | 69.8 | 30.1 | 82.3 | 45.8 | 6.2 |
| Anthropic Haiku 4.5 | 45.2 | 77.2 | 13.8 | 78.7 | 49.4 | 6.7 |
| Google Gemini 3 Flash | 71.0 | 89.9 | 64.8 | 86.2 | 58.4 | 56.0 |
| Qwen 3 VL | 62.0 | 74.7 | 28.2 | 87.6 | 64.2 | 55.2 * |
| Dots OCR 1.5 | 55.8 | 85.2 | 0.9 | 90.0 | 47.0 | 55.8 |
Table 5: Main ParseBench results (%). Overall is the unweighted mean across all five dimensions. Bold marks the best score across all providers in each column; underlined values mark the second-best. * For visual grounding, Qwen uses a separate layout-only pipeline; 4 pages where that pipeline failed to return usable output are excluded. See Appendix C.2 . Docling’s visual-grounding score excludes 13 pages with pipeline failures.
Figure 5: Quality vs. cost for all evaluated providers ( Section 4.3 ). Per-page costs use publicly listed pay-as-you-go prices [ 23 , 9 , 4 , 18 , 1 , 10 , 20 , 27 , 7 , 14 ] .
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Distribution of chart characteristics in the benchmark dataset. Per-chart dimensions are computed over all charts; the chart count dimension is computed per page.
| Provider | Category | Model / Mode | Notes |
| OpenAI GPT-5 Mini | VLM (commercial) | gpt-5-mini | Reasoning: minimal (default exceeds cost target) |
| Anthropic Haiku 4.5 | VLM (commercial) | claude-haiku-4-5 | Thinking: disabled (default) |
| Google Gemini 3 Flash | VLM (commercial) | gemini-3-flash | Thinking: minimal (default exceeds cost target) |
| Qwen 3 VL | VLM (open) | Qwen3-VL-8B-Instruct | H100, vLLM 0.11.2, BF16 |
| Dots OCR 1.5 | VLM (open) | dots-ocr-1.5 | H100, vLLM 0.16.0, BF16 |
| Docling | OSS Pipeline | Default pipeline | v2.73.1, H100 |
Table 6: Per-provider evaluation configurations. All VLMs use the same standardized prompt per benchmark dimension.
| Model | GPU | Serving Framework | Version | Notes |
| Qwen 3 VL (8B) | NVIDIA H100 | vLLM | 0.11.2 | Full precision (BF16) |
| Dots OCR 1.5 | NVIDIA H100 | vLLM | 0.16.0 | Full precision (BF16) |
| Docling | NVIDIA H100 | Native pipeline | v2.73.1 | Chart extraction via Granite Vision 3.3 2B [ 12 ] |
Table 7: Hardware and software configurations for self-hosted models. All deployments use Modal serverless GPU containers with on-demand scaling.
| Provider | Reasoning | Charts | Tables | Text | Layout | Average |
| (¢/page) | ||||||
| Anthropic Haiku 4.5 | disabled † | 0.97 | 2.46 | 1.53 | 1.60 | 1.64 |
| thinking | 3.49 | 4.26 | 3.16 | 3.83 | 3.69 | |
| Gemini 3 Flash | minimal | 0.49 | 0.84 | 0.56 | 0.69 | 0.65 |
| high † | 2.38 | 2.29 | 2.25 | 2.70 | 2.41 | |
| GPT-5 Mini | minimal | 0.70 | 1.22 | 0.67 | 0.91 | 0.88 |
Table 8: Per-page cost (¢/page) across dataset categories for proprietary VLM configurations. All configurations include layout extraction. marks default reasoning settings.
Figure 7: Tables: quality vs. cost.
Figure 8: Charts: quality vs. cost.
Figure 9: Content Faithfulness: quality vs. cost.
Figure 10: Semantic Formatting: quality vs. cost.
Figure 11: Visual Grounding: quality vs. cost.
| Provider | Pages | Element Pass | Cls. | Loc. | Attr. | Reading Order | All Layout Rules |
| VLMs | |||||||
| OpenAI GPT-5 Mini | 500/500 | 6.1 | 11.5 | 17.8 | 16.6 | 32.3 | 15.2 |
| Anthropic Haiku 4.5 | 499/500 | 6.7 | 13.4 | 18.8 | 11.0 | 39.3 | 14.5 |
| Google Gemini 3 Flash | 497/500 | 56.0 | 60.0 | 68.5 | 71.9 | 82.2 | 66.5 |
| Qwen 3 VL | 496/500 | 55.2 | 60.7 | 68.0 | 70.4 | 79.9 | 66.1 |
| Dots OCR 1.5 | 492/500 | 55.8 | 60.0 | 64.6 | 65.3 | 81.0 | 63.2 |
Table 9: Fine-grained rule-based layout metrics (%). Bold marks the best score in each column; underlined values mark the second-best.
| Provider | Pages | mAP@[.50:.95] | AF1 | LAP | LAR |
| VLMs | |||||
| OpenAI GPT-5 Mini | 500/500 | 2.1 | 40.8 | 39.5 | 44.2 |
| Anthropic Haiku 4.5 | 499/500 | 1.9 | 42.4 | 40.8 | 47.2 |
| Google Gemini 3 Flash | 497/500 | 42.8 | 83.8 | 84.0 | 85.4 |
| Qwen 3 VL | 496/500 | 32.1 | 86.7 | 91.4 | 85.1 |
| Dots OCR 1.5 | 492/500 | 39.4 | 77.8 | 84.9 | 76.0 |
Table 10: Detection and attribution-only layout diagnostics (%). Bold marks the best score in each column; underlined values mark the second-best.
Figure 12: Difficulty-stratified layout element pass rate on two reported buckets, Easy and Hard , for LlamaParse and the three strongest specialized parsers. The right panel reports LlamaParse’s absolute lead over the next-best method in each bucket.
| Signal family | Feature and thresholds | Contribution |
| Layout density | rule_count >= 45, 60, 80, 110 | +1 each |
| Layout density | picture_count >= 4, 10 | +1 each |
| Layout density | table_count >= 1, 3 | +1 each |
| Structural fragmentation | fragmented_row_count >= 5, 9 | +1 each |
| Structural fragmentation | horizontal_band_count >= 3, 4 | +1 each |
| Structural fragmentation | upward_reset_ratio >= 0.12 | +1 |
Table 11: Deterministic page-level heuristic used to score page difficulty for the appendix Easy / Hard analysis. Pages with scores are reported as Easy ; all higher-scoring pages are reported as Hard .
Figure 13: Two contrasting predictions where GriTS and TEDS disagree with the table’s downstream usefulness. (a) A semantically wrong prediction that GriTS and TEDS round to . (b) A semantically equivalent rewrite that GriTS and TEDS heavily penalize. TRM ranks both correctly.
| Labels | Value | Relative tolerance | Accepted range |
| Below upper secondary, Sweden, Adjusted | 10 | 0.1 | |
| Below upper secondary, Sweden, Unadjusted | 3 | 0.5 |
Table 12: Two representative test rules for the OECD chart example (Sweden / Below upper secondary). The wide 0.5 relative tolerance on the Unadjusted rule reflects that the bar is very short ( 3 score points) and difficult to read precisely from the 3D chart; any estimate between 1.5 and 4.5 is accepted.
| Provider | Passed | Primary failure mode |
| LlamaParse Agentic | 8/10 | Minor label association gaps |
| Gemini 3 Flash (high thinking) | 3/10 | Labels not associated with values |
| Reducto | 3/10 | Incorrect values |
| GPT-5 Mini (minimal reasoning) | 0/10 | Incorrect values |
| Gemini 3 Flash (minimal thinking) | 0/10 | No table extracted |
Table 13: Per-provider results on the OECD chart example (10 test rules).
Figure 14: Source chart and provider outputs for the OECD chart example. (a) Source chart with Sweden / Below upper secondary marked in red. (b)–(e) Rendered table outputs (top portion shown); Sweden row highlighted in green (pass) or pink (fail). Gemini 3 Flash with minimal thinking omitted (no table, 0/10).
| Provider | Content Faithfulness | Text Correctness | Order |
| LlamaParse Agentic | 0.966 | 0.949 | 1.000 |
| Textract | 0.780 | 0.720 | 0.901 |
| Haiku 4.5 | 0.659 | 0.876 | 0.225 |
| Extend | 0.577 | 0.648 | 0.437 |
Table 14: Per-provider Content Faithfulness scores on the Federal Register example. Each competitor exhibits a distinct failure mode: Textract duplicates content, Haiku 4.5 hallucinates text and jumbles column order, and Extend reads across columns row-wise.
Figure 15: Source page and provider outputs for the Federal Register example. (a) Source 3-column page. (b)–(e) Rendered outputs. LlamaParse Agentic correctly linearizes columns; Textract wraps them into an HTML table; Extend reads row-wise across columns; Haiku 4.5 hallucinates words and jumbles column order.
| Provider | Semantic Formatting | Text Styling | Title Accuracy |
| LlamaParse Agentic | 1.000 | 1.000 | 1.000 |
| GPT-5 Mini | 0.829 | 1.000 | 0.658 |
| Haiku 4.5 | 0.174 | 0.000 | 0.348 |
| Textract | 0.000 | 0.000 | 0.000 |
Table 15: Per-provider Semantic Formatting scores on the infographic example. GPT-5 Mini preserves bold styling but flattens the heading hierarchy; Haiku 4.5 uses HTML tags instead of Markdown; Textract produces plain text with no formatting at all.
Figure 16: Source page and provider outputs for the infographic example. (a) Source page with 3-level heading hierarchy. (b)–(e) Rendered outputs. LlamaParse Agentic preserves hierarchy and bold titles; GPT-5 Mini flattens headings; Haiku 4.5 mixes HTML with Markdown; Textract produces plain text.
| Provider | Element Pass Rate | Localization | Classification | Attribution |
| LlamaParse Agentic | 86.4% | 93.2% | 86.4% | 100.0% |
| Gemini 3 Flash | 43.2% | 45.5% | 45.5% | 62.1% |
| LandingAI | 2.3% | 4.5% | 2.3% | 3.4% |
| Haiku 4.5 | 0.0% | 0.0% | 0.0% | 0.0% |
Table 16: Per-provider Element Pass Rate and sub-metric scores on the Sappi annual report page (44 ground truth elements). The element pass rate requires all three sub-checks to pass simultaneously. Gemini 3 Flash detects some elements but misses over half; LandingAI and Haiku 4.5 produce only 6 predictions each and fail almost entirely.
Figure 17: Ground truth and predicted layout overlays for the Sappi annual report page. (a) Ground truth with 44 elements across Picture , Text , Section , and Page-footer classes. (b)–(e) Provider predictions. LlamaParse Agentic produces 48 predictions closely matching GT; Gemini detects only 21; LandingAI and Haiku 4.5 produce only 6 each.