Beyond String Matching: Semantic Evaluation of PDF Table Extraction

Pius Horn; Janis Keuper

cs.CVMar 19, 2026

Beyond String Matching: Semantic Evaluation of PDF Table Extraction

Authors: Pius Horn, Janis Keuper

Organizations: Institute for Machine Learning and Analytics (IMLA), Offenburg University, Offenburg, Germany · University of Mannheim, Mannheim, Germany

Abstract

Reliably extracting tables from PDFs is essential for large-scale scientific data mining and knowledge base construction, yet existing evaluation approaches rely on rule-based metrics that fail to capture semantic equivalence of table content. We present a benchmarking framework based on synthetically generated PDFs with precise LaTeX ground truth, using tables sourced from arXiv to ensure realistic complexity and diversity. As our central methodological contribution, we apply LLM-as-a-judge for semantic table evaluation, integrated into a matching pipeline that accommodates inconsistencies in parser outputs. Through a human validation study comprising over 1,500 quality judgments on extracted table pairs, we show that LLM-based evaluation achieves substantially higher correlation with human judgment (Pearson r=0.93) compared to currently used Tree Edit Distance-based Similarity (TEDS, r=0.68) and Grid Table Similarity (GriTS, r=0.70). Evaluating 21 contemporary PDF parsers across 100 synthetic documents containing 451 tables reveals significant performance disparities. Our results offer practical guidance for selecting parsers for tabular data extraction and establish a reproducible, scalable evaluation methodology for this critical task. Code and data: https://github.com/phorn1/pdf-parse-bench Metric study and human evaluation: https://github.com/phorn1/table-metric-study

Figures & tables

Panel of Figure 1 — Figure 1: Rule-based metrics both over-reward semantically broken outputs and penalize benign representational variation (a); an LLM-as-a-judge approach validated on 1,554 human ratings aligns substantially better with human judgment (Pearson $r{=}0.93$ vs. $r{=}0.68$ for TEDS) (b); and used to benchmark 21 parsers on 100 synthetic PDFs that embed 451 real arXiv tables with their LaTeX source as exact ground truth (c).

Figure 2: Overview of the benchmark generation pipeline: arXiv L a T e X tables are collected, cleaned, and validated, then placed alongside filler text under a sampled layout, yielding a PDF page paired with a JSON ground truth.

Metric	Type	Pearson $r$	Spearman $\rho$	Kendall $\tau$	Cost ($)
TEDS	Rule-based	0.684	0.717	0.558	—
$\mathrm{GriTS}_{\mathrm{Top}}$	Rule-based	0.633	0.735	0.597	—
$\mathrm{GriTS}_{\mathrm{Con}}$	Rule-based	0.701	0.745	0.598	—
GriTS-Avg	Rule-based	0.698	0.765	0.606	—
SCORE Index	Rule-based	0.558	0.684	0.561	—
SCORE Content	Rule-based	0.642	0.657	0.524	—

Table 1: Correlation of automated metrics with averaged human scores (

n=518

pairs, three evaluators each). LLM costs: OpenRouter API per 1,000 pairs as of April 2026.

Figure 5: Scatter plots comparing automated metrics with human scores. Left column: rule-based metrics (TEDS, GriTS-Avg, SCORE-Avg); right column: LLM judges spanning the score range (DeepSeek-v3.2, Gemma-4-31b-it, Claude Opus 4.6). Bubble size indicates point count.

Model	Tuned	Tuned (no CoT)	Naive
Claude Opus 4.6	0.939	0.933 ( $-$ 0.006)	0.891 ( $-$ 0.048)
Gemma-4-31b-it	0.929	0.916 ( $-$ 0.013)	0.907 ( $-$ 0.022)
Gemini 3 Flash Preview	0.924	0.907 ( $-$ 0.017)	0.851 ( $-$ 0.073)
Gemma-4-26b-a4b-it	0.909	0.777 ( $-$ 0.132)	0.856 ( $-$ 0.053)
Gemini 3.1 Flash Lite Preview	0.909	0.863 ( $-$ 0.046)	0.807 ( $-$ 0.102)
GPT-5.4 nano	0.809	0.795 ( $-$ 0.014)	0.718 ( $-$ 0.091)

Table 2: Pearson correlation with averaged human scores across three prompt variants on the same 518 pairs. Parenthetical values show

\Delta r

relative to the Tuned variant.

	LLM Score (0–10)
Parser	Overall	Simple	Moderate	Complex	TEDS	Inference	Cost / Time
Gemini 3 Pro	9.55	9.58	9.57	9.49	0.85	API	$10.00
Gemini 3 Flash	9.50	9.53	9.38	9.61	0.85	API	$0.57
LightOnOCR-2-1B	9.08	9.41	8.90	8.91	0.83	GPU	30 min
Mistral OCR 3	8.89	8.92	8.69	9.07	0.88	API	$0.20
dots.ocr	8.73	9.01	8.43	8.76	0.81	GPU	20 min

Table 3: Table extraction performance across 451 tables from 100 synthetic pages, scored 0–10 by Gemma-4-31b-it and broken down by structural complexity, with TEDS scores (0–1 scale) for comparison. Parsers are ranked by overall score.

Figure 6: Per-parser score distributions across 451 tables. Each subplot shows the percentage of tables receiving each integer score (0–10); the dashed line marks the mean. Parsers are ordered by mean score (top-left to bottom-right).

Document Parsing Benchmarks

See all 15 papers

Tabular PDF Information Extraction with Local LLMs and Layout-Aware Parsing: A Reliability Evaluation
Mar 9, 2026Muhammad Anis Al Hilmi, Neelansh Khare, Noel Framil Iglesias +3Document Information Extraction Document Parsing
Informatics, Faculty of Engineering, Universitas Swadaya Gunung Jati, Cirebon, Indonesia · Computer Science, University of California, Irvine, USA · Computer Engineering, UNIR, La Rioja, Spain +1
PulseBench-Tab: A Multilingual Benchmark for Table Extraction with Graph-Based Evaluation
Apr 21, 2026Ritvik Pandey, Sid Manchkanti, Mohammed Wazir Adain +2Table Structure Recognition Document Information Extraction
1Pulse AI · 2Georgia Institute of Technology · 3S&P Global, Enterprise Data Organization
From Diagnosis to Correction: Benchmarking and Improving Real-World Table Parsing
Aug 10, 2026Jutao Xiao, Yuan Qu, Dongsheng Ma +7Table Structure Recognition Document Image Understanding
Zhejiang University · Shanghai Artificial Intelligence Laboratory · 1Zhejiang University, 2Shanghai Artificial Intelligence Laboratory +3

Mar 9, 2026J/K move · Enter open · S save

Tabular PDF Information Extraction with Local LLMs and Layout-Aware Parsing: A Reliability Evaluation

Muhammad Anis Al Hilmi, Neelansh Khare, Noel Framil Iglesias, Kurnia Adi Cahyanto+2

Informatics, Faculty of Engineering, Universitas Swadaya Gunung Jati, Cirebon, Indonesia · Computer Science, University of California, Irvine, USA · Computer Engineering, UNIR, La Rioja, Spain+1

Document Information Extraction Document Parsing LLM Information Extraction LLM Reliability

Open paper AskPDF arXiv

Extracting structured information from academic PDF documents is non trivial: a single page typically combines free text metadata with tabular regions, exhibits cross program variation, and is susceptible to Unicode encoding artifacts that interfere with downstream parsing. This study evaluates the reliability of information extraction approaches for tabular PDF documents, using academic course registration documents (Kartu Rencana Studi or KRS) from Indonesian higher education as a case study. Three strategies are compared: LLM only, Hybrid Deterministic - LLM (regex & LLM), and a Camelot based pipeline with LLM fallback. Experiments were conducted on 140 documents for the LLM based test and 860 documents for the Camelot based pipeline evaluation, covering four study programs with varying data in tables and metadata. Three 12 - 14B LLM models (Gemma 3, Phi 4, and Qwen 2.5) were run locally using Ollama and a consumer grade CPU without a GPU. Evaluations used exact match (EM) and Levenshtein similarity (LS) metrics with a threshold of 0.7. Although not applicable to all models, the results show that the hybrid approach can improve efficiency compared to LLM only, especially for deterministic metadata. The Camelot based pipeline with LLM fallback produced the best combination of accuracy (EM and LS up to 0.99 - 1.00) and computational efficiency (less than 1 second per PDF in most cases). The Qwen 2.5:14b model demonstrated the most consistent performance across all scenarios. These findings confirm that integrating deterministic and LLM based methods is a reliable and efficient strategy for information extraction from tabular text based PDF documents in computationally constrained environments.

Abstract

Document Parsing Benchmarks

Explore similar work

Tabular PDF Information Extraction with Local LLMs and Layout-Aware Parsing: A Reliability Evaluation

PulseBench-Tab: A Multilingual Benchmark for Table Extraction with Graph-Based Evaluation

From Diagnosis to Correction: Benchmarking and Improving Real-World Table Parsing