Document Parsing Benchmarks

Follow this topic Ask about this topic

Latest papers

18 papers4 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

RankShift: Evaluating Ranking Instability Across Benchmark Dimensions in Document Parsing
Nov 7, 2026Halimat Afolabi, Daeyoung LeeDocument Parsing Benchmarks VLM Evaluation
Stratified Precision · University College London
AmbiHTMLBench: A Benchmark for HTML-based Visually Rich Document parsing
Nov 4, 2026Boyu Niu, Yingjie Xiong, Shurui Li +8Document Parsing Benchmarks Document Parsing
Shanghai Jiao Tong University · Theseus Labs · Xiaohongshu
TreasuryQA Bench: Can AI Agents Work an Archive of Scanned Financial Records?
Oct 28, 2026Ritiz Tambi, Sahar Shayegan, Joshua Meyer +5AI Agent Benchmarks Document QA
Veris AI
Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation
Oct 14, 2026Yuheng Chen, Yufan Chen, Zhuojun Cai +8Document Parsing Benchmarks Document Parsing
Karlsruhe Institute of Technology · University of Alberta · Hunan University
WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans
Aug 24, 2026Jun Zhang, Qiao Zhao, Cheng Cui +6Document Understanding Document Parsing Benchmarks
Baidu Inc.
Code as Representation: A Compilable Parsing Paradigm for Academic Documents
Aug 18, 2026Rihui Jin, Jun Wang, chengyuan zhu +11Document Parsing Document Parsing Benchmarks
Southeast University School of Computer Science and Engineering Nanjing, Jiangsu, China · Nanyang Technological University Singapore, Singapore · Nanjing University State Key Laboratory for Novel Software Technology Nanjing, Jiangsu, China +1
LongDocBench: Benchmarking TOC Hierarchy and Contextual Relationship Recovery in Long Documents
Aug 15, 2026Yuefeng Zou, Yichen Lu, Jingxiao Yang +5Document Parsing Benchmarks Document Parsing
Ant Group · Zhejiang University
MORE: A Multilingual Document Parsing Benchmark and Evaluation
Jul 3, 2026Long Xu, Binghong Wu, Tinghao Yu +6Document Parsing Benchmarks Document Parsing
Tencent, Shenzhen, China
RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents
Jun 5, 2026Ameya Joshi, Joon Kim, Gus Eggert +5Document Parsing Benchmarks Benchmark Design
Extend AI
Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing
May 31, 2026Minglai Yang, Xinyan Velocity Yu, Pengyuan Li +22VLM Evaluation Document Parsing Benchmarks
2077AI · University of Arizona · University of Southern California +8
MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing
May 21, 2026Bangbang Zhou, Hangdi Xing, Yifan Chen +8Document Parsing Benchmarks Document Parsing
University of Science and Technology of China · Tongyi Lab, Alibaba Group
How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings
May 8, 2026Zhiheng Li, Zongyang Ma, Jiaxian Chen +12Document Parsing Benchmarks Document Parsing
CASIA · UCAS · Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information +6
ParseBench: A Document Parsing Benchmark for AI Agents
Apr 9, 2026Boyang Zhang, Sebastián G. Acosta, Preston Carlson +4Document Parsing Benchmarks Document Parsing
MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios
Mar 30, 2026Zhang Li, Zhibo Lin, Qiang Liu +7Document Parsing Benchmarks Document Image Understanding
Huazhong University of Science and Technology · Kingsoft Office
RenoBench: A Citation Parsing Benchmark
Mar 26, 2026Parth Sarin, Juan Pablo Alperin, Adam Buttrick +1Document Parsing Benchmarks Document Information Extraction
Graduate School of Education, Stanford University, Stanford, California, USA · Public Knowledge Project, Simon Fraser University, Vancouver, British Columbia, Canada · California Digital Library, University of California Office of the President, Oakland, California, USA +1
Beyond String Matching: Semantic Evaluation of PDF Table Extraction
Mar 19, 2026Pius Horn, Janis KeuperDocument Parsing Benchmarks Document Information Extraction
Institute for Machine Learning and Analytics (IMLA), Offenburg University, Offenburg, Germany · University of Mannheim, Mannheim, Germany
Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild
Mar 4, 2026Changda Zhou, Ziyue Gao, Xueqing Wang +4VLM Evaluation Document Parsing Benchmarks
PaddlePaddle Team, Baidu Inc. · The Hong Kong University of Science and Technology (Guangzhou)
LLMStructBench: Benchmarking Large Language Model Structured Data Extraction
Feb 16, 2026Sönke Tenckhoff, Mario Koddenbrock, Erik RodnerLLM Information Extraction Document Parsing Benchmarks
KI-Werkstatt / FB2, University of Applied Sciences Berlin

Nov 7, 2026J/K move · Enter open · S save

RankShift: Evaluating Ranking Instability Across Benchmark Dimensions in Document Parsing

Halimat Afolabi, Daeyoung Lee

Stratified Precision · University College London

Document Parsing Benchmarks VLM Evaluation Optical Character Recognition

Open paper AskPDF arXiv

Document parsing benchmarks are often treated as stable measures of model quality, yet rankings may depend substantially on benchmark design choices. In this work, we present RankShift, a systematic study comparing OCR and vision-language models (VLMs) under controlled variation in dataset composition, matching strategy, and scoring methodology. Using OmniDocBench as a case study, we evaluate the ranking stability of 16 models across 3 dataset configurations, 6 matching strategies, and 3 scoring metrics. We quantify rank stability using pairwise rank correlation, top-k churn, and pairwise reversal rate. Our results show that the interaction between matching strategy and ground truth format is a substantial but understudied driver of instability: scoring against OmniDocBench's JSON annotations versus its Markdown rendering yields a mean Kendall's tau of approximately 0.33 under region-level matching strategies, with individual pairs as low as tau = 0.23, compared with tau = 0.77 when each page is scored as a single concatenated string. Dataset composition produces comparable instability across page-type strata (mean tau = 0.48) and layout-type strata (mean tau = 0.49), while the cross-dataset shift from OmniDocBench to Real5's Scanning reconstruction of the same pages is more stable (tau = 0.80). Metric choice is the most stable dimension, with mean tau of approximately 0.78 across NED, BLEU, and METEOR. These findings indicate that leaderboard position is not solely a property of model capability, motivating leaderboards that report not only headline scores but also the robustness of rankings across reasonable design choices.