RankShift: Evaluating Ranking Instability Across Benchmark Dimensions in Document Parsing

Halimat Afolabi; Daeyoung Lee

cs.AINov 7, 2026

RankShift: Evaluating Ranking Instability Across Benchmark Dimensions in Document Parsing

Authors: Halimat Afolabi, Daeyoung Lee

Organizations: Stratified Precision · University College London

Abstract

Document parsing benchmarks are often treated as stable measures of model quality, yet rankings may depend substantially on benchmark design choices. In this work, we present RankShift, a systematic study comparing OCR and vision-language models (VLMs) under controlled variation in dataset composition, matching strategy, and scoring methodology. Using OmniDocBench as a case study, we evaluate the ranking stability of 16 models across 3 dataset configurations, 6 matching strategies, and 3 scoring metrics. We quantify rank stability using pairwise rank correlation, top-k churn, and pairwise reversal rate. Our results show that the interaction between matching strategy and ground truth format is a substantial but understudied driver of instability: scoring against OmniDocBench's JSON annotations versus its Markdown rendering yields a mean Kendall's tau of approximately 0.33 under region-level matching strategies, with individual pairs as low as tau = 0.23, compared with tau = 0.77 when each page is scored as a single concatenated string. Dataset composition produces comparable instability across page-type strata (mean tau = 0.48) and layout-type strata (mean tau = 0.49), while the cross-dataset shift from OmniDocBench to Real5's Scanning reconstruction of the same pages is more stable (tau = 0.80). Metric choice is the most stable dimension, with mean tau of approximately 0.78 across NED, BLEU, and METEOR. These findings indicate that leaderboard position is not solely a property of model capability, motivating leaderboards that report not only headline scores but also the robustness of rankings across reasonable design choices.

Figures & tables

Figure 1: Evaluation dataset, alignment strategy and scoring metric are varied independently, and the resulting leaderboards compared using Kendall’s $\tau$ , top- $k$ churn, and pairwise reversals.

Model	Type	Params	Open	Doc-Specialised	Output
Docling	P	–	$\checkmark$	$\checkmark$	Markdown
MinerU2.5	S-VLM	1.2B	$\checkmark$	$\checkmark$	Markdown
Chandra 2	S-VLM	5B	$\checkmark$	$\checkmark$	Markdown
DeepSeek-OCR 2	S-VLM	3B	$\checkmark$	$\checkmark$	Markdown
Dolphin 1.5	S-VLM	0.5B	$\checkmark$	$\checkmark$	Markdown
dots.ocr	S-VLM	1.7B	$\checkmark$	$\checkmark$	Markdown

Table 1: Models evaluated in RankShift. Type: P = Pipeline, S-VLM = Specialist VLM, G-VLM = General-Purpose VLM. Output format indicates the primary structured output produced.

Dimension	Comparison	Kendall’s $\tau$ [95% CI]	Category
Alignment	quick_match vs simple_match (E2E)	0.767 [0.575, 0.929]	E2E vs E2E
	quick_match vs no_split (E2E)	0.550 [0.229, 0.821]	E2E vs E2E
	simple_match vs no_split (E2E)	0.583 [0.256, 0.860]	E2E vs E2E
	quick_match vs no_split (M2M)	0.817 [0.593, 0.965]	M2M vs M2M
	simple_match vs no_split (M2M)	0.683 [0.449, 0.892]	M2M vs M2M
	quick_match vs simple_match (M2M)	0.667 [0.368, 0.911]	M2M vs M2M

Table 2: Pairwise Kendall’s

\tau

between model rankings (mean page-level scores). 95% CIs from bootstrap resampling of models (

n=16

B=2000

replicates.

Figure 2: Experiment 2 bump chart showing rank trajectories under alignment variation.

Figure 3: Rank-displacement heatmap for Experiment 1 across layout strata. Each cell shows how many leaderboard positions a model gains when ranked on B-pages instead of A-pages (rank A $-$ rank B). $+12$ (blue) means the model moved 12 places toward #1; $-3$ (red) means it fell 3 places.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Variation	Configurations	Comparisons	Evidence relationship
Page type	PPT2PDF; academic literature; book; colorful textbook; exam paper; magazine; newspaper; note; research report	All pairwise ( $36$ )	Disjoint page subsets
Layout	1 $+$ column; double column; other layout; single column; three column	All pairwise ( $10$ )	Disjoint page subsets
Capture condition	OmniDocBench v1.5 digital pages; Real5 Scanning reconstructions	Digital vs. scanned ( $1$ )	Paired versions of the same 1,355 pages

Appendix

Table A1 : Dataset configurations used in Experiment 1. Partition-based comparisons use disjoint page subsets, while the capture-condition comparison pairs two versions of the same pages.

Model	OmniDocBench rank	Real5 rank	$\Delta$ Real5 position
dots.ocr	1	6	$-5$
GLMOCR	2	2	0
PaddleOCR-VL-1.5	3	1	$+2$
Youtu-Parsing	4	4	0
MinerU2.5	5	3	$+2$
DeepSeek-OCR 2	6	5	$+1$

Appendix

Table A2 : Experiment 1 model rankings on OmniDocBench v1.5 vs. Real5 Scanning (

n=16

models, E2E quick_match text NED).

\Delta={}

OmniDocBench rank

-

Real5 rank (1 = best); negative

\Delta

means a worse rank under Real5.

Comparison	Rate	95% CI	$q$ -value
Cross-paradigm (E2E $\leftrightarrow$ M2M)
E2E quick_match vs. M2M simple_match	35.8%	[30.0%, 39.2%]	$4.8\times 10^{-6}$ *
E2E simple_match vs. M2M simple_match	35.0%	[27.5%, 36.7%]	$6.8\times 10^{-6}$ *
E2E simple_match vs. M2M quick_match	30.0%	[23.3%, 32.5%]	0.001 *
E2E simple_match vs. M2M no_split	29.2%	[21.7%, 31.7%]	0.002 *
E2E quick_match vs. M2M no_split	28.3%	[21.7%, 31.7%]	0.003 *

Appendix

Table A3 : Alignment-pair rank reversal rates (

n=16

models, 120 model pairs per alignment pair). One-sided binomial tests vs. within-E2E baseline (

\sim 16.7\%

);

q

-values from Benjamini–Hochberg FDR over 15 alignment pairs (

\alpha=0.05

). 95% CIs from page bootstrap. An asterisk indicates significance after FDR correction (

q<0.05

Figure B1 : Cross-dataset rank displacement from OmniDocBench v1.5 to Real5 Scanning in Experiment 1. Values report OmniDocBench rank minus Real5 rank: positive (blue) values indicate movement toward rank 1 on Real5, negative (red) values indicate a lower rank on Real5, and zero indicates no change.

Figure B2 : Pairwise reversal-rate heatmap for Experiment 1. Each cell gives the share of the 120 model pairs whose ordering reverses between two of the nine OmniDocBench page-type strata (36 comparisons).

Figure B3 : Full rank heatmap for Experiment 1: each cell represents a model’s rank within each of the nine OmniDocBench page-type strata, where 1 is the best-scoring model on that page type and 16 the worst.

Figure B4 : Full rank heatmap for Experiment 1: each cell gives a model’s rank within one of the five OmniDocBench layout strata, where 1 is the best-scoring model on that layout type and 16 the worst.

Figure B5 : Top-1 churn across Experiment 1 comparison settings. Each cell indicates whether the highest-scoring model differs between two page-type strata: 1.00 for a different top-ranked model, 0.00 for the same.

Figure B6 : Top-3 churn across Experiment 1 comparison settings. Each cell gives the proportion of the top-three set that changes between two page-type strata, from 1.00 where none of the three models are retained to 0.00 where all top-three models are the same.

Figure B7 : Top-5 churn across Experiment 1 comparison settings. Each cell gives the proportion of the top-five set that changes between two page-type strata, ranging from 0.20, where one of the five models differs, to 0.80, where four differ.

Figure B8 : Cross-dataset rank scatter for Experiment 1, comparing model ranks on OmniDocBench v1.5 with the Real5 Scanning subset.

Figure B9 : Focused bump chart comparing RolmOCR and GLMOCR across the six alignment settings in Experiment 2. GLMOCR ranks higher under every E2E condition and RolmOCR under every M2M condition, reversing their relative order.

Figure B10 : Alignment signatures used in the Experiment 2 diagnostic procedure: the 20 most frequent patterns, covering 78.5% of complete page $\times$ model rows. Each row is one signature, with columns in the fixed order E2E quick_match ( $E_{q}$ ), simple_match ( $E_{s}$ ) and no_split ( $E_{n}$ ), followed by the M2M counterparts $M_{q}$ , $M_{s}$ and $M_{n}$ .

Figure B11 : Experiment 3 bump chart showing each model’s rank trajectory across the three scoring metrics: NED, BLEU and METEOR. Rankings are largely preserved, though the top-ranked model changes, with dots.ocr leading under NED and Youtu-Parsing under both BLEU and METEOR.

Figure B12 : Heatmap of pairwise rank agreement across the three Experiment 3 scoring metrics. All three pairs show high agreement, with Kendall’s $\tau$ between 0.700 and 0.867.

Observed pattern	Likely cause	What to check	Example signatures	Freq.	Recommendation
E2E high, M2M lower	Excess content (hallucinations, captions, duplication)	Pred. length vs. GT; extra spans	HHMMMM , HHHMMM , HHMHMM , HMHHMM , HMMMMM , HHLLLL , HHMMML	$\sim 12\%$	Report both protocols; inspect extra output when E2E $\gg$ M2M
E2E high, M2M low/medium, heavy syntax	Formatting overhead (Markdown/HTML penalised byte-wise)	Headers/HTML; stripped vs. full Markdown	HHHHHM , HHHHMH , HMHHLM , HHHHLH , HLHHLH , HHMHLM	$\sim 13\%$	Pair region-based scores with M2M; report syntax-stripped variants

Appendix

Table C1 : Diagnostic families where E2E

>

M2M (OmniDocBench v1.5, 16 models;

N=20{,}610

page

\times

model rows). Frequencies are the combined share of listed signatures.

Observed pattern	Likely cause	What to check	Example signatures	Freq.	Recommendation
E2E low, M2M high (math)	Equation/block mis-pair under E2E	Equation blocks in GT	LLLHHH , LLLHMM , LLMHMM	$\sim 0.02\%$	Stratify math-heavy pages; audit block matching
E2E low, M2M high (layout)	Column/reading-order break under concatenation	Block order vs. M2M concatenation	MMMHHH , MMMHMH , MMLHMH , HLMHLH , LLLMML , 00LLLL	$\sim 0.8\%$	Report M2M on layout-heavy pages
E2E title/box slot empty, M2M body correct	Region-slot failure (title dropped)	GT title region vs. E2E match	00MHHH , 00HHHH , 00MHHM	$\sim 0.03\%$	Inspect region-level matching, not page OCR quality

Appendix

Table C2 : Diagnostic families where M2M

>

E2E (same corpus; rare on OmniDocBench v1.5).

Document Parsing Benchmarks

See all 18 papers

May 8, 2026

How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings

Document parsing models are improving rapidly, but existing benchmarks are still insufficient for a comprehensive evaluation of their capabilities. We introduce PureDocBench, a benchmark with 10 domains, 66 subcategories, and 1,475 pages. Each page has matched Clean, Digital, and Real versions, covering clean renders, simulated degradation, and real capture or sharing conditions. The resulting 4,425 images have source-derived annotations for text, formulas, tables, and reading order, checked through automated validation and human review. Keeping page content and annotations fixed allows direct comparisons across image conditions. We evaluate 58 models, including pipeline-based parsers, end-to-end parsers, and general vision-language models. The best three-track average score is 79.69 out of 100, and the mean across models is 63.84. Scores drop by 3.34 points on Digital and 11.61 points on Real on average, relative to Clean. TeleOCR leads on Clean, GLM-5.3-Flash on Digital, and Gemini 3.6 Flash on Real. The benchmark dataset, source documents, generation tools, and evaluation code are publicly available at https://github.com/zhihengli-casia/PureDocBench.

Zhiheng Li, Zongyang Ma, Jiaxian Chen +12

CASIA · UCAS · Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information +6

Document Parsing Benchmarks Document Parsing Document Image Understanding

Ask PDF

Jul 15, 2026

OvisOCR2 Technical Report

We introduce OvisOCR2, a 0.8B document parsing model. OvisOCR2 is designed as an end-to-end parser: given a document page image, it generates a Markdown representation in natural reading order, covering text, formulas, tables, and visual regions. We build a data engine that combines filtered real-document annotations with synthetic pages whose rendered images and Markdown targets are derived from the same HTML source. The training recipe includes supervised fine-tuning, reinforcement learning on a 4B branch with a multi-component reward design, on-policy distillation into the 0.8B model, and model fusion. On OmniDocBench v1.6, OvisOCR2 achieves a state-of-the-art overall score of 96.58, placing an end-to-end model at the top of this leaderboard previously dominated by pipeline methods and highlighting the potential of end-to-end document parsing. On PureDocBench, OvisOCR2 also achieves the highest Avg3 score of 75.06. Beyond these two public benchmarks, we evaluate OvisOCR2 on an in-house benchmark designed to cover a broader set of long-tail and challenging scenarios. OvisOCR2 obtains the best overall performance among the compared methods, providing further evidence of its generalization and robustness. OvisOCR2 is available at https://huggingface.co/ATH-MaaS/OvisOCR2.

Shiyin Lu, Yinglun Li, Yu Xia +10

ATH-MaaS, Alibaba Group

Document Parsing Optical Character Recognition Document Image Understanding

Ask PDF

May 31, 2026

Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing

Document parsing and recognition are fundamental capabilities for vision-language models (VLMs) and document processing systems. However, existing Optical Character Recognition (OCR) and document parsing benchmarks are increasingly limited in coverage and difficulty: many focus on common document genres or uniformly sampled pages where modern parsers already perform strongly, while offering limited annotation for expert-domain structures such as chemical formula, music notation, complex tables, and cross-page layouts. We introduce Dr. DocBench, a difficulty-aware benchmark for expert-level document parsing. Built from a large-scale multilingual book corpus, Dr. DocBench spans 52 BISAC subject domains and selects challenging documents through parser-failure-based sampling, targeting cases where multiple state-of-the-art systems struggle. It contains 4,514 annotated pages from long documents averaging around 100 pages, with 65k high-quality page- and block-level annotations for layout, reading order, hierarchical relations, and domain-specific visual contents. Evaluations of pipeline-based parsers and general-purpose VLMs show that strong performance on existing benchmarks does not transfer to our expert-level document parsing. Our analysis reveals substantial failures across subjects, content types, and structural attributes, highlighting Dr. DocBench as a comprehensive testbed for diagnosing and advancing document intelligence.

Minglai Yang, Xinyan Velocity Yu, Pengyuan Li +22

2077AI · University of Arizona · University of Southern California +8

VLM Evaluation Document Parsing Benchmarks Document Parsing Document Layout Analysis Document Image Understanding

Ask PDF

Abstract

Document Parsing Benchmarks

Explore similar work

How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings

OvisOCR2 Technical Report

Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing