Document understanding systems are typically evaluated as separate components via layout analysis, optical text recognition, reading-order prediction, and image captioning. This separation leaves scientific figures, captions, and visual semantics weakly connected, and fails to measure whether a model can produce a single page-level parse that is both structurally faithful and semantically accessible. We introduce Read-Parse-Describe, a unified dataset for document parsing with visual element description generation. Given a page image, models are required to output an ordered list of layout elements containing bounding boxes, semantic labels, reading-order indices, recognized textual content, and alternative text descriptions for visual elements. The benchmark is built from 2,402 documents, comprising 18,163 continuous pages, 125,218 layout elements, and 11,338 visual instances. Besides, we further propose a plug-and-play space-awareness enhancer that reuses multi-layer visual features and injects grid-aware spatial guidance into vision-language tokens for layout-aware generation. Experiments across end-to-end VLMs and pipeline parsers show that existing systems still struggle to jointly recover reading order and visual semantics, while our enhanced model achieves strong joint performance. These results highlight the need to move beyond isolated document parsing modules toward unified models that jointly read, parse, and describe complex documents. The new dataset and code will be made publicly available at https://github.com/yufanchen96/RPD
Figures & tables
(a) Unified document parsing with different document elements.
(b) Domain distribution.
Dataset
Docs
Layout
Reading Order
Textual Elements
Visual Elements
Caption Links
Alternative Text
Structured Output
PubLayNet ( Zhong et al., 2019 )
PDF
✓
–
–
✓
–
–
–
DocLayNet ( Pfitzmann et al., 2022 )
Mixed
✓
–
–
✓
–
–
–
DocBank ( Li et al., 2020 )
PDF
✓
–
–
✓
–
–
–
M6Doc ( Cheng et al., 2023 )
Mixed
✓
–
–
✓
–
–
–
D4LA ( Da et al., 2023 )
Mixed
✓
–
–
✓
–
–
–
SciTSR ( Chi et al., 2019 )
Images
–
–
✓
–
–
–
–
Table 1: Comparison of dataset target contracts. A checkmark indicates that the benchmark explicitly requires the corresponding field as part of its evaluation target. Caption Links refer to explicit associations between visual objects and their captions. Structured Output denotes a single page-level prediction that jointly contains layout, order, textual content, and caption links.
Score
Generated
Refined
Score
Generated
Refined
1
0
0
4
2,419
226
2
146
0
5
7,511
11,112
3
1,262
0
Score-5 rate
66.2%
98.1%
Table 4
Figure 1: Document parsing model with SARA adapter, which is a plug-and-play spatial enhancer that leverages multi-layer visual features. Features from encoder layers are concatenated, followed by a grid-aware multi-crop alignment mechanism that preserves spatial structure. The aligned features serve as spatial guidance and are injected into the original tokens via a gated residual connection.
Model
Trainable Params
DLA
OCR Acc. ↑
RO Score ↑
Alt. Text
F1 ↑
mIoU ↑
METEOR ↑
BERT ↑
Zero-shot Models
GOT-OCR 2.0
--
--
--
0.8652
0.5330
--
--
Nougat
--
--
--
0.7327
0.4640
--
--
Dolphin-v2
--
0.9124
0.9265
0.9307
0.8221
--
--
MonkeyOCR
--
0.6030
0.8949
0.9684
0.6254
--
--
Table 2: Main test-set results under the unified Read-Parse-Describe output schema. Rows are grouped by evaluation setting to separate Zero-shot Models , Full Fine-Tuning , which means fine-tuning the whole model, and Adapter Tuning , which means only fine-tuning the adapter part. -- indicates that the corresponding field is not produced by the evaluated system. bold and underlined scores denote the best and second-best results for each metric, respectively.
Table 7
Figure 2: Qualitative comparison of annotation mapping results. All model-based methods use Qwen3.5-2B as the backbone. We compare the ground-truth annotations with outputs from the base model, full fine-tuning, and the proposed SARA adapter.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Description
Label
ID
Description
Label
1
Paper Title
sec_0
13
Table
tab
2
Level-1 Heading
sec_1
14
Figure-Caption
figure-cap
3
Level-2 Heading
sec_2
15
Table-Caption
table-cap
4
Level-3 Heading
sec_3
16
Annotation
anno
5
Level-4 Heading
sec_4
17
Formula
equ
6
Paragraph
para
18
Code Block
code
Appendix
Table 5: Semantic label taxonomy used in Read-Parsing-Describe.
Category
Count
Percentage
AI & ML
271
12.60%
Accessibility
255
11.85%
AR/VR/XR
235
10.93%
Design & HCI Methods
149
6.93%
Privacy & Social
103
4.79%
Health & Wellbeing
95
4.42%
Appendix
Table 6: Primary Domain Distribution of Analyzed Articles from 2020 - 2024
Element Category
Pages 1–8 (%)
Pages ≥9 (%)
Density (1–8)
Density ( ≥9 )
Title, Abstract & Metadata
100.00%
0.00%
1.2 / page
0.0 / page
Equations & Math Blocks
78.42%
21.58%
4.5 / page
1.2 / page
Figures & Captions
69.15%
30.85%
2.1 / page
0.9 / page
Body Paragraphs
62.30%
37.70%
12.4 / page
7.5 / page
Tables & Algorithms
54.80%
45.20%
0.8 / page
0.7 / page
References
8.25%
91.75%
0.5 / page
15.6 / page
Appendix
Table 7: Impact of the 8-Page Truncation Strategy on Layout Element Distribution. By comparing the element proportions in Pages 1–8 versus Pages ≥9 (estimated on a full-text subset), we observe that the truncation effectively preserves high-value multimodal objects while filtering out visually monotonous reference lists.
Category
Count
Percentage
Computer Science & AI
6,443
56.8%
Social Science
2,181
19.2%
Engineering
1,359
12.0%
Medicine & Healthcare
652
5.8%
Other
245
2.2%
Biology
110
1.0%
Appendix
Table 8: Image Categories Distribution
Image Type
Count
Percentage
Table
2758
24.3%
Diagrammatic Instruction / Concept
2086
18.4%
User Interface / Screenshot
1743
15.4%
Photographic Image
1529
13.5%
Flowchart / Block Diagram
1000
8.8%
Bar Chart / Histogram
942
8.3%
Appendix
Table 9: Image Type Distribution
Category
Label
Count
Percentage
Structural Headers
sec_1 , sec_2 , sec_0
44,617
35.64%
Academic Metadata
author-info , fnote
28,381
22.66%
Scholarly Objects
fig , tab , equ , code
12,336
9.85%
Captions and References
figure-cap , table-cap , reference
21,381
17.08%
Layout Fragments
half_para , list , icon
17,564
14.03%
Others
anno , sec_3 , sec_4 , watermark , etc.
939
0.74%
Appendix
Table 10: Distribution of high-value layout elements.
Configuration
Detailed Content / Implementation
Generation Prompt
“Describe this image in 1–3 concise sentences for accessibility. If the image contains multiple panels or subfigures, summarize them as a single combined figure rather than describing each panel separately. Focus only on the key visual information, such as the type of chart or image, the main subjects, and clearly visible labels or visual patterns. Do not use phrases like ’image of’ or ’picture of’. Provide a factual description based only on what is visible, without interpreting or explaining the underlying data.”
Generation Parameters
Temperature: 0.2 Top-p: 0.9 Max New Tokens: 256 Repetition Penalty: 1.05
Evaluation Role
System Prompt: You are a rigorous Image Accessibility Auditor specializing in scientific and technical content. Your goal is to determine if the provided ’Alt-text’ accurately represents the visual information for a visually impaired user. Evaluate strictly based on Accuracy, Completeness, and Constraint Adherence.
Evaluation Rubric
Score 1 (Failure - Misleading/Hallucination): Completely irrelevant, contains gross data hallucinations, or generates pure technical noise. Fails to identify the basic type of the image. Actively misleads the visually impaired user. Score 2 (Poor - Severe Omissions/Violations): Identifies the general chart/image type but suffers from severe omissions (e.g., missing critical axes, legends, or main subjects). Alternatively, it severely violates negative constraints (e.g., provides subjective interpretations of data instead of visual facts). Score 3 (Acceptable - Basic but Flawed): Correctly identifies the main subjects and chart type. However, it lacks specific critical details (e.g., distinct visual trends or key labels). May fail minor constraints (e.g., exceeds the 3-sentence limit, or includes filler phrases like "This is a picture of"). Score 4 (Good - Professional & Factual): Clear, objective, and accurate. Successfully captures the chart type, major variables, key labels, and significant visual patterns. Strictly adheres to negative constraints. May only miss minor, non-essential visual nuances or slightly lack extreme conciseness. Score 5 (Excellent - Perfect Accessibility): Highly concise (strictly 1-3 sentences), factual, and deeply informative. Seamlessly summarizes complex or multi-panel figures into a cohesive whole. Provides a visually impaired user with the exact informational equivalent of a quick visual scan, devoid of any over-interpretation or fluff.
Appendix
Table 11: Detailed Prompt Configurations and Expanded Evaluation Rubric for Alt-text Generation
category
LLM generated (Count)
LLM + human_modified (Count)
Score 1
0
0
Score 2
146
0
Score 3
1262
0
Score 4
2419
226
Score 5
7511
11112
Score 5
66.2%
98.1%
Appendix
Table 12: Alt text generation:LLM generated vs LLM+human_modified
Figure 3: Human-LLM agreement audit over 200 revised samples. Agreement is measured after low-score descriptions are refined and re-scored.
Data
Method
F1
mIoU
OCR
RO
METEOR
BERTScore
10%
Full FT
0.6487
0.7563
0.8476
0.7996
0.1256
0.8106
10%
SARA
0.5731
0.7144
0.7311
0.7805
0.1672
0.8002
30%
Full FT
0.6771
0.7861
0.8756
0.8504
0.2651
0.8288
30%
SARA
0.6567
0.8061
0.8087
0.8226
0.2871
0.8372
50%
Full FT
0.6991
0.8232
0.9362
0.9623
0.3072
0.8540
50%
SARA
0.8781
0.8533
0.9562
0.9623
0.3340
0.8533
Appendix
Table 13: Training-data scaling on Qwen3.5-2B. Fractions indicate the reported proportion of the training data.
Resolution
F1
mIoU
OCR
RO
100%
0.9189
0.8919
0.9636
0.9774
50%
0.8389
0.8673
0.9248
0.9534
25%
0.7589
0.8194
0.7648
0.8864
Appendix
Table 14: Resolution sensitivity of Qwen3.5-2B+SARA under the reported crop-resolution.
Figure 4: Representative failure cases of SARA. Top row: a borderless composite figure where intra-image text can cause a spatial shift between figure regions and their captions. Bottom row: a dense table where localization remains relatively accurate, but decoding may truncate long content and introduce minor hallucinated artifacts.
PDF documents contain critical visual elements such as figures, tables, and forms whose accurate extraction is essential for document understanding and multimodal retrieval-augmented generation (RAG). Existing PDF parsers often miss complex visuals, extract non-informative artifacts (e.g., watermarks, logos), produce fragmented elements, and fail to reliably associate captions with their corresponding elements, which degrades downstream retrieval and question answering. We present a lightweight and production level PDF parsing framework that can accurately detect visual elements and associates captions using a combination of spatial heuristics, layout analysis, and semantic similarity. On popular benchmark datasets and internal product data, the proposed solution achieves ≥96% visual element detection accuracy and 93% caption association accuracy. When used as a preprocessing step for multimodal RAG, it significantly outperforms state-of-the-art parsers and large vision-language models on both internal data and the MMDocRAG benchmark, while reducing latency by over 2×. We have deployed the proposed system in challenging production environment.
Efficient teamwork typically combines global coordination with parallel execution, a principle not yet fully reflected in unified Vision-Language Model (VLM)-based document parsers. Existing unified parsers process an entire page jointly but generate its output through a single token-by-token autoregressive trajectory, creating a sequential bottleneck that grows with document length. Such full-page sequential generation overlooks a key property of document parsing: layout must be analyzed globally, whereas block content can be parsed in parallel. Based on this observation, we introduce HPD-Parsing, which replaces full-page autoregressive generation with a Hierarchical Parallel Decoding paradigm. A main layout branch organizes the overall document structure and dynamically assigns block-level content decoding to concurrent branches, while progressive multi-token prediction (P-MTP) further reduces the decoding steps within each branch. Experiments on public benchmarks show that HPD-Parsing achieves 4,752 tokens per second, delivering 2.62× the throughput of the fastest existing document parsing model and 3.06× that of the vanilla autoregressive baseline, while maintaining competitive parsing accuracy. These results establish hierarchical parallel decoding as an effective alternative to full-page autoregressive generation, opening a new direction for efficient unified document parsing.
We introduce OvisOCR2, a 0.8B document parsing model. OvisOCR2 is designed as an end-to-end parser: given a document page image, it generates a Markdown representation in natural reading order, covering text, formulas, tables, and visual regions. We build a data engine that combines filtered real-document annotations with synthetic pages whose rendered images and Markdown targets are derived from the same HTML source. The training recipe includes supervised fine-tuning, reinforcement learning on a 4B branch with a multi-component reward design, on-policy distillation into the 0.8B model, and model fusion. On OmniDocBench v1.6, OvisOCR2 achieves a state-of-the-art overall score of 96.58, placing an end-to-end model at the top of this leaderboard previously dominated by pipeline methods and highlighting the potential of end-to-end document parsing. On PureDocBench, OvisOCR2 also achieves the highest Avg3 score of 75.06. Beyond these two public benchmarks, we evaluate OvisOCR2 on an in-house benchmark designed to cover a broader set of long-tail and challenging scenarios. OvisOCR2 obtains the best overall performance among the compared methods, providing further evidence of its generalization and robustness. OvisOCR2 is available at https://huggingface.co/ATH-MaaS/OvisOCR2.