cs.CVOct 14, 2026
SaveRead, Parse, Describe: Unified Document Parsing with Visual Element Description Generation
Organizations: Karlsruhe Institute of Technology · University of Alberta · Hunan University
Abstract
Document understanding systems are typically evaluated as separate components via layout analysis, optical text recognition, reading-order prediction, and image captioning. This separation leaves scientific figures, captions, and visual semantics weakly connected, and fails to measure whether a model can produce a single page-level parse that is both structurally faithful and semantically accessible. We introduce Read-Parse-Describe, a unified dataset for document parsing with visual element description generation. Given a page image, models are required to output an ordered list of layout elements containing bounding boxes, semantic labels, reading-order indices, recognized textual content, and alternative text descriptions for visual elements. The benchmark is built from 2,402 documents, comprising 18,163 continuous pages, 125,218 layout elements, and 11,338 visual instances. Besides, we further propose a plug-and-play space-awareness enhancer that reuses multi-layer visual features and injects grid-aware spatial guidance into vision-language tokens for layout-aware generation. Experiments across end-to-end VLMs and pipeline parsers show that existing systems still struggle to jointly recover reading order and visual semantics, while our enhanced model achieves strong joint performance. These results highlight the need to move beyond isolated document parsing modules toward unified models that jointly read, parse, and describe complex documents. The new dataset and code will be made publicly available at https://github.com/yufanchen96/RPD
Figures & tables
(a) Unified document parsing with different document elements.
(b) Domain distribution.
| Dataset | Docs | Layout | Reading Order | Textual Elements | Visual Elements | Caption Links | Alternative Text | Structured Output |
| PubLayNet ( Zhong et al., 2019 ) | ✓ | – | – | ✓ | – | – | – | |
| DocLayNet ( Pfitzmann et al., 2022 ) | Mixed | ✓ | – | – | ✓ | – | – | – |
| DocBank ( Li et al., 2020 ) | ✓ | – | – | ✓ | – | – | – | |
| M6Doc ( Cheng et al., 2023 ) | Mixed | ✓ | – | – | ✓ | – | – | – |
| D4LA ( Da et al., 2023 ) | Mixed | ✓ | – | – | ✓ | – | – | – |
| SciTSR ( Chi et al., 2019 ) | Images | – | – | ✓ | – | – | – | – |
Table 1: Comparison of dataset target contracts. A checkmark indicates that the benchmark explicitly requires the corresponding field as part of its evaluation target. Caption Links refer to explicit associations between visual objects and their captions. Structured Output denotes a single page-level prediction that jointly contains layout, order, textual content, and caption links.
| Score | Generated | Refined | Score | Generated | Refined |
| 1 | 0 | 0 | 4 | 2,419 | 226 |
| 2 | 146 | 0 | 5 | 7,511 | 11,112 |
| 3 | 1,262 | 0 | Score-5 rate | 66.2% | 98.1% |
Figure 1: Document parsing model with SARA adapter, which is a plug-and-play spatial enhancer that leverages multi-layer visual features. Features from encoder layers are concatenated, followed by a grid-aware multi-crop alignment mechanism that preserves spatial structure. The aligned features serve as spatial guidance and are injected into the original tokens via a gated residual connection.
| Model | Trainable Params | DLA | OCR Acc. | RO Score | Alt. Text | ||
| F1 | mIoU | METEOR | BERT | ||||
| Zero-shot Models | |||||||
| GOT-OCR 2.0 | -- | -- | -- | 0.8652 | 0.5330 | -- | -- |
| Nougat | -- | -- | -- | 0.7327 | 0.4640 | -- | -- |
| Dolphin-v2 | -- | 0.9124 | 0.9265 | 0.9307 | 0.8221 | -- | -- |
| MonkeyOCR | -- | 0.6030 | 0.8949 | 0.9684 | 0.6254 | -- | -- |
Table 2: Main test-set results under the unified Read-Parse-Describe output schema. Rows are grouped by evaluation setting to separate Zero-shot Models , Full Fine-Tuning , which means fine-tuning the whole model, and Adapter Tuning , which means only fine-tuning the adapter part. -- indicates that the corresponding field is not produced by the evaluated system. bold and underlined scores denote the best and second-best results for each metric, respectively.
Figure 2: Qualitative comparison of annotation mapping results. All model-based methods use Qwen3.5-2B as the backbone. We compare the ground-truth annotations with outputs from the base model, full fine-tuning, and the proposed SARA adapter.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| ID | Description | Label | ID | Description | Label |
| 1 | Paper Title | sec_0 | 13 | Table | tab |
| 2 | Level-1 Heading | sec_1 | 14 | Figure-Caption | figure-cap |
| 3 | Level-2 Heading | sec_2 | 15 | Table-Caption | table-cap |
| 4 | Level-3 Heading | sec_3 | 16 | Annotation | anno |
| 5 | Level-4 Heading | sec_4 | 17 | Formula | equ |
| 6 | Paragraph | para | 18 | Code Block | code |
Table 5: Semantic label taxonomy used in Read-Parsing-Describe.
| Category | Count | Percentage |
| AI & ML | 271 | 12.60% |
| Accessibility | 255 | 11.85% |
| AR/VR/XR | 235 | 10.93% |
| Design & HCI Methods | 149 | 6.93% |
| Privacy & Social | 103 | 4.79% |
| Health & Wellbeing | 95 | 4.42% |
Table 6: Primary Domain Distribution of Analyzed Articles from 2020 - 2024
| Element Category | Pages 1–8 (%) | Pages (%) | Density (1–8) | Density ( ) |
| Title, Abstract & Metadata | 100.00% | 0.00% | 1.2 / page | 0.0 / page |
| Equations & Math Blocks | 78.42% | 21.58% | 4.5 / page | 1.2 / page |
| Figures & Captions | 69.15% | 30.85% | 2.1 / page | 0.9 / page |
| Body Paragraphs | 62.30% | 37.70% | 12.4 / page | 7.5 / page |
| Tables & Algorithms | 54.80% | 45.20% | 0.8 / page | 0.7 / page |
| References | 8.25% | 91.75% | 0.5 / page | 15.6 / page |
Table 7: Impact of the 8-Page Truncation Strategy on Layout Element Distribution. By comparing the element proportions in Pages 1–8 versus Pages (estimated on a full-text subset), we observe that the truncation effectively preserves high-value multimodal objects while filtering out visually monotonous reference lists.
| Category | Count | Percentage |
| Computer Science & AI | 6,443 | 56.8% |
| Social Science | 2,181 | 19.2% |
| Engineering | 1,359 | 12.0% |
| Medicine & Healthcare | 652 | 5.8% |
| Other | 245 | 2.2% |
| Biology | 110 | 1.0% |
Table 8: Image Categories Distribution
| Image Type | Count | Percentage |
| Table | 2758 | 24.3% |
| Diagrammatic Instruction / Concept | 2086 | 18.4% |
| User Interface / Screenshot | 1743 | 15.4% |
| Photographic Image | 1529 | 13.5% |
| Flowchart / Block Diagram | 1000 | 8.8% |
| Bar Chart / Histogram | 942 | 8.3% |
Table 9: Image Type Distribution
| Category | Label | Count | Percentage |
| Structural Headers | sec_1 , sec_2 , sec_0 | 44,617 | 35.64% |
| Academic Metadata | author-info , fnote | 28,381 | 22.66% |
| Scholarly Objects | fig , tab , equ , code | 12,336 | 9.85% |
| Captions and References | figure-cap , table-cap , reference | 21,381 | 17.08% |
| Layout Fragments | half_para , list , icon | 17,564 | 14.03% |
| Others | anno , sec_3 , sec_4 , watermark , etc. | 939 | 0.74% |
Table 10: Distribution of high-value layout elements.
| Configuration | Detailed Content / Implementation |
| Generation Prompt | “Describe this image in 1–3 concise sentences for accessibility. If the image contains multiple panels or subfigures, summarize them as a single combined figure rather than describing each panel separately. Focus only on the key visual information, such as the type of chart or image, the main subjects, and clearly visible labels or visual patterns. Do not use phrases like ’image of’ or ’picture of’. Provide a factual description based only on what is visible, without interpreting or explaining the underlying data.” |
| Generation Parameters | Temperature: 0.2 Top-p: 0.9 Max New Tokens: 256 Repetition Penalty: 1.05 |
| Evaluation Role | System Prompt: You are a rigorous Image Accessibility Auditor specializing in scientific and technical content. Your goal is to determine if the provided ’Alt-text’ accurately represents the visual information for a visually impaired user. Evaluate strictly based on Accuracy, Completeness, and Constraint Adherence. |
| Evaluation Rubric | Score 1 (Failure - Misleading/Hallucination): Completely irrelevant, contains gross data hallucinations, or generates pure technical noise. Fails to identify the basic type of the image. Actively misleads the visually impaired user. Score 2 (Poor - Severe Omissions/Violations): Identifies the general chart/image type but suffers from severe omissions (e.g., missing critical axes, legends, or main subjects). Alternatively, it severely violates negative constraints (e.g., provides subjective interpretations of data instead of visual facts). Score 3 (Acceptable - Basic but Flawed): Correctly identifies the main subjects and chart type. However, it lacks specific critical details (e.g., distinct visual trends or key labels). May fail minor constraints (e.g., exceeds the 3-sentence limit, or includes filler phrases like "This is a picture of"). Score 4 (Good - Professional & Factual): Clear, objective, and accurate. Successfully captures the chart type, major variables, key labels, and significant visual patterns. Strictly adheres to negative constraints. May only miss minor, non-essential visual nuances or slightly lack extreme conciseness. Score 5 (Excellent - Perfect Accessibility): Highly concise (strictly 1-3 sentences), factual, and deeply informative. Seamlessly summarizes complex or multi-panel figures into a cohesive whole. Provides a visually impaired user with the exact informational equivalent of a quick visual scan, devoid of any over-interpretation or fluff. |
Table 11: Detailed Prompt Configurations and Expanded Evaluation Rubric for Alt-text Generation
| category | LLM generated (Count) | LLM + human_modified (Count) |
| Score 1 | 0 | 0 |
| Score 2 | 146 | 0 |
| Score 3 | 1262 | 0 |
| Score 4 | 2419 | 226 |
| Score 5 | 7511 | 11112 |
| Score 5 | 66.2% | 98.1% |
Table 12: Alt text generation:LLM generated vs LLM+human_modified
Figure 3: Human-LLM agreement audit over 200 revised samples. Agreement is measured after low-score descriptions are refined and re-scored.
| Data | Method | F1 | mIoU | OCR | RO | METEOR | BERTScore |
| 10% | Full FT | 0.6487 | 0.7563 | 0.8476 | 0.7996 | 0.1256 | 0.8106 |
| 10% | SARA | 0.5731 | 0.7144 | 0.7311 | 0.7805 | 0.1672 | 0.8002 |
| 30% | Full FT | 0.6771 | 0.7861 | 0.8756 | 0.8504 | 0.2651 | 0.8288 |
| 30% | SARA | 0.6567 | 0.8061 | 0.8087 | 0.8226 | 0.2871 | 0.8372 |
| 50% | Full FT | 0.6991 | 0.8232 | 0.9362 | 0.9623 | 0.3072 | 0.8540 |
| 50% | SARA | 0.8781 | 0.8533 | 0.9562 | 0.9623 | 0.3340 | 0.8533 |
Table 13: Training-data scaling on Qwen3.5-2B. Fractions indicate the reported proportion of the training data.
| Resolution | F1 | mIoU | OCR | RO |
| 100% | 0.9189 | 0.8919 | 0.9636 | 0.9774 |
| 50% | 0.8389 | 0.8673 | 0.9248 | 0.9534 |
| 25% | 0.7589 | 0.8194 | 0.7648 | 0.8864 |
Table 14: Resolution sensitivity of Qwen3.5-2B+SARA under the reported crop-resolution.
Figure 4: Representative failure cases of SARA. Top row: a borderless composite figure where intra-image text can cause a spatial shift between figure regions and their captions. Bottom row: a dense table where localization remains relatively accurate, but decoding may truncate long content and introduce minor hallucinated artifacts.