Multimodal OCR: Parse Anything from Documents

Handong Zheng; Yumeng Li; Kaile Zhang; Liang Xin; Guangwei Zhao; Hao Liu; Jiayu Chen; Jie Lou

cs.CVMar 13, 2026

Multimodal OCR: Parse Anything from Documents

Authors: Handong Zheng, Yumeng Li, Kaile Zhang, Liang Xin, Guangwei Zhao, Hao Liu, Jiayu Chen, Jie Lou, +17 more

Organizations: Huazhong University of Science and Technology · hi lab, Xiaohongshu Inc

Abstract

We present Multimodal OCR (MOCR), a document parsing paradigm that jointly parses text and graphics into unified textual representations. Unlike conventional OCR systems that focus on text recognition and leave graphical regions as cropped pixels, our method, termed dots.mocr, treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets, enabling systems to parse documents while preserving semantic relationships across elements. It offers several advantages: (1) it reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction; (2) it supports end-to-end training over heterogeneous document elements, allowing models to exploit semantic relations between textual and visual components; and (3) it converts previously discarded graphics into reusable code-level supervision, unlocking multimodal supervision embedded in existing documents. To make this paradigm practical at scale, we build a comprehensive data engine from PDFs, rendered webpages, and native SVG assets, and train a compact 3B-parameter model through staged pretraining and supervised fine-tuning. We evaluate dots.mocr from two perspectives: document parsing and structured graphics parsing. On document parsing benchmarks, it ranks second only to Gemini 3 Pro on our OCR Arena Elo leaderboard, surpasses existing open-source document parsing systems, and sets a new state of the art of 83.9 on olmOCR Bench. On structured graphics parsing, our model achieves higher reconstruction quality than Gemini 3 Pro across image-to-SVG benchmarks, demonstrating strong performance on charts, UI layouts, scientific figures, and chemical diagrams. These results show a scalable path toward building large-scale image-to-code corpora for multimodal pretraining. Code and models are publicly available at https://github.com/rednote-hilab/dots.mocr.

Figures & tables

Figure 1: Overview of MOCR. Given a document image, MOCR parses anything on the page into unified, ordered textual representations, capturing both textual and visual structures and enabling faithful reconstruction of the original document.

Figure 2: Overall performance comparison results. (a) reports metrics for image parsing tasks within documents. dots.mocr-svg are further enhanced with additional task-specific training for graphics parsing. (b) reports metrics for general document parsing tasks (olmOCR-Bench, OmniDocBench 1.5, and XDocParse).

Figure 3: Comparison between traditional text-only OCR and MOCR paradigms. Traditional OCR treats graphics as pixels and often discards them, while MOCR parses graphics into structured code (e.g. SVG), enabling faithful reconstruction and broader downstream applications.

Model Type	Inference Type	Model Name	Task
Model Type	Inference Type	Model Name	Layout-Det	Parsing	Spotting	VQA	IE	Graphics
Cascade Pipeline	Multi-Step	PaddleOCR-V5 [ 5 ]	✗	✗	✓	✗	✗	✗
		Marker-1.8.2	✓	✓	✗	✗	✗	✗
		PP-StructureV3 [ 5 ]	✓	✓	✗	✗	✗	✗
Specialized VLMs (Modular)	Multi-step	MonkeyOCR-pro-3B [ 16 ]	✓	✓	✗	✗	✗	✗
		MinerU2.5 [ 25 ]	✓	✓	✗	✗	✗	✗

Table 1: Comparison of different VLMs and OCR systems across multiple tasks. ✓ indicates Supported (good),

\triangle

indicates Supported but Underperforming, and ✗ indicates Not Supported.

Models	olmOCR-Bench	OmniDocBench1.5	XDocParse	Average
Gemini 3 Pro [ 28 ]	1180.4	1128.0	1323.7	1210.7
dots.mocr	1104.4	1059.0	1210.7	1124.7
dots.ocr [ 15 ]	1041.1	1027.2	1190.3	1086.2
HunyuanOCR [ 33 ]	997.6	1003.9	951.1	984.2
PaddleOCR-VL-1.5 [ 4 ]	897.3	997.9	866.4	920.5
GLM-OCR [ 6 ]	884.2	972.6	820.7	892.5

Table 2: Elo comparison on olmOCR-Bench [ 29 ] , OmniDocBench(v1.5) [ 26 ] , and XDocParse [ 15 ] . All models are evaluated under the same Elo protocol with Gemini 3 Flash as judge, on benchmark-provided document images.

Model	ArXiv	Old scans math	Tables	Old scans	Headers & footers	Multi column	Long tiny text	Base	Overall
MinerU 2.5.4 [ 25 ]	76.6	54.6	84.9	33.7	96.6	78.2	83.5	93.7	75.2 $\pm$ 1.1
MonkeyOCR-pro-3B [ 16 ]	83.8	68.8	74.6	36.1	91.2	76.6	80.1	95.3	75.8 $\pm$ 1.0
dots.ocr [ 15 ]	82.1	64.2	88.3	40.9	94.1	82.4	81.2	99.5	79.1 $\pm$ 1.0
DeepSeek-OCR [ 39 ]	77.2	73.6	80.2	33.3	96.1	66.4	79.4	99.8	75.7 $\pm$ 1.0
Nanonets-OCR2-3B [ 21 ]	75.4	46.1	86.8	40.9	32.1	81.9	93.0	99.6	69.5 $\pm$ 1.1
PaddleOCR-VL [ 3 ]	85.7	71.0	84.1	37.8	97.0	79.9	85.7	98.5	80.0 $\pm$ 1.0

Table 3: Performance comparison on olmOCR-Bench.

Methods	UniSVG Low-Level	UniSVG High-Level	UniSVG Score	ChartMimic	Design2Code	GenExam	SciGen	ChemDraw
OCRVerse [ 47 ]	0.632	0.852	0.763	0.799	-	-	-	0.881
Gemini 3 Pro [ 28 ]	0.563	0.850	0.735	0.788	0.760	0.756	0.783	0.839
dots.mocr	0.850	0.923	0.894	0.772	0.801	0.664	0.660	0.790
dots.mocr-svg	0.860	0.931	0.902	0.905	0.834	0.800	0.797	0.901

Table 4: ISVGEN reconstruction scores on UniSVG (low-level, high-level, overall) and downstream visual-language benchmarks (higher is better).

Model	CharXiv Descriptive	CharXiv Reasoning	OCR Reasoning	InfoVQA	DocVQA	ChartQA	OCRBench	AI2D	CountBenchQA	RefCOCO
Qwen3-VL-2B-Instruct [ 1 ]	62.3	26.8	-	72.4	93.3	-	85.8	76.9	88.4	-
Qwen3-VL-4B-Instruct [ 1 ]	76.2	39.7	-	80.3	95.3	-	88.1	84.1	84.9	-
dots.mocr	77.4	55.3	22.85	73.76	91.85	83.2	86.0	82.16	94.46	80.03

Table 5: Performance comparison on CharXiv, OCR reasoning, and a suite of document and vision-language benchmarks.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model Type	Methods	Size	OmniDocBench(v1.5) TextEdit $\downarrow$	OmniDocBench(v1.5) Read OrderEdit $\downarrow$	pdf-parse-bench
GeneralVLMs	Gemini 2.5 Pro	-	0.075	0.097	9.06
	Qwen3-VL-235B-A22B-Instruct [ 1 ]	235B	0.069	0.068	9.71
	Gemini 3 Pro [ 28 ]	-	0.066	0.079	9.68
SpecializedVLMs	Mistral OCR	-	0.164	0.144	8.84
	DeepSeek-OCR [ 39 ]	3B	0.073	0.086	8.26
	MonkeyOCR-3B [ 16 ]	3B	0.075	0.129	9.27

Appendix

Table 6: Comparison of general-purpose and specialized VLMs on OmniDocBench(v1.5) and pdf-parse-bench. Lower is better for edit metrics.

Figure 4: Qualitative layout analysis results of dots.mocr on heterogeneous documents. The model identifies and partitions structural elements such as titles, paragraphs, multi-column regions, dense tables, mathematical formulas, scanned text, and handwritten content, demonstrating robust document-level structural understanding across diverse formats and languages.

Figure 5: Parsing results of dots.mocr on full-length web screenshots and real-world scene images. The model preserves global reading order and structured components in long webpage layouts, while accurately recognizing and organizing scene text in complex real-world environments, highlighting generalization beyond conventional OCR benchmarks.

Figure 6: SVG parsing outputs and rendered reconstructions produced by dots.mocr-svg for icon images. The model converts raster icons into concise and executable SVG code, capturing geometric primitives, hierarchical grouping, and spatial relationships for scalable vector-level reconstruction.

Figure 7: Rendered SVG reconstruction results of dots.mocr-svg for diverse statistical charts, including bar, line, scatter, and composite visualizations. The model recovers chart structure, axes, legends, data encodings, and semantic grouping through executable SVG representations, enabling editable and reusable vector outputs.

Figure 8: Rendered SVG reconstruction results of dots.mocr-svg for complex cross-disciplinary illustrations in science and design. The model captures intricate geometric structures, semantic components, and hierarchical organization, demonstrating its ability to generalize beyond standard charts to domain-specific structured graphics.

Figure 9: General-purpose visual question answering results of dots.mocr. While retaining strong document parsing and SVG reconstruction capabilities, the model produces coherent and context-aware responses across documents, charts, UI screenshots, and complex illustrations, indicating competitive performance in broader vision–language tasks.

Figure 10: Prompt used in OCR Arena

Figure 11: Example judging results in OCR Arena, showing document images, OCR outputs from two models, and the explanations produced by the VLM judge.

Explore similar work

Jul 15, 2026

OvisOCR2 Technical Report

We introduce OvisOCR2, a 0.8B document parsing model. OvisOCR2 is designed as an end-to-end parser: given a document page image, it generates a Markdown representation in natural reading order, covering text, formulas, tables, and visual regions. We build a data engine that combines filtered real-document annotations with synthetic pages whose rendered images and Markdown targets are derived from the same HTML source. The training recipe includes supervised fine-tuning, reinforcement learning on a 4B branch with a multi-component reward design, on-policy distillation into the 0.8B model, and model fusion. On OmniDocBench v1.6, OvisOCR2 achieves a state-of-the-art overall score of 96.58, placing an end-to-end model at the top of this leaderboard previously dominated by pipeline methods and highlighting the potential of end-to-end document parsing. On PureDocBench, OvisOCR2 also achieves the highest Avg3 score of 75.06. Beyond these two public benchmarks, we evaluate OvisOCR2 on an in-house benchmark designed to cover a broader set of long-tail and challenging scenarios. OvisOCR2 obtains the best overall performance among the compared methods, providing further evidence of its generalization and robustness. OvisOCR2 is available at https://huggingface.co/ATH-MaaS/OvisOCR2.

Shiyin Lu, Yinglun Li, Yu Xia +10

ATH-MaaS, Alibaba Group

Document Parsing Optical Character Recognition Document Image Understanding

Ask PDF

Jul 8, 2026

Infinity-Parser2 Technical Report

We present Infinity-Parser2, a large multimodal model that couples a controllable data-synthesis pipeline with multi-task reinforcement learning for end-to-end document parsing, addressing the persistent scarcity of faithfully annotated parsing corpora. Our contributions are threefold. First, we build a scalable synthesis engine, pairing a controllable rendering framework with an iterative refinement loop, and use it to construct and open-source Infinity-Doc2-5M: a 5-million-sample bilingual (Chinese/English) corpus spanning diverse document types, annotated with element bounding boxes, canonical content forms (Markdown, HTML, LaTeX, SMILES, structured charts), and full-page reading order. Second, we introduce a verifiable, multi-task reward system that enables Joint Reinforcement Learning across eight co-trained objectives (document parsing, layout analysis, table parsing, math formula parsing, chart parsing, chemical formula parsing, document VQA, and general multimodal understanding), unifying perception, structure, and reasoning in a single optimization signal. Third, we release two variants under a shared architecture: Infinity-Parser2-Flash, optimized for low-latency inference with a 3.68x throughput gain over Infinity-Parser-7B, and Infinity-Parser2-Pro, engineered for precision-critical settings. Infinity-Parser2-Pro reaches state-of-the-art 87.6% on olmOCR-Bench and 74.3% on ParseBench, surpassing DeepSeek-OCR-2, PaddleOCR-VL-1.5, and MinerU2.5, with strong generalization to charts, chemical formulas, and document VQA.

Zuming Huang, Jun Huang, Kexuan Ren +12

Synthetic Data Generation Multimodal Large Language Models Document Parsing Multi-Task RL Multimodal Foundation Models Multimodal Understanding

Ask PDF

Mar 11, 2026

Qianfan-OCR: A Unified End-to-End Model for Document Intelligence

We present Qianfan-OCR, a 4B-parameter end-to-end vision-language model that unifies document parsing, layout analysis, and document understanding within a single architecture. It performs direct image-to-Markdown conversion and supports diverse prompt-driven tasks including table extraction, chart understanding, document QA, and key information extraction. To address the loss of explicit layout analysis in end-to-end OCR, we propose Layout-as-Thought, an optional thinking phase triggered by special think tokens that generates structured layout representations -- bounding boxes, element types, and reading order -- before producing final outputs, recovering layout grounding capabilities while improving accuracy on complex layouts. Qianfan-OCR ranks first among end-to-end models on OmniDocBench v1.5 (93.12) and OlmOCR Bench (79.8), achieves competitive results on OCRBench, CCOCR, DocVQA, and ChartQA against general VLMs of comparable scale, and attains the highest average score on public key information extraction benchmarks, surpassing Gemini-3.1-Pro, Seed-2.0, and Qwen3-VL-235B. The model is publicly accessible via the Baidu AI Cloud Qianfan platform.

Daxiang Dong, Mingming Zheng, Dong Xu +17

Document Image Understanding Optical Character Recognition Document Information Extraction Vision-Language Models Document Layout Analysis

Ask PDF