Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

Yunhang Shen; Zhangquan Chen; Xiaotian Li; Zhongpeng Cai; Qing Zhou; Zekai Zhang; Kun Zhang; Zezhong Wu

cs.CVOct 29, 2026

Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

Authors: Yunhang Shen, Zhangquan Chen, Xiaotian Li, Zhongpeng Cai, Qing Zhou, Zekai Zhang, Kun Zhang, Zezhong Wu,

Abstract

We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.

Figures & tables

Figure 1: Youtu-Parsing-Omni ranks first on OmniDocBench v1.6 (a), is the strongest open-weight model on OmniParsingBench (b), and also leads on most domain-specific benchmarks (c).

Figure 2: Omni-modal parsing with one encoder and one schema. Youtu-Parsing-Omni parses documents, natural images, audio, and video with Youtu-Omni-Encoder shared across all modalities, and unifies their parsing tasks in OmniSchema.

Figure 3: Overview of omni-modal parsing architectures. We compare how visual and audio inputs are encoded before reaching the language model in the conventional design and in Youtu-Parsing-Omni. (a) Existing omni-modal parsers encode pixels and audio with two separately pre-trained towers, whose streams first meet inside the language model. (b) Youtu-Parsing-Omni replaces both towers with a single encoder, Youtu-Omni-Encoder. Thin modality-specific stems map pixels and log-mel frames to tokens; a shared bidirectional block then contextualizes an arbitrary packed mix of images, audio chunks, and videos under one $(t,h,w)$ positional encoding; and a few of its layers (red) attend across modalities within short audio-visual windows (Fig. 4 ). As a result, pixels and sound share weights and positional encoding, and interact already inside the encoder.

Figure 4: Concurrent visual and audio content share fusion windows and a common temporal coordinate system. (a) A video with eight frames (I) and four audio chunks (A), packed in temporal order, with each chunk placed after the frames it overlaps. A non-fusion layer gives every frame and every chunk its own window; a fusion layer starts a new window at every audio-to-frame boundary, so the video forms four

(\mathrm{I}^{+}\mathrm{A}^{*})

windows; red cells mark the cross-modal pairs, and off-diagonal blue cells the frame-to-frame pairs, that only fusion layers admit. (b) Vision tokens take

t

from the frame index and

(h,w)

from the patch grid; audio tokens take

t

from their offset and write into the same temporal rotary slots. Inside a fusion window both advance on the common time scale of Equ. 1 .

Figure 5: Perception first, output format second, on-policy refinement last. Four pre-training stages (Stage 0: perception alignment; Stage 1: omni-modal pre-training; Stage 2: annealing; Stage 3: long-context extension) are followed by three post-training stages: supervised fine-tuning, schema-routed RLVR, and OSAD. Rows give each stage’s token budget, trainable parameters, sequence-length limit, number of encoder fusion layers, and maximum frames per training video. Stage 1 uses sequence lengths of

8

k–

32

k. LoRA denotes low-rank adaptation of frozen base weights.

Figure 6: OSAD combines token-adaptive on-policy supervision with teacher refresh at each round boundary. Admitted student rollouts provide shared prefixes for a LoRA-trained student and a frozen, reference-conditioned teacher. The objective combines content-specific JS divergence at $\tau_{c}$ , structure-specific forward KL divergence at $\tau_{s}$ , and an all-token JS term at $\tau$ . Within each round, only the student’s adapters are updated. At the round boundary, the learned adapters are merged into the base weights to initialize the next teacher and student; the student receives fresh adapters, while only the teacher conditions on the reference. The lower panel illustrates the update and refresh mechanism rather than a measured model-state distance.

Benchmark	Split / task	Size	Reported metric	Baselines
OmniDocBench v1.6 [ 42 ]	full set	1,651	Overall, Edit, CDM, TEDS	leaderboard
PureDocBench [ 43 ]	clean, degraded, recaptured	1,475 $\times$ 3	per-track Overall, Avg 3	leaderboard
OmniParsingBench [ 11 ]	natural image	1,000	Overall: parse rate $\times$ mean of Perception and Cognition	re-scored
	chart, geometry	1,000
	audio	938
	natural video (general)	550

Table 1: Evaluation protocol. Size is the number of evaluated samples. Every benchmark is scored with its official code (the LEGATO protocol for PDMX-Synth), except OmniParsingBench, which we score with our own toolkit (App. D ). Baselines : re-scored means that every system is run and scored by us with the same code; leaderboard means that baseline numbers are taken from the official public leaderboard. ∗ ChemDFM-X scores on ChemOCR are taken from TinyChemVL [ 41 ] ; LEGATO scores on PDMX-Synth are taken from its original report [ 8 ] (Tab. 12 ).

Multi-stage specialists
Model	Params	Overall $\uparrow$	Text Edit $\downarrow$	Formula CDM $\uparrow$	TEDS $\uparrow$	TEDS-S $\uparrow$	Read. Order Edit $\downarrow$
TeleOCR [ 47 ]	$1.2$ B	96.91	0.0267	96.59	96.82	98.18	0.1184
PaddleOCR-VL-1.6 [ 6 ]	$0.9$ B	96.34	0.0326	97.53	94.76	97.10	0.1278
MinerU2.5-Pro [ 20 ]	$1.2$ B	95.75	0.0360	97.45	93.42	95.92	0.1200
GLM-OCR [ 48 ]	$0.9$ B	95.22	0.0440	97.18	92.83	95.39	0.1330
PaddleOCR-VL-1.5 [ 49 ]	$0.9$ B	94.93	0.0380	96.89	91.67	94.37	0.1300

Table 2: Performance comparison of Youtu-Parsing-Omni with other document parsing models and proprietary models on OmniDocBench v1.6. Bold and underline : best and second best in each column. † Proprietary model.

Figure 7: Youtu-Parsing-Omni is the only model above

96

on OmniDocBench v1.6 that also parses images, audio, and video, and its gain over Youtu-Parsing comes mostly from tables and formulas. (a) OmniDocBench v1.6 Overall against total parameters (teal: open-weight), with API systems in a strip at the right; the document specialists at our level have

0.8

–

1.2

B parameters. (b) Per-metric difference to Youtu-Parsing [ 4 ] , our previous model, and to TeleOCR [ 47 ] , the next-best system overall; edit distances are shown as

1-\text{Edit}

so that a positive difference always favors Youtu-Parsing-Omni.

Multi-stage specialists
		Clean				Digitally degraded				Physically recaptured
Model	Params	Overall $\uparrow$	Text Edit $\downarrow$	Formula CDM $\uparrow$	Table TEDS $\uparrow$	Overall $\uparrow$	Text Edit $\downarrow$	Formula CDM $\uparrow$	Table TEDS $\uparrow$	Overall $\uparrow$	Text Edit $\downarrow$	Formula CDM $\uparrow$	Table TEDS $\uparrow$	Avg ${}_{3}\uparrow$
TeleOCR [ 47 ]	$1.2$ B	86.90	0.111	81.01	91.09	77.47	0.206	72.59	80.45	70.85	0.302	65.11	77.66	78.41
dots.mocr [ 68 ]	$3$ B	76.27	0.151	66.23	77.65	73.16	0.198	64.32	74.95	61.73	0.312	54.39	61.97	70.39
MinerU2.5-Pro [ 20 ]	$1.2$ B	75.87	0.222	65.14	84.68	71.77	0.272	61.79	80.73	62.56	0.375	52.70	72.47	70.07
Youtu-Parsing [ 4 ]	$2.5$ B	75.02	0.230	67.34	80.74	69.66	0.270	61.44	74.49	60.29	0.360	52.20	64.69	68.32

Table 3: Comparison with existing document parsing models and proprietary models on PureDocBench under clean and degraded conditions. Numbers are from the public leaderboard [ 43 ] . † Proprietary model. Bold and underline : best and second best of each Overall column.

Proprietary / API
Model	Chart	Geometry	Nat. image	Audio	Nat. video	Text-rich video	Avg.
Gemini-3-Pro [ 62 ]	92.79	85.43	69.73	76.74	74.82	76.80	77.44
GPT-5.4 [ 67 ]	90.92	81.16	60.50	–	–	–	–
Open-weight
Logics-Parsing-Omni [ 11 ]	94.65	76.34	67.53	75.58	67.34	65.36	72.26
Qwen3-Omni-30B-A3B [ 10 ]	87.07	58.60	59.35	75.23	65.19	59.59	66.44

Table 4: Comparison with open-weight and proprietary models on each split of OmniParsingBench. Scores are the per-split Overall (Subsec. 5.1 ); higher is better. Bold and underline : best and second best in each column. Avg. is the average over five columns, in which chart and geometry enter together as one graphics column. All systems are scored with the same toolkit. Perception and Cognition are in Tab. 5 – 10 .

Model	Params	Avg. Sim. $\uparrow$	[email protected] $\uparrow$
DeepSeek-OCR 2 [ 24 ]	$3$ B	74.9	49.6
ChemDFM-X ∗ [ 7 ]	$13$ B	70.9	36.5
GPT-4o [ 76 ]	–	36.8	3.4
Qwen2.5-VL-7B [ 77 ]	$7$ B	25.5	0.4
InternVL2.5-4B [ 78 ]	$4$ B	4.4	0.0
Youtu-Parsing-Omni	$5$ B	74.6	54.2

Table 12: Comparison with existing specialized, general, and proprietary models on ChemOCR and PDMX-Synth. Left: ChemOCR (

1{,}000

molecules;

990

scored after excluding

10

invalid references). Right: PDMX-Synth (

800

score images). Bold and underline : best and second best in each column. ∗ ChemDFM-X [ 7 ] scores on ChemOCR are taken from TinyChemVL [ 41 ] ; LEGATO scores on PDMX-Synth are taken from its original report [ 8 ] .

Figure 8: Encoder architecture and initialization studies. (a) General multimodal (Avg-6); (b) OCRBench; (c) BLINK; (d) LibriSpeech test-clean WER. Blue, green, and pink denote the text-initialized unified encoder, randomly initialized unified encoder, and pre-trained SigLIP-2 [ 80 ] and SenseVoice-small [ 81 ] towers, respectively.

Figure 9: OSAD supplies

1.75

of the

1.99

points gained after SFT (

88\%

), with positive but diminishing increments. OmniDocBench v1.6 Overall (higher is better). (a) Score after SFT, RLVR, and each OSAD round; the dashed line marks TeleOCR, the strongest other system (

96.91

). (b) Gain over the preceding checkpoint.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Scope	Semantics
id	element	Unique element identifier.
category	element	Element type: text, table, formula, code, caption, header, footer, title, chemical, score, chart, flowchart, figure, seal, or geometric.
bbox	element	Bounding box of the element in page coordinates.
content.text	element	Transcription for text-like elements, code, chemical structures, and ABC notation.
content.latex	formula	Formula transcription in L a T e X .
content.otsl	table	Table structure and content in OTSL.

Appendix

Table 13: Annotation fields of the document parsing dataset and their semantics.

Field	Scope	Semantics
id	element	Unique element identifier.
category	element	Element type, fixed to entity .
bbox	element	Bounding box of a salient object.
tag	element	A word naming the object inside the bounding box.
content.text	element	OCR result inside the bounding box; "" when the annotation pipeline returns no text.
content.caption	element	Precise and concise caption of the region inside the bounding box.

Appendix

Table 14: Annotation fields of the natural image parsing dataset and their semantics.

Field	Scope	Semantics
id	segment	Unique segment identifier.
kind	segment	Segment type: vocal or nonvocal .
audio.asr_text	vocal segment	Verbatim transcription of the speech in the segment.
audio.speaker_id	vocal segment	Speaker identifier, consistent across the chunk.
audio.timbre_caption	vocal segment	Voice timbre: gender, age, pitch, breathiness, etc.
audio.instruct_caption	vocal segment	Speech delivery characteristics: emotion, pace, prosody, etc.

Appendix

Table 15: Annotation fields of the audio parsing dataset and their semantics.

Field	Scope	Category	Semantics
id	segment	shared	Unique segment identifier.
subtype	video	shared	Video type: natural_video or text_rich_video .
start/end_timestamp	segment	shared	Temporal boundaries relative to video onset.
visual.elements	segment	shared	Salient objects and on-screen text; each item has a category ( object or text ) and, for text, an ocr_type (e.g., title ).
visual.atomic_actions	segment	shared	Fine-grained actions within the segment.
visual.object_actor_interactions	segment	shared	Interactions between actors and objects.

Appendix

Table 16: Annotation fields of the video parsing dataset and their semantics.

Component	Quantity	Value
Youtu-Omni-Encoder	layers $L$	$28$
	hidden size $d$	$1{,}024$
	FFN hidden size $d_{\mathrm{ffn}}$	$3{,}072$
	query heads $H$ / KV groups $H_{kv}$	$16$ / $8$
	head dimension $d_{h}$	$128$
	attention	GQA, bidirectional, packed

Appendix

Table 17: Complete configuration of Youtu-Parsing-Omni.

Model	Encoder paradigm	Vision	Audio	Language model	Doc.	Image	Audio	Video	End-to-end	Params
		Perception			Parsing capability
MinerU2.5-Pro [ 20 ]	Modality-specific	ViT $0.6$ B	–	Qwen2-0.5B	✓	–	–	–	–	$1.2$ B
PaddleOCR-VL-1.6 [ 6 ]	Modality-specific	ViT $0.4$ B	–	ERNIE-4.5-0.3B	✓	–	–	–	–	$0.9$ B
TeleOCR [ 47 ]	Modality-specific	ViT $0.6$ B	–	Qwen3-0.6B	✓	–	–	–	–	$1.2$ B
DeepSeek-OCR 2 [ 24 ]	Modality-specific	LLM $0.5$ B	–	DeepSeek-3B-MoE	✓	–	–	–	✓	$3$ B
Youtu-Parsing [ 4 ]	Modality-specific	ViT $0.4$ B	–	Youtu-LLM-2B	✓	–	–	–	–	$2.5$ B

Appendix

Table 18: Every other representative parser uses a separate encoder per modality, and only Logics-Parsing-Omni covers all three modalities, with two towers; Youtu-Parsing-Omni covers them with one encoder. Modality-specific : every modality has its own encoder; Unified : one encoder with shared weights for all modalities. End-to-end : a page or clip is parsed in one forward pass rather than by a layout-then-recognition pipeline.

	Natural image	Graphics	Audio	Natural video	Text-rich video	Total
Original	1,000	1,000	1,014	601	259	3,874
Media excluded	0	0	76	51	9	136
Kept	1,000	1,000	938	550	250	3,738

Appendix

Table 19: OmniParsingBench sample counts within our evaluation scope. Natural video counts only the general subset.

Figure 10: Unified encoder versus separate towers. Green solid curves denote the randomly initialized unified encoder (

0.4

B); pink dashed curves denote pre-trained SigLIP-2 [ 80 ] and SenseVoice-small [ 81 ] towers (

0.6

B combined) with random projectors. Panels report individual vision benchmarks, the general multimodal average (Avg-6), and ASR error rates.

Figure 11: Encoder initialization on vision and ASR benchmarks. Blue solid curves denote the encoder initialized from Qwen3-0.6B-Base, and green dashed curves denote the encoder trained from scratch. Shaded bands indicate the performance gap, colored by the better-performing branch. Lower WER and CER indicate better ASR performance.

Panel of Figure 12 — Figure 12: Document. A conventional single-column page: the parse recovers one linear reading order over headings, body paragraphs, and inline elements.

Panel of Figure 14 — Figure 14: Document. A two-column page combining several element categories—text, formulas, tables, and figures. The parse has to assign every element to its column and serialize the two columns into a single reading order rather than interleaving them line by line.

Panel of Figure 16 — Figure 16: Flowchart. Anomaly-Handling Workflow of a Medical System.

Panel of Figure 18 — Figure 18: Audio. Sci-Fi Dialogue with Mechanical Sound Effects.

Panel of Figure 20 — Figure 20: Natural Video. Raindrops Coalescing on a Glass Surface.

Panel of Figure 22 — Figure 22: Geometry. Triangle with Perpendicular Foot and Angle Annotation.

Figure 25: Text-Rich Video. Data Analysis Problem-Solving Framework. Frames, timeline, and narrative; the report and segments follow on the next page.

Figure 26: Text-Rich Video. Data Analysis Problem-Solving Framework (continued).

Figure 27: Text-Rich Video. System Security Course on Attack Types and Threats. Frames, timeline, narrative, report, and the first segment; the last frame and the remaining segments follow on the next page.

Figure 28: Text-Rich Video. System Security Course on Attack Types and Threats (continued).

Document Parsing

Document parsing converts PDFs and page images into structured, machine-readable text, tables, formulas, layouts, and reading order.

See all 64 papers

Logics-Parsing-Omni Technical Report
Mar 10, 2026Xin An, Jingyi Cai, Xiangyang Chen +22Multimodal Understanding Semantic Parsing
Alibaba Group
OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder
May 2, 2026Detao Bai, Shimin Yao, Weixuan Chen +4Cross-Modal Learning Video Understanding
1Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology
Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decoding
Jan 28, 2026Haoyu Cao, Kun Yin, Yunfei Wu +16Efficient VLM Inference Document Parsing
Youtu-Parsing Team*

Mar 10, 2026J/K move · Enter open · S save

Logics-Parsing-Omni Technical Report

Xin An, Jingyi Cai, Xiangyang Chen, Huayao Liu+21

Alibaba Group

Multimodal Understanding Semantic Parsing Multimodal Reasoning Multimodal Grounding Multimodal Model Evaluation

Open paper AskPDF arXiv

Addressing the challenges of fragmented task definitions and the heterogeneity of unstructured data in multimodal parsing, this paper proposes the Omni Parsing framework. This framework establishes a Unified Taxonomy covering documents, images, and audio-visual streams, introducing a progressive parsing paradigm that bridges perception and cognition. Specifically, the framework integrates three hierarchical levels: 1) Holistic Detection, which achieves precise spatial-temporal grounding of objects or events to establish a geometric baseline for perception; 2) Fine-grained Recognition, which performs symbolization (e.g., OCR/ASR) and attribute extraction on localized objects to complete structured entity parsing; and 3) Multi-level Interpreting, which constructs a reasoning chain from local semantics to global logic. A pivotal advantage of this framework is its evidence anchoring mechanism, which enforces a strict alignment between high-level semantic descriptions and low-level facts. This enables ``evidence-based'' logical induction, transforming unstructured signals into standardized knowledge that is locatable, enumerable, and traceable. Building on this foundation, we constructed a standardized dataset and released the Logics-Parsing-Omni model, which successfully converts complex audio-visual signals into machine-readable structured knowledge. Experiments demonstrate that fine-grained perception and high-level cognition are synergistic, effectively enhancing model reliability. Furthermore, to quantitatively evaluate these capabilities, we introduce OmniParsingBench. Code, models and the benchmark are released at https://github.com/alibaba/Logics-Parsing/tree/master/Logics-Parsing-Omni.

Abstract

Document Parsing

Explore similar work

Logics-Parsing-Omni Technical Report

OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder

Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decoding