We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.
Figures & tables
Figure 1: Youtu-Parsing-Omni ranks first on OmniDocBench v1.6 (a), is the strongest open-weight model on OmniParsingBench (b), and also leads on most domain-specific benchmarks (c).
Figure 2: Omni-modal parsing with one encoder and one schema. Youtu-Parsing-Omni parses documents, natural images, audio, and video with Youtu-Omni-Encoder shared across all modalities, and unifies their parsing tasks in OmniSchema.
Figure 3: Overview of omni-modal parsing architectures. We compare how visual and audio inputs are encoded before reaching the language model in the conventional design and in Youtu-Parsing-Omni. (a) Existing omni-modal parsers encode pixels and audio with two separately pre-trained towers, whose streams first meet inside the language model. (b) Youtu-Parsing-Omni replaces both towers with a single encoder, Youtu-Omni-Encoder. Thin modality-specific stems map pixels and log-mel frames to tokens; a shared bidirectional block then contextualizes an arbitrary packed mix of images, audio chunks, and videos under one (t,h,w) positional encoding; and a few of its layers (red) attend across modalities within short audio-visual windows (Fig. 4 ). As a result, pixels and sound share weights and positional encoding, and interact already inside the encoder.
Figure 4: Concurrent visual and audio content share fusion windows and a common temporal coordinate system. (a) A video with eight frames (I) and four audio chunks (A), packed in temporal order, with each chunk placed after the frames it overlaps. A non-fusion layer gives every frame and every chunk its own window; a fusion layer starts a new window at every audio-to-frame boundary, so the video forms four (I+A∗) windows; red cells mark the cross-modal pairs, and off-diagonal blue cells the frame-to-frame pairs, that only fusion layers admit. (b) Vision tokens take t from the frame index and (h,w) from the patch grid; audio tokens take t from their offset and write into the same temporal rotary slots. Inside a fusion window both advance on the common time scale of Equ. 1 .
Figure 5: Perception first, output format second, on-policy refinement last. Four pre-training stages (Stage 0: perception alignment; Stage 1: omni-modal pre-training; Stage 2: annealing; Stage 3: long-context extension) are followed by three post-training stages: supervised fine-tuning, schema-routed RLVR, and OSAD. Rows give each stage’s token budget, trainable parameters, sequence-length limit, number of encoder fusion layers, and maximum frames per training video. Stage 1 uses sequence lengths of 8 k– 32 k. LoRA denotes low-rank adaptation of frozen base weights.
Figure 6: OSAD combines token-adaptive on-policy supervision with teacher refresh at each round boundary. Admitted student rollouts provide shared prefixes for a LoRA-trained student and a frozen, reference-conditioned teacher. The objective combines content-specific JS divergence at τc , structure-specific forward KL divergence at τs , and an all-token JS term at τ . Within each round, only the student’s adapters are updated. At the round boundary, the learned adapters are merged into the base weights to initialize the next teacher and student; the student receives fresh adapters, while only the teacher conditions on the reference. The lower panel illustrates the update and refresh mechanism rather than a measured model-state distance.
Benchmark
Split / task
Size
Reported metric
Baselines
OmniDocBench v1.6 [ 42 ]
full set
1,651
Overall, Edit, CDM, TEDS
leaderboard
PureDocBench [ 43 ]
clean, degraded, recaptured
1,475 × 3
per-track Overall, Avg 3
leaderboard
OmniParsingBench [ 11 ]
natural image
1,000
Overall: parse rate × mean of Perception and Cognition
re-scored
chart, geometry
1,000
audio
938
natural video (general)
550
Table 1: Evaluation protocol. Size is the number of evaluated samples. Every benchmark is scored with its official code (the LEGATO protocol for PDMX-Synth), except OmniParsingBench, which we score with our own toolkit (App. D ). Baselines : re-scored means that every system is run and scored by us with the same code; leaderboard means that baseline numbers are taken from the official public leaderboard. ∗ ChemDFM-X scores on ChemOCR are taken from TinyChemVL [ 41 ] ; LEGATO scores on PDMX-Synth are taken from its original report [ 8 ] (Tab. 12 ).
Model
Params
Overall ↑
Text Edit ↓
Formula CDM ↑
TEDS ↑
TEDS-S ↑
Read. Order Edit ↓
Multi-stage specialists
TeleOCR [ 47 ]
1.2 B
96.91
0.0267
96.59
96.82
98.18
0.1184
PaddleOCR-VL-1.6 [ 6 ]
0.9 B
96.34
0.0326
97.53
94.76
97.10
0.1278
MinerU2.5-Pro [ 20 ]
1.2 B
95.75
0.0360
97.45
93.42
95.92
0.1200
GLM-OCR [ 48 ]
0.9 B
95.22
0.0440
97.18
92.83
95.39
0.1330
PaddleOCR-VL-1.5 [ 49 ]
0.9 B
94.93
0.0380
96.89
91.67
94.37
0.1300
Table 2: Performance comparison of Youtu-Parsing-Omni with other document parsing models and proprietary models on OmniDocBench v1.6. Bold and underline : best and second best in each column. † Proprietary model.
Figure 7: Youtu-Parsing-Omni is the only model above 96 on OmniDocBench v1.6 that also parses images, audio, and video, and its gain over Youtu-Parsing comes mostly from tables and formulas. (a) OmniDocBench v1.6 Overall against total parameters (teal: open-weight), with API systems in a strip at the right; the document specialists at our level have 0.8 – 1.2 B parameters. (b) Per-metric difference to Youtu-Parsing [ 4 ] , our previous model, and to TeleOCR [ 47 ] , the next-best system overall; edit distances are shown as 1−Edit so that a positive difference always favors Youtu-Parsing-Omni.
Clean
Digitally degraded
Physically recaptured
Model
Params
Overall ↑
Text Edit ↓
Formula CDM ↑
Table TEDS ↑
Overall ↑
Text Edit ↓
Formula CDM ↑
Table TEDS ↑
Overall ↑
Text Edit ↓
Formula CDM ↑
Table TEDS ↑
Avg 3↑
Multi-stage specialists
TeleOCR [ 47 ]
1.2 B
86.90
0.111
81.01
91.09
77.47
0.206
72.59
80.45
70.85
0.302
65.11
77.66
78.41
dots.mocr [ 68 ]
3 B
76.27
0.151
66.23
77.65
73.16
0.198
64.32
74.95
61.73
0.312
54.39
61.97
70.39
MinerU2.5-Pro [ 20 ]
1.2 B
75.87
0.222
65.14
84.68
71.77
0.272
61.79
80.73
62.56
0.375
52.70
72.47
70.07
Youtu-Parsing [ 4 ]
2.5 B
75.02
0.230
67.34
80.74
69.66
0.270
61.44
74.49
60.29
0.360
52.20
64.69
68.32
Table 3: Comparison with existing document parsing models and proprietary models on PureDocBench under clean and degraded conditions. Numbers are from the public leaderboard [ 43 ] . † Proprietary model. Bold and underline : best and second best of each Overall column.
Model
Chart
Geometry
Nat. image
Audio
Nat. video
Text-rich video
Avg.
Proprietary / API
Gemini-3-Pro [ 62 ]
92.79
85.43
69.73
76.74
74.82
76.80
77.44
GPT-5.4 [ 67 ]
90.92
81.16
60.50
–
–
–
–
Open-weight
Logics-Parsing-Omni [ 11 ]
94.65
76.34
67.53
75.58
67.34
65.36
72.26
Qwen3-Omni-30B-A3B [ 10 ]
87.07
58.60
59.35
75.23
65.19
59.59
66.44
Table 4: Comparison with open-weight and proprietary models on each split of OmniParsingBench. Scores are the per-split Overall (Subsec. 5.1 ); higher is better. Bold and underline : best and second best in each column. Avg. is the average over five columns, in which chart and geometry enter together as one graphics column. All systems are scored with the same toolkit. Perception and Cognition are in Tab. 5 – 10 .
Table 12: Comparison with existing specialized, general, and proprietary models on ChemOCR and PDMX-Synth. Left: ChemOCR ( 1,000 molecules; 990 scored after excluding 10 invalid references). Right: PDMX-Synth ( 800 score images). Bold and underline : best and second best in each column. ∗ ChemDFM-X [ 7 ] scores on ChemOCR are taken from TinyChemVL [ 41 ] ; LEGATO scores on PDMX-Synth are taken from its original report [ 8 ] .
Figure 8: Encoder architecture and initialization studies. (a) General multimodal (Avg-6); (b) OCRBench; (c) BLINK; (d) LibriSpeech test-clean WER. Blue, green, and pink denote the text-initialized unified encoder, randomly initialized unified encoder, and pre-trained SigLIP-2 [ 80 ] and SenseVoice-small [ 81 ] towers, respectively.
Figure 9: OSAD supplies 1.75 of the 1.99 points gained after SFT ( 88% ), with positive but diminishing increments. OmniDocBench v1.6 Overall (higher is better). (a) Score after SFT, RLVR, and each OSAD round; the dashed line marks TeleOCR, the strongest other system ( 96.91 ). (b) Gain over the preceding checkpoint.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Scope
Semantics
id
element
Unique element identifier.
category
element
Element type: text, table, formula, code, caption, header, footer, title, chemical, score, chart, flowchart, figure, seal, or geometric.
bbox
element
Bounding box of the element in page coordinates.
content.text
element
Transcription for text-like elements, code, chemical structures, and ABC notation.
content.latex
formula
Formula transcription in L a T e X .
content.otsl
table
Table structure and content in OTSL.
Appendix
Table 13: Annotation fields of the document parsing dataset and their semantics.
Field
Scope
Semantics
id
element
Unique element identifier.
category
element
Element type, fixed to entity .
bbox
element
Bounding box of a salient object.
tag
element
A word naming the object inside the bounding box.
content.text
element
OCR result inside the bounding box; "" when the annotation pipeline returns no text.
content.caption
element
Precise and concise caption of the region inside the bounding box.
Appendix
Table 14: Annotation fields of the natural image parsing dataset and their semantics.
Field
Scope
Semantics
id
segment
Unique segment identifier.
kind
segment
Segment type: vocal or nonvocal .
audio.asr_text
vocal segment
Verbatim transcription of the speech in the segment.
audio.speaker_id
vocal segment
Speaker identifier, consistent across the chunk.
audio.timbre_caption
vocal segment
Voice timbre: gender, age, pitch, breathiness, etc.
audio.instruct_caption
vocal segment
Speech delivery characteristics: emotion, pace, prosody, etc.
Appendix
Table 15: Annotation fields of the audio parsing dataset and their semantics.
Field
Scope
Category
Semantics
id
segment
shared
Unique segment identifier.
subtype
video
shared
Video type: natural_video or text_rich_video .
start/end_timestamp
segment
shared
Temporal boundaries relative to video onset.
visual.elements
segment
shared
Salient objects and on-screen text; each item has a category ( object or text ) and, for text, an ocr_type (e.g., title ).
visual.atomic_actions
segment
shared
Fine-grained actions within the segment.
visual.object_actor_interactions
segment
shared
Interactions between actors and objects.
Appendix
Table 16: Annotation fields of the video parsing dataset and their semantics.
Component
Quantity
Value
Youtu-Omni-Encoder
layers L
28
hidden size d
1,024
FFN hidden size dffn
3,072
query heads H / KV groups Hkv
16 / 8
head dimension dh
128
attention
GQA, bidirectional, packed
Appendix
Table 17: Complete configuration of Youtu-Parsing-Omni.
Perception
Parsing capability
Model
Encoder paradigm
Vision
Audio
Language model
Doc.
Image
Audio
Video
End-to-end
Params
MinerU2.5-Pro [ 20 ]
Modality-specific
ViT 0.6 B
–
Qwen2-0.5B
✓
–
–
–
–
1.2 B
PaddleOCR-VL-1.6 [ 6 ]
Modality-specific
ViT 0.4 B
–
ERNIE-4.5-0.3B
✓
–
–
–
–
0.9 B
TeleOCR [ 47 ]
Modality-specific
ViT 0.6 B
–
Qwen3-0.6B
✓
–
–
–
–
1.2 B
DeepSeek-OCR 2 [ 24 ]
Modality-specific
LLM 0.5 B
–
DeepSeek-3B-MoE
✓
–
–
–
✓
3 B
Youtu-Parsing [ 4 ]
Modality-specific
ViT 0.4 B
–
Youtu-LLM-2B
✓
–
–
–
–
2.5 B
Appendix
Table 18: Every other representative parser uses a separate encoder per modality, and only Logics-Parsing-Omni covers all three modalities, with two towers; Youtu-Parsing-Omni covers them with one encoder. Modality-specific : every modality has its own encoder; Unified : one encoder with shared weights for all modalities. End-to-end : a page or clip is parsed in one forward pass rather than by a layout-then-recognition pipeline.
Natural image
Graphics
Audio
Natural video
Text-rich video
Total
Original
1,000
1,000
1,014
601
259
3,874
Media excluded
0
0
76
51
9
136
Kept
1,000
1,000
938
550
250
3,738
Appendix
Table 19: OmniParsingBench sample counts within our evaluation scope. Natural video counts only the general subset.
Figure 10: Unified encoder versus separate towers. Green solid curves denote the randomly initialized unified encoder ( 0.4 B); pink dashed curves denote pre-trained SigLIP-2 [ 80 ] and SenseVoice-small [ 81 ] towers ( 0.6 B combined) with random projectors. Panels report individual vision benchmarks, the general multimodal average (Avg-6), and ASR error rates.
Figure 11: Encoder initialization on vision and ASR benchmarks. Blue solid curves denote the encoder initialized from Qwen3-0.6B-Base, and green dashed curves denote the encoder trained from scratch. Shaded bands indicate the performance gap, colored by the better-performing branch. Lower WER and CER indicate better ASR performance.
Figure 12: Document. A conventional single-column page: the parse recovers one linear reading order over headings, body paragraphs, and inline elements.
Figure 14: Document. A two-column page combining several element categories—text, formulas, tables, and figures. The parse has to assign every element to its column and serialize the two columns into a single reading order rather than interleaving them line by line.
Figure 16: Flowchart. Anomaly-Handling Workflow of a Medical System.
Figure 18: Audio. Sci-Fi Dialogue with Mechanical Sound Effects.
Figure 20: Natural Video. Raindrops Coalescing on a Glass Surface.
Figure 22: Geometry. Triangle with Perpendicular Foot and Angle Annotation.
Figure 25: Text-Rich Video. Data Analysis Problem-Solving Framework. Frames, timeline, and narrative; the report and segments follow on the next page.
Figure 26: Text-Rich Video. Data Analysis Problem-Solving Framework (continued).
Figure 27: Text-Rich Video. System Security Course on Attack Types and Threats. Frames, timeline, narrative, report, and the first segment; the last frame and the remaining segments follow on the next page.
Figure 28: Text-Rich Video. System Security Course on Attack Types and Threats (continued).
Addressing the challenges of fragmented task definitions and the heterogeneity of unstructured data in multimodal parsing, this paper proposes the Omni Parsing framework. This framework establishes a Unified Taxonomy covering documents, images, and audio-visual streams, introducing a progressive parsing paradigm that bridges perception and cognition. Specifically, the framework integrates three hierarchical levels: 1) Holistic Detection, which achieves precise spatial-temporal grounding of objects or events to establish a geometric baseline for perception; 2) Fine-grained Recognition, which performs symbolization (e.g., OCR/ASR) and attribute extraction on localized objects to complete structured entity parsing; and 3) Multi-level Interpreting, which constructs a reasoning chain from local semantics to global logic. A pivotal advantage of this framework is its evidence anchoring mechanism, which enforces a strict alignment between high-level semantic descriptions and low-level facts. This enables ``evidence-based'' logical induction, transforming unstructured signals into standardized knowledge that is locatable, enumerable, and traceable. Building on this foundation, we constructed a standardized dataset and released the Logics-Parsing-Omni model, which successfully converts complex audio-visual signals into machine-readable structured knowledge. Experiments demonstrate that fine-grained perception and high-level cognition are synergistic, effectively enhancing model reliability. Furthermore, to quantitatively evaluate these capabilities, we introduce OmniParsingBench. Code, models and the benchmark are released at https://github.com/alibaba/Logics-Parsing/tree/master/Logics-Parsing-Omni.
Recent advances in omni-modal large language models have enabled remarkable progress in joint vision-audio understanding. However, prevailing architectures rely on modality-specific encoders with a video-coarse, audio-dense design -- sampling visual frames at 1--2 fps while processing audio waveforms at 25 fps -- resulting in systems that perceive video frame by frame, modality by modality rather than holistically as humans do. Such a discrepancy leaves models with impoverished cross-modal interaction during encoding and an inability to capture fine-grained visual motion. To bridge this gap, we present Omni-Encoder, a unified Transformer backbone designed to co-embed visual and audio signals at a symmetrical 25 fps within a shared latent space. This architecture leverages three core innovations -- the Omni-Encoder Token Template, Omni-RoPE, and Temporal Window Shifting -- to effectively reconcile the dual challenges of modality disentanglement and computational efficiency. Experiments demonstrate that, compared to the modality-specific baseline Qwen2.5-Omni under the same input token budget to the LLM decoder, Omni-Encoder delivers substantial gains on visual continuous understanding tasks -- such as sign language recognition and fine-grained sports action analysis -- while maintaining competitive performance on established audio-visual benchmarks such as AVQA and Speaker Identification and Localization. These results suggest that unified omnivorous encoding offers a promising direction for building omni-modal models that more closely reflect the integrated nature of human perception.
Detao Bai, Shimin Yao, Weixuan Chen +4
1Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology
This paper presents Youtu-Parsing, an efficient and versatile document parsing model designed for high-performance content extraction. The architecture employs a native Vision Transformer (ViT) featuring a dynamic-resolution visual encoder to extract shared document features, coupled with a prompt-guided Youtu-LLM-2B language model for layout analysis and region-prompted decoding. Leveraging this decoupled and feature-reusable framework, we introduce a high-parallelism decoding strategy comprising two core components: token parallelism and query parallelism. The token parallelism strategy concurrently generates up to 64 candidate tokens per inference step, which are subsequently validated through a verification mechanism. This approach yields a 5--11x speedup over traditional autoregressive decoding and is particularly well-suited for highly structured scenarios, such as table recognition. To further exploit the advantages of region-prompted decoding, the query parallelism strategy enables simultaneous content prediction for multiple bounding boxes (up to five), providing an additional 2x acceleration while maintaining output quality equivalent to standard decoding. Youtu-Parsing encompasses a diverse range of document elements, including text, formulas, tables, charts, seals, and hierarchical structures. Furthermore, the model exhibits strong robustness when handling rare characters, multilingual text, and handwritten content. Extensive evaluations demonstrate that Youtu-Parsing achieves state-of-the-art (SOTA) performance on both the OmniDocBench and olmOCR-bench benchmarks. Overall, Youtu-Parsing demonstrates significant experimental value and practical utility for large-scale document intelligence applications.