We present Dynin-Omni, the first masked-diffusion-based omnimodal foundation model that unifies text, image, and speech understanding and generation, together with video understanding, within a single architecture. Unlike autoregressive unified models that serialize heterogeneous modalities, or compositional unified models that require orchestration with external modality-specific decoders, Dynin-Omni natively formulates omnimodal modeling as masked diffusion over a shared discrete token space, enabling iterative refinement under bidirectional context. Dynin-Omni adopts a multi-stage training strategy with model-merging-based modality expansion and omnimodal alignment. We evaluate Dynin-Omni across 19 multimodal benchmarks spanning language reasoning, image generation and editing, video understanding, and speech recognition and synthesis. Dynin-Omni achieves 87.6 on GSM8K, 1733.6 on MME-P, 61.4 on VideoMME, 0.87 on GenEval, and 2.1 WER on LibriSpeech test-clean, consistently outperforming existing open-source unified models while remaining competitive with strong modality-specific expert systems. These results demonstrate the potential of masked diffusion as a unified paradigm for any-to-any modeling, providing a flexible foundation for real-time omnimodal systems, unified cross-modal retrieval and generation, and embodied multimodal agents.
Figures & tables
Figure 1: Unified omnimodal performance of Dynin-Omni across understanding and generation benchmarks.
Figure 2: Overview of Dynin-Omni . Natively unified masked-diffusion backbone enables omnimodal understanding and generation across text, image, video, and speech within a single architecture, while keeping strong generation quality.
Understanding Tasks
Generation Tasks
Methods
Modeling
Textual Reasoning
Image
Video
Speech
Image
Image Editing
Speech
Perception-centric
EMOVA-7B [ 25 ]
AR
✓
✓
✓
VITA-1.5 [ 26 ]
AR
✓
✓
✓
✓
MiniCPM-o-2.6 [ 27 ]
AR
✓
✓
✓
✓
Baichuan-Omni-1.5 [ 28 ]
AR
✓
✓
✓
✓
✓
Table 1: Supported modality capabilities and architectural characteristics of representative perception-centric and unified perception–generation models. Diff. denotes diffusion-based generation and FM denotes flow-matching-based generation. Shaded cells indicate omnimodal capability (supporting text, vision, and speech concurrently) within each task group (Understanding or Generation). Entries written in parentheses, such as (+ Diff.), indicate auxiliary external generation modules attached to the primary architecture.
Figure 3: Architectural comparison of omnimodal modeling paradigms. (a) Perception-centric models mainly focus on text-centric multimodal understanding and reasoning, where generation tasks are typically limited. (b) Compositional unified models combine a shared backbone with external modality-specific expert generators, resulting in staged generation and heterogeneous objectives. (c) Dynin-Omni adopts a native unified formulation, enabling deeper cross-modal representation sharing and end-to-end token-level refinement.
Figure 4: Training and inference pipeline of Dynin-Omni . During training, text, vision, and speech inputs are discretized into a shared token space and optimized under a unified masked-diffusion objective with random masking. At inference, modality-specific parallel decoding strategies are applied (block-wise for text and speech, fully parallel for images), followed by deterministic detokenization, without relying on external generators.
Figure 5: Multi-stage training strategy of Dynin-Omni . Stage 1 performs text-centric modality adaptation for newly introduced modalities. We then apply modality-disentangled model merging to retain backbone knowledge while initializing Stage 2. Stage 2 conducts native omnimodal supervised fine-tuning under a unified token objective, enabling combined omnimodal training and flexible-length generation. Finally, Stage 3 enhances advanced capabilities such as reasoning, high-resolution image generation, video QA, and extended speech synthesis.
Table 2: Input–output template families across training stages. Red tokens denote masked target responses that are supervised throughout the stages. Blue tokens indicate the global termination token <EOS> , which also serves as padding token in masked diffusion language models, and is supervised only from Stage 2 onward.
Figure 6: Training loss over the first 20K steps of Stage 2 for text-to-text (T→T), text-to-image (T→I), and video-to-text (V→T). T→T and T→I are native to the backbone, while V→T is introduced in Stage 1. w/o Model Merging denotes directly performing omnimodal SFT, whereas Merging 4:6 and Merging 6:4 indicate linear modality disentangled merging between the backbone and Stage-1 model. Model merging consistently accelerates convergence and leads to lower final loss across all modalities. Notably, for V→T—unsupported by the original backbone— the 6:4 merging strategy enables stable adaptation from high initial loss to low-loss convergence.
Figure 7: Impact of stage-aware <EOS> supervision on ASR and TTS Word Error Rate (WER).
Figure 8: Distribution of training data across stages and tasks. Each stage is represented by a pair of horizontal bars: the upper bar shows the composition of training tasks ( e.g. Text→Text, Image→Text), while the lower bar depicts the proportion of real and synthetic data used in that stage. Percentages indicate the relative contribution of each task within a stage.
Model
MMLU [ 110 ]
ARC-C [ 111 ]
GSM8K [ 112 ]
MATH [ 113 ]
GPQA [ 114 ]
Text-Experts
Llama-3-8B [ 115 ]
66.6
-
55.3
20.5
25.8
Ministral-3-8B [ 116 ]
76.1
88.0
-
62.6
-
Qwen3-8B [ 77 ]
76.9
-
89.8
60.8
44.4
LLaDA-8B [ 17 ]
65.9
47.9
70.7
27.3
26.1
Trida-7B [ 117 ]
67.2
-
65.1
33.6
-
Table 3: Evaluation on textual reasoning benchmarks (higher is better for all metrics). ∗ Indicates support for video generation. † Indicates native speech input/output capability within the unified architecture. ‡ Indicates results reproduced.
Model
POPE [ 61 ]
MME-P [ 118 ]
GQA [ 119 ]
MMMU [ 60 ]
MMB [ 120 ]
MMU-Experts
InternVL3-8B [ 121 ]
90.4
1748.4
–
62.2
83.6
Ovis2-8B [ 58 ]
88.6
1628.1
–
57.4
84.8
Qwen2.5-VL-7B [ 122 ]
85.9
1698.1
–
58.0
83.2
LLaDA-V [ 54 ]
–
1507.0
–
48.6
82.9
Perception-centric
Table 4: Evaluation on multimodal understanding (MMU) benchmarks ( image ). Higher is better for all metrics. ∗ Indicates support for video generation. † Indicates native speech input/output capability within the unified architecture. ‡ Indicates results reproduced.
Figure 9: Qualitative examples of Dynin-Omni for textual reasoning (top-left), image understanding (top-right), and video understanding (bottom).
Model
GenEval [ 129 ]
DPG-Bench [ 130 ]
Single Obj.
Two Obj.
Count.
Colors
Pos.
Color Attr.
Overall
Generation-Experts
HiDream-I1-Full [ 131 ]
1.00
0.98
0.79
0.91
0.60
0.72
0.83
85.9
FLUX.1 [Dev] [ 64 ]
0.98
0.81
0.74
0.79
0.22
0.45
0.66
83.8
Qwen-Image [ 107 ]
0.99
0.92
0.89
0.88
0.76
0.77
0.87
88.3
GPT-Image 1 [High] [ 132 ]
0.99
0.92
0.85
0.92
0.75
0.61
0.84
85.2
Table 6: Evaluation on image generation benchmarks (higher is better for all metrics). ∗ Indicates support for video generation. † Indicates native speech input/output capability within the unified architecture. ‡ Indicates results reproduced.
Model
Add
Adjust
Extract
Replace
Remove
Background
Style
Action
Overall
Editing-Experts
Step1X-Edit [ 135 ]
3.88
3.14
1.76
3.40
2.41
3.16
4.63
2.52
3.11
OmniGen2 [ 136 ]
3.57
3.06
1.77
3.74
3.20
3.57
4.81
4.68
3.55
Flux.1 Kontext [Pro] [ 106 ]
3.76
3.45
2.15
3.98
2.94
3.78
4.38
4.26
3.59
GPT-Image 1 [High] [ 132 ]
4.61
4.33
2.90
4.35
3.66
4.57
4.93
4.89
4.28
Qwen-Image [ 107 ]
4.38
4.16
3.43
4.66
4.14
4.38
4.81
4.69
4.33
Table 7: Evaluation on the ImgEdit benchmark [ 134 ] . Higher is better. All metrics are evaluated using GPT-4.1. Overall denotes the average score across all tasks. † Indicates native speech input/output capability within the unified architecture.
Table 8: Evaluation on automatic speech recognition (ASR) and text-to-speech (TTS) benchmarks. Lower is better. All metrics report word error rate (WER) and are evaluated using Whisper-large-v3 [ 137 ] . ∗ Indicates support for video generation. ‡ Indicates results reproduced.
Figure 17
Figure 11: Effect of diffusion sampling steps on reasoning and generation tasks, illustrating that larger refinement benefits reasoning while moderate steps suffice for high-quality image generation and editing.
Table 10: Detailed data composition across training stages for Dynin-Omni . Tasks are described using unified input–output directions (e.g., V → T denotes video-to-text). Synthetic indicate the internally generated data.
Hyperparameters
Stage 1
Stage 2
Stage 3
(New Modality Adapt.)
(Omni. SFT)
(Cont. SFT)
Learning Rate
1.0×10−5
2.0×10−5
2.0×10−5
LR Scheduler
Cosine
Cosine
Cosine
Weight Decay
0.1
0.1
0.1
Gradient Norm Clip
1.0
1.0
1.0
Optimizer
AdamW ( β1=0.9 , β2=0.999 )
Appendix
Table 11: Detailed hyperparameters and configurations of the training recipe across different stages.
Figure 12: Comparison of model merging strategies under vocabulary extension. (a) Shared Merging: Original vocabulary parameters are interpolated between the backbone and Stage 1, while new speech embeddings come from Stage 1. (b) Stage 1-Only Merging: All vocabulary-related parameters are taken from Stage 1. (c) Modality-Disentangled Merging (Ours): Original vocabulary parameters remain from the backbone, while new speech-specific dimensions come from Stage 1. Remaining backbone weights are interpolated.
While recent multimodal large language models (MLLMs) have made impressive strides, they predominantly employ a conventional autoregressive architecture as their backbone, leaving significant room to explore effective and efficient alternatives in architectural design. Concurrently, recent studies have successfully applied discrete diffusion models to various domains, such as visual understanding and image generation, revealing their considerable potential as a promising backbone for multimodal systems. Drawing inspiration from these pioneering studies, we introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models, which unifies understanding and generation across text, speech, and images. Omni-Diffusion employs a unified mask-based discrete diffusion model to directly capture the joint distribution over discrete multimodal tokens. This approach supports not only bimodal tasks but also more complex scenarios involving multiple modalities. On a diverse set of benchmarks, our method outperforms or performs on par with existing multimodal systems that process two or more modalities, highlighting the significant promise of diffusion models in powering the next generation of multimodal foundation models. Project webpage: https://omni-diffusion.github.io.
Unified multimodal diffusion large language models (dLLMs) offer a single architecture for both image generation and multimodal understanding, but their iterative decoding requires tens to hundreds of forward passes. Existing few-step distillation methods largely focus on either image generation or text generation, making it unclear how to compress a fully discrete multimodal dLLM into a single efficient student while preserving both generation and understanding. We introduce Omni-Diffusion-Distill, a unified two-stage distillation framework that retains strong generation and understanding capabilities while substantially reducing the inference cost of a unified multimodal dLLM. Omni-Diffusion-Distill aligns the distillation of both generation and understanding, for both images and text, in the discrete token space. In the first stage, the student is trained to skip decoding steps by replaying cached teacher trajectories, and in the second stage the student is refined on intermediate states along its own rollouts. We further remedy two sources of degradation in unified distillation with a pairwise collision penalty that reduces repetition under parallel text decoding, and entropy-matched guidance that prevents entropy collapse caused by fitting the sharpened teacher distribution in image generation. Omni-Diffusion-Distill achieves state-of-the-art trade-offs between decoding efficiency and generation and understanding performance for multimodal dLLMs, reducing image generation from 128 to 8 decoding steps and multimodal understanding from 512 to 64, giving 18.2x and 21.2x wall-clock speedups. Under these budgets, it scores 0.828 on GenEval and 83.0 on DPG-Bench for text-to-image generation, while reaching GPT judge scores of 20.0 on MM-Vet and 57.2 on COCO captioning (twice the teacher's 28.4 at the same steps) for multimodal understanding.
We present LLaDA-o, an effective and length-adaptive omni diffusion model for multimodal understanding and generation. LLaDA-o is built on a Mixture of Diffusion (MoD) framework that decouples discrete masked diffusion for text understanding and continuous diffusion for visual generation, while coupling them through a shared, simple, and efficient attention backbone that reduces redundant computation for fixed conditions. Building on MoD, we further introduce a data-centric length adaptation strategy that enables flexible-length decoding in multimodal settings without architectural changes. Extensive experiments show that LLaDA-o achieves state-of-the-art performance among omni-diffusion models on multimodal understanding and generation benchmarks, and reaches 87.04 on DPG-Bench for text-to-image generation, supporting the effectiveness of unified omni diffusion modeling. Code is available at https://github.com/ML-GSAI/LLaDA-o.
Zebin You, Xiaolu Zhang, Jun Zhou +2
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +1