Dynin-Omni: Omnimodal Unified Large Diffusion Language Model

Jaeik Kim; Woojin Kim; Jihwan Hong; Yejoon Lee; Sieun Hyeon; Mintaek Lim; Yunseok Han; Dogeun Kim

cs.CLMar 9, 2026

Dynin-Omni: Omnimodal Unified Large Diffusion Language Model

Authors: Jaeik Kim, Woojin Kim, Jihwan Hong, Yejoon Lee, Sieun Hyeon, Mintaek Lim, Yunseok Han, Dogeun Kim,

Organizations: AIDAS Lab Seoul National University

Abstract

We present Dynin-Omni, the first masked-diffusion-based omnimodal foundation model that unifies text, image, and speech understanding and generation, together with video understanding, within a single architecture. Unlike autoregressive unified models that serialize heterogeneous modalities, or compositional unified models that require orchestration with external modality-specific decoders, Dynin-Omni natively formulates omnimodal modeling as masked diffusion over a shared discrete token space, enabling iterative refinement under bidirectional context. Dynin-Omni adopts a multi-stage training strategy with model-merging-based modality expansion and omnimodal alignment. We evaluate Dynin-Omni across 19 multimodal benchmarks spanning language reasoning, image generation and editing, video understanding, and speech recognition and synthesis. Dynin-Omni achieves 87.6 on GSM8K, 1733.6 on MME-P, 61.4 on VideoMME, 0.87 on GenEval, and 2.1 WER on LibriSpeech test-clean, consistently outperforming existing open-source unified models while remaining competitive with strong modality-specific expert systems. These results demonstrate the potential of masked diffusion as a unified paradigm for any-to-any modeling, providing a flexible foundation for real-time omnimodal systems, unified cross-modal retrieval and generation, and embodied multimodal agents.

Figures & tables

Figure 1: Unified omnimodal performance of Dynin-Omni across understanding and generation benchmarks.

Figure 2: Overview of Dynin-Omni . Natively unified masked-diffusion backbone enables omnimodal understanding and generation across text, image, video, and speech within a single architecture, while keeping strong generation quality.

		Understanding Tasks				Generation Tasks
Methods	Modeling	Textual Reasoning	Image	Video	Speech	Image	Image Editing	Speech
Perception-centric
EMOVA-7B [ 25 ]	AR		✓		✓			✓
VITA-1.5 [ 26 ]	AR		✓	✓	✓			✓
MiniCPM-o-2.6 [ 27 ]	AR		✓	✓	✓			✓
Baichuan-Omni-1.5 [ 28 ]	AR	✓	✓	✓	✓			✓

Table 1: Supported modality capabilities and architectural characteristics of representative perception-centric and unified perception–generation models. Diff. denotes diffusion-based generation and FM denotes flow-matching-based generation. Shaded cells indicate omnimodal capability (supporting text, vision, and speech concurrently) within each task group (Understanding or Generation). Entries written in parentheses, such as (+ Diff.), indicate auxiliary external generation modules attached to the primary architecture.

Figure 3: Architectural comparison of omnimodal modeling paradigms. (a) Perception-centric models mainly focus on text-centric multimodal understanding and reasoning, where generation tasks are typically limited. (b) Compositional unified models combine a shared backbone with external modality-specific expert generators, resulting in staged generation and heterogeneous objectives. (c) Dynin-Omni adopts a native unified formulation, enabling deeper cross-modal representation sharing and end-to-end token-level refinement.

Figure 4: Training and inference pipeline of Dynin-Omni . During training, text, vision, and speech inputs are discretized into a shared token space and optimized under a unified masked-diffusion objective with random masking. At inference, modality-specific parallel decoding strategies are applied (block-wise for text and speech, fully parallel for images), followed by deterministic detokenization, without relying on external generators.

Figure 5: Multi-stage training strategy of Dynin-Omni . Stage 1 performs text-centric modality adaptation for newly introduced modalities. We then apply modality-disentangled model merging to retain backbone knowledge while initializing Stage 2. Stage 2 conducts native omnimodal supervised fine-tuning under a unified token objective, enabling combined omnimodal training and flexible-length generation. Finally, Stage 3 enhances advanced capabilities such as reasoning, high-resolution image generation, video QA, and extended speech synthesis.

Template Family	S1	S2	S3	Input-Output Template
Video $\rightarrow$ Text	✓	✓	✓	$\underbrace{\texttt{<image> \ldots<image>}}_{\text{video frames}}\;\texttt{<\|user\|>}\;\{\text{prompt}\}\;\texttt{<\|assistant\|>}\;{\color[rgb]{1,0,0}\{\text{response}\}}\;{\color[rgb]{0,0,1}\texttt{<EOS>}}$
Speech $\rightarrow$ Text	✓	✓	✓	$\texttt{<\|startofspeech\|>}\;\{\text{speech}\}\;\texttt{<\|endofspeech\|>}\;\texttt{<\|startoftext\|>}\;{\color[rgb]{1,0,0}\{\text{text}\}}\;{\color[rgb]{0,0,1}\texttt{<EOS>}}$
Text $\rightarrow$ Speech	✓	✓	✓	$\texttt{<\|startoftext\|>}\;\{\text{text}\}\;\texttt{<\|endoftext\|>}\;\texttt{<\|startofspeech\|>}\;{\color[rgb]{1,0,0}\{\text{speech}\}}\;\texttt{<\|endofspeech\|>}\;{\color[rgb]{0,0,1}\texttt{<EOS>}}$
Text Chat		✓	✓	$\texttt{<\|user\|>}\;\{\text{prompt}\}\;\texttt{<\|assistant\|>}\;{\color[rgb]{1,0,0}\{\text{response}\}}\;{\color[rgb]{0,0,1}\texttt{<EOS>}}$
Image $\rightarrow$ Text		✓	✓	$\texttt{<image>}\;\texttt{<\|user\|>}\;\{\text{prompt}\}\;\texttt{<\|assistant\|>};{\color[rgb]{1,0,0}\{\text{response}\}}\;{\color[rgb]{0,0,1}\texttt{<EOS>}}$
Text $\rightarrow$ Image		✓	✓	$\texttt{<\|startoftext\|>}\;\{\text{instruction}\}\;\texttt{<\|endoftext\|>}$ {image}

Table 2: Input–output template families across training stages. Red tokens denote masked target responses that are supervised throughout the stages. Blue tokens indicate the global termination token <EOS> , which also serves as padding token in masked diffusion language models, and is supervised only from Stage 2 onward.

Figure 6: Training loss over the first 20K steps of Stage 2 for text-to-text (T→T), text-to-image (T→I), and video-to-text (V→T). T→T and T→I are native to the backbone, while V→T is introduced in Stage 1. w/o Model Merging denotes directly performing omnimodal SFT, whereas Merging 4:6 and Merging 6:4 indicate linear modality disentangled merging between the backbone and Stage-1 model. Model merging consistently accelerates convergence and leads to lower final loss across all modalities. Notably, for V→T—unsupported by the original backbone— the 6:4 merging strategy enables stable adaptation from high initial loss to low-loss convergence.

Figure 7: Impact of stage-aware <EOS> supervision on ASR and TTS Word Error Rate (WER).

Figure 8: Distribution of training data across stages and tasks. Each stage is represented by a pair of horizontal bars: the upper bar shows the composition of training tasks ( e.g. Text→Text, Image→Text), while the lower bar depicts the proportion of real and synthetic data used in that stage. Percentages indicate the relative contribution of each task within a stage.

Model	MMLU [ 110 ]	ARC-C [ 111 ]	GSM8K [ 112 ]	MATH [ 113 ]	GPQA [ 114 ]
Text-Experts
Llama-3-8B [ 115 ]	66.6	-	55.3	20.5	25.8
Ministral-3-8B [ 116 ]	76.1	88.0	-	62.6	-
Qwen3-8B [ 77 ]	76.9	-	89.8	60.8	44.4
LLaDA-8B [ 17 ]	65.9	47.9	70.7	27.3	26.1
Trida-7B [ 117 ]	67.2	-	65.1	33.6	-

Table 3: Evaluation on textual reasoning benchmarks (higher is better for all metrics). ∗ Indicates support for video generation. † Indicates native speech input/output capability within the unified architecture. ‡ Indicates results reproduced.

Model	POPE [ 61 ]	MME-P [ 118 ]	GQA [ 119 ]	MMMU [ 60 ]	MMB [ 120 ]
MMU-Experts
InternVL3-8B [ 121 ]	90.4	1748.4	–	62.2	83.6
Ovis2-8B [ 58 ]	88.6	1628.1	–	57.4	84.8
Qwen2.5-VL-7B [ 122 ]	85.9	1698.1	–	58.0	83.2
LLaDA-V [ 54 ]	–	1507.0	–	48.6	82.9
Perception-centric

Table 4: Evaluation on multimodal understanding (MMU) benchmarks ( image ). Higher is better for all metrics. ∗ Indicates support for video generation. † Indicates native speech input/output capability within the unified architecture. ‡ Indicates results reproduced.

Figure 9: Qualitative examples of Dynin-Omni for textual reasoning (top-left), image understanding (top-right), and video understanding (bottom).

Model	GenEval [ 129 ]							DPG-Bench [ 130 ]
	Single Obj.	Two Obj.	Count.	Colors	Pos.	Color Attr.	Overall
Generation-Experts
HiDream-I1-Full [ 131 ]	1.00	0.98	0.79	0.91	0.60	0.72	0.83	85.9
FLUX.1 [Dev] [ 64 ]	0.98	0.81	0.74	0.79	0.22	0.45	0.66	83.8
Qwen-Image [ 107 ]	0.99	0.92	0.89	0.88	0.76	0.77	0.87	88.3
GPT-Image 1 [High] [ 132 ]	0.99	0.92	0.85	0.92	0.75	0.61	0.84	85.2

Table 6: Evaluation on image generation benchmarks (higher is better for all metrics). ∗ Indicates support for video generation. † Indicates native speech input/output capability within the unified architecture. ‡ Indicates results reproduced.

Model	Add	Adjust	Extract	Replace	Remove	Background	Style	Action	Overall
Editing-Experts
Step1X-Edit [ 135 ]	3.88	3.14	1.76	3.40	2.41	3.16	4.63	2.52	3.11
OmniGen2 [ 136 ]	3.57	3.06	1.77	3.74	3.20	3.57	4.81	4.68	3.55
Flux.1 Kontext [Pro] [ 106 ]	3.76	3.45	2.15	3.98	2.94	3.78	4.38	4.26	3.59
GPT-Image 1 [High] [ 132 ]	4.61	4.33	2.90	4.35	3.66	4.57	4.93	4.89	4.28
Qwen-Image [ 107 ]	4.38	4.16	3.43	4.66	4.14	4.38	4.81	4.69	4.33

Table 7: Evaluation on the ImgEdit benchmark [ 134 ] . Higher is better. All metrics are evaluated using GPT-4.1. Overall denotes the average score across all tasks. † Indicates native speech input/output capability within the unified architecture.

Table 8: Evaluation on automatic speech recognition (ASR) and text-to-speech (TTS) benchmarks. Lower is better. All metrics report word error rate (WER) and are evaluated using Whisper-large-v3 [ 137 ] . ∗ Indicates support for video generation. ‡ Indicates results reproduced.

Figure 11: Effect of diffusion sampling steps on reasoning and generation tasks, illustrating that larger refinement benefits reasoning while moderate steps suffice for high-quality image generation and editing.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Task	Dataset Source	Scale (# pair)
Stage 1	V $\rightarrow$ T	WebVid-10M	10.7M
	S $\rightarrow$ T	GigaSpeech; LibriSpeech; Common Voice	10M
	T $\rightarrow$ S	GigaSpeech; LibriSpeech; Common Voice	10M
Stage 2	T $\rightarrow$ T	Evol-Instruct; Magpie-Pro; Open-Platypus; OpenR1-Math; OpenHermes-2.5	1.6M
	I $\rightarrow$ T	Mantis-Instruct; Cambrian10M	8M
	T $\rightarrow$ I	JourneyDB; FLUX-Reason-6M; PickaPic	10M

Appendix

Table 10: Detailed data composition across training stages for Dynin-Omni . Tasks are described using unified input–output directions (e.g., V

\rightarrow

T denotes video-to-text). Synthetic indicate the internally generated data.

Hyperparameters	Stage 1	Stage 2	Stage 3
	(New Modality Adapt.)	(Omni. SFT)	(Cont. SFT)
Learning Rate	$1.0\times 10^{-5}$	$2.0\times 10^{-5}$	$2.0\times 10^{-5}$
LR Scheduler	Cosine	Cosine	Cosine
Weight Decay	0.1	0.1	0.1
Gradient Norm Clip	1.0	1.0	1.0
Optimizer	AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.999$ )

Appendix

Table 11: Detailed hyperparameters and configurations of the training recipe across different stages.

Figure 12: Comparison of model merging strategies under vocabulary extension. (a) Shared Merging: Original vocabulary parameters are interpolated between the backbone and Stage 1, while new speech embeddings come from Stage 1. (b) Stage 1-Only Merging: All vocabulary-related parameters are taken from Stage 1. (c) Modality-Disentangled Merging (Ours): Original vocabulary parameters remain from the backbone, while new speech-specific dimensions come from Stage 1. Remaining backbone weights are interpolated.

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 290 papers

Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
Mar 6, 2026Lijiang Li, Zuwei Long, Yunhang Shen +6Masked Diffusion Models Unified Multimodal Models
Nanjing University · Tencent Youtu Lab · CASIA
Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Models
Oct 7, 2026Hong Huang, Chenhongyi Yang, Junzhe Sun +3Diffusion Language Model Distillation Diffusion Model Inference Acceleration
Simon Fraser University · Meta
LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Model
Mar 1, 2026Zebin You, Xiaolu Zhang, Jun Zhou +2Diffusion Models Multimodal Diffusion Models
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +1

Mar 6, 2026J/K move · Enter open · S save

Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion

Lijiang Li, Zuwei Long, Yunhang Shen, Heting Gao+5

Nanjing University · Tencent Youtu Lab · CASIA

Masked Diffusion Models Unified Multimodal Models Multimodal Generation Multimodal Foundation Models Multimodal Learning Diffusion Models+2

Open paper AskPDF arXiv

While recent multimodal large language models (MLLMs) have made impressive strides, they predominantly employ a conventional autoregressive architecture as their backbone, leaving significant room to explore effective and efficient alternatives in architectural design. Concurrently, recent studies have successfully applied discrete diffusion models to various domains, such as visual understanding and image generation, revealing their considerable potential as a promising backbone for multimodal systems. Drawing inspiration from these pioneering studies, we introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models, which unifies understanding and generation across text, speech, and images. Omni-Diffusion employs a unified mask-based discrete diffusion model to directly capture the joint distribution over discrete multimodal tokens. This approach supports not only bimodal tasks but also more complex scenarios involving multiple modalities. On a diverse set of benchmarks, our method outperforms or performs on par with existing multimodal systems that process two or more modalities, highlighting the significant promise of diffusion models in powering the next generation of multimodal foundation models. Project webpage: https://omni-diffusion.github.io.

Abstract

Unified Multimodal Models

Explore similar work

Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion

Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Models

LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Model