Modaliff: A Unified Token Diffusion Model for Any-to-Any Generation

Duc Kieu; Kien Do; Minh Khoa Le; Tung Kieu; Dang Nguyen; Thin Nguyen

cs.CVOct 18, 2026

Modaliff: A Unified Token Diffusion Model for Any-to-Any Generation

Authors: Duc Kieu, Kien Do, Minh Khoa Le, Tung Kieu, Dang Nguyen, Thin Nguyen

Organizations: Applied Artificial Intelligence Initiative, Deakin University · VinSmart Future · Aalborg University

Abstract

Any-to-any generation seeks a single model that translates flexibly among text, image, and audio, yet existing approaches either require fully aligned multimodal tuples or a shared representation that suppresses modality-specific structure. We introduce Modaliff, a novel token-based diffusion model that learns from widely available bimodal pairs while preserving each modality in its native latent sequence. Modaliff assembles conditions and targets into a unified token sequence, with per-token attributes to disambiguate token identities and role-based positional embedding to preserve each modality's internal structure across sequence layouts. This design enables a single denoiser to support diverse generation tasks without task-specific architectural changes or a shared cross-modal latent space. Beyond one-to-one generation, Modaliff supports multiple conditions either by combining single-condition predictions at inference or by lightweight finetuning on captions that jointly describe multiple inputs. Experiments across text, image, and audio show that Modaliff achieves the best fidelity among any-to-any models on most generation tasks while using substantially fewer parameters and training data.

Figures & tables

Panel of Figure 1 — Figure 1: Fine-grain illustration of our proposed Modaliff.

Task	T $\to$ I	A $\to$ I	I $\to$ T	A $\to$ T	T $\to$ A	I $\to$ A
Method	FID $\downarrow$		CIDEr $\uparrow$		FAD $\downarrow$
SD3-Medium	25.40	-	-	-	-	-
Sound2Vision	-	42.55	-	-	-	-
LLaVA-Next	-	-	109.3	-	-	-
Qwen2-Audio	-	-	-	4.64	-	-
AudioLDM-L	-	-	-	-	1.96	-

Table 1: Fidelity metrics for cross-modal generation. * indicates results that we reproduced using the official repository.

Task	T $\to$ I	I $\to$ T	A $\to$ T	T $\to$ A	A $\to$ I	I $\to$ A
Method	CLIP $\uparrow$		CLAP $\uparrow$		ImgBind $\uparrow$
SD3-Medium	31.60	-	-	-	-	-
Sound2Vision	-	-	-	-	26.71	-
LLaVA-Next	-	32.14	-	-	-	-
Qwen2-Audio	-	-	17.09	-	-	-
AudioLDM-L	-	-	-	36.05	-	-

Table 2: Alignment scores for cross-modal generation. * indicates results that we reproduced using the official repository.

Figure 2: Qualitative results of Modaliff and generalists on (T + A) $\to$ I and (I + A) $\to$ T.

Method	T-I	A-I	I-T	A-T	T-A	I-A
	(T+A) $\to$ I		(I+A) $\to$ T		(T+I) $\to$ A
CoDi	22.97	22.93	15.05	19.51	8.21	3.30
FlowBind	27.71	20.82	20.21	20.54	12.92	9.60
Modaliff (CFG)	23.39	20.93	21.92	17.13	20.89	13.89
Modaliff (Finetuned)	29.41	16.36	24.28	18.35	22.10	14.08

Table 3: ImgBind scores for multi-condition generation. This capability is unavailable in OmniFlow’s released code.

Figure 3: Qualitative results of Modaliff and generalists on (T + I) $\to$ I.

Panel of Figure 4 — Figure 4: Learning curves comparing role-based positional encoding and local indexing.

$\gamma_{\text{min}}$	T-I	A-I	I-T	A-T	T-A	I-A
	(T+A) $\to$ I		(I+A) $\to$ T		(T+I) $\to$ A
0	26.80	16.42	23.23	14.98	16.28	12.71
0.2	27.13	16.36	23.55	14.35	16.43	12.33
0.4	27.25	15.60	23.90	13.65	16.84	11.78
0.6	27.78	15.37	23.98	12.13	17.04	11.20

Table 4: ImgBind score between the target and each conditions with

\gamma_{\text{min}}\in\left\{0,0.2,0.4,0.6\right\}

and

\gamma_{\text{max}}=0.8

. Each setting finetunes on 10K samples.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 5 — Table 5: Comparison between the components for each modality used in our methods and baselines.

Multimodal Generation

Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.

See all 243 papers

MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation
Jun 15, 2026Kyeongmin Yeo, Yunhong Min, Minhyuk SungLatent Diffusion Models Multimodal Generation
KAIST
MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation
Oct 7, 2026Kyeongmin Yeo, Minhyuk SungMultimodal Generation Unified Multimodal Models
KAIST
Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
Mar 6, 2026Lijiang Li, Zuwei Long, Yunhang Shen +6Masked Diffusion Models Unified Multimodal Models
Nanjing University · Tencent Youtu Lab · CASIA

Jun 15, 2026J/K move · Enter open · S save

MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation

Kyeongmin Yeo, Yunhong Min, Minhyuk Sung

KAIST

Latent Diffusion Models Multimodal Generation Multimodal Learning Multimodal Diffusion Models

Open paper AskPDF arXiv

We introduce MUNI, an end-to-end multimodal latent diffusion framework for any-to-any generation that unifies subset-conditioned cross-modal generation and unconditional joint sampling through a shared stochastic latent. Existing multimodal generative models are largely LLM-based, which limits leveraging modality-specific generators and requires text-paired data for training. Recent diffusion- and flow-based any-to-any extensions take a different direction but still rely on text-aligned embeddings, fully-paired training, or matched-dimensionality deterministic mappings. MUNI rests on two complementary contributions, one architectural and one in the training objective. First, we extend latent diffusion to multimodal any-to-any generation end-to-end: instead of the standard two-stage recipe that precomputes a frozen latent space and then fits a prior over it, MUNI jointly trains modality-specific encoders, expressive decoders, and a single shared flow-based prior under one objective. Second, we identify that the standard aggregation rules of multimodal variational inference are insufficient once coupled with a learned prior and expressive decoders. A suitable shared latent must simultaneously satisfy coherence across generated modalities, predictive sufficiency of subset latents, and minimality of the latent content. We propose a routed training objective whose structural choices align the latent with these criteria and admit a minimal-sufficiency characterization in the realizable setting. Experiments on PolyMNIST-Quadrant-Labels and a large-scale image-text-audio benchmark show MUNI matching or exceeding the strongest baselines on conditional generation while opening its largest margins on unconditional coherence. Project page: https://muni-proj.github.io/.

Abstract

Multimodal Generation

Explore similar work

MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation

MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation

Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion