Any-to-any generation seeks a single model that translates flexibly among text, image, and audio, yet existing approaches either require fully aligned multimodal tuples or a shared representation that suppresses modality-specific structure. We introduce Modaliff, a novel token-based diffusion model that learns from widely available bimodal pairs while preserving each modality in its native latent sequence. Modaliff assembles conditions and targets into a unified token sequence, with per-token attributes to disambiguate token identities and role-based positional embedding to preserve each modality's internal structure across sequence layouts. This design enables a single denoiser to support diverse generation tasks without task-specific architectural changes or a shared cross-modal latent space. Beyond one-to-one generation, Modaliff supports multiple conditions either by combining single-condition predictions at inference or by lightweight finetuning on captions that jointly describe multiple inputs. Experiments across text, image, and audio show that Modaliff achieves the best fidelity among any-to-any models on most generation tasks while using substantially fewer parameters and training data.
Figures & tables
Figure 1: Fine-grain illustration of our proposed Modaliff.
Task
T → I
A → I
I → T
A → T
T → A
I → A
Method
FID ↓
CIDEr ↑
FAD ↓
SD3-Medium
25.40
-
-
-
-
-
Sound2Vision
-
42.55
-
-
-
-
LLaVA-Next
-
-
109.3
-
-
-
Qwen2-Audio
-
-
-
4.64
-
-
AudioLDM-L
-
-
-
-
1.96
-
Table 1: Fidelity metrics for cross-modal generation. * indicates results that we reproduced using the official repository.
Task
T → I
I → T
A → T
T → A
A → I
I → A
Method
CLIP ↑
CLAP ↑
ImgBind ↑
SD3-Medium
31.60
-
-
-
-
-
Sound2Vision
-
-
-
-
26.71
-
LLaVA-Next
-
32.14
-
-
-
-
Qwen2-Audio
-
-
17.09
-
-
-
AudioLDM-L
-
-
-
36.05
-
-
Table 2: Alignment scores for cross-modal generation. * indicates results that we reproduced using the official repository.
Figure 2: Qualitative results of Modaliff and generalists on (T + A) → I and (I + A) → T.
(T+A) → I
(I+A) → T
(T+I) → A
Method
T-I
A-I
I-T
A-T
T-A
I-A
CoDi
22.97
22.93
15.05
19.51
8.21
3.30
FlowBind
27.71
20.82
20.21
20.54
12.92
9.60
Modaliff (CFG)
23.39
20.93
21.92
17.13
20.89
13.89
Modaliff (Finetuned)
29.41
16.36
24.28
18.35
22.10
14.08
Table 3: ImgBind scores for multi-condition generation. This capability is unavailable in OmniFlow’s released code.
Figure 3: Qualitative results of Modaliff and generalists on (T + I) → I.
Figure 4: Learning curves comparing role-based positional encoding and local indexing.
(T+A) → I
(I+A) → T
(T+I) → A
γmin
T-I
A-I
I-T
A-T
T-A
I-A
0
26.80
16.42
23.23
14.98
16.28
12.71
0.2
27.13
16.36
23.55
14.35
16.43
12.33
0.4
27.25
15.60
23.90
13.65
16.84
11.78
0.6
27.78
15.37
23.98
12.13
17.04
11.20
Table 4: ImgBind score between the target and each conditions with γmin∈{0,0.2,0.4,0.6} and γmax=0.8 . Each setting finetunes on 10K samples.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Table 5: Comparison between the components for each modality used in our methods and baselines.
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.
We introduce MUNI, an end-to-end multimodal latent diffusion framework for any-to-any generation that unifies subset-conditioned cross-modal generation and unconditional joint sampling through a shared stochastic latent. Existing multimodal generative models are largely LLM-based, which limits leveraging modality-specific generators and requires text-paired data for training. Recent diffusion- and flow-based any-to-any extensions take a different direction but still rely on text-aligned embeddings, fully-paired training, or matched-dimensionality deterministic mappings. MUNI rests on two complementary contributions, one architectural and one in the training objective. First, we extend latent diffusion to multimodal any-to-any generation end-to-end: instead of the standard two-stage recipe that precomputes a frozen latent space and then fits a prior over it, MUNI jointly trains modality-specific encoders, expressive decoders, and a single shared flow-based prior under one objective. Second, we identify that the standard aggregation rules of multimodal variational inference are insufficient once coupled with a learned prior and expressive decoders. A suitable shared latent must simultaneously satisfy coherence across generated modalities, predictive sufficiency of subset latents, and minimality of the latent content. We propose a routed training objective whose structural choices align the latent with these criteria and admit a minimal-sufficiency characterization in the realizable setting. Experiments on PolyMNIST-Quadrant-Labels and a large-scale image-text-audio benchmark show MUNI matching or exceeding the strongest baselines on conditional generation while opening its largest margins on unconditional coherence. Project page: https://muni-proj.github.io/.
We introduce MUNITE, a latent-variable framework for flexible any-to-any multimodal generation that treats encoding and latent generation as the same inference problem under different amounts of observed evidence. Given any subset of modalities, MUNITE models the conditional distribution over the latent representation associated with the complete observation. Full observation recovers deterministic encoding, no observation recovers the latent marginal, and intermediate subsets define conditional latent inference, all within a single conditional flow model. A shared latent sample captures variation that must remain consistent across generated targets, while modality-specific generative decoders model the remaining uncertainty independently. To learn these conditional distributions from incomplete training examples, we extend conditional flow matching through self-distillation: predictions conditioned on richer available observations supervise the same model conditioned on smaller subsets at the same intermediate latent state. When the richer-evidence trajectory follows the exact conditional flow, this provides the same expected learning signal as full-target denoising. Across PolyMNIST-D-Q, FFHQ64, and image-text-audio, MUNITE achieves competitive or better generation quality and source-target alignment, with higher joint-generation coherence. In particular, it attains the highest coherence in all one-to-many and unconditional image-text-audio comparisons, showing the effectiveness of unified latent inference across diverse multimodal settings.
While recent multimodal large language models (MLLMs) have made impressive strides, they predominantly employ a conventional autoregressive architecture as their backbone, leaving significant room to explore effective and efficient alternatives in architectural design. Concurrently, recent studies have successfully applied discrete diffusion models to various domains, such as visual understanding and image generation, revealing their considerable potential as a promising backbone for multimodal systems. Drawing inspiration from these pioneering studies, we introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models, which unifies understanding and generation across text, speech, and images. Omni-Diffusion employs a unified mask-based discrete diffusion model to directly capture the joint distribution over discrete multimodal tokens. This approach supports not only bimodal tasks but also more complex scenarios involving multiple modalities. On a diverse set of benchmarks, our method outperforms or performs on par with existing multimodal systems that process two or more modalities, highlighting the significant promise of diffusion models in powering the next generation of multimodal foundation models. Project webpage: https://omni-diffusion.github.io.