Multimodal Diffusion Models

These models use information from one or more modalities—such as text, images, audio, video, motion, or sensor data—to generate or understand content in another modality or several at once. Common approaches include continuous latent diffusion and flow-matching transformers, discrete masked-token diffusion, and architectures that combine shared representations with modality-specific streams, cross-attention, or adapters. They are used for tasks ranging from text-guided image and video creation to audio-video synchronization, motion generation, medical imaging, and multimodal reasoning, with evaluation spanning benchmarks such as GenEval, MMMU, VBench, and HumanML3D. A central challenge is maintaining coherence and precise alignment across modalities while preserving each modality’s quality, supporting missing or noisy inputs, and keeping inference and evaluation practical.