Multimodal Diffusion Models

Overview

These models use information from one or more modalities—such as text, images, audio, video, motion, or sensor data—to generate or understand content in another modality or several at once. Common approaches include continuous latent diffusion and flow-matching transformers, discrete masked-token diffusion, and architectures that combine shared representations with modality-specific streams, cross-attention, or adapters. They are used for tasks ranging from text-guided image and video creation to audio-video synchronization, motion generation, medical imaging, and multimodal reasoning, with evaluation spanning benchmarks such as GenEval, MMMU, VBench, and HumanML3D. A central challenge is maintaining coherence and precise alignment across modalities while preserving each modality’s quality, supporting missing or noisy inputs, and keeping inference and evaluation practical.

Latest papers

CardsList

186 papers13 papers in the last four weeks, down 54% on the four weeks before. 0.1% of all new papers.

  1. ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion

    Jul 8, 2025Yuhan Li, Wei Zhang, Juan Chen +3Cross-Modal LearningMissing-Modality Learning

  2. Hyper-modal Imputation Diffusion Embedding with Dual-Distillation for Federated Multimodal Knowledge Graph Completion

    Jun 27, 2025Ying Zhang, Yu Zhao, Xuhui Sui +5Multimodal Federated LearningMissing-Modality Learning

  3. HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation

    Jun 10, 2025Ziyao Huang, Zixiang Zhou, Juan Cao +9HOI GenerationPhysical Consistency in Video Generation

  4. LaViDa: A Large Diffusion Language Model for Multimodal Understanding

    May 22, 2025Shufan Li, Konstantinos Kallidromitis, Hritik Bansal +7Multimodal UnderstandingVision-Language Models

  5. Zero-Shot Paragraph-level Handwriting Imitation with Latent Diffusion Models

    Sep 1, 2024Martin Mayr, Marcel Dreier, Florian Kordon +5Latent Diffusion ModelsImage Generation