Multimodal Diffusion Models

Ask about this topic

Overview

These models use information from one or more modalities—such as text, images, audio, video, motion, or sensor data—to generate or understand content in another modality or several at once. Common approaches include continuous latent diffusion and flow-matching transformers, discrete masked-token diffusion, and architectures that combine shared representations with modality-specific streams, cross-attention, or adapters. They are used for tasks ranging from text-guided image and video creation to audio-video synchronization, motion generation, medical imaging, and multimodal reasoning, with evaluation spanning benchmarks such as GenEval, MMMU, VBench, and HumanML3D. A central challenge is maintaining coherence and precise alignment across modalities while preserving each modality’s quality, supporting missing or noisy inputs, and keeping inference and evaluation practical.

Latest papers

CardsList

175 papers26 papers in the last four weeks, up 73% on the four weeks before. 0.2% of all new papers.

  1. COP-GEN: Latent Diffusion Transformer for Copernicus Earth Observation Data

    Mar 3, 2026Miguel Espinosa, Eva Gmelich Meijling, Valerio Marsocci +2Conditional Generative ModelingMultimodal Diffusion Models

    School of Engineering, University of Edinburgh · European Space Agency (ESA) · Asterisk Labs

  2. LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Model

    Mar 1, 2026Zebin You, Xiaolu Zhang, Jun Zhou +2Diffusion ModelsMultimodal Diffusion Models

    Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +1

  3. DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generation

    Feb 26, 2026Yichen Peng, Jyun-Ting Song, Siyeol Jung +7Human Motion GenerationCo-Speech Gesture Generation

    Institute of Science Tokyo · Carnegie Mellon Unversity · UNIST +2

  4. SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model

    Feb 25, 2026Guibin Chen, Dixuan Lin, Jiangping Yang +48Multimodal Video GenerationAudio-Video Generation

  5. The Design Space of Tri-Modal Masked Diffusion Models

    Feb 25, 2026Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec +21Diffusion ModelsMultimodal Diffusion Models

    Apple · University of Cambridge · MIT +1

  6. RelA-Diffusion: Relativistic Adversarial Diffusion for Multi-Tracer PET Synthesis from Multi-Sequence MRI

    Feb 24, 2026Minhui Yu, Yongheng Sun, David S. Lalush +3Medical Image SynthesisDiffusion Models

    Department of Radiology and Biomedical Research Imaging Center (BRIC), University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA. · Joint Department of Biomedical Engineering, University of North Carolina at Chapel Hill and North Carolina State University, Chapel Hill, NC 27599, USA. · Department of Exercise and Sport Science, UNC-CH, Chapel Hill, NC 27599, USA.

  7. Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Models

    Feb 24, 2026Guangnian Wan, Qi Li, Gongfan Fang +2Diffusion Language ModelsMultimodal Diffusion Models

    National University of Singapore

  8. MultiDiffSense: Diffusion-Based Multi-Modal Visuo-Tactile Image Generation Conditioned on Object Shape and Contact Pose

    Feb 22, 2026Sirine Bhouri, Lan Wei, Jian-Qing Zheng +1Synthetic Data Generation for RoboticsMultimodal Diffusion Models

    the Department of Bioengineering, Imperial-X Initiative, Imperial College London, London, United Kingdom. · CAMS-Oxford Institute, University of Oxford, Oxford, United Kingdom.

  9. Art2Mus: Artwork-to-Music Generation via Visual Conditioning and Large-Scale Cross-Modal Alignment

    Feb 19, 2026Ivan Rinaldi, Matteo Mendula, Nicola Fanelli +4Video-to-Music GenerationMusic Generation

    University of Bari Aldo Moro, Bari, Italy · Catalonia’s Telecommunications Technology Centre, Barcelona, Spain · University of Picardie Jules Verne, Amiens, France +1

  10. Efficient Text-Guided Convolutional Adapter for the Diffusion Model

    Feb 16, 2026Aryan Das, Koushik Biswas, Swalpa Kumar Roy +2Diffusion Model AdaptationConditional Image Generation

    VIT Bhopal · IIIT Delhi · Tezpur University Assam +1

  11. LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Models

    Feb 15, 2026Shufan Li, Yuchen Zhu, Jiuxiang Gu +6Unified Multimodal ModelsMultimodal Reasoning

    1Adobe · 2UCLA · 3Georgia Tech

  12. DiTS: Multimodal Diffusion Transformers Are Time Series Forecasters

    Feb 6, 2026Haoran Zhang, Haixuan Liu, Yong Liu +4Exogenous Time Series ForecastingTime Series Forecasting

    School of Software, BNRist, Tsinghua University

  13. DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter

    Feb 5, 2026Xukun Li, Yu Sun, Lei Zhang +10Bimanual Robotic ManipulationContact-Rich Robotic Manipulation

    XYZ Embodied AI, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China · School of Computation, Information and Technology, Technical University of Munich, Garching, Germany +3

  14. TexTailor: Inference-Time Textual Guidance Tailoring for Multimodal Diffusion Transformers

    Jan 5, 2026Binglei Li, Mengping Yang, Zhiyu Tan +2Diffusion TransformerCompositional T2I Generation

    Fudan University, Shanghai 200433, China · Shanghai Academy of AI for Science, Shanghai 200030, China · Shanghai Innovation Institute, Shanghai 200030, China

  15. Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models

    Dec 16, 2025Shufan Li, Jiuxiang Gu, Kangning Liu +4Masked Diffusion ModelsDiffusion Language Model Inference

    Adobe · UCLA

  16. UniLayDiff: A Unified Diffusion Transformer for Content-Aware Layout Generation

    Dec 9, 2025Zeyang Liu, Le Wang, Sanping Zhou +4Diffusion TransformerConditional Generative Modeling

    College of Artificial Intelligence, Xi’an Jiaotong University · Amazon.com, Inc. · Pixocial Technology

  17. HAGI++: Head-Assisted Gaze Imputation and Generation

    Nov 4, 2025Chuhan Jiao, Zhiming Hu, Andreas BullingGaze EstimationTime Series Imputation

    University of Stuttgart, Germany · The Hong Kong University of Science and Technology (Guangzhou), China

  18. UniVideo: Unified Understanding, Generation, and Editing for Videos

    Oct 9, 2025Cong Wei, Quande Liu, Zixuan Ye +5Video GenerationUnified Multimodal Models

    University of Waterloo · Kling Team, Kuaishou Technology

  19. Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation

    Sep 23, 2025Shufan Li, Jiuxiang Gu, Kangning Liu +4Masked Diffusion ModelsUnified Multimodal Models

    Adobe · UCLA

  20. ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion

    Jul 8, 2025Yuhan Li, Wei Zhang, Juan Chen +3Cross-Modal LearningMissing-Modality Learning

    National University of Defense Technology · The Fifth Electronic Research Institute of MIIT · University of Chinese Academy of Sciences +2

  21. Hyper-modal Imputation Diffusion Embedding with Dual-Distillation for Federated Multimodal Knowledge Graph Completion

    Jun 27, 2025Ying Zhang, Yu Zhao, Xuhui Sui +5Multimodal Federated LearningMissing-Modality Learning

    College of Computer Science, VCIP, DISSec, Nankai University, Tianjin, China · School of Software, Tiangong University, Tianjin, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China +1

  22. HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation

    Jun 10, 2025Ziyao Huang, Zixiang Zhou, Juan Cao +9HOI GenerationPhysical Consistency in Video Generation

    University of Chinese Academy of Sciences · Tencent Hunyuan

  23. LaViDa: A Large Diffusion Language Model for Multimodal Understanding

    May 22, 2025Shufan Li, Konstantinos Kallidromitis, Hritik Bansal +7Multimodal UnderstandingVision-Language Models

    1UCLA · 2Panasonic AI Research · 4Salesforce Research +1

  24. TF-TI2I: Training-Free Text-and-Image-to-Image Generation via Multi-Modal Implicit-Context Learning in Text-to-Image Models

    Mar 19, 2025Teng-Fang Hsiao, Bo-Kai Ruan, Yi-Lun Wu +2Conditional Image GenerationCompositional T2I Generation

    National Yang Ming Chiao Tung University

  25. Zero-Shot Paragraph-level Handwriting Imitation with Latent Diffusion Models

    Sep 1, 2024Martin Mayr, Marcel Dreier, Florian Kordon +5Latent Diffusion ModelsImage Generation

    Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Erlangen, Germany. · Planet AI GmbH, Rostock, Germany.