Multimodal Diffusion Models

Ask about this topic

Overview

These models use information from one or more modalities—such as text, images, audio, video, motion, or sensor data—to generate or understand content in another modality or several at once. Common approaches include continuous latent diffusion and flow-matching transformers, discrete masked-token diffusion, and architectures that combine shared representations with modality-specific streams, cross-attention, or adapters. They are used for tasks ranging from text-guided image and video creation to audio-video synchronization, motion generation, medical imaging, and multimodal reasoning, with evaluation spanning benchmarks such as GenEval, MMMU, VBench, and HumanML3D. A central challenge is maintaining coherence and precise alignment across modalities while preserving each modality’s quality, supporting missing or noisy inputs, and keeping inference and evaluation practical.

Latest papers

CardsList

175 papers26 papers in the last four weeks, up 73% on the four weeks before. 0.2% of all new papers.

  1. Uncertainty-Guided Cross-Modal Alignment for Diffusion-Based IoT Intrusion Detection

    Oct 29, 2026Mahdi Arab Loodaricheh, Anita Raja, Mohammad Hossein ManshaeiIoT Intrusion DetectionCross-Modal Alignment

    Department of Computer Science, Hunter College and The Graduate Center, CUNY, New York, NY, USA

  2. Diffusion-Based Synthesis of Complete 3D Biparametric Prostate MRI

    Oct 22, 2026Claudia Giardina, Montse Pardàs, Verónica VilaplanaMedical Image SynthesisMagnetic Resonance Imaging

    Universitat Politècnica de Catalunya, Barcelona, Spain

  3. Trion-1: Unifying Language, Protein Sequence, and Structure with Diffusion Language Models

    Oct 22, 2026Tianyu Lu, Kaicheng Yang, Jiangtao Feng +7Protein Language ModelsProtein Structure Generation

  4. SciFusion: Physics-Guided Compositional Diffusion for Multimodal 3D Scientific Image Generation

    Oct 21, 2026Luoxi Wang, Alessio Quercia, Hanno Scharr +1Image GenerationMultimodal Diffusion Models

    Princeton University, Princeton, NJ, USA · Forschungszentrum Jülich, Jülich, Germany

  5. Modaliff: A Unified Token Diffusion Model for Any-to-Any Generation

    Oct 18, 2026Duc Kieu, Kien Do, Minh Khoa Le +3Multimodal GenerationUnified Multimodal Models

    Applied Artificial Intelligence Initiative, Deakin University · VinSmart Future · Aalborg University

  6. Multimodal-Conditioned Latent Diffusion Models for Spatiotemporal Brain MRI Forecasting

    Oct 18, 2026Antonio Scardace, Silvia Ingala, Sebastiano Battiato +1Brain MRIAlzheimer's Disease Progression Prediction

    University of Catania · Copenhagen University Hospital – Rigshospitalet · Copenhagen University Hospital – Herlev and Gentofte +2

  7. UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling

    Oct 17, 2026Kien T. Pham, Yuhao Liu, Chong Hou Choi +6Audio-Video GenerationUnified Multimodal Models

    HKUST · Huawei Research

  8. CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective

    Oct 16, 2026Qiyan Zhao, Xiaofeng Zhang, Xu-Yao Zhang +4Diffusion Language Model DecodingDiffusion Language Models

    State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China. · Q. Zhao and X.-Y. Zhang are the School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing 100049, China · Shanghai Jiao Tong University, Shanghai 200240, China. +1

  9. Single-stage joint diffusion of sky images and clear-sky index for ultra-short-term irradiance forecasting: what the frame-to-irradiance stage discards

    Oct 15, 2026Kanawut Suwandee, Suwichaya SuwanwimolkulSolar Irradiance ForecastingTime Series Forecasting

    Chulalongkorn University, 254 Phaya Thai Rd, Wang Mai, Pathum Wan, Bangkok, 10330, Thailand

  10. Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond

    Oct 15, 2026Zhiyuan Ma, Jiabao Wei, Tianyu Wang +21RL for Diffusion ModelsRL for Generative Models

    Huazhong University of Science and Technology · Beijing Institute of Technology · Tsinghua University +9

  11. LiFi: LiDAR Generation from Multi-View Images via Geometric and Semantic Collaborative Guidance

    Oct 14, 2026Sizhuo Zhou, Xiaosong Jia, Fanrui Zhang +10Point Cloud GenerationDiffusion Model Guidance

    University of Science and Technology of China · Shanghai Innovation Institute · Fudan University +1

  12. QK-Edit: Revisiting Attention-based Injection in MM-DiT for Image and Video Editing

    Oct 13, 2026Tiancheng Shen, Zilong Huang, Xiangtai Li +6Diffusion-Based Image EditingAttention Control in Diffusion Models

    UC Merced · ByteDance Seed · ByteDance Intelligent Creation +1

  13. HairFlow: Modeling 3D Hair Strands with Multi-Modal Conditioned Diffusion

    Oct 12, 2026Yating Wang, Kui Wu, Ran Yi +5Conditional Diffusion Models3D Diffusion Models

    Shanghai Jiao Tong University · LIGHTSPEED

  14. Joint Discrete and SE(3) Diffusion for Protein Interface Design

    Oct 10, 2026Zhenqiao Song, Jianwen Xie, Lei LiProtein Binder DesignMolecular Generation

    Language Technologies Institute, Carnegie Mellon University · Lambda Inc.

  15. Multimodal Co-Generation With Continuous Bitstream Diffusion

    Oct 9, 2026Georgios Batzolis, Georgios Kontos, Bruno Trentini +4Multimodal GenerationMultimodal Diffusion Models

    Radboud University

  16. Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback

    Oct 9, 2026Yucheng Yang, Xiaoxiao Sun, Shusheng Yang +2Multimodal Diffusion ModelsDiffusion Model Guidance

    Stanford University · New York University · University of California, Berkeley

  17. AV-FlowEdit: Training-Free Prompt-Driven Audio-Visual Editing with Event-Aware Spectral Trajectory Preservation

    Oct 9, 2026Shaohua Pan, Xiao LiuInstruction-Based Video EditingMultimodal Diffusion Models

  18. Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Models

    Oct 7, 2026Hong Huang, Chenhongyi Yang, Junzhe Sun +3Diffusion Language Model DistillationDiffusion Model Inference Acceleration

    Simon Fraser University · Meta

  19. Learning to Read the Contextual Tokens in Diffusion Transformers

    Oct 5, 2026Omer Dahary, Etai Sella, Hadar Averbuch-Elor +2Diffusion Model AlignmentDiffusion Transformer

    Tel Aviv University · Cornell University

  20. CleanMDM: Clean Motion Diffusion Model for Multimodal Motion Cleanup

    Oct 4, 2026Zhe Li, Shicheng Wang, Bowen Cai +1Masked Diffusion ModelsMotion Diffusion Models

    Peking University · Light Origins · Alibaba

  21. COSMI: COmpositional Synthesis of Multi-object Interactions

    Oct 2, 2026Daniel Eskandar, Ilya A. Petrov, Gerard Pons-MollHOI GenerationMultimodal Diffusion Models

    University of Tübingen, Germany · Tübingen AI Center, Germany · Zuse School ELIZA, Germany +1

  22. Gestalt: Large Multimodal Interplay Model

    Sep 30, 2026Zequn Yang, Yu Miao, Haotian Ni +8Unified Multimodal ModelsCross-Modal Alignment

    Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Beihang University +3

  23. UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement

    Sep 30, 2026Fang Wu, Da Xing, Yanjie Huang +16Unified Multimodal ModelsOn-Policy Self-Distillation

    Stanford University · Johns Hopkins University · Independent Researcher +7

  24. Adaptive Reward Routing: Dynamic Multi-Reward Optimization for Joint Audio-Video Diffusion via Forward-Process RL

    Sep 29, 2026Songlin Yang, Xiaotong Zhao, Jiacheng Zhang +5Multi-Objective Reinforcement LearningCross-Modal Alignment

    Tencent Video · The University of Hong Kong

  25. World2Motion: Turning Video World Models into 3D Human Motion Generators

    Sep 29, 2026Fangyuan Tu, Xiangyue Zhang, Yiyi Cai +8Human Motion GenerationDiffusion Models for Motion Generation

    Japan Advanced Institute of Science and Technology · The University of Tokyo · Institute of Science Tokyo +1

  26. Trajectory-Level Mode Guidance for Controllable Diffusion-Based Multi-Robot Motion Planning

    Sep 29, 2026Tianyou Yu, Shengze Cai, Chao XuMulti-Robot Motion PlanningTrajectory Generation

    Institute of Cyber-System and Control, College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China

  27. BiMoGen: Bidirectional Motion-Text Generation via Unified Masked Discrete Diffusion

    Sep 28, 2026Wanjiang Weng, Yongliang Wu, Xiaofeng Tan +3Masked Diffusion ModelsText-to-Motion Generation

    Department of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, Nanjing, China · National University of Singapore +1

  28. Enhancing Photogrammetric Digital Surface Models with Pretrained Diffusion Models and Multimodal Conditioning

    Sep 25, 2026Antoine Lorentz, Stéphane May, Valentine Bellet +2Multimodal Diffusion Models3D Reconstruction

    Thales, 92098 Paris, France · Centre National d’Études Spatiales (CNES), 75039 Paris, France

  29. Timo: Taming Multimodal Diffusion Transformers for Human Motion Generation

    Sep 25, 2026Zhao Wang, Jiangtao Hu, Jack Yu +1Text-to-Motion GenerationHuman Motion Generation

    LimX Dynamics

  30. Panoptic Scene Program Diffusion Transformer

    Sep 24, 2026Chika MaduabuchiImage GenerationCompositional T2I Generation

    William & Mary, Williamsburg, VA, USA