Multimodal Diffusion Models

Overview

These models use information from one or more modalities—such as text, images, audio, video, motion, or sensor data—to generate or understand content in another modality or several at once. Common approaches include continuous latent diffusion and flow-matching transformers, discrete masked-token diffusion, and architectures that combine shared representations with modality-specific streams, cross-attention, or adapters. They are used for tasks ranging from text-guided image and video creation to audio-video synchronization, motion generation, medical imaging, and multimodal reasoning, with evaluation spanning benchmarks such as GenEval, MMMU, VBench, and HumanML3D. A central challenge is maintaining coherence and precise alignment across modalities while preserving each modality’s quality, supporting missing or noisy inputs, and keeping inference and evaluation practical.

Latest papers

CardsList

186 papers13 papers in the last four weeks, down 54% on the four weeks before. 0.1% of all new papers.

  1. Why Multi-Source Audio Fails to Compose: A Geometric Diagnosis and Its Remedy

    Nov 18, 2026Anurag Singh, Prajwal Singh, Shanmuganathan RamanCompositional Image GenerationMultimodal Diffusion Models

  2. Multimodal Masked Diffusion Autoencoder for Modeling Impaired Driving

    Nov 17, 2026John Gideon, Anshul Gupta, Megan Applegate-Kenton +7Multimodal Diffusion ModelsMultimodal SSL

  3. Nahual: A Sequence Model for Language and Atoms

    Nov 14, 2026Austin H. Cheng, Marcel Müller, Andreas Burger +10Chemical Language Models3D Molecular Generation

  4. Découpage: Directorial Audio-Visual Control for Long Multi-Shot Video Generation

    Nov 13, 2026Yifan Wang, Hao Ouyang, Qiuyu Wang +9Audio-Video GenerationMulti-Shot Video Generation

  5. Split and Bridge: Multimodal Generation via Diffusion Bridging

    Nov 7, 2026Ahmad Arrabi, Xiaohan Zhang, Xingyu Li +1Multimodal GenerationMultimodal Diffusion Models

  6. Scenario Augmentation for Rare Financial Events via Multimodal Latent Diffusion

    Nov 5, 2026Mahdi Arab Loodaricheh, Anita Raja, Indranil SenGuptaSynthetic Data AugmentationConditional Generative Modeling

  7. Spatially and Compositionally Controlled Multimodal Glioblastoma MRI Synthesis with Latent Diffusion

    Nov 4, 2026Steven Glass, Max Van Puyvelde, Marija Pizurica +3Medical Image SynthesisMedical Imaging

  8. C-WFDM: A ControlNet-Guided Wavelet-Fusion Diffusion Model for Multimodal Brain MRI Completion

    Nov 3, 2026Muhammad Nabi Yasinzai, Remika Mito, Mangor PedersenMedical Image SynthesisBrain MRI

  9. NitroFusion2: Bringing Adversarial Loss to the Text Stream of Multimodal Diffusion Transformers

    Nov 3, 2026Dar-Yen Chen, Yi-Zhe SongDiffusion Model DistillationMultimodal Diffusion Models

  10. From Tokens to Trajectories: Parameter-Efficient Continuous Adaptation of Unified Multimodal Models

    Nov 3, 2026Gayatri Deshmukh, Onkar Susladkar, Adheesh Juvekar +2Unified Multimodal ModelsMultimodal Diffusion Models

  11. Uncertainty-Guided Cross-Modal Alignment for Diffusion-Based IoT Intrusion Detection

    Oct 29, 2026Mahdi Arab Loodaricheh, Anita Raja, Mohammad Hossein ManshaeiIoT Intrusion DetectionCross-Modal Alignment

  12. Diffusion-Based Synthesis of Complete 3D Biparametric Prostate MRI

    Oct 22, 2026Claudia Giardina, Montse Pardàs, Verónica VilaplanaMedical Image SynthesisMagnetic Resonance Imaging

  13. SciFusion: Physics-Guided Compositional Diffusion for Multimodal 3D Scientific Image Generation

    Oct 21, 2026Luoxi Wang, Alessio Quercia, Hanno Scharr +1Image GenerationMultimodal Diffusion Models

  14. Modaliff: A Unified Token Diffusion Model for Any-to-Any Generation

    Oct 18, 2026Duc Kieu, Kien Do, Minh Khoa Le +3Multimodal GenerationUnified Multimodal Models

  15. Multimodal-Conditioned Latent Diffusion Models for Spatiotemporal Brain MRI Forecasting

    Oct 18, 2026Antonio Scardace, Silvia Ingala, Sebastiano Battiato +1Brain MRIAlzheimer's Disease Progression Prediction

  16. CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective

    Oct 16, 2026Qiyan Zhao, Xiaofeng Zhang, Xu-Yao Zhang +4Diffusion Language Model DecodingDiffusion Language Models

  17. Single-stage joint diffusion of sky images and clear-sky index for ultra-short-term irradiance forecasting: what the frame-to-irradiance stage discards

    Oct 15, 2026Kanawut Suwandee, Suwichaya SuwanwimolkulSolar Irradiance ForecastingTime Series Forecasting

  18. Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond

    Oct 15, 2026Zhiyuan Ma, Jiabao Wei, Tianyu Wang +21RL for Diffusion ModelsRL for Generative Models

  19. LiFi: LiDAR Generation from Multi-View Images via Geometric and Semantic Collaborative Guidance

    Oct 14, 2026Sizhuo Zhou, Xiaosong Jia, Fanrui Zhang +10Point Cloud GenerationDiffusion Model Guidance

  20. QK-Edit: Revisiting Attention-based Injection in MM-DiT for Image and Video Editing

    Oct 13, 2026Tiancheng Shen, Zilong Huang, Xiangtai Li +6Diffusion-Based Image EditingAttention Control in Diffusion Models

  21. HairFlow: Modeling 3D Hair Strands with Multi-Modal Conditioned Diffusion

    Oct 12, 2026Yating Wang, Kui Wu, Ran Yi +5Conditional Diffusion Models3D Diffusion Models

  22. Joint Discrete and SE(3) Diffusion for Protein Interface Design

    Oct 10, 2026Zhenqiao Song, Jianwen Xie, Lei LiProtein Binder DesignMolecular Generation

  23. Multimodal Co-Generation With Continuous Bitstream Diffusion

    Oct 9, 2026Georgios Batzolis, Georgios Kontos, Bruno Trentini +4Multimodal GenerationMultimodal Diffusion Models

  24. Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback

    Oct 9, 2026Yucheng Yang, Xiaoxiao Sun, Shusheng Yang +2Multimodal Diffusion ModelsDiffusion Model Guidance

  25. Learning to Read the Contextual Tokens in Diffusion Transformers

    Oct 5, 2026Omer Dahary, Etai Sella, Hadar Averbuch-Elor +2Diffusion Model AlignmentDiffusion Transformer