Multimodal Diffusion Models

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 110

All topics

Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Models
Oct 7, 2026Hong Huang, Chenhongyi Yang, Junzhe Sun +3Diffusion Language Model Distillation Diffusion Model Inference Acceleration
Simon Fraser University · Meta
Learning to Read the Contextual Tokens in Diffusion Transformers
Oct 5, 2026Omer Dahary, Etai Sella, Hadar Averbuch-Elor +2Diffusion Model Alignment Diffusion Transformer
Tel Aviv University · Cornell University
CleanMDM: Clean Motion Diffusion Model for Multimodal Motion Cleanup
Oct 4, 2026Zhe Li, Shicheng Wang, Bowen Cai +1Masked Diffusion Models Motion Diffusion Models
Peking University · Light Origins · Alibaba
Gestalt: Large Multimodal Interplay Model
Sep 30, 2026Zequn Yang, Yu Miao, Haotian Ni +8Unified Multimodal Models Cross-Modal Alignment
Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Beihang University +3
UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement
Sep 30, 2026Fang Wu, Da Xing, Yanjie Huang +16Unified Multimodal Models On-Policy Self-Distillation
Stanford University · Johns Hopkins University · Independent Researcher +7
Adaptive Reward Routing: Dynamic Multi-Reward Optimization for Joint Audio-Video Diffusion via Forward-Process RL
Sep 29, 2026Songlin Yang, Xiaotong Zhao, Jiacheng Zhang +5Multi-Objective Reinforcement Learning Cross-Modal Alignment
Tencent Video · The University of Hong Kong
World2Motion: Turning Video World Models into 3D Human Motion Generators
Sep 29, 2026Fangyuan Tu, Xiangyue Zhang, Yiyi Cai +8Human Motion Generation Diffusion Models for Motion Generation
Japan Advanced Institute of Science and Technology · The University of Tokyo · Institute of Science Tokyo +1
Trajectory-Level Mode Guidance for Controllable Diffusion-Based Multi-Robot Motion Planning
Sep 29, 2026Tianyou Yu, Shengze Cai, Chao XuMulti-Robot Motion Planning Trajectory Generation
Institute of Cyber-System and Control, College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China
BiMoGen: Bidirectional Motion-Text Generation via Unified Masked Discrete Diffusion
Sep 28, 2026Wanjiang Weng, Yongliang Wu, Xiaofeng Tan +3Masked Diffusion Models Text-to-Motion Generation
Department of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, Nanjing, China · National University of Singapore +1
Diff-RF: Mutually Reinforced Image Registration and Fusion via Degradation-Aware Learning
Sep 23, 2026Xunpeng Yi, Zaixi Du, Qinglong Yan +3Multi-Source Image Fusion Cross-Modal Image Registration
Electronic Information School, Wuhan University, Wuhan 430072, China
All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation
Sep 23, 2026Ohad Rahamim, Dvir Samuel, Idan Schwartz +1Video Diffusion Models Modality Imbalance
Bar-Ilan University · NVIDIA
MM-Future: Multi-Mode Joint World-Action Modeling for Autonomous Driving
Sep 17, 2026Shuai Liu, Hechangle Gong, Hao Jiang +5World Models for Autonomous Driving Autonomous Driving Planning
NIO · School of Computer Science and Engineering, Sun Yat-sen University · Beihang University +1
Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning
Sep 17, 2026Haoqiang Kang, Yizhe Zhang, Nikki Lijing Kuang +3Unified Multimodal Models Multimodal Reasoning
University of California, San Diego · Meta · University of Pennsylvania
Learning Interaction between Image and Layout Priors for Joint Image-Layout Generation in Design Templates
Sep 12, 2026Shirong Yang, Bo Yang, Ying CaoMultimodal Diffusion Models
School of Information Science and Technology, ShanghaiTech University, 393 Middle Huaxia Road, Pudong New Area, Shanghai, China
Multi-Modal Controlled Coherent Motion Generation
Sep 11, 2026Yifei Liu, Qiong Cao, Hongwei Yi +2Human Motion Generation 3D Avatar Generation
South China University of Technology · Joy Future Academy · Peking University
VoT: Vision-of-Thought for Unified Multimodal Representation Alignment
Sep 7, 2026Jingxiang Sun, Chao Liao, Zhengxiong Luo +6T2I Generation Compositional T2I Generation
ByteDance Seed
The Attention Triangle in Audio-Video Models
Sep 3, 2026Sagi Polaczek, Noa Kraicer, Gal Metzer +4Audio-Video Generation Attention Control in Diffusion Models
1Tel Aviv University · 2Simon Fraser University
TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models
Sep 1, 2026Chao Zhou, Yiling Chen, Qi Chu +3Audio-Video Generation Latent Diffusion Models
Differentially Private Paired Table-Image Multimodal Synthesis
Sep 1, 2026Kai Chen, Josephine Lamp, Somesh Jha +1Synthetic Data Generation Cross-Modal Alignment
University of Virginia · Dexcom · University of Wisconsin-Madison
DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution
Aug 31, 2026Jiashu Zhu, Yanhao Zheng, Ruitian Tian +7Audio-Video Generation Video Diffusion Models
DreamX Team, Alibaba Group
Toward a Foundation Plug-and-Play Prior for Computed Tomography Reconstruction via a Multimodal Diffusion Model
Aug 24, 2026Haley Duba-Sullivan, Patxi Fernandez-Zelaia, Obaidullah Rahman +1Computed Tomography Diffusion Models
Oak Ridge National Laboratory Oak Ridge, TN, USA
Semantic Steering for Controllable Generation: Tuning-Free Concept Erasure in Multimodal Diffusion Transformers
Aug 13, 2026Qiao Li, Xiaomeng Fu, Yuanshu Zhao +3Diffusion Model Unlearning Activation Steering
Institute of Information Engineering, Chinese Academy of Sciences Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences Beijing, China
TimeRoute: Time-Aware Modality Routing and Diffusion for Multi-Modal Recommendation
Aug 11, 2026Pengyu Zhang, Yangqin Jiang, Klim Zaporojets +2Recommender Systems Multimodal Recommendation
University of Amsterdam Amsterdam, The Netherlands · University of Hong Kong Hong Kong, China · Aarhus University Aarhus, Denmark
Surg-UniWorld: A Unified Surgical World Model with Multimodal Control Experts
Aug 7, 2026Rulin Zhou, Wanhao Liu, Guoheng Ma +8Controllable Video Generation Temporal Consistency in Video Generation
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR 999077, China · Shenzhen Loop Area Institute, Shen Zhen 518055, China · School of Electrical and Computer Engineering, Faculty of Engineering, the University of Sydney, Sydney, NSW 2006, Australia
Vorch-Omni: Multi-Task Orchestration of Sight and Sound
Aug 6, 2026Vorch Team, Xiaoyu Chen, Yang Ding +25Audio-Video Generation Unified Multimodal Models
CSGen: A Multi-Domain Curvilinear Structure Generation Model via Hierarchical Multimodal Diffusion
Aug 5, 2026Zhe Shan, Ziming Yang, Lei Zhou +3Image Generation Conditional Generative Modeling
Hainan University Haikou, China · Guangdong Ocean University Zhanjiang, China
Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition
Aug 5, 2026Zehao Bao, Shujun Guo, Bruce X. B. YuSkeleton-Based Action Recognition Diffusion Transformer
The University of Hong Kong Hong Kong SAR, China · Zhejiang University Hangzhou, China
Coupled Continuous-Discrete Generation for Scene Text Image Super-Resolution
Aug 5, 2026Axi Niu, Knag Zhang, Qingsen Yan +3Image Super-Resolution Diffusion Models
Northwestern Polytechnical University, Xi’an, China · Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
DiffImaginE: Imagine to Verify Entity Types with Diffusion
Aug 4, 2026Feng Zhang, Feiyu Han, Rongxin Yang +11Named Entity Recognition Conditional Diffusion Models
1Fuzhou University · 2Chinese Academy of Sciences · 6Fullive Innovation (Beijing) AI Technology Co., Ltd. +5
EchoChange: A Diffusion Language Model with Dual Pass Remasking for Factual Remote Sensing Disaster Change Captioning
Aug 3, 2026Dongwei Sun, Bowen Yao, Yujie Zhang +3Remote Sensing Image Understanding Masked Diffusion Models
School of Computer Science and Technology and the Ministry of Education Key Lab for Intelligent Networks and Network Security, Xi’an Jiaotong University, 710049, China · School of Computer Science and Technology, Faculty of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China · State Key Laboratory of Remote Sensing and Digital Earth, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, 100094, China

Oct 7, 2026, cs.CVJ/K move · Enter open · S save

Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Models

Hong Huang, Chenhongyi Yang, Junzhe Sun, Animesh Sinha+2

Simon Fraser University · Meta

Diffusion Language Model Distillation Diffusion Model Inference Acceleration Multimodal Diffusion Models T2I Diffusion Models Multimodal Understanding

Open paper AskPDF arXiv

Unified multimodal diffusion large language models (dLLMs) offer a single architecture for both image generation and multimodal understanding, but their iterative decoding requires tens to hundreds of forward passes. Existing few-step distillation methods largely focus on either image generation or text generation, making it unclear how to compress a fully discrete multimodal dLLM into a single efficient student while preserving both generation and understanding. We introduce Omni-Diffusion-Distill, a unified two-stage distillation framework that retains strong generation and understanding capabilities while substantially reducing the inference cost of a unified multimodal dLLM. Omni-Diffusion-Distill aligns the distillation of both generation and understanding, for both images and text, in the discrete token space. In the first stage, the student is trained to skip decoding steps by replaying cached teacher trajectories, and in the second stage the student is refined on intermediate states along its own rollouts. We further remedy two sources of degradation in unified distillation with a pairwise collision penalty that reduces repetition under parallel text decoding, and entropy-matched guidance that prevents entropy collapse caused by fitting the sharpened teacher distribution in image generation. Omni-Diffusion-Distill achieves state-of-the-art trade-offs between decoding efficiency and generation and understanding performance for multimodal dLLMs, reducing image generation from 128 to 8 decoding steps and multimodal understanding from 512 to 64, giving 18.2x and 21.2x wall-clock speedups. Under these budgets, it scores 0.828 on GenEval and 83.0 on DPG-Bench for text-to-image generation, while reaching GPT judge scores of 20.0 on MM-Vet and 57.2 on COCO captioning (twice the teacher's 28.4 at the same steps) for multimodal understanding.

Momentum

Related topics

Latest papers 110