Multimodal Diffusion Models

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 110

All topics

ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment
May 29, 2026Jun-Hak Yun, Seung-Bin Kim, Seong-Whan LeeCross-Modal Representation Learning TTS Synthesis
Department of Artificial Intelligence, Korea University, Seoul, Korea
Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation
May 28, 2026Chrysa Pratikaki, Pablo Ruiz-Ponce, Jiankang Deng +2HOI Generation Human Motion Generation
Imperial College London, UK · University of Alicante, Spain
Native Audio-Visual Alignment for Generation
May 28, 2026Longbin Ji, Guan Wang, Xuan Wei +6Audio-Video Generation Audio-Visual Synchronization
ERNIE Team, Baidu Inc.
Masked Diffusion Vision-Language Models for Temporal Action Localization
May 28, 2026Fengshun Wang, Zhengbo Zhang, Zhigang TuHuman Activity Recognition Video-Language Models
Wuhan University · Singapore University of Technology and Design
Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion
May 28, 2026Yizhuo Lu, Changde Du, Qingyu Shi +5Brain-Computer Interfaces Unified Multimodal Models
NeuBCI Lab, State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China +2
AnchorDiff: Training-Free Concept Grounding for MM-DiTs via Anchor-Based Graph Propagation
May 26, 2026Jian Zhang, Zhijun ZhangMultimodal Grounding Multimodal Diffusion Models
School of Automation Science and Engineering South China University of Technology Guang Zhou, China
StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration
May 25, 2026Linrui Tian, Qi Wang, Bang ZhangAudio-Video Generation Long-Horizon Video Generation
Tongyi Lab, Alibaba Group
Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation
May 24, 2026Shuyuan Tu, Qi Tian, Zihan Yang +9Cross-Modal Learning Audio-Video Generation
Fudan University · Tencent Hunyuan
Vision-Language Binding in In-Context Image Generation
May 23, 2026Chris Ge, Rohit Gandikota, Antonio Torralba +1Image Generation Mechanistic Interpretability
MIT CSAIL · Northeastern University
Composing People Together: Iterative Pose-Image Generation for Multi-Person Interaction Scenes
May 22, 2026Wenxuan Peng, Bharath Hariharan, Hadar Averbuch-ElorImage Generation Compositional Image Generation
Cornell University, Ithaca, USA · Cornell University, New York, New York, USA
Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving
May 21, 2026Jiahao Wang, Bo Sun, Yijing Bai +12Autonomous Driving Datasets Autonomous Driving
Waymo · Johns Hopkins University · Google DeepMind +1
MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery
May 16, 2026Reese Kneeland, Cesar Kadir Torrico Villanueva, Jordyn Ojeda +4Neural Decoding Image Reconstruction
University of Minnesota, Minneapolis, Minnesota, United States of America · Medical AI Research Center (MedARC) · Stanford University, Palo Alto, California, United States of America +2
Diagnosing and Correcting Concept Omission in Multimodal Diffusion Transformers
May 14, 2026Kanghyun Baek, Jaihyun Lew, Chaehun Shin +2Diffusion Transformer T2I Generation
Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, South Korea · Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Computer Science & Engineering, Korea University, Seoul, South Korea +1
OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation
May 12, 2026Guohui Zhang, XiaoXiao Ma, Jie Huang +9Audio-Video Generation Audio-Visual Synchronization
University of Science and Technology of China · JD Explore Academy · Peking University
UniCustom: Unified Visual Conditioning for Multi-Reference Image Generation
May 12, 2026Yiyan Xu, Qiulin Wang, Wenjie Wang +5Subject-Driven Image Generation Image Generation
University of Science and Technology of China · Kling Team, Kuaishou Technology
Qwen-Image-2.0 Technical Report
May 11, 2026Bing Zhao, Chenfei Wu, Deqing Li +72T2I Generation Compositional T2I Generation
Marrying Generative Model of Healthcare Events with Digital Twin of Social Determinants of Health for Disease Reasoning
May 10, 2026Ziquan Wei, Tingting Dan, Guorong WuHealthcare Digital Twins
Department of Computer Science 2Department of Psychiatry University of North Carolina at Chapel Hill
Cross-Modal Semantic-Enhanced Diffusion Framework for Diabetic Retinopathy Grading
May 10, 2026Yiqun WangDiabetic Retinopathy Grading Medical VLMs
Beijing Jiaotong University · School of Software Engineering Beijing Jiaotong University Beijing, China
JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generation
May 5, 2026Lin Song, Wenbo Li, Guoqing Ma +16Unified Multimodal Models Multimodal Foundation Models
Joy Future Academy, JD
Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention
May 5, 2026Daniel Mensing, Jan Kapar, Jochen G. Hirsch +3Synthetic Tabular Data Generation Multimodal Generation
Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany · Leibniz Institute for Prevention Research and Epidemiology – BIPS, Bremen, Germany · Faculty of Mathematics and Computer Science, University of Bremen, Bremen, Germany +1
Semantic-Structural Alignment for Generative Pictorial Charts
May 5, 2026Zhida Sun, Yulin Zhang, Zheng Gu +4Data Visualization Controllable Image Generation
Visual Computing Research Center (VCC), College of Computer Science and Software Engineering (CSSE), Shenzhen University, China · Yonsei University, South Korea
A-CODE: Fully Atomic Protein Co-Design with Unified Multimodal Diffusion
May 5, 2026Chaoran Cheng, Jiaqi Guan, Milong Ren +5Generative Modeling Molecular Generation
University of Illinois Urbana-Champaign · 2ByteDance Seed
Divide and Conquer: Decoupled Representation Alignment for Multimodal World Models
May 3, 2026Junyuan Xiao, Dingkang Liang, Xin Zhou +9World Models Cross-Modal Alignment
Tsinghua University, China · Huazhong University of Science and Technology, China · CSU, China +4
UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors
May 1, 2026Houyuan Chen, Hong Li, Xianghao Kong +8Video Diffusion Models Unified Multimodal Models
Beihang University, Beijing, China · Nanjing University, Nanjing, China · BAAI, Beijing, China +3
IdentiFace: Multi-Modal Iterative Diffusion Framework for Identifiable Suspect Face Generation in Crime Investigations
May 1, 2026Weichen Liu, Yixin Yang, Changsheng Chen +1Diffusion Sampling Image Generation
Southeast University · Nanjing, China · Shenzhen MSU-BIT University +1
The Thinking Pixel: Recursive Sparse Reasoning in Multimodal Diffusion Latents
Apr 28, 2026Yuwei Sun, Yuxuan Yao, Hui Li +1Diffusion Transformer Conditional Image Generation
Shanghai Academy of AI for Science · Fudan University
Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
Apr 26, 2026Zhen Ye, Xu Tan, Aoxiong Yin +8Cross-Modal Learning Audio-Video Generation
Hong Kong University of Science and Technology · Independent Researcher · Zhejiang University +3
Multimodal Diffusion to Mutually Enhance Polarized Light and Low Resolution EBSD Data
Apr 24, 2026Harry Dong, Timofey Efimov, Megna Shah +4Diffusion Models for Image Restoration Materials Science
Department of Electrical and Computer Engineering, Carnegie Mellon University, USA · Materials and Manufacturing Directorate, Air Force Research Laboratory, USA · Department of Materials Science and Engineering, Carnegie Mellon University, USA +1
GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers
Apr 22, 2026Yuxuan Xue, Ruofan Liang, Egor Zakharov +6Image-to-Image Translation Diffusion Transformer
Codec Avatars Lab, Meta · University of Tübingen · Max Planck Institute for Informatics, Saarland Informatics Campus
Wan-Image: Pushing the Boundaries of Generative Visual Intelligence
Apr 21, 2026Chaojie Mao, Chen-Wei Xie, Chongyang Zhong +55T2I Generation Identity-Preserving Image Generation

May 29, 2026, eess.ASJ/K move · Enter open · S save

ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment

Jun-Hak Yun, Seung-Bin Kim, Seong-Whan Lee

Department of Artificial Intelligence, Korea University, Seoul, Korea

Cross-Modal Representation Learning TTS Synthesis Text-to-Speech Multimodal Diffusion Models

Open paper AskPDF arXiv

Recent advancements in text-guided audio generation have yielded promising results in diverse domains, including sound effects, speech, and music. However, jointly generating speech with environmental audio remains challenging due to the inherent disparities in their acoustic patterns and temporal dynamics. We propose ImmersiveTTS, an environment-aware text-to-speech (TTS) model that generates natural speech seamlessly integrated within environmental contexts by explicitly modeling cross-modal interactions. Our model builds on a multimodal diffusion transformer and fuses transcript-aligned speech latent with text-conditioned environmental context via joint attention. To enhance semantic consistency, we introduce a domain-specific representation alignment objective tailored to environment-aware TTS, leveraging complementary self-supervised representations from speech and audio encoders. Experimental results show that ImmersiveTTS achieves higher naturalness, intelligibility, and audio fidelity than existing approaches across objective metrics and human listening tests.

Momentum

Related topics

Latest papers 110