Unified Multimodal Models

Follow this topic Ask about this topic

Unified multimodal models study how one system can process and generate text, images, video, audio, and sometimes spatial or action data. Common approaches use shared tokenizers and autoregressive or diffusion-based backbones, often alongside modality-specific encoders, decoders, or experts; training may combine supervised instruction tuning, alignment, and reinforcement learning. Evaluation spans tasks such as visual question answering, text-to-image generation, video understanding, editing, and cross-modal consistency. A central challenge is that shared architecture does not guarantee useful transfer: understanding and generation can interfere or remain semantically misaligned, while efficiency, safety, and reliable evaluation across interacting modalities remain open concerns.

Latest papers

295 papers35 papers in the last four weeks, up 59% on the four weeks before. 0.2% of all new papers.

DiFuse: Fusing Native Diffusion Representations for Spatial Reasoning in Unified Multimodal Models
Oct 29, 2026Ruixin Fan, Weining Wang, Jiahui Sun +2Multimodal Spatial Reasoning Unified Multimodal Models
Institute of Automation, Chinese Academy of Sciences
Calibrating Attention-Based Token Pruning for Efficient Multi-Image Synthesis in UMMs
Oct 29, 2026Zimeng Wu, Qijian Tian, Penghui Du +3Efficient Multimodal Inference Multimodal Token Compression
Beihang University · KlingAI Research · Shanghai Jiao Tong University
Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities
Oct 29, 2026Yunhang Shen, Zhangquan Chen, Xiaotian Li +27Document Parsing Unified Multimodal Models
Continual Adaptation Across Modalities: Learning and Forgetting in Unified Multimodal Models
Oct 28, 2026Xinrui Wang, Zhihao Chen, Jia Tang +4Unified Multimodal Models Continual Learning
Nanjing University of Aeronautics and Astronautics · Computer Vision Center, Barcelona · EPFL +3
IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation
Oct 28, 2026Shaohan Yu, Jie Qin, Wende Tan +4Multimodal Generation Multimodal Dataset
PRLab, Nanjing University · Meituan · Imperial College London +1
EmbeddingGemma 2: Unified and Efficient Multimodal Representations
Oct 27, 2026Henrique Schechter Vera, Sahil Dua, Jovan Andonov +70Multimodal Embedding Unified Multimodal Models
PailitaoMMSearch2.0 Technical Report: Scaling Generative Retrieval to Unified End-to-End Multimodal Search
Oct 27, 2026Xiaohan Ye, Zihan Gong, Jingqian Zhao +9Multimodal IR Generative Retrieval
Taobao & Tmall Group of Alibaba
Cross-modal Steering Vectors Exist Across Both Image and Text Generation
Oct 26, 2026Donghao Zeng, Nishant Subramani, Tamar Rott Shaham +1Cross-Modal Representation Transfer Unified Multimodal Models
Mila, McGill University · Carnegie Mellon University · MIT CSAIL +1
DriveUniTok: A Unified Multimodal Tokenizer for Driving World-Action Models
Oct 26, 2026Jingbo Xu, Hanchao Leng, JiaHui Huang +7World Models for Autonomous Driving World Model Learning
Tsinghua University · Xiaomi EV
RynnUnify: Towards Unified Embodied Brain
Oct 23, 2026Minghao Zhu, Bohan Hou, Kun Xiang +20Robot Foundation Models Embodied World Models
DAMO Academy, Alibaba Group Hupan Lab
JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces
Oct 20, 2026Jiahan Li, Huangyuan Su, Shengbang Tong +9Unified Multimodal Models Multimodal Generation
New York University · Harvard University · University of Illinois Urbana-Champaign +2
Beyond Omni: A Unified MLLM for Embodied Sensory Understanding
Oct 19, 2026Shiyun Xiong, Dongming Wu, Peiwen Sun +5Unified Multimodal Models Multimodal Large Language Models
MMLab, The Chinese University of Hong Kong · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen +2
UniJev: Separating Accuracy from Calibration in Trimodal System One Decision Models
Oct 19, 2026Jianing Wang, Youwen Jian, Derong Wang +5Model Calibration VLM Calibration
Fudan University · Institute of Information Engineering, Chinese Academy of Sciences
Modaliff: A Unified Token Diffusion Model for Any-to-Any Generation
Oct 18, 2026Duc Kieu, Kien Do, Minh Khoa Le +3Multimodal Generation Unified Multimodal Models
Applied Artificial Intelligence Initiative, Deakin University · VinSmart Future · Aalborg University
UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling
Oct 17, 2026Kien T. Pham, Yuhao Liu, Chong Hou Choi +6Audio-Video Generation Unified Multimodal Models
HKUST · Huawei Research
FlexUMM: Flexible Representation Learning for Unified Multimodal Models
Oct 15, 2026Yinyi Luo, Haoyue Bai, Yi Ge +3Task Interference Unified Multimodal Models
Carnegie Mellon University · William & Mary · University of Maryland
PW-Pruning: Phase-Aware Expert Selection for Speech-Centric Omni-Modal Models
Oct 15, 2026Junjie Cao, Fengping Tian, Wenjie Zhou +5Mixture-of-Experts Pruning Mixture-of-Experts Models
Alibaba, Hangzhou, China · University of Macau
Can Visual Generation Expand What Unified Multimodal Models Know?
Oct 15, 2026Wenkai Li, Wenyi Hong, Zhuoyi Yang +1Unified Multimodal Models Transfer Learning
Tsinghua University
Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation
Oct 14, 2026Qian Yu, Zongyang Qiu, Juncheng Yan +40Unified Multimodal Models Diffusion Language Models
ReasonUni: Reasoning-Augmented Reconstruction Advances Unified Multimodal Models
Oct 14, 2026Feize Wu, Zhiyuan Yan, Yuhuan Hong +10Unified Multimodal Models Multimodal Generation
Peking University · Meituan · Tsinghua University +1
What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models
Oct 14, 2026Tengju Ru, Weitong Lian, Zecong Tang +6Multimodal Understanding Unified Multimodal Models
Zhejiang University · Shenzhen Loop Area Institute · The University of Hong Kong +1
Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence
Oct 13, 2026Thinker Team, Peng Gao, Jichao Jiao +8World Models for Robotics Embodied World Models
Ubtech
Omni-RS: Unifying Multi-Task Understanding and Generation in Remote Sensing
Oct 13, 2026Pan Yin, Shuai Yang, Yuanhao Liu +7Image Generation Unified Multimodal Models
Xi’an Jiaotong University
Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation
Oct 12, 2026Ziqiao Peng, Yutao Cui, Wang Lin +12Multimodal Generation Unified Multimodal Models
Renmin University of China · Tencent Hy · Tsinghua University
OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation
Oct 11, 2026Hebaixu Wang, Jing Zhang, Haonan Guo +3Multimodal Generation Unified Multimodal Models
Wuhan University & Zhongguancun Academy
Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation
Oct 10, 2026Zhengyang Luo, Xiaoyu Qiu, Zhiheng Tan +3Unified Multimodal Models Multimodal Generation
University of Michigan, Ann Arbor · The Ohio State University · Indiana University Indianapolis
TartanMatch: Towards Universal Dense Correspondence Across Modalities
Oct 10, 2026Hyeokjoon Kwon, Jiting Cai, Ruogu Li +7Cross-Modal Feature Matching Unified Multimodal Models
The authors are with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA · Kritan Bhandari is with Fisk University, Nashville, TN, USA
CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing
Oct 7, 2026William Chen, Prem Seetharaman, Ke Chen +10Instruction-Based Video Editing Unified Multimodal Models
Adobe Research · Carnegie Mellon University
MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation
Oct 7, 2026Kyeongmin Yeo, Minhyuk SungMultimodal Generation Unified Multimodal Models
KAIST
Visual Abstention in Unified Multimodal Models
Oct 6, 2026Chufan Shi, Cheng Yang, Tiannuo Yang +4Unified Multimodal Models Image Editing
University of Southern California · University of California San Diego

Oct 29, 2026J/K move · Enter open · S save

DiFuse: Fusing Native Diffusion Representations for Spatial Reasoning in Unified Multimodal Models

Ruixin Fan, Weining Wang, Jiahui Sun, Tailai Chen+1

Institute of Automation, Chinese Academy of Sciences

Multimodal Spatial Reasoning Unified Multimodal Models Multimodal Fusion Cross-Modal Representation Transfer

Open paper AskPDF arXiv

Unified multimodal models (UMMs) jointly support visual understanding and image generation, yet information flow between these two capabilities remains largely asymmetric: understanding representations routinely condition generation, whereas generative representations are rarely reused for multimodal reasoning. In this work, we investigate whether intermediate representations from a UMM's native diffusion branch can enhance spatial reasoning without compromising general multimodal understanding. We propose DiFuse, which selectively incorporates intermediate diffusion representations into the understanding pathway of UMMs. Through systematic layer-wise analysis, we identify which diffusion representations are most informative and where they can be integrated with minimal interference to semantic processing. Guided by these findings, DiFuse spatially aligns diffusion features with visual tokens and injects them into selected multimodal decoder layers via residual fusion. Extensive experiments on UniWorld-V1 show that, DiFuse consistently improves spatial reasoning over a matched SFT baseline while maintaining broadly comparable performance on general multimodal benchmarks. Further analyses show that these gains depend critically on the choice of diffusion representations, injection positions, and the preservation of spatial correspondence between diffusion features and visual tokens. Results on OmniGen2 further demonstrate the transferability of DiFuse across unified multimodal architectures. Overall, our findings show that native diffusion representations can provide complementary spatial cues for multimodal reasoning when selectively integrated in a structure-preserving manner.