Overview

Unified multimodal models study how one system can process and generate text, images, video, audio, and sometimes spatial or action data. Common approaches use shared tokenizers and autoregressive or diffusion-based backbones, often alongside modality-specific encoders, decoders, or experts; training may combine supervised instruction tuning, alignment, and reinforcement learning. Evaluation spans tasks such as visual question answering, text-to-image generation, video understanding, editing, and cross-modal consistency. A central challenge is that shared architecture does not guarantee useful transfer: understanding and generation can interfere or remain semantically misaligned, while efficiency, safety, and reliable evaluation across interacting modalities remain open concerns.

Latest papers

CardsList

314 papers26 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.

  1. MetaEarth-MM: Unified Multimodal Remote Sensing Image Generation with Scene-centered Joint Modeling

    May 19, 2026Zhiping Yu, Chenyang Liu, Jinqi Cao +4Cross-Modal LearningUnified Multimodal Models

  2. Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models

    May 19, 2026Tobias Braun, Jonas Henry Grebe, Hossein Shakibania +2Multimodal Foundation ModelsBackdoor Attacks

  3. Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding

    May 18, 2026Paul Quinlan, Jeremy Levasseur, Qingguo Li +1Cross-Modal LearningMultivariate Time Series Forecasting

  4. Semantic Generative Tuning for Unified Multimodal Models

    May 18, 2026Songsong Yu, Yuxin Chen, Ying Shan +1Image SegmentationUnified Multimodal Models

  5. Lance: Unified Multimodal Modeling by Multi-Task Synergy

    May 18, 2026Fengyi Fu, Mengqi Huang, Shaojin Wu +10Unified Multimodal ModelsMulti-Task Learning

  6. WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens

    May 18, 2026Yiwei Guo, Shaobin Zhuang, Zhipeng Huang +4Image TokenizationUnified Multimodal Models

  7. LatentUMM: Dual Latent Alignment for Unified Multimodal Models

    May 18, 2026Yinyi Luo, Wenwen Wang, Hayes Bai +2Cross-Modal LearningUnified Multimodal Models

  8. Latent Action Control for Reasoning-Guided Unified Image Generation

    May 16, 2026Fuxiang Zhai, Sixiang Chen, Yingjin Li +4Unified Multimodal ModelsCompositional T2I Generation

  9. Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Models

    May 15, 2026Yujun Tong, Dongliang Chang, Zijin Yin +3Unified Multimodal ModelsMultimodal Reasoning

  10. Pelican-Unify 1.0: A Unified Embodied Intelligence Model for Understanding, Reasoning, Imagination and Action

    May 14, 2026Yi Zhang, Yinda Chen, Che Liu +26Action-Conditioned Video GenerationUnified Multimodal Models

  11. UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars

    May 14, 2026Xiaoyu Zhan, Xinyu Fu, Chenghao Yang +9Unified Multimodal ModelsHuman Motion Generation

  12. Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners

    May 14, 2026Qingyang Liu, Bingjie Gao, Canmiao Fu +9Unified Multimodal ModelsVisual Reasoning

  13. G2^2TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Models

    May 12, 2026Junxian Li, Kai Liu, Zizhong Ding +4Efficient Multimodal InferenceUnified Multimodal Models

  14. SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignment

    May 9, 2026Weiren Zhao, Yi Dong, Cheng ChenHealthcareUnified Multimodal Models

  15. SGC-RML: A reliable and interpretable longitudinal assessment for PD in real-world DNS

    May 8, 2026Wenbin Wei, Ruixiang Gao, Suyuan Yao +3Selective PredictionUnified Multimodal Models

  16. Steering Visual Generation in Unified Multimodal Models with Understanding Supervision

    May 7, 2026Zeyu Liu, Zanlin Ni, Yang Yue +5Unified Multimodal ModelsImage Generation

  17. Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

    May 5, 2026Lingyi Hong, Jinglun Li, Xinyu Zhou +6Cross-Modal LearningMultimodal Visual Object Tracking

  18. OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder

    May 2, 2026Detao Bai, Shimin Yao, Weixuan Chen +4Cross-Modal LearningVideo Understanding

  19. Valley3: Scaling Omni Foundation Models for E-commerce

    May 2, 2026Zeyu Chen, Guanghao Zhou, Min Yang +6Unified Multimodal ModelsMultimodal Pretraining

  20. UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors

    May 1, 2026Houyuan Chen, Hong Li, Xianghao Kong +8Video Diffusion ModelsUnified Multimodal Models