Overview

Unified multimodal models study how one system can process and generate text, images, video, audio, and sometimes spatial or action data. Common approaches use shared tokenizers and autoregressive or diffusion-based backbones, often alongside modality-specific encoders, decoders, or experts; training may combine supervised instruction tuning, alignment, and reinforcement learning. Evaluation spans tasks such as visual question answering, text-to-image generation, video understanding, editing, and cross-modal consistency. A central challenge is that shared architecture does not guarantee useful transfer: understanding and generation can interfere or remain semantically misaligned, while efficiency, safety, and reliable evaluation across interacting modalities remain open concerns.

Latest papers

CardsList

314 papers26 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.

  1. SegBanana: Steering Unified Multimodal Models into Medical Segmenters

    Sep 28, 2026Xiaoye Liang, Ye Yan, Mingze Yin +5Unified Multimodal ModelsMedical Image Segmentation

  2. OPERA: A Unified Omnimodal Progressive Spatio-Temporal Reasoning Agent for Referring Video Segmentation

    Sep 27, 2026Jingchen Ni, Yuji Wang, Shannan Yan +3Unified Multimodal ModelsReferring Video Object Segmentation

  3. OneSign: Unifying Sign Language Understanding Tasks with One Model

    Sep 27, 2026Shiwei Gan, Yafeng Yin, Xiao Liu +3Sign Language RecognitionUnified Multimodal Models

  4. UniCache: Task- and Type-Aware KV Cache Compression for Unified Multimodal Models

    Sep 26, 2026Wanqi Yang, Yuexiao Ma, Mei Xie +2KV-Cache CompressionEfficient Multimodal Inference

  5. Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Tasks

    Sep 25, 2026Nilotpal Kakati, Daniel Murnane, Baran Hashemi +5Unified Multimodal ModelsHigh-Energy Physics

  6. TimeBraid: Unifying Time Series and Language for Understanding and Forecasting

    Sep 24, 2026Xinyue Wang, Jiacheng Pang, Kun Zhou +7Unified Multimodal ModelsMultimodal Pretraining

  7. GeoNLI - A Natural Language Interpreter for Satellite Imagery

    Sep 23, 2026Ashutosh Gandhe, Anupam Rawat, Geet Sethi +5Remote Sensing Visual GroundingRemote Sensing Visual Question Answering

  8. PartLLM: A Unified Multimodal Foundation for 3D Part Segmentation

    Sep 22, 2026Zhe Zhu, Yiheng Zhang, Peng Li +93D Part Segmentation3D VLMs

  9. Grounded Product Understanding in Livestream Videos

    Sep 17, 2026Xinyu Zhang, Junjie Chen, Jiawei Ge +4Temporal Video GroundingStreaming Video Understanding

  10. Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning

    Sep 17, 2026Haoqiang Kang, Yizhe Zhang, Nikki Lijing Kuang +3Unified Multimodal ModelsMultimodal Reasoning

  11. Efficient Unified Multimodal Understanding (EUMU): Winning Solution for the MUMU Track at the 8th LSVOS Challenge

    Sep 16, 2026Dayoung Kil, Seong-heum KimUnified Multimodal ModelsEfficient Multimodal Inference

  12. Can MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Model

    Sep 16, 2026Haoyu Zhao, Zihao Zhao, Tianyu Deng +10Multimodal ReasoningMultimodal Model Evaluation

  13. AURA: Unified Multimodal Framework for Conversational Music Editing

    Sep 13, 2026Quoc-Huy Trinh, Minh-Van Nguyen, Debesh JhaAudio EditingControllable Music Generation

  14. Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

    Sep 8, 2026Siting Li, Zhengyang Wang, Simon Shaolei Du +2Image TokenizationUnified Multimodal Models

  15. Multimodal Duplex Interaction Agent

    Sep 8, 2026Orantqing, Shengpeng Ji, Junlong Tong +19Unified Multimodal ModelsLLM Agent Orchestration

  16. Evolution of Multimodal Question Answering: From Modality-Adaptive Extraction to Unified Language Representation

    Sep 8, 2026Abdullah Al ShafiUnified Multimodal ModelsMultimodal QA

  17. Dreaming in Flow: Bidirectional Self-Refinement for Unified Multimodal Models

    Sep 8, 2026Ke Hao, Yuanzhi Liang, Tingxi Chen +5Multimodal GroundingUnified Multimodal Models

  18. Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition

    Sep 7, 2026Peng Xie, Amr AlanwarUnified Multimodal ModelsMultimodal Fusion

  19. SimpleDesign: A Joint Model for Protein Sequence and Structure Codesign

    Sep 3, 2026Jiarui Lu, Yuyang Wang, Yizhe Zhang +4Unified Multimodal ModelsGenerative Modeling

  20. Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

    Sep 1, 2026Penghao Wu, Haiwen Diao, Weichen Fan +3Unified Multimodal ModelsMultimodal Generation

  21. UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm

    Aug 25, 2026Zhehan Kan, Xinghua Jiang, Yubo Zhu +10Discrete Visual TokenizationUnified Multimodal Models

  22. WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report

    Aug 25, 2026Junjie Zhou, Ke Mei, Lei Li +3Multimodal EmbeddingUnified Multimodal Models

  23. Object-Uni: A Unified Model for Object-Centric Spatial Understanding and Controllable Generation

    Aug 24, 2026Mining Tan, Yinuo Wang, Ziqi Zhou +6Unified Multimodal ModelsConditional Image Generation

  24. When Does Visual Generation Help Visual Understanding in Unified Multimodal Models?

    Aug 23, 2026Yubo Zhu, Zhehan Kan, Jingyi Yang +6Unified Multimodal ModelsMultimodal Understanding