Unified Multimodal Models
Overview
Unified multimodal models study how one system can process and generate text, images, video, audio, and sometimes spatial or action data. Common approaches use shared tokenizers and autoregressive or diffusion-based backbones, often alongside modality-specific encoders, decoders, or experts; training may combine supervised instruction tuning, alignment, and reinforcement learning. Evaluation spans tasks such as visual question answering, text-to-image generation, video understanding, editing, and cross-modal consistency. A central challenge is that shared architecture does not guarantee useful transfer: understanding and generation can interfere or remain semantically misaligned, while efficiency, safety, and reliable evaluation across interacting modalities remain open concerns.
Latest papers
314 papers26 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.
Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
Semantic Generative Tuning for Unified Multimodal Models
Lance: Unified Multimodal Modeling by Multi-Task Synergy
WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens
LatentUMM: Dual Latent Alignment for Unified Multimodal Models
Latent Action Control for Reasoning-Guided Unified Image Generation
Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Models
Pelican-Unify 1.0: A Unified Embodied Intelligence Model for Understanding, Reasoning, Imagination and Action
UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars
Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners
Unified Pix Token And Word Token Generative Language Model
SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture
AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward
GTR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Models
Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generation
Images iN SEnTences (a.k.a, INSET), a unified generation model that seamlessly embeds images as native vocabulary within textual instructions. By positioning visual features directly at their corresponding semantic slots, INSET leverages the contextual locality of transformers for precise object binding, effectively treating images as dense, expressive language tokens. Furthermore, we introduce a scalable data engine that synthesizes 15M high-quality interleaved samples from standard image and video datasets, utilizing VLMs and LLMs to construct rich, long-horizon sequences. Evaluation results on InterleaveBench demonstrate that INSET significantly outperforms state-of-the-art methods in multi-image consistency and text alignment, with performance gaps widening as input complexity increases. Beyond standard generation, our approach inherently extends to multimodal image editing, integrating visual content as part of the instruction to facilitate highly expressive and creative visual manipulations.