Unified Multimodal Models
Overview
Unified multimodal models study how one system can process and generate text, images, video, audio, and sometimes spatial or action data. Common approaches use shared tokenizers and autoregressive or diffusion-based backbones, often alongside modality-specific encoders, decoders, or experts; training may combine supervised instruction tuning, alignment, and reinforcement learning. Evaluation spans tasks such as visual question answering, text-to-image generation, video understanding, editing, and cross-modal consistency. A central challenge is that shared architecture does not guarantee useful transfer: understanding and generation can interfere or remain semantically misaligned, while efficiency, safety, and reliable evaluation across interacting modalities remain open concerns.
Latest papers
314 papers26 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.
Next-Dense-Stride Prediction for Multimodal Autoregressive Visual Modeling
ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts
Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing
DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Models
ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation
Vision as Unified Multimodal Generation
Unified Audio Intelligence Without Regressing on Text Intelligence
Transferability Between Understanding and Generation in Unified Multimodal Models
Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process
WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modeling
WorldBagel, a unified VLAW framework built on BAGEL, a modern multimodal unified model, and use it to systematically investigate the role of unification in world modeling. Across multi-task robotic manipulation and cross-domain experiments, WorldBagel consistently outperforms task-specific alternatives and learns action representations that are more structured and semantically aligned with visual and linguistic context. Experiments on LIBERO, Language Table, and Franka show that unification is not only an architectural convenience, but also a key factor in learning effective VLAW models, leading to consistent empirical gains and deeper insights into multimodal world modeling. Code and model checkpoints will be released upon acceptance.